CN115063416B - Rail fastener state detection method and system - Google Patents
Rail fastener state detection method and system Download PDFInfo
- Publication number
- CN115063416B CN115063416B CN202210940958.3A CN202210940958A CN115063416B CN 115063416 B CN115063416 B CN 115063416B CN 202210940958 A CN202210940958 A CN 202210940958A CN 115063416 B CN115063416 B CN 115063416B
- Authority
- CN
- China
- Prior art keywords
- rail
- fastener
- area
- coordinate information
- state
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/0002—Inspection of images, e.g. flaw detection
- G06T7/0004—Industrial image inspection
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/60—Analysis of geometric attributes
- G06T7/62—Analysis of geometric attributes of area, perimeter, diameter or volume
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/60—Analysis of geometric attributes
- G06T7/66—Analysis of geometric attributes of image moments or centre of gravity
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/20—Image preprocessing
- G06V10/26—Segmentation of patterns in the image field; Cutting or merging of image elements to establish the pattern region, e.g. clustering-based techniques; Detection of occlusion
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/40—Extraction of image or video features
- G06V10/44—Local feature extraction by analysis of parts of the pattern, e.g. by detecting edges, contours, loops, corners, strokes or intersections; Connectivity analysis, e.g. of connected components
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
- G06V10/764—Arrangements for image or video recognition or understanding using pattern recognition or machine learning using classification, e.g. of video objects
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
- G06V10/77—Processing image or video features in feature spaces; using data integration or data reduction, e.g. principal component analysis [PCA] or independent component analysis [ICA] or self-organising maps [SOM]; Blind source separation
- G06V10/774—Generating sets of training patterns; Bootstrap methods, e.g. bagging or boosting
Landscapes
- Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Multimedia (AREA)
- Computing Systems (AREA)
- Artificial Intelligence (AREA)
- Health & Medical Sciences (AREA)
- Databases & Information Systems (AREA)
- Evolutionary Computation (AREA)
- General Health & Medical Sciences (AREA)
- Medical Informatics (AREA)
- Software Systems (AREA)
- Geometry (AREA)
- Quality & Reliability (AREA)
- Image Analysis (AREA)
Abstract
The application provides a method and a system for detecting the state of a rail fastener, wherein the method for detecting the state of the rail fastener comprises the steps of obtaining an image of the rail fastener to be identified, inputting the image into a first rail environment segmentation model and a second rail environment segmentation model, and obtaining central point coordinate information and contour point coordinate information of a nut, a rail and the fastener; obtaining a local image of the rail fastener to be identified according to the coordinate information of the central point and the contour point of the fastener, and inputting the local image into a fastener state detection model to obtain an initial prediction state; and obtaining state auxiliary judgment information according to the coordinate information of the central points and the contour points of the nuts, the rails and the fasteners, and further obtaining the state type of the rail fastener image to be identified. According to the detection method, the position and the form of the fastener are obtained through the coordinate information of the central point and the contour point of the nut, the rail and the fastener, the state auxiliary judgment information is obtained, and the high-precision detection of the state of the rail fastener is realized by combining the initial prediction state of the fastener.
Description
Technical Field
The application relates to the technical field of rail state detection, in particular to a method and a system for detecting the state of a rail fastener.
Background
The rail fastener is used as a fastener for connecting the rail and the soft sleeper, has the functions of fixing the rail on the soft sleeper, keeping the gauge and preventing the rail from moving longitudinally and transversely relative to the soft sleeper, and is a key component for ensuring the operation safety of the railway. Therefore, detection of the rail clip condition is important in railway automated inspection operations.
The existing rail fastener state detection method is to acquire rail fastener images in real time through an industrial camera carried by a computer-based rail inspection vehicle and transmit the rail fastener images to a server with computing capacity. And carrying out state identification on the acquired fastener image, and judging whether the fastener is abnormal or not.
Railway tracks are typically made up of clips, nuts, soft sleepers and tracks, so the acquired rail clip images also include clips, nuts, soft sleepers and tracks, see fig. 1. When carrying out state identification to the rail clip image, need earlier to gather the regional discernment of rail clip in the rail clip image, carry out rail clip state detection again. Rail clip conditions generally include normal, clip lean, clip defect, see fig. 2 a-2 d. Figure 2a shows the rail clip normal, figure 2b shows the rail clip inclined and figures 2c and 2d show the rail clip defect.
However, the focus of the current rail clip state detection method is on the recognition of the rail clip profile itself, which makes it difficult to capture the relatively small abnormal states such as the rail clip inclination shown in fig. 2b and the rail clip defect shown in fig. 2c, and thus the accuracy of rail clip state detection is not high.
Disclosure of Invention
In order to solve the problem that the rail clip inclination, defect and other tiny abnormal states are difficult to capture by the existing rail clip state detection method, and the accuracy of rail clip state detection is not high, the application provides a rail clip state detection method and system through the following aspects.
A first aspect of the present application provides a method of rail clip condition detection, the method comprising:
acquiring an image of a rail fastener to be identified;
inputting the rail fastener image to be identified into the first rail environment segmentation model and the second rail environment segmentation model;
the first rail environment segmentation model responds to an input rail fastener image to be identified and outputs first coordinate information, wherein the first coordinate information comprises central point coordinate information and contour point coordinate information of a nut, and central point coordinate information and contour point coordinate information of a rail;
the second rail environment segmentation model responds to the input rail fastener image to be identified and outputs second coordinate information, and the second coordinate information comprises center point coordinate information and contour point coordinate information of the rail fastener to be identified;
obtaining a local image of the rail fastener to be identified according to the second coordinate information;
inputting a local image of the rail fastener to be identified into a fastener state detection model;
the clip state detection model responds to the input local image of the rail clip to be identified and outputs the initial prediction state of the image of the rail clip to be identified;
obtaining auxiliary state judgment information of the rail fastener image to be identified according to the first coordinate information and the second coordinate information, wherein the auxiliary state judgment information is defect, inclination or normal;
when the state auxiliary judgment information is defect, outputting the state type of the rail fastener image to be identified as defect;
when the state auxiliary judgment information is inclination, outputting the state type of the image of the rail fastener to be identified as inclination;
and when the state auxiliary judgment information is normal, outputting the state type of the rail fastener image to be identified as an initial prediction state.
Optionally, the obtaining of the state auxiliary judgment information of the rail clip image to be identified according to the first coordinate information and the second coordinate information includes:
obtaining a first area, a second area, a third area and a fourth area according to the first coordinate information and the second coordinate information, wherein the first area is the area of one side area of the fastener, which is divided by taking the nut as the center, overlapped with the rail, the second area is the area of the other side area of the fastener, which is divided by taking the nut as the center, overlapped with the rail, the third area is the area of one side area of the fastener, which is divided by taking the nut as the center, and the fourth area is the area of the other side area of the fastener, which is divided by taking the nut as the center;
when the first ratio is larger than a first preset ratio, the state auxiliary judgment information is inclination, wherein the first ratio is the ratio of the difference value of the first area and the second area and the sum value of the first area and the second area;
when the second ratio is larger than a second preset ratio, the state auxiliary judgment information is defective, wherein the second ratio is the ratio of the difference value of the third area and the fourth area and the sum value of the third area and the fourth area;
when the coordinate information of the center point of the nut in the first coordinate information is empty and the coordinate information of the center point of the fastener in the second coordinate information is not empty, the state auxiliary judgment information is defective.
Optionally, the first rail environment segmentation model is trained according to the following method:
acquiring a plurality of rail images;
marking the nut outline and the rail outline in the rail image by adopting a pixel-level marking mode to obtain a first marked rail image;
performing a preprocessing process on the plurality of first labeled rail images to obtain a first training data set;
training a first environment segmentation model according to an example segmentation algorithm based on a polar coordinate system by using a first training data set;
the second rail environment segmentation model is trained according to the following method;
acquiring a plurality of rail images;
marking the outline of the fastener in the rail image by adopting a pixel-level marking mode to obtain a second marked rail image;
performing a preprocessing process on the plurality of second labeled rail images to obtain a second training data set;
and training a second environment segmentation model according to an example segmentation algorithm based on the polar coordinate system by using a second training data set.
Optionally, the number of the contour points of the nut obtained by the first rail environment segmentation model is 48, and the number of the contour points of the rail is 48;
the second rail environment segmentation model resulted in a clip with 48 contour points.
Optionally, the rail fastening status detection method further comprises: and sending the image of the rail fastener to be identified and the corresponding state category to a server.
A second aspect of the present application provides a rail clip condition detection system. The rail clip state detection system is used for realizing a rail clip state detection method provided by the first aspect of the application. The rail fastener state detection system comprises a rail fastener image acquisition module, a rail environment segmentation module and a fastener state identification module which are sequentially connected;
the image acquisition module of the rail fastener is used for acquiring an image of the rail fastener to be identified;
the rail environment segmentation module comprises a first rail environment segmentation unit, a second rail environment segmentation unit and a local image acquisition unit, wherein the first rail environment segmentation unit is used for responding to an input rail fastener image to be identified and outputting first coordinate information, and the first coordinate information comprises central point coordinate information and contour point coordinate information of a nut, and central point coordinate information and contour coordinate point information of a rail; the second rail environment segmentation model responds to the input image of the rail fastener to be identified and outputs second coordinate information, and the second coordinate information comprises the coordinate information of the central point and the coordinate information of the contour point of the rail fastener to be identified; the local image acquisition unit is used for acquiring a local image of the rail fastener to be identified according to the second coordinate information;
the fastener state identification module comprises an initial prediction unit, an auxiliary judgment unit and a result output unit; the initial prediction unit is used for responding to the input partial image of the rail fastener to be recognized and outputting an initial prediction state of the image of the rail fastener to be recognized; the auxiliary judgment unit is used for obtaining auxiliary judgment information of the state of the image of the rail fastener to be identified according to the first coordinate information and the second coordinate information, wherein the auxiliary judgment information of the state is defect, inclination or normal; the result output unit is used for determining the state type of the image of the rail fastener to be identified according to the state auxiliary judgment information and the initial prediction state, wherein when the state auxiliary judgment information is defect, the state type of the image of the rail fastener to be identified is defect; when the state auxiliary judgment information is inclination, outputting the state type of the image of the rail fastener to be identified as inclination; and when the state auxiliary judgment information is normal, outputting the state type of the rail fastener image to be identified as an initial prediction state.
Optionally, the auxiliary judging unit is configured to perform the following operations:
obtaining a first area, a second area, a third area and a fourth area according to the first coordinate information and the second coordinate information, wherein the first area is the area of one side area of the fastener divided by taking the nut as the center and overlapped with the rail, the second area is the area of the other side area of the fastener divided by taking the nut as the center and overlapped with the rail, the third area is the area of one side area of the fastener divided by taking the nut as the center, and the fourth area is the area of the other side area of the fastener divided by taking the nut as the center;
when the first ratio is larger than a first preset ratio, the state auxiliary judgment information is inclination, wherein the first ratio is the ratio of the difference value of the first area and the second area and the sum value of the first area and the second area;
when the second ratio is larger than a second preset ratio, the state auxiliary judgment information is defective, wherein the second ratio is the ratio of the difference value of the third area and the fourth area and the sum value of the third area and the fourth area;
when the coordinate information of the center point of the nut in the first coordinate information is empty and the coordinate information of the center point of the fastener in the second coordinate information is not empty, the state auxiliary judgment information is defective.
Optionally, the fastener state detection system further includes a storage module;
the storage module is used for storing the image of the rail fastener to be identified and the corresponding state category.
A third aspect of the present application provides a computer device comprising:
a memory for storing a computer program;
a processor configured to implement a method of detecting a state of a rail fastener provided by a first aspect of the present application when executing a computer program.
A fourth aspect of the present application provides a computer-readable storage medium. A computer readable storage medium has stored thereon a computer program which, when executed, implements a rail clip status detection method as provided in the first aspect of the present application.
The application provides a method and a system for detecting the state of a rail fastener, wherein the method for detecting the state of the rail fastener obtains an image of the rail fastener to be identified, and inputs the image into a first rail environment segmentation model and a second rail environment segmentation model to obtain central point coordinate information and contour point coordinate information of a nut, a rail and the fastener; obtaining a local image of the rail fastener to be identified according to the coordinate information of the central point and the contour point of the fastener, and inputting the local image into a fastener state detection model to obtain an initial prediction state; and obtaining state auxiliary judgment information according to the coordinate information of the central points and the contour points of the nuts, the rails and the fasteners, and further obtaining the state type of the rail fastener image to be identified. According to the detection method, the position and the form of the fastener are obtained through the coordinate information of the central point and the contour point of the nut, the rail and the fastener, the state auxiliary judgment information is obtained, and the high-precision detection of the state of the rail fastener is realized by combining the initial prediction state of the fastener.
Drawings
FIG. 1 is a schematic view of a rail clip image captured;
FIG. 2a is an exemplary schematic illustration of an image of a rail clip in a normal state;
FIG. 2b is an exemplary schematic illustration of an image of a rail clip in an inclined condition;
FIG. 2c is an exemplary schematic illustration of an image of a rail clip in a defective condition;
FIG. 2d is a schematic illustration of an example rail clip image in a defective condition of a rail clip;
fig. 3 is a schematic view of a working flow of a method for detecting a state of a rail clip according to an embodiment of the present application;
FIG. 4 is an exemplary illustration of an image of a rail clip labeled with pixel-level labeling;
fig. 5 is a schematic model structure diagram of a first environment segmentation model in a rail clip status detection method according to an embodiment of the present application;
fig. 6 is a schematic diagram illustrating first coordinate information output by the first environment segmentation model, and labeled nut and rail positions according to an embodiment of the present application;
fig. 7 is a schematic diagram of a fastener position marked according to second coordinate information output by the second environment segmentation model provided in the embodiment of the present application;
fig. 8 is a schematic diagram illustrating an example of obtaining the auxiliary judgment information in a method for detecting a state of a rail clip according to an embodiment of the present application;
fig. 9 is a schematic structural diagram of a rail clip status detection system according to an embodiment of the present disclosure.
Detailed Description
In order to solve the problem that the detection accuracy of the rail clip state is not high due to the fact that the existing rail clip state detection method is difficult to capture the tiny abnormal states such as the inclination and the defect of the rail clip, the application provides a rail clip state detection method and a rail clip state detection system through the following embodiments.
Referring to fig. 3, a method for detecting a state of a rail clip according to a first embodiment of the present application includes steps 11 to 19.
And step 11, acquiring an image of the rail fastener to be identified. In this embodiment, an industrial camera may be used to collect depth information from the rail, resulting in a 16 bit depth 1 lane rail clip depth map of 1280 x 1000. Reshape processes the depth map of the rail clip to obtain an image of the rail clip to be identified which is 1024 x 800. Wherein Reshape is a matrix transformation function in Matlab.
Where bit depth refers to the number of binary valued bits that a computer actually needs for each pixel when recording the color of a digital image. The normally visible color RGB image is usually 8 bit deep (2 ^16=256 colors) 3 channels, its data form is usually [ height of picture, width of picture, 3 (channels) ], and the value of each pixel in the matrix is between (0-255). The single-channel image with 16 bit depth (2 ^16=65535 colors) used in this embodiment is a gray-scale image (not in color), the data form is [ height of picture, width of picture, 1 (channel) ], and the value of each pixel in the matrix is between (0-65535).
And 12, inputting the image of the rail fastener to be identified into a first rail environment segmentation model and a second rail environment segmentation model.
The first environment segmentation model and the second environment segmentation model have the same model structure, and are different in that input training data sets are different and outputs are different, so that parameters of the first environment segmentation model and the second environment segmentation model obtained after training are different.
The present embodiment takes the first environment segmentation model as an example, and describes the structure and training process (steps 21 to 24) of the first environment segmentation model.
Step 21, a plurality of rail images are acquired. In this embodiment, an industrial camera is used to collect depth information of the rail, and a 16-bit depth 1 channel depth map of 1280 × 1000 is obtained.
And step 22, marking the nut outline and the rail outline in the rail image in a pixel-level marking mode to obtain a first marked rail image.
And after the depth map data is obtained, carrying out pixel-level labeling on the image rail. The pixel-level labeling refers to a labeling format for segmentation, and is an example of pixel-level labeling of the rail image, see fig. 4. In fig. 4, the profile of the rail, clip, and nut is divided into a plurality of points connected together. When the first marked rail image for training the first environment segmentation model is obtained, only the nut contour and the rail contour in the rail image need to be marked. Similarly, when the second labeled rail image of the second environment segmentation model is obtained, only the fastener contour in the rail image needs to be labeled in a pixel level.
And 23, performing a preprocessing process on the plurality of first labeled rail images to obtain a first training data set. The preprocessing process includes data augmentation, image size filling and the like, and finally, the sample data in the first training data set are marked images with sizes of 1024 × 800.
In this embodiment, the process of labeling first and then preprocessing is adopted, which facilitates the pipeline work during training and improves the training efficiency.
And 24, training a first environment segmentation model according to an example segmentation algorithm based on a polar coordinate system by using the first training data set.
In this embodiment, an example segmentation algorithm based on a polar coordinate system, which may also be referred to as a polar mask algorithm, is a one-stage example segmentation framework. Unlike the example segmentation algorithm based on the rectangular coordinate system, the example segmentation algorithm based on the polar coordinate system performs representation of the target contour by taking the position of the target center point and n rays from the center point as the standard. The representation mode can simplify the spatial overlapping relation among different instances in the railway track environment into the problem of contour point coordinate interpolation, and the operation speed is high. However, for a concave object such as a rail clip, if the number of rays used to represent the contour is small, the object cannot be represented accurately, so that the contour points are represented by 48 rays in the present embodiment, and only the distance between the contour points in the direction of the rays and the center point is required if the center point is known. The rail fastener, the nut and the rail are segmented through the polar coordinate system modeling outline, and the problem of instance segmentation is converted into the problem of instance center point classification and the problem of dense distance regression.
Further, the first environment segmentation model includes a backbone network, a Feature Pyramid Network (FPN), and a header network.
Referring to fig. 5, in the present embodiment, the backbone Network is a first Convolutional Neural Network (CNN), and is configured to perform CNN feature extraction with a downsampling ratio of 8, 16, and 32 on 1024 × 800 pictures to obtain initial feature maps C1, C2, and C3 with sizes of 128 × 100 × 256, 64 × 50, 512, and 32 × 25 × 1024, respectively, and convolve the initial feature maps by 1 × 1 to obtain first feature maps C1, C2, and C3 with different sizes but the same number of channels (256), where C1 is size of 128 × 100 × 256, C2 is size of 64 × 50 256, and C3 is size of 32 × 25.
The FPN network enables the feature maps with different sizes to share information, namely, the backbone network obtains the first feature map, up-down sampling is carried out on the first feature map by taking 2 as a sampling ratio, and concat connection is carried out on the first feature map and the original first feature map, so that five second feature maps P1-P5 used for final prediction are obtained. Wherein, C3 directly migrated from P3 (32 × 25 × 256), P2 (64 × 50 × 256) was obtained by adding up C2 and P3 upsampling, and P1 (128 × 100 × 256) was obtained by adding up C1 and Feature Map (Feature Map) obtained by P2 upsampling; p4 (16 × 13 × 256) is obtained by deconvoluting P2 with step size 2, and P5 (8 × 7 × 256) is obtained by deconvoluting P3 with step size 2.
The head network is a second convolutional neural network and comprises five detection heads, and the second characteristic diagrams P1-P5 are respectively input into the corresponding detection heads to obtain prediction results of corresponding sizes. Each detection head outputs three prediction results, namely prediction (H × W × 3) of pixel point types, prediction (H × W × 1) of whether the pixel points are central points or not, and regression prediction (H × W × 48) of distance edge points, wherein H is the height of the second feature map, and W is the width of the second feature map. The output of the final head network is the prediction of the pixel point class (1024 × 800 × 3), the prediction of whether it is the center point (1024 × 800 × 1), and the regression prediction of the distance edge points (1024 × 800 × 48). The pixel point types output by the first neural network are respectively a background, a nut and a rail.
Since the polar coordinate system representation example cannot be calculated by using a conventional IOU (interaction over Union) calculation method, this embodiment adopts an IOU loss calculation method suitable for polar coordinates, and the calculation method is that
Wherein d is the predicted contour point and true in the ith ray directionDistance of real-valued contour points. To be provided with
As the cross-over ratio of the prediction to the real mask,
、
the minimum value and the maximum value of the distance between the real contour and the predicted contour between two adjacent ray directions are obtained.
The structure of the second environment segmentation model and the training process can be obtained by those skilled in the art with reference to steps 21 to 24. The difference is that when pixel-level labeling is performed, the fastener outline is labeled in the second training data set; and the output of the second environment segmentation model is the prediction of the pixel point category (1024 × 800 × 2), whether it is the prediction of the center point (1024 × 800 × 1), and the regression prediction of the distance edge points (1024 × 800 × 48), wherein the pixel point category is the background and the fastener.
In the embodiment, two rail environment segmentation models are adopted to respectively segment the rail and the nut and the fastener, so that the negative influence on the segmentation caused by the overlapping problem of different types of parts in the rail environment can be solved.
The first rail environment segmentation model and the second rail environment segmentation model are both example segmentation models based on a full convolution neural network. The contour modeling based on the rectangular coordinate system is improved into the contour modeling based on the polar coordinate system, the example segmentation problem is simplified into a general form of full convolution, and high-precision and quick example segmentation can be realized.
And step 13, the first rail environment segmentation model outputs first coordinate information in response to the input rail fastener image to be identified, wherein the first coordinate information comprises central point coordinate information and contour point coordinate information of the nut, and central point coordinate information and contour point coordinate information of the rail. Referring to fig. 6, the obtained nut position and rail position example are labeled according to the first coordinate information output by the first rail environment segmentation model.
And 14, responding to the input image of the rail fastener to be identified by the second rail environment segmentation model, and outputting second coordinate information, wherein the second coordinate information comprises the coordinate information of the center point and the coordinate information of the contour point of the rail fastener to be identified. Referring to fig. 7, an example of a location of a clip is labeled according to the second coordinate information output by the second rail environment segmentation model.
And step 15, obtaining a local image of the rail fastener to be identified according to the second coordinate information. In this embodiment, the position of the rail clip is determined and intercepted according to the clip center point coordinate information in the second coordinate information and the distance between the center point and the contour point farthest from the center point, so as to obtain the local image of the rail clip to be identified.
Because the subsequent clip state detection model requires the size of the input image, in one implementation, the captured image is subjected to reshape processing in step 15 to obtain the local image of the rail clip to be identified for input into the clip state detection model.
And step 16, inputting the local image of the fastener to be identified to a fastener state detection model. In this embodiment, the fastener state detection model employs a resnet18 network. The resnet18 network in this embodiment requires the size of the input image to be 224 × 224, so that when the reshape process is performed in step 15, the local image of the rail clip to be identified is obtained according to the requirement of 224 × 224.
And step 17, responding to the input local image of the rail fastener to be identified by the fastener state detection model, and outputting an initial prediction state of the image of the rail fastener to be identified. Wherein the initial prediction states include normal, lean, and defective.
And 18, obtaining auxiliary state judgment information of the rail fastener image to be identified according to the first coordinate information and the second coordinate information, wherein the auxiliary state judgment information is defect, inclination or normal.
In this embodiment, referring to fig. 8, a first area, a second area, a third area and a fourth area are obtained according to the first coordinate information and the second coordinate information, wherein the first area is an area a where one side region of the clip divided by using the nut as the center overlaps with the rail, the second area is an area b where the other side region of the clip divided by using the nut as the center overlaps with the rail, the third area is an area c of one side region of the clip divided by using the nut as the center, and the fourth area is an area d of the other side region of the clip divided by using the nut as the center.
And when the first ratio is larger than a first preset ratio, the state auxiliary judgment information is inclination, wherein the first ratio is the ratio of the difference value of the first area and the second area and the sum value of the first area and the second area.
And when a second ratio is larger than a second preset ratio, the state auxiliary judgment information is defective, wherein the second ratio is the ratio of the difference value of the third area and the fourth area to the sum value of the third area and the fourth area.
And when the coordinate information of the center point of the nut in the first coordinate information is empty and the coordinate information of the center point of the fastener in the second coordinate information is not empty, the state auxiliary judgment information is defective.
Illustratively, when the difference (| a-b |) between the areas on both sides of the rail clip divided by taking the nut as the center and the rail overlapping area is larger than one third of the sum (a + b) of the areas on both sides and the rail overlapping area, the determination of the inclined state is assisted. When the fastener lacks the nut or the difference (| c-d |) of the two side areas with the nut(s) as the center is larger than one tenth of the two side areas (c + d), the auxiliary judgment is that the fastener is in a defect state.
It should be noted that, the first preset value in this example is one third, which can better reflect that the difference between the overlapping areas of the rail and the areas on two sides of the rail clip divided by using the nut as the center is too large, and identify the clip inclination; the second preset value is one tenth, so that the area difference of the rail fasteners taking the nuts as the two sides of the center can be better reflected, and the defect of the fastener can be identified. The specific values of the first preset value and the second preset value are not specifically limited in the present application. For example, the first preset value may also be set to 0.35, or 0.37, etc.; the second preset value may also be set to 0.12,0.9, etc.
It should be noted that, in the present embodiment, the above numerical examples and the examples in the drawings both adopt WJ-type rail fasteners, and other types of rail fasteners may also adopt the method provided by the present embodiment to realize high-precision state detection, and the specific first preset value and the second preset value may be set according to specific situations.
In this embodiment, when the rail clip is present but the nut is not present, it is also determined that the rail clip is defective. The rail fastener defect state judging method comprises the following steps of judging whether the rail fastener is in a defect state or not according to the rail fastener center point coordinate information, wherein the rail fastener center point coordinate information exists, but the nut center point coordinate information does not exist, and judging that the rail fastener is in the defect state in an auxiliary manner.
In one implementation, the state auxiliary judgment information can be used as weight information of the initial state prediction value, and the initial prediction state is subjected to nonlinear mapping so as to achieve the purpose of auxiliary rail fastener state classification. The specific implementation process can refer to the following formula:
wherein, the first and the second end of the pipe are connected with each other,
showing rail clip to be identifiedThe initial state prediction value of the piece image,
representing a predicted value of the status category of the image of the rail clip to be identified,xfor indicating the normal condition of the rail clip,yfor indicating the state of inclination of the rail clip,zfor indicating a rail clip defect condition.
In one implementation, the method for detecting the state of the rail fastener further includes sending the image of the rail fastener to be identified and the state category obtained by processing and identifying to a server. The server, upon receiving the relevant information, can retain the information, issue an alarm according to status category, collect the rail clip images to facilitate updating the training data set, and so forth.
The embodiment provides a rail fastener state detection method. The rail fastener state detection method obtains a rail fastener image to be identified, and inputs the rail fastener image to the first rail environment segmentation model and the second rail environment segmentation model to obtain central point coordinate information and contour point coordinate information of nuts, rails and fasteners; obtaining a local image of the rail fastener to be identified according to the coordinate information of the central point and the contour point of the fastener, and inputting the local image into the fastener state detection model to obtain an initial prediction state; and obtaining state auxiliary judgment information according to the coordinate information of the central points and the contour points of the nuts, the rails and the fasteners, and further obtaining the state type of the rail fastener image to be identified. According to the method, the position and the form of the fastener are obtained through the coordinate information of the central point and the contour point of the nut, the rail and the fastener, state auxiliary judgment information is obtained, and the method is used for realizing high-precision detection of the state of the rail fastener by combining the initial prediction state of the fastener.
In accordance with a second embodiment of the present invention, a rail clip status detection system is provided. Referring to fig. 9, the rail clip state detection system is used for implementing the rail clip detection method according to the first embodiment of the present application, and includes a rail clip image acquisition module, a rail environment segmentation module, and a clip state identification module, which are connected in sequence.
The rail fastener image acquisition module is used for acquiring an image of a rail fastener to be identified.
The rail environment segmentation module comprises a first rail environment segmentation unit, a second rail environment segmentation unit and a local image acquisition unit, wherein the first rail environment segmentation unit is used for responding to the input rail fastener image to be identified and outputting first coordinate information, and the first coordinate information comprises central point coordinate information and contour point coordinate information of a nut, central point coordinate information and contour point coordinate information of a rail; the second rail environment segmentation model responds to the input image of the rail fastener to be identified and outputs second coordinate information, and the second coordinate information comprises the coordinate information of the central point and the coordinate information of the contour point of the rail fastener to be identified; the local image acquisition unit is used for obtaining a local image of the rail fastener to be identified according to the second coordinate information.
The fastener state identification module comprises an initial prediction unit, an auxiliary judgment unit and a result output unit; the initial prediction unit is used for responding to the input local image of the rail fastener to be identified and outputting an initial prediction state of the image of the rail fastener to be identified; the auxiliary judgment unit is used for obtaining auxiliary judgment information of the state of the image of the rail fastener to be identified according to the first coordinate information and the second coordinate information, wherein the auxiliary judgment information of the state is defective, inclined or normal; the result output unit is used for determining the state type of the image of the rail fastener to be identified according to the state auxiliary judgment information and the initial prediction state, wherein when the state auxiliary judgment information is defect, the state type of the image of the rail fastener to be identified is defect; when the state auxiliary judgment information is inclination, outputting the state type of the image of the rail fastener to be identified as inclination; and when the state auxiliary judgment information is normal, outputting the state type of the rail fastener image to be identified as the initial prediction state.
In one implementation, the auxiliary determination unit is configured to perform the following operations: obtaining a first area, a second area, a third area and a fourth area according to the first coordinate information and the second coordinate information, wherein the first area is an area where one side area of the fastener divided by taking the nut as the center is overlapped with the rail, the second area is an area where the other side area of the fastener divided by taking the nut as the center is overlapped with the rail, the third area is an area of one side area of the fastener divided by taking the nut as the center, and the fourth area is an area of the other side area of the fastener divided by taking the nut as the center; when a first ratio is larger than a first preset ratio, the state auxiliary judgment information is inclination, wherein the first ratio is the ratio of the difference value of the first area and the second area and the sum value of the first area and the second area; when a second ratio is larger than a second preset ratio, the state auxiliary judgment information is defective, wherein the second ratio is the ratio of the difference value of the third area and the fourth area to the sum value of the third area and the fourth area; and when the coordinate information of the center point of the nut in the first coordinate information is empty and the coordinate information of the center point of the fastener in the second coordinate information is not empty, the state auxiliary judgment information is defective.
In one implementation, the fastener status detection system further comprises a storage module; the storage module is used for storing the image of the rail fastener to be identified and the corresponding state category. Illustratively, the storage module is a server.
The operation and effect of the system when applying the method can be referred to the description of the embodiment of the method, and will not be described herein again.
A third embodiment of the present application provides a computer apparatus, including:
a memory for storing a computer program;
a processor for implementing a method of detecting a condition of a rail clip according to the first embodiment of the present application when the computer program is executed.
A fourth embodiment of the present application provides a computer readable storage medium, wherein the computer readable storage medium stores a computer program, and the computer program is configured to implement a method for detecting a rail clip status according to the first embodiment of the present application when the computer program is executed.
Other embodiments of the present application will be apparent to those skilled in the art from consideration of the specification and practice of the application disclosed herein. This application is intended to cover any variations, uses, or adaptations of the invention following, in general, the principles of the application and including such departures from the present disclosure as come within known or customary practice in the art to which the invention pertains. It is intended that the specification and examples be considered as exemplary only, with a true scope and spirit of the application being indicated by the following claims.
Similar parts in all embodiments in the specification are referred to each other.
Claims (8)
1. A method of detecting a condition of a rail clip, comprising:
acquiring an image of a rail fastener to be identified;
inputting the rail fastener image to be identified into a first rail environment segmentation model and a second rail environment segmentation model;
the first rail environment segmentation model responds to the input rail fastener image to be identified and outputs first coordinate information, wherein the first coordinate information comprises central point coordinate information and contour point coordinate information of a nut, and central point coordinate information and contour point coordinate information of a rail;
the second rail environment segmentation model responds to the input image of the rail fastener to be identified and outputs second coordinate information, and the second coordinate information comprises the coordinate information of the central point and the coordinate information of the contour point of the rail fastener to be identified;
obtaining a local image of the rail fastener to be identified according to the second coordinate information;
inputting the local image of the rail fastener to be identified into a fastener state detection model;
the clip state detection model is used for responding to the input local image of the rail clip to be identified and outputting an initial prediction state of the image of the rail clip to be identified;
obtaining state auxiliary judgment information of the rail fastener image to be identified according to the first coordinate information and the second coordinate information, wherein the state auxiliary judgment information is defect, inclination or normal;
when the state auxiliary judgment information is defective, outputting the state type of the rail fastener image to be identified as defective;
when the state auxiliary judgment information is inclination, outputting the state type of the image of the rail fastener to be identified as inclination;
when the state auxiliary judgment information is normal, outputting the state type of the rail fastener image to be identified as the initial prediction state;
the method for obtaining the state auxiliary judgment information of the rail fastener image to be identified according to the first coordinate information and the second coordinate information comprises the following steps:
obtaining a first area, a second area, a third area and a fourth area according to the first coordinate information and the second coordinate information, wherein the first area is an area where one side area of the fastener divided by taking the nut as the center is overlapped with the rail, the second area is an area where the other side area of the fastener divided by taking the nut as the center is overlapped with the rail, the third area is an area of one side area of the fastener divided by taking the nut as the center, and the fourth area is an area of the other side area of the fastener divided by taking the nut as the center;
when a first ratio is greater than a first preset ratio, the state auxiliary judgment information is inclination, wherein the first ratio is a ratio of a difference value between the first area and the second area and a sum value of the first area and the second area;
when a second ratio is larger than a second preset ratio, the state auxiliary judgment information is defective, wherein the second ratio is the ratio of the difference value of the third area and the fourth area to the sum value of the third area and the fourth area;
and when the coordinate information of the center point of the nut in the first coordinate information is empty and the coordinate information of the center point of the fastener in the second coordinate information is not empty, the state auxiliary judgment information is defective.
2. A rail clip condition detection method according to claim 1, wherein said first rail environment segmentation model is trained in accordance with the following method:
acquiring a plurality of rail images;
marking the nut outline and the rail outline in the rail image in a pixel-level marking mode to obtain a first marked rail image;
performing a preprocessing process on the plurality of first labeled rail images to obtain a first training data set;
training a first environment segmentation model according to an example segmentation algorithm based on a polar coordinate system by using the first training data set;
the second rail environment segmentation model is trained according to the following method;
acquiring a plurality of rail images;
marking the fastener outline in the rail image by adopting a pixel-level marking mode to obtain a second marked rail image;
performing a preprocessing process on the plurality of second labeled rail images to obtain a second training data set;
and training a second environment segmentation model according to an example segmentation algorithm based on a polar coordinate system by using the second training data set.
3. The method according to claim 2, wherein the number of contour points of the nut obtained by the first rail environment segmentation model is 48, and the number of contour points of the rail is 48;
the number of contour points of the fasteners obtained by the second rail environment segmentation model is 48.
4. The method of detecting the status of a rail clip according to claim 1, further comprising: and sending the image of the rail fastener to be identified and the corresponding state category to a server.
5. A rail clip state detection system for implementing a rail clip state detection method as claimed in any one of claims 1 to 4, the rail clip state detection system comprising a rail clip image acquisition module, a rail environment segmentation module, and a clip state identification module connected in series;
the image acquisition module of the rail fastener is used for acquiring an image of the rail fastener to be identified;
the rail environment segmentation module comprises a first rail environment segmentation unit, a second rail environment segmentation unit and a local image acquisition unit, wherein the first rail environment segmentation unit is used for responding to the input image of the rail fastener to be identified and outputting first coordinate information, and the first coordinate information comprises central point coordinate information and contour point coordinate information of a nut, central point coordinate information and contour point coordinate information of a rail; the second rail environment segmentation model responds to the input image of the rail fastener to be identified and outputs second coordinate information, and the second coordinate information comprises the coordinate information of the central point and the coordinate information of the contour point of the rail fastener to be identified; the local image acquisition unit is used for acquiring a local image of the rail fastener to be identified according to the second coordinate information;
the fastener state identification module comprises an initial prediction unit, an auxiliary judgment unit and a result output unit; the initial prediction unit is used for responding to the input local image of the rail fastener to be identified and outputting an initial prediction state of the image of the rail fastener to be identified; the auxiliary judgment unit is used for obtaining auxiliary judgment information of the state of the image of the rail fastener to be identified according to the first coordinate information and the second coordinate information, wherein the auxiliary judgment information of the state is defective, inclined or normal; the result output unit is used for determining the state type of the image of the rail fastener to be identified according to the state auxiliary judgment information and the initial prediction state, wherein when the state auxiliary judgment information is defect, the state type of the image of the rail fastener to be identified is defect; when the state auxiliary judgment information is inclination, outputting the state type of the rail fastener image to be identified as inclination; when the state auxiliary judgment information is normal, outputting the state type of the rail fastener image to be identified as the initial prediction state;
wherein the auxiliary judgment unit is configured to perform the following operations:
obtaining a first area, a second area, a third area and a fourth area according to the first coordinate information and the second coordinate information, wherein the first area is an area where one side area of the fastener divided by taking the nut as the center is overlapped with the rail, the second area is an area where the other side area of the fastener divided by taking the nut as the center is overlapped with the rail, the third area is an area of one side area of the fastener divided by taking the nut as the center, and the fourth area is an area of the other side area of the fastener divided by taking the nut as the center;
when a first ratio is greater than a first preset ratio, the state auxiliary judgment information is inclination, wherein the first ratio is a ratio of a difference value between the first area and the second area and a sum value of the first area and the second area;
when a second ratio is larger than a second preset ratio, the state auxiliary judgment information is defective, wherein the second ratio is the ratio of the difference value of the third area and the fourth area to the sum value of the third area and the fourth area;
and when the coordinate information of the center point of the nut in the first coordinate information is empty and the coordinate information of the center point of the fastener in the second coordinate information is not empty, the state auxiliary judgment information is defective.
6. The rail clip condition detection system of claim 5, further comprising a memory module;
the storage module is used for storing the image of the rail fastener to be identified and the corresponding state category.
7. A computer device, comprising:
a memory for storing a computer program;
a processor for implementing a method of rail clip condition detection as claimed in any one of claims 1 to 4 when said computer program is executed.
8. A computer readable storage medium having stored thereon a computer program which when processed and executed carries out a method of detecting the condition of a rail clip according to any one of claims 1 to 4.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210940958.3A CN115063416B (en) | 2022-08-08 | 2022-08-08 | Rail fastener state detection method and system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210940958.3A CN115063416B (en) | 2022-08-08 | 2022-08-08 | Rail fastener state detection method and system |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN115063416A CN115063416A (en) | 2022-09-16 |
| CN115063416B true CN115063416B (en) | 2022-11-04 |
Family
ID=83208205
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202210940958.3A Active CN115063416B (en) | 2022-08-08 | 2022-08-08 | Rail fastener state detection method and system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN115063416B (en) |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107576666A (en) * | 2017-10-13 | 2018-01-12 | 成都精工华耀机械制造有限公司 | A kind of double light spectrum image-forming rails and fastener method for detecting abnormality |
| CN109916912A (en) * | 2019-04-01 | 2019-06-21 | 北京通运腾达科技发展有限公司 | A kind of railway rail clip Defect inspection method and system |
| CN111391881A (en) * | 2020-04-03 | 2020-07-10 | 中国铁道科学研究院集团有限公司基础设施检测研究所 | Method and device for detecting fastening state of steel rail fastener |
-
2022
- 2022-08-08 CN CN202210940958.3A patent/CN115063416B/en active Active
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107576666A (en) * | 2017-10-13 | 2018-01-12 | 成都精工华耀机械制造有限公司 | A kind of double light spectrum image-forming rails and fastener method for detecting abnormality |
| CN109916912A (en) * | 2019-04-01 | 2019-06-21 | 北京通运腾达科技发展有限公司 | A kind of railway rail clip Defect inspection method and system |
| CN111391881A (en) * | 2020-04-03 | 2020-07-10 | 中国铁道科学研究院集团有限公司基础设施检测研究所 | Method and device for detecting fastening state of steel rail fastener |
Non-Patent Citations (1)
| Title |
|---|
| 基于轨道图像位置距离特征的扣件状态识别;冯婷立等;《智能计算机与应用》;20200930;第10卷(第9期);第178-181页 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN115063416A (en) | 2022-09-16 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN112419299B (en) | Bolt missing detection method, device, equipment and storage medium | |
| CN113658132B (en) | Computer vision-based structural part weld joint detection method | |
| CN110264444B (en) | Damage detection method and device based on weak segmentation | |
| CN111178236A (en) | Parking space detection method based on deep learning | |
| US20080232715A1 (en) | Image processing apparatus | |
| CN112419297A (en) | Bolt looseness detection method, device, equipment and storage medium | |
| CN111191611B (en) | Traffic sign label identification method based on deep learning | |
| CN112419298B (en) | Bolt node plate rust detection method, device, equipment and storage medium | |
| CN112330593A (en) | Building surface crack detection method based on deep learning network | |
| CN109829458B (en) | Method for automatically generating log file for recording system operation behavior in real time | |
| CN115063618B (en) | Defect positioning method, system, equipment and medium based on template matching | |
| CN108694716B (en) | Workpiece detection method, model training method and equipment | |
| CN111696072A (en) | Color image line detection method, color image line detection device, electronic device, and storage medium | |
| CN114820679B (en) | Image labeling method and device electronic device and storage medium | |
| CN113989794A (en) | License plate detection and recognition method | |
| CN106875430A (en) | Single movement target method for tracing and device based on solid form under dynamic background | |
| CN113781537A (en) | Track elastic strip fastener defect identification method and device and computer equipment | |
| CN114399518A (en) | Method for monitoring tightness of vehicle bottom bolt and electronic equipment | |
| CN115063416B (en) | Rail fastener state detection method and system | |
| CN112053407B (en) | Automatic lane line detection method based on AI technology in traffic law enforcement image | |
| CN114331961A (en) | Method for defect detection of an object | |
| CN115631197B (en) | Image processing method, device, medium, equipment and system | |
| CN112434582A (en) | Lane line color identification method and system, electronic device and storage medium | |
| CN116403200A (en) | License plate real-time identification system based on hardware acceleration | |
| CN115291141A (en) | Fault diagnosis method and system for cable and accessories thereof |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |