CN101706274B - Device for automatically detecting nut loss of rail fastener system - Google Patents

Abstract

An automatic detection device for missing nuts in a rail fastener system, comprising a vision acquisition device, a GPS locator, a lighting device and a computer system. The lighting fixture increases nut lighting. The computer system is equipped with a video data storage software module, a display software module and a missing nut identification and positioning software module, and its execution steps are as follows: the video data storage software module receives and stores the location information of the device obtained by the GPS locator, and visual collection The device obtains continuous video images containing the nuts of the fastener system, and records the corresponding time values of each video image and position data acquisition; the missing nut identification and positioning software module preprocesses the obtained video images, cuts sub-images, extracts features, Feature classification and position calculation realize the identification and location of missing nuts; the display software module displays and prints images and position information of missing nuts. When the invention works, it moves with the train to realize the automatic detection and positioning of the missing nuts of the rail fastener system.

Description

Automatic detection device for missing nuts in rail fastener system

technical field

The patent of the present invention relates to an automatic detection device for missing nuts in a rail fastener system, in particular to an automatic detection device for missing nuts in a rail fastener system based on computer vision.

Background technique

Line maintenance plays an important role in the normal work and safe operation of subway and railway systems. The railway and urban rail transit departments of all countries in the world attach great importance to line maintenance work, and have formulated detailed plans and strict regulations. One of the important tasks of line maintenance is to check whether the nuts of the fastener system are missing and whether the fastening parts are loose. Fastener system The function of nuts, spikes and fasteners is to achieve a tight and effective connection between rails and sleepers, and they are important accessories for railway and subway lines. In practice, due to insufficient installation and maintenance, vibration of the train running, and human theft, the nuts of the fastener system on the line may be missing, which poses hidden dangers to the safe running of the train.

For a long time, line maintenance has mainly relied on the manual inspection and visual inspection of the maintenance team to determine whether there are line abnormalities such as damaged road bolsters, worn rails, and missing fastener system nuts. Some foreign railway companies also rely on experienced maintenance personnel to watch the road surface video to judge whether there are road ties, rails and fastener system nuts that need to be repaired. The video is recorded by the camera installed on the moving train. This method of manual inspection of line rails, road sleepers and fastening system nuts requires a long time, high labor intensity, low efficiency, and high missed detection rate.

Computer vision uses various imaging systems instead of visual organs as input sensitive means, and the computer replaces the brain to complete processing and interpretation. The ultimate goal of computer vision is to enable computers to observe and understand the world through vision like humans, and to have the ability to judge and recognize autonomously. In the past two or three decades, computer vision technology has gradually moved from experimental research to practical application. At present, computer vision systems are widely used in the field of detection. The detection method based on computer vision technology has the advantages of high efficiency, low cost and good reliability, and is an important development direction of nondestructive testing. The use of computer vision recognition methods to replace manual visual inspection is a new trend in railway and subway line maintenance.

After searching the literature of the existing technology, it is found that there is no utility model or invention patent for the automatic detection device for the missing nut of the rail fastener system, and only a small number of foreign scholars have carried out the automatic detection technology for the missing nut of the fastener system based on computer vision. preliminary research. Mazzeo et al. (Mazzeo et al., Pattern Recognition Letters, 2004, 25: 669-677; Mazzeo et al., Proceedings of the 2004 IEEE Intelligent Transportation Systems Conference, 417-422; Mazzeo et al., Proceedings of the 2003 IEEE Intelligent Transportation Systems Conference on Symposium on Symposium Intelligent Control, 636-641) studied the use of principal component analysis, wavelet transform and neural network technology to process video images of track lines to automatically detect the absence of nuts in fastener systems. In recent literature (Ruvo et al., Proceedings of the7th international Workshop on Computer Architecture for Machine Perception, 2005, 1-6; Marino et al., IEEE Transactions on Systems, Man, and Cybernetics, Part C, 2007), research The staff introduced the nut missing detection device of the fastener system developed based on FPGA, using the aforementioned neural network and other algorithms, and the experimental results show that it has a certain real-time performance. The above-mentioned researchers used computer vision to detect missing nuts. Although they have achieved preliminary results, they have the following shortcomings:

(1) The method of feature classification using neural network tends to cause the classification and identification results to fall into a local optimum, and generally requires large samples, and the training results are also unstable, so that the applicability and working reliability of the visual recognition system need to be improved;

(2) The position of the missing nut is calculated based on the number of nuts in the fastener system in the video image and the distance between two adjacent nuts. This calculation method results in a large error in the location of the nut.

Contents of the invention

The purpose of the present invention is to overcome the deficiencies in the prior art, to provide an automatic detection device for the lack of nuts in the rail fastener system, so that the system structure is simple, the detection and positioning are accurate, and it is used to replace or assist the nuts of the rail fastener system. Manual inspection.

In order to achieve the stated purpose of the present invention, the present invention provides an automatic detection device for missing nuts in a rail fastener system. The automatic detection device includes: a visual acquisition device, a GPS locator, a lighting device and a computer system. The missing nut automatic detection device is installed at the bottom of the train. When the train is moving, it collects and stores the video images of the nuts of the fastener system on both sides of the rail, and the identification and positioning of the missing nuts are completed online or offline by the computer system.

The vision collection device of the present invention realizes the collection of continuous video images including nuts of each fastener system, and transmits the obtained video images to a computer system.

The lighting unit was used to increase the illumination of the fastener system nuts, thereby improving the quality of the video images obtained.

The GPS locator determines the geographic location of the visual acquisition device at each moment, and transmits the geographic location information corresponding to each moment to the computer system.

The computer system is provided with a video data storage software module, a display software module and a missing nut identification and positioning software module, and the following steps are performed:

a) The video data storage software module receives and stores the continuous video containing the nut of the fastener system transmitted by the visual acquisition device, and records the corresponding time value of each video image;

b) The video data storage software module receives and stores the geographic location information of the device at each moment transmitted by the GPS locator, and records the corresponding time value of each geographic location information;

c) The missing nut identification and positioning software module performs image preprocessing, sub-image cropping, feature extraction and feature classification on the video image, thereby completing the identification of missing nuts, and recording the acquisition time values of the images corresponding to these missing nuts;

d) The missing nut identification and positioning software module calculates the geographic location of the missing nut according to the acquisition time value of the image corresponding to the missing nut and the corresponding geographic location information at each moment, so as to realize the location of the missing nut;

e) The display software module displays the collected video as required, and summarizes, displays and prints the image and geographic location information of the missing nut.

The missing nut identification and positioning software module includes five functional submodules such as image preprocessing submodule, subimage cutting submodule, feature extraction submodule, feature classification submodule and missing nut position calculation submodule, and the specific working process is:

1) The video image read from the video data storage software module is first pre-processed by the image pre-processing sub-module such as filter denoising and image restoration, and the pre-processed video image is sent to the sub-image cutting sub-module;

2) The purpose of the sub-image cropping sub-module is to cut as small a sub-image as possible containing the nut of the fastener system. The processing process first uses the template matching method to determine the sub-image containing the nut, and uses exhaustive search to determine the first fastener system The position of the nut in the video image; then predict the time when the nut of the fastening system on the next sleeper will appear according to the train speed, so that a sub-image of the nut can be cropped from the corresponding video image, where the train speed is based on the geographic location data and time values are calculated;

3) The feature extraction sub-module uses an image transformation algorithm to perform feature transformation on each sub-image transmitted by the sub-image cropping sub-module, obtains the low-dimensional feature vector of each sub-image, and sends it to the feature classification sub-module;

4) The feature classification processing process carried out by the feature classification sub-module uses the support vector machine algorithm to classify the features of the sub-images, judges whether the fastener system nuts in the sub-images corresponding to each feature vector are missing according to the classification results, and replaces the missing nuts The acquisition time of the image is sent to the missing nut position calculation submodule;

5) The missing nut position calculation sub-module searches and calculates the corresponding geographic location data from the video data storage software module according to the time values of these missing nut images, and after storing these time values and position data, generates a report and sends it to the display software module.

The present invention is an automatic detection device for missing nuts in a rail fastener system. The automatic detection device realizes automatic identification of missing nuts in the fastener system according to the principle of computer vision, uses a GPS locator to locate missing nuts, and has a high recognition rate for missing nuts. , and has the characteristics of simple positioning method and small positioning error. It is an on-line or off-line automatic detection device for missing nuts.

Description of drawings

Fig. 1 is a block diagram of the present invention;

Fig. 2 is a schematic diagram of connections between modules and sub-modules of the present invention.

Detailed ways

As shown in FIG. 1 , the present invention includes four components: a visual acquisition device, a GPS locator 3 , an illumination device 4 and a computer system 5 . The visual acquisition device includes a CCD image sensor 1 and an image acquisition card 2 . The visual collection device is used to collect continuous video images including the fastener system nuts on both sides of the rail at the bottom of the train, and transmit the obtained continuous video images to the computer system 5 .

The GPS locator 3 determines the geographic location of the visual acquisition device at each moment according to the GPS positioning principle, the geographic location data includes longitude and latitude values, and transmits the geographic location information of each moment to the video data storage software module 11 in the computer system 5 At the same time, the video data storage software module 11 also stores the corresponding time value when each geographical location information is acquired. The GPS locator 3 is connected and communicated with the computer system 5 through interfaces such as USB or RS-232.

The lighting device 4 is installed beside the CCD image sensor 1 of the visual acquisition device, and is used to increase the illumination of the fastener system nut, thereby improving the quality of the video image obtained.

As shown in FIG. 2 , the computer system 5 is provided with a video data storage software module 11 , a display software module 12 and a missing nut identification and positioning software module 13 . The video data storage software module 11 of computer system 5 receives in real time and stores the geographical location information of the current device obtained by GPS locator 3, and the fastener system nut continuous video image obtained by visual acquisition device, and completes geographic location data, video Storage of images and corresponding time values. The missing nut identification and positioning software module 13 completes the automatic identification and positioning of missing nuts by performing steps such as preprocessing, sub-image cropping, feature transformation, feature classification, and position calculation on the obtained video images. The display software module 12 can not only display the obtained continuous video images in real time, but also display the missing nut images processed by the missing nut identification and positioning software module 13 and all missing nut position information reports.

When the device of the present invention is installed, the optical axis of the CCD image sensor 1 is as vertical as possible to the plane of the rail subgrade, and is directly above the nut of the fastener system. Since each road sleeper is generally equipped with four fastener nuts, the present invention needs to install four CCD image sensors side by side at the bottom of the train. The lens magnification of the CCD image sensor 1 is properly selected according to the distance between the bottom fastener system nut and the image sensor. When the train is moving, the CCD image sensor 1 realizes the acquisition of continuous video images including nuts of the fastener system, and the obtained continuous video images are transmitted to the computer system 5 through the image acquisition card 2 .

The visual acquisition device is equipped with a high-speed CCD image sensor with as short an exposure time as possible, so that when the device of the present invention works, the train is allowed to travel at high speed. Except that the computer system platform adopts Windows XP, the video data storage software module 11 of the present invention, the display software module 12 and the missing nut identification and positioning software module 13 all adopt Visual C++ software programming to realize.

As shown in FIG. 2 , the missing nut identification and positioning software module 13 includes an image preprocessing submodule 6 , a subimage cropping submodule 7 , a feature extraction submodule 8 , a feature classification submodule 9 and a missing nut position calculation submodule 10 . Each sub-module respectively obtains the image or data from the previous module, and processes the image or data, and transmits the processed image or data to the next module.

Each software module and each submodule of the computer system 5 are further described in detail as follows:

1) Video data storage software module 11. CCD image sensor 1 collects required video images, and transmits to video data storage software module 11 through image acquisition card 2, on the other hand GPS locator 3 also periodically sends the geographical location information data of current moment to video data storage software module 11 The video data storage software module 11 receives and stores these video image data, geographic location data and corresponding time values for use by the display software module 12 and the missing nut identification and positioning software module 13.

2) The software module 12 is displayed. The display software module 12 is used to display the continuous video images obtained by the video data storage software module 11, and can also display and print all missing nut position information reports output by the missing nut identification and positioning software module 13 and images of corresponding missing nuts.

3) Missing nut identification and positioning software module 13. This module completes the identification and location of missing nuts online or offline according to the video image and geographic location data provided by the video data storage software module 11, and it includes an image preprocessing submodule 6, a sub-image cutting submodule 7, and a feature extraction submodule 8. Feature classification sub-module 9 and missing nut position calculation sub-module 10. Image pre-processing sub-module 6 reads the original video image and performs image pre-processing for use by sub-image cutting sub-module 7; sub-image cutting sub-module 7 uses a corresponding algorithm to determine the position of each fastener system nut in each frame image of continuous video position, and simultaneously cut the sub-images containing the nuts of the fastener system from each frame image, and determine the time information obtained by each sub-image; the feature extraction sub-module 8 applies an image transformation algorithm to each sub-image transmitted by the sub-image cutting sub-module 7 The image is subjected to feature transformation, and the low-dimensional feature vectors of each sub-image are calculated to be used by the feature classification sub-module 9; the feature classification sub-module 9 uses the corresponding feature classification method to classify each low-dimensional feature vector, thereby identifying each feature vector. Whether the fastener system nut in the corresponding sub-image is missing, and the acquisition time of the image of the missing nut is sent to the missing nut position calculation submodule 10; the missing nut position calculation submodule 10 is based on the time value of these missing nut images, from the video The data storage software module finds and calculates the corresponding geographic location data, and after storing these locations, a report is generated for use by the display software module 12 .

a) Image preprocessing sub-module 6. The image pre-processing sub-module 6 acquires the original continuous video images of fastener system nuts from the video data storage software module 11 for pre-processing, and provides the required images for the sub-image cutting sub-module 7 . Since electronic devices such as CCD image sensor 1 and image acquisition card 2 themselves will introduce image noise, the main tasks of the image preprocessing sub-module 6 include converting image formats of different formats into grayscale image formats, and using image filtering such as sliding window averaging The denoising algorithm filters and denoises the image.

b) The sub-image cropping sub-module 7 . The function realized by the sub-image cutting sub-module 7 is to determine the position of the nut of the fastener system in the video image, and to cut the sub-image containing the nut of the fastener system as small as possible. This sub-module uses an exhaustive search to determine the position of the first fastener system nut in the video image based on the template matching method used to determine the small area containing the nut. After determining the time and position of the first fastening system nut, the time and position of the fastening system nut on the next road sleeper are predicted according to the train speed and the distance between two adjacent road sleepers, so that in the corresponding video image Cropping yields a subimage of a nut. Note that the time when the nth nut appears is t _n , the current speed of the train is v _tn , and the distance between the centerlines of the sleepers is a constant L, then the calculation formula for the time when the n+1th nut appears is:

t _n+1 =t _n +L/v _tn (1)

The current speed v _tn of the train is calculated by the sub-image cutting sub-module 7 according to the geographic position data and the time variation value. Assume that the direction on the right side of the image is the train speed direction, set the video image as k frames/second, and the ratio of v _tn to k is recorded as P=v _tn /k, and assume that the nth nut appears for the first time in the k _nth sheet On the image, the actual distance from the right edge of the image is q, and the n+1th nut appears for the first time on the k _n+1th image, then:

k _n+1 =k _n +[(Lq)/p] (2)

Among them, "[]" indicates rounding down. The distance from the center point of the n+1th nut to the right edge of the _kn+1th image is p-((Lq)mod p), and mod represents the remainder operation. The distance from the top edge of the image remains unchanged. The corresponding relationship between the actual distance and the image pixel can be obtained by analyzing in advance, and then the q value and the specific position of the n+1 nut in the k _n+1 image can be obtained. According to the position of the nut of the fastener system, the sub-image containing the nut of the fastener system is cropped, and the pixels are recorded as n×m. The values of n and m are related to the distance between the CCD image sensor and the nut and the magnification of the CCD image sensor lens.

c) Feature extraction sub-module 8. The feature extraction sub-module 8 applies Principal Components Analysis (PCA, or principal component analysis) to extract the sub-image features provided by the sub-image cropping sub-module 7 including the fastener system nut. Assume that during the training process of the missing nut identification and positioning software module 13, a total of W nut sample images (including nut missing and nut normal images) are used, and each image has been converted into a gray image with a size of m×n pixels The image is represented by an m×n matrix A, and the value of the corresponding element of the matrix is the gray value of the corresponding pixel of the image. Principal component analysis is a linear dimensionality reduction technique. Note N=m×n, then each image is represented by a one-dimensional vector x _k =(a ₁ , a ₂ ,...,a _N ) of size N, where k=1, 2,... , W. Taking these W sample images as the training set, the overall covariance matrix is:

${\mathrm{<span\; class="notranslate">\; S</span>}}_{\mathrm{<span\; class="notranslate">\; T</span>}}<span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; E.</span>}<span\; class="notranslate">\; \{</span><span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; x</span>}<span\; class="notranslate">\; -</span>\mathrm{<span\; class="notranslate">\; u</span>}<span\; class="notranslate">\; )</span>{<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; x</span>}<span\; class="notranslate">\; -</span>\mathrm{<span\; class="notranslate">\; u</span>}<span\; class="notranslate">\; )</span>}^{\mathrm{<span\; class="notranslate">\; T</span>}}<span\; class="notranslate">\; \}</span><span\; class="notranslate">\; =</span>\frac{\mathrm{<span\; class="notranslate">\; 1</span>}}{\mathrm{<span\; class="notranslate">\; W</span>}}\underset{\mathrm{<span\; class="notranslate">\; k</span>}<span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; 1</span>}}{\overset{\mathrm{<span\; class="notranslate">\; k</span>}<span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; W</span>}}{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; x</span>}}_{\mathrm{<span\; class="notranslate">\; k</span>}}<span\; class="notranslate">\; -</span>\mathrm{<span\; class="notranslate">\; u</span>}<span\; class="notranslate">\; )</span>{<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; x</span>}}_{\mathrm{<span\; class="notranslate">\; k</span>}}<span\; class="notranslate">\; -</span>\mathrm{<span\; class="notranslate">\; u</span>}<span\; class="notranslate">\; )</span>}^{\mathrm{<span\; class="notranslate">\; T</span>}}<span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; A</span>}{\mathrm{<span\; class="notranslate">\; A</span>}}^{\mathrm{<span\; class="notranslate">\; T</span>}}<span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; 3</span>}<span\; class="notranslate">\; )</span>$

where u is the mean vector of all x _k , $A=[(x_1-u),(x_2-u),...,(x_W-u)]/\sqrt{W}.$ If the eigenvalues and corresponding unit eigenvectors of S _j are set to be _λk and _ηk respectively, k=1, 2,..., W, then according to principal component analysis, the linear transformation from the original image to the new feature space is y _k =D ^T x _k , D is a transformation matrix (k=1, 2, . . . , W) composed of eigenvectors η _k . Therefore, the eigenvectors corresponding to J (J≤W) eigenvalues can be selected from the W eigenvalues (for simplicity, the largest irrelevant group is directly selected), and it is orthogonalized to make it a new space Orthogonal basis of . The J eigenvalues are the principal components (principal components), and the coordinate coefficients of each sub-image under the new orthogonal basis are used as the low-dimensional feature parameters of the sub-image, which are provided for the next step of feature classification processing.

d) Feature classification sub-module 9. The trained feature classification sub-module 9 classifies the low-dimensional feature vectors sent by the feature extraction sub-module 8 to determine whether the nuts in the corresponding sub-images are missing. The feature classification sub-module 9 transmits the time information corresponding to the missing nut to the missing nut position calculation sub-module 10 . In the training phase, the feature classification sub-module 9 uses the support vector machine method to classify and train the features of the nut sample image extracted by the feature extraction sub-module 8, so as to identify missing nuts in other non-sample images. The support vector machine adopts the principle of structural risk minimization to improve the generalization ability of the learning machine. It is a classification method that can obtain small errors from the decision rules obtained from fewer training samples and can still obtain small errors for the independent test set. It basically solves the problem of over-learning. problems, problems of nonlinearity and the curse of dimensionality, and problems of local convergence. The support vector machine is dedicated to finding a hyperplane, so that the points belonging to different categories in the training set (each point corresponds to the principal component of each sample image) are located on different sides of the hyperplane, and at the same time, these points should be as far away from the hyperplane as possible. It may be far, that is, to maximize the classification interval. Suppose the training sample input is z _i , i=1, 2,..., W, z _i ∈ R ^j , and the corresponding expected output is b _i ∈ {+1, -1}, where +1 and -1 are respectively Sample z _i is the category identification of the image feature parameters of the sample images with intact nuts and missing nuts. The goal of the support vector machine is to construct an objective function according to the principle of structural risk minimization, so as to distinguish the intact and missing nuts of the rail fastener system as correctly as possible. The feature classification sub-module 9 must correctly select the kernel function according to the characteristics of the nut image of the fastener system, so as to achieve a satisfactory recognition effect of missing nuts.

e) Missing nut position calculation sub-module 10 . The missing nut position calculation sub-module 10 searches and calculates the corresponding geographical position data from the video data storage software module 11 according to the time value of the missing nut image transmitted by the feature classification sub-module 9, and after storing these position data, generates a report Send to the display software module 12.

After the above processing, the automatic detection and positioning of the missing nut of the rail fastener system is finally completed.

Claims (3)

1. device for automatically detecting nut loss of rail fastener system; Comprise: vision collecting device, GPS steady arm (3), lighting device (4) and computer system (5); Wherein computer system (5) comprises video data storing software module (11), software for display module (12) and disappearance nut identification positioning software module (13); It is characterized in that, be as the criterion the really identification and the location of nut loss of rail fastener system at present, this computer system (5) is carried out following steps:

A) continuous videos that comprises the fastener system nut that video data storing software module (11) receives and storage is sent by the vision collecting device, and write down each video image time corresponding value;

B) each moment vision collecting device geographical location information of living in that video data storing software module (11) receives and storage is sent by GPS steady arm (3), and write down each geographical location information time corresponding value;

C) disappearance nut identification positioning software module (13) to video image carry out that image pre-service, subimage cut, feature extraction and tagsort, thereby accomplish the identification of disappearance nut, and write down the acquisition time value of the corresponding image of these disappearance nuts;

D) disappearance nut identification positioning software module (13) is according to acquisition time value and each moment corresponding geographical location information of the corresponding image of disappearance nut, the geographic position of calculating the disappearance nut, the location of realizing the disappearance nut;

E) software for display module (12) shows the video of gathering on request, and gathers, shows and print the image and the geographical location information of disappearance nut.

2. device for automatically detecting nut loss of rail fastener system according to claim 1; It is characterized in that; The subimage that said disappearance nut identification positioning software module (13) is carried out cuts processing procedure; Purpose is to cut the as far as possible little subimage that comprises the fastener system nut, and this processing procedure at first uses template matching method to confirm to comprise the subimage of nut, adopts exhaustive search to confirm first position of fastener system nut in video image; Predict the time that the fastener system nut on the pillow of next road occurs according to train speed then; So just can from the corresponding video image, cut the subimage that obtains a nut, wherein train speed calculates with these geographic position data time corresponding values according to each geographic position data that GPS steady arm (3) provides.

3. device for automatically detecting nut loss of rail fastener system according to claim 1; It is characterized in that; The tagsort processing procedure that said disappearance nut identification positioning software module (13) is carried out; Adopt algorithm of support vector machine to carry out the tagsort of subimage, judge according to classification results whether the fastener system nut in the pairing subimage of each proper vector lacks.

Images