JP7299041B2 - Overhead wire fitting detection device and overhead wire fitting detection method - Google Patents

Description

The present invention relates to an overhead wire fitting detection device and an overhead wire fitting detection method for detecting overhead wire fittings such as hangers by image processing.

A railway operator needs to inspect contact lines installed along a long-distance railway track. Conventionally, overhead wire fittings such as hangers as overhead contact lines are visually checked by maintenance personnel from commercial vehicles, rail vehicles, or on railroad tracks. In order to reduce the labor of maintenance personnel involved in this work, it is desired to save labor by automating the inspection of contact lines. For example, in the inspection of overhead wire fittings, there is a method of detecting an image of overhead wire fittings using an image of the vicinity of a trolley wire captured by a line sensor camera installed on the roof of a railcar running on a railroad track. Then, the image of the hanger is detected from the image of the wire fitting, and the inspection and quality judgment are performed.

In general, normal hangers are installed vertically, but if they are tilted from the vertical direction due to temperature changes, etc., the current collection characteristics of the train will deteriorate, so it is also required to know the angle and position of the hangers.

Here, Patent Literature 1 listed below discloses a technique for detecting a hanger from a photographed image by template matching processing or vertical edge detection processing. Further, Patent Literature 2 listed below discloses a technique for detecting a hanger from a photographed image by Hough transform processing.

Further, Patent Document 3 below discloses a technique for detecting a hanger by applying GSTH processing. According to Patent Document 3, it is possible to detect a rectangular hanger that intersects the overhead wire.

In addition, in Patent Document 4, the applicant of the present application proposed a technique that makes it possible to detect not only hangers but also wire fittings such as connectors. While the techniques disclosed in Patent Documents 1 to 3 are techniques for detecting objects such as hangers that extend in a direction intersecting overhead wires, Patent Document 4 first removes the overhead wires and then removes the remaining structure. By detecting , it is possible to detect an overhead wire fitting extending in a direction different from that of the overhead wire when the overhead contact line is viewed from the sleeper direction (FIG. 13).

Japanese Patent No. 5402096 JP 2010-285054 A JP 2016-168937 A Japanese Patent Application No. 2017-247234

Yusuke Uchida, Takayoshi Yamashita, "[Survey Paper] Research Trends in Convolutional Neural Networks", IEICE Technical Report, Institute of Electronics, Information and Communication Engineers Publishing, December 2017, vol. 117, no. 362, pp. 25- 38

However, the inventions disclosed in Patent Documents 1 and 2 are capable of detecting a hanger with a simple filament structure. Difficult to detect.
That is, Patent Document 1 detects the hanger by edge detection in the vertical direction. In general edge detection, a portion having a large differential value between two pixels is detected as an edge.
Further, Patent Literature 2 detects a hanger by Hough transform processing. Since the Hough transform detects hangers by using the relative amount of votes between the features of the entire image and the hanger features, there is a problem that it is difficult to detect hangers that appear small in the entire image.

In addition, Patent Document 3 has a problem that it cannot detect an object having a curved shape instead of a rectangular shape, such as a connector.

Patent Document 4 has a problem that, when there is a background object extending in a direction different from that of the overhead wire or noise in the background, it may be erroneously detected as an overhead wire fitting.

In view of the above, an object of the present invention is to provide an overhead wire fitting detection device and an overhead wire fitting detection method capable of correctly detecting only wire fittings.

A wire fitting detection device according to a first invention for solving the above problems,
a line sensor camera that captures an image of overhead wires and overhead wire equipment perpendicularly to the traveling direction of the railway vehicle while the railway vehicle is running;
an image processing unit that performs image processing on a line sensor image created by arranging image signals output from the line sensor camera in time series, and detects an overhead wire fitting from the image;
The image processing unit
An arithmetic device equipped with an artificial intelligence function learned using an image of the overhead wire fitting obtained in advance and an image other than the overhead wire fitting,
The computing device is
a preprocessing unit that learns the artificial intelligence function;
a line sensor image creation unit that creates a line sensor image from an image signal input from the line sensor camera while the vehicle is running;
a brightness correction unit that performs a brightness correction process for increasing the difference between brightness and darkness in the image signal with respect to the line sensor image;
an image clipping unit that creates a clipped image clipped from the line sensor image that has undergone the brightness correction process and the overhead wire fitting or other than the overhead wire fitting;
and a fitting identification processing unit that identifies the overhead wire fitting by inputting the clipped image to the artificial intelligence function.

Further, a wire fitting detection device according to a second invention for solving the above problems is the first invention,
The image clipping unit
The judgment image clipping unit splits the line sensor image that has undergone the brightness correction process into a grid pattern, and clips an image of the overhead wire fitting or an image other than the overhead wire fitting from the divided line sensor image. Characterized by

Further, a wire fitting detection device according to a third invention for solving the above problems is, in the second invention,
The artificial intelligence function divides the line sensor image obtained in advance into a grid pattern, and for each of the divided images, an image in which the overhead wire fitting exists and a correct flag is attached, and an incorrect answer in which a wire fitting other than the overhead wire fitting exists. The convolutional neural network is a trained convolutional neural network in which 2-class discrimination learning for the overhead wire fitting and other than the overhead wire fitting is carried out in advance based on the flagged image.

In addition, a wire fitting detection device according to a fourth invention for solving the above problems is the first invention,
The computing device is
a day/night determining unit that determines whether the brightness of each of the image signals subjected to the brightness correction process is equivalent to daytime or equivalent to nighttime;
an image brightness reversing unit that performs a brightness reversal process on the image signal determined by the day/night determining unit to be equivalent in brightness to daytime;
a median filter processing unit that performs median filter processing on the line sensor image that has been subjected to the brightness inversion processing by the image brightness inversion unit;
a GSTH processing unit that performs GSTH processing in the extending direction of the overhead wire on the line sensor image that has been subjected to median filtering by the median filtering unit;
a grayscale opening processing unit that performs grayscale opening processing in a direction perpendicular to the extension direction of the overhead wire on the line sensor image that has been subjected to the GSTH processing by the GSTH processing unit;
a binarization processing unit that performs binarization processing on the line sensor image that has undergone grayscale opening processing in the grayscale opening processing unit;
a closing processing unit that performs closing processing in a direction perpendicular to the extension direction of the overhead wire on the line sensor image that has been subjected to the binarization processing by the binarization processing unit;
a metal fitting candidate detection processing unit that detects a white mass from the line sensor image that has been subjected to the closing processing by the closing processing unit, and detects the overhead wire metal fitting candidate from the size of the detected white mass. ,
The image clipping unit is a metal fitting candidate image clipping unit that cuts out an image of the metal fitting candidate from the image based on the information of the metal fitting candidate detected by the metal fitting candidate detection processing unit.

Further, a wire fitting detection device according to a fifth invention for solving the above problems is the fourth invention,
The artificial intelligence function extracts an image with a correct flag attached by cutting out the wire fitting part existing in the line sensor image acquired in advance, and extracting the part other than the overhead wire fitting existing in the line sensor image acquired in advance. It is a pre-trained convolutional neural network in which two-class discrimination learning of the wire fitting and other than the wire fitting is carried out in advance based on an image flagged with an incorrect answer.

Further, a wire fitting detection device according to a sixth invention for solving the above problems is the fourth or fifth invention,
The preprocessing unit is
a brightness correction unit that performs a brightness correction process for increasing a difference between light and dark in the image signal with respect to the previously obtained line sensor image;
a day/night determination unit that determines whether the brightness of each image signal subjected to brightness correction processing is equivalent to daytime or equivalent to nighttime;
an image brightness reversing unit that performs a brightness reversal process on the image signal determined by the day/night determination unit to have brightness equivalent to that of daytime with respect to the line sensor image subjected to the day/night determination by the day/night determination unit;
a median filter processing unit that performs median filter processing on the line sensor image that has been subjected to the brightness inversion processing by the image brightness inversion unit;
a GSTH processing unit that performs GSTH processing in the extending direction of the overhead wire on the line sensor image that has been subjected to median filtering by the median filtering unit;
a grayscale opening processing unit that performs grayscale opening processing in a direction perpendicular to the extension direction of the overhead wire on the line sensor image that has been subjected to the GSTH processing by the GSTH processing unit;
a binarization processing unit that performs binarization processing on the line sensor image that has undergone grayscale opening processing in the grayscale opening processing unit;
a closing processing unit that performs closing processing in a direction perpendicular to the extension direction of the overhead wire on the line sensor image that has been subjected to the binarization processing by the binarization processing unit;
a metal fitting candidate detection processing unit that detects a white mass from the line sensor image that has been subjected to the closing processing by the closing processing unit, and detects the wire metal fitting candidate from the size of the detected white mass;
a metal fitting candidate image clipping unit that cuts out an image of a metal fitting candidate from the line sensor image based on information on the metal fitting candidate detected by the metal fitting candidate detection processing unit;
The image clipped by the metal fitting candidate image clipping unit and flagged with a correct answer and the image clipped by the metal fitting candidate image clipping unit and flagged with an incorrect flag are input to the artificial intelligence function to perform the artificial intelligence function. and a weight learning unit that performs learning.

In addition, a wire fitting detection method according to a seventh invention for solving the above problems,
A pre-processing step of learning an artificial intelligence function using an image of an overhead wire fitting obtained in advance and an image other than the overhead wire fitting;
an image creation step of creating a line sensor image by arranging in time series image signals output from a line sensor camera that captures an image of an overhead wire and overhead wire equipment perpendicular to the traveling direction of the rail vehicle while the rail vehicle is running;
a brightness correction step of increasing a difference between light and dark in the image signal with respect to the line sensor image;
an image clipping step of clipping the overhead wire fitting or other than the overhead wire fitting from the line sensor image subjected to brightness correction;
and a fitting determination step of inputting the clipped image to the artificial intelligence function learned in the preprocessing step to identify the overhead wire fitting.

Further, an overhead wire fitting detection method according to an eighth invention for solving the above problems is, in the seventh invention,
The image clipping step divides the image of the overhead wire fitting or the image other than the overhead wire fitting into a grid pattern, and cuts out the image of the overhead wire fitting or the image other than the overhead wire fitting from the divided line sensor image. It is characterized by being a cutting process.

Further, a wire fitting detection method according to a ninth invention for solving the above problems is, in the eighth invention,
The artificial intelligence function divides the line sensor image obtained in advance into a grid pattern, and for each of the divided images, an image in which the overhead wire fitting exists and a correct flag is attached, and an incorrect answer in which a wire fitting other than the overhead wire fitting exists. The convolutional neural network is a trained convolutional neural network in which 2-class discrimination learning for the overhead wire fitting and other than the overhead wire fitting is carried out in advance based on the flagged image.

Further, a wire fitting detection method according to a tenth invention for solving the above problems is, in the seventh invention,
a day/night determination step of determining whether the brightness of each image signal subjected to the brightness correction process is equivalent to daytime or equivalent to nighttime;
an image brightness reversal step of performing a brightness reversal process on the image signal determined to have brightness equivalent to daytime in the day/night determination step with respect to the line sensor image;
a median filter processing step of performing median filtering on the line sensor image subjected to the luminance inversion processing in the image luminance inversion step;
A GSTH processing step of performing GSTH processing in the extension direction of the overhead wire on the line sensor image that has undergone median filtering in the median filtering step;
A grayscale opening processing step of performing a grayscale opening processing in a direction perpendicular to the extension direction of the overhead wire on the line sensor image subjected to the GSTH processing in the GSTH processing step;
a binarization processing step of performing binarization processing on the line sensor image that has undergone grayscale opening processing in the grayscale opening processing step;
A closing processing step of performing closing processing in a direction perpendicular to the extension direction of the overhead wire on the line sensor image that has undergone the binarization processing in the binarization processing step;
a metal fitting candidate detection processing step of detecting a white mass from the line sensor image subjected to the closing processing in the closing processing step, and detecting the wire metal fitting candidate from the size of the detected white mass. ,
The image clipping step is a metal fitting candidate image clipping step of cutting out an image of the metal fitting candidate from the image based on information of the metal fitting candidate detected by the metal fitting candidate detection processing section.

Further, an overhead wire fitting detection method according to an eleventh invention for solving the above problems is, in the tenth invention,
The artificial intelligence function extracts an image with a correct flag attached by cutting out the wire fitting part existing in the line sensor image acquired in advance, and extracting the part other than the overhead wire fitting existing in the line sensor image acquired in advance. It is a pre-trained convolutional neural network in which two-class discrimination learning of the wire fitting and other than the wire fitting is carried out in advance based on an image flagged with an incorrect answer.

Further, a wire fitting detection method according to a twelfth invention for solving the above problems is the tenth or eleventh invention,
The pretreatment step includes
a brightness correction step of performing a brightness correction process for increasing the difference between brightness and darkness in the image signal on the line sensor image obtained in advance;
a day/night determination step of determining whether the brightness of each image signal subjected to brightness correction processing is equivalent to daytime or equivalent to nighttime;
an image brightness reversing step of performing a brightness reversal process on the image signal whose brightness is determined to be equivalent to daytime in the day/night determination step with respect to the line sensor image subjected to the day/night determination in the day/night determination step;
a median filter processing step of performing median filtering on the line sensor image subjected to the luminance inversion processing in the image luminance inversion step;
A GSTH processing step of performing GSTH processing in the extension direction of the overhead wire on the line sensor image that has undergone median filtering in the median filtering step;
A grayscale opening processing step of performing a grayscale opening processing in a direction perpendicular to the extension direction of the overhead wire on the line sensor image subjected to the GSTH processing in the GSTH processing step;
a binarization processing step of performing binarization processing on the line sensor image that has undergone grayscale opening processing in the grayscale opening processing step;
A closing processing step of performing closing processing in a direction perpendicular to the extension direction of the overhead wire on the line sensor image that has undergone the binarization processing in the binarization processing step;
a metal fitting candidate detection processing step of detecting a white mass from the line sensor image subjected to the closing processing in the closing processing step and detecting the wire metal fitting candidate from the size of the detected white mass;
a metal fitting candidate image cutting step of cutting out an image of a metal fitting candidate from the line sensor image based on information of the metal fitting candidate detected in the metal fitting candidate detection processing step;
The image clipped in the metal fitting candidate image clipping step and flagged with a correct answer and the image clipped by the metal clip candidate image clipping unit and flagged with an incorrect flag are input to the artificial intelligence function to perform the artificial intelligence function. and a weight learning step of performing learning.

According to the overhead wire fitting detection device and the overhead wire fitting detection method according to the present invention, by performing fitting determination using an artificial intelligence function, parts other than overhead wire fittings are excluded from detection locations including erroneous detection, and only overhead wire fittings are detected. can be detected correctly.

1 is an explanatory diagram showing an installation example of an overhead wire fitting detection device according to Embodiment 1 of the present invention; FIG. 1 is a block diagram showing the configuration of an overhead wire fitting detection device according to Embodiment 1 of the present invention; FIG. It is a flowchart which shows the flow of an overhead wire metal fitting detection process in Example 1 of this invention. FIG. 4 is an explanatory diagram schematically showing an example of a line sensor image acquired in advance in Example 1 of the present invention; 5 is an explanatory diagram showing an example of creating correct data from the line sensor image shown in FIG. 4 in Embodiment 1 of the present invention; FIG. FIG. 2 is an explanatory diagram showing an example of a convolutional neural network applied in Example 1 of the present invention; FIG. 4 is an explanatory diagram showing an example of convolution filter processing in Example 1 of the present invention; It is a block diagram which shows the structure of the overhead wire metal fitting detection apparatus which concerns on Example 2 of this invention. It is a flowchart which shows the flow of an overhead wire metal fitting detection process in Example 2 of this invention. 5 is an explanatory diagram showing an example of creating correct data from the line sensor image shown in FIG. 4 in Embodiment 2 of the present invention; FIG. FIG. 10 is a block diagram showing the configuration of an overhead wire fitting detection device according to Example 3 of the present invention; It is a flowchart which shows the flow of an overhead wire metal fittings detection process in Example 3 of this invention. It is an explanatory view showing an example of an overhead wire fitting, and is a view seen from the side of the overhead wire.

DESCRIPTION OF THE PREFERRED EMBODIMENTS An overhead wire fitting detection device and an overhead wire fitting detection method according to the present invention will be described below with reference to the drawings.

Here, in the following description, the term "wire fitting" refers to the following.
Hangers, droppers, connectors, double-ear/curved wire fittings, supports These can be detected by the wire fitting detection device and the wire fitting detection method according to the present invention.

The present invention first removes the overhead wire from the image acquired by the line sensor camera, and detects the remaining structures. In addition to the overhead wire fittings extending in the same direction as the overhead wires, such as the crossing wire fittings 8 shown in FIG. can. In other words, it is possible to detect structures other than the overhead wire fittings (the hanger 6 and the connector 7) as long as they extend in a direction different from that of the overhead wire. Furthermore, by identifying overhead wire fittings by deep learning (machine learning), it is possible to exclude background objects that extend in a direction different from the overhead wire, which was conventionally erroneously detected as overhead wire fittings.

In this specification, "line" means the following.
Line: A measurement surface obtained when scanning along a direction perpendicular to the vehicle's traveling direction with a line sensor camera. For example, when a line sensor camera measures at 1000 Hz, the number of lines obtained per second is 1000 lines.

Embodiment 1 A wire fitting detection device and a wire fitting detection method according to a first embodiment of the present invention will be described with reference to FIGS. 1 to 7. FIG.

As shown in FIG. 1, the overhead wire fitting detection device in this embodiment includes a line sensor camera 2 installed on the roof of a train vehicle (hereinafter referred to as a vehicle) 1, and an image processing unit 3 installed inside the vehicle 1. and
The trolley wire (trolley wire) 4 is supported by a plurality of hangers 6 installed on the catenary wire 5 at approximately equal intervals.

The line sensor camera 2 is installed on the side of the roof of the vehicle 1, and its installation angle and elevation angle are such that its optical axis is perpendicular to the traveling direction of the vehicle 1 and that one hanger 6 is within its field of view. is set. 1 is a photographing line by the line sensor camera 2. As shown in FIG. An image signal (line) acquired by the line sensor camera 2 is input to the image processing section 3 .

The image processing unit 3 analyzes a line sensor image obtained by arranging image signals (lines) input from the line sensor camera 2 in time series to detect overhead wire fittings. It is configured.

The computing device 31 is equipped with an artificial intelligence function learned by using pre-obtained images of overhead wire fittings and images other than overhead wire fittings. As shown in FIG. It has a sensor image creation unit 31b, a brightness correction unit 31c, a judgment image extraction unit (image extraction unit) 31d, a metal fitting identification processing unit 31e, a memory M1 and a memory M2. Specific functions of the image processing unit 3 will be described with reference to the flowchart shown in FIG.

(Step S1)
The pre-processing unit 31a is a part that learns artificial intelligence functions, and includes an image database 31aa, a correct data creation unit 31ab, and a weight learning unit 31ac. In the image processing section 3, first, the preprocessing section 31a is used to perform the preprocessing of the following steps S1-1 to S1-3.

(Step S1-1)
In the image database 31aa, an input image (hereinafter referred to as a line sensor Image) A large number of L0 are stored in advance.

Therefore, first, the line sensor image L0 stored in the image database 31aa is acquired.

(Step S1-2)
Subsequently, using the line sensor image L0 acquired from the image database 31aa, correct data is created by the correct data creating unit 31ab and manually.

In this embodiment, the correct data includes a clipped image and a correct flag of a portion where an overhead wire fitting exists in the line sensor image L0, and a structure other than the overhead wire fitting in the line sensor image L0 (hereinafter simply referred to as a non-overhead wire fitting). It refers to the clipped image of the part to be answered and the incorrect answer flag.

For example, when there is a line sensor image L0 as shown in FIG. 4, if reference numerals 91, 92, and 93 are overhead wire fittings, and 94 is a non-overhead wire fitting, the correct data are _A91 , _A92 , _A93 and It becomes _A94 . That is, in this embodiment, as shown in FIG. 5, the correct data generation unit 31ab divides the line sensor image L0 into a grid pattern. After that, the area where the overhead wire fittings 91, 92, or 93 exist (divided area surrounded by a dashed line in FIG. 5) is manually labeled (correct flag), and the area where the wire fittings 94 other than the overhead wire fittings exist (see FIG. 5). 5), a label (wrong flag) other than the wiring bracket is given to the divided area indicated by the two-dot chain line. The correct data creation unit 31ab cuts out the part to which the wiring fitting label or the label other than the wiring fitting is given, and resizes it into a shape that can be input to the convolutional neural network (see Non-Patent Document 1 above) (hereinafter referred to as the cut out image (correct answer data) is referred to as clipped image A).

(Step S1-3)
The weight learning unit 31ac inputs the resized clipped image A to a convolutional neural network (CNN), and performs learning (machine learning) for two-class classification of wire fittings and non-wire fittings.

The contents of machine learning in the weight learning unit 31ac will be roughly described below with reference to FIG. As shown in FIG. 6, in the present embodiment, the convolutional neural network CNN includes five convolutional layers C1 to C5 (generically the convolutional layer C) and three fully connected layers D1 to D3 (generically It is composed of a fully connected layer D).

The convolution layer C performs convolution filter processing, pooling processing, and normalization processing.

In the convolution filter process, as shown in FIG. 7, for example, a convolution process is applied to an a×a input image (here, the resized clipped image A). The convolution process is a process of obtaining a weighted sum for each fixed size of the resized clipped image A. FIG. The weighting coefficient at this time is called a filter F. When the size of the filter F is f×f [pixel], the weighted sum is calculated for each area of f×f [pixel] while moving the filter F on the resized clipped image A. This process is performed on the entire resized clipped image A. FIG.

Specifically, the convolution filter processing is performed by taking a weighted sum of the input with a filter F representing a weighting factor such as which part should be emphasized and which part should be smoothed, so that the resized clipped image A is obtained. This is the process of creating the feature map M. The feature map M makes it possible to extract the density pattern of the image such as the edge feature. In addition, since features are extracted from a local area through the filter F, it is robust against spatial displacement of the position of the object.

Pooling processing is processing for compressing information in order to make input data easier to handle. In this embodiment, a max pooling process is performed in which a portion having the maximum output is selected from the local region and inherited to the next layer. This makes it possible to inherit only the optimal features specific to the features of the image to subsequent layers.

Normalization processing is processing for performing normalization so as to emphasize changes in gradation. The above-described convolution layer is a process of extracting image-like features (edges, trajectory shading patterns, etc.) for the resized clipped image A, rather than extracting features for two-class classification.

In the fully connected layer D, weights for connecting all the feature maps M are learned in order to classify each feature map M obtained through the convolution layer into two classes, that is, the wire fittings 91 to 93 and the non-wire fittings 94 .

The result of learning by the weight learning unit 31ac (hereinafter referred to as a learned network) is stored in the memory M2.

When the wire fitting is actually detected while the vehicle 1 is running, the processing from step S2 to step S6 is performed.

(Step S2)
The line sensor image creation unit 31b arranges image signals (lines) input from the line sensor camera 2 while the vehicle 1 is running in time series to create a line sensor image L1. The created line sensor image L1 is stored in the memory M1 and then stored in the memory M2.

(Step S3)
When the line sensor image L1 is input, the brightness correction unit 31c performs brightness correction processing for correcting the brightness value so as to increase the difference between white and black (brightness and darkness). The sensor image (metal fitting determination image) L2 is stored in the memory M2.
(Step S4)
When the line sensor image L2 is input, the determination image clipping unit 31d divides the line sensor image L2 into a grid pattern and stores the divided line sensor image (determination image) L3 in the memory M2.

(Step S5)
The metal fitting identification processing unit 31e reads out the learned network and the line sensor image L3 from the memory M2, inputs the line sensor image L3 to the learned network, calculates the metal fitting identification label and the metal fitting identification probability at that time, and calculates the metal fitting identification probability. are stored in the memory M2 as metal fitting identification label data and metal fitting identification probability data, and a portion determined to be an overhead wire metal fitting label based on the metal fitting identification label and the metal fitting identification probability is determined as an overhead wire metal fitting existing location, and metal fitting determination results and metal fitting position information are stored. is stored in memory M2.
(Step S6)
The processing from step S2 to step S5 described above is repeated until the inspection is completed.

According to the overhead wire fitting detection device and the overhead wire fitting detection method according to the present embodiment configured as described above, by performing fitting determination using the artificial intelligence function, detection points other than the overhead wire fitting are selected from among detection points including erroneous detection. can be excluded and only overhead wire fittings can be detected correctly. Further, by setting the entire line sensor image L1 as the metal fitting identification target area, it becomes possible to detect even the metal fittings that have not been detected in the conventional metal fitting detection process. In addition, since the number of identification target locations is constant, it is possible to detect overhead wire fittings in a constant processing time.

In this embodiment, an example in which the line sensor camera 2 and the image processing unit 3 are installed in the vehicle 1 is shown, but the present invention is not limited to the above-described embodiment, and a railway vehicle other than the vehicle 1 (for example, Needless to say, various modifications can be made without departing from the scope of the present invention, such as installing the line sensor camera 2 and the image processing unit 3 in the railcar) to detect overhead wire fittings.

Embodiment 2 A wiring fitting detection device and a wiring fitting detection method according to a second embodiment of the present invention will be described with reference to FIGS. 8 to 10. FIG.

As shown in FIG. 8, the arithmetic unit 31 in this embodiment includes the preprocessing unit 31a, the line sensor image generation unit 31b, the brightness correction unit 31c, the metal fitting identification processing unit 31e, and the day/night determination unit shown in FIG. 31f, an image brightness inversion unit 31g, a median filter processing unit 31h, a GSTH processing unit 31i, a grayscale opening processing unit 31j, a binarization processing unit 31k, a closing processing unit 31l, a labeling processing unit 31m, a metal fitting candidate detection processing unit 31n, and and a metal fitting candidate image clipping unit 31o. Concrete functions of the image processing section 3 in this embodiment will be described with reference to the flow chart shown in FIG.

Since the configuration other than the arithmetic device 31 is substantially the same as that of the first embodiment, redundant description will be omitted.

(Step S11)
The preprocessing unit 31a includes an image database 31aa, a correct data generation unit 31ab, and a weight learning unit 31ac, as in the first embodiment. In the image processing section 3, the preprocessing section 31a is first used to perform the preprocessing of the following steps S11-1 to S11-3.
(Step S11-1)
First, the line sensor image L0 stored in the image database 31aa is obtained.
(Step S11-2)
This embodiment differs from the first embodiment in the procedure for creating correct data. That is, in this embodiment, the line sensor image L0 is not divided into a grid pattern, and as shown in FIG. The wiring fitting label is placed in the area where the wiring fitting 91, 92 or 93 exists (the area surrounded by the solid line in FIG. 10), and the area where the wiring fitting 94 other than the wiring fitting exists (the area surrounded by the broken line in FIG. 10). Labels other than overhead wire fittings are given to . The correct data creation unit 31ab cuts out the manually surrounded portions (hereinafter referred to as cutout images) _B91 , _B92 , _B93 , and _B94 , and resizes them into a shape that can be input to the convolutional neural network.

(Step S11-3)
The weight learning unit 31ac inputs the resized clipped images B ₉₁ , B ₉₂ , B ₉₃ , and B ₉₄ to a convolutional neural network (CNN), and learns (machine learning) two-class classification of overhead wire fittings and non-overhead wire fittings. Let

When the wire fitting is actually detected while the vehicle 1 is running, the processing from step S12 to step S25 is performed.

(Step S12)
The line sensor image creation unit 31b arranges image signals (lines) input from the line sensor camera 2 while the vehicle 1 is running in time series to create a line sensor image L1. The created line sensor image L1 is stored in the memory M1 and then stored in the memory M2.

(Step S13)
When the line sensor image L1 is input, the brightness correction unit 31c performs brightness correction processing for correcting the brightness value so as to increase the difference between white and black (brightness and darkness). The sensor image L12 is saved in the memory M2.

(Step S14)
When the line sensor image L12 is input, the day/night determination unit 31f performs day/night determination processing on each line forming the line sensor image L12. That is, it is determined whether the brightness of each line is daytime (equivalent to daytime) or nighttime (equivalent to nighttime). As a criterion for determination, if the median brightness of each line is equal to or greater than the threshold, it is considered daytime, and if it is less than the threshold value, it is considered nighttime. The result of the day/night determination process for each line is stored in the memory M2 as day/night flag data.

(Step S15)
When the line sensor image L12 and the day/night flag data are input, the image brightness inversion unit 31g performs a brightness inversion process for inverting the brightness value of the line sensor image L12. At this time, based on the day/night flag data, the brightness is reversed only for the lines whose brightness is determined to be daytime (equivalent to daytime) by the day/night determination unit 31f, and the line sensor image L13 after the brightness reversal processing is stored in the memory M2. save.

(Step S16)
When the line sensor image L13 is input, the median filter processing unit 31h performs median filtering to remove white spots generated in the background due to processing in the brightness correction unit 31c, for example. Image L14 is stored in memory M2.

(Step S17)
When the line sensor image L14 is input, the GSTH processing unit 31i performs GSTH processing on the line sensor image L14 not in the horizontal direction but in the vertical direction. Specifically, first, horizontal lines of the hanger 6, the connector 7, and the like are erased by performing a grayscale opening process (a process of shrinking and then expanding) in the vertical direction. By setting the number of contractions and expansions performed here to a value larger than the vertical length of the hanger 6 and the connector 7, the horizontal lines of the hanger 6 and the connector 7 can be erased. Subsequently, by subtracting the grayscale opening processed image from the line sensor image L14, it is possible to obtain a line sensor image L15 in which only the edges of the hanger 6, the connector 7, and the contact wire 4 and the catenary wire 5 remain. , is stored in the memory M2.

(Step S18)
When the line sensor image L15 is input, the grayscale opening processing unit 31j performs grayscale opening processing in the horizontal direction in order to erase the edges of the contact wire 4 and the catenary wire 5 remaining in the line sensor image L15. Specifically, by performing grayscale opening processing with a value larger than the edges of the contact wire 4 and the messenger wire 5 remaining in the line sensor image L15, the edges of the contact wire 4 and the messenger wire 5 are removed. An image L16 can be obtained and stored in memory M2.

(Step S19)
When the line sensor image L16 is input, the binarization processing unit 31k performs binarization processing for converting the gray scale into black and white binary. The binarization threshold value at this time is set to a luminance value (for example, 10) that is larger than the background and is very small. That is, all values greater than the luminance value of the background are converted to 255. By this processing, a binarized line sensor image L17 can be obtained and stored in the memory M2.

(Step S20)
When the line sensor image L17 is input, the closing processing unit 31l performs a closing process (a process of expanding and then contracting) in the horizontal direction in order to connect the disconnected blocks. At this time, the number of times of expansion and contraction in the closing process is set to a value larger than the width of the contact wire 4 and the catenary wire 5 . By this processing, it is possible to obtain a binarized line sensor image L18 in which the wire fittings of the hanger 6 and the connector 7, which were once interrupted by the width of the trolley wire 4 and the catenary wire 5, are reconnected, and this is stored in the memory M2. .

(Step S21)
When the line sensor image L18 is input, the labeling processing unit 31m detects white blocks in the binarized image, performs a labeling process of labeling each block, and labels each of the detected white blocks. The labeled line sensor image L19 is stored in the memory M2.

(Step S22)
When the labeled line sensor image L19 is input, the metal fitting candidate detection processing unit 31n determines whether it is an overhead wire metal fitting or noise based on the size of the labeled white block. A small size is removed as noise, a white mass of a certain size or more is determined as an overhead wire fitting, and position information of the wire fitting determined as the wire fitting is stored in a memory M2 as metal fitting candidate position information.

(Step S23)
The metal fitting candidate image clipping unit 31o cuts out metal fitting candidate images from the line sensor image L1 based on the metal fitting candidate position information, and stores them in the memory M2 as metal fitting candidate cutout images.

(Step S24)
The metal fitting identification processing unit 31e reads out the learned network and the metal fitting candidate cutout images from the memory M2, inputs the metal fitting candidate cutout images to the learned network, calculates the metal fitting identification label and the metal fitting identification probability at that time, and calculates each of them. are stored in the memory M2 as metal fitting identification label data and metal fitting identification probability data, and a portion determined to be an overhead wire metal fitting label based on the metal fitting identification label and the metal fitting identification probability is determined as an overhead wire metal fitting existing location, and metal fitting determination results and metal fitting position information are stored. is stored in memory M2.
(Step S25)
The processing from step S12 to step S24 described above is repeated until the inspection is completed.

According to the overhead wire fitting detection device and the overhead wire fitting detection method according to the present embodiment configured as described above, by performing the metal fitting determination using the artificial intelligence function, the detected parts including false detections can be selected from the detected parts including false detection. It is possible to remove background objects that have been erroneously detected in 1 and correctly detect only overhead wire fittings. In addition, it is possible to unify the appearance of image data to be learned. As a result, variations in images to be learned are reduced, learning is more likely to converge, and identification accuracy is improved.

Note that the above-described labeling processing unit 31m is desirable for visibility, but is not essential. In the absence of the labeling processing unit 31m, the metal fitting candidate detection processing unit 31n receives the line sensor image L18, and the metal fitting candidate detection processing unit 31n detects white lumps in the line sensor image L18. The size of the white block can be used to determine whether it is an overhead wire fitting or noise.

Embodiment 3 A wire fitting detection device and a wire fitting detection method according to a third embodiment of the present invention will be described with reference to FIGS. 11 and 12. FIG. This embodiment differs from the above-described second embodiment in the method of creating correct data by the preprocessing unit 31a. The rest of the configuration is the same as that of the second embodiment, and the description that overlaps with the second embodiment will be omitted.

As shown in FIG. 11, in this embodiment, the preprocessing unit 31a includes the above-described image database 31aa, correct data generation unit 31ab, weight learning unit 31ac, and metal fitting candidate detection unit 31ad. The functions of the image processing section 3 in this embodiment will be described with reference to the flow chart shown in FIG.
(Step S31)
In the image processing section 3, first, the preprocessing section 31a is used to perform preprocessing of the following steps S31-1 to S31-5.

(Step S31-1)
First, the line sensor image L0 stored in the image database 31aa is acquired.
(Step S31-2)
The metal fitting candidate detection unit 31ad performs the same processing as steps S13 to S23 shown in FIG. 9 and described above. 11, for simplicity, the brightness correction unit 31c, the day/night determination unit 31f, the image brightness inversion unit 31g, the median filter processing unit 31h, the GSTH processing unit 31i, the grayscale opening processing unit 31j, and the A binarization processing unit 31k, a closing processing unit 31l, a labeling processing unit 31m, an overhead wire metal fitting detection processing unit 31n, and a metal fitting candidate image clipping unit 31o are collectively shown as a metal fitting candidate detection unit 31ad.

In short, in the present embodiment, the pre-processing unit 31a cuts out a metal fitting candidate cut-out image from the line sensor image L0 acquired from the image database 31aa through the above-described processing from step S13 to step S23. That is, in the second embodiment, metal fitting positions are manually detected and labeled, but in the present embodiment, metal fitting candidates are detected and labeled by performing image processing on the line sensor image L0 acquired from the image database 31aa. Based on the positional information of the detected metal fitting candidate, the detected part is cut out as a metal fitting candidate cut-out image.

(Step S31-3)
For the metal fitting candidate cutout image cut out by the metal fitting candidate detection unit 31ad, the area where the overhead wire fitting 91, 92, or 93 exists (the area surrounded by the solid line in FIG. 10) is manually labeled, and the overhead wire fitting A region where there is a non-wire 94 (a region surrounded by a dashed line in FIG. 10) is given a label other than wire fitting.

(Step S31-4)
The correct data creation unit ab resizes the labeled metal fitting candidate clipped image into a shape that can be input to the convolutional neural network to obtain correct data.

(Step S31-5)
The weight learning unit 31ac inputs the resized metal fitting candidate clipped image to a convolutional neural network (CNN), and performs learning (machine learning) for two-class classification of wire metal fittings and non-wire metal fittings.

When the wire fitting is actually detected while the vehicle 1 is running, the processing from the next step S32 to step S34 is performed.

(Step S32)
The line sensor image creation unit 31b arranges image signals (lines) input from the line sensor camera 2 while the vehicle 1 is running in time series to create a line sensor image L1. The created line sensor image L1 is stored in the memory M1 and then stored in the memory M2.

(Step S33)
The metal fitting identification unit 31p performs the same processing as steps S13 to S24 described in the second embodiment. 11, for simplicity, the brightness correction unit 31c, the day/night determination unit 31f, the image brightness inversion unit 31g, the median filter processing unit 31h, the GSTH processing unit 31i, the grayscale opening processing unit 31j, and the A binarization processing unit 31k, a closing processing unit 31l, a labeling processing unit 31m, an overhead wire metal fitting detection processing unit 31n, a metal fitting candidate image extraction unit 31o, and a metal fitting identification processing unit 31e are collectively shown as a metal fitting candidate detection unit 31p.
(Step S34)
The processing from step S32 to step S33 is repeated until the inspection is completed.

With this configuration, for example, in the second embodiment, variations in images to be learned are reduced, learning is more likely to converge, and learning accuracy is improved. Therefore, there is a possibility that the accuracy of identifying an unknown image may be degraded. By labeling the image detected by the overhead wire fitting detection process, machine learning can be performed with an image that is close to the image actually input to the fitting identification unit 31p, thereby improving the identification accuracy.

1 train vehicle 2 line sensor camera 3 image processing unit 4 trolley wire 5 catenary wire 6 hanger 7 connector 8 crossing wire metal fittings 31 arithmetic unit 31a preprocessing unit 31aa image database 31ab correct data creation unit 31ac weight learning unit 31ad metal fitting candidate detection unit 31b Line sensor image creation unit 31c Brightness correction unit 31d Judgment image extraction unit 31e Metal fitting identification processing unit 31f Day/night judgment unit 31g Image brightness inversion unit 31h Median filter processing unit 31i GSTH processing unit 31j Grayscale opening processing unit 31k Binarization processing unit 31l closing processing unit 31m labeling processing unit 31n metal fitting candidate detection processing unit 31o metal fitting candidate image extraction unit 31p metal fitting identification unit 32 recording device 91 to 93 overhead wire fittings 94 other than overhead wire fittings A ₉₁ to A ₉₃ , B ₉₁ to B ₉₃ overhead wire fitting labels A ₉₄ , B ₉₄ Labels other than overhead wire fittings M1, M2 Memory

Claims (10)

a line sensor camera that captures an image of overhead wires and overhead wire equipment perpendicularly to the traveling direction of the railway vehicle while the railway vehicle is running;
an image processing unit that performs image processing on a line sensor image created by arranging image signals output from the line sensor camera in time series, and detects an overhead wire fitting from the image;
The image processing unit
An arithmetic device equipped with an artificial intelligence function learned using an image of the overhead wire fitting obtained in advance and an image other than the overhead wire fitting,
The computing device is
a preprocessing unit that learns the artificial intelligence function;
a line sensor image creation unit that creates a line sensor image from an image signal input from the line sensor camera while the vehicle is running;
a brightness correction unit that performs a brightness correction process for increasing the difference between brightness and darkness in the image signal with respect to the line sensor image;
an image clipping unit that creates a clipped image clipped from the line sensor image that has undergone the brightness correction process and the overhead wire fitting or other than the overhead wire fitting;
a fitting identification processing unit that identifies the overhead wire fitting by inputting the clipped image to the artificial intelligence function ,
The image clipping unit
a determination image clipping unit that divides the line sensor image that has undergone the brightness correction process into a grid pattern, and cuts out an image of the overhead wire fitting or an image other than the overhead wire fitting from the divided line sensor image.
An overhead wire fitting detection device characterized by: The artificial intelligence function divides the line sensor image obtained in advance into a grid pattern, and for each of the divided images, an image in which the overhead wire fitting exists and a correct flag is attached, and an incorrect answer in which a wire fitting other than the overhead wire fitting exists. 2. The overhead wire fitting detection according to claim 1 , wherein the trained convolutional neural network is a pre-trained convolutional neural network in which two-class discrimination learning of the overhead wire fitting and the wire fitting other than the overhead wire fitting is carried out in advance based on the flagged image. Device. The computing device is
a day/night determining unit that determines whether the brightness of each of the image signals subjected to the brightness correction process is equivalent to daytime or equivalent to nighttime;
an image brightness reversing unit that performs a brightness reversal process on the image signal determined by the day/night determining unit to be equivalent in brightness to daytime;
a median filter processing unit that performs median filter processing on the line sensor image that has been subjected to the brightness inversion processing by the image brightness inversion unit;
a GSTH processing unit that performs GSTH processing in the extending direction of the overhead wire on the line sensor image that has been subjected to median filtering by the median filtering unit;
a grayscale opening processing unit that performs grayscale opening processing in a direction perpendicular to the extension direction of the overhead wire on the line sensor image that has been subjected to the GSTH processing by the GSTH processing unit;
a binarization processing unit that performs binarization processing on the line sensor image that has undergone grayscale opening processing in the grayscale opening processing unit;
a closing processing unit that performs closing processing in a direction perpendicular to the extension direction of the overhead wire on the line sensor image that has been subjected to the binarization processing by the binarization processing unit;
a metal fitting candidate detection processing unit that detects a white mass from the line sensor image that has been subjected to the closing processing by the closing processing unit, and detects the overhead wire metal fitting candidate from the size of the detected white mass. ,
2. The overhead wire according to claim 1, wherein the image clipping unit is a metal fitting candidate image clipping unit that cuts out an image of the metal fitting candidate from the image based on the metal fitting candidate information detected by the metal fitting candidate detection processing unit. Hardware detection device. The artificial intelligence function extracts an image with a correct flag attached by cutting out the wire fitting part existing in the line sensor image acquired in advance, and extracting the part other than the overhead wire fitting existing in the line sensor image acquired in advance. 4. The overhead wire according to claim 3 , wherein the trained convolutional neural network is a pre-trained convolutional neural network in which two-class discrimination learning of the wire fitting and the wire fitting other than the wire fitting is performed in advance based on the image to which an incorrect answer flag is attached. Hardware detection device. The preprocessing unit is
a brightness correction unit that performs a brightness correction process for increasing a difference between light and dark in the image signal with respect to the previously acquired line sensor image;
a day/night determination unit that determines whether the brightness of each image signal subjected to brightness correction processing is equivalent to daytime or equivalent to nighttime;
an image brightness reversing unit that performs a brightness reversal process on the image signal determined by the day/night determination unit to have brightness equivalent to that of daytime with respect to the line sensor image subjected to the day/night determination by the day/night determination unit;
a median filter processing unit that performs median filter processing on the line sensor image that has been subjected to the brightness inversion processing by the image brightness inversion unit;
a GSTH processing unit that performs GSTH processing in the extending direction of the overhead wire on the line sensor image that has been subjected to median filtering by the median filtering unit;
a grayscale opening processing unit that performs grayscale opening processing in a direction perpendicular to the extension direction of the overhead wire on the line sensor image that has been subjected to the GSTH processing by the GSTH processing unit;
a binarization processing unit that performs binarization processing on the line sensor image that has undergone grayscale opening processing in the grayscale opening processing unit;
a closing processing unit that performs closing processing in a direction perpendicular to the extension direction of the overhead wire on the line sensor image that has been subjected to the binarization processing by the binarization processing unit;
a metal fitting candidate detection processing unit that detects a white mass from the line sensor image that has been subjected to the closing processing by the closing processing unit, and detects the wire metal fitting candidate from the size of the detected white mass;
a metal fitting candidate image clipping unit that cuts out an image of a metal fitting candidate from the line sensor image based on information on the metal fitting candidate detected by the metal fitting candidate detection processing unit;
The image clipped by the metal fitting candidate image clipping unit and flagged with a correct answer and the image clipped by the metal fitting candidate image clipping unit and flagged with an incorrect flag are input to the artificial intelligence function to perform the artificial intelligence function. 5. The overhead wire fitting detection device according to claim 3 , further comprising a weight learning unit that performs learning. A pre-processing step of learning an artificial intelligence function using an image of an overhead wire fitting obtained in advance and an image other than the overhead wire fitting;
an image creation step of creating a line sensor image by arranging in time series image signals output from a line sensor camera that captures an image of an overhead wire and overhead wire equipment perpendicular to the traveling direction of the rail vehicle while the rail vehicle is running;
a brightness correction step of increasing a difference between light and dark in the image signal with respect to the line sensor image;
an image clipping step of clipping the overhead wire fitting or other than the overhead wire fitting from the line sensor image subjected to brightness correction;
a metal fitting determination step of inputting the clipped image to the artificial intelligence function learned in the preprocessing step to identify the overhead wire metal fitting ,
The image clipping step divides the image of the overhead wire fitting or the image other than the overhead wire fitting into a grid pattern, and cuts out the image of the overhead wire fitting or the image other than the overhead wire fitting from the divided line sensor image. cutting process
An overhead wire fitting detection method characterized by: The artificial intelligence function divides the line sensor image obtained in advance into a grid pattern, and for each of the divided images, an image in which the overhead wire fitting exists and a correct flag is attached, and an incorrect answer in which a wire fitting other than the overhead wire fitting exists. 7. The wire fitting detection according to claim 6 , wherein the trained convolutional neural network is a pre-trained convolutional neural network in which two-class discrimination learning of the wire fitting and the wire fitting other than the wire fitting is carried out in advance based on the flagged image. Method. a day/night determination step of determining whether the brightness of each image signal subjected to the brightness correction process is equivalent to daytime or equivalent to nighttime;
an image brightness reversal step of performing a brightness reversal process on the image signal determined to have brightness equivalent to daytime in the day/night determination step with respect to the line sensor image;
a median filter processing step of performing median filtering on the line sensor image subjected to the luminance inversion processing in the image luminance inversion step;
A GSTH processing step of performing GSTH processing in the extension direction of the overhead wire on the line sensor image that has undergone median filtering in the median filtering step;
A grayscale opening processing step of performing a grayscale opening processing in a direction perpendicular to the extension direction of the overhead wire on the line sensor image subjected to the GSTH processing in the GSTH processing step;
a binarization processing step of performing binarization processing on the line sensor image that has undergone grayscale opening processing in the grayscale opening processing step;
A closing processing step of performing closing processing in a direction perpendicular to the extension direction of the overhead wire on the line sensor image that has undergone the binarization processing in the binarization processing step;
a metal fitting candidate detection processing step of detecting a white mass from the line sensor image subjected to the closing processing in the closing processing step, and detecting the wire metal fitting candidate from the size of the detected white mass. ,
7. The overhead wire according to claim 6 , wherein the image clipping step is a metal fitting candidate image clipping step of cutting out an image of the metal fitting candidate from the image based on the metal fitting candidate information detected in the metal fitting candidate detection processing step. Hardware detection method. The artificial intelligence function extracts an image with a correct flag attached by cutting out the wire fitting part existing in the line sensor image acquired in advance, and extracting the part other than the overhead wire fitting existing in the line sensor image acquired in advance. 9. The overhead wire according to claim 8 , wherein the trained convolutional neural network is a pre-trained convolutional neural network that has previously performed two-class discrimination learning of the overhead wire fitting and the other than the overhead wire fitting based on the image to which an incorrect flag is attached. Hardware detection method. The pretreatment step includes
a brightness correction step of performing a brightness correction process for increasing the difference between brightness and darkness in the image signal on the line sensor image obtained in advance;
a day/night determination step of determining whether the brightness of each image signal subjected to brightness correction processing is equivalent to daytime or equivalent to nighttime;
an image brightness reversing step of performing a brightness reversal process on the image signal whose brightness is determined to be equivalent to daytime in the day/night determination step with respect to the line sensor image subjected to the day/night determination in the day/night determination step;
a median filter processing step of performing median filtering on the line sensor image subjected to the luminance inversion processing in the image luminance inversion step;
A GSTH processing step of performing GSTH processing in the extension direction of the overhead wire on the line sensor image that has undergone median filtering in the median filtering step;
A grayscale opening processing step of performing a grayscale opening processing in a direction perpendicular to the extension direction of the overhead wire on the line sensor image subjected to the GSTH processing in the GSTH processing step;
a binarization processing step of performing binarization processing on the line sensor image that has undergone grayscale opening processing in the grayscale opening processing step;
A closing processing step of performing closing processing in a direction perpendicular to the extension direction of the overhead wire on the line sensor image that has undergone the binarization processing in the binarization processing step;
a metal fitting candidate detection processing step of detecting a white mass from the line sensor image subjected to the closing processing in the closing processing step and detecting the wire metal fitting candidate from the size of the detected white mass;
a metal fitting candidate image cutting step of cutting out an image of a metal fitting candidate from the line sensor image based on information of the metal fitting candidate detected in the metal fitting candidate detection processing step;
The image cut out in the metal fitting candidate image cutout step and marked with a correct flag and the image cut out in the metal fitting candidate image cutout step and marked with an incorrect flag are input to the artificial intelligence function to perform the artificial intelligence function. 10. The wire fitting detection method according to claim 8 , further comprising a weight learning step of performing learning.

Images