JP7473491B2 - Anomaly detection device and anomaly detection method - Google Patents

Description

The present invention relates to an abnormality detection device that detects abnormalities in electric train wire fittings.

In the railway industry, there is a demand for improving efficiency by automatically inspecting overhead train lines, which are long structures, using traveling trains to reduce the workload of maintenance personnel. One of the inspection items for overhead train lines that would like to be automated is the inspection of the overhead train line fittings. To meet this demand, progress is being made in the development and practical application of technology for detecting abnormalities in overhead train line fittings using image processing, such as photographing the overhead train line using a photographing device (e.g., a line sensor camera) installed on the roof of a train, detecting the overhead train line fittings in the photographed images, and detecting any abnormalities in appearance (see, for example, Patent Document 1).

JP 2020-149286 A

In order to prevent localized wear on the pantograph's contact strips, overhead wires are stretched in a zigzag pattern from left to right (in the direction of the sleepers) with respect to the center of the track. For this reason, the camera installed on the roof of the railway vehicle approaches or moves away from the overhead wire depending on the train's position, and the relative positional relationship changes. This can result in reduced accuracy in detecting abnormalities, as the position and size of the overhead wire fittings shown in the captured images of the wire vary depending on the train's position.

The present invention was made in consideration of the above circumstances, and its purpose is to improve the accuracy of detecting abnormalities in electric train wire fittings by processing captured images that show the electric train wire fittings.

The first invention for solving the above problem is:
an abnormality detection device that converts a pre-conversion image showing an overhead contact line on which an overhead contact line fitting is provided so as to include an overhead contact line fitting having a first engagement portion that engages with a first wire, a second engagement portion that engages with a second wire, and an intermediate portion that connects the first engagement portion and the second engagement portion, into a preparatory image for determination of a predetermined size, and then performs an abnormality detection process for the overhead contact line fitting by image recognition processing,
a division unit (e.g., a division unit 206 in FIG. 7 ) that divides the pre-conversion image into a plurality of ranges in a direction along the longitudinal direction of the electric train wire fitting, the division unit dividing the range in which the intermediate portion is captured (hereinafter, this range will be referred to as an “intermediate range”) into one range;
a preparatory image for determination generating means for generating the preparatory image for determination of the predetermined size by enlarging or reducing the pre-conversion image at least along the longitudinal direction of the electric train wire fittings, the preparatory image for determination generating means (e.g., the preparatory image for determination generating unit 208 in FIG. 7 ) generating the preparatory image for determination of the predetermined size by enlarging or reducing the pre-conversion image at least along the longitudinal direction of the electric train wire fittings, the preparatory image for determination generating means generating the preparatory image for determination of the predetermined size by enlarging or reducing the preparatory image for determination with an allowable change rate of the enlargement or reduction set to be larger in the intermediate range than in a range other than the intermediate range;
The abnormality detection device includes:

Other inventions include:
1. An anomaly detection method for detecting an anomaly in an overhead contact line, the method comprising: converting a pre-conversion image showing an overhead contact line provided with an overhead contact line fitting, the overhead contact line fitting including a first engagement portion that engages with a first wire, a second engagement portion that engages with a second wire, and an intermediate portion that connects the first engagement portion and the second engagement portion, into a preparatory image for determination of a predetermined size, and then performing an anomaly detection process for the overhead contact line fitting by image recognition processing,
a division step of dividing the pre-conversion image into a plurality of ranges in a direction along a longitudinal direction of the train rail metal fitting, the division step dividing the range in which the intermediate portion is captured (hereinafter, this range is referred to as an "intermediate range") as one range;
a preparatory image for determination generating step of generating the preparatory image for determination of the predetermined size by enlarging or reducing the pre-conversion image at least along a longitudinal direction of the electric train wire fitting, in which the preparatory image for determination of the predetermined size is generated by enlarging or reducing the allowable change rate of the enlargement or reduction in the intermediate range larger than in a range other than the intermediate range;
The anomaly detection method may include the following.

According to the first invention, the accuracy of detecting abnormalities in the electric train line fittings can be improved by image processing of the captured image in which the electric train line fittings are shown. Many electric train line fittings, such as hangers, droppers, and connectors, have an elongated shape. In the pre-conversion image, the longitudinal length of the electric train line fittings is often longer in the intermediate range in which the intermediate parts connecting the engaging parts with the wires are shown than in the range in which the engaging parts are shown. In addition, the probability of abnormalities occurring is higher in the engaging parts of the electric train line fittings than in the intermediate parts. For this reason, when the pre-conversion image is enlarged or reduced at least along the longitudinal direction of the electric train line fittings, the allowable change rate of enlargement or reduction along the longitudinal direction is increased in the intermediate range in which the intermediate parts are shown than in the range in which the engaging parts of the electric train line fittings are shown. In this way, in the generated preliminary image for judgment, the range in which the engaging parts, which have a high probability of abnormalities, are shown is relatively larger than the intermediate range in which the intermediate parts, which have a low probability of occurrence, which is advantageous in terms of size and resolution. This can improve the accuracy of abnormality detection by image recognition processing using the preliminary image for judgment.

The second invention is the first invention,
the dividing means divides the pre-conversion image into a first engagement portion range in which the first engagement portion appears, the intermediate range, and a second engagement portion range in which the second engagement portion appears;
It is an anomaly detection device.

According to the second invention, the pre-conversion image is divided into three ranges: a first engagement portion range, a second engagement portion range, and an intermediate range.

A third aspect of the present invention relates to the second aspect of the present invention,
the determination preparation image generating means generates the determination preparation image of the predetermined size by enlarging or reducing only the intermediate range without enlarging or reducing the first engagement portion range and the second engagement portion range.
It is an anomaly detection device.

According to the third invention, the first and second engagement part ranges of the electric train wire fittings, which include the engagement parts with a high probability of abnormality, are not enlarged or reduced, and only the intermediate range, which includes the intermediate parts with a low probability of abnormality, is enlarged or reduced to generate a preparatory image for judgment of a predetermined size. This makes the image size and resolution of the first and second engagement parts, which have a high probability of abnormality, more advantageous than the intermediate parts, thereby improving the accuracy of abnormality detection, and as a result, improving the accuracy of abnormality detection in electric train wire fittings.

A fourth aspect of the present invention is the second or third aspect of the present invention,
a pre-conversion image extraction means (e.g., the pre-conversion image extraction unit 204 in FIG. 7 ) that performs scaling and cropping of an original photographed image of the overhead train line to extract the pre-conversion image from the original photographed image, the pre-conversion image extraction means performing the scaling and cropping so that the first engagement partial range and the second engagement partial range satisfy a predetermined size condition;
The abnormality detection device further comprises:

According to the fourth invention, by performing scaling and cropping of an original photographed image of an overhead train line, a pre-conversion image in which the first and second engagement parts satisfy a predetermined size condition is extracted from the original photographed image. In other words, since the overhead train line is stretched in a zigzag pattern to the left and right of the center of the track, the position and size of the overhead train line in the original photographed image of the overhead train line differs depending on the train position. In response to this, by performing a scaling process that enlarges or reduces the original photographed image so that the positions and sizes of the engagement parts of the overhead train line fittings in the original photographed image are the same, and cropping of the pre-conversion image, a pre-conversion image that satisfies a predetermined size condition, such as the first and second engagement parts being the same size, can be extracted.

A fifth aspect of the present invention relates to the fourth aspect of the present invention,
the pre-conversion image extraction means performs the extraction process so that the first line and the second line are captured at predetermined positions in the pre-conversion image.
It is an anomaly detection device.

According to the fifth aspect of the invention, the cutting process is performed so that the first and second lines are captured at predetermined positions in the pre-conversion image. This makes it possible to extract the pre-conversion image in which the engagement portion with the lines is captured at the predetermined position.

A sixth aspect of the present invention is the fourth or fifth aspect of the present invention,
The original photographed image is associated with data indicating a relative positional relationship between the photographing device that photographed the image and the overhead train line,
the pre-conversion image extraction means performs the scaling process and the cut-out process using the data indicating the relative positional relationship.
It is an anomaly detection device.

According to the sixth invention, since the overhead train wires are stretched in a zigzag pattern to the left and right of the center of the track, the position and size of the overhead train wire fittings in the original photographed image differs depending on the train position. However, by using data indicating the relative positional relationship between the photographing device and the overhead train wires, the position and size of the overhead train wire fittings in the original photographed image can be determined. This makes it possible to determine the degree of enlargement or reduction of the original photographed image in the scaling process, and to perform a cutout process so that the first and second wires are located at predetermined positions in the pre-conversion image.

A seventh aspect of the present invention is the method according to any one of the second to sixth aspects of the present invention,
A normalized image generating means (e.g., the normalized image generating unit 210 in FIG. 7 ) that generates a normalized image for the image recognition processing by converting the size of the determination preparation image;
The abnormality detection device further comprises:

According to the seventh invention, a normalized image for image recognition processing can be generated by converting the size of the preparation image for judgment.

An eighth aspect of the present invention is the method according to any one of the second to sixth aspects of the present invention,
a normalized image generating means for generating a multi-channel image by converting the image relating to the first engagement portion range, the image relating to the intermediate range, and the image relating to the second engagement portion range, among the determination preparation images, into single-channel images, thereby generating a normalized image for the image recognition processing;
The abnormality detection device further comprises:

According to the eighth invention, among the preparatory images for judgment, the image relating to the first engagement portion range, the image relating to the intermediate range, and the image relating to the second engagement portion range can each be made into a single channel image to generate a multi-channel image as a normalized image for image recognition processing. In this case, since there is no need to convert the size of the preparatory images for judgment, it is possible to suppress the loss of image information that occurs when the image is reduced. As the multi-channel image, for example, a color image in which each of the images relating to the three ranges is made into a single channel image of RGB (red, green, blue).

FIG. 4 is an explanatory diagram of a method for capturing an original captured image. 13 is a flowchart of an abnormality detection process. 5A and 5B are explanatory diagrams of a scaling process for an original photographed image. FIG. 4 is an explanatory diagram of how to calculate the elevation angle θ. FIG. 11 is an explanatory diagram of a process for cutting out a pre-conversion image. 6A and 6B are explanatory diagrams for generating a preliminary image for judgment and a normalized image. FIG. 2 is a functional configuration diagram of the abnormality detection device. 13 is an example of a normalized image to which the present embodiment is applied. An example of a comparative example image. An example of experimental results.

Below, an example of a preferred embodiment of the present invention will be described with reference to the drawings. Note that the present invention is not limited to the embodiment described below, and the forms to which the present invention can be applied are not limited to the following embodiment. In addition, in the description of the drawings, the same elements are given the same reference numerals.

[overview]
In this embodiment, an abnormality in the overhead contact wire fittings is detected by image recognition processing using an original photographed image of an overhead contact wire on which the overhead contact wire fittings are installed.

In this embodiment, the electric wire fitting has a first engagement part that engages with the first wire, a second engagement part that engages with the second wire, and an intermediate part that connects the first engagement part and the second engagement part. The electric wire fitting in this embodiment is, for example, a hanger that suspends a contact wire from a catenary or an auxiliary catenary, a dropper that suspends an auxiliary catenary from a catenary, a connector that electrically connects a catenary and a contact wire, and of course, may include other parts. The relevant parts of the engagement part and intermediate part are as follows: for the hanger, the bent part of the hanger that contacts the catenary and the ear are the engagement parts with the wire, and the straight part (bar) of the hanger is the intermediate part that connects the engagement parts. For the dropper, the clip is the engagement part with the wire, and the wire is the intermediate part. For the connector, the clamp and the ear are the engagement parts with the wire, and the lead wire is the intermediate part.

First, we will explain the original captured image. In this embodiment, the original captured image is an image of an overhead train line captured using a camera installed on the roof of a railway vehicle.

Figure 1 is a diagram explaining an example of a method for capturing an original image. As shown in Figure 1, as a system for capturing images of an overhead train line 20, two image capturing devices, a line sensor camera 10 (10a, 10b), two laser range sensors 12 (12a, 12b), and multiple LED lights 14 (14a, 14b), are arranged on the roof of a railway vehicle 5.

The line sensor cameras 10a and 10b are arranged at a predetermined distance on a straight line extending horizontally (in the vehicle width direction and sleeper direction) approximately perpendicular to the traveling direction (vehicle front-rear direction) of the railway vehicle 5. The line sensor cameras 10a and 10b each capture an image of a space including the overhead train line 20 to generate first image data and second image data representing a one-dimensional line sensor image. The line sensor cameras 10a and 10b each include a line sensor having a plurality of pixels arranged along a straight line extending horizontally approximately perpendicular to the traveling direction of the railway vehicle 5, and a lens attached to the front of the line sensor to focus light from the surrounding space onto the plurality of pixels of the line sensor. This lens can make the imaging angle in a plane approximately perpendicular to the traveling direction of the railway vehicle 5 correspond 1:1 to the position of the pixel on the line sensor. Therefore, it is possible to obtain the imaging angle of the overhead train line 20 as seen from the line sensor cameras 10a and 10b from the position of the pixel on the line sensor.

The laser range sensors 12a and 12b are arranged at a predetermined distance on a straight line extending horizontally and approximately perpendicular to the traveling direction of the railcar 5. The laser range sensors 12a and 12b project laser beams at a projection angle within a predetermined range on a plane approximately perpendicular to the traveling direction of the railcar 5 to scan a space including the overhead train line 20. The laser range sensors 12a and 12b measure the position of the overhead train line 20 and generate angle data indicating the direction in which the overhead train line 20 exists and distance data indicating the distance to the overhead train line 20. By using the two laser range sensors 12a and 12b, even if multiple overhead train lines 20 are arranged overlapping each other as viewed from one of the laser range sensors 12, the overhead train lines can be measured separately.

The LED lights 14a and 14b illuminate the space captured by the line sensor cameras 10a and 10b, for example at night.

While the railway vehicle 5 is traveling, the line sensor cameras 10a and 10b each capture an image of the space including the overhead train line 20 to sequentially generate first image data and second image data. The generated first image data and second image data are input to the captured image generating device 3. The captured image generating device 3 generates an original captured image, which is a single two-dimensional image captured by the line sensor camera 10a, by combining a predetermined number of lines (e.g., 1000 lines) of the inputted first image data. Similarly, the captured image generating device 3 generates an original captured image, which is a single two-dimensional image captured by the line sensor camera 10b, by combining a predetermined number of lines (e.g., 1000 lines) of the inputted second image data.

Also, while the railway vehicle 5 is traveling, the laser range sensors 12a and 12b sequentially generate angle data and distance data representing the approximate position of the overhead contact line 20. The generated angle data and distance data are input to the image capture device 3. The image capture device 3 determines the three-dimensional position of the overhead contact line 20 based on the input angle data and distance data. This three-dimensional position is a relative three-dimensional position based on the positions of the laser range sensors 12a and 12b. Since the installation interval between the laser range sensors 12a and 12b and the line sensor cameras 10a and 10b is known, the relative three-dimensional position of the overhead contact line 20 based on the line sensor cameras 10a and 10b can be determined. Therefore, the angle data and distance data sequentially generated by the laser range sensors 12a and 12b can also be said to be data indicating the relative positional relationship between the line sensor cameras 10a and 10b and the overhead contact line 20. Furthermore, this three-dimensional position can be corrected to a more accurate value using image data captured by line sensor cameras 10a and 10b.

Then, by comparing the image capture time by the line sensor cameras 10a and 10b with the scanning time by the laser range sensors 12a and 12b, the original captured image can be associated with data indicating the relative positional relationship between the line sensor cameras 10a and 10b that captured the image and the overhead train line 20 at the time the image was captured.

The original photographed image generated by the photographed image generating device 3 is input to the abnormality detection device 1, which detects abnormalities in the overhead train line fittings by image recognition processing using the original photographed image, together with data indicating the relative positional relationship between the overhead train line 20 and the line sensor cameras 10a and 10b, which are the photographing devices associated with the image.

1) Both the photographic image generating device 3 and the abnormality detection device 1 may be mounted on the railway vehicle 5, 2) the photographic image generating device 3 may be mounted on the railway vehicle 5, and the abnormality detection device 1 may be installed on the ground, such as at a command center, and the data of the original photographic image generated by the photographic image generating device 3 may be stored in a storage device inside the railway vehicle 5 or in a storage device connected to the railway vehicle 5 via communication, and the data stored in the storage device may be input to the abnormality detection device 1 at any time, such as after the operation ends, or 3) both the photographic image generating device 3 and the abnormality detection device 1 may be installed on the ground, such as at a command center, and the data generated by each of the line sensor cameras 10a, 10b and the laser range sensors 12a, 12b (first image data, second image data, angle data, distance data) may be stored in a storage device inside the railway vehicle 5 or in a storage device connected to the railway vehicle 5 via communication, and the data stored in the storage device may be input to the photographic image generating device 3 at any time, such as after the operation ends.

Figure 2 is a flowchart explaining the flow of the train line metal fitting abnormality detection process performed by the abnormality detection device 1. Figure 2 also shows examples of images obtained during the abnormality detection process. Details of each step will be described later.

As shown in FIG. 2, the abnormality detection device 1 first acquires an original photographed image of an overhead electric train line on which electric train line fittings are installed from the photographed image generating device 3 (step S1). Next, a scaling process is performed on the acquired original photographed image (step S3). Details of this scaling process will be described later. Next, a cutout process is performed on the original photographed image after scaling, and a pre-conversion image showing the electric train line fittings is extracted from the original photographed image (step S5). Details of the extraction of this pre-conversion image will be described later. Then, a preparatory image for judgment of a predetermined size is generated by enlarging or reducing the extracted pre-conversion image (step S7). Details of the generation of this preparatory image for judgment will be described later. Furthermore, the size of the preparatory image for judgment is converted to generate a normalized image for image processing (step S9). The normalized image is a square image with the length of one side (number of pixels) N being N=2 ⁿ (n is a natural number). Details of the generation of this normalized image will be described later.

Thereafter, abnormality detection of the electric train line fittings is performed by image recognition processing using the normalized image (step S11). For example, machine learning can be used as the image recognition processing. For example, an algorithm for machine learning can be used that learns only from normal data, since it is difficult to collect a large amount of abnormal data on the electric train line fittings that are the target of abnormality detection. In a general-purpose machine learning algorithm, an input image is converted into a square and a convolution operation is performed, so the normalized image is a square image with the length of one side (number of pixels) N (= ²ⁿ (n is a natural number)).

Figure 3 is a diagram explaining the scaling process (step S3 in Figure 2) for the original photographed image. Figure 3 shows an example of the relative positional relationship between the overhead train line 20 and the imaging device 10 (line sensor camera) installed on the roof of the railway vehicle 5. For ease of understanding, the overhead train line 20 is shown stretched in an extremely zigzag manner. The upper side of Figure 3 shows a top view with the horizontal direction being the direction of train travel, and the lower side shows a front view with respect to the direction of train travel, and also shows the elevation angle θ of the overhead train line 20 from the imaging device 10. The positional relationship between the imaging device 10 and the overhead train line 20 shown in the top view is shown in the front view.

As shown in Fig. 3, the overhead contact wire 20 is stretched in a zigzag pattern in the left-right direction with respect to the center of the track. This is to prevent the pantograph contact strip from being locally worn. For this reason, the relative position between the camera 10 and the overhead contact wire 20 changes depending on the train position. That is, as shown in the left side of Fig. 3, the elevation angle θ when the position of the overhead contact wire 20 coincides with the center of the track (deviation center) is set as the reference elevation angle _θ0 . Then, as shown in the center of Fig. 3, the elevation angle _θ1 when the position of the overhead contact wire 20 is deviated toward the camera 10 (deviation +) is larger than the reference elevation angle _θ0 . Also, as shown in the right side of Fig. 3, the elevation angle _θ2 when the position of the overhead contact wire 20 is deviated away from the camera 10 (deviation -) is smaller than the reference elevation angle _θ0 .

In this way, the relative positional relationship between the camera 10 and the overhead train line 20 varies depending on the train position. In other words, because the distance from the camera 10 to the overhead train line 20 and the elevation angle θ vary, the positions and sizes of the overhead train line 20 and the overhead train line metal fittings vary in the original photographed image captured by the camera 10. In the scaling process, the original photographed image is enlarged or reduced (enlarged and reduced) so that the size of the overhead train line metal fittings in the original photographed image is the same or approximately the same regardless of the train position. The scaling coefficient, which is the enlargement/reduction rate at this time, is given by the following equation (1).
Scaling coefficient=sinθ/sinθ ₀ (1)

In other words, the original photographed image is enlarged or reduced so as to match the size of the overhead contact line metal fittings in the photographed image when the elevation angle θ is the reference elevation angle _θ0 (when the position of the overhead contact line 20 coincides with the center of the track (center of deviation) as shown on the left side of Fig. 3). The elevation angle θ can be found from the relative positional relationship between the imaging device 10 and the overhead contact line 20 when the original photographed image was taken.

Fig. 4 is a diagram for explaining an example of how to determine the elevation angle θ. Fig. 4 shows an example of the relative positional relationship between the camera 10 and the overhead contact wire 20 in a front view in the direction of train travel. The elevation angle θ of the overhead contact wire 20 from the camera 10 is determined by the following formula (2) using the contact wire height (trolley wire height) L1 from the rail level, the camera height (camera height) L2 from the rail level, and the position h of the camera 10 from the track center toward the sleepers (horizontal direction).
θ=tan ⁻¹ {(L1−L2)/h} (2)

Figure 5 is a diagram illustrating the cropping process (step S5 in Figure 2) for extracting a pre-conversion image from an original photographed image. Figure 5 shows an example of an original photographed image and an extracted pre-conversion image. The original photographed image is an image of the overhead train line 20 viewed from diagonally below. In Figure 5, the up-down direction (vertical direction) of the original photographed image is the direction along the train's travel direction, and the left-right direction (horizontal direction) is the direction along the width of the train.

The original photographed image shows the first wire 21 and the second wire 22, which are the two wires that make up the overhead train line 20. The train line fitting 30, which is provided between the two wires 21 and 22, consists of a first engaging portion 31 that engages with the first wire 21, a second engaging portion 32 that engages with the second wire 22, and an intermediate portion 33 that connects the engaging portions.

In the cutout process, a rectangular pre-conversion image is extracted from the original captured image so that the electric train line fitting 30 is captured and the first wire 21 and the second wire 22 are captured at predetermined positions in the image. Because the electric train line fitting 30 has an elongated shape, the pre-conversion image is also an elongated rectangular image (in FIG. 5, it is a horizontally long rectangular image).

Specifically, the long side of the pre-conversion image (the longitudinal direction of the electric train line fitting 30; the horizontal direction in FIG. 5) is set to a range outside each of the two wires 21, 22, which are captured as substantially straight lines, and extending a predetermined length (N/2 in terms of the number of pixels) from the wires 21, 22. The short side of the pre-conversion image (the direction perpendicular to the longitudinal direction of the electric train line fitting 30; the vertical direction in FIG. 5) is set to a range whose length (number of pixels) is equal to the length of one side of the normalized image (N (= ²ⁿ ) in terms of the number of pixels) and whose central position is the position of the middle part 33 of the electric train line fitting 30. Note that the relative positional relationship between the imaging device 10 and the overhead electric train line 20 differs depending on the train position, and therefore the length between the wires 21, 22 in the pre-conversion image differs, and therefore the length of the long side of the pre-conversion image is not constant.

Figure 6 is a diagram explaining the generation of a preparatory image for judgment and a normalized image from a pre-conversion image (steps S7 to S9 in Figure 2). First, the pre-conversion image is divided into three ranges: a first engagement portion range in which the first engagement portion 31 of the electric train line fitting 30 is captured, a second engagement portion range in which the second engagement portion 32 is captured, and an intermediate range in which the intermediate portion 33 is captured. Specifically, a square-shaped range at one end of the elongated rectangular pre-conversion image and including the first line 21 is defined as the first engagement portion range, and a square-shaped range at the other end and including the second line 22 is defined as the second engagement portion range. Then, the range of the pre-conversion image other than the first engagement portion range and the second engagement portion range is defined as the intermediate range. In the pre-transformation image, the first line 21 and the second line 22 are located N/2 pixels from the ends, so the first line 21 is located approximately in the center of the first engagement portion range, and the second line 22 is located approximately in the center of the second engagement portion range.

Then, the intermediate range is enlarged or reduced (scaled) in the longitudinal direction of the electric train line metal fittings 30 and converted into a square image to generate a preliminary image for determination. In other words, the intermediate range is scaled so that its longitudinal length (number of pixels) is the same as its lateral length (number of pixels N). In the example of FIG. 6, the intermediate range is compressed in the longitudinal direction at a compression rate of 52.5% so that its longitudinal length of 244 pixels becomes 128 pixels, and converted into a square image. As described above, because the length (number of pixels) of the long side of the image before conversion is not constant, the longitudinal length (number of pixels) of the intermediate range is not constant either, and the compression rate is not constant either.

In this way, the preparatory image for judgment is an image in which the three ranges, the first engagement partial range, the second engagement partial range, and the intermediate range, are the same size. In other words, the preparatory image for judgment of a predetermined size is generated by making the allowable change rate of the enlargement/reduction for the pre-conversion image larger for the intermediate range than for the first engagement partial range and the second engagement partial range, which are ranges other than the intermediate range. Here, the preparatory image for judgment of a predetermined size is generated by enlarging/reducing only the intermediate range without enlarging/reducing the first engagement partial range and the second engagement partial range, but it may be enlarged/reduced for the first engagement partial range and the second engagement partial range. Since enlarging/reducing an image impairs the information of the original image, it is better to keep the original image as it is, that is, the size (number of pixels) as it is, in order to increase the accuracy of subsequent image processing. In the example of FIG. 6, the image information of the first engagement partial range and the second engagement partial range is stored without impairing in the preparatory image for judgment, but the image information of the intermediate range is impairing in the preparatory image for judgment.

Then, the preparatory image for judgment is compressed in the longitudinal direction of the electric train wire fitting 30 to convert the size, thereby generating a square-shaped normalized image. The normalized image is an image used in image recognition processing for abnormality detection, and has a fixed image size. Therefore, the size of the preparatory image for judgment is converted to generate the normalized image. Since each of the first engagement portion range, the second engagement portion range, and the intermediate range that constitute the preparatory image for judgment is a square-shaped image, the normalized image is generated by compressing each of these three ranges to 1/3 in the longitudinal direction. Since the entire image is compressed at a predetermined compression rate, it can be said that the image information of the first engagement portion range and the second engagement portion range preserves more information of the original captured image than the intermediate range. The preparatory image for judgment may be used as the normalized image without being enlarged or reduced.

[Functional configuration]
Fig. 7 shows an example of the functional configuration of the abnormality detection device 1. According to Fig. 7, the abnormality detection device 1 is configured to include an operation unit 102, a display unit 104, a sound output unit 106, a communication unit 108, a processing unit 200, and a storage unit 300, and is realized as a type of computer.

The operation unit 102 is realized by an input device such as a keyboard, mouse, touch panel, various switches, etc., and outputs an operation signal corresponding to the operation input to the processing unit 200. The display unit 104 is realized by a display device such as a liquid crystal display or touch panel, and performs various displays based on a display signal from the processing unit 200. The sound output unit 106 is realized by an audio output device such as a speaker, and performs various audio outputs based on an audio signal from the processing unit 200. The communication unit 108 is a communication device realized by a wireless communication module, router, modem, jack of a wired communication cable, control circuit, etc., and performs data communication with an external device.

The processing unit 200 is a processor realized by an arithmetic device or arithmetic circuit such as a CPU (Central Processing Unit) or an FPGA (Field Programmable Gate Array), and performs overall control of the anomaly detection device 1 based on programs and data stored in the memory unit 300, input data from the operation unit 102 and the communication unit 108, etc. The processing unit 200 also has, as functional processing blocks, a captured image acquisition unit 202, a pre-conversion image extraction unit 204, a division unit 206, a determination preparation image generation unit 208, a normalized image generation unit 210, and an anomaly detection unit 212. Each of these functional units possessed by the processing unit 200 can be realized in software by the processing unit 200 executing a program, or can be realized by a dedicated arithmetic circuit. In this embodiment, the former software realization will be described.

The photographed image acquisition unit 202 acquires an original photographed image of the overhead train line 20 (step S1 in FIG. 2). The original photographed image can be acquired, for example, from the photographed image generating device 3 (see FIG. 1). The acquired original photographed image is stored as original photographed image data 320 in association with data indicating the relative positional relationship between the photographing device 10 that captured the image and the overhead train line 20 at the time the image was captured.

The pre-conversion image extraction unit 204 performs scaling and cropping of the original photographed image of the overhead train line 20 to extract a pre-conversion image from the original photographed image (steps S3 to S5 in FIG. 2). The pre-conversion image extraction unit 204 performs scaling and cropping so that the first engagement portion range and the second engagement portion range satisfy a predetermined size condition. The original photographed image is associated with data indicating the relative positional relationship between the photographing device 10 that captured the image and the overhead train line 20, and the pre-conversion image extraction unit 204 performs scaling and cropping using the data indicating this relative positional relationship. The pre-conversion image extraction unit 204 also performs cropping so that the first line 21 and the second line 22 are captured in predetermined positions in the pre-conversion image.

Specifically, the pre-conversion image extraction unit 204 calculates the elevation angle θ from the photographing device 10 to the overhead train line 20 from the data of the relative positional relationship between the photographing device 10 and the overhead train line 20 when the image associated with the original photographed image was photographed, calculates a scaling coefficient according to formula (1) based on the elevation angle θ, and performs scaling by enlarging or reducing the original photographed image with the scaling coefficient as the enlargement or reduction ratio (see Figs. 3 and 4). Next, a cut-out process is performed to cut out the pre-conversion image in a range in which the train line metal fittings 30 are captured from the original photographed image after scaling, where the two lines (first line 21 and second line 22) engaged with the two engaging parts (first engaging part 31 and second engaging part 32) are located at a position a predetermined length (number of pixels N/2) from the end of the image, and the middle part 33 is located in the center (see Fig. 5).

The division unit 206 divides the pre-conversion image into multiple ranges along the longitudinal direction of the electric train line fitting 30: a first engagement portion range in which the first engagement portion 31 is captured, an intermediate range in which the intermediate portion 33 is captured, and a second engagement portion range in which the second engagement portion 32 is captured.

Specifically, the division unit 206 divides the pre-conversion image by defining the first and second engagement partial ranges as square-shaped ranges, and defining the range other than the first and second engagement partial ranges as an intermediate range (see FIG. 6).

The determination preparation image generating unit 208 generates a determination preparation image of a predetermined size by scaling the pre-conversion image at least along the longitudinal direction of the train line fittings 30 (step S7 in FIG. 2). At this time, the determination preparation image of a predetermined size is generated by scaling the intermediate range with a larger allowable change rate for scaling than the ranges other than the intermediate range. In this embodiment, the determination preparation image of a predetermined size is generated by scaling only the intermediate range without scaling the first engagement portion range and the second engagement portion range. Specifically, the determination preparation image is generated by scaling the intermediate range to make it square (see FIG. 6).

The normalized image generating unit 210 generates a normalized image for image recognition processing by converting the size of the determination preparation image (step S9 in FIG. 2). Specifically, the first engagement portion range, the second engagement portion range, and the intermediate range that constitute the determination preparation image are compressed in the longitudinal direction of the train line metal fittings at a 1/3 expansion/reduction ratio to generate a square normalized image (see FIG. 6). The generated normalized image is stored as normalized image data 330.

The anomaly detection unit 212 detects anomalies in the electric train line fittings 30 by image recognition processing using the normalized image (step S11 in FIG. 2). Specifically, machine learning is used as the image recognition processing. That is, the normalized image is input to a two-class classification anomaly detection model that distinguishes between normal and abnormal electric train line fittings 30, and anomalies in the electric train line fittings 30 shown in the pre-conversion image are detected from the output. This anomaly detection model is a machine learning model trained using previously prepared abnormal data (normalized image generated from a pre-conversion image showing an abnormal electric train line fitting 30) and normal data (normalized image generated from a pre-conversion image showing a normal electric train line fitting 30), and parameters that define the anomaly detection model are stored as anomaly detection model data 310.

The storage unit 300 is realized by IC (Integrated Circuit) memory such as ROM (Read Only Memory) or RAM (Random Access Memory), or a storage device such as a hard disk or SSD (Solid State Drive), and stores programs and data for the processing unit 200 to comprehensively control the anomaly detection device 1, and is used as a working area for the processing unit 200 to temporarily store the results of calculations performed by the processing unit 200, input data from the operation unit 102 and communication unit 108, etc. In this embodiment, the storage unit 300 stores an anomaly detection program 302, anomaly detection model data 310, original captured image data 320, and normalized image data 330.

The anomaly detection program 302 is a program for implementing the anomaly detection process (see FIG. 2) executed by the processing unit 200.

[Experimental Example]
The following describes the results of an experiment on anomaly detection using the anomaly detection device 1 of this embodiment. Fig. 8 shows an example of a normalized image (image to which this embodiment is applied) used for anomaly detection. The image (1) on the left is a normalized image generated from a pre-transformation image in which a normal electric train wire fitting is shown, and the image (2) on the right is a normalized image generated from a pre-transformation image in which an abnormal electric train wire fitting is shown.

Figure 9 shows an example of an image for comparison (comparison image). The comparison image is a square image generated by uniformly reducing the length of a pre-transformation image showing a train line fitting. The image on the left (1) is a comparison image generated by uniformly reducing the pre-transformation image showing a normal train line fitting used to generate the normalized image in Figure 8 (1), and the image on the right (2) is a comparison image generated by uniformly reducing the pre-transformation image showing an abnormal train line fitting used to generate the normalized image in Figure 8 (2).

When comparing the image of this embodiment with the image of the comparative example, the area of engagement with the stripes appears larger in the image of this embodiment than in the image of the comparative example. This is because the image of this embodiment shows a larger allowable rate of change in longitudinal expansion/contraction in the middle range than in the area of the engaged portion.

Figures 8 and 9 show examples of images to which this embodiment is applied and comparative example images for each of the two pre-transformed images, but 25 pre-transformed images were similarly prepared, and the results of anomaly detection using these pre-transformed images are shown in Figure 10.

Figure 10 (1) is a graph showing the results of anomaly detection for images to which this embodiment is applied, and Figure 10 (2) is a graph showing the results of anomaly detection for images of a comparative example. In this experiment, AnoGAN was used as the machine learning algorithm, and a numerical value (indicated as "Total Loss" in Figure 10) indicating the difference from normal data was output as the result of anomaly detection. The larger the output numerical value, the higher the degree to which the input data can be considered abnormal. Of the 25 pre-conversion images, 20 were normal data and 5 were abnormal data. In both graphs of Figures 10 (1) and (2), the horizontal axis represents the numerical value that is the result of anomaly detection, and the vertical axis represents the frequency (the number of images that correspond).

Comparing the graphs in Figures 10 (1) and (2), in the application example of this embodiment, the numerical values resulting from anomaly detection are significantly different between the group of abnormal data (normalized images generated from pre-conversion images showing abnormal train line fittings) and the group of normal data (normalized images generated from pre-conversion images showing normal train line fittings). Therefore, by setting an appropriate threshold value for the numerical values, it is possible to accurately detect abnormality and normality. In contrast, in the comparative example, the numerical values resulting from anomaly detection are partially overlapping between the group of abnormal data and the group of normal data, and the groups partially overlap with each other. For this reason, it is difficult to set a threshold value to clearly distinguish between abnormality and normality, and it can be seen that the detection accuracy is low.

[Action and Effect]
Thus, according to the present embodiment, it is possible to improve the accuracy of detecting abnormalities in the electric train line fittings 30 by performing image processing on the captured image showing the electric train line fittings 30. Many electric train line fittings 30, such as hangers, droppers, and connectors, have an elongated shape. In the pre-conversion image, the longitudinal length of the electric train line fittings 30 is often longer in the intermediate range in which the intermediate portion 33 connecting the engaging portions 31, 32 with the wires 21, 22 is shown than in the range in which the engaging portions 31, 32 are shown. Also, in the electric train line fittings 30, the probability of abnormalities occurring is higher in the engaging portions 31, 32 than in the intermediate portion 33. For this reason, when the pre-conversion image is enlarged or reduced at least along the longitudinal direction of the electric train line fittings 30, the allowable change rate of enlargement or reduction along the longitudinal direction is set to be larger in the intermediate range in which the intermediate portion is shown than in the range in which the engaging portions of the electric train line fittings 30 are shown. In this way, in the generated preliminary image for determination, the range in which the engagement parts 31, 32, which have a high probability of occurrence of an abnormality, are captured is relatively large compared to the intermediate range in which the intermediate part 33, which has a low probability of occurrence, is captured, which is advantageous in terms of size and resolution. This makes it possible to improve the accuracy of abnormality detection by image recognition processing using the preliminary image for determination.

[Modification]
Incidentally, the applicable embodiments of the present invention are not limited to the above-described embodiments, and can of course be modified as appropriate without departing from the spirit of the present invention.

(A) Normalized image is made into a multi-channel image For example, a normalized image for image recognition processing may be generated by generating a multi-channel image in which the image related to the first engagement part range, the image related to the intermediate range, and the image related to the second engagement part range among the judgment preparation images are each made into a single channel image. For example, three square-shaped images ((3) judgment preparation image in FIG. 6) related to the first engagement part range, the second engagement part range, and the intermediate range constituting the judgment preparation image are each converted into a single channel image of RGB (red, green, blue) while keeping the size as it is. Then, the three images are synthesized into one color image (multi-channel image). This color image is made into a normalized image.

1: Anomaly detection device 200: Processing unit 202: Photographed image acquisition unit 204: Pre-conversion image extraction unit 206: Division unit 208: Judgment preparation image generation unit 210: Normalized image generation unit 212: Anomaly detection unit 300: Storage unit 302: Anomaly detection program 310: Anomaly detection model data 320: Original photographed image data 330: Normalized image data 3: Photographed image generation device 5: Railway vehicle 10 (10a, 10b)... Line sensor camera (photographing device)
12 (12a, 12b)... Laser range sensor 14 (14a, 14b)... LED lighting 20... Overhead train line 21... First wire 22... Second wire 30... Train line fitting 31... First engagement portion 32... Second engagement portion 33... Intermediate portion

Claims (9)

an abnormality detection device that converts a pre-conversion image showing an overhead contact line on which an overhead contact line fitting is provided so as to include an overhead contact line fitting having a first engagement portion that engages with a first wire, a second engagement portion that engages with a second wire, and an intermediate portion that connects the first engagement portion and the second engagement portion, into a preparatory image for determination of a predetermined size, and then performs an abnormality detection process for the overhead contact line fitting by image recognition processing,
a division means for dividing the pre-conversion image into a plurality of ranges in a direction along a longitudinal direction of the electric train wire fitting, the division means dividing the range in which the intermediate portion is shown (hereinafter, this range will be referred to as an "intermediate range") as one range;
a preparatory image for determination generating means for generating the preparatory image for determination of the predetermined size by enlarging or reducing the pre-conversion image at least along a longitudinal direction of the electric train wire fitting, the preparatory image for determination generating means generating the preparatory image for determination of the predetermined size by enlarging or reducing the preparatory image for determination by setting an allowable change rate of the enlargement or reduction to be larger in the intermediate range than in a range other than the intermediate range;
An abnormality detection device comprising: the dividing means divides the pre-conversion image into a first engagement portion range in which the first engagement portion appears, the intermediate range, and a second engagement portion range in which the second engagement portion appears;
The abnormality detection device according to claim 1 . the determination preparation image generating means generates the determination preparation image of the predetermined size by enlarging or reducing only the intermediate range without enlarging or reducing the first engagement portion range and the second engagement portion range.
The abnormality detection device according to claim 2 . a pre-conversion image extraction means for extracting the pre-conversion image from an original photographed image obtained by photographing the overhead train line by performing scaling processing and cropping processing, the pre-conversion image extraction means performing the scaling processing and the cropping processing so that the first engagement partial range and the second engagement partial range satisfy a predetermined size condition;
The abnormality detection device according to claim 2 or 3, further comprising: the pre-conversion image extraction means performs the extraction process so that the first line and the second line are captured at predetermined positions in the pre-conversion image.
The abnormality detection device according to claim 4. The original photographed image is associated with data indicating a relative positional relationship between the photographing device that photographed the image and the overhead train line,
the pre-conversion image extraction means performs the scaling process and the cut-out process using the data indicating the relative positional relationship.
The abnormality detection device according to claim 4 or 5. A normalized image generating means for generating a normalized image for the image recognition processing by converting the size of the judgment preparation image;
The abnormality detection device according to any one of claims 2 to 6, further comprising: a normalized image generating means for generating a multi-channel image by converting the image relating to the first engagement portion range, the image relating to the intermediate range, and the image relating to the second engagement portion range, among the determination preparation images, into single-channel images, thereby generating a normalized image for the image recognition processing;
The abnormality detection device according to any one of claims 2 to 6, further comprising: 1. An anomaly detection method for detecting an anomaly in an overhead contact line, the method comprising: converting a pre-conversion image showing an overhead contact line provided with an overhead contact line fitting, the overhead contact line fitting including a first engagement portion that engages with a first wire, a second engagement portion that engages with a second wire, and an intermediate portion that connects the first engagement portion and the second engagement portion, into a preparatory image for determination of a predetermined size, and then performing an anomaly detection process for the overhead contact line fitting by image recognition processing,
a division step of dividing the pre-conversion image into a plurality of ranges in a direction along a longitudinal direction of the train rail metal fitting, the division step dividing the range in which the intermediate portion is captured (hereinafter, this range is referred to as an "intermediate range") as one range;
a preparatory image for determination generating step of generating the preparatory image for determination of the predetermined size by enlarging or reducing the pre-conversion image at least along a longitudinal direction of the electric train wire fitting, in which the preparatory image for determination of the predetermined size is generated by enlarging or reducing the allowable change rate of the enlargement or reduction in the intermediate range larger than in a range other than the intermediate range;
The anomaly detection method includes:

Images