RU2815254C1 - Method for automatic detection of missing elements of freight car in round-the-clock mode - Google Patents

Abstract

FIELD: control devices; optics; data processing.

SUBSTANCE: invention relates to methods of optical control of rolling stock. In the method, the pre-trained neural network is launched in the cycle of image processing from the cameras to detect the specified elements of the car, detecting the required car elements, determining the car number, after which the obtained car number is queried into the car database, obtaining a list and the number of elements the presence of which must be monitored in a given car; further, by comparing this list with the compiled set, it is determined whether all the car elements to be controlled are found on the car images; if there are missing or inadequate elements, they are entered into the list of faults of this car.

EFFECT: enabling detection of missing elements of a freight car.

1 cl, 13 dwg

Description

The invention relates to automatic means of optical monitoring of rolling stock and can be used for automatic registration and recognition of numbers of railway cars and tanks at railway junctions, transit stations, entrances to unloading/loading points, as well as for counting the number of cars, monitoring the condition of cars and tanks, also ensure that the type of brake pad matches the model of the car, and identify the absence of elements of the freight car.

From the prior art, a solution is known according to patent RU2508216, publication: 2014.02.27, which describes an automatic car number identifier, characterized in that it contains a GLONASS/GPS navigation receiver, a non-volatile real-time clock, a GSM/GPRS modem with two SIM cards, controller, non-volatile memory, controller launch unit when external power appears or the car starts moving, power module and autonomous power supply, and a GLONASS/GPS navigation receiver, non-volatile real-time clock, GSM/GPRS modem are connected to the controller, controller launch unit when external power appears power supply or at the beginning of the movement of the car and non-volatile memory, and the power module from the on-board electrical network and the autonomous power supply are connected independently of each other to all energy-consuming elements of the identifier.

The disadvantage of the solution is the dependence on the presence of an autonomous power supply, which, in severe frost conditions, quickly discharges after a short time. In addition, not everywhere in Russia there is cellular communication and in taiga zones

The solution https://controleng.ru/wp-content/uploads/4960.pdf is also known, which describes the HARDWARE-SOFTWARE COMPLEX OF OPTOELECTRONIC IDENTIFICATION OF RAILWAY TRANSPORT ROLLING STOCK OBJECTS, which allows you to automatically recognize the numbers of freight cars of railway trains. The first implementation took place in June 2005 at the station. Cherepovets-1 Northern Railway. The development is called ARSCIS (Automated RollingStocked Cars Identification System).

The closest analogue is the SOK-TEG optical monitoring system for freight cars and tanks (https://teg.ru.com/wp-content/uploads/2020/06/sok-teg.pdf, published: June 2020), which is designed for automatic registration and recognition of numbers of railway cars and tanks at railway junctions, transit stations, entrances to unloading/loading points, as well as for registration of numbers of transport containers moved by rail in order to obtain operational information in automatic mode about the movement of rolling stock railway facilities, searching for cars by their inventory numbers, counting the number of cars, monitoring the condition of cars and tanks.

The technical problem of the known solutions is the ability to only count the cars and record their numbers.

The objective of the invention is to improve known systems.

The technical result of the invention is the ability, during the movement of rolling stock, in addition to recognizing car numbers, analyzing and generating the received data taking into account the car model, to also identify the absence of necessary elements of a freight car or determine their inconsistency.

The specified technical result is achieved due to the fact that a method is claimed for automatically identifying missing elements of a freight car around the clock, characterized by the use of a control post equipped with video cameras and lights for round-the-clock recording, with the help of which images from the cameras are automatically loaded during the passage of the train to the processing server, where images of cars are formed from different angles, car numbers are recognized, data is generated taking into account the model of the car, characterized in that video cameras with lights at the control post are placed on both sides of the railway track in height at the level of the necessary elements of the freight car, that need to be identified, and the method is carried out in the following sequence of steps:

- using a pre-trained neural network, they start processing all images from all cameras in a loop to detect specified elements of the car, and after processing the image, a list of objects is formed from the specified elements of the car, with each object being described by the same numbers as in the markings on the image;

- based on the results of processing all images for each type of object, a data table is created, consisting of rows and columns, where the elements of the rows are cars, and the elements of the columns are: camera, image number from this camera, time of formation of this image, location of the element in the image;

- analyzing the resulting data table for the automatic coupling element of the car, and, having the frame numbers for each of the automatic couplers, select for each car a set of photographs from each of the cameras, forming for each of the cars a data array from sets of three numbers: camera number, number of the first photo for of this car from this camera, the number of the last photo for this car from this camera; wherein the number of such sets corresponds to the number of cameras used;

- by comparing the data table and the data array, a compiled set and the number of elements that were detected for a particular car are obtained;

- at the next stage, for each of the cars, photographs are selected from cameras configured to recognize the car number, this set of photographs is loaded for compilation into the program for recognizing eight-digit numbers of railway cars, at the output they receive the car number, and if the car number is not recognized, then they are identified by it a mark that the car was not processed;

- after which they make a request for the received car number to the car database, receiving a list and number of elements whose presence on a given car needs to be monitored;

- further, comparing this list with the compiled set, it is determined whether all the elements of the car that need to be monitored are found in the images of the car; if there are missing or inappropriate elements for the car, they are included in the list of faults of this car.

Preferably, upon completion of processing of the train, a list of unprocessed cars, if any, is generated, indicating the serial number from the head, a list of cars with missing elements of interest, indicating the car number and a list of not found elements, and a list of car numbers for which all elements were found.

Brief description of drawings

In FIG. 1 shows a diagram of a control post with installed video cameras according to the claimed invention.

In FIG. 2 - Fig. 5, Fig. 9 - Fig. Figure 11 shows examples of detecting car elements.

In FIG. 6 and Fig. Figure 7 shows examples of marking using the LabelImg program.

In FIG. Figure 8 shows an example of car number detection.

In FIG. 12 and Fig. 13 shows examples of determining the type of brake pad.

In the drawings: 1 - control post, 2 - video cameras at the level of the car number, 3 - video cameras at the level of the car's automatic lining, 4 - video cameras at the level of the axle box of the car's wheelset, 5 - auxiliary post, 6 - axle box unit, 7 - automatic lining of the car, 8 - video cameras 8 to monitor the contents of cars.

Carrying out the invention

The method is designed to automatically identify missing elements of a freight car around the clock, recognize car numbers, analyze and generate the received data taking into account the car model, as well as transfer the received readings to the customer’s information systems.

The method can be implemented using video recording modules installed at control post 1 and connected wired or wirelessly to an operator’s automated workstation (AWS), where a server with the appropriate software is installed. Video cameras 2, 3, 4 with illuminators 5 (for operation in night mode) at control post 1 are placed in height at the level of the necessary elements of the freight car that need to be identified.

For example, cameras 2 are installed at the level of the car number on both sides of the railway track on the supports of control post 1, cameras 3 - at the level of the car's automatic lining 7, cameras 4 - at the level of the axle box unit 6 of the car's wheel pair. If necessary, control post 1 is additionally equipped with auxiliary pillars 5, on which the same video cameras 2, 3, 4 are placed at the same level. Also, video cameras 8 can be placed on the top beam of control post 1 to monitor the contents of cars. The location of cameras 2, 3, 4 at a given height relative to the elements of the car is necessary for the purpose of accurately identifying the elements of the car, so that they are not at a large angle of inclination from above or below, which greatly distorts the identified object and complicates its recognition.

When a train passes through control post 1, which is equipped with at least video cameras 2, 3, 4 with illuminators (can be either separate or built into cameras) for round-the-clock recording, images of the car are automatically generated from different angles. Specified mathematical algorithms using neural networks process the results of searching for objects in images of cars. The data analysis algorithm allows you to recognize car numbers, generate data taking into account the car model, determine the correspondence of the type of brake pad to the car model, and identify the absence of elements of a freight car.

The main stages of the program for identifying missing elements on a car are characterized by the following stages:

After the next train passes, images from the cameras are uploaded to the processing server, which is connected to video cameras 2, 3, 4, 8 by wire or wireless connection. The processing server is installed on the operator's workstation. The server has the appropriate software installed, in particular neural network software.

At the second stage, a neural network is used. For example, you can use the detect script from the YOLO library, which starts processing all images from all cameras in a loop in order to detect elements of interest to us.

To identify missing or inappropriate elements of the car, the neural network must first be trained.

To do this, a set of images with objects of interest to us is pre-selected.

Next, for example, in the YOLO neural network, the so-called marking procedure is performed - for each image <image_name>.jpg, a file <image_name>.txt is created, which lists the elements present in this image and the parameters of the rectangles within which these elements are located.

To simplify this procedure, you can use, for example, the LabelImg program. It correlates the detected object with its location in the image.

Figure 6 shows an example of marking using the LabelImg program,
the contents of the markup file are as follows, as shown in Table 1:

Table 1

0 0.276562 0.395625 0.189062 0.218750 6 0.414062 0.453125 0.190625 0.313750 3 0.695703 0.185000 0.116406 0.250000 2 0.893359 0.204375 0.077344 0.241250 4 0.732812 0.610000 0.104688 0.192500

In Table 1, the first column is the element's numeric ID, the second and third columns are the relative coordinates of the center of the bounding box, and the fourth and fifth columns are the relative sizes of the bounding box.

After selecting a sufficient number of images (at least 1000 occurrences of each element) and marking them using the train.py script from the YOLO library, the neural network is trained.

An array of more than 1000 pictures is prepared for training.

Using the YOLOv5 library and a pre-trained weight system, they are retrained for the required objects. That is, they manually look through the pictures, indicate to the neural network what each object intended for search looks like and is called. The train program is used for image processing.

To mark coordinates, use the LabelImg program (see https://russianblogs.com/article/24391005208/). It correlates the detected object with its location in the image.

The detect function processes all images and generates a list: photo name - object name, coordinate; name of object 2 - coordinate 2; etc.

The result is such images with the designation of detected objects as in Fig.7.

Examples of detailed configuration of a neural network for object detection are shown, for example, in https://newtechaudit.ru/nastrojka-nejroseti-detekczia-obektov/ or in https://develop-nil.com/kak-sozdat-svoj-sobstvennyj-object- detector/.

An example of detection is shown in Figure 5, where two hooks (lhook, rhook), a steering wheel (helm), an axle box (oldbox) 6 and an automatic coupler (coupler) 7 were found. The result of image processing is a list of objects, with each object described by the same numbers (numeric identifier of the element, relative coordinates of the center of the bounding box, relative sizes of the bounding box), as in markup.

Example 1 of the list entry result for the given image:

[[2,0.828516,0.164375,0.0976562,0.24375],[0,0.214844,0.4175,0.2,0.21],[4,0.598047,0.14375,0.122656,0.2675],[6,0.677734,0.621875 ,0.110156,0.18875],[ 7,0.317578,0.523125,0.180469,0.27875]].

Based on the results of processing all images within the program, for each type of object, a data table is created, consisting of rows and columns, where the elements of the rows are cars, and the elements of the columns are: camera, image number from this camera, time of formation of this image, location of the element in the image.

Analyzing the array from the previous paragraph for the automatic coupler element 7, and having the frame numbers for each of the automatic couplers 7, a set of photographs from each of the cameras is selected for each car, forming for each of the cars an array of sets of three numbers (camera number, number of the first photo for of a given car from a given camera, the number of the last photo for a given car from a given camera). The number of such triplets corresponds to the number of cameras. As a result, as a result of performing the functions from the above steps, for each of the elements a set of photo numbers is obtained, where it was found, and for each of the cars - a set of photo numbers related to this car.

By comparing the data table and the data array, a compiled set and the number of elements that were detected for a particular car are obtained.

For examples of working with YOLOv5 models, see https://vc.ru/newtechaudit/326571-ispolzovanie-yolov5-dlya-zadachi-detekcii, as well as https://habr.com/ru/articles/576738/).

Thanks to the use of a neural network, it is possible to detect, that is, “view” the image once, which significantly speeds up data processing to obtain the result.

Examples of searching for objects in images of cars using neural networks are shown in Figure 2-Figure 5.

A set of images for each of the cars is processed using the iANRCR Pyhton library (also implemented based on the YOLO architecture, see: https://intbusoft.com/ianrcr/) in order to recognize the car number (an example of such number recognition is shown in Fig. 8) .

At the next stage, for each of the cars, photographs are selected from cameras configured to recognize the car number, this set of photographs is loaded for compilation into a program for recognizing eight-digit numbers of railway cars, and at the output they receive the car number, and if the car number is not recognized, then a mark is placed on it that the car was not processed.

As a program for recognizing eight-digit numbers of railway cars, you can use the development kit for recognizing numbers of railway cars iANRCR SDK, for example: https://intbusoft.com/2023/01/ianrcrpython/. iANRCR SDK is a development kit for recognizing eight-digit railway car numbers. The main goal is to provide automated license plate recognition based on the OpenCV computer vision library. The main language for using the library is C/C++. Version 2.0.1.0 is compiled with OpenCV 3.4 versions.

For example, photographs from cameras configured to recognize the car number are passed to the ia.process function from the iANRCR library and the car number is received as output. If the car number is not recognized, then it indicates that the car was not processed. An example of how the iANRCR SDK works is described in https://intbusoft.com/wp-content/uploads/2021/01/iANRCR.pdf.

By making a request for the received car number in the customer’s database, they receive a list and number of elements whose presence on this car is important to the customer.

Thus, the iANRCR SDK library determines the car number in accordance with Russian Railways car marking standards. The recognized number (with its probability) is assigned to an already recognized array of objects. After which it is no longer tied to the serial number, but to the inventory number of the car.

Example 1 of determining the car number.

The iANRCR SDK software package finds a line in Excel using the car number and reads the entire line.

If the car number is not recognized in its entirety (in the 8-digit Russian Railways format), then the algorithm stops.

For example, in Table 2 below you can see that the row was found and selected by a command using an SQL query like: SELECT * FROM Wagon WHERE number=61621843

The software makes a request from the database automatically. The car database can be stored both on the operator's workstation server and on any other server, from where it can be uploaded via the Internet to the operator's workstation server. The database is formed on the basis of data provided by the customer.

The database is represented by the Wagon table with the following fields: number-number of the wagon, type-type of the wagon (gondola car, tank, platform, etc.). Next is a list of elements (expanded as needed and the capabilities of the neural network) for which the neural network is trained to find in photographs. If the value of this field is zero, then there is no need to monitor the presence of this element on the car. A non-zero field value is the number of elements of this type, the presence of which must be monitored on this car.

Example 2.

From Table 3 it can be seen that car number 61621843 is subject to control according to all criteria of elements, and car 50016260 has no hooks and is not checked.

Table 3

number type helm hook break_cyl double_chamber_tank spare_tank 61621843 box 1 24 1 1 1 . . . 50016260 tank 1 0 1 1 1

Next, by comparing the resulting list and the number of elements with the compiled set, it is determined whether all the elements of the car that need to be monitored are found in the images of the car. If there are missing or inappropriate elements for the car, they are included in the list of faults of this car.

The system must at least detect the absence of the following elements of a freight car (see examples of Fig.9-Fig.11):

- parking brake (steering wheel);

- auto mode;

- auto mode support beam;

- brake pad;

- hatch cover locking device;

- spring (external for springs visible from the side of the car);

- auto regulator;

- blocker;

- shunt.

The found discrepancy is highlighted in color in the images.

To determine the match, subtract the number of searched elements from the number of elements found. If the resulting value is greater than or equal to 0, then the quantity matches. If the number is negative, it does not match.

Example 3 definition of non-conformity.

In Table 2, as well as from a comparison of Fig. 3 (the initial state of the car according to the customer) and Fig. 4 (the control state of the car at the time of inspection), one can see the discrepancy.

From Table 2 for car 50016260 and Fig. 3 it is clear that according to the customer, it should have a helm (helm), since the data indicates the value 1 for this element.

But during the control check (Fig. 4) of the same car 50016260, the helm was not detected, since the data indicates a value of 0 for this element.

As a result, the program records the value 0 in red for the helm element of car 50016260 and records its absence in the log.

Data generation is also carried out taking into account the car model.

After the train passes, the sets of images obtained from the cameras are processed using a neural network.

One of the objects that the neural network detects is automatic coupler 7. Having information in which frames automatic coupler 7 was found, sets of images are identified for each of the cars. Accordingly, it is possible to determine which objects and in what quantity were detected on each of the cars.

There are several models of carriages. And for each there is its own “set” of elements. Such as: steering wheel, brake pad, brake cylinder, two-chamber reservoir, spare tank, automatic coupler 7, ladder, axle box unit 6.

Thus, it is possible to determine whether the brake pad type matches the car model. There are two types of brake pads: long and short. The algorithm determines them by length. If the database indicates that the car is equipped with a long block, but there is a short block, then the operator should know this (the short block is less effective and problems with braking may arise; also, using a long block instead of a short one is impractical). For example, in FIG. 12, a short brake shoe is detected, and in FIG. 13, a long brake shoe is detected.

Upon completion of processing of the train, a list of unprocessed wagons (if any) is generated, indicating the serial number from the head, a list of wagons with missing elements of interest, indicating the wagon number and a list of not found elements, and a list of wagon numbers for which all elements were found.

Claims (2)

1. A method for automatically identifying missing elements of a freight car in a round-the-clock mode, characterized by the use of a control post equipped with video cameras and lights for round-the-clock recording, with the help of which images from the cameras are automatically loaded during the passage of the train to a processing server, where images of the cars are generated from different angles, recognize car numbers, generate data taking into account the model of the car, characterized in that video cameras with lights at the control post are placed on both sides of the railway track in height at the level of the location of the necessary elements of the freight car that need to be identified, while the method is carried out in the following sequence of steps: using a pre-trained neural network, they start processing all images from all cameras in a loop to detect specified elements of the car, and after processing the image, a list of objects is formed from the specified elements of the car, with each object being described by the same numbers as in markings on the image; based on the results of processing all images for each type of object, a data table is created, consisting of rows and columns, where the elements of the rows are cars, and the elements of the columns are: camera, image number from this camera, time of formation of this image, location of the element in the image; Analyzing the resulting data table for the automatic coupling element of the car and having the frame numbers for each of the automatic couplers, a set of photographs from each of the cameras is selected for each car, forming for each of the cars a data array from sets of three numbers: camera number, number of the first photo for this car with of this camera, the number of the last photo for this car from this camera; wherein the number of such sets corresponds to the number of cameras used; by comparing the data table and the data array, a compiled set and the number of elements that were detected for a particular car are obtained; at the next stage, for each of the cars, photographs are selected from cameras configured to recognize the car number, this set of photographs is loaded for compilation into the program for recognizing eight-digit numbers of railway cars, at the output they receive the car number, and if the car number is not recognized, then a mark is placed on it that the car was not processed; after which they make a request for the received car number to the car database, receiving a list and number of elements whose presence on a given car needs to be monitored; further, by comparing this list with the compiled set, it is determined whether all elements of the car that need to be monitored are found in the images of the car; if there are missing or inappropriate elements for the car, they are included in the list of faults of this car. 2. The method according to claim 1, characterized in that upon completion of processing the train, a list of unprocessed cars, if any, is generated, indicating the serial number from the head, a list of cars with missing elements of interest, indicating the car number and a list of unfound elements, and a list of car numbers , for which all elements were found.