CN114529811A - Rapid and automatic identification and positioning method for foreign matters in subway tunnel - Google Patents

Abstract

The invention relates to a method for quickly and automatically identifying and positioning foreign matters in a subway tunnel, which comprises the following steps: two identical cameras are arranged on the robot; acquiring subway tunnel images of two camera monitoring areas; processing the acquired image and judging whether foreign matters exist in the image or not; when the foreign body is detected, judging the type of the foreign body and calculating the position of the foreign body and the distance between the foreign body and the camera; according to the type, position and distance of the foreign matters, the robot grabs the foreign matters. The foreign matter is identified by using a YOLO v3 target identification algorithm. The method has the advantages that the deep network model trained by the YOLO object detection framework is used for real-time foreign matter detection, and compared with the traditional detection algorithm, the deep network model trained by the YOLO object detection framework is more accurate, and meanwhile, the identification process is very rapid, and the target of real-time detection can be achieved.

Description

A fast automatic identification and localization method of foreign objects in subway tunnels

technical field

The invention belongs to the related field of foreign body detection in subway tunnels, and more particularly, relates to a method and system for foreign body detection in subway tunnels.

Background technique

During the normal operation of the subway, foreign objects intrude into the safe driving range of the subway and collide and rub against the subway. Due to the characteristics of foreign body intrusion, such as suddenness, uncertainty, and great harm, once the train conflicts with foreign bodies, it will interrupt the operation at light level, and endanger the traffic safety at worst. Safe operation of subway trains. Therefore, how to detect the intrusion of foreign objects into the boundary in time during the operation of the vehicle, give early warning and alarm before the arrival of the train, and take effective measures in time to avoid the collision between the subway and the intruding object is a very important part to ensure the normal operation of the subway.

At present, the detection of foreign objects in subway tunnels is mainly completed by manual inspection, which not only consumes a lot of manpower and material resources, but also has low efficiency. For the current rapid development of subway rail transit, the subway speed is accelerated and the rail lines are extended, and it is difficult for workers to work in limited capacity. Complete the inspection within time.

For the detection of foreign objects in tunnels with robots as the carrier, a Chinese invention patent specification CN 109131444 A discloses an intelligent detection device for foreign objects in the subway track section. By using a monorail trolley to drive on the track, image recognition is used to judge whether the subway tunnel is intruded by foreign objects. If it is found, it will be reported to the monitoring center; however, the use and maintenance of this method consumes a lot of resources. Once the car encounters a failure, it requires manpower and material resources to check and repair the system. In the Chinese invention patent specification CN108248635 A, an intelligent detection system for rail transit tunnels is disclosed. The real-time monitoring of the inside of the tunnel is carried out by a real-time monitoring intelligent inspection robot installed on the guide rail, and the detected data and information The central control system of the station to which it is transmitted. It is possible to discover potential safety hazards in rail transit tunnels; however, although this method can detect the tunnel status in real time, it cannot identify and grasp foreign objects. In the Chinese invention patent specification CN 109688388 A, a method for all-round real-time monitoring using a tunnel inspection robot is disclosed. After receiving instructions, the tunnel inspection robot monitors the tunnel in real-time and all-round, including illegal driving, traffic accidents, vehicle congestion, etc. In order to detect and deal with the abnormal and illegal situations of the tunnel in time; but this method can realize the identification of foreign objects, but it cannot realize the combination of identification and automatic grasping, and it cannot be identified and grasped at any time. Pick. In the Chinese invention patent specification CN 108657223A, an automatic patrol inspection system for urban rail transit and a method for detecting tunnel deformation are disclosed. By comparing the three-dimensional model of the tunnel after real-time detection and processing with the standard model, the deformation map of the tunnel at the corresponding position is obtained. It can detect various parameters including tunnel equipment and facilities and cable temperature, tunnel deformation, and track damage, and can automatically identify and clean foreign objects in the track, which greatly saves labor costs; the invention invents an automatic inspection robot to The system modules are integrated together, and there is no technical method for automatic identification and positioning of specific foreign objects.

For the existing railway track foreign body intrusion limit detection technology, in the Chinese invention patent specification CN 108549087 A, an on-line detection method based on laser radar is disclosed. The method cannot realize the type identification and timely grasping of foreign bodies. The Chinese utility model patent specification CN207712053 U discloses a robot for rail transportation, which can realize rapid inspection of rail transportation, remove foreign objects on the track, and adopt a special-structured inspection device to realize a clearer inspection of the track. ; but this patent does not identify the part of the system, only describes the working principle and way of the robot. In the Chinese invention patent specification CN 206115276 U, an underground foreign object inspection robot system is disclosed, which includes a robot body, a robot controller, a control terminal, and a power supply. The robot performs inspection and transmits the inspection results to the central control system, which is convenient for The staff can keep abreast of the current situation; this system cannot realize the identification and timely grasp of foreign objects, and it still requires a certain amount of manpower and material resources. In the Chinese invention patent specification CN108197610 A, a deep learning-based track foreign body detection system is disclosed. The track foreign body detection system includes: a vehicle-mounted image acquisition device, an image transmission unit, a foreign body detection device, a machine learning device and image data. Real-time detection of foreign objects on the track is realized; however, the system cannot determine the specific location of the foreign objects and pick up the foreign objects in time.

In summary, domestic research on foreign objects in subway tunnels has achieved certain results, but the technology research and development of real-time identification of foreign object types, location distances, and real-time foreign object pickup using existing robots as a carrier still needs to be explored.

SUMMARY OF THE INVENTION

In view of this, the present invention provides a rapid automatic identification and positioning technology of foreign objects in subway tunnels, which can quickly and effectively identify and locate the type and location of foreign objects automatically, and at the same time, combined with robots, return foreign object data in real time, and realize the grasping of foreign objects by manipulators. Pick.

The present invention adopts the following technical scheme: a method for rapid automatic identification and positioning of foreign objects in a subway tunnel, comprising the following steps:

S1: Set up two identical cameras on the robot;

S2: Obtain the subway tunnel image of the monitoring area of the two cameras;

S3: Process the acquired image, and determine whether there is foreign matter in the image;

S4: When a foreign body is detected, determine the type of the foreign body and calculate the position of the foreign body and the distance from the camera;

S5: According to the type, position and distance of the foreign body, the robot grabs the foreign body.

The S3 includes the following steps:

Any one of the two cameras captures images, and performs image enhancement and filtering processing;

The type and location of foreign objects in the image are judged and predicted by neural network.

The training of the neural network includes the following steps:

(1) Acquire the foreign body image data of the subway tunnel through two cameras in advance. For the images collected by each camera, mark the foreign body, use the label frame containing the foreign body to indicate the position of the foreign body, and use the label name to indicate the type of foreign body;

(2) Establish a YOLO neural network model for foreign object detection, perform YOLO neural network model training for the images collected and labeled by each camera, and obtain a trained YOLO neural network model for real-time monitoring of any camera. Image to identify foreign objects.

The calculation of the distance between the foreign object and the camera uses binocular ranging to calculate the distance between the foreign object and the camera, including the following steps:

According to the camera parameters, the parallax of the same foreign object between the binocular cameras is obtained, and then the depth information of the foreign object in the image is obtained, that is, the distance between the foreign object and the camera.

The calculation of the distance between the foreign object and the camera uses binocular ranging to calculate the distance between the foreign object and the camera, including the following steps:

The binocular camera is calibrated in advance, the camera parameters of the two cameras are obtained, and the images obtained by the two cameras are corrected, so that the corrected two images are located on the same plane and parallel to each other. The matching result calculates the depth of each pixel to obtain a depth map, and obtains the distance between the foreign object and the camera according to the depth map.

The distance between the foreign object and the camera is obtained by the following formula:

Among them, c is the focal length of the camera, b is the length of the line connecting the imaging points of the two cameras, d is the parallax, and Z is the distance from the foreign object to the midpoint of the line connecting the two cameras.

A rapid automatic identification and positioning system for foreign objects in a subway tunnel, comprising:

The image processing module processes the subway tunnel image obtained by the binocular camera, and determines whether there is foreign matter in the image;

The image detection module is used to judge the type of the foreign body and calculate the position of the foreign body and the distance from the camera when the foreign body is detected, so that the robot can grasp the foreign body.

Overall, compared with the prior art, the above technical solutions according to the present invention mainly have the following technical advantages:

1. Use YOLO v3 target recognition algorithm to identify foreign objects. The advantage is that the deep network model trained by the YOLO object detection framework is used for real-time foreign object detection. Compared with the traditional detection algorithm, the deep network model trained based on the YOLO object detection framework is more accurate, and the recognition process is very fast, which can achieve real-time detection. Target.

2. Use binocular ranging to calculate the distance of foreign objects. Compared with monocular ranging, binocular ranging can directly use parallax to calculate distance, and the accuracy is higher than that of monocular. More accurate position distance information can be obtained through images, which can well meet our recognition and calculation requirements.

3. This technology uses the robot as the carrier. While it can quickly and accurately identify and locate the position of foreign objects, it can be combined with robots to realize real-time detection and real-time grasping of foreign objects, ensuring that there are no foreign objects in the subway tunnel at any time, and ensuring the safe operation of the subway. , to ensure the health and safety of drivers and passengers.

Description of drawings

Fig. 1 is the principle flow chart of the technology of the present invention;

Figure 2 is a schematic diagram of the pinhole model;

Figure 3 is a schematic diagram of the binocular imaging model.

Detailed ways

In order to make the objectives, technical solutions and advantages of the present invention clearer, the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention, but not to limit the present invention. In addition, the technical features involved in the various embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.

The present invention includes the following steps:

S1: Install the camera as required, and install two identical cameras in the appropriate position of the robot;

S2: obtain the image of the camera monitoring area;

S3: Process the acquired image, and determine whether there is foreign matter in the image;

S4: When a foreign body is detected, determine the type of the foreign body and calculate the position and distance between the foreign body and the camera (robot);

S5: The robot grabs the foreign object according to the type, position and distance data of the foreign object.

Preferably, for step S1, the system is combined with the existing robot technology, and the grasping function of the robot's manipulator is used to realize timely detection and grasping of foreign objects.

Preferably, for step S1, the robot as the system carrier should be selected to have a moving function and a grasping function. The moving function should be able to adjust the direction and speed of the movement so as to approach the foreign object correctly, and the grasping function should be able to adjust the size, angle and strength of the manipulator to successfully grasp the foreign object.

Preferably, for step S1, the positions of the two cameras should be installed at the appropriate positions of the robot, so as to monitor the global image of the tunnel.

Preferably, for step S3, the image obtained by the camera is first subjected to image enhancement, filtering and other processing to make it a picture that meets the identification requirements.

Preferably, for step S3, the object recognition and localization algorithm YOLO v3 based on the deep neural network is used to realize the judgment and prediction of the type and position of the foreign object in the image. The typical feature of YOLO v3 is that it runs very fast and can be used for real-time detection. At the same time, the background false detection rate is relatively low, and the recognition accuracy rate is relatively high, which meets the technical requirements.

The implementation of the YOLO recognition algorithm needs to include the following steps: (1) Collect and obtain a large amount of image data in advance, and mark the foreign body in the image, use the label box containing the foreign body to indicate the location of the foreign body, and use the label name to indicate the type of foreign body. (2) Establish a foreign body detection and recognition model, and use the obtained tunnel foreign body images to train the YOLO neural network model to obtain a trained foreign body detection and detection model. (3) Use the trained foreign object detection and recognition model to test the monitoring image collected by the camera to test the detection model to identify foreign objects.

When the robot inspects, the camera captures image data, and transmits the data as input to the neural network to detect it. If there is a foreign object, it will identify the specific location and category. If there is no foreign object, it will continue to process the next frame of image.

Preferably, for step S4, binocular ranging is used to calculate the distance between the foreign object and the camera (robot). According to the camera imaging model, for the monitoring of the same object, the position of the object in the image is different when observed at different positions. The principle of binocular ranging is similar to that of the human eye. The human eye can perceive the distance of an object due to the difference between the images presented by the two eyes to the same object, also known as "parallax". The farther the object is, the smaller the parallax; conversely, the greater the parallax. By calculating the parallax of the same object between the binocular cameras, the depth information of the object in the camera image can be calculated, that is, the distance between the object and the camera. Binocular ranging generally requires the following steps: (1) Calibrate the binocular camera to obtain the internal and external parameters and homography matrices of the two cameras. (2) Correct the original image according to the calibration result, and the two corrected images are located on the same plane and parallel to each other. (3) Perform pixel point matching on the two corrected images. (4) Calculate the depth of each pixel according to the matching result, thereby obtaining a depth map.

Preferably, for step S5, the foreign object type, position and distance data obtained in step S4 will be returned to the robot, and the robot will adjust its position in real time according to the information, and approach and grasp the foreign object.

Fig. 1 illustrates the principle frame structure diagram of a kind of fast automatic identification and positioning technology of foreign objects in subway tunnels of the present invention, and the specific steps are as follows:

S1: Install the camera as required, and install two identical cameras in the appropriate position of the robot;

Due to the needs of detection, in order to realize the detection of foreign objects in subway tunnels, a camera is installed on the robot. In the present invention, the acquisition of the monitoring image of the subway tunnel is realized by using dual cameras, and the image collected by a single camera is used to identify and detect foreign objects in the subway tunnel. In the captured image, the position of the foreign object in the image is different and the principle of parallax is generated, which realizes the calculation of the time distance between the foreign object and the camera;

At the same time, the picking up of foreign objects in the present invention is realized by a robot hand. The technology of the present invention takes the robot as the carrier. When the system calculates the specific type and distance of the foreign object, the data is transmitted to the robot, and the robot will make corresponding actions to approach and pick up the foreign object, thereby realizing the real-time detection and real-time grasping of the foreign object. .

When adding a camera to the robot, considering the cost and structural reasons, the structure of the robot should not be changed when adding a camera to the robot. While considering the structure, the installation position and angle of the camera should also be considered. The positions of the two cameras installed should be as symmetrical as possible about the center line of the robot, and their directions should be on the same horizontal line as possible, so as to achieve a farther field of view and easier calculation.

S2: obtain the image of the camera monitoring area;

During the experiment, the camera parameters were continuously adjusted, and the parameters of the two cameras were fixed before the final work to ensure that the images collected by the cameras were clear and accurate, and prepared for the subsequent image recognition. The collected images are transmitted to the system through the image transmission unit, so as to further process the images.

S3: Process the acquired image, and determine whether there is foreign matter in the image;

S4: When a foreign body is detected, determine the type of the foreign body and calculate the position and distance between the foreign body and the camera (robot);

In step S3, the collected images are processed by image enhancement, filtering, etc. Since the subway tunnel is relatively dark, the collected images are made clearer through image enhancement, the brightness is improved, and the foreign objects and the background are clearer. Separate; prepare for the identification of foreign objects by filtering out noise in the background that may interfere with detection.

The rapid detection and identification of foreign objects in subway tunnels mentioned in steps S3 and S4. The object recognition and localization algorithm YOLO v3 based on deep neural network is used to realize the judgment and prediction of the type and location of foreign objects in the image.

For the detection of foreign objects using the YOLO algorithm, the following steps are required: 1) Collect a large number of data sets, find on the Internet or take pictures of subway tunnels with foreign objects on the spot, foreign objects include notebook bags, backpacks for men and women, handbags, and paper bags , plastic bags with items, express packaging bags, water cups, medicine bottles, mineral water bottles, cannonball-shaped cylinders, etc. These images will be used as our training dataset. Label the collected pictures with an annotation tool such as labelimg, the label frame surrounds the object, identifies the location and size of the foreign object, and the label name fills in the type of foreign object. The file format of the labeled picture is xml. 2) Modify the Makefile to train our dataset with GPU, which can greatly improve the speed of our training. For network training on our data set, use the network structure and training algorithm provided by the YOLO framework to train the model to get the weights we need; 3) Perform network prediction for the trained model, and the test result returns the bounding box coordinates of the foreign object (boundary). The center coordinates of the box, the length and width of the bounding box, and the types of foreign objects, the training parameters are continuously adjusted according to the test results, and the network structure is optimized, so that the final model meets the actual needs.

The calculation of the distance between the foreign object and the camera (robot) mentioned in step S4 is realized by binocular ranging.

Binocular ranging can directly use parallax to calculate distance, and the accuracy is higher than that of monocular, and more accurate position distance information can be obtained through images; there is no restriction on the recognition rate, because in principle, there is no need to recognize and then measure, but for all Obstacles are measured directly; binocular vision also does not need to maintain a sample database, because there is no concept of samples for binoculars.

A pinhole model is used to approximate the imaging mechanism of the camera, as shown in Figure 2. M is an object point in the real scene, O is the optical center of the camera, O' is the projection of the optical center on the image plane, OO' is the optical axis of the camera, and M' is the image point of M on the image plane P.

For the sake of simplicity, consider two cameras spaced at an appropriate distance and with parallel optical axes (the camera parameters are the same). This is the most ideal binocular model. The imaging of the same object point P in the two cameras is shown in Figure 3.

As shown in Figure 3, P is a certain point on the object to be measured, and O' and O" are the optical centers of the left and right cameras. The imaging points of point P on the two camera photoreceptors are P' and P" respectively (the camera The imaging plane is rotated and placed in front of the lens), c is the focal length of the camera, b is the center line (baseline) of the two cameras, and the depth direction distance from point P to point O' is set to Z, then the triangle similarity principle:

The X/Y coordinates of point P are:

In depth direction:

So it can be deduced that:

(d为视差)(3)

(d is parallax) (3)

Among them, x' and x" are the distances of point P in the X direction of the image point offset on the imaging plane, and the difference between the two is defined as parallax, that is, the difference between the imaging positions of the image points in different cameras, which is represented by d.

The binocular measurement technology determines the target position through the principle of parallax.

Binocular ranging generally requires the following steps: (1) Calibrate the binocular camera to obtain the internal and external parameters and homography matrices of the two cameras. (2) Correct the original image according to the calibration result, and the two corrected images are located on the same plane and parallel to each other. (3) Perform pixel point matching on the two corrected images. (4) Calculate the depth of each pixel according to the matching result, thereby obtaining a depth map.

S5: The robot grabs the foreign object according to the type, position and distance data of the foreign object.

In summary, the present invention realizes the rapid identification of the type and location of foreign objects through the YOLO v3 target object identification and positioning algorithm, and realizes the calculation of the distance of foreign objects through binocular ranging. Real-time and rapid identification and positioning of foreign objects in subway tunnels, and timely pickup of foreign objects to ensure the safety of subway trains.

Those skilled in the art can easily understand that the above are only preferred embodiments of the present invention, and are not intended to limit the present invention. Any modifications, equivalent replacements and improvements made within the spirit and principles of the present invention, etc., All should be included within the protection scope of the present invention.

Claims (7)

1. a subway tunnel foreign body fast automatic identification and positioning method, is characterized in that, comprises the following steps: S1: Set up two identical cameras on the robot; S2: Obtain the subway tunnel image of the monitoring area of the two cameras; S3: Process the acquired image, and determine whether there is foreign matter in the image; S4: When a foreign body is detected, determine the type of the foreign body and calculate the position of the foreign body and the distance from the camera; S5: According to the type, position and distance of the foreign body, the robot grabs the foreign body. 2. a kind of subway tunnel foreign body quick automatic identification and positioning method as claimed in claim 1, is characterized in that, described S3 comprises the following steps: Any one of the two cameras captures images, and performs image enhancement and filtering processing; The type and location of foreign objects in the image are judged and predicted by neural network. 3. a kind of subway tunnel foreign body fast automatic identification and localization method as claimed in claim 2 is characterized in that, the training of described neural network, comprises the following steps: (1) Acquire the foreign body image data of the subway tunnel through two cameras in advance. For the images collected by each camera, mark the foreign body, use the label frame containing the foreign body to indicate the position of the foreign body, and use the label name to indicate the type of foreign body; (2) Establish a YOLO neural network model for foreign object detection, perform YOLO neural network model training for the images collected and labeled by each camera, and obtain a trained YOLO neural network model for real-time monitoring of any camera. Image to identify foreign objects. 4. a kind of subway tunnel foreign body fast automatic identification and positioning method as claimed in claim 1 is characterized in that, described calculating the distance between foreign body and camera, uses binocular ranging to realize the distance between foreign body and camera Perform the calculation, which includes the following steps: According to the camera parameters, the parallax of the same foreign object between the binocular cameras is obtained, and then the depth information of the foreign object in the image is obtained, that is, the distance between the foreign object and the camera. 5. a kind of subway tunnel foreign body fast automatic identification and positioning method as claimed in claim 1 is characterized in that, described calculating the distance between foreign body and camera, uses binocular ranging to realize the distance between foreign body and camera Perform the calculation, which includes the following steps: The binocular camera is calibrated in advance, the camera parameters of the two cameras are obtained, and the images obtained by the two cameras are corrected, so that the corrected two images are located on the same plane and parallel to each other. The matching result calculates the depth of each pixel to obtain a depth map, and obtains the distance between the foreign object and the camera according to the depth map. 6. a kind of subway tunnel foreign body quick automatic identification and positioning method as claimed in claim 4 or 5, is characterized in that, the distance between described foreign body and camera is obtained by following formula:

Among them, c is the focal length of the camera, b is the length of the line connecting the imaging points of the two cameras, d is the parallax, and Z is the distance from the foreign object to the midpoint of the line connecting the two cameras. 7. A rapid automatic identification and positioning system for foreign objects in a subway tunnel, characterized in that, comprising: The image processing module processes the subway tunnel image obtained by the binocular camera, and determines whether there is foreign matter in the image; The image detection module is used to judge the type of the foreign body and calculate the position of the foreign body and the distance from the camera when the foreign body is detected, so that the robot can grasp the foreign body.

Images