CN116721093B - Subway rail obstacle detection method and system based on neural network - Google Patents
Subway rail obstacle detection method and system based on neural network Download PDFInfo
- Publication number
- CN116721093B CN116721093B CN202310968414.2A CN202310968414A CN116721093B CN 116721093 B CN116721093 B CN 116721093B CN 202310968414 A CN202310968414 A CN 202310968414A CN 116721093 B CN116721093 B CN 116721093B
- Authority
- CN
- China
- Prior art keywords
- detected
- track
- illumination
- segments
- neural network
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000001514 detection method Methods 0.000 title claims abstract description 83
- 238000013528 artificial neural network Methods 0.000 title claims abstract description 32
- 238000005286 illumination Methods 0.000 claims abstract description 152
- 238000003062 neural network model Methods 0.000 claims abstract description 46
- 238000012544 monitoring process Methods 0.000 claims abstract description 45
- 239000003086 colorant Substances 0.000 claims abstract description 26
- 238000012545 processing Methods 0.000 claims abstract description 22
- 238000000034 method Methods 0.000 claims abstract description 8
- 230000004888 barrier function Effects 0.000 claims description 2
- 238000009826 distribution Methods 0.000 description 5
- 238000010801 machine learning Methods 0.000 description 5
- 238000012549 training Methods 0.000 description 4
- 230000015572 biosynthetic process Effects 0.000 description 3
- 230000008569 process Effects 0.000 description 3
- 238000003786 synthesis reaction Methods 0.000 description 3
- 238000004458 analytical method Methods 0.000 description 2
- 238000004364 calculation method Methods 0.000 description 2
- 230000008859 change Effects 0.000 description 2
- 238000006243 chemical reaction Methods 0.000 description 2
- 238000013527 convolutional neural network Methods 0.000 description 2
- 238000010586 diagram Methods 0.000 description 2
- 238000005516 engineering process Methods 0.000 description 2
- 238000000605 extraction Methods 0.000 description 2
- 239000000463 material Substances 0.000 description 2
- 229920006395 saturated elastomer Polymers 0.000 description 2
- 238000000547 structure data Methods 0.000 description 2
- 238000001308 synthesis method Methods 0.000 description 2
- 230000000007 visual effect Effects 0.000 description 2
- 238000000576 coating method Methods 0.000 description 1
- 125000004122 cyclic group Chemical group 0.000 description 1
- 238000013461 design Methods 0.000 description 1
- 230000006870 function Effects 0.000 description 1
- 238000011478 gradient descent method Methods 0.000 description 1
- 230000007774 longterm Effects 0.000 description 1
- 238000010606 normalization Methods 0.000 description 1
- 230000009467 reduction Effects 0.000 description 1
- 238000000926 separation method Methods 0.000 description 1
- 239000013598 vector Substances 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/0002—Inspection of images, e.g. flaw detection
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/044—Recurrent networks, e.g. Hopfield networks
- G06N3/0442—Recurrent networks, e.g. Hopfield networks characterised by memory or gating, e.g. long short-term memory [LSTM] or gated recurrent units [GRU]
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
- G06N3/098—Distributed learning, e.g. federated learning
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T5/00—Image enhancement or restoration
- G06T5/40—Image enhancement or restoration by the use of histogram techniques
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/90—Determination of colour characteristics
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/10—Image acquisition modality
- G06T2207/10024—Color image
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/10—Image acquisition modality
- G06T2207/10032—Satellite or aerial image; Remote sensing
- G06T2207/10044—Radar image
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/10—Image acquisition modality
- G06T2207/10048—Infrared image
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20076—Probabilistic image processing
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20081—Training; Learning
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20084—Artificial neural networks [ANN]
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/30—Subject of image; Context of image processing
- G06T2207/30232—Surveillance
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/30—Subject of image; Context of image processing
- G06T2207/30248—Vehicle exterior or interior
- G06T2207/30252—Vehicle exterior; Vicinity of vehicle
- G06T2207/30256—Lane; Road marking
Abstract
The invention provides a subway rail obstacle detection method and system based on a neural network, and relates to the technical field of rail obstacle detection, wherein the method comprises the steps of obtaining a plurality of rail pictures under different colors of illumination; processing a plurality of track pictures under different colors of illumination based on a graph neural network model to determine a plurality of track segments to be detected; acquiring monitoring videos of a plurality of track segments to be detected and pictures of the track segments to be detected under different colors of illumination; determining the existence probability of the obstacle of the track segments to be detected based on the monitoring videos of the track segments to be detected and the picture use probability determining model of the track segments to be detected under different colors of illumination; the detection mode of each track segment to be detected in the track segments to be detected is determined based on the existence probabilities of the obstacles in the track segments to be detected, and the detection efficiency of the track obstacles can be improved.
Description
Technical Field
The invention relates to the technical field of rail obstacle detection, in particular to a subway rail obstacle detection method and system based on a neural network.
Background
In order to overcome the frequent occurrence of rail traffic safety accidents, rail obstacles need to be detected, and in the existing subway rail obstacle detection technology, rail pictures are usually processed by using a computer vision algorithm to detect possible obstacles. In the traditional track obstacle detection, all tracks need to be scanned and detected, and due to the fact that the tracks are longer, the problem of missed detection or false detection possibly occurs in the detection process, the accuracy of detection results is affected, and the scanning time and the subsequent detection time of the longer tracks are also longer. And because the processing of each track picture needs to carry out complex calculation, a great deal of calculation resources and time are consumed, and because the occurrence of the obstacle in the track is a small probability event, a great deal of manpower and material resources are consumed for detecting the obstacle in all the tracks at a time, and the efficiency is low.
Therefore, how to improve the detection efficiency of the track obstacle is a current urgent problem to be solved.
Disclosure of Invention
The invention mainly solves the technical problem of how to improve the detection efficiency of the track obstacle.
According to a first aspect, the invention provides a subway rail obstacle detection method based on a neural network, comprising the following steps: acquiring a plurality of track pictures under different color illumination, wherein each track picture is obtained by shooting a track under one color illumination; processing the plurality of track pictures under different color illumination to determine a plurality of track segments to be detected based on a graph neural network model, wherein the input of the graph neural network model comprises a plurality of nodes and a plurality of edges between the plurality of nodes, each node in the plurality of nodes represents a track under each color illumination, each node in the plurality of nodes comprises a plurality of node features, each node feature comprises a track picture under each color illumination, illumination intensity of a plurality of light sources corresponding to each color illumination, and a color histogram of the track picture under each color illumination, and the output of the graph neural network model is the plurality of track segments to be detected; acquiring monitoring videos of the track segments to be detected and pictures of the track segments to be detected under different colors of illumination; determining the existence probability of the obstacle of the track segments to be detected based on the monitoring videos of the track segments to be detected and the picture use probability determining model of the track segments to be detected under different color illumination; and determining the detection mode of each track section to be detected in the track sections to be detected based on the existence probabilities of the obstacles of the track sections to be detected.
Still further, the detection mode of the track section to be detected includes that the track section to be detected is detected in a cooperative manner by adopting a visible light camera, a laser radar and an infrared camera, the track section to be detected is detected in a cooperative manner by adopting a visible light camera and an infrared camera, the track section to be detected is detected by adopting an ultrasonic sensor, the resolution of the visible light camera is larger than 1080P, and the detection mode of each track section to be detected in the track sections to be detected based on the existence probabilities of the barriers of the track sections to be detected comprises: if the existence probability of the obstacle of the track section to be detected is greater than 0.1, the visible light camera, the laser radar and the infrared camera are adopted to cooperatively detect the track section to be detected; if the existence probability of the obstacle of the track section to be detected is more than 0.01 and less than 0.1, the visible light cameras and the infrared cameras are adopted to cooperatively detect the track section to be detected; and if the existence probability of the obstacle of the track section to be detected is smaller than or equal to 0.01, detecting the track section to be detected by adopting an ultrasonic sensor.
Further, the plurality of track pictures under different color illumination include a plurality of track pictures under white light illumination, a plurality of track pictures under red light illumination, a plurality of track pictures under green light illumination, a plurality of track pictures under blue light illumination, and a plurality of track pictures under ultraviolet light illumination.
Furthermore, the probability determination model is a long-short period neural network model, the inputs of the probability determination model are the monitoring videos of the track segments to be detected and pictures of the track segments to be detected under different colors of illumination, and the probability determination model outputs the existence probabilities of the obstacles of the track segments to be detected.
Still further, the method further comprises: and acquiring the position information of each track segment to be detected and the detection mode corresponding to each track segment to be detected, and determining the total detection time based on the position information of each track segment to be detected and the detection mode corresponding to each track segment to be detected by using a time determination model, wherein the time determination model is a deep neural network model.
According to a second aspect, the present invention provides a subway rail obstacle detection system based on a neural network, comprising: the first acquisition module is used for acquiring a plurality of track pictures under illumination of different colors, wherein each track picture is a picture obtained by shooting a track under illumination of one color; the track segment determining module is used for processing the plurality of track pictures under different color illumination to determine a plurality of track segments to be detected based on a graph neural network model, wherein the input of the graph neural network model comprises a plurality of nodes and a plurality of edges between the plurality of nodes, each node in the plurality of nodes represents a track under each color illumination, each node in the plurality of nodes comprises a plurality of node characteristics, each node characteristic comprises a track picture under each color illumination, illumination intensity of a plurality of light sources corresponding to each color illumination, and a color histogram of the track picture under each color illumination, and the output of the graph neural network model is the plurality of track segments to be detected; the second acquisition module is used for acquiring the monitoring videos of the plurality of track segments to be detected and the pictures of the plurality of track segments to be detected under different colors of illumination; the probability determining module is used for determining the existence probability of the obstacle of the plurality of track segments to be detected based on the monitoring videos of the plurality of track segments to be detected and the pictures of the plurality of track segments to be detected under different colors of illumination by using a probability determining model; the detection mode determining module is used for determining the detection mode of each track segment to be detected in the track segments to be detected based on the existence probabilities of the obstacles of the track segments to be detected.
Still further, the detection mode of treating the track section of detecting includes adopting visible light camera, laser radar, infrared camera to treat the track section of detecting jointly and detect, adopting visible light camera, infrared camera to treat the track section of detecting jointly and detect, adopting ultrasonic sensor to treat the track section of detecting, the resolution ratio of visible light camera is greater than 1080P, detection mode determination module still is used for:
if the existence probability of the obstacle of the track section to be detected is greater than 0.1, the visible light camera, the laser radar and the infrared camera are adopted to cooperatively detect the track section to be detected;
if the existence probability of the obstacle of the track section to be detected is more than 0.01 and less than 0.1, the visible light cameras and the infrared cameras are adopted to cooperatively detect the track section to be detected;
and if the existence probability of the obstacle of the track section to be detected is smaller than or equal to 0.01, detecting the track section to be detected by adopting an ultrasonic sensor.
Further, the plurality of track pictures under different color illumination include a plurality of track pictures under white light illumination, a plurality of track pictures under red light illumination, a plurality of track pictures under green light illumination, a plurality of track pictures under blue light illumination, and a plurality of track pictures under ultraviolet light illumination.
Furthermore, the probability determination model is a long-short period neural network model, the inputs of the probability determination model are the monitoring videos of the track segments to be detected and pictures of the track segments to be detected under different colors of illumination, and the probability determination model outputs the existence probabilities of the obstacles of the track segments to be detected.
Still further, the system is further configured to: and acquiring the position information of each track segment to be detected and the detection mode corresponding to each track segment to be detected, and determining the total detection time based on the position information of each track segment to be detected and the detection mode corresponding to each track segment to be detected by using a time determination model, wherein the time determination model is a deep neural network model.
The invention provides a subway rail obstacle detection method and system based on a neural network, wherein the method comprises the steps of obtaining a plurality of rail pictures under illumination of different colors, wherein each rail picture is obtained by shooting a rail under illumination of one color; processing the plurality of track pictures under different color illumination to determine a plurality of track segments to be detected based on a graph neural network model, wherein the input of the graph neural network model comprises a plurality of nodes and a plurality of edges between the plurality of nodes, each node in the plurality of nodes represents a track under each color illumination, each node in the plurality of nodes comprises a plurality of node features, each node feature comprises a track picture under each color illumination, illumination intensity of a plurality of light sources corresponding to each color illumination, and a color histogram of the track picture under each color illumination, and the output of the graph neural network model is the plurality of track segments to be detected; acquiring monitoring videos of the track segments to be detected and pictures of the track segments to be detected under different colors of illumination; determining the existence probability of the obstacle of the track segments to be detected based on the monitoring videos of the track segments to be detected and the picture use probability determining model of the track segments to be detected under different color illumination; the detection mode of each track segment to be detected in the track segments to be detected is determined based on the existence probabilities of the obstacles in the track segments to be detected, and the detection efficiency of the track obstacles can be improved.
Drawings
Fig. 1 is a schematic flow chart of a subway rail obstacle detection method based on a neural network according to an embodiment of the present invention;
fig. 2 is a schematic diagram of a subway rail obstacle detection system based on a neural network according to an embodiment of the present invention.
Detailed Description
In an embodiment of the present invention, a subway rail obstacle detection method based on a neural network is provided as shown in fig. 1, where the subway rail obstacle detection method based on the neural network includes steps S1 to S5:
step S1, a plurality of track pictures under different color illumination are obtained, wherein each track picture is obtained by shooting a track under one color illumination.
The track is the track that the subway was driven, and the track includes straight line portion and curved portion.
Each of the plurality of track pictures under different color illumination is a picture obtained by shooting a track under one color illumination.
In some embodiments, a plurality of light sources with different colors can be arranged beside the subway track to irradiate the track and obtain a plurality of irradiated track pictures.
In some embodiments, a monitoring camera may be used to capture a track to obtain a picture of a plurality of sections of tracks, and a picture synthesis method may be used to synthesize a track picture obtained by capturing the track under illumination of one color. The picture synthesis method comprises superposition synthesis, collage synthesis, mask synthesis and the like.
The plurality of track pictures under different color illumination comprise a plurality of track pictures under white light illumination, a plurality of track pictures under red light illumination, a plurality of track pictures under green light illumination, a plurality of track pictures under blue light illumination and a plurality of track pictures under ultraviolet light illumination.
Illuminating the train track with different colored illumination can provide a variety of information and features that can also be used as input to the graphic neural network model to determine obstructions. For example, white light is a comprehensive lighting condition that provides a relatively realistic and natural lighting environment. Therefore, the track picture under white light illumination can provide real visual information for the object, including the surface texture, color, shape and other characteristics of the object. The track picture under white light illumination can be used as a reference image for comparing and referencing pictures under other color illumination. Through contrast analysis, the change condition of the reflection characteristic of the object surface under other illumination conditions can be detected, and then the obstacle is detected.
The picture under red light mainly highlights the red component of the object, so that the red object is more saturated and brighter in the image. Therefore, the rail picture under the illumination of red light can highlight the characteristics of the red obstacle, and the red obstacle can be conveniently detected and identified. The picture under red light illumination can be used for detecting and identifying red obstacles.
The picture under green light can enhance the contrast and brightness of the green object, so that the green obstacle is more prominent in the image. Thus, the track picture under green light helps to detect and identify green obstacles. The picture under green light illumination can be used for detecting and identifying green obstacles.
The picture under blue light mainly highlights the features of the blue object, making the blue obstacle brighter and saturated in the image. The track picture under blue light illumination can be used to detect and identify blue obstacles. The picture under blue light illumination can be used for detection and identification of blue obstacles.
The picture under ultraviolet light mainly highlights the ultraviolet reflection characteristic of the surface of the object, so that certain specific materials or coatings show special reflection characteristics under ultraviolet light. The track pictures under ultraviolet light can be used for detecting and identifying some obstacles which are not easy to detect under visible light.
Step S2, a plurality of track pictures under different color illumination are processed based on a graph neural network model to determine a plurality of track segments to be detected, wherein the input of the graph neural network model comprises a plurality of nodes and a plurality of edges among the plurality of nodes, each node in the plurality of nodes represents a track under each color illumination, each node in the plurality of nodes comprises a plurality of node features, each node feature comprises a track picture under each color illumination, illumination intensity of a plurality of light sources corresponding to each color illumination, and a color histogram of the track picture under each color illumination, and the output of the graph neural network model is a plurality of track segments to be detected.
Because the definition of the plurality of track pictures under different color illumination shot by the monitoring camera is low, hiding the plurality of track pictures under different color illumination cannot be completely used for displaying the obstacle in the track, for example, the separation rate of the monitoring camera is 480P, but the plurality of track pictures under different color illumination can be used as the input of the graph neural network model to determine a plurality of track segments to be detected.
The track segment to be detected refers to a specific track area where obstacle detection is required. A small section of track in the whole subway track of the track section to be detected. For example, a certain curve area on a section of straight track may be a section of track to be detected.
The graphic neural network model comprises a graphic neural network (Graph Neural Network, GNN) and a full connection layer, wherein the graphic neural network is a neural network directly acting on graphic structure data, and the graphic structure data is a data structure consisting of nodes and edges.
The track picture under illumination of each color, the illumination intensity of a plurality of light sources corresponding to each color and the color histogram of the track picture under illumination of each color can be obtained, and a graph structure is constructed according to the information, wherein the graph structure comprises a plurality of nodes and a plurality of edges between the nodes, and each node represents a track under illumination of one color. In a graph neural network, a node may represent any object or entity, including an image, text, a user, and so forth. The track pictures under each color illumination can be used as node characteristics of nodes in the graph neural network model. Node features are vectors or matrices that describe node attributes. For the track picture under each color illumination, the track picture under each color illumination can be encoded into node characteristics, so that the node characteristics become the input of the graph neural network model, for example, a digitized pixel array corresponding to the track picture under each color illumination can be input into the graph neural network model.
The illumination intensity of the plurality of light sources corresponding to illumination of each color can be the illumination intensity set by the staff when illuminating the track.
The color histogram of the track picture under each color illumination is calculated to obtain the color histogram of the track picture under each color illumination. In some embodiments, the histogram may be calculated by counting the number of pixels for each color value or color channel in the picture. In some embodiments, the histogram of the track picture may be calculated using the calcHist function in OpenCV.
The color histogram can be used as the characteristic of the track node under each color illumination, and the color histogram can capture the distribution condition of different colors in each track segment to be detected. These color features may be used to distinguish between different objects or obstacles. The graphic neural network model can identify and classify different obstacles by learning these color features. The color histogram may provide a color distribution of various locations within the track segment to be detected. These color distributions may help the model understand the surrounding environment and assist in obstacle detection. The graph neural network model can fuse the color histogram with the attributes and the relations of other nodes and perform joint learning. The accuracy and robustness of obstacle detection can be improved.
In some embodiments, the characteristics of the edges in the plurality of edges may be the similarity between the track pictures under each color illumination. In some embodiments, the similarity between the track pictures under each color illumination may be calculated by cosine similarity, euclidean distance.
Because the definition of the track pictures under different illumination is not high enough, if the track pictures are directly used for detecting the obstacle, the result is not very accurate, the track pictures under different illumination only can provide less accurate obstacle distribution information, and the less accurate obstacle distribution information needs to be further confirmed and detected. The rail pictures under different illumination are processed by using the graph neural network model to determine a plurality of rail segments to be detected, and then subsequent obstacle detection is carried out, so that the efficiency of obstacle detection can be improved, and detection of all rails is avoided.
And step S3, acquiring the monitoring videos of the plurality of track segments to be detected and the pictures of the plurality of track segments to be detected under different colors of illumination.
The monitoring videos of the track segments to be detected refer to continuous time series image data collected in the range of the track segments to be detected. And the monitoring videos of the track segments to be detected are recorded by the monitoring cameras. Because the resolution of the monitoring camera is low, if the monitoring camera is directly used for detecting the rail obstacle, the accuracy is not enough. The monitoring video can provide continuous real-time information, and can display how the obstacle reaches the track because the obstacle is not in the track due to the fact that the obstacle is not in the gap, but can not completely and clearly judge whether the obstacle exists or not through the monitoring video because the resolution of the monitoring video is insufficient. For example, a surveillance video of a track segment to be detected may show that a subway passenger's clothing falls into the track, and the surveillance video may show that there is an obstacle to the subway passenger's clothing due to the larger clothing. For another example, if the headphones of the passenger drop into the track, it is difficult for the monitoring video of the track segment to be detected to clearly show the obstacle of the headphones due to the smaller headphones. However, since the earphone has a corresponding small motion track in the monitoring video, although it is difficult to clearly determine whether the obstacle exists or not through the monitoring video, the probability of the obstacle exists can be determined by processing the monitoring video. For example, the larger the obstacle is, the clearer the display of the monitoring video is, and the higher the probability of judging that the obstacle exists after the monitoring video is processed is.
In some embodiments, after determining a plurality of track segments to be detected, a monitoring video corresponding to the plurality of track segments to be detected recorded by a monitoring camera may be obtained, where the monitoring camera covers the entire track.
The pictures of the plurality of track segments to be detected under different color illumination represent the pictures of the plurality of track segments to be detected under different color illumination. In some embodiments, the plurality of track segments to be detected may be cut out from the plurality of track pictures under different color illumination to obtain pictures of the plurality of track segments to be detected under different color illumination.
And S4, determining the existence probability of the obstacle of the track segments to be detected based on the monitoring videos of the track segments to be detected and the pictures of the track segments to be detected under different colors of illumination by using a probability determination model.
The existence probability of the obstacle of the track section to be detected represents the existence probability of the obstacle of the track section to be detected, the existence probability of the obstacle of the track section to be detected can be a numerical value between 0 and 1, and the larger the numerical value is, the higher the existence probability of the obstacle is. It will be appreciated that the larger the obstacle, the higher the probability of the obstacle being obtained by processing the surveillance videos of the plurality of track segments to be detected and the pictures of the plurality of track segments to be detected under different color illumination, and vice versa, and if the obstacle is smaller than a certain threshold, although the lower the probability of the obstacle being obtained by processing the surveillance videos of the plurality of track segments to be detected and the pictures of the plurality of track segments to be detected under different color illumination, although the obstacle may not be identified, since if the number of the obstacles in the subway track is smaller than the set threshold, and the obstacles do not cause substantial interference or damage to the train and the track, the subway will not have a direct influence on the operation of the subway after passing through the obstacles in general. The subway system design and management considers the safe and smooth running of the train, and adopts multiple protection measures. If the obstacle is smaller than the threshold value and does not cause an emergency or destroy the track structure, the subway track is designed in general in consideration of safety in various situations and space allowed to pass through. The space above the track and the side clearance are typically designed to be sufficient to accommodate common obstructions such as small branches, small garbage, etc.
The probability determination model is a long-short-period neural network model, the input of the probability determination model is a monitoring video of the track segments to be detected and pictures of the track segments to be detected under different colors of illumination, and the probability determination model outputs the existence probability of the obstacle of the track segments to be detected.
The probability determination model is a long-short-term neural network model. The Long-Short Term neural network model includes a Long-Short Term neural network (LSTM). The long-term and short-term neural network model can process sequence data with any length, capture sequence information and output results based on the association relationship of front data and rear data in the sequence. The probability determination model comprehensively considers the monitoring videos of the track segments to be detected and the pictures of the track segments to be detected at different time points under different colors of illumination, and finally determines the existence probability of the obstacle of the track segments to be detected. The probability determination model can be obtained by training a training sample through a gradient descent method.
In some embodiments, the probability determination model includes a surveillance video processing model, a picture processing model, and a weighted summation algorithm, where the surveillance video processing model and the picture processing model are both long and short term neural network models. The input of the monitoring video processing model is the monitoring video of the track segments to be detected, the output of the monitoring video processing model is the first obstacle existence probability of the track segments to be detected, the input of the picture processing model is the pictures of the track segments to be detected under different colors of illumination, and the output of the picture processing model is the second obstacle existence probability of the track segments to be detected. The inputs of the weighted summation algorithm are a first obstacle existence probability and a second obstacle existence probability, and the output of the weighted summation algorithm is the obstacle existence probability.
Each sub-model can be focused on own tasks, the precision and efficiency of the model are improved, the training speed is faster during model training, the single sub-model can be adjusted and optimized more conveniently, and meanwhile, the new sub-model is integrated into the whole model conveniently.
And S5, determining the detection mode of each track section to be detected in the track sections to be detected based on the existence probabilities of the obstacles of the track sections to be detected.
It will be appreciated that the greater the probability of the presence of an obstacle, the more accurate detection will need to be taken to detect each track segment to be detected.
The detection mode of the track section to be detected comprises the steps of adopting a visible light camera, a laser radar and an infrared camera to cooperatively detect the track section to be detected together, adopting the visible light camera and the infrared camera to cooperatively detect the track section to be detected together and adopting an ultrasonic sensor to detect the track section to be detected, wherein the resolution of the visible light camera is larger than 1080P.
If the existence probability of the obstacle of the track section to be detected is greater than 0.1, the track section to be detected is cooperatively detected by adopting a visible light camera, a laser radar and an infrared camera, as an example, the track section to be detected can be scanned by using the visible light camera, the laser radar and the infrared camera together, and the scanning result is input into a trained first machine learning model to obtain the recognition result of the obstacle. The machine learning model can obtain an output result after carrying out feature extraction and conversion on input data, wherein the feature extraction and conversion can comprise the technologies of dimension reduction, normalization, text processing, image processing and the like. The machine learning model used may be a Neural Network (NN), convolutional Neural Network (CNN), cyclic neural network (RNN), or the like.
If the existence probability of the obstacle of the track section to be detected is more than 0.01 and less than 0.1, the visible light cameras and the infrared cameras are adopted to cooperatively detect the track section to be detected; as an example, a visible light camera and an infrared camera can be used for scanning the track section to be detected together, and the scanning result is input into a trained second machine learning model to obtain the recognition result of the obstacle.
And if the existence probability of the obstacle of the track section to be detected is smaller than or equal to 0.01, detecting the track section to be detected by adopting an ultrasonic sensor. As an example, an ultrasonic sensor may be used to scan the track segment to be detected, and the scan result may be input into a trained third machine learning model to obtain the recognition result of the obstacle.
In some embodiments, the position information of each track segment to be detected and the detection mode corresponding to each track segment to be detected may be further obtained, and the total detection time is determined based on the position information of each track segment to be detected and the detection mode corresponding to each track segment to be detected by using a time determination model, where the time determination model is a deep neural network model. The deep neural network model includes a deep neural network (Deep Neural Networks, DNN).
The position information of each track section to be detected comprises a total length of each track section to be detected, a length of a bent section, a length of a straight section and a plurality of position coordinates from a starting point to an ending point.
The total length of the track segment can directly affect the detection time, as the detection system requires data acquisition and analysis over the entire track segment. Longer track segments may require longer time to complete the detection operation, while shorter track segments may require less time.
Curved road segments generally require more time to detect than straight road segments. This is because at bends, the shape of the subway track may change, and more visual information and data processing may be required to detect obstacles or anomalies.
Straight road segments are generally easier to detect than curved road segments because they have a simpler and regular structure. For straight road sections, the detection system can capture various conditions on the track more quickly and process the conditions correspondingly.
The start point to end point position coordinates may be used to represent the position of the entire track segment to be detected and the start point to end point position coordinates.
The total detection time represents the sum of the detection times of each track segment to be detected.
Based on the same inventive concept, fig. 2 is a schematic diagram of a subway rail obstacle detection system based on a neural network according to an embodiment of the present invention, where the subway rail obstacle detection system based on the neural network includes:
a first obtaining module 21, configured to obtain a plurality of track pictures under illumination of different colors, where each track picture is a picture obtained by photographing a track under illumination of one color;
a track segment determining module 22, configured to determine a plurality of track segments to be detected by processing the plurality of track pictures under different color illuminations based on a graph neural network model, where an input of the graph neural network model includes a plurality of nodes and a plurality of edges between the plurality of nodes, each node of the plurality of nodes represents a track under each color illumination, each node of the plurality of nodes includes a plurality of node features, the plurality of node features includes a track picture under each color illumination, illumination intensities of a plurality of light sources corresponding to each color illumination, and a color histogram of the track picture under each color illumination, and an output of the graph neural network model is the plurality of track segments to be detected;
the second obtaining module 23 is configured to obtain the monitoring videos of the plurality of track segments to be detected and the pictures of the plurality of track segments to be detected under different color illumination;
the probability determining module 24 is configured to determine the existence probabilities of the obstacles of the plurality of track segments to be detected based on the monitoring videos of the plurality of track segments to be detected and the pictures of the plurality of track segments to be detected under different color illumination using a probability determining model;
the detection mode determining module 25 is configured to determine a detection mode of each of the plurality of track segments to be detected based on the existence probabilities of the obstacles of the plurality of track segments to be detected.
Claims (8)
1. The subway rail obstacle detection method based on the neural network is characterized by comprising the following steps of:
acquiring a plurality of track pictures under different color illumination, wherein each track picture is obtained by shooting a track under one color illumination;
processing the plurality of track pictures under different color illumination to determine a plurality of track segments to be detected based on a graph neural network model, wherein the input of the graph neural network model comprises a plurality of nodes and a plurality of edges between the plurality of nodes, each node in the plurality of nodes represents a track under each color illumination, each node in the plurality of nodes comprises a plurality of node features, each node feature comprises a track picture under each color illumination, illumination intensity of a plurality of light sources corresponding to each color illumination, and a color histogram of the track picture under each color illumination, and the output of the graph neural network model is the plurality of track segments to be detected;
acquiring monitoring videos of the track segments to be detected and pictures of the track segments to be detected under different colors of illumination;
determining the existence probability of the obstacle of the track segments to be detected based on the monitoring videos of the track segments to be detected and the picture use probability determining model of the track segments to be detected under different color illumination;
determining a detection mode of each track segment to be detected in the track segments to be detected based on the existence probabilities of the obstacles of the track segments to be detected, wherein the detection mode of the track segments to be detected comprises the steps of adopting a visible light camera, a laser radar and an infrared camera to cooperatively detect the track segments to be detected, adopting the visible light camera and the infrared camera to cooperatively detect the track segments to be detected, adopting an ultrasonic sensor to detect the track segments to be detected, the resolution of the visible light camera is larger than 1080P, and determining the detection mode of each track segment to be detected in the track segments to be detected based on the existence probabilities of the obstacles of the track segments to be detected, wherein the detection mode of the track segments to be detected comprises the following steps:
if the existence probability of the obstacle of the track section to be detected is greater than 0.1, the visible light camera, the laser radar and the infrared camera are adopted to cooperatively detect the track section to be detected;
if the existence probability of the obstacle of the track section to be detected is more than 0.01 and less than 0.1, the visible light cameras and the infrared cameras are adopted to cooperatively detect the track section to be detected;
and if the existence probability of the obstacle of the track section to be detected is smaller than or equal to 0.01, detecting the track section to be detected by adopting an ultrasonic sensor.
2. The method for detecting subway rail obstacles based on a neural network according to claim 1, wherein the plurality of rail pictures under different color illumination include a plurality of rail pictures under white light illumination, a plurality of rail pictures under red light illumination, a plurality of rail pictures under green light illumination, a plurality of rail pictures under blue light illumination, and a plurality of rail pictures under ultraviolet light illumination.
3. The neural network-based subway rail obstacle detection method as claimed in claim 1, wherein the probability determination model is a long-short neural network model, the inputs of the probability determination model are the monitoring videos of the rail segments to be detected and pictures of the rail segments to be detected under different colors of illumination, and the probability determination model outputs the obstacle existence probabilities of the rail segments to be detected.
4. The neural network-based subway rail obstacle detection method as claimed in claim 1, wherein the method further comprises: and acquiring the position information of each track segment to be detected and the detection mode corresponding to each track segment to be detected, and determining the total detection time based on the position information of each track segment to be detected and the detection mode corresponding to each track segment to be detected by using a time determination model, wherein the time determination model is a deep neural network model.
5. A subway rail obstacle detection system based on a neural network, comprising:
the first acquisition module is used for acquiring a plurality of track pictures under illumination of different colors, wherein each track picture is a picture obtained by shooting a track under illumination of one color;
the track segment determining module is used for processing the plurality of track pictures under different color illumination to determine a plurality of track segments to be detected based on a graph neural network model, wherein the input of the graph neural network model comprises a plurality of nodes and a plurality of edges between the plurality of nodes, each node in the plurality of nodes represents a track under each color illumination, each node in the plurality of nodes comprises a plurality of node characteristics, each node characteristic comprises a track picture under each color illumination, illumination intensity of a plurality of light sources corresponding to each color illumination, and a color histogram of the track picture under each color illumination, and the output of the graph neural network model is the plurality of track segments to be detected;
the second acquisition module is used for acquiring the monitoring videos of the plurality of track segments to be detected and the pictures of the plurality of track segments to be detected under different colors of illumination;
the probability determining module is used for determining the existence probability of the obstacle of the plurality of track segments to be detected based on the monitoring videos of the plurality of track segments to be detected and the pictures of the plurality of track segments to be detected under different colors of illumination by using a probability determining model;
the detection mode determining module is used for determining the detection mode of each track segment to be detected in the track segments to be detected based on the existence probabilities of the barriers of the track segments to be detected, wherein the detection mode of the track segments to be detected comprises the steps of adopting a visible light camera, a laser radar and an infrared camera to cooperatively detect the track segments to be detected, adopting the visible light camera and the infrared camera to cooperatively detect the track segments to be detected, adopting an ultrasonic sensor to detect the track segments to be detected, and the resolution of the visible light camera is larger than 1080P, and the detection mode determining module is further used for:
if the existence probability of the obstacle of the track section to be detected is greater than 0.1, the visible light camera, the laser radar and the infrared camera are adopted to cooperatively detect the track section to be detected;
if the existence probability of the obstacle of the track section to be detected is more than 0.01 and less than 0.1, the visible light cameras and the infrared cameras are adopted to cooperatively detect the track section to be detected;
and if the existence probability of the obstacle of the track section to be detected is smaller than or equal to 0.01, detecting the track section to be detected by adopting an ultrasonic sensor.
6. The neural network-based subway rail obstacle detection system of claim 5, wherein the plurality of rail pictures under different color illumination include a plurality of rail pictures under white light illumination, a plurality of rail pictures under red light illumination, a plurality of rail pictures under green light illumination, a plurality of rail pictures under blue light illumination, and a plurality of rail pictures under ultraviolet light illumination.
7. The neural network-based subway rail obstacle detection system according to claim 5, wherein the probability determination model is a long-short neural network model, the inputs of the probability determination model are the monitoring videos of the rail segments to be detected and pictures of the rail segments to be detected under different colors of illumination, and the probability determination model outputs the obstacle existence probabilities of the rail segments to be detected.
8. The neural network-based subway rail obstacle detection system as set forth in claim 5, further configured to: and acquiring the position information of each track segment to be detected and the detection mode corresponding to each track segment to be detected, and determining the total detection time based on the position information of each track segment to be detected and the detection mode corresponding to each track segment to be detected by using a time determination model, wherein the time determination model is a deep neural network model.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310968414.2A CN116721093B (en) | 2023-08-03 | 2023-08-03 | Subway rail obstacle detection method and system based on neural network |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310968414.2A CN116721093B (en) | 2023-08-03 | 2023-08-03 | Subway rail obstacle detection method and system based on neural network |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN116721093A CN116721093A (en) | 2023-09-08 |
| CN116721093B true CN116721093B (en) | 2023-10-31 |
Family
ID=87868209
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202310968414.2A Active CN116721093B (en) | 2023-08-03 | 2023-08-03 | Subway rail obstacle detection method and system based on neural network |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN116721093B (en) |
Citations (11)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US9830503B1 (en) * | 2014-12-31 | 2017-11-28 | Morphotrust Usa, Llc | Object detection in videos |
| CN109544990A (en) * | 2018-12-12 | 2019-03-29 | 惠州市德赛西威汽车电子股份有限公司 | A kind of method and system that parking position can be used based on real-time electronic map identification |
| CN111126262A (en) * | 2019-12-24 | 2020-05-08 | 中国科学院自动化研究所 | Video highlight detection method and device based on graph neural network |
| CN111462192A (en) * | 2020-02-24 | 2020-07-28 | 江苏大学 | Space-time double-current fusion convolutional neural network dynamic obstacle avoidance method for sidewalk sweeping robot |
| CN112966609A (en) * | 2021-03-05 | 2021-06-15 | 北京百度网讯科技有限公司 | Target detection method and device |
| CN113029154A (en) * | 2021-04-01 | 2021-06-25 | 北京深睿博联科技有限责任公司 | Navigation method and device for blind person |
| CN113553916A (en) * | 2021-06-30 | 2021-10-26 | 广西大学 | Orbit dangerous area obstacle detection method based on convolutional neural network |
| CN113689483A (en) * | 2021-07-23 | 2021-11-23 | 浙江零跑科技股份有限公司 | Distance measurement method based on monocular camera and laser radar |
| CN114266993A (en) * | 2021-12-10 | 2022-04-01 | 广州海格星航信息科技有限公司 | Image-based road environment detection method and device |
| CN115729245A (en) * | 2022-11-25 | 2023-03-03 | 青岛慧拓智能机器有限公司 | Obstacle fusion detection method, chip and terminal for mine ramp |
| CN116117817A (en) * | 2023-02-28 | 2023-05-16 | 广东电网有限责任公司 | Obstacle clearing robot control method, obstacle clearing robot control device, robot and storage medium |
Family Cites Families (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110298298B (en) * | 2019-06-26 | 2022-03-08 | 北京市商汤科技开发有限公司 | Target detection and target detection network training method, device and equipment |
-
2023
- 2023-08-03 CN CN202310968414.2A patent/CN116721093B/en active Active
Patent Citations (11)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US9830503B1 (en) * | 2014-12-31 | 2017-11-28 | Morphotrust Usa, Llc | Object detection in videos |
| CN109544990A (en) * | 2018-12-12 | 2019-03-29 | 惠州市德赛西威汽车电子股份有限公司 | A kind of method and system that parking position can be used based on real-time electronic map identification |
| CN111126262A (en) * | 2019-12-24 | 2020-05-08 | 中国科学院自动化研究所 | Video highlight detection method and device based on graph neural network |
| CN111462192A (en) * | 2020-02-24 | 2020-07-28 | 江苏大学 | Space-time double-current fusion convolutional neural network dynamic obstacle avoidance method for sidewalk sweeping robot |
| CN112966609A (en) * | 2021-03-05 | 2021-06-15 | 北京百度网讯科技有限公司 | Target detection method and device |
| CN113029154A (en) * | 2021-04-01 | 2021-06-25 | 北京深睿博联科技有限责任公司 | Navigation method and device for blind person |
| CN113553916A (en) * | 2021-06-30 | 2021-10-26 | 广西大学 | Orbit dangerous area obstacle detection method based on convolutional neural network |
| CN113689483A (en) * | 2021-07-23 | 2021-11-23 | 浙江零跑科技股份有限公司 | Distance measurement method based on monocular camera and laser radar |
| CN114266993A (en) * | 2021-12-10 | 2022-04-01 | 广州海格星航信息科技有限公司 | Image-based road environment detection method and device |
| CN115729245A (en) * | 2022-11-25 | 2023-03-03 | 青岛慧拓智能机器有限公司 | Obstacle fusion detection method, chip and terminal for mine ramp |
| CN116117817A (en) * | 2023-02-28 | 2023-05-16 | 广东电网有限责任公司 | Obstacle clearing robot control method, obstacle clearing robot control device, robot and storage medium |
Also Published As
| Publication number | Publication date |
|---|---|
| CN116721093A (en) | 2023-09-08 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| JP6797860B2 (en) | Water intrusion detection system and its method | |
| EP1675066A1 (en) | Face image candidate area search method, face image candidate area search system, and face image candidate area search program | |
| WO2012022744A2 (en) | Multi-mode video event indexing | |
| KR101040049B1 (en) | Fast and robust method for tracking the multiple moving objects in video-based surveillance system | |
| Nurhadiyatna et al. | Background subtraction using gaussian mixture model enhanced by hole filling algorithm (gmmhf) | |
| CN110490043A (en) | A kind of forest rocket detection method based on region division and feature extraction | |
| Salehi et al. | An automatic video-based drowning detection system for swimming pools using active contours | |
| CN110096945B (en) | Indoor monitoring video key frame real-time extraction method based on machine learning | |
| Naufal et al. | Preprocessed mask RCNN for parking space detection in smart parking systems | |
| CN112287823A (en) | Facial mask identification method based on video monitoring | |
| Liang et al. | Methods of moving target detection and behavior recognition in intelligent vision monitoring. | |
| CN115131880A (en) | Multi-scale attention fusion double-supervision human face in-vivo detection method | |
| Tao et al. | Smoky vehicle detection based on range filtering on three orthogonal planes and motion orientation histogram | |
| CN101976342B (en) | Space-time collaborative determination-based cat eye effect target identification method | |
| Shafie et al. | Smart video surveillance system for vehicle detection and traffic flow control | |
| CN116721093B (en) | Subway rail obstacle detection method and system based on neural network | |
| CN113221603A (en) | Method and device for detecting shielding of monitoring equipment by foreign matters | |
| CN115147450B (en) | Moving target detection method and detection device based on motion frame difference image | |
| CN111583341B (en) | Cloud deck camera shift detection method | |
| Hussien | Detection and tracking system of moving objects based on MATLAB | |
| Patel et al. | Depthwise convolution for compact object detector in nighttime images | |
| Bao et al. | A new approach to mouth detection using neural network | |
| Xu | An efficient crowd estimation method using convolutional neural network with thermal images | |
| Frías-Velázquez et al. | A mathematical morphology-based approach for vehicle detection in road tunnels | |
| Hayat et al. | Mask R-CNN based real time near drowning person detection system in swimming pools |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |