CN112686215B - Track tracking monitoring and early warning system and method for carrier vehicle - Google Patents
Track tracking monitoring and early warning system and method for carrier vehicle Download PDFInfo
- Publication number
- CN112686215B CN112686215B CN202110105837.2A CN202110105837A CN112686215B CN 112686215 B CN112686215 B CN 112686215B CN 202110105837 A CN202110105837 A CN 202110105837A CN 112686215 B CN112686215 B CN 112686215B
- Authority
- CN
- China
- Prior art keywords
- carrier vehicle
- track
- early warning
- target
- vehicle
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Classifications
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02T—CLIMATE CHANGE MITIGATION TECHNOLOGIES RELATED TO TRANSPORTATION
- Y02T10/00—Road transport of goods or passengers
- Y02T10/10—Internal combustion engine [ICE] based vehicles
- Y02T10/40—Engine management systems
Landscapes
- Image Analysis (AREA)
Abstract
The invention provides a track tracking monitoring early warning system and method of a carrier vehicle, which relate to the technical field of track tracking monitoring of targets, wherein real-time videos of the carrier vehicle are collected through video collecting equipment and transmitted to a preprocessing module for preprocessing, the area of the carrier vehicle is preliminarily positioned, a target detection module detects the area of the current frame of the real-time videos which is preliminarily positioned, HOG characteristics of each frame of images are extracted through a characteristic extraction unit, the real-time positions of the carrier vehicle are determined by utilizing a carrier vehicle SVM classifier, the carrier vehicle is tracked by adopting a KCF algorithm, the comprehensive processing module judges whether the carrier vehicle is abnormal or not, and abnormal information is sent to a server; the server receives and stores the abnormal information sent by the comprehensive processing module, sends an early warning instruction to the early warning module, generates emergency response measures, and sends the early warning information to staff by the early warning module, so that the cost of manual inspection is reduced, and the risk of personal accidental injury is reduced.
Description
Technical Field
The invention relates to the technical field of target track tracking and monitoring, in particular to a track tracking and monitoring early warning system and method of a carrier vehicle.
Background
With new demands of global markets for enterprise profitability and competitiveness, how to effectively ensure the safety of production processes and improve the quality and production efficiency of products is a major challenge facing modern process industry. Along with the rising of labor cost, the manufacturing industry starts to develop towards automation and intellectualization, and the carrier vehicle gradually becomes one of important equipment for safe and efficient production of factories, and can automatically carry goods according to a track preset on the ground.
However, for tracking and monitoring during the running of the carrier vehicle, a factory often adopts a manual inspection mode, and the following disadvantages exist: (1) A plurality of carrier vehicles exist in a factory at the same time, the transport paths of different carrier vehicles are different, and the manual inspection efficiency is low; (2) When a certain carrier vehicle fails, such as power failure, speed reduction, overspeed and derailment, manual inspection is difficult to find in time and immediately early-warning, and more inspection manpower and response time are required for the condition of multiple carrier vehicle failures; (3) In some scenes with harm or potential danger to human bodies, a manual inspection method cannot be adopted, for example, raw materials of chemical industry factories possibly contain toxic and harmful substances, and once the raw materials leak, the health of workers can be endangered; (4) Manual inspection is unfavorable for the development of the whole automation and the intellectualization of the factory.
Some target tracking algorithms are continuously proposed at present to overcome the defect of manual inspection, for example, chinese patent (publication number: CN 107341820A) discloses a mutation moving target tracking method integrating Cukoo search and KCF, a Cukoo search mechanism is adopted to obtain a globally optimal target prediction state, a new base image sample is generated, a KCF method is executed to track a target, and the precise tracking of the moving mutation target existing between frames is realized from a theoretical level.
Disclosure of Invention
In order to solve the problems of high cost, low efficiency and long response time of manual inspection, the invention provides a track tracking, monitoring and early warning system and method for a carrier vehicle, which are used for reducing the cost and dangerous factors of manual inspection, shortening the emergency response time and improving the reliability of safe production of factories.
In order to achieve the technical effects, the technical scheme of the invention is as follows:
a track following monitoring and early warning system for a carrier vehicle, comprising:
the video acquisition equipment is used for acquiring real-time videos of the traveling of the carrier vehicle;
the preprocessing module is used for receiving real-time continuous videos of the vehicle traveling transmitted by the video acquisition equipment, carrying out sliding window preprocessing on the real-time continuous videos to generate a pre-inspection frame, scanning images in the pre-inspection frame through the sliding window, extracting HOG characteristics and carrying out preliminary positioning on the vehicle;
the target detection module comprises a feature extraction unit and a carrier vehicle SVM classifier, wherein the feature extraction unit extracts HOG features of each frame of image, and the real-time position of the carrier vehicle is determined by classifying by the carrier vehicle SVM classifier;
the target tracking module is used for tracking the carrier vehicle by adopting a KCF algorithm;
the comprehensive processing module is used for judging whether the traveling of the carrier vehicle is abnormal and sending abnormal information to the server;
the server receives and stores the abnormal information sent by the comprehensive processing module, issues an early warning instruction to the early warning module and generates emergency response measures;
and the early warning module is used for receiving an early warning instruction issued by the server and sending early warning information to staff.
In the technical scheme, a video acquisition device acquires real-time video of the running of the carrier vehicle, the real-time video is transmitted to a preprocessing module for preprocessing, the area of the carrier vehicle is preliminarily positioned, a target detection module detects the area of the current frame of the real-time video which is preliminarily positioned, HOG characteristics of each frame of image are extracted through a characteristic extraction unit, the real-time position of the carrier vehicle is determined by classifying by using a carrier vehicle SVM classifier, a target tracking module tracks the carrier vehicle by adopting a KCF algorithm, a comprehensive processing module judges whether the running of the carrier vehicle is abnormal, and abnormal information is sent to a server; the server receives and stores the abnormal information sent by the comprehensive processing module, sends an early warning instruction to the early warning module, generates emergency response measures, and sends early warning information to staff.
Preferably, before classifying by using the vehicle SVM classifier, a worker determines a vehicle data set to be tracked and monitored, and the target detection module performs HOG feature extraction by using the feature extraction unit, and the extracted HOG feature is used as an input of the vehicle SVM classifier to train the vehicle SVM classifier.
Here, the SVM refers to a support vector machine, which is a common discrimination method. In the field of machine learning, the method is a supervised learning model which is commonly used for pattern recognition, classification and regression analysis, the method is used as a classifier for distinguishing a carrier vehicle target from a non-carrier vehicle target during carrier vehicle monitoring and tracking in the technical scheme, the SVM is a classification model on a feature space, the optimal classification plane is solved by maximizing a gap, the hardware cost can be reduced based on an image processing method, the adaptability and the configurability are higher, the expandability of a software algorithm is better, a data set or related algorithms can be replaced according to requirements to realize monitoring of various targets, and the diversified and intelligent requirements under an industrial scene are met.
Preferably, the abnormal information sent by the integrated processing module comprises information of running track deviation, overspeed, speed reduction and unexpected stop of the carrier vehicle.
Preferably, the early warning module includes:
the first-level early warning unit is used for sending first-level early warning information to staff when the carrier vehicle runs off the track;
the secondary early warning unit sends secondary early warning information to staff when the carrier vehicle is not started on time or stopped in the running process;
the three-level early warning unit is used for sending three-level early warning information to staff when the running speed of the carrier vehicle is lower than the standard speed or higher than the standard speed; the early warning module can send out three different early warning information according to different running states of the carrier vehicle.
The invention also provides a track tracking, monitoring and early warning method of the carrier vehicle, which is realized based on the track tracking, monitoring and early warning system of the carrier vehicle and at least comprises the following steps:
s1, taking one or more times of correct running of a carrier vehicle as a reference to obtain a reference track coordinate of the running of the carrier vehicle;
s2, the video acquisition equipment acquires real-time continuous videos of the carrier vehicle running and transmits the real-time continuous videos to the preprocessing module, the preprocessing module carries out sliding window preprocessing on the real-time continuous videos to generate a pre-inspection frame, and images in the pre-inspection frame are scanned through the sliding window;
s3, extracting HOG features of each frame of image in the real-time video of the vehicle running through a feature extraction unit of the target detection module, wherein the extracted HOG features are used for classifying the SVM classifier of the vehicle, and the target detection module confirms the real-time position of the target vehicle;
s4, tracking the track of the target carrier vehicle by the target tracking module through a KCF algorithm to obtain track coordinates of the target carrier vehicle;
s5, comparing the tracked track coordinates of the target carrier vehicle with the reference track coordinates by the comprehensive processing module, judging whether the track of the target carrier vehicle is abnormal, if so, sending abnormal information to the server, sending an early warning instruction to the early warning module by the server, executing the step S6, and generating emergency response measures; otherwise, the comprehensive processing module confirms that the running track of the target carrier vehicle is normal;
s6, the early warning module receives an early warning instruction issued by the server and sends early warning information to staff.
Preferably, the process of obtaining the reference track coordinates of the vehicle in step S1 includes:
s11, a worker makes a carrier vehicle data set to be tracked and monitored, HOG feature extraction is carried out by using a feature extraction unit, the extracted HOG feature is used as input of a carrier vehicle SVM classifier, and the carrier vehicle SVM classifier is trained;
s12, acquiring a video of the correct running of the carrier vehicle by using video acquisition equipment;
s13, carrying out sliding window pretreatment on real-time continuous videos by a pretreatment module on videos which are correctly run by the carrier vehicle, generating a pre-inspection frame, scanning images in the pre-inspection frame by the sliding window, and carrying out preliminary positioning on the carrier vehicle;
s14, extracting HOG characteristics of each frame of image in the real-time video of the vehicle running through a characteristic extraction unit of the target detection module, and classifying by using a trained vehicle SVM classifier to obtain the reference track coordinates of the vehicle running.
Here, since the intelligent vehicles are of a large variety and have few network resources, a worker is required to make a data set of the vehicle to be tested, perform feature extraction on the samples, train an SVM classifier, and further, HOG features are feature descriptions for object detection in image processing, which form features by calculating and counting gradient direction histograms of local areas of images.
Preferably, the preprocessing module in step S2 performs sliding window preprocessing on the real-time continuous video, and the process of generating the pre-inspection frame includes:
S21.let the maximum speed of the carrier be V max The frame rate of the video acquisition equipment is F, and the distance maximum value V of the carrier vehicle running per second on two continuous video images of the real-time continuous video is obtained max /F；
S22, setting the resolution ratio of a video image as m multiplied by n, wherein the area shot by video acquisition equipment is a rectangle with the length of a and the width of b, and the distance between a carrier vehicle in the next frame of image and the carrier vehicle in the previous frame of image is x pixels, so that the following conditions are satisfied:
wherein m represents the number of horizontal pixels of the video image, and n represents the number of vertical pixels of the video image;
s23, in the next frame of image, taking the carrier vehicle as a central target, expanding x pixels from the central target to the periphery, and generating a pre-detection frame.
The preprocessing module can realize the rough positioning of the carrier vehicle target, and when the traditional target is detected, a sliding window method is firstly used for scanning all sub-windows in each frame of image of the video. Under industrial scenes, the installation position of the camera is generally higher, even though the camera is a high-definition camera, the proportion of the images of the carrier vehicle in the picture to the whole picture is smaller, the whole picture is scanned, the operation amount is large, the average frame rate of identification and tracking is low, the real-time performance is poor, and the maximum speed V of the carrier vehicle is determined max Generating a configurable pre-inspection frame, improving a sliding window, taking a carrier vehicle as a central target in a next frame image, and moving x pixels around at most when the carrier vehicle appears in the next frame, so that x pixels are outwards expanded from the current target center to form a pre-inspection frame, and scanning only in the pre-inspection frame when the image is scanned in the next frame, thereby greatly reducing the number of windows required to be scanned, reducing the calculated amount, and further improving the detection speed and the real-time performance of the system.
Preferably, the step S3 of the target detection module determining the real-time position of the target carrier comprises: the target vehicle is lost and the target vehicle is present and the location can be tracked.
Preferably, if the target carrier vehicle exists and the position of the target carrier vehicle can be tracked, the step S5 of comparing the tracked track coordinates of the target carrier vehicle with the reference track coordinates by the integrated processing module, and the process of determining whether the track of the target carrier vehicle is abnormal includes:
s51, when the carrier vehicle is correctly driven, the area of a reference track target detection frame taking the carrier vehicle as the center is S A The area of the actual track target detection frame taking the tracked target carrier vehicle as the center is S B Determining the intersection ratio IOU of the reference track target detection frame and the actual track target detection frame:
s52, the comprehensive processing module judges whether the IOU and the fault tolerance allowance lambda meet the following conditions: if the IOU is more than or equal to lambda, the target carrier vehicle does not deviate from the track at present, and the overspeed, deceleration and accidental stopping judging process is executed; otherwise, the target carrier vehicle is already off track at present.
Preferably, the determining process of overspeed, deceleration and unexpected stopping in step S52 includes:
s521, setting a real-time continuous video of the tracked target carrier vehicle running and a real-time continuous video of the carrier vehicle correctly running, which are acquired by video acquisition equipment, as T frames, determining a fixed-length container T, storing T frames of the real-time continuous video of the carrier vehicle correctly running in real time, wherein each frame shifts leftwards, discarding the leftmost point, and storing the reference track point with the maximum IOU at the tail end of the fixed-length container;
s522, setting the area of a reference track target detection frame in the ith frame as S Ai The area of the track target detection frame of the i-th frame tracked target carrier vehicle is S Bi Calculating the cross ratio;
s523, the comprehensive processing module judges whether the intersection ratio is smaller than the fault tolerance allowance lambda, if yes, the tracked target carrier vehicle runs abnormally, and the area S of a track target detection frame of the tracked target carrier vehicle is calculated Bi Face of target detection frame with other reference trackProduct S A Finding out the j frame with the largest cross-over ratio in the real-time continuous video track of the correct running of the carrier vehicle, updating the fixed-length container T, and executing step S524; otherwise, updating the fixed-length container T, and enabling the tracked target carrier vehicle to be normal;
s524, judging whether each frame in the long container T is j or not by the comprehensive processing module, if so, stopping the tracked target carrier vehicle accidentally; otherwise, the tracked target carrier vehicle is not stopped, and step S525 is executed;
s525, judging the area S of a track target detection frame of the tracked target carrier vehicle of the current ith frame by the comprehensive processing module Bi Area S of target detection frame with j-th frame reference track Aj If the intersection ratio of the target carrier vehicle is smaller than the fault tolerance margin lambda, if so, the tracked target carrier vehicle deviates from the track; otherwise, step S526 is performed;
s526, the comprehensive processing module judges whether j is greater than i, if yes, the tracked target carrier vehicle overspeed; otherwise, the tracked target carrier vehicle slows down.
If the intersection ratio IOU of the corresponding frame is smaller than the fault tolerance margin lambda, the intersection ratio with other correct target detection frames is calculated, and the correct track point with the maximum intersection ratio is found out. If the time of the point is positioned before the current frame, the speed of the carrier vehicle is reduced; if the time of the point is positioned after the current frame, the overspeed of the carrier vehicle is indicated; if the point is the point with the maximum intersection ratio in the continuous t frames, the vehicle is judged to stop.
Compared with the prior art, the technical scheme of the invention has the beneficial effects that:
the invention provides a track tracking, monitoring and early warning system and method of a carrier vehicle, wherein a real-time video of the running of the carrier vehicle is acquired through video acquisition equipment and is transmitted to a preprocessing module for preprocessing, so that the preliminary positioning of the area of the carrier vehicle is realized, a target detection module detects the area of the preliminary positioning of the current frame of the real-time video, HOG characteristics of each frame of image are extracted through a characteristic extraction unit, the real-time position of the carrier vehicle is determined by classifying by using a carrier vehicle SVM classifier, the target tracking module tracks the carrier vehicle by adopting a KCF algorithm, and a comprehensive processing module judges whether the running of the carrier vehicle is abnormal and sends abnormal information to a server; the server receives and stores the abnormal information sent by the comprehensive processing module, sends an early warning instruction to the early warning module, generates emergency response measures, and sends early warning information to staff.
Drawings
FIG. 1 shows a schematic diagram of a track tracking, monitoring and early warning system for a carrier vehicle according to an embodiment of the present invention;
FIG. 2 shows an initial schematic diagram of the maximum classification plane of an SVM according to an embodiment of the invention;
FIG. 3 is another schematic diagram of a maximum classification plane of an SVM according to an embodiment of the invention;
FIG. 4 is a schematic diagram of a pre-inspection box according to an embodiment of the present invention;
FIG. 5 is a schematic diagram of the cross-over ratio detection proposed in the embodiment of the present invention;
fig. 6 shows a schematic view of a fixed length container T according to an embodiment of the present invention.
Detailed Description
The drawings are for illustrative purposes only and are not to be construed as limiting the present patent;
for better illustration of the present embodiment, some parts of the drawings may be omitted, enlarged or reduced, and do not represent actual dimensions;
it will be appreciated by those skilled in the art that some well known descriptions in the figures may be omitted.
The technical scheme of the invention is further described below with reference to the accompanying drawings and examples.
The positional relationship depicted in the drawings is for illustrative purposes only and is not to be construed as limiting the present patent;
example 1
The track tracking, monitoring and early warning system of the carrier vehicle shown in fig. 1 comprises the following structural schematic diagram:
the video acquisition equipment is used for acquiring real-time videos of the traveling of the carrier vehicle;
the preprocessing module is used for receiving real-time continuous videos of the vehicle traveling transmitted by the video acquisition equipment, carrying out sliding window preprocessing on the real-time continuous videos to generate a pre-inspection frame, scanning images in the pre-inspection frame through the sliding window, extracting HOG characteristics and carrying out preliminary positioning on the vehicle;
the target detection module comprises a feature extraction unit and a carrier vehicle SVM classifier, wherein the feature extraction unit extracts HOG features of each frame of image, and the real-time position of the carrier vehicle is determined by classifying by the carrier vehicle SVM classifier;
the target tracking module is used for tracking the carrier vehicle by adopting a KCF algorithm;
the comprehensive processing module is used for judging whether the traveling of the carrier vehicle is abnormal and sending abnormal information to the server;
the server receives and stores the abnormal information sent by the comprehensive processing module, issues an early warning instruction to the early warning module and generates emergency response measures;
and the early warning module is used for receiving an early warning instruction issued by the server and sending early warning information to staff.
The track tracking monitoring early warning system of the carrier vehicle comprises video acquisition equipment, a preprocessing module, a target detection module, a target tracking module, a comprehensive processing module, a server and an early warning module, wherein in the embodiment, the video acquisition equipment is a high-definition camera on the occasion where the carrier vehicle is located, the video acquisition equipment acquires real-time video of the running of the carrier vehicle, the real-time video is transmitted to the preprocessing module for preprocessing, the preliminary positioning of the area of the carrier vehicle is realized, the target detection module detects the area of the preliminary positioning of the current frame of the real-time video, the HOG characteristic of each frame of image is extracted through a characteristic extraction unit, and the process of extracting the HOG characteristic of each frame of image by the characteristic extraction unit comprises the following steps:
A. carrying out normalization processing on each frame of image in the real-time video of the carrier vehicle running, confirming the image gradient of each frame of image according to the abscissa direction and the ordinate direction of each frame of image, and calculating the gradient direction value of each pixel position according to the image gradient;
B. voting the histogram channel based on the direction by taking the gradient direction value of each pixel position as a weight value to form a histogram based on the direction;
C. all histograms are combined to form HOG characteristics of each frame of image, in the embodiment, undirected gradients and 9 histogram channels are adopted to achieve the optimal effect of carrier vehicle detection, and the gradient change range among each cell unit has larger difference due to difference of illumination, foreground and background contrast, so that normalization processing is required for gradient strength. By combining the cell units into large spatially connected regions, the descriptions of overlapping cells ultimately act on the whole, further reducing the effects of illumination, shadowing, and edges.
Then, classifying by using a carrier SVM classifier, identifying a target carrier, determining the real-time position of the carrier, referring to FIG. 2 and FIG. 3, wherein a circle represents the target carrier, a square represents other objects, l in FIG. 2 is a classification line, l 1 、l 2 To distinguish between vehicles and other objects and a straight line parallel to the sorting line, the sorting interval is the straight line 1 And l 2 The greater the distance between them, the better the classification, see fig. 3, straight line l' 1 And l' 2 The distance between the two planes is increased, the weight coefficient in the SVM classifier is optimized through sample training, the plane with the largest classification interval is found to be the optimal classification plane, the plane decision is used for classifying the image, and the target class is judged;
the target tracking module tracks the carrier vehicle by adopting a KCF algorithm, the KCF algorithm abstracts the tracking problem into a solution of a linear regression model, the processing speed of the algorithm is greatly improved by dot product instead of matrix inversion, when the nonlinear problem is faced, the nonlinear problem can be converted into a linear problem solution by a kernel function, and the sampling data of a sample is assumed to be x i The linear regression function is:
f(x i )＝ω T x i
wherein ω is a column vector representing a linear regression coefficient, let y be i To predict the distance between the position of the carrier vehicle and the real position of the carrier vehicle, y i The smaller the value of (2), the higher the accuracy, i.e.:
lambda is a regularization parameter. The bias is calculated and is equal to 0, and the following steps can be obtained:
ω＝(X T X+λI) -1 X T y
since complex matrices may occur, the up-conversion is to conjugate matrices whose complex domain form is as follows:
ω＝(X H X+λI) -1 X H y
the conjugate and inversion operations exist, and the complexity is high, so that the nonlinear problem is converted into the linear problem by introducing a kernel function. And (3) making:
let K denote the kernel space matrix, it is possible to obtain:
α＝(K+λI) -1 y
where α is a linear regression coefficient, i.e., a tracker template. The method comprises the following steps:
the combined cyclic matrix properties can be obtained:
the correlation is obtained by the current input frame and the filter, and then the regression coefficient is multiplied, and the maximum correlation is the tracking result; then the comprehensive processing module judges whether the traveling of the carrier vehicle is abnormal or not and sends abnormal information to the server; the server receives and stores the abnormal information sent by the comprehensive processing module, sends an early warning instruction to the early warning module, generates emergency response measures, and sends early warning information to staff.
Before classifying by using the vehicle SVM classifier, a worker determines a vehicle data set to be tracked and monitored, and the target detection module extracts HOG features by using the feature extraction unit, wherein the extracted HOG features are used as the input of the vehicle SVM classifier to train the vehicle SVM classifier. The SVM refers to a support vector machine, and is a common distinguishing method. In the field of machine learning, the method is a supervised learning model which is commonly used for pattern recognition, classification and regression analysis, the method is used as a classifier for distinguishing a carrier vehicle target from a non-carrier vehicle target during carrier vehicle monitoring and tracking in the technical scheme, the SVM is a classification model on a feature space, the optimal classification plane is solved by maximizing a gap, the hardware cost can be reduced based on an image processing method, the adaptability and the configurability are higher, the expandability of a software algorithm is better, a data set or related algorithms can be replaced according to requirements to realize monitoring of various targets, and the diversified and intelligent requirements under an industrial scene are met.
The abnormal information sent by the comprehensive processing module comprises information of track deviation, overspeed, deceleration and accidental stop of the carrier vehicle. Referring to fig. 1, the early warning module includes:
the first-level early warning unit is used for sending first-level early warning information to staff when the carrier vehicle runs off the track;
the secondary early warning unit sends secondary early warning information to staff when the carrier vehicle is not started on time or stopped in the running process;
the three-level early warning unit is used for sending three-level early warning information to staff when the running speed of the carrier vehicle is lower than the standard speed or higher than the standard speed; the early warning module can send out three different early warning information according to different running states of the carrier vehicle.
The invention also provides a track tracking, monitoring and early warning method of the carrier vehicle, which is realized based on the track tracking, monitoring and early warning system of the carrier vehicle and comprises the following steps:
s1, taking one or more times of correct running of a carrier vehicle as a reference to obtain a reference track coordinate of the running of the carrier vehicle; firstly, carrying out correct running of one or more carrier vehicles so as to facilitate the abnormality discrimination of the comprehensive processing module;
s2, the video acquisition equipment acquires real-time continuous videos of the carrier vehicle running and transmits the real-time continuous videos to the preprocessing module, the preprocessing module carries out sliding window preprocessing on the real-time continuous videos to generate a pre-inspection frame, and images in the pre-inspection frame are scanned through the sliding window;
s3, extracting HOG features of each frame of image in the real-time video of the vehicle running through a feature extraction unit of the target detection module, wherein the extracted HOG features are used for classifying the SVM classifier of the vehicle, and the target detection module confirms the real-time position of the target vehicle;
s4, tracking the track of the target carrier vehicle by the target tracking module through a KCF algorithm to obtain track coordinates of the target carrier vehicle;
s5, comparing the tracked track coordinates of the target carrier vehicle with the reference track coordinates by the comprehensive processing module, judging whether the track of the target carrier vehicle is abnormal, if so, sending abnormal information to the server, sending an early warning instruction to the early warning module by the server, executing the step S6, and generating emergency response measures; otherwise, the comprehensive processing module confirms that the running track of the target carrier vehicle is normal;
s6, the early warning module receives an early warning instruction issued by the server and sends early warning information to staff.
In this embodiment, the process of obtaining the reference track coordinates of the vehicle in step S1 includes:
s11, a worker makes a carrier vehicle data set to be tracked and monitored, HOG feature extraction is carried out by using a feature extraction unit, the extracted HOG feature is used as input of a carrier vehicle SVM classifier, and the carrier vehicle SVM classifier is trained;
s12, acquiring a video of the correct running of the carrier vehicle by using video acquisition equipment;
s13, carrying out sliding window pretreatment on real-time continuous videos by a pretreatment module on videos which are correctly run by the carrier vehicle, generating a pre-inspection frame, scanning images in the pre-inspection frame by the sliding window, and carrying out preliminary positioning on the carrier vehicle;
s14, extracting HOG characteristics of each frame of image in the real-time video of the vehicle running through a characteristic extraction unit of the target detection module, classifying by using a trained vehicle SVM classifier to obtain reference track coordinates of the vehicle running, and firstly carrying out one or more times of correct running of the vehicle so as to facilitate the subsequent abnormality judgment.
In this embodiment, the preprocessing module in step S2 performs sliding window preprocessing on the real-time continuous video, and the process of generating the pre-inspection frame includes:
s21, setting the maximum speed of the carrier vehicle as V max The frame rate of the video acquisition equipment is F, and the distance maximum value V of the carrier vehicle running per second on two continuous video images of the real-time continuous video is obtained max /F；
S22, setting the resolution ratio of a video image as m multiplied by n, wherein the area shot by video acquisition equipment is a rectangle with the length of a and the width of b, and the distance between a carrier vehicle in the next frame of image and the carrier vehicle in the previous frame of image is x pixels, so that the following conditions are satisfied:
wherein m represents the number of horizontal pixels of the video image, and n represents the number of vertical pixels of the video image;
s23, in the next frame of image, taking the carrier vehicle as a central target, expanding x pixels from the central target to the periphery to generate a pre-inspection frame, wherein a schematic diagram of the generated pre-inspection frame is shown in fig. 4, the central point is the central coordinate of the carrier vehicle of the current frame, the resolution ratio of the video image is m multiplied by n, and when the carrier vehicle appears in the next frame, the carrier vehicle moves by x pixels to the periphery at most.
In this embodiment, the result of the target detection module determining the real-time position of the target carrier in step S3 includes: the target vehicle is lost and the target vehicle is present and the location can be tracked. If the target carrier vehicle exists and the position can be tracked, the step S5 of the comprehensive processing module compares the tracked track coordinates of the target carrier vehicle with the reference track coordinates, and the process of judging whether the track of the target carrier vehicle is abnormal comprises the following steps:
s51, when the carrier vehicle is correctly driven, the area of a reference track target detection frame taking the carrier vehicle as the center is S A The area of the actual track target detection frame taking the tracked target carrier vehicle as the center is S B Determining the intersection ratio IOU of the reference track target detection frame and the actual track target detection frame:
s52, the comprehensive processing module judges whether the IOU and the fault tolerance allowance lambda meet the following conditions: if the IOU is more than or equal to lambda, the target carrier vehicle does not deviate from the track at present, and the overspeed, deceleration and accidental stopping judging process is executed; otherwise, the target carrier vehicle is already off track at present. Specifically, as further described in conjunction with FIG. 5, A as shown in FIG. 5 1 、A 2 、A 3 The center point coordinates of the carrier vehicle on the track for correct running are B 1 、B 2 、B 3 Calculating the area of a target detection frame of the correct track of each frame when the system monitors the coordinates of the central point of the carrier vehicle on the actual track of the corresponding moment (frame), and then determining the intersection ratio; the actual track point cannot coincide with the correct track point due to the influences of ground unevenness, shooting angles, detection precision and the like, so that the judgment of the intersection ratio needs to set a fault tolerance margin lambda, and lambda is influenced by the actual track width, so that a proper value can be selected in the actual detection. When IOU is more than or equal to lambda, the carrier vehicle is considered to have no offset rail in the current frame; when IOU is less than lambda, the carrier vehicle is considered to be off track in the current frame, the abnormal condition of the carrier vehicle is judged by using the intersection ratio of the reference track point and the actual track point detection frame, and compared with the three-dimensional modeling and sensor judgment, the method has the advantages of small calculated amount, suitability for embedded implementation and reduction of the whole vehicleThe cost of the software and hardware is high, and the method has great advantages in occasions with low requirements on precision and important real-time performance.
However, considering that the cross-over ratio with the corresponding frame is small when the vehicle has problems such as overspeed, deceleration and unexpected stop, a decision mechanism is required to determine different anomalies of the vehicle, in this embodiment, the determining process of overspeed, deceleration and unexpected stop in step S52 includes:
s521, setting a real-time continuous video of the tracked target carrier vehicle running and a real-time continuous video of the carrier vehicle correctly running, which are acquired by video acquisition equipment, as T frames, determining a fixed-length container T, storing T frames of the real-time continuous video of the carrier vehicle correctly running in real time, wherein each frame shifts leftwards, discarding the leftmost point, and storing the reference track point with the maximum IOU at the tail end of the fixed-length container;
s522, setting the area of a reference track target detection frame in the ith frame as S Ai The area of the track target detection frame of the i-th frame tracked target carrier vehicle is S Bi Calculating the cross ratio;
s523, the comprehensive processing module judges whether the intersection ratio is smaller than the fault tolerance allowance lambda, if yes, the tracked target carrier vehicle runs abnormally, and the area S of a track target detection frame of the tracked target carrier vehicle is calculated Bi Area S of target detection frame with other reference track A Finding out the j frame with the largest cross-over ratio in the real-time continuous video track of the correct running of the carrier vehicle, updating the fixed-length container T, and executing step S524; otherwise, updating the fixed-length container T, and enabling the tracked target carrier vehicle to be normal;
s524, judging whether each frame in the long container T is j or not by the comprehensive processing module, if so, stopping the tracked target carrier vehicle accidentally; otherwise, the tracked target carrier vehicle is not stopped, and step S525 is executed;
s525, judging the area S of a track target detection frame of the tracked target carrier vehicle of the current ith frame by the comprehensive processing module Bi Area S of target detection frame with j-th frame reference track Aj If the intersection ratio of the target carrier vehicle is smaller than the fault tolerance margin lambda, if so, the tracked target carrier vehicle deviates from the track; otherwise, executeLine step S526;
s526, the comprehensive processing module judges whether j is greater than i, if yes, the tracked target carrier vehicle overspeed; otherwise, the tracked target carrier vehicle slows down, namely firstly judging the intersection ratio IOU of the corresponding frame, if the intersection ratio is smaller than the fault tolerance margin lambda, calculating the intersection ratio with other correct target detection frames, and finding out the correct track point with the maximum intersection ratio. If the time of the point is positioned before the current frame, the speed of the carrier vehicle is reduced; if the time of the point is positioned after the current frame, the overspeed of the carrier vehicle is indicated; if the points are all points with the largest intersection ratio in the continuous T frames, the carrier vehicle is judged to stop, specifically, as shown in fig. 6, considering that when monitoring whether the carrier vehicle stops or not, the continuous T frames in the video image of the carrier vehicle running need to be judged, and whether the points with the largest intersection ratio with the current i frame detection frame are the same or not needs to be judged, therefore, a fixed-length container T is designed and used for storing the point with the largest correct track of the T frame IOU in real time, as shown in fig. 6, the length of the fixed-length container T is T, each frame is shifted leftwards, the leftmost point is discarded, and the point with the largest correct track of the IOU is stored at the tail end (right side) of the container at the moment, as shown in fig. 6. When judging whether the carrier vehicle stops, only the t points need to be traversed, if the t points are the same, the fact that the carrier vehicle is not changed, namely the vehicle stops, the value of t can be determined by actual demands, various actual running states of the carrier vehicle can be judged through abnormal decision, early warning information can be sent timely, and the safety production is guaranteed through linkage with emergency management.
The positional relationship depicted in the drawings is for illustrative purposes only and is not to be construed as limiting the present patent;
it is to be understood that the above examples of the present invention are provided by way of illustration only and are not intended to limit the scope of the invention. Other variations or modifications of the above teachings will be apparent to those of ordinary skill in the art. It is not necessary here nor is it exhaustive of all embodiments. Any modification, equivalent replacement, improvement, etc. which come within the spirit and principles of the invention are desired to be protected by the following claims.
Claims (7)
1. The utility model provides a track tracking monitoring early warning system of carrier loader which characterized in that includes:
the video acquisition equipment is used for acquiring real-time video of the running of the carrier vehicle;
the pretreatment module is used for receiving real-time continuous video transmitted by the video acquisition equipment and used for carrying out sliding window pretreatment on the real-time continuous video to generate a pre-detection frame, scanning images in the pre-detection frame through the sliding window, extracting HOG characteristics and carrying out preliminary positioning on the carrier;
the preprocessing module carries out sliding window preprocessing on the real-time continuous video, and the process of generating the pre-detection frame comprises the following steps:
s21, setting the maximum speed of the carrier vehicle as V max The frame rate of the video acquisition equipment is F, and the distance maximum value V of the carrier vehicle running per second on two continuous video images of the real-time continuous video is obtained max /F；
S22, setting the resolution ratio of a video image as m multiplied by n, wherein the area shot by video acquisition equipment is a rectangle with the length of a and the width of b, and the distance between a carrier vehicle in the next frame of image and the carrier vehicle in the previous frame of image is x pixels, so that the following conditions are satisfied:
wherein m represents the number of horizontal pixels of the video image, and n represents the number of vertical pixels of the video image;
s23, in the next frame of image, taking a carrier vehicle as a central target, expanding x pixels from the central target to the periphery, and generating a pre-detection frame;
the target detection module comprises a feature extraction unit and a carrier vehicle SVM classifier, wherein the feature extraction unit extracts HOG features of each frame of image, and the real-time position of the carrier vehicle is determined by classifying by the carrier vehicle SVM classifier;
the result of the target detection module confirming the real-time position of the target carrier vehicle comprises the following steps: the target carrier is lost and the target carrier exists and the position can be tracked;
if the target carrier vehicle exists and the position can be tracked, the comprehensive processing module compares the tracked track coordinates of the target carrier vehicle with the reference track coordinates, and the process for judging whether the running track of the target carrier vehicle is abnormal comprises the following steps:
s51, setting the area of a reference track target detection frame taking the carrier as the center as S when the carrier runs correctly A The area of the actual track target detection frame taking the tracked target carrier vehicle as the center is S B Determining the intersection ratio IOU of the reference track target detection frame and the actual track target detection frame:
s52, the comprehensive processing module judges whether the IOU and the fault tolerance allowance lambda meet the following conditions: if the IOU is more than or equal to lambda, the target carrier vehicle does not deviate from the track at present, and the overspeed, deceleration and accidental stopping judging process is executed; otherwise, the target carrier vehicle is already off track at present;
the target tracking module is used for tracking the carrier vehicle by adopting a KCF algorithm;
the comprehensive processing module is used for judging whether the running of the carrier vehicle is abnormal and sending abnormal information to the server;
the server receives and stores the abnormal information sent by the comprehensive processing module, issues an early warning instruction to the early warning module and generates emergency response measures;
and the early warning module is used for receiving an early warning instruction issued by the server and sending early warning information to staff.
2. The vehicle trajectory tracking monitoring and early warning system according to claim 1, wherein a worker determines a vehicle dataset to be tracked and monitored before classifying by the vehicle SVM classifier, the object detection module performs HOG feature extraction by using the feature extraction unit, and the extracted HOG feature is used as an input of the vehicle SVM classifier to train the vehicle SVM classifier.
3. The vehicle track following monitoring and early warning system according to claim 1, wherein the anomaly information sent by the integrated processing module comprises vehicle running track deviation, overspeed, deceleration and unexpected stop information.
4. The track following monitoring and early warning system of claim 3, wherein the early warning module comprises:
the first-level early warning unit is used for sending first-level early warning information to staff when the carrier vehicle runs off the track;
the secondary early warning unit sends secondary early warning information to staff when the carrier vehicle is not started on time or stopped in the running process;
and the three-level early warning unit is used for sending three-level early warning information to staff when the running speed of the carrier vehicle is lower than the standard speed or higher than the standard speed.
5. The track tracking monitoring and early warning method for the carrier vehicle is characterized by being realized based on the track tracking monitoring and early warning system of the carrier vehicle according to claim 1 and at least comprising the following steps:
s1, taking one or more times of correct running of a carrier vehicle as a benchmark to obtain a reference track coordinate of the running of the carrier vehicle;
s2, the video acquisition equipment acquires real-time continuous videos of the carrier vehicle running and transmits the real-time continuous videos to the preprocessing module, the preprocessing module carries out sliding window preprocessing on the real-time continuous videos to generate a pre-inspection frame, and images in the pre-inspection frame are scanned through the sliding window;
s3, extracting HOG features of each frame of image in the real-time video of the vehicle running through a feature extraction unit of the target detection module, wherein the extracted HOG features are used for classifying the SVM classifier of the vehicle, and the target detection module confirms the real-time position of the target vehicle;
s4, tracking the track of the target carrier vehicle by the target tracking module through a KCF algorithm to obtain track coordinates of the target carrier vehicle;
s5, comparing the tracked track coordinates of the target carrier vehicle with the reference track coordinates by the comprehensive processing module, judging whether the track of the target carrier vehicle is abnormal, if so, sending abnormal information to a server, sending an early warning instruction to the early warning module by the server, executing the step S6, and generating emergency response measures; otherwise, the comprehensive processing module confirms that the running track of the target carrier vehicle is normal;
s6, the early warning module receives an early warning instruction issued by the server and sends early warning information to staff.
6. The method for tracking, monitoring and pre-warning of a vehicle according to claim 5, wherein the process of obtaining the reference track coordinates of the vehicle in step S1 includes:
s11, a worker makes a carrier vehicle data set to be tracked and monitored, HOG feature extraction is carried out by using a feature extraction unit, the extracted HOG feature is used as input of a carrier vehicle SVM classifier, and the carrier vehicle SVM classifier is trained;
s12, acquiring a video of correct running of the carrier vehicle by using video acquisition equipment;
s13, carrying out sliding window pretreatment on real-time continuous videos by a pretreatment module on videos of correct running of the carrier vehicle, generating a pre-inspection frame, scanning images in the pre-inspection frame by the sliding window, and carrying out primary positioning on the carrier vehicle;
s14, extracting HOG characteristics of each frame of image in the real-time video of the vehicle running through a characteristic extraction unit of the target detection module, and classifying by using a trained vehicle SVM classifier to obtain the reference track coordinates of the vehicle running.
7. The method for tracking, monitoring and pre-warning of a carrier vehicle according to claim 6, wherein the determining process of overspeed, deceleration and unexpected stopping in step S52 includes:
s521, setting a real-time continuous video of the tracked target carrier vehicle running and a real-time continuous video of the carrier vehicle correctly running, which are acquired by video acquisition equipment, as T frames, determining a fixed-length container T, storing T frames of the real-time continuous video of the carrier vehicle correctly running in real time, namely, a reference track point with the maximum IOU, shifting each frame leftwards, discarding the leftmost point, and storing the reference track point with the maximum IOU at the tail end of the fixed-length container;
s522, setting the area of a reference track target detection frame in the ith frame as S Ai The area of the track target detection frame of the i-th frame tracked target carrier vehicle is S Bi Calculating the cross ratio;
s523, the comprehensive processing module judges whether the intersection ratio is smaller than the fault tolerance allowance lambda, if yes, the tracked target carrier vehicle runs abnormally, and the area S of a track target detection frame of the tracked target carrier vehicle is calculated Bi Area S of target detection frame with other reference track A Finding out the j frame with the largest cross-over ratio in the real-time continuous video track of the correct running of the carrier vehicle, updating the fixed-length container T, and executing step S524; otherwise, updating the fixed-length container T, and enabling the tracked target carrier vehicle to run normally;
s524, judging whether each frame in the long container T is j or not by the comprehensive processing module, if so, stopping the tracked target carrier vehicle accidentally; otherwise, the tracked target carrier vehicle is not stopped, and step S525 is executed;
s525, judging the area S of a track target detection frame of the tracked target carrier vehicle of the current ith frame by the comprehensive processing module Bi Area S of target detection frame with j-th frame reference track Aj If the intersection ratio of the target carrier vehicle is smaller than the fault tolerance margin lambda, if so, the tracked target carrier vehicle deviates from the track; otherwise, step S526 is performed;
s526, the comprehensive processing module judges whether j is greater than i, if yes, the tracked target carrier vehicle overspeed; otherwise, the tracked target carrier vehicle slows down.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110105837.2A CN112686215B (en) | 2021-01-26 | 2021-01-26 | Track tracking monitoring and early warning system and method for carrier vehicle |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110105837.2A CN112686215B (en) | 2021-01-26 | 2021-01-26 | Track tracking monitoring and early warning system and method for carrier vehicle |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN112686215A CN112686215A (en) | 2021-04-20 |
| CN112686215B true CN112686215B (en) | 2023-07-25 |
Family
ID=75459346
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110105837.2A Active CN112686215B (en) | 2021-01-26 | 2021-01-26 | Track tracking monitoring and early warning system and method for carrier vehicle |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112686215B (en) |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109146923A (en) * | 2018-07-13 | 2019-01-04 | 高新兴科技集团股份有限公司 | The processing method and system of disconnected frame are lost in a kind of target following |
| CN109948582A (en) * | 2019-03-28 | 2019-06-28 | 湖南大学 | A kind of retrograde intelligent detecting method of vehicle based on pursuit path analysis |
| CN111784744A (en) * | 2020-07-06 | 2020-10-16 | 天津理工大学 | Automatic target detection and tracking method based on video monitoring |
-
2021
- 2021-01-26 CN CN202110105837.2A patent/CN112686215B/en active Active
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109146923A (en) * | 2018-07-13 | 2019-01-04 | 高新兴科技集团股份有限公司 | The processing method and system of disconnected frame are lost in a kind of target following |
| CN109948582A (en) * | 2019-03-28 | 2019-06-28 | 湖南大学 | A kind of retrograde intelligent detecting method of vehicle based on pursuit path analysis |
| CN111784744A (en) * | 2020-07-06 | 2020-10-16 | 天津理工大学 | Automatic target detection and tracking method based on video monitoring |
Also Published As
| Publication number | Publication date |
|---|---|
| CN112686215A (en) | 2021-04-20 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110390691B (en) | Ore dimension measuring method based on deep learning and application system | |
| CN111310645B (en) | Method, device, equipment and storage medium for warning overflow bin of goods accumulation | |
| CN105424330B (en) | A kind of lorry bolts at axle end fault detection method and device | |
| CN111260629A (en) | Pantograph structure abnormity detection algorithm based on image processing | |
| CN108038424B (en) | Visual automatic detection method suitable for high-altitude operation | |
| CN105160340A (en) | Vehicle brand identification system and method | |
| CN109165541A (en) | Coding method for vehicle component in intelligent recognition rail traffic vehicles image | |
| CN112001878A (en) | Deep learning ore scale measuring method based on binarization neural network and application system | |
| CN112184773A (en) | Helmet wearing detection method and system based on deep learning | |
| CN111161292A (en) | Ore size measurement method and application system | |
| CN110135383A (en) | Loading goods train video intelligent monitoring system | |
| CN110245592A (en) | A method of for promoting pedestrian's weight discrimination of monitoring scene | |
| Pham et al. | A YOLO-based real-time packaging defect detection system | |
| CN114740086A (en) | Monorail crane track defect detection method | |
| CN115512134A (en) | Express item stacking abnormity early warning method, device, equipment and storage medium | |
| CN117094609B (en) | Intelligent management system for aluminum profile production quality based on machine vision | |
| Li et al. | Integrating deformable convolution and pyramid network in cascade R-CNN for fabric defect detection | |
| CN112686215B (en) | Track tracking monitoring and early warning system and method for carrier vehicle | |
| Lu et al. | Automatic visual inspection of a missing split pin in the China railway high-speed | |
| Tang et al. | Visual abnormality detection framework for train-mounted pantograph headline surveillance | |
| CN114120175A (en) | Method for identifying foreign matters on coal conveying belt based on computer vision | |
| Lv et al. | Railway train inspection robot based on intelligent recognition technology | |
| CN110516551B (en) | Vision-based line patrol position deviation identification system and method and unmanned aerial vehicle | |
| Liu et al. | Detection of railway tunnel lining based on adaptive background learning | |
| CN117557570B (en) | Rail vehicle abnormality detection method and system |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |