CN111144247A - Escalator passenger reverse-running detection method based on deep learning - Google Patents
Escalator passenger reverse-running detection method based on deep learning Download PDFInfo
- Publication number
- CN111144247A CN111144247A CN201911292323.1A CN201911292323A CN111144247A CN 111144247 A CN111144247 A CN 111144247A CN 201911292323 A CN201911292323 A CN 201911292323A CN 111144247 A CN111144247 A CN 111144247A
- Authority
- CN
- China
- Prior art keywords
- passenger
- detection
- retrograde
- target
- escalator
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V20/00—Scenes; Scene-specific elements
- G06V20/50—Context or environment of the image
- G06V20/52—Surveillance or monitoring of activities, e.g. for recognising suspicious objects
- G06V20/53—Recognition of crowd images, e.g. recognition of crowd congestion
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/20—Image preprocessing
- G06V10/25—Determination of region of interest [ROI] or a volume of interest [VOI]
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
- G06V40/10—Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02B—CLIMATE CHANGE MITIGATION TECHNOLOGIES RELATED TO BUILDINGS, e.g. HOUSING, HOUSE APPLIANCES OR RELATED END-USER APPLICATIONS
- Y02B50/00—Energy efficient technologies in elevators, escalators and moving walkways, e.g. energy saving or recuperation technologies
Landscapes
- Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Multimedia (AREA)
- Theoretical Computer Science (AREA)
- Human Computer Interaction (AREA)
- Image Analysis (AREA)
Abstract
The invention discloses a method for detecting the passenger reverse running of an escalator based on deep learning, which comprises the following steps of firstly, obtaining image frames from a monitoring video stream of the escalator, and setting a detection area (ROI); secondly, detecting the head position of the passenger in the ROI specified in the step one by using a target detection algorithm; step three, judging the direction of the head detected in the step two by using a classifier, and judging whether the passenger possibly has a retrograde motion phenomenon; step four, tracking each possible retrograde target (head) in the step three by using a multi-target tracking algorithm to obtain a tracking track; and step five, analyzing each track in the step four, and judging whether the passenger has a retrograde motion behavior. The algorithm can be used for effectively judging the passenger reverse running behavior on the escalator with higher accuracy and real-time performance under the complex condition, and avoiding the occurrence of accidents.
Description
Technical Field
The invention belongs to the technical field of computer vision, and particularly relates to an escalator passenger converse detection method based on deep learning.
Background
With the rise of domestic smart phones, almost all people develop the habit of walking and seeing mobile phones, so that when people take the escalator, people often miss the elevator entrance without paying attention to the elevator and are prone to running wrong, the people do not stop the escalator in time, and serious elevator taking accidents such as falling down and the like often happen after the people enter the escalator if the people do not stop the escalator in time. According to statistics, the rate of the elevator-taking safety accidents caused by the retrograde motion is the largest of all safety accidents, so that the detection of whether passengers enter the escalator in the retrograde motion or not and the reminding of the passengers in the retrograde motion are necessary measures for ensuring the safety of the passengers.
At present most elevators do not all set up equipment and the device that can detect the retrograde motion, and staff's maintenance order can be arranged in the place that a small amount of flows of people are big, and it is tired very much that still to examine whether the passenger takes place the retrograde motion when the staff maintenance order is obvious, and the automatic escalator of most public occasions all does not arrange the staff moreover, takes place the accident very easily. In some public occasions, an infrared detection instrument is additionally arranged on an elevator, and when a pedestrian is detected at an elevator opening, an alarm is given, so that the device has the defect that firstly, false detection is often caused because no behavior recognition is carried out.
The invention with publication number 106503632A discloses an escalator intelligent safety monitoring method based on video analysis, which comprises the following steps: firstly, acquiring a real-time video image sequence installed in a camera in a monitoring area; secondly, establishing a mixed Gaussian background model, and inhibiting the shadow in the video image to extract the foreground; then, the classifier is used for identifying the human body object of the extracted foreground, and the angular point light stream of the human body object is calculated, so that whether abnormal behaviors such as retrograde movement or falling down of passengers exist or not is judged, measures such as emergency stop of the escalator are taken in time, and the personal safety of people riding the escalator is guaranteed. According to the method, the abnormal behavior of the human body object is judged through the angular point light stream, so that the probability of trampling accidents caused by retrograde motion or falling of people can be effectively prevented, and the personal safety of passengers is greatly protected. However, the method has a single judgment mode and has low accuracy in judging complex retrograde conditions.
Therefore, by applying a computer vision technology and a monitoring camera of the elevator, the person who drives in the wrong direction is tracked and the behavior is analyzed by an artificial intelligence method, and whether the passenger drives in the wrong direction or not is judged accurately and rapidly, so that the method has great significance for reducing the accidents of the escalator.
Disclosure of Invention
The invention aims to effectively judge the passenger converse behavior on an escalator under the complex condition, has higher accuracy and real-time performance, and provides an escalator passenger converse detection method based on deep learning.
The method for detecting the passenger in the escalator in the reverse direction based on deep learning comprises the following steps:
step one, acquiring image frames from a monitoring video stream of an escalator, and setting a detection Region (ROI);
secondly, detecting the head position of the passenger in the detection area specified in the first step by using a target detection algorithm;
step three, judging the direction of the head detected in the step two by using a classifier, and judging whether the passenger possibly has a retrograde motion phenomenon;
step four, tracking each possible retrograde target (head) in the step three by using a multi-target tracking algorithm to obtain a tracking track;
step five, analyzing each track in the step four, and judging whether the passenger has a retrograde motion behavior;
in the first step, the video stream comes from a monitoring camera of the escalator, and the camera is arranged right opposite to the escalator opening and can clearly shoot images of passengers riding the escalator getting on and off the escalator. After the camera is installed, a rectangular area is set, the rectangular area is an exit of the escalator and serves as a detection area (ROI) of the invention, and when passengers enter from the exit, the phenomenon of reverse driving is considered to occur.
In the second step, the adopted target detection algorithm is a Yolov3 algorithm, a head detection model is trained by using a transfer learning algorithm, the head in a video frame is detected, and some misdetected targets and targets outside the ROI are filtered out through post-processing, wherein the head detection comprises the following steps:
(2.1) during training, firstly building a model structure of YOLOv3, wherein a YOLOv3 network structure comprises a dark net53 network for feature extraction, three full-convolution networks called yolo serve as output layers, each yolo layer outputs results on different feature maps, and in the yolo layer, the output dimensions are as follows:
S×S×(B×(5+C))
where S is the size of the feature map, B represents the number of frames output per trellis, 5 represents the 4 frame parameters and a confidence score of whether an object is contained, and C represents the number of categories.
In step two, there is only one category of head, so C is 1, the resolution of the picture used in this invention is 416x416, and (52x52) + (26x26) +13x13)) x3 is predicted to be 10647 bounding boxes.
The method is characterized in that a darknet53 model trained on an ILSVRC image data set is used as initial weight of a network backbone, and the rest parameters adopt a kaiming initialization method, so that transfer learning is performed equivalently, and the capability of extracting the network features from the convolution features is enhanced.
And training the network by using the public human head data set, wherein the loss function used in the training comprises a category loss function and a frame position regression loss function, the category loss function adopts binary cross entropy, and the frame position regression loss function adopts a square loss function.
(2.2) during detection, the pictures are input into a trained network after being normalized, and the output of the network can be obtained at a yolo layer, wherein the prediction result of the output of the network to a frame is converted as follows:
bx＝σ(tx)+cx
by＝σ(ty)+cy
wherein t isx，ty，tw，thBorder information output for the network, cx，cy，pw，phThe coordinate corresponding to the upper left corner of the feature map grid and the width and height of the anchor set, respectively.
Wherein the probability of a certain class predicted by each frame is given by the following formula:
P＝objectness*Pi
objectness indicates the confidence that the current bbox contains objects;
Pirepresenting the conditional probability of predicting as the ith category in the current bbox;
(2.3) post-processing, predicting 10647 bounding boxes by YOLOv3 for the image with the resolution size of 416x416, and processing the output bounding boxes by adopting a non-maximum value suppression NMS algorithm in S2 so as to screen out effective boxes with confidence degrees larger than a certain threshold value from a plurality of output mutually overlapped bounding boxes. In addition, according to the particularity of the scene, the length-width ratio and the prediction of the size in a possible range are screened out through an NMS algorithm, and the error of the network is reduced through priori knowledge. And finally outputting the screened frames in the R0I area set in the step one.
In the third step, the used classifier is a CNN classifier and is used for judging whether the face of a person faces towards the elevator, and whether a retrograde motion phenomenon exists can be judged more accurately through the face of the person. The method specifically comprises the following steps:
and (3.1) training a classifier, training the network by adopting transfer learning, initializing the convolution layer of the network by adopting the weight trained on the ILSVRC image data set, initializing the parameters of the full connection layer of the network by adopting random initialization, and finally training to obtain an effective model.
And (3.2) detecting, namely intercepting the head region output in the step two, normalizing the picture, outputting the picture to a network to obtain a classification result, and defining the head in the direction towards an outlet Region (ROI) of the elevator as a passenger possibly having retrograde motion.
In the fourth step, the multi-target tracking algorithm adopts a deppsort algorithm, and a tracker is generated for the head of the person who may move in the wrong direction in the third step to obtain a tracking track. The method comprises the following steps:
(4.1) estimating the prediction result by using a Kalman filter, and describing the motion state of the human head by using eight state parameters of (x, y, r, h, x ^ y, y ^ r, h ^) wherein x and y are the central coordinates of the human head detection frame, r represents the length-width ratio of the human head frame, and h represents the height of the human head frame, wherein x ^ y, y ^ r, h ^ represents the variation of x, y, r, h, namely speed information. In which the pedestrian walking is considered to be at a constant speed, the kalman filter of the linear observation model is used here to estimate the motion state of the passenger.
And (4.2) target association, wherein the deppsort algorithm simultaneously considers motion information association and appearance information association.
And (3) correlating the motion information by adopting the Mahalanobis distance between the detected target position and the position predicted by the Kalman filter, wherein the Mahalanobis distance calculation method comprises the following steps:
d(1)(i，j)＝(dj-yi)TSi -1(dj，yi)
wherein: djHead position, y, of a passenger detected in step three, who may be in a wrong directioniRepresenting the prediction result of the i-th Kalman filter in the previous frame, SiRepresenting a covariance matrix between the detected position and the average tracked position.
And (3) correlating appearance information, namely inputting the picture di of the head part of each passenger which is detected in the third step and is possible to run in the wrong direction into a convolutional neural network to obtain a feature vector ri, wherein | | | ri | | | 1. If di association is successful, it is put into the feature set with successful association, and the feature set retains the feature vector of the nearest k frames with successful association.
The specific process is as follows:
calculating the distance between the picture of each head in each Kalman filter and each feature in the feature set which is successfully associated recently, wherein the calculation formula is as follows:
wherein r isjInputting the picture di representing the header part into a convolutional neural network to obtain a feature vector rj,
if di association is successful, it is put into the feature set with successful association, and the feature set retains the feature vector of the nearest k frames with successful association.
If the calculation is less than a specified threshold, the association is deemed to be successful.
Finally, taking the result of linear weighting of the above two association modes as the metric of the final association, the calculation formula is as follows:
ci，j＝λd(1)(i，j)+(1-λ)d(2)(i，j)
wherein λ represents a linear weighting coefficient for two distances;
in addition, a cascade matching method is adopted to give a large priority to a target appearing recently, so that errors caused by position updating uncertainty due to long-time shielding are solved.
(4.3) starting and ending of the tracking track, when 3 continuous frames of a tracking path are not matched with the detection frame, the tracking is considered to be ended, when the detection frame is not matched with the target in any tracker, a new target is considered to possibly appear, if the Kalman prediction result of the new target can be matched with the detection result in 3 continuous frames, the tracker of the target is added into the tracking list, otherwise, the target is considered to be detected by a detector in error, and the target is required to be deleted.
In the fifth step, each track in the fourth step is analyzed to judge whether the passenger has the retrograde motion behavior, and the process is as follows:
(5.1) initializing a record Δ Y of 0 for each track, representing the total displacement in the Y axis (parallel to the direction of installation of the stairs);
(5.2) recording the coordinate (x) of the tracked head of the ith frame imagei，yi) And (x) of the previous pointi-1，yi-1) Pixel difference (Δ x)i，Δyi)。
(5.3) when Δ Y is equal to Δ Y + Δ YiDetermining Δ Y and THΔY(threshold for reverse) if DeltaY is greater than THΔYIt indicates that the retrograde motion phenomenon occurs and sends a retrograde motion signal.
Compared with the prior art, the invention has the following advantages and beneficial effects:
the method and the device have the advantages that the video frame is directly obtained from the monitoring camera, and then whether the passenger in the monitoring video has the behavior of going backwards or not is analyzed. Secondly, the method is based on the tracking of passengers in the video, and then the behaviors of the passengers are understood and analyzed, so that the method has the advantages of high accuracy and stable effect.
Compared with the method for detecting the whole pedestrian, the method for directly detecting the head of the pedestrian has the advantages that:
1) the number of pedestrians on the elevator is large, so that the problem of shielding is caused when the pedestrians are detected, and the problem of shielding of the pedestrians is not serious when the monitoring camera is in a overlooking visual angle;
2) the detection of the human head is more beneficial to the next judgment of the walking direction of the passenger, because the human face is obviously distinguished from the characteristics of the back of the human head, but the whole front and back characteristics of the human are not obviously distinguished, especially under the condition of shielding;
3) the head of a person shelters from fewly, consequently for whole pedestrian, the tracking effect is better, and the orbit more can reflect passenger's true orbit. In addition, the invention can store the video segment of the passenger who has the phenomenon of reverse running and can generate the report, thereby being beneficial to the design of the escalator in public places and the arrangement of personnel.
Drawings
Fig. 1 is a flow chart of an escalator passenger reverse detection method based on deep learning according to the invention.
Fig. 2 is a schematic photograph of an acquired image frame and a set ROI region.
Fig. 3 is a photograph showing the detection of the occurrence of the retrograde motion phenomenon.
Detailed Description
For purposes of making the objects, aspects and advantages of the present invention more apparent, the present invention will be described in further detail with reference to the accompanying drawings and examples, it being understood that the specific examples described herein are for purposes of illustration only and are not intended to be all embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
As shown in fig. 1, the method for detecting passenger reverse movement of an escalator based on deep learning provided by the invention comprises the following steps:
step one, acquiring image frames from a monitoring video stream of an escalator, and setting a detection area (ROI):
the video stream comes from a monitoring camera of the escalator, the monitoring camera can be a Haikang or Dahua network camera, the frame rate is 25fps, and the resolution is 1920 x 1080. The camera is arranged right opposite to the escalator entrance, and can clearly shoot images of passengers riding the escalator for getting on and off the escalator. After the camera is installed, a rectangular area is set to serve as a detection area (ROI), the rectangular area is an exit of the escalator, and when passengers enter the escalator from the exit, the phenomenon of reverse driving is considered to occur.
Because the processing speed is lower than the reading speed of the video stream, the currently processed video frame is not the latest read video frame at this time, a multithreading method is adopted to reduce the time delay, one thread is used to read the video frame in the video stream, the other thread processes the image, and when the processing speed is lower than the reading speed, the thread reading the video frame automatically updates the buffer to ensure that the processing thread processes the latest video frame, thereby reducing the time delay.
Step two, detecting the head position of the passenger in the ROI area specified in the step one by using an object detection algorithm:
the adopted target detection algorithm is a YOLOv3 algorithm, a human head detection model is trained by using a transfer learning algorithm, the human head in a video frame is detected, and some false detection targets and targets outside the ROI are filtered out through post-processing. The invention adopts direct detection of human head, and comprises the following steps:
during training, a model structure of YOLOv3 is built, the resolution of the picture used in the invention is 416x416 images, and ((52x52) + (26x26) +13x13)) x3 is predicted to be 10647 bounding boxes. The method is characterized in that a darknet53 model trained on an ILSVRC image data set is used as initial weight of a network backbone, and the rest parameters adopt a kaiming initialization method, so that transfer learning is performed equivalently, and the capability of extracting the network features from the convolution features is enhanced.
Training the network using the public head data set, the braiwash data set, the loss function used in the training is as follows:
the loss function comprises a category loss function and a border position regression loss function, the category loss function adopts binary cross entropy, and the border position regression loss function adopts a square loss function.
During prediction, after the pictures are normalized, the pictures are input into a trained network, and the output of the network can be obtained at a yolo layer, wherein the prediction result from the output of the network to a frame is converted as follows:
bx＝σ(tx)+cx
by＝σ(ty)+cy
wherein t isx，ty，tw，thBorder information output for the network, cx，cy，pw，phCoordinates corresponding to the upper left corner of the feature map lattice, respectivelyThe width and height of the anchor are set.
Wherein the probability of a certain class predicted by each frame is given by the following formula:
P＝objectness*Pi
objectness indicates the confidence that the current bbox contains objects;
Pirepresenting the conditional probability of predicting the ith category in the current bbox.
In the post-processing, for the image with the size of 416x416, the YOLO predicts ((52x52) + (26x26) +13x13)) x3 as 10647 bounding boxes, and in the second step, the non-maximum suppression NMS algorithm is used for processing the output bounding boxes, so that effective boxes with the confidence coefficient greater than 0.6 are screened out from the output numerous overlapped bounding boxes. In addition, according to the particularity of the scene, the length-width ratio and the prediction of the size in a possible range are screened out through an NMS algorithm, and the error of the network is reduced through priori knowledge. And finally, outputting the screened frame positioned in the ROI area set in the step one.
Step three, using a classifier to judge the direction of the human head detected in the step two:
a CNN classifier is added for judging whether the face faces towards the elevator or not, and the reverse phenomenon can be more accurately judged through the face direction. The method comprises the following steps:
training the network, wherein the used classification network is resnet50, the training input sample size is 34 × 34, the network is trained by adopting transfer learning, the convolution layer of the network is initialized by adopting the weight trained on the ILSVRC image data set, the parameters of the full connection layer of the network are initialized randomly, and finally an effective model is obtained by training.
And (3) detecting, namely intercepting the head region output in the step two, deforming the size to 34 x 34, normalizing the picture, outputting the picture to a network to obtain a classification result, and defining the head facing the direction of an elevator outlet Region (ROI) as a passenger possibly having reverse running.
Step four, tracking each possible retrograde target (head) in the step three by using a multi-target tracking algorithm to obtain a tracking track:
the multi-target tracking algorithm adopts a deppsort algorithm, and a tracker is generated for the head of the person who may move in the wrong direction in the third step to obtain a tracking track. The method comprises the following steps:
and estimating the prediction result by using a Kalman filter, and describing the motion state of the human head by adopting eight state parameters of (x, y, r, h, x ^ y, y ^ r, r ^ h), wherein x and y are the central coordinates of the human head detection frame, r represents the aspect ratio of the human head frame, and h represents the height of the human head frame, wherein x ^ y, y ^, r ^ h represents the variation of x, y, r and h, namely speed information. In which the pedestrian walking is considered to be at a constant speed, the kalman filter of the linear observation model is used here to estimate the motion state of the passenger.
And (3) target association, wherein the deppsort algorithm simultaneously considers motion information association and appearance information association:
in the motion information association, the Mahalanobis distance between the detected target position and the position predicted by the Kalman filter is adopted for association, and the Mahalanobis distance calculation method comprises the following steps:
d(1)(i，j)＝(dj-yi)TSi -1(dj-yi)
where dj denotes the head position of the passenger detected in step three, possibly in the wrong direction, yi denotes the prediction result of the i-th kalman filter in the previous frame, and Si denotes the covariance matrix between the detected position and the average tracking position.
And (3) appearance information association, namely inputting the picture dj of the head part of each passenger which is detected in the third step and is possible to run in the opposite direction into a convolutional neural network to obtain a feature vector ri, wherein | ri | | | | is 1, if dj association is successful, the dj association is put into a feature set which is successfully associated, and the feature set retains the feature vector of the nearest k frames which are successfully associated. Calculating the distance between the picture of each head in each Kalman filter and each feature in the feature set which is successfully associated recently, wherein the calculation formula is as follows:
if the calculation is less than a specified threshold, the association is deemed to be successful. Finally, taking the result of linear weighting of the above two association modes as the metric of the final association, the calculation formula is as follows:
ci，j＝λd(1)(i，j)+(1-λ)d(2)(i，j)
in addition, a cascade matching method is adopted to give a large priority to a target appearing recently, so that errors caused by position updating uncertainty due to long-time shielding are solved.
And (3) starting and ending tracking tracks, when 3 continuous frames of a tracking path are not matched with a detection frame, considering that the tracking is ended, when the detection frame is not matched with a target in any tracker, considering that a new target possibly appears, if the Kalman prediction result of the new target can be matched with the detection result in 3 continuous frames, adding the tracker of the target into a tracking list, otherwise, considering that the detector performs false detection, and deleting the target.
Step five, analyzing each track in the step four, and judging whether the passenger has a retrograde motion:
analyzing each track to judge whether the passenger has the behavior of going backwards or not, wherein the process is as follows:
initializing a record Δ Y ═ 0 for each track, representing the total displacement in the Y axis (parallel to the direction of escalator installation);
recording the coordinate (x) of the tracked human head of the ith frame imagei，yi) And (x) of the previous pointi-1，yi-1) Pixel difference (Δ x)i，Δyi)。
Let Δ Y be Δ Y + Δ YiDetermining Δ Y and THΔY(threshold for reverse) if DeltaY is greater than THΔYIt indicates that the retrograde motion phenomenon occurs and sends a retrograde motion signal.
Claims (10)
1. The escalator passenger converse running detection method based on deep learning is characterized by comprising the following steps: the method comprises the following steps:
step one, acquiring image frames from a monitoring video stream of an escalator, and setting a detection area;
secondly, detecting the head position of the passenger in the detection area specified in the first step by using a target detection algorithm;
step three, judging the direction of the head detected in the step two by using a classifier, and judging whether the passenger possibly has a retrograde motion phenomenon;
step four, tracking each head target which can be retrograde in step three by using a multi-target tracking algorithm to obtain a tracking track;
and step five, analyzing each track in the step four, and judging whether the passenger has a retrograde motion behavior.
2. The deep learning based escalator passenger retrograde detection algorithm of claim 1, wherein: in the first step, the monitoring video stream comes from a monitoring camera of the escalator, the camera is installed right opposite to the escalator entrance, a rectangular area is set as a detection area, and the rectangular area is an exit of the escalator.
3. The deep learning based escalator passenger retrograde detection algorithm of claim 1, wherein: in the second step, the adopted target detection algorithm uses a transfer learning algorithm to train a human head detection model, the human head in the video frame is detected, and some false detection targets and targets outside the detection area are filtered out through post-processing.
4. The deep learning based escalator passenger retrograde detection algorithm of claim 3, wherein: the second step specifically comprises the following steps:
(2.1) during training, firstly building a target detection model structure, adopting a transfer learning method, using a model trained on an ILSVRC image data set as an initial value weight of a network backbone, and adopting a random initialization method for the rest parameters; training a network by using a public human head data set, wherein the loss function comprises a category loss function and a frame position regression loss function;
(2.3) during detection, normalizing the pictures and inputting the normalized pictures into a trained network to obtain the output of the network;
and (2.3) post-processing, outputting a plurality of bounding boxes by the network, and processing the output bounding boxes by adopting a non-maximum value inhibition NMS algorithm, thereby screening effective boxes with confidence degrees larger than a certain threshold value from a plurality of output mutually overlapped bounding boxes.
5. The deep learning based escalator passenger retrograde detection algorithm of claim 1, wherein: in the third step, a classifier is added for judging whether the face is oriented, namely, whether the face is oriented towards the elevator, and whether the phenomenon of retrograde motion exists is judged according to the orientation of the face.
6. The deep learning based escalator passenger retrograde detection algorithm of claim 5, wherein: the third step specifically comprises:
(3.1) training a classifier, namely taking the image of the human head part detected by the target detector as training data, taking whether the human head faces the elevator direction as a label, training the network by adopting transfer learning, initializing the convolution layer of the network by adopting the weight trained on the ILSVRC image data set, initializing the parameters of the full connection layer of the network randomly, and finally training to obtain an effective model;
and (3.2) detecting, namely intercepting the head area output in the step two, normalizing the picture, outputting the picture to a network to obtain a classification result, and defining the head facing the direction of the elevator exit detection area as a passenger possibly having a retrograde motion.
7. The deep learning based escalator passenger retrograde detection algorithm of claim 1, wherein: in the fourth step, the multi-target tracking algorithm adopts a deppsort algorithm, and a tracker is generated for the head of the person who may move in the wrong direction in the third step to obtain a tracking track.
8. The deep learning based escalator passenger retrograde detection algorithm of claim 7, wherein: the fourth step comprises the following specific steps:
(4.1) estimating a prediction result by using a Kalman filter, and describing the motion state of the human head by using eight state parameters of x, y, r, h, x ^ y, y ^ r, r ^ h, wherein x and y are the central coordinates of a human head detection frame, r represents the length-width ratio of the human head frame, and h represents the height of the human head frame, wherein x ^ y, y ^ r, h ^ h represents the variation of x, y, r, h, namely speed information;
(4.2) performing target association, wherein the deppsort algorithm simultaneously considers motion information association and appearance information association;
(4.3) starting and ending of the tracking track, when a plurality of continuous frames of a tracking path are not matched with the detection frame, the tracking is considered to be ended, when the detection frame is not matched with the target in any tracker, a new target is considered to possibly appear, if the Kalman prediction results of the new target in a plurality of continuous frames can be matched with the detection results, the tracker of the target is added into the tracking list, otherwise, the target is considered to be detected by a detector in error, and the target is required to be deleted.
9. The deep learning based escalator passenger retrograde detection algorithm of claim 1, wherein: and step five, analyzing each track in the step four, and judging whether the passenger has a retrograde motion behavior.
10. The deep learning based escalator passenger retrograde detection algorithm of claim 9, wherein: the concrete process of the step five is as follows:
(5.1) initializing a record Δ Y of 0 for each trace, representing the total displacement on the Y-axis;
(5.2) recording the coordinate (x) of the tracked head of the ith frame imagei，yi) And (x) with the last pointi-1，yi-1) Pixel difference (Δ x)t，Δyt)；
(5.3) when Δ Y is ═ Δ Y + Δ γiDetermining Δ Y and THΔYIf DeltaY is greater than THΔYWherein TH isΔYA threshold value for whether or not retrograde motion is present; it indicates that the retrograde motion phenomenon occurs and sends a retrograde motion signal.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911292323.1A CN111144247B (en) | 2019-12-16 | 2019-12-16 | Escalator passenger reverse detection method based on deep learning |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911292323.1A CN111144247B (en) | 2019-12-16 | 2019-12-16 | Escalator passenger reverse detection method based on deep learning |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN111144247A true CN111144247A (en) | 2020-05-12 |
| CN111144247B CN111144247B (en) | 2023-10-13 |
Family
ID=70518425
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201911292323.1A Active CN111144247B (en) | 2019-12-16 | 2019-12-16 | Escalator passenger reverse detection method based on deep learning |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111144247B (en) |
Cited By (16)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111582243A (en) * | 2020-06-05 | 2020-08-25 | 上海商汤智能科技有限公司 | Countercurrent detection method, device, electronic equipment and storage medium |
| CN111860282A (en) * | 2020-07-15 | 2020-10-30 | 中国电子科技集团公司第三十八研究所 | Subway section passenger flow volume statistics and pedestrian retrograde motion detection method and system |
| CN111986231A (en) * | 2020-08-10 | 2020-11-24 | 深思考人工智能科技(上海)有限公司 | Multi-target tracking method and system |
| CN112613365A (en) * | 2020-12-11 | 2021-04-06 | 北京影谱科技股份有限公司 | Pedestrian detection and behavior analysis method and device and computing equipment |
| CN112668432A (en) * | 2020-12-22 | 2021-04-16 | 上海幻维数码创意科技股份有限公司 | Human body detection tracking method in ground interactive projection system based on YoloV5 and Deepsort |
| CN112785625A (en) * | 2021-01-20 | 2021-05-11 | 北京百度网讯科技有限公司 | Target tracking method and device, electronic equipment and storage medium |
| CN112801072A (en) * | 2021-04-14 | 2021-05-14 | 浙江大学 | Elevator non-flat-layer door opening fault recognition device and method based on computer vision |
| CN112800841A (en) * | 2020-12-28 | 2021-05-14 | 深圳市捷顺科技实业股份有限公司 | Pedestrian counting method, device and system and computer readable storage medium |
| CN112836667A (en) * | 2021-02-20 | 2021-05-25 | 上海吉盛网络技术有限公司 | Method for judging falling and retrograde of passenger on ascending escalator |
| CN112875481A (en) * | 2021-01-13 | 2021-06-01 | 深圳英飞拓科技股份有限公司 | Operation control method and device of escalator, terminal equipment and medium |
| CN113256690A (en) * | 2021-06-16 | 2021-08-13 | 中国人民解放军国防科技大学 | Pedestrian multi-target tracking method based on video monitoring |
| CN113361351A (en) * | 2021-05-27 | 2021-09-07 | 湖南信达通信息技术有限公司 | Image recognition-based retrograde determination method and system |
| CN113435402A (en) * | 2021-07-14 | 2021-09-24 | 深圳市比一比网络科技有限公司 | Method and system for detecting non-civilized behavior of train compartment |
| CN113723372A (en) * | 2021-11-01 | 2021-11-30 | 北京卓建智菡科技有限公司 | Prompting method and device, computer equipment and computer readable storage medium |
| CN114120210A (en) * | 2022-01-29 | 2022-03-01 | 通号通信信息集团有限公司 | Pedestrian detection method, electronic device, and computer-readable medium |
| CN117315550A (en) * | 2023-11-29 | 2023-12-29 | 南京市特种设备安全监督检验研究院 | Detection method for dangerous behavior of escalator passengers |
Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JP2014028667A (en) * | 2012-07-31 | 2014-02-13 | West Japan Railway Co | Escalator monitoring system |
| CN106127148A (en) * | 2016-06-21 | 2016-11-16 | 华南理工大学 | A kind of escalator passenger's unusual checking algorithm based on machine vision |
| CN106503632A (en) * | 2016-10-10 | 2017-03-15 | 南京理工大学 | A kind of escalator intelligent and safe monitoring method based on video analysis |
| CN107220992A (en) * | 2017-06-16 | 2017-09-29 | 华南理工大学 | With reference to machine vision and the escalator floor plates video frequency monitoring method of infrared array |
| WO2019025872A2 (en) * | 2018-11-26 | 2019-02-07 | Wasfi Alshdaifat | Autonomous city transportation means with artificial telepathy |
| CN109522793A (en) * | 2018-10-10 | 2019-03-26 | 华南理工大学 | More people's unusual checkings and recognition methods based on machine vision |
| CN110532852A (en) * | 2019-07-09 | 2019-12-03 | 长沙理工大学 | Subway station pedestrian's accident detection method based on deep learning |
-
2019
- 2019-12-16 CN CN201911292323.1A patent/CN111144247B/en active Active
Patent Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JP2014028667A (en) * | 2012-07-31 | 2014-02-13 | West Japan Railway Co | Escalator monitoring system |
| CN106127148A (en) * | 2016-06-21 | 2016-11-16 | 华南理工大学 | A kind of escalator passenger's unusual checking algorithm based on machine vision |
| CN106503632A (en) * | 2016-10-10 | 2017-03-15 | 南京理工大学 | A kind of escalator intelligent and safe monitoring method based on video analysis |
| CN107220992A (en) * | 2017-06-16 | 2017-09-29 | 华南理工大学 | With reference to machine vision and the escalator floor plates video frequency monitoring method of infrared array |
| CN109522793A (en) * | 2018-10-10 | 2019-03-26 | 华南理工大学 | More people's unusual checkings and recognition methods based on machine vision |
| WO2019025872A2 (en) * | 2018-11-26 | 2019-02-07 | Wasfi Alshdaifat | Autonomous city transportation means with artificial telepathy |
| CN110532852A (en) * | 2019-07-09 | 2019-12-03 | 长沙理工大学 | Subway station pedestrian's accident detection method based on deep learning |
Non-Patent Citations (1)
| Title |
|---|
| NICOLAI WOJKE: ""Simple Online and Realtime Tracking with a Deep Association Metric"", pages 1 - 5 * |
Cited By (23)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111582243A (en) * | 2020-06-05 | 2020-08-25 | 上海商汤智能科技有限公司 | Countercurrent detection method, device, electronic equipment and storage medium |
| CN111582243B (en) * | 2020-06-05 | 2024-03-26 | 上海商汤智能科技有限公司 | Countercurrent detection method, countercurrent detection device, electronic equipment and storage medium |
| CN111860282A (en) * | 2020-07-15 | 2020-10-30 | 中国电子科技集团公司第三十八研究所 | Subway section passenger flow volume statistics and pedestrian retrograde motion detection method and system |
| CN111986231A (en) * | 2020-08-10 | 2020-11-24 | 深思考人工智能科技(上海)有限公司 | Multi-target tracking method and system |
| CN112613365A (en) * | 2020-12-11 | 2021-04-06 | 北京影谱科技股份有限公司 | Pedestrian detection and behavior analysis method and device and computing equipment |
| CN112668432A (en) * | 2020-12-22 | 2021-04-16 | 上海幻维数码创意科技股份有限公司 | Human body detection tracking method in ground interactive projection system based on YoloV5 and Deepsort |
| CN112800841B (en) * | 2020-12-28 | 2024-05-17 | 深圳市捷顺科技实业股份有限公司 | Pedestrian counting method, device and system and computer readable storage medium |
| CN112800841A (en) * | 2020-12-28 | 2021-05-14 | 深圳市捷顺科技实业股份有限公司 | Pedestrian counting method, device and system and computer readable storage medium |
| CN112875481A (en) * | 2021-01-13 | 2021-06-01 | 深圳英飞拓科技股份有限公司 | Operation control method and device of escalator, terminal equipment and medium |
| CN112785625A (en) * | 2021-01-20 | 2021-05-11 | 北京百度网讯科技有限公司 | Target tracking method and device, electronic equipment and storage medium |
| CN112785625B (en) * | 2021-01-20 | 2023-09-22 | 北京百度网讯科技有限公司 | Target tracking method, device, electronic equipment and storage medium |
| CN112836667B (en) * | 2021-02-20 | 2022-11-15 | 上海吉盛网络技术有限公司 | Method for judging falling and reverse running of passengers going upstairs escalator |
| CN112836667A (en) * | 2021-02-20 | 2021-05-25 | 上海吉盛网络技术有限公司 | Method for judging falling and retrograde of passenger on ascending escalator |
| CN112801072A (en) * | 2021-04-14 | 2021-05-14 | 浙江大学 | Elevator non-flat-layer door opening fault recognition device and method based on computer vision |
| CN113361351A (en) * | 2021-05-27 | 2021-09-07 | 湖南信达通信息技术有限公司 | Image recognition-based retrograde determination method and system |
| CN113256690A (en) * | 2021-06-16 | 2021-08-13 | 中国人民解放军国防科技大学 | Pedestrian multi-target tracking method based on video monitoring |
| CN113256690B (en) * | 2021-06-16 | 2021-09-17 | 中国人民解放军国防科技大学 | Pedestrian multi-target tracking method based on video monitoring |
| CN113435402A (en) * | 2021-07-14 | 2021-09-24 | 深圳市比一比网络科技有限公司 | Method and system for detecting non-civilized behavior of train compartment |
| CN113723372B (en) * | 2021-11-01 | 2022-01-18 | 北京卓建智菡科技有限公司 | Prompting method and device, computer equipment and computer readable storage medium |
| CN113723372A (en) * | 2021-11-01 | 2021-11-30 | 北京卓建智菡科技有限公司 | Prompting method and device, computer equipment and computer readable storage medium |
| CN114120210A (en) * | 2022-01-29 | 2022-03-01 | 通号通信信息集团有限公司 | Pedestrian detection method, electronic device, and computer-readable medium |
| CN117315550A (en) * | 2023-11-29 | 2023-12-29 | 南京市特种设备安全监督检验研究院 | Detection method for dangerous behavior of escalator passengers |
| CN117315550B (en) * | 2023-11-29 | 2024-02-23 | 南京市特种设备安全监督检验研究院 | Detection method for dangerous behavior of escalator passengers |
Also Published As
| Publication number | Publication date |
|---|---|
| CN111144247B (en) | 2023-10-13 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111144247B (en) | Escalator passenger reverse detection method based on deep learning | |
| CN112257557B (en) | High-altitude parabolic detection and identification method and system based on machine vision | |
| CN110765964B (en) | Method for detecting abnormal behaviors in elevator car based on computer vision | |
| KR101375583B1 (en) | Object Density Estimation in Video | |
| Faro et al. | Adaptive background modeling integrated with luminosity sensors and occlusion processing for reliable vehicle detection | |
| US9569531B2 (en) | System and method for multi-agent event detection and recognition | |
| CN111260693B (en) | High-altitude parabolic detection method | |
| US20060245618A1 (en) | Motion detection in a video stream | |
| EP2093698A1 (en) | Crowd congestion analysis | |
| CN111401311A (en) | High-altitude parabolic recognition method based on image detection | |
| CN107977646B (en) | Partition delivery detection method | |
| US8599261B1 (en) | Vision-based car counting for multi-story carparks | |
| Sharma | Human detection and tracking using background subtraction in visual surveillance | |
| CN114926422B (en) | Method and system for detecting passenger flow of getting on and off vehicles | |
| Ahmad et al. | Robust background subtraction based person’s counting from overhead view | |
| CN110713082B (en) | Elevator control method, system, device and storage medium | |
| CN106056078A (en) | Crowd density estimation method based on multi-feature regression ensemble learning | |
| JP7125843B2 (en) | Fault detection system | |
| JP5110246B2 (en) | Fall detection device, program, fall detection method, and fall detection system | |
| Liu et al. | Metro passenger flow statistics based on yolov3 | |
| KR20030018487A (en) | Method and apparatus for counting the number of entering people at the gate using image | |
| Wickramasinghe et al. | Pedestrian Detection, Tracking, Counting, Waiting Time Calculation and Trajectory Detection for Pedestrian Crossings Traffic light systems | |
| CN113223081A (en) | High-altitude parabolic detection method and system based on background modeling and deep learning | |
| CN112580633A (en) | Public transport passenger flow statistical device and method | |
| Hradiš et al. | Real-time tracking of participants in meeting video |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |