CN115497056B - Method for detecting lost articles in region based on deep learning - Google Patents
Method for detecting lost articles in region based on deep learning Download PDFInfo
- Publication number
- CN115497056B CN115497056B CN202211462573.7A CN202211462573A CN115497056B CN 115497056 B CN115497056 B CN 115497056B CN 202211462573 A CN202211462573 A CN 202211462573A CN 115497056 B CN115497056 B CN 115497056B
- Authority
- CN
- China
- Prior art keywords
- person
- personnel
- foreign matter
- foreign
- frame
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000000034 method Methods 0.000 title claims abstract description 44
- 238000013135 deep learning Methods 0.000 title claims abstract description 27
- 238000001514 detection method Methods 0.000 claims abstract description 74
- 238000009826 distribution Methods 0.000 claims description 17
- 230000006399 behavior Effects 0.000 claims description 16
- 238000012549 training Methods 0.000 claims description 14
- 238000001914 filtration Methods 0.000 claims description 11
- 238000004364 calculation method Methods 0.000 claims description 10
- 238000000605 extraction Methods 0.000 claims description 6
- 230000003247 decreasing effect Effects 0.000 claims description 3
- 238000012545 processing Methods 0.000 claims description 3
- 230000003068 static effect Effects 0.000 claims description 3
- 238000012360 testing method Methods 0.000 claims description 3
- 230000002596 correlated effect Effects 0.000 claims 1
- 230000000875 corresponding effect Effects 0.000 claims 1
- 238000010586 diagram Methods 0.000 description 4
- 239000011159 matrix material Substances 0.000 description 4
- 238000004519 manufacturing process Methods 0.000 description 3
- 230000009471 action Effects 0.000 description 2
- 238000012790 confirmation Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000008569 process Effects 0.000 description 2
- 230000007306 turnover Effects 0.000 description 2
- 230000003044 adaptive effect Effects 0.000 description 1
- 238000004458 analytical method Methods 0.000 description 1
- 238000013473 artificial intelligence Methods 0.000 description 1
- 238000004891 communication Methods 0.000 description 1
- 238000005520 cutting process Methods 0.000 description 1
- 230000007547 defect Effects 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 239000000284 extract Substances 0.000 description 1
- 230000006870 function Effects 0.000 description 1
- 230000007246 mechanism Effects 0.000 description 1
- 239000013598 vector Substances 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V20/00—Scenes; Scene-specific elements
- G06V20/50—Context or environment of the image
- G06V20/52—Surveillance or monitoring of activities, e.g. for recognising suspicious objects
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
- G06V10/74—Image or video pattern matching; Proximity measures in feature spaces
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
- G06V10/82—Arrangements for image or video recognition or understanding using pattern recognition or machine learning using neural networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V20/00—Scenes; Scene-specific elements
- G06V20/60—Type of objects
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
- G06V40/10—Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V2201/00—Indexing scheme relating to image or video recognition or understanding
- G06V2201/07—Target detection
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Multimedia (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Evolutionary Computation (AREA)
- Computing Systems (AREA)
- Artificial Intelligence (AREA)
- Databases & Information Systems (AREA)
- Health & Medical Sciences (AREA)
- General Health & Medical Sciences (AREA)
- Medical Informatics (AREA)
- Software Systems (AREA)
- Human Computer Interaction (AREA)
- Image Analysis (AREA)
Abstract
The invention discloses a method for detecting lost articles in an area based on deep learning, which specifically comprises the following steps: s1: in the selected area, detecting personnel and foreign matters by adopting a deep learning target detection algorithm model, continuously judging whether personnel enter or not, and locking personnel ID (identity); meanwhile, detecting foreign matters, and if no foreign matters appear, continuously judging the detection of the foreign matters; s2: after the foreign matter is detected, locking the foreign matter ID, performing association matching on the foreign matter ID and the person ID in the step S1, and dynamically matching the foreign matter and the person with the nearest distance to obtain the person ID with the foreign matter association attribute; s3: and judging whether the person ID with the foreign matter association attribute leaves the selected area, if so, judging whether the foreign matter associated with the person ID leaves the selected area, if so, judging whether the foreign matter is the lost matter, and outputting an alarm result to finish the detection of the lost matter.
Description
Technical Field
The invention belongs to the technical field of artificial intelligence, and particularly relates to a method for detecting lost articles in an area based on deep learning, which is used for personnel tracking and target article detection.
Background
The prior art adopts a background difference method or an interframe difference method to judge whether an article belongs to a lost article. The method has poor detection effect on the lost objects, the detection is easily influenced by the environment, a large amount of noise is generated, the application scene is single when the detection of the lost objects is realized, the method depends on a fixed camera and an image background, and the method has great defects when used in a production scene.
In chinese patent document CN111553414A, a method for detecting a lost object in a vehicle based on improved fast R-CNN is disclosed, which uses a target detection algorithm in the field of deep learning computer vision, and as can be seen from the process, the method uses fast R-CNN as a target detection method, and directly judges the detected object as the lost object in the vehicle to give an alarm according to the result obtained by the target detection; the method mainly improves the network structure of FasterR-CNN, adds a ResNet101-FPN path enhancement method, extracts ROI (region of interest) for a multi-scale feature map, and finally predicts a bounding box and a category.
The method is characterized in that the detected object is directly judged as the lost object to give an alarm by the result obtained by the existing target detection algorithm, various false alarms and missed alarms are always generated, and the problem that the lost object detection false alarms are serious in various scenes due to the fact that the object stays in the region for a certain time and is obviously unreasonable is solved.
Disclosure of Invention
The invention aims to provide a method for detecting lost articles in an area based on deep learning, which combines the correlation between personnel behaviors and articles with time sequence characteristics to judge the dependency relationship of the lost articles.
In order to solve the technical problems, the invention adopts the technical scheme that: the method for detecting lost articles in the region based on deep learning specifically comprises the following steps:
s1: in the selected area, detecting personnel and foreign matters by adopting a deep learning target detection algorithm model, continuously judging whether personnel enter or not, and locking personnel ID (identity); meanwhile, detecting foreign matters, and if no foreign matters appear, continuously judging the detection of the foreign matters;
s2: after the foreign matter is detected, locking the foreign matter ID, performing association matching on the foreign matter ID and the person ID in the step S1, and dynamically matching the foreign matter and the person with the nearest distance to obtain the person ID with the foreign matter association attribute;
s3: and judging whether the person ID with the foreign matter association attribute leaves the selected area, if so, judging whether the foreign matter associated with the person ID leaves the selected area, if so, judging whether the foreign matter is the lost matter, and outputting an alarm result to finish the detection of the lost matter.
It is to be understood that the appearance of loss requires the following three features to be satisfied:
(1) Along with the appearance of people, if no people appear, the foreign matters are not forgotten or lost; foreign bodies and the like in the area into which wind blows are possible, which obviously does not accord with the definition of the lost object of the invention;
(2) People enter and leave, which is usually a precondition for judging lost articles, no people enter and leave, and even if foreign matters are generated in an area, the foreign matters can also be articles carried by the people and still can be taken away after a period of time; the generation of lost objects always occurs along with the appearance of foreign matters after people enter and leave an area;
(3) When a foreign matter appears, one more object is certainly arranged in a certain area after the foreign matter appears in the area; therefore, it is determined whether the object is a lost object, and at least one foreign object is present in the area.
In the technical scheme, if foreign matters appear in a selected area and along with the entering and leaving behaviors (time sequence characteristics) of personnel, the foreign matters are locked and the relevance analysis of the personnel and the articles is carried out, if a certain article belonging to a certain person is left, the early warning output is carried out on the appearance of the lost articles, namely, the personnel with the object subordination relation leave the selected area, and the lost article warning is generated; the method has the advantages that the dependency relationship locking between the articles is completed by means of the time-series personnel in-and-out judgment method and the distance characteristics between the personnel and the articles, so that the problem of serious false alarm of the lost article detection under various scenes is solved, personnel tracking and target article detection can be realized, and the result of generating the lost article output alarm is more accurate.
Preferably, in step S1, a deep learning YOLOV5 target detection algorithm model is used to detect people and foreign objects, and the method for establishing the YOLOV5 target detection algorithm model is as follows: s11 preparing a data set: collecting data, and segmenting the data to generate a data set; the data set comprises the type of the foreign matter needing to be detected;
s12, enhancement treatment: then, data enhancement is carried out by using a Mosaic data enhancement mode to obtain an enhanced data set, and the enhanced data set is divided into a training set and a test set;
s13, building a YOLOV5 target detection algorithm model: and inputting and training data of the training set, and obtaining the model weight of the Yolov5 target detection algorithm.
Preferably, in step S2, a deep learning Deepsort target tracking algorithm is adopted to track the person ID and the item ID, and the Deepsort target tracking algorithm specifically includes: predicting the track by adopting a Kalman filter; performing cascade matching and IOU matching on the predicted track and the detection result in the current frame by adopting a Hungarian algorithm; and determining the foreign matter ID, the person ID and the person ID with the foreign matter association attribute by using the Kalman filtering updating track determination tracking result. The method for tracking the Deepsort target increases Cascade Matching (Matching Cascade) and confirmation (confirmed) of a new track on the basis of the Sort algorithm, and improves the accuracy of target tracking.
Preferably, in the step S3, the determining whether the person ID having the foreign object association attribute leaves the selected area specifically includes: constructing a personnel time sequence feature list on personnel ID results tracked by a Deepsort target tracking algorithm to store personnel in-out features and time sequence features, marking the ID of personnel not leaving the entering area as 1 and marking the ID of personnel leaving the area as 0 in a certain time sequence; the in-out behavior and the time sequence relevance of the personnel ID are analyzed, the in-out state of the personnel ID can be judged when the in-out characteristic meets normal distribution under the sequence time sequence, and the in-out characteristic is specifically represented as 010 characteristic when the in-out characteristic of the personnel ID meets the normal distribution, and the characteristic represents that the personnel ID enters and leaves an area in the sequence time sequence; and when the personnel ID meeting the 010 characteristic is judged by utilizing the time sequence correlation, marking the corresponding personnel ID as a state of leaving after entering the area.
Preferably, in step S2, the distance between the center points is used to determine the dependency relationship between the foreign object ID and the person ID, the center point position of the person ID is calculated on the result of the person ID tracked by the Deepsort target tracking algorithm, the center point position of the foreign object ID is calculated on the result detected by the YOLOV5 target detection algorithm model, the dependency relationship between the foreign object ID and the person ID is calculated according to the euclidean distance between the two center point positions, and when the foreign object ID is closest to the person ID, the foreign object ID is determined to be the foreign object ID; and then carrying out association matching between the foreign matter ID and the person ID.
The personnel appear with personnel's business turn over action as the precondition that the lost article was judged, personnel's business turn over action and whether the lost article exists objective correlation, the detection accuracy of lost article is directly decided to the production of foreign matter, can accurately detect the production of foreign matter and algorithm accuracy become direct correlation, the subordinate logic that judges the foreign matter belongs to and is the auxiliary judgement, can be better when knowing that the foreign matter belongs to who the probability that the judgement is who produces the foreign matter, can improve the judgement accuracy.
Preferably, in step S13, the method for detecting people by using the built YOLOV5 target detection algorithm model includes: acquiring image data in the selected area, and acquiring each frame of image data through a video or an rtsp stream; and detecting whether a person appears in each frame of image data.
Preferably, the manner of generating the data set in step S11 is specifically: performing frame extraction processing on the video by adopting a video frame extraction mode to generate a data set; the data enhancement using the Mosaic data enhancement mode in the step S12 specifically includes: the data set is enhanced by rotating, cropping and increasing or decreasing the brightness of the image.
Preferably, in step S13, if the YOLOV5 target detection algorithm model in the current image detects that there is a person, the detection result is saved, that is, coordinates of four points of the rectangular frame are saved in a list, and coordinates of a center point of the rectangular frame are calculated, where the calculation formula is:
wherein xmin, xmax, ymin, and ymax represent coordinates of four points of the rectangular frame, and c _ x and c _ y represent coordinates of a center point of the rectangular frame; judging whether the personnel enter a specified area or not according to the coordinates of the central point, drawing the specified area by using opencv, and transmitting a parameter polygon coordinate point to obtain a specific area; and if the central point coordinate is in the selected area, judging that the detected person enters the selected area.
Preferably, in the step S2, a built YOLOV5 target detection algorithm model is used to detect people and articles appearing in each frame of image and give an ID; and then using the detection result of the YOLOV5 target detection algorithm model as a target frame of a pre-trained Deepsort target tracking algorithm model for input, using the obtained section of track as the current frame image track, performing IOU intersection and comparison matching through the target frame and the current frame image track, predicting the state of the next frame image target frame according to the track state by Kalman filtering, and updating the states of all tracks by using Kalman filtering observation values and estimation values, thereby completing the tracking of the foreign matter ID and the personnel ID.
Preferably, the dependency relationship between the foreign object ID and the person ID is determined, and the euclidean distance between the foreign object ID and the person ID is calculated using the coordinates of the center points of the person ID detection frame and the foreign object ID detection frame, and the calculation formula is:
wherein,
is a point
And point
When the foreign matter is static, the central coordinate of the foreign matter ID is calculated, the central coordinate of the current frame person ID is calculated, the central coordinate of the foreign matter ID and the central coordinate of the current frame person ID are used as two points, the person ID with the minimum distance is obtained through calculation of a formula of the Euclidean distance, the foreign matter is judged to belong to the person ID closest to the foreign matter, the person ID is marked as the person carrying the foreign matter and named as X _ ID, and the central coordinate of the person X _ ID with the foreign matter association attribute is obtained;
in the step S3, according to the obtained coordinates of the central point of the person X _ ID with the foreign matter associated attribute, the YOLOV5 target detection algorithm model in the step S1 is used to detect whether the person X _ ID with the foreign matter associated attribute is in the selected area, when the person X _ ID with the foreign matter associated attribute is not in the selected area, the time-sequence associated entry and exit feature is used to determine the entry and exit behaviors of the person X _ ID with the foreign matter associated attribute, and when the person X _ ID with the foreign matter associated attribute satisfies the normal distribution of 010 feature, the person X _ ID with the foreign matter associated attribute is determined to leave the selected area.
Drawings
FIG. 1 is a diagram of the attribute definition of lost objects in the deep learning-based method for detecting lost objects in an area according to the present invention;
FIG. 2 is a flowchart of a method for detecting lost objects in an area based on deep learning according to the present invention;
FIG. 3a is a diagram illustrating a time-series characteristic distribution of the person entering and exiting behavior according to an embodiment of the present invention;
FIG. 3b is a density plot of a time series signature distribution of the person ingress and egress behavior according to FIG. 3 a;
FIG. 4 is a display of personnel IDs initially detecting an access area;
fig. 5 is a graph showing the detection result of fig. 4.
Detailed Description
The following embodiments of the present invention are described in detail with reference to the accompanying drawings, and the following embodiments are only used to more clearly illustrate the technical solutions of the present invention, but not to limit the scope of the present invention.
The method for detecting the lost articles in the area based on the deep learning specifically comprises the following steps:
s1: in the selected area, detecting personnel and foreign matters by adopting a deep learning target detection algorithm model, continuously judging whether personnel enter or not, and locking personnel ID (identity); meanwhile, detecting foreign matters, and if no foreign matters appear, continuously judging the detection of the foreign matters;
s2: after the foreign matter is detected, locking the foreign matter ID, carrying out association matching on the foreign matter ID and the personnel ID in the step S1, and dynamically matching the foreign matter and the personnel with the nearest distance to obtain the personnel ID with the foreign matter association attribute;
s3: and judging whether the person ID with the foreign matter association attribute leaves the selected area, if so, judging whether the foreign matter associated with the person ID leaves the selected area, if so, judging whether the foreign matter is the lost matter, and outputting an alarm result to finish the detection of the lost matter.
It is to be understood that the appearance of loss requires the following three features to be satisfied:
(1) Along with the appearance of people, if no people appear, the foreign matters are not forgotten or lost; foreign bodies and the like in the area into which wind blows are possible, which obviously does not accord with the definition of the lost object of the invention;
(2) People enter and leave, which is usually a precondition for judging lost articles, no people enter and leave, and even if foreign matters are generated in an area, the foreign matters can also be articles carried by the people and still can be taken away after a period of time; the generation of lost objects always occurs along with the appearance of foreign matters after people enter and leave an area;
(3) When a foreign matter appears, one more object is certainly arranged in a certain area after the foreign matter appears in the area; therefore, it is determined whether the foreign object is a lost object, and at least one foreign object is present in the region.
And the fourth characteristic and the foreign matter dependency relationship of the lost article are judged according to the three characteristics.
The method for establishing the YOLOV5 target detection algorithm model comprises the following steps:
s11 preparing a data set: collecting data, segmenting the data and generating a data set; the data set comprises the type of the foreign matter needing to be detected;
s12, enhancement treatment: then, data enhancement is carried out by using a Mosaic data enhancement mode to obtain an enhanced data set, and the enhanced data set is divided into a training set and a test set;
s13, building a YOLOV5 target detection algorithm model: and inputting and training data of the training set, and obtaining the model weight of the Yolov5 target detection algorithm.
The method comprises the steps that a Mosaic data enhancement mode is used for an input data set, a self-adaptive anchor training mode is added, a Focus structure is added to perform slicing operation on a characteristic diagram on a backbone network similar to the YOLOV4, two CSP structures are used, and GIOU _ loss is used as a training loss function; in the Mosaic data enhancement-Yoloov 5, a Mosaic data enhancement method is still used in the stage of training a model, and the algorithm is improved on the basis of a CutMix data enhancement method; cutMix only uses two pictures for splicing, while the Mosaic data enhancement method adopts 4 pictures and splices the pictures according to the modes of random scaling, random cutting and random arrangement, and the enhancement method can combine several pictures into one picture, thereby not only enriching the data set, but also greatly improving the training speed of the network, and reducing the memory requirement of the Yolov5 target detection algorithm model.
Focus structure: the structure mainly cuts an input picture through a slicing operation. If the original input picture size is 608 × 3, a feature map of 304 × 12 is output after the slicing and stitching operations; then, a feature map with a size of 304 × 32 is output through a stack with a channel number of 32 (the channel number is only for the YOLOv5s structure, and other structures have corresponding changes).
CSP structure: in the YOLOV4 network structure, the CSP structure is designed only in the backbone network; two CSP structures are designed in the YOLOV5, wherein the CSP1_ X structure is applied to a Backbone network of the backhaul network, and the other CSP2_ X structure is applied to a hack network.
So-called IOU Loss, i.e., the intersection between the prediction box and the GT box/the union between the prediction box and the GT box; a measuring mode of an intersection scale is added in the GIOU _ Loss, and the purpose is to solve the problem that in the IOU Loss, when a prediction box and a GT box are not intersected, the Loss is 0, and the IOU _ Loss cannot optimize the condition that the two boxes are not intersected.
In addition, the manner of generating the data set in step S11 is specifically: performing frame extraction processing on the video by adopting a video frame extraction mode to generate a data set; the data enhancement using the Mosaic data enhancement mode in step S12 specifically includes: the data set is enhanced by rotating, cropping and increasing or decreasing the brightness of the image.
In step S13, the method for detecting people by using the built YOLOV5 target detection algorithm model includes: acquiring image data in the selected area, and acquiring each frame of image data through a video or an rtsp stream; and detecting whether a person appears in each frame of image data.
If the YOLOV5 target detection algorithm model in the current image detects that a person is present, storing a detection result, namely storing coordinates of four points of the rectangular frame in a list, and calculating coordinates of a center point of the rectangular frame, wherein the calculation formula is as follows:
wherein xmin, xmax, ymin, and ymax represent coordinates of four points of the rectangular frame, and c _ x and c _ y represent coordinates of a center point of the rectangular frame; judging whether the personnel enter a specified area or not according to the coordinates of the central point, drawing the specified area by using opencv, and transmitting parameter polygon coordinate points to obtain a specific area; and if the coordinates of the central point are in the selected area, judging that the detected personnel enter the selected area.
In step S2, a deep learning Deepsort target tracking algorithm is used to track the person ID and the article ID, and the Deepsort target tracking algorithm adds cascade matching and confirmation of a new track on the basis of the Sort algorithm, and specifically includes: predicting the track by adopting a Kalman filter; performing cascade matching and IOU matching on the predicted track and the detection result in the current frame by adopting a Hungarian algorithm; and determining the foreign matter ID, the person ID and the person ID with the foreign matter association attribute by using the Kalman filtering updating track determination tracking result. In addition, a REID judgment (REID pedestrian re-identification) can be performed according to a target detection result before cascade matching and IOU matching are performed by adopting the Hungarian algorithm, and the tracking accuracy is further improved.
The predecessor of the Deepsort target Tracking algorithm is the Sort algorithm, which is called Simple Online and real Tracking; the Sort algorithm is mainly characterized in that the target detection method is based on fast R-CNN, and the Kalman filtering algorithm and the Hungary algorithm are utilized, so that the multi-target tracking speed is greatly increased, and the accuracy of SOTA is achieved.
The Deepsort target tracking algorithm model of the embodiment has the main characteristics that: appearance information is added on the basis of an SORT algorithm, and a ReID field model is used for extracting appearance characteristics, so that the number of times of ID switch is reduced; the matching mechanism is changed from the original matching based on the IOU cost matrix into cascade matching + IOU matching.
The core of the Deepsort target tracking algorithm model is a Kalman filter and a Hungarian algorithm, wherein the Kalman filter (Kalman filtering algorithm) is divided into two processes: predicting and updating, wherein the algorithm defines the motion state of the target as 8 normally distributed vectors; and (3) prediction: when the target moves, predicting parameters such as the position and the speed of the target frame of the current frame through parameters such as the target frame and the speed of the previous frame; updating: and the predicted value and the observed value are obtained, and the two normally distributed states are subjected to linear weighting to obtain the predicted state.
The Hungarian algorithm solves the bipartite graph distribution problem, and an IOU cost matrix for calculating the similarity in the main MOT step obtains a similarity matrix of two frames before and after; the Hungarian algorithm solves the real matching target of the front frame and the rear frame by solving the similarity matrix.
In the depsort target tracking algorithm model of this embodiment, adaptive anchor frame calculation is adopted, and anchor frames with specific length and width need to be set for different data sets. In the training phase, the model outputs a corresponding prediction frame on the basis of the initial anchor frame, calculates the difference between the prediction frame and the GT frame, and performs a reverse updating operation, thereby updating the parameters of the whole network, so that setting the initial anchor frame is also a relatively critical ring.
In step S2, the distance between the center points is used to determine the membership between the foreign object ID and the person ID, the center point position of the person ID is calculated on the result of the person ID tracked by the Deepsort target tracking algorithm, the center point position of the foreign object ID is calculated on the result of the detection by the YOLOV5 target detection algorithm model, the membership between the foreign object ID and the person ID is calculated according to the euclidean distance between the two center point positions, when the foreign object ID is closest to the person ID, the foreign object ID is determined to be the foreign object ID, and the foreign object ID and the person ID are associated and matched.
In the step S2, detecting persons and articles appearing in each frame of image by adopting a built YOLOV5 target detection algorithm model and giving IDs; and then using the detection result of the YOLOV5 target detection algorithm model as a target frame of a pre-trained Deepsort target tracking algorithm model for input, using the obtained section of track as the current frame image track, performing IOU intersection and comparison matching through the target frame and the current frame image track, predicting the state of the next frame image target frame according to the track state by Kalman filtering, and updating the states of all tracks by using Kalman filtering observation values and estimation values, thereby completing the tracking of the foreign matter ID and the personnel ID.
In step S2, the membership relationship between the foreign object ID and the person ID is determined, and the euclidean distance between the foreign object ID and the person ID is calculated using the coordinates of the center points of the person ID detection frame and the foreign object ID detection frame, and the calculation formula is:
wherein,
is a point
And point
The Euclidean distance between the two people is calculated according to the formula of the Euclidean distance, the central coordinate of the foreign matter ID is calculated when the foreign matter is static, the coordinate of the central point of the current frame person ID is calculated, the coordinate of the central point of the current frame person ID is used as two points, the person ID with the minimum distance is obtained through calculation of the Euclidean distance formula, the foreign matter is judged to belong to the person ID nearest to the foreign matter, and the person ID is recorded as the person carrying the foreign matterNamed as X _ ID, namely the coordinates of the center point of the person X _ ID with the foreign matter association attribute are obtained;
in the step S3, according to the obtained coordinates of the center point of the person X _ ID with the foreign matter associated attribute, the YOLOV5 target detection algorithm model in the step S1 is used to detect whether the person X _ ID with the foreign matter associated attribute is in the selected area, when the person X _ ID with the foreign matter associated attribute is not in the selected area, the time-sequence associated entry and exit feature is used to determine whether the entry and exit behavior of the person X _ ID with the foreign matter associated attribute, that is, whether the person X _ ID is away from the selected area, and when the person X _ ID with the foreign matter associated attribute satisfies the normal distribution of the 010 feature, the person X _ ID with the foreign matter associated attribute is determined to be away from the selected area.
In step S3, the determining whether the person ID having the foreign object association attribute leaves the selected area specifically includes: constructing a personnel time sequence feature list on personnel ID results tracked by a Deepsort target tracking algorithm to store personnel in-out features and time sequence features, marking the ID of personnel not leaving the entering area as 1 and marking the ID of personnel leaving the area as 0 in a certain time sequence; the in-out behavior and the time sequence relevance of the personnel ID are analyzed, the in-out state of the personnel ID can be judged when the in-out characteristic meets normal distribution under sequential time sequence, and the in-out characteristic is specifically represented as 010 characteristic when the in-out characteristic of the personnel ID meets normal distribution, and the characteristic represents that the personnel ID enters and leaves an area within sequential time sequence; and when the personnel ID meeting the 010 characteristic is judged by utilizing the time sequence correlation, marking the corresponding personnel ID as a state of leaving after entering the area.
The specific experimental environment of the method for detecting lost articles in the deep learning-based region according to the present embodiment is shown in table 1 below:
TABLE 1 Experimental Environment
Pairs of algorithms are shown in table 2 below:
TABLE 2 comparison of algorithms
As shown in fig. 3a and 3b, the time sequence characteristic distribution of the person access behavior in the embodiment of the present invention is recorded, and a density curve graph is drawn according to the recorded time sequence characteristic of the person access behavior, wherein fig. 3a shows a time sequence characteristic distribution diagram of the person access behavior of a behavior, a characteristic value is given according to a center of a person distance region when the time sequence characteristic is constructed, and fig. 3a mainly shows a part of the characteristic distribution to provide data support for the characteristic density curve of fig. 3 b; fig. 3b shows a time sequence characteristic density curve of the person entering and exiting behavior, wherein the curve shows that data is in a peak wave shape, similar to a signal 010 in communication, and conforms to a normal distribution characteristic, and when the person entering and exiting time sequence characteristic meets the distribution characteristic, the person is judged to have the entering and exiting behavior; in fig. 4, the ID of the entering area is detected at the beginning of the algorithm, two IDs, ID1 and ID2, are present in fig. 4, but only ID1 enters the selected area, and in fig. 5, when ID1 leaves, a mobile phone is left in the screen, and the mobile phone is judged to be a forgotten object (lost object) and gives an alarm.
It is obvious to those skilled in the art that the present invention is not limited to the above embodiments, and it is within the scope of the present invention to adopt various insubstantial modifications of the method concept and technical scheme of the present invention, or to directly apply the concept and technical scheme of the present invention to other occasions without modification.
Claims (7)
1. A method for detecting lost articles in an area based on deep learning is characterized by comprising the following steps:
s1: in the selected area, detecting personnel and foreign matters by adopting a deep learning target detection algorithm model, continuously judging whether personnel enter or not, and locking personnel ID (identity); meanwhile, detecting foreign matters, and if no foreign matters appear, continuously judging the detection of the foreign matters;
s2: after the foreign matter is detected, locking the foreign matter ID, performing association matching on the foreign matter ID and the person ID in the step S1, and dynamically matching the foreign matter and the person with the nearest distance to obtain the person ID with the foreign matter association attribute;
s3: judging whether the personnel ID with the foreign matter correlation attribute leaves a selected area, if the personnel ID with the foreign matter correlation attribute is detected to leave the selected area, judging whether the foreign matter correlated with the personnel ID is still in the selected area, if so, judging that the foreign matter is the lost matter, outputting an alarm result, and completing the detection of the lost matter;
in the step S1, a deep learning YOLOV5 target detection algorithm model is used to detect people and foreign objects, and the method for establishing the YOLOV5 target detection algorithm model is as follows:
s11 preparing a data set: collecting data, segmenting the data and generating a data set; the data set comprises the type of the foreign matter needing to be detected;
s12, enhancement treatment: then, data enhancement is carried out by using a Mosaic data enhancement mode to obtain an enhanced data set, and the enhanced data set is divided into a training set and a test set;
s13, building a YOLOV5 target detection algorithm model: inputting and training data of a training set, and obtaining a model weight of a Yolov5 target detection algorithm;
in the step S2, a deep learning Deepsort target tracking algorithm is adopted to track the person ID and the article ID, and the Deepsort target tracking algorithm specifically includes: predicting the track by adopting a Kalman filter; performing cascade matching and IOU matching on the predicted track and the detection result in the current frame by adopting a Hungarian algorithm; determining foreign matter ID, person ID, and person ID having foreign matter-associated attribute using Kalman filtering to update trajectory determination tracking result
In step S3, the determining whether the person ID having the foreign object association attribute leaves the selected area specifically includes: constructing a personnel time sequence feature list on personnel ID results tracked by a Deepsort target tracking algorithm to store personnel in-out features and time sequence features, marking the ID of personnel not leaving the entering area as 1 and marking the ID of personnel leaving the area as 0 in a certain time sequence; the in-out behavior and the time sequence relevance of the personnel ID are analyzed, the in-out state of the personnel ID can be judged when the in-out characteristic meets normal distribution under sequential time sequence, and the in-out characteristic is specifically represented as 010 characteristic when the in-out characteristic of the personnel ID meets normal distribution, and the characteristic represents that the personnel ID enters and leaves an area within sequential time sequence; and when the personnel ID meeting the 010 characteristic is judged by utilizing the time sequence association, marking the corresponding personnel ID as a state of leaving after entering the area.
2. The method for detecting lost articles in area based on deep learning of claim 1, wherein in step S2, the distance between the center points is used to determine the membership between the foreign object ID and the person ID, the center point position of the person ID is calculated on the result of the person ID tracked by the Deepsort target tracking algorithm, the center point position of the foreign object ID is calculated on the result of the detection by the YOLOV5 target detection algorithm model, the membership between the foreign object ID and the person ID is calculated according to the euclidean distance between the two center point positions, when the foreign object ID is closest to the person ID, the foreign object ID is determined to be the foreign object ID, and the association matching between the foreign object ID and the person ID is performed.
3. The method for detecting lost articles in the area based on deep learning of claim 2, wherein in the step S13, the method for detecting the persons by using the built YOLOV5 target detection algorithm model is as follows: acquiring image data in the selected area, and acquiring each frame of image data through a video or an rtsp stream; and detecting whether a person appears in each frame of image data.
4. The method for detecting lost articles in an area based on deep learning of claim 3, wherein the manner of generating the data set in step S11 is specifically as follows: performing frame extraction processing on the video by adopting a video frame extraction mode to generate a data set; the data enhancement using the Mosaic data enhancement mode in the step S12 specifically includes: the data set is enhanced by rotating, cropping and increasing or decreasing the brightness of the image.
5. The method for detecting lost articles in the area based on the deep learning of claim 3, wherein in the step S13, if the YOLOV5 target detection algorithm model in the current image detects the presence of a person, the detection result is saved, that is, coordinates of four points of the rectangular frame are saved in a list, and coordinates of a center point of the rectangular frame are calculated, and the calculation formula is as follows:
wherein xmin, xmax, ymin, and ymax represent coordinates of four points of the rectangular frame, and c _ x and c _ y represent coordinates of a center point of the rectangular frame; judging whether the personnel enter a specified area or not according to the coordinates of the central point, drawing the specified area by using opencv, and transmitting parameter polygon coordinate points to obtain a specific area; and if the central point coordinate is in the selected area, judging that the detected person enters the selected area.
6. The method for detecting lost articles in the area based on deep learning of claim 3, wherein in the step S2, a built YOLOV5 target detection algorithm model is adopted to detect the persons and articles appearing in each frame of image and give IDs; and then using the detection result of the YOLOV5 target detection algorithm model as a target frame of a pre-trained Deepsort target tracking algorithm model for input, using the obtained section of track as the current frame image track, performing IOU intersection and comparison matching through the target frame and the current frame image track, predicting the state of the next frame image target frame according to the track state by Kalman filtering, and updating the states of all tracks by using Kalman filtering observation values and estimation values, thereby completing the tracking of the foreign matter ID and the personnel ID.
7. The method for detecting lost articles in an area based on deep learning of claim 6, wherein in the step S2, the membership relationship between the foreign object ID and the person ID is determined, and the euclidean distance between the foreign object ID and the person ID is calculated by using the coordinates of the center points of the person ID detection frame and the foreign object ID detection frame, and the calculation formula is:
where ρ is a point (x) 2 ，y 2 ) And point (x) 1 ，y 1 ) The Euclidean distance between the personnel ID and the foreign object is calculated when the foreign object is static, the coordinate of the center point of the personnel ID of the current frame is calculated, the coordinate of the center point of the foreign object ID and the coordinate of the center point of the personnel ID of the current frame are used as two points, the personnel ID of the minimum distance is obtained through calculation of a formula of the Euclidean distance, the foreign object is judged to belong to the personnel ID closest to the foreign object, the personnel ID is recorded as the personnel carrying the foreign object and named as X _ ID, and the coordinate of the center point of the personnel X _ ID with the foreign object correlation attribute is obtained;
in the step S3, according to the obtained coordinates of the central point of the person X _ ID with the foreign matter associated attribute, the YOLOV5 target detection algorithm model in the step S1 is used to detect whether the person X _ ID with the foreign matter associated attribute is in the selected area, when the person X _ ID with the foreign matter associated attribute is not in the selected area, the time-sequence associated entry and exit feature is used to determine the entry and exit behaviors of the person X _ ID with the foreign matter associated attribute, and when the person X _ ID with the foreign matter associated attribute satisfies the normal distribution of 010 feature, the person X _ ID with the foreign matter associated attribute is determined to leave the selected area.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202211462573.7A CN115497056B (en) | 2022-11-21 | 2022-11-21 | Method for detecting lost articles in region based on deep learning |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202211462573.7A CN115497056B (en) | 2022-11-21 | 2022-11-21 | Method for detecting lost articles in region based on deep learning |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN115497056A CN115497056A (en) | 2022-12-20 |
| CN115497056B true CN115497056B (en) | 2023-03-24 |
Family
ID=85114944
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202211462573.7A Active CN115497056B (en) | 2022-11-21 | 2022-11-21 | Method for detecting lost articles in region based on deep learning |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN115497056B (en) |
Families Citing this family (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN117037045B (en) * | 2023-10-08 | 2024-04-26 | 成都考拉悠然科技有限公司 | Anomaly detection system based on fusion clustering and deep learning |
| CN117710903B (en) * | 2024-02-05 | 2024-05-03 | 南京信息工程大学 | Visual specific pedestrian tracking method and system based on ReID and Yolov5 double models |
Family Cites Families (13)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US8164443B2 (en) * | 2009-04-08 | 2012-04-24 | International Business Machines Corporation | Group aware tracking of personal property within a bounded zone |
| WO2012141663A1 (en) * | 2011-04-13 | 2012-10-18 | Alptekin Temizel | A method for individual tracking of multiple objects |
| KR20200057923A (en) * | 2018-11-19 | 2020-05-27 | 최영준 | Multi-purpose anti-lost sticker |
| CN110532852B (en) * | 2019-07-09 | 2022-10-18 | 长沙理工大学 | Subway station pedestrian abnormal event detection method based on deep learning |
| CN111199182A (en) * | 2019-11-12 | 2020-05-26 | 恒大智慧科技有限公司 | Lost object method, system and storage medium based on intelligent community |
| CN113671907B (en) * | 2020-05-14 | 2023-01-24 | 华晨宝马汽车有限公司 | Production monitoring method and device for monitoring vehicle production and storage medium |
| CN113807450A (en) * | 2021-09-23 | 2021-12-17 | 怀化学院 | Unmanned aerial vehicle power line patrol fault detection method based on ultrahigh resolution picture |
| CN113920392A (en) * | 2021-09-23 | 2022-01-11 | 同济大学 | Library mobile personnel target detection and multi-target tracking method and management system |
| CN114926757A (en) * | 2022-04-20 | 2022-08-19 | 上海商汤科技开发有限公司 | Method and system for retrieving lost article, electronic equipment and computer storage medium |
| CN114862837A (en) * | 2022-06-02 | 2022-08-05 | 西京学院 | Human body security check image detection method and system based on improved YOLOv5s |
| CN115131821A (en) * | 2022-06-29 | 2022-09-30 | 大连理工大学 | Improved YOLOv5+ Deepsort-based campus personnel crossing warning line detection method |
| CN115115675A (en) * | 2022-07-26 | 2022-09-27 | 通号通信信息集团有限公司 | Method for detecting carry-over, electronic device, computer readable medium |
| CN115297306B (en) * | 2022-10-10 | 2023-03-24 | 深圳市旗扬特种装备技术工程有限公司 | Pedestrian personal belonging anti-loss monitoring method, device, equipment and storage medium |
-
2022
- 2022-11-21 CN CN202211462573.7A patent/CN115497056B/en active Active
Also Published As
| Publication number | Publication date |
|---|---|
| CN115497056A (en) | 2022-12-20 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN115497056B (en) | Method for detecting lost articles in region based on deep learning | |
| US10990191B2 (en) | Information processing device and method, program and recording medium for identifying a gesture of a person from captured image data | |
| CN106164980B (en) | Information processing apparatus and information processing method | |
| Senior et al. | Appearance models for occlusion handling | |
| CN101482923B (en) | Human body target detection and sexuality recognition method in video monitoring | |
| WO2017150032A1 (en) | Method and system for detecting actions of object in scene | |
| JP2002157599A (en) | Method for detecting and recognizing object, recording medium recording its program and object monitoring and tracking device | |
| Ruta et al. | Detection, tracking and recognition of traffic signs from video input | |
| CN108846852B (en) | Monitoring video abnormal event detection method based on multiple examples and time sequence | |
| Denman et al. | Multi-spectral fusion for surveillance systems | |
| CN113378675A (en) | Face recognition method for simultaneous detection and feature extraction | |
| CN110929670A (en) | Muck truck cleanliness video identification and analysis method based on yolo3 technology | |
| Xiao et al. | Vehicle and person tracking in aerial videos | |
| JP6283008B2 (en) | Tracking device | |
| Afonso et al. | Automatic estimation of multiple motion fields from video sequences using a region matching based approach | |
| Yang et al. | A method of pedestrians counting based on deep learning | |
| CN116935325A (en) | Bus stop legacy detection method based on detection and tracking | |
| CN114581847B (en) | Community pedestrian abnormal behavior detection method and device based on GAM tracker | |
| Belmouhcine et al. | Robust deep simple online real-time tracking | |
| CN115909497A (en) | Human body posture recognition method and device | |
| Sridevi et al. | Automatic generation of traffic signal based on traffic volume | |
| Saif et al. | Adaptive long term motion pattern analysis for moving object detection using UAV aerial images | |
| CN114821441A (en) | Deep learning-based airport scene moving target identification method combined with ADS-B information | |
| Sujatha et al. | An innovative moving object detection and tracking system by using modified region growing algorithm | |
| JP3763279B2 (en) | Object extraction system, object extraction method, and object extraction program |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |