CN111813995A - Pedestrian article extraction behavior detection method and system based on space-time relationship - Google Patents
Pedestrian article extraction behavior detection method and system based on space-time relationship Download PDFInfo
- Publication number
- CN111813995A CN111813995A CN202010619517.4A CN202010619517A CN111813995A CN 111813995 A CN111813995 A CN 111813995A CN 202010619517 A CN202010619517 A CN 202010619517A CN 111813995 A CN111813995 A CN 111813995A
- Authority
- CN
- China
- Prior art keywords
- pedestrian
- article
- face
- detection
- detection result
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000001514 detection method Methods 0.000 title claims abstract description 225
- 238000000605 extraction Methods 0.000 title claims abstract description 27
- 238000000034 method Methods 0.000 claims abstract description 23
- 238000012544 monitoring process Methods 0.000 claims abstract description 22
- 230000006399 behavior Effects 0.000 claims description 62
- 238000013135 deep learning Methods 0.000 claims description 11
- 238000012549 training Methods 0.000 claims description 11
- 238000004590 computer program Methods 0.000 claims description 9
- 238000012937 correction Methods 0.000 claims description 4
- 238000010586 diagram Methods 0.000 description 7
- 210000003128 head Anatomy 0.000 description 6
- 238000004891 communication Methods 0.000 description 5
- 210000000887 face Anatomy 0.000 description 3
- 238000007689 inspection Methods 0.000 description 3
- 238000011897 real-time detection Methods 0.000 description 3
- 230000009471 action Effects 0.000 description 2
- 238000013528 artificial neural network Methods 0.000 description 2
- 230000008569 process Effects 0.000 description 2
- 230000001174 ascending effect Effects 0.000 description 1
- 230000005540 biological transmission Effects 0.000 description 1
- 238000004364 calculation method Methods 0.000 description 1
- 238000013527 convolutional neural network Methods 0.000 description 1
- 238000002372 labelling Methods 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/70—Information retrieval; Database structures therefor; File system structures therefor of video data
- G06F16/78—Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually
- G06F16/783—Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually using metadata automatically derived from the content
- G06F16/7837—Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually using metadata automatically derived from the content using objects detected or recognised in the video content
- G06F16/784—Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually using metadata automatically derived from the content using objects detected or recognised in the video content the detected or recognised objects being people
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/70—Information retrieval; Database structures therefor; File system structures therefor of video data
- G06F16/78—Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually
- G06F16/783—Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually using metadata automatically derived from the content
- G06F16/7837—Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually using metadata automatically derived from the content using objects detected or recognised in the video content
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/70—Information retrieval; Database structures therefor; File system structures therefor of video data
- G06F16/78—Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually
- G06F16/7867—Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually using information manually generated, e.g. tags, keywords, comments, title and artist information, manually generated time, location and usage information, user ratings
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/70—Information retrieval; Database structures therefor; File system structures therefor of video data
- G06F16/78—Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually
- G06F16/787—Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually using geographical or spatial information, e.g. location
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/22—Matching criteria, e.g. proximity measures
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/045—Combinations of networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
Abstract
The invention provides a pedestrian article extraction behavior detection method and system based on a space-time relationship, wherein the method comprises the following steps: the method comprises the steps of detecting an acquired monitoring video frame image in real time based on a trained pedestrian key point detection model, a face key point detection model and an article detection model to respectively obtain a pedestrian detection result, a face detection result and an article detection result; matching the pedestrian detection result with the face detection result based on the space-time relation of the object in the monitoring video frame image to obtain a related pedestrian and face matching result, and matching the pedestrian detection result with the object detection result to obtain a related pedestrian and object matching result; and detecting the behavior of extracting the article, and associating the pedestrian corresponding to the behavior of extracting the article according to the matching result of the pedestrian and the article. The pedestrian detection method can combine the space-time relationship to realize real-time accurate detection of pedestrian extracted articles.
Description
Technical Field
The invention relates to the field of target detection, in particular to a pedestrian article extraction behavior detection method and system based on a space-time relationship.
Background
At present, in places such as airports, railway stations, bus stations, subway stations, ports, frontier inspection, customs, government buildings, embassy halls, convention centers and the like, article security inspection equipment is arranged to detect suspicious articles. When the suspicious goods relate to the illegal crime, the physical evidence that the related staff can collect is generally limited to the suspicious goods, and the evidence that can prove the relevance between the suspicious goods and the suspicious people is difficult to collect, so that the suspicious people cannot be effectively controlled by the evidence.
At present, in a real security inspection scene, due to the fact that a background is quite complex, pedestrians are mobile, article types are various, and a single type of target detection model is not enough to meet detection judgment of association between the pedestrians and the articles. However, in the prior art, no target detection method can effectively relate the action of the pedestrian and the action of extracting the article.
Therefore, a method capable of accurately detecting the behavior of the pedestrian extracting the article is needed to solve the above problems.
Disclosure of Invention
The invention provides a pedestrian article extraction behavior detection method and system based on a spatiotemporal relationship, which overcomes or at least partially solves the problems.
According to a first aspect provided by the invention, the invention provides a pedestrian article extraction behavior detection method based on a spatiotemporal relationship, which comprises the following steps:
the method comprises the steps of detecting an acquired monitoring video frame image in real time based on a trained pedestrian key point detection model, a face key point detection model and an article detection model to respectively obtain a pedestrian detection result, a face detection result and an article detection result;
matching the pedestrian detection result with the face detection result based on the space-time relation of the object in the monitoring video frame image to obtain a related pedestrian and face matching result, and matching the pedestrian detection result with the object detection result to obtain a related pedestrian and object matching result;
and detecting the behavior of extracting the article, and associating the pedestrian corresponding to the behavior of extracting the article according to the matching result of the pedestrian and the article.
Wherein the method further comprises:
carrying out deep learning training by utilizing the constructed pedestrian data set to obtain the pedestrian key point detection model;
carrying out deep learning training by using the constructed face data set to obtain the face key point detection model;
and carrying out deep learning training by utilizing the constructed object data set to obtain the object detection model, wherein the pedestrian data set, the human face data set and the object data set are video frame images captured from a historical monitoring video, and pedestrians, human faces and objects in the video frame images are marked and classified.
Wherein, based on pedestrian key point detection model, face key point detection model and the article detection model that train, carry out real-time detection to the surveillance video frame image who obtains, obtain pedestrian detection result, face detection result and article detection result respectively, include:
acquiring a human body positioning rectangular frame and human body skeleton information of the pedestrian on a frame image based on the trained pedestrian key point detection model;
acquiring a face positioning rectangular frame on a frame image and a face image after face alignment correction based on a trained face key point detection model;
and acquiring an article positioning rectangular frame on the frame image based on the trained article detection model.
Wherein, the matching the pedestrian detection result and the face detection result to obtain the associated pedestrian and face matching result comprises:
in the same frame of image, if the overlapping degree of the area of the face in the image and the head area in the human skeleton information of the pedestrian is larger than a preset threshold value, the pedestrian and the face are judged to be successfully matched in an associated mode.
In the same frame of image, the areas of rectangular frames of a face region and a head region in the pedestrian human skeleton information are respectively A1 and A2, the area of an overlapping region is A0, and the overlapping rate R is:
and the overlapping rate threshold value is T, and if R is larger than T, the pedestrian and the face are judged to be successfully associated and matched.
Wherein, the matching the pedestrian detection result and the article detection result to obtain the associated pedestrian and article matching result comprises:
in the same frame of image, if the object has an arm vertex in the pedestrian human skeleton information in the region of the image, judging that the association matching between the pedestrian and the object is successful.
Wherein the detecting the behavior of extracting the article comprises:
and comparing the object detection result in the current frame image with the object detection results in the previous frames of images, and if the coordinates of the center point of the object positioning rectangular frame rise in the images, judging that the behavior of extracting the object exists.
Setting the coordinate of the central point of the article positioning rectangle frame in the vertical direction at the current frame as YtY in the previous framet-1Coordinate variation difference Δ Y ═Yt-Yt-1(ii) a When DeltaY is larger than threshold value TyWhen the counter is over, the counter C is accumulated; the counter threshold is TcWhen C is greater than TcAnd judging that the coordinates of the center point of the article positioning rectangular frame rise in the image.
Wherein the method further comprises:
and if the pedestrian corresponding to the behavior of the article is extracted and the associated pedestrian and face matching result exists, outputting the aligned corrected face image corresponding to the pedestrian.
According to a second aspect provided by the present invention, the present invention provides a pedestrian extraction article behavior detection system based on spatiotemporal relationship, comprising:
the data detection module is used for detecting the acquired monitoring video frame images in real time based on the trained pedestrian key point detection model, the face key point detection model and the object detection model to respectively obtain a pedestrian detection result, a face detection result and an object detection result;
the data matching module is used for matching the pedestrian detection result with the face detection result based on the space-time relation of the object in the monitoring video frame image to obtain a related pedestrian and face matching result, and matching the pedestrian detection result with the object detection result to obtain a related pedestrian and object matching result;
and the behavior detection module is used for detecting the behavior of extracting the article and associating the pedestrian corresponding to the behavior of extracting the article according to the matching result of the pedestrian and the article.
According to a third aspect provided by the present invention, an embodiment of the present invention provides an electronic device, which includes a memory, a processor, and a computer program stored on the memory and executable on the processor, and the processor executes the program to implement the steps of the pedestrian extraction item behavior detection method based on spatiotemporal relationship as provided in the first aspect.
In a fourth aspect, an embodiment of the present invention provides a non-transitory computer-readable storage medium, on which a computer program is stored, where the computer program, when executed by a processor, implements the steps of the pedestrian extraction item behavior detection method based on spatiotemporal relationship as provided in the above first aspect.
According to the pedestrian article extraction behavior detection method and system based on the space-time relationship, the behavior of pedestrian article extraction is identified through a human body key point detection model, a human face key point detection model and an article detection model based on deep learning and coordinate information on time and space is combined, and real-time and accurate detection of pedestrian article extraction is achieved.
Drawings
Fig. 1 is a schematic flow chart of a pedestrian article extraction behavior detection method based on a spatiotemporal relationship provided by an embodiment of the present invention.
Fig. 2 is a schematic structural diagram of a pedestrian article extraction behavior detection system based on spatiotemporal relationship according to an embodiment of the present invention.
Fig. 3 illustrates a schematic structural diagram of an electronic device.
Detailed Description
In order to make the objects, technical solutions and advantages of the present invention more apparent, the present invention is described in further detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.
Fig. 1 is a schematic flow chart of a pedestrian article extraction behavior detection method based on spatiotemporal relationship, as shown in fig. 1, including:
101. the method comprises the steps of detecting an acquired monitoring video frame image in real time based on a trained pedestrian key point detection model, a face key point detection model and an article detection model to respectively obtain a pedestrian detection result, a face detection result and an article detection result;
102. matching the pedestrian detection result with the face detection result based on the space-time relation of the object in the monitoring video frame image to obtain a related pedestrian and face matching result, and matching the pedestrian detection result with the object detection result to obtain a related pedestrian and object matching result;
103. and detecting the behavior of extracting the article, and associating the pedestrian corresponding to the behavior of extracting the article according to the matching result of the pedestrian and the article.
Specifically, in step 101, three detection models, namely a pedestrian key point detection model, a face key point detection model and an article detection model, are trained in advance in the embodiment of the present invention, and the pedestrian key point detection model is specifically a human body positioning rectangular frame and human body key point positions, such as a head, shoulders, arms, a trunk and legs, of a pedestrian in an image by labeling information. When the pedestrian key point detection model is used, firstly, a convolutional neural network operation is carried out on an input image to obtain a feature map, then the feature map is respectively input into a key point confidence coefficient prediction network and a key point affinity vector field prediction network to obtain the key point confidence coefficient and affinity vector of each key point, according to the key point confidence coefficient network, a position with the highest confidence coefficient is selected as a key point position of network prediction, finally, according to the key point affinity vector field network, the correlation of the two key points is calculated, a connection line mode with the highest affinity is output, and clustering is carried out on the key points.
The training process of the face key point detection model is realized by positioning a rectangular frame and the positions of face key points, such as eyes, a nose and a mouth, for the faces of pedestrians in the image by marking information. When the face key point detection model is used, firstly, two-stage cascade convolution neural network operation is carried out on an image, then a face region is obtained through calculation according to the first-stage convolution neural network, and finally the face region is used as the input of the second stage to obtain face key point information (position and confidence).
The training process of the article detection model is realized by marking information as a positioning rectangular frame of an article in an image, and the article detection model is mainly divided into three parts: a base layer, an extension layer, and a prediction layer. And calculating the input image through a basic layer to obtain two characteristic diagrams, calculating the input image through an extended layer to obtain four characteristic diagrams, and finally obtaining the position coordinate and the confidence coefficient of the article through the six characteristic diagrams through a prediction layer.
Further, in step 102, the embodiment of the present invention matches the pedestrian detection result and the face detection result in sequence by using the spatiotemporal relationship in the monitoring video frame image to obtain a related pedestrian and face matching result, and then matches the pedestrian face matching result and the object detection result to obtain a related pedestrian and object matching result.
Finally, in step 103, the behavior of extracting the article is automatically output, and the pedestrian extracting the article is determined according to the association of the article and the pedestrian before, so that the evidence is saved.
According to the pedestrian article extraction behavior detection method based on the space-time relationship, the behavior of extracting articles from pedestrians is identified through a human body key point detection model, a human face key point detection model and an article detection model based on deep learning and by combining coordinate information on time and space, and real-time and accurate detection of the pedestrian article extraction is achieved.
On the basis of the above embodiment, the method further includes:
carrying out deep learning training by utilizing the constructed pedestrian data set to obtain the pedestrian key point detection model;
carrying out deep learning training by using the constructed face data set to obtain the face key point detection model;
and carrying out deep learning training by utilizing the constructed object data set to obtain the object detection model, wherein the pedestrian data set, the human face data set and the object data set are video frame images captured from a historical monitoring video, and pedestrians, human faces and objects in the video frame images are marked and classified.
On the basis of the above embodiment, the real-time detection is performed on the acquired surveillance video frame image based on the trained pedestrian key point detection model, the face key point detection model and the article detection model, so as to obtain a pedestrian detection result, a face detection result and an article detection result respectively, including:
acquiring a human body positioning rectangular frame and human body skeleton information of the pedestrian on a frame image based on the trained pedestrian key point detection model;
acquiring a face positioning rectangular frame on a frame image and a face image after face alignment correction based on a trained face key point detection model;
and acquiring an article positioning rectangular frame on the frame image based on the trained article detection model.
The pedestrian detection result comprises a human body positioning rectangular frame and human body skeleton information of the pedestrian on the frame image, and comprises coordinate information of the head, the shoulders, the arms, the trunk and the legs; the detection result of the human face is a human face positioning rectangular frame and a human face picture after the human face is aligned and corrected, and the object detection result is an object positioning rectangular frame.
On the basis of the above embodiment, the matching the pedestrian detection result and the face detection result to obtain a related pedestrian and face matching result includes:
in the same frame of image, if the overlapping degree of the area of the face in the image and the head area in the human skeleton information of the pedestrian is larger than a preset threshold value, the pedestrian and the face are judged to be successfully matched in an associated mode.
When the pedestrian and the face appear in the detection result at the same time, the embodiment of the invention judges whether the pedestrian is matched with the face according to the plane coordinate information of the frame image, specifically, whether the region of the face in the image is overlapped with the head region in the human skeleton information of the pedestrian, and when the overlapping degree reaches a certain threshold value, the detected pedestrian and the face are matched.
On the basis of the above embodiment, the matching the pedestrian detection result and the article detection result to obtain a related pedestrian and article matching result includes:
in the same frame of image, if the object has an arm vertex in the pedestrian human skeleton information in the region of the image, judging that the association matching between the pedestrian and the object is successful.
And when the pedestrian and the article appear in the detection result at the same time, judging whether the pedestrian is matched with the article according to the plane coordinate information of the frame image, namely whether the article has an arm vertex in the pedestrian human body skeleton information in the region of the image, and if so, pairing the detected pedestrian and article.
On the basis of the above embodiment, the detecting the behavior of extracting the article includes:
and comparing the object detection result in the current frame image with the object detection results in the previous frames of images, and if the coordinates of the center point of the object positioning rectangular frame rise in the images, judging that the behavior of extracting the object exists.
The embodiment of the invention identifies the behavior of extracting the article by comparing the article detection result of the current frame with the article detection results of the previous frames, thereby judging whether the position of the article in the frame image rises. Specifically, when an article appears in the detection result, coordinate information of the center point of each frame of article rectangular frame is recorded, and a plurality of frames are taken as 5 frames for illustration, and if the upper left corner of the image is taken as the origin of coordinates and the coordinate value of the center point of the article rectangular frame of the 5 frames in the front and back directions in the vertical direction is reduced, the article is judged to be ascending.
It can be understood that if the coordinates of the center point of the rectangular frame of the object are moving upwards in the frame image, the object is considered to be extracted by a pedestrian who is successfully paired, the object is associated with the corresponding pedestrian, and the result is output.
On the basis of the above embodiment, the method further includes:
and if the pedestrian corresponding to the behavior of the article is extracted and the associated pedestrian and face matching result exists, outputting the aligned corrected face image corresponding to the pedestrian.
It should be noted that, if the behavior of the pedestrian for extracting the object is successfully detected, the embodiment of the present invention may further output and extract a face image corresponding to the pedestrian of the object according to the result of matching between the pedestrian and the face, where the face image is a face image after face alignment correction output by the face key point detection model. When the article is a suspicious article, the method can be convenient for a law enforcement department to capture the suspicious person after face recognition in time.
Fig. 2 is a schematic structural diagram of a pedestrian article extraction behavior detection system based on spatiotemporal relationship according to an embodiment of the present invention, as shown in fig. 2, including: a data detection module 201, a data matching module 202, and a behavior detection module 203, wherein:
the data detection module 201 is configured to perform real-time detection on the acquired surveillance video frame image based on the trained pedestrian key point detection model, the trained face key point detection model and the trained article detection model, and obtain a pedestrian detection result, a face detection result and an article detection result respectively;
the data matching module 202 is configured to match the pedestrian detection result and the face detection result based on a spatiotemporal relationship of the object in the surveillance video frame image to obtain an associated pedestrian and face matching result, and match the pedestrian detection result and the object detection result to obtain an associated pedestrian and object matching result;
the behavior detection module 203 is configured to detect a behavior of extracting an article, and associate a pedestrian corresponding to the behavior of extracting the article according to the result of matching the pedestrian with the article.
For a specific way of detecting the behavior of the pedestrian by using the data detection module 201, the data matching module 202 and the behavior detection module 203, reference may be made to the embodiment shown in fig. 1, and details of the embodiment of the present invention are not repeated herein.
Fig. 3 illustrates a schematic structural diagram of an electronic device, and as shown in fig. 3, the server may include: a processor (processor)301, a communication Interface (communication Interface)302, a memory (memory)303 and a bus 304, wherein the processor 301, the communication Interface 302 and the memory 303 complete communication with each other through the bus 304. The communication interface 302 may be used for information transmission between the server and the smart television. Processor 301 may call logic instructions in memory 303 to perform the following method: the method comprises the steps of detecting an acquired monitoring video frame image in real time based on a trained pedestrian key point detection model, a face key point detection model and an article detection model to respectively obtain a pedestrian detection result, a face detection result and an article detection result; matching the pedestrian detection result with the face detection result based on the space-time relation of the object in the monitoring video frame image to obtain a related pedestrian and face matching result, and matching the pedestrian detection result with the object detection result to obtain a related pedestrian and object matching result; and detecting the behavior of extracting the article, and associating the pedestrian corresponding to the behavior of extracting the article according to the matching result of the pedestrian and the article.
The present embodiments also provide a computer program product comprising a computer program stored on a non-transitory computer-readable storage medium, the computer program comprising program instructions that, when executed by a computer, enable the computer to perform the methods provided by the above-described method embodiments, for example, including: the method comprises the steps of detecting an acquired monitoring video frame image in real time based on a trained pedestrian key point detection model, a face key point detection model and an article detection model to respectively obtain a pedestrian detection result, a face detection result and an article detection result; matching the pedestrian detection result with the face detection result based on the space-time relation of the object in the monitoring video frame image to obtain a related pedestrian and face matching result, and matching the pedestrian detection result with the object detection result to obtain a related pedestrian and object matching result; and detecting the behavior of extracting the article, and associating the pedestrian corresponding to the behavior of extracting the article according to the matching result of the pedestrian and the article.
The present embodiments provide a non-transitory computer-readable storage medium storing computer instructions that cause the computer to perform the methods provided by the above method embodiments, for example, including: the method comprises the steps of detecting an acquired monitoring video frame image in real time based on a trained pedestrian key point detection model, a face key point detection model and an article detection model to respectively obtain a pedestrian detection result, a face detection result and an article detection result; matching the pedestrian detection result with the face detection result based on the space-time relation of the object in the monitoring video frame image to obtain a related pedestrian and face matching result, and matching the pedestrian detection result with the object detection result to obtain a related pedestrian and object matching result; and detecting the behavior of extracting the article, and associating the pedestrian corresponding to the behavior of extracting the article according to the matching result of the pedestrian and the article.
The above description is only for the purpose of illustrating the preferred embodiments of the present invention and is not to be construed as limiting the invention, and any modifications, equivalents and improvements made within the spirit and principle of the present invention are intended to be included within the scope of the present invention.
Claims (10)
1. A pedestrian article extraction behavior detection method based on a spatiotemporal relationship is characterized by comprising the following steps:
the method comprises the steps of detecting an acquired monitoring video frame image in real time based on a trained pedestrian key point detection model, a face key point detection model and an article detection model to respectively obtain a pedestrian detection result, a face detection result and an article detection result;
matching the pedestrian detection result with the face detection result based on the space-time relation of the object in the monitoring video frame image to obtain a related pedestrian and face matching result, and matching the pedestrian detection result with the object detection result to obtain a related pedestrian and object matching result;
and detecting the behavior of extracting the article, and associating the pedestrian corresponding to the behavior of extracting the article according to the matching result of the pedestrian and the article.
2. The pedestrian extraction article behavior detection method based on the spatiotemporal relationship as claimed in claim 1, wherein the method further comprises:
carrying out deep learning training by utilizing the constructed pedestrian data set to obtain the pedestrian key point detection model;
carrying out deep learning training by using the constructed face data set to obtain the face key point detection model;
and carrying out deep learning training by utilizing the constructed object data set to obtain the object detection model, wherein the pedestrian data set, the human face data set and the object data set are video frame images captured from a historical monitoring video, and pedestrians, human faces and objects in the video frame images are marked and classified.
3. The pedestrian extraction article behavior detection method based on the spatiotemporal relationship as claimed in claim 2, wherein the detecting the acquired surveillance video frame image in real time based on the trained pedestrian key point detection model, the face key point detection model and the article detection model to respectively obtain the pedestrian detection result, the face detection result and the article detection result comprises:
acquiring a human body positioning rectangular frame and human body skeleton information of the pedestrian on a frame image based on the trained pedestrian key point detection model;
acquiring a face positioning rectangular frame on a frame image and a face image after face alignment correction based on a trained face key point detection model;
and acquiring an article positioning rectangular frame on the frame image based on the trained article detection model.
4. The pedestrian extraction article behavior detection method based on the spatiotemporal relationship as claimed in claim 3, wherein the matching of the pedestrian detection result and the face detection result to obtain the associated pedestrian and face matching result comprises:
in the same frame of image, if the overlapping degree of the area of the face in the image and the head area in the human skeleton information of the pedestrian is larger than a preset threshold value, the pedestrian and the face are judged to be successfully matched in an associated mode.
5. The pedestrian extraction article behavior detection method based on the spatiotemporal relationship as claimed in claim 4, wherein the matching of the pedestrian detection result and the article detection result to obtain the associated pedestrian and article matching result comprises:
in the same frame of image, if the object has an arm vertex in the pedestrian human skeleton information in the region of the image, judging that the association matching between the pedestrian and the object is successful.
6. The pedestrian extracted article behavior detection method based on spatiotemporal relationship according to claim 5, wherein the detecting the behavior of the extracted article comprises:
and comparing the object detection result in the current frame image with the object detection results in the previous frames of images, and if the coordinates of the center point of the object positioning rectangular frame rise in the images, judging that the behavior of extracting the object exists.
7. The pedestrian extraction article behavior detection method based on the spatiotemporal relationship as claimed in claim 1, wherein the method further comprises:
and if the pedestrian corresponding to the behavior of the article is extracted and the associated pedestrian and face matching result exists, outputting the aligned corrected face image corresponding to the pedestrian.
8. A pedestrian extraction article behavior detection system based on spatiotemporal relationship is characterized by comprising:
the data detection module is used for detecting the acquired monitoring video frame images in real time based on the trained pedestrian key point detection model, the face key point detection model and the object detection model to respectively obtain a pedestrian detection result, a face detection result and an object detection result;
the data matching module is used for matching the pedestrian detection result with the face detection result based on the space-time relation of the object in the monitoring video frame image to obtain a related pedestrian and face matching result, and matching the pedestrian detection result with the object detection result to obtain a related pedestrian and object matching result;
and the behavior detection module is used for detecting the behavior of extracting the article and associating the pedestrian corresponding to the behavior of extracting the article according to the matching result of the pedestrian and the article.
9. An electronic device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, wherein the processor when executing the program implements the steps of the pedestrian extraction article behavior detection method based on spatiotemporal relationship as claimed in any one of claims 1 to 7.
10. A non-transitory computer readable storage medium, on which a computer program is stored, wherein the computer program, when executed by a processor, implements the steps of the pedestrian extraction article behavior detection method based on spatiotemporal relationship according to any one of claims 1 to 7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010619517.4A CN111813995A (en) | 2020-07-01 | 2020-07-01 | Pedestrian article extraction behavior detection method and system based on space-time relationship |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010619517.4A CN111813995A (en) | 2020-07-01 | 2020-07-01 | Pedestrian article extraction behavior detection method and system based on space-time relationship |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN111813995A true CN111813995A (en) | 2020-10-23 |
Family
ID=72855682
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010619517.4A Pending CN111813995A (en) | 2020-07-01 | 2020-07-01 | Pedestrian article extraction behavior detection method and system based on space-time relationship |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111813995A (en) |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114022143A (en) * | 2022-01-06 | 2022-02-08 | 深圳爱莫科技有限公司 | Code scanning payment behavior model training method, detection method and processing equipment |
| CN115631464A (en) * | 2022-11-17 | 2023-01-20 | 北京航空航天大学 | Pedestrian three-dimensional representation method oriented to large space-time target association |
Citations (13)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102938058A (en) * | 2012-11-14 | 2013-02-20 | 南京航空航天大学 | Method and system for video driving intelligent perception and facing safe city |
| CN105516656A (en) * | 2015-11-27 | 2016-04-20 | 上海博康智能信息技术有限公司 | Article real condition viewing method and system |
| CN107516127A (en) * | 2017-08-21 | 2017-12-26 | 山东大学 | Service robot independently obtains people and wears the method and system for taking article ownership semanteme |
| CN108711086A (en) * | 2018-05-09 | 2018-10-26 | 连云港伍江数码科技有限公司 | Man-machine interaction method, device, article-storage device and storage medium in article-storage device |
| CN108830251A (en) * | 2018-06-25 | 2018-11-16 | 北京旷视科技有限公司 | Information correlation method, device and system |
| CN108898109A (en) * | 2018-06-29 | 2018-11-27 | 北京旷视科技有限公司 | The determination methods, devices and systems of article attention rate |
| WO2018215861A1 (en) * | 2017-05-24 | 2018-11-29 | Kpit Technologies Limited | System and method for pedestrian detection |
| CN109544838A (en) * | 2019-01-14 | 2019-03-29 | 博拉网络股份有限公司 | A kind of artificial intelligence cognition identifying system for special area |
| WO2019090769A1 (en) * | 2017-11-13 | 2019-05-16 | 深圳和而泰智能控制股份有限公司 | Human face shape recognition method and apparatus, and intelligent terminal |
| CN109977782A (en) * | 2019-02-27 | 2019-07-05 | 浙江工业大学 | Across shop operation detection method based on target position information reasoning |
| CN110287867A (en) * | 2019-06-24 | 2019-09-27 | 广州织点智能科技有限公司 | Unmanned convenience store enters recognition methods, device, equipment and storage medium |
| CN110688987A (en) * | 2019-10-16 | 2020-01-14 | 山东建筑大学 | Pedestrian position detection and tracking method and system |
| CN111209868A (en) * | 2020-01-08 | 2020-05-29 | 中国铁道科学研究院集团有限公司电子计算技术研究所 | Passenger and luggage information association method and device for passenger station |
-
2020
- 2020-07-01 CN CN202010619517.4A patent/CN111813995A/en active Pending
Patent Citations (13)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102938058A (en) * | 2012-11-14 | 2013-02-20 | 南京航空航天大学 | Method and system for video driving intelligent perception and facing safe city |
| CN105516656A (en) * | 2015-11-27 | 2016-04-20 | 上海博康智能信息技术有限公司 | Article real condition viewing method and system |
| WO2018215861A1 (en) * | 2017-05-24 | 2018-11-29 | Kpit Technologies Limited | System and method for pedestrian detection |
| CN107516127A (en) * | 2017-08-21 | 2017-12-26 | 山东大学 | Service robot independently obtains people and wears the method and system for taking article ownership semanteme |
| WO2019090769A1 (en) * | 2017-11-13 | 2019-05-16 | 深圳和而泰智能控制股份有限公司 | Human face shape recognition method and apparatus, and intelligent terminal |
| CN108711086A (en) * | 2018-05-09 | 2018-10-26 | 连云港伍江数码科技有限公司 | Man-machine interaction method, device, article-storage device and storage medium in article-storage device |
| CN108830251A (en) * | 2018-06-25 | 2018-11-16 | 北京旷视科技有限公司 | Information correlation method, device and system |
| CN108898109A (en) * | 2018-06-29 | 2018-11-27 | 北京旷视科技有限公司 | The determination methods, devices and systems of article attention rate |
| CN109544838A (en) * | 2019-01-14 | 2019-03-29 | 博拉网络股份有限公司 | A kind of artificial intelligence cognition identifying system for special area |
| CN109977782A (en) * | 2019-02-27 | 2019-07-05 | 浙江工业大学 | Across shop operation detection method based on target position information reasoning |
| CN110287867A (en) * | 2019-06-24 | 2019-09-27 | 广州织点智能科技有限公司 | Unmanned convenience store enters recognition methods, device, equipment and storage medium |
| CN110688987A (en) * | 2019-10-16 | 2020-01-14 | 山东建筑大学 | Pedestrian position detection and tracking method and system |
| CN111209868A (en) * | 2020-01-08 | 2020-05-29 | 中国铁道科学研究院集团有限公司电子计算技术研究所 | Passenger and luggage information association method and device for passenger station |
Non-Patent Citations (1)
| Title |
|---|
| 陈乙雄等: "移动设备新技术简明教程", 31 August 2016, 重庆大学出版社, pages: 54 - 58 * |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114022143A (en) * | 2022-01-06 | 2022-02-08 | 深圳爱莫科技有限公司 | Code scanning payment behavior model training method, detection method and processing equipment |
| CN115631464A (en) * | 2022-11-17 | 2023-01-20 | 北京航空航天大学 | Pedestrian three-dimensional representation method oriented to large space-time target association |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN109740413B (en) | Pedestrian re-identification method, device, computer equipment and computer storage medium | |
| CN110738101B (en) | Behavior recognition method, behavior recognition device and computer-readable storage medium | |
| CN103824070B (en) | A kind of rapid pedestrian detection method based on computer vision | |
| WO2021051601A1 (en) | Method and system for selecting detection box using mask r-cnn, and electronic device and storage medium | |
| CN108960067B (en) | Real-time train driver action recognition system and method based on deep learning | |
| CN105590097B (en) | Dual camera collaboration real-time face identification security system and method under the conditions of noctovision | |
| KR20160088224A (en) | Method for recognizing an object and apparatus thereof | |
| CN114187665B (en) | Multi-person gait recognition method based on human skeleton heat map | |
| US20220180534A1 (en) | Pedestrian tracking method, computing device, pedestrian tracking system and storage medium | |
| CN111241932A (en) | Automobile exhibition room passenger flow detection and analysis system, method and storage medium | |
| KR102406804B1 (en) | Method and system for automatically detecting objects in image based on deep learning | |
| CN111611874A (en) | Face mask wearing detection method based on ResNet and Canny | |
| US10496874B2 (en) | Facial detection device, facial detection system provided with same, and facial detection method | |
| JP2012083855A (en) | Object recognition device and object recognition method | |
| CN103020992A (en) | Video image significance detection method based on dynamic color association | |
| CN112183472A (en) | Method for detecting whether test field personnel wear work clothes or not based on improved RetinaNet | |
| CN111813995A (en) | Pedestrian article extraction behavior detection method and system based on space-time relationship | |
| CN112270381A (en) | People flow detection method based on deep learning | |
| CN112200056A (en) | Face living body detection method and device, electronic equipment and storage medium | |
| KR20170006356A (en) | Method for customer analysis based on two-dimension video and apparatus for the same | |
| KR20210100937A (en) | Device for identifying the situaton of object's conduct using sensor fusion | |
| CN111310720A (en) | Pedestrian re-identification method and system based on graph metric learning | |
| CN115690046B (en) | Article carry-over detection and tracing method and system based on monocular depth estimation | |
| CN114529979A (en) | Human body posture identification system, human body posture identification method and non-transitory computer readable storage medium | |
| US20220076005A1 (en) | Person authentication apparatus, control method, and non-transitory storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |