CN112163469B - Smoking behavior recognition method, system, equipment and readable storage medium - Google Patents
Smoking behavior recognition method, system, equipment and readable storage medium Download PDFInfo
- Publication number
- CN112163469B CN112163469B CN202010955978.9A CN202010955978A CN112163469B CN 112163469 B CN112163469 B CN 112163469B CN 202010955978 A CN202010955978 A CN 202010955978A CN 112163469 B CN112163469 B CN 112163469B
- Authority
- CN
- China
- Prior art keywords
- frame
- video
- cigarette
- neural network
- key points
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
- G06V40/20—Movements or behaviour, e.g. gesture recognition
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/045—Combinations of networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V20/00—Scenes; Scene-specific elements
- G06V20/40—Scenes; Scene-specific elements in video content
- G06V20/41—Higher-level, semantic clustering, classification or understanding of video scenes, e.g. detection, labelling or Markovian modelling of sport events or news items
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V20/00—Scenes; Scene-specific elements
- G06V20/40—Scenes; Scene-specific elements in video content
- G06V20/44—Event detection
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02P—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN THE PRODUCTION OR PROCESSING OF GOODS
- Y02P90/00—Enabling technologies with a potential contribution to greenhouse gas [GHG] emissions mitigation
- Y02P90/30—Computing systems specially adapted for manufacturing
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- General Health & Medical Sciences (AREA)
- Health & Medical Sciences (AREA)
- Software Systems (AREA)
- Computational Linguistics (AREA)
- Computing Systems (AREA)
- Mathematical Physics (AREA)
- Data Mining & Analysis (AREA)
- Molecular Biology (AREA)
- Biophysics (AREA)
- General Engineering & Computer Science (AREA)
- Biomedical Technology (AREA)
- Evolutionary Computation (AREA)
- Artificial Intelligence (AREA)
- Life Sciences & Earth Sciences (AREA)
- Multimedia (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Psychiatry (AREA)
- Social Psychology (AREA)
- Human Computer Interaction (AREA)
- Image Analysis (AREA)
Abstract
The embodiment of the application discloses a smoking behavior identification method, a system, equipment and a readable storage medium, which aim at collected videos to be analyzed, detect whether cigarettes exist in video frames, label the cigarettes in the video frames with the cigarettes, and form a cigarette data set; inputting the video to be analyzed into a human body posture estimation neural network to obtain key points of a human body in a frame sequence, and inputting the key points into a graph convolution neural network to obtain a behavior label of the human body in each frame; inputting the frame sequence of the cigarette data set into a target detection neural network to detect the cigarette position in a frame image and obtain the cigarette confidence of each frame; and judging whether smoking behaviors exist in each frame according to the behavior label of each frame and the cigarette confidence coefficient, and marking a smoking label in the video with the smoking behaviors. The smoking behavior is accurately identified, and the fire is effectively avoided.
Description
Technical Field
The embodiment of the application relates to the technical field of artificial intelligence, in particular to a smoking behavior identification method, a system, equipment and a readable storage medium.
Background
The number of global smoking people is increasing every year, smoking not only seriously harms the health of people, but also increases the fire hazard, especially for the areas of no smoking fire, such as gas stations, public places, dangerous goods areas, inflammable goods warehouses and the like, once a fire happens, immeasurable loss is caused. Therefore, it is important to quickly and effectively detect smoking behavior in the above-mentioned area, and to prevent the smoking behavior from occurring in the past.
Current methods for determining smoking behavior can be divided into two broad categories, the first category relying primarily on sensor detection, such as: collecting cigarette end infrared thermal data by using infrared sensing equipment; the determination is made using a smoke sensor device. The second category mainly relies on real-time video to detect smoking behavior, and particularly in recent years, with the rapid development of deep neural networks, the identification of occurring smoking behavior in video by means of intelligent video analysis algorithm shows a variety of solutions, such as: classifying the video single-frame image by using a convolutional neural network; segmenting the head and hand regions of a person in a video, and detecting whether cigarettes exist in the segmented regions by using a depth algorithm; partial key points (such as faces and hands) of people are detected, and the position relation of the key points between frames is analyzed.
The method for detecting smoking by a sensor is only suitable for detecting a closed space, is greatly influenced by environmental illumination and ventilation, and basically cannot meet the detection requirement of an outdoor smoking ban. And the sensitivity of the sensor is difficult to adapt, frequent false alarm is easy to occur when the sensitivity is too high, and the late alarm is easy to miss when the sensitivity is too low, so that the sensor cannot play a role in prevention. The existing method for detecting smoking behavior by means of real-time video analysis basically only considers the posture of a human body in a single-frame image, is easy to generate false alarm, is easy to be interfered by factors such as external light and the like, and is poor in real-time performance.
Disclosure of Invention
Therefore, the embodiment of the application provides a smoking behavior identification method, a system, equipment and a readable storage medium, a human body key point sequence is used as input, and a method for identifying smoking behaviors is completed by using a graph volume and a target detection deep neural network algorithm together, so that the potential smoking behaviors are detected in real time while the accuracy is ensured, and a fire disaster is effectively avoided.
In order to achieve the above object, the embodiments of the present application provide the following technical solutions:
according to a first aspect of embodiments of the present application, there is provided a smoking behaviour identification method, the method comprising:
detecting whether cigarettes exist in the video frames or not according to the collected video to be analyzed, and labeling the cigarettes in the video frames with the cigarettes to form a cigarette data set;
inputting the video to be analyzed into a human body posture estimation neural network to obtain key points of a human body in a frame sequence, and inputting the key points into a graph convolution neural network to obtain a behavior label of the human body in each frame;
inputting the frame sequence of the cigarette data set into a target detection neural network to detect the cigarette position in a frame image and obtain the cigarette confidence of each frame;
and judging whether smoking behaviors exist in each frame according to the behavior label and the cigarette confidence coefficient of each frame, and marking a smoking label in the video with the smoking behaviors.
Optionally, inputting the video to be analyzed into a human body posture estimation neural network to obtain key points of a human body in a frame sequence, including:
inputting the video to be analyzed into a human body posture estimation neural network, predicting and acquiring the whole body joint point connection of each person in each frame of video, taking a frame sequence with a set length for each section of video, connecting the frames and the same joint point to form a graph structure, and thus obtaining the key points of the human body in the frame sequence.
Optionally, the inputting the key points into a graph convolution neural network to obtain a behavior tag of the person in each frame includes:
constructing a skeleton space-time diagram G-V, E through each video frame; the node matrix set V ═ vti | T ═ 1., T, i ═ 1., N }, the frame number is T, the number of key points is N, E includes Es and Ef, Es refers to connectivity of key points in a single frame according to human body structures, edges are connected, and Ef refers to a single key point being connected to a node of the same type on consecutive video frames;
automatically capturing the spatial configuration and the temporal dynamics among the key points by using the graph convolution neural network, wherein in space, local features of adjacent key points in the same frame are learned by using convolution, and in time, features among corresponding key points among the convolution learning frames are also used for carrying out convolution alternation in space and time, so as to extract a higher-level feature graph;
and classifying the feature map into corresponding behavior categories by using a classifier to obtain the behavior label of the person in each frame.
Optionally, before detecting whether a cigarette is present in the video frame, the method further comprises:
and aiming at the collected video to be analyzed, normalizing the frame rate and the resolution of the video so as to enable the time lengths in the graph convolution neural network to be consistent.
According to a second aspect of embodiments of the present application, there is provided a smoking behaviour recognition system, the system comprising:
the data preprocessing module is used for detecting whether cigarettes exist in the video frames aiming at the collected video to be analyzed and marking the cigarettes in the video frames with the cigarettes to form a cigarette data set;
the key point detection module is used for inputting the video to be analyzed into a human body posture estimation neural network to obtain key points of a human body in a frame sequence, and inputting the key points into a graph convolution neural network to obtain a behavior tag of the human body in each frame;
the cigarette detection module is used for inputting the frame sequence of the cigarette data set into a target detection neural network so as to detect the cigarette position in a frame image and obtain the cigarette confidence of each frame;
and the smoking action recognition module is used for judging whether smoking actions exist in each frame according to the behavior label of each frame and the cigarette confidence coefficient, and printing a smoking label in the video with the smoking actions.
Optionally, the key point detection module is specifically configured to:
inputting the video to be analyzed into a human body posture estimation neural network, predicting and acquiring the whole body joint point connection of each person in each frame of video, taking a frame sequence with a set length for each section of video, connecting the frames and the same joint point to form a graph structure, and thus obtaining the key points of the human body in the frame sequence.
Optionally, the key point detection module is specifically configured to:
constructing a skeleton space-time diagram G-V, E through each video frame; the node matrix set V ═ vti | T ═ 1., T, i ═ 1., N }, the frame number is T, the number of key points is N, E includes Es and Ef, Es refers to connectivity of key points in a single frame according to human body structures, edges are connected, and Ef refers to a single key point being connected to a node of the same type on consecutive video frames;
automatically capturing the spatial configuration and the time dynamics among the key points by utilizing the graph convolution neural network, wherein in space, local features of adjacent key points in the same frame are learned by using convolution, and in time, features among corresponding key points among the convolution learning frames are also utilized to perform convolution alternation in space and time so as to extract a higher-level feature graph;
and classifying the feature map into corresponding behavior categories by using a classifier to obtain the behavior label of the person in each frame.
Optionally, the data preprocessing module is further configured to:
and aiming at the collected video to be analyzed, normalizing the frame rate and the resolution of the video so as to enable the time lengths in the graph convolution neural network to be consistent.
According to a third aspect of embodiments herein, there is provided an apparatus comprising: the device comprises a data acquisition device, a processor and a memory; the data acquisition device is used for acquiring data; the memory is to store one or more program instructions; the processor is configured to execute one or more program instructions to perform the method of any of the first aspect.
According to a fourth aspect of embodiments herein, there is provided a computer-readable storage medium having one or more program instructions embodied therein for performing the method of any of the first aspects.
In summary, the present application provides a smoking behavior identification method, system, device, and readable storage medium, which detect whether a cigarette exists in a video frame for an acquired video to be analyzed, and mark the cigarette in the video frame where the cigarette exists to form a cigarette data set; inputting the video to be analyzed into a human body posture estimation neural network to obtain key points of a human body in a frame sequence, and inputting the key points into a graph convolution neural network to obtain a behavior tag of the human body in each frame; inputting the frame sequence of the cigarette data set into a target detection neural network to detect the cigarette position in a frame image and obtain the cigarette confidence of each frame; and judging whether smoking behaviors exist in each frame according to the behavior label of each frame and the cigarette confidence coefficient, and marking a smoking label in the video with the smoking behaviors. The method for identifying smoking behaviors by using the key point sequence of the human body as input and using the graph volume and the target detection deep neural network algorithm together achieves real-time detection of the potential smoking behaviors while ensuring the precision, and effectively avoids fire.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below. It should be apparent that the drawings in the following description are merely exemplary, and that other embodiments can be derived from the drawings provided by those of ordinary skill in the art without inventive effort.
The structures, ratios, sizes, and the like shown in the present specification are only used for matching with the contents disclosed in the specification, so that those skilled in the art can understand and read the present invention, and do not limit the conditions for implementing the present invention, so that the present invention has no technical significance, and any structural modifications, changes in the ratio relationship, or adjustments of the sizes, without affecting the functions and purposes of the present invention, should still fall within the scope of the present invention.
Fig. 1 is a schematic flow chart of a smoking behavior identification method according to an embodiment of the present disclosure;
FIG. 2 is a schematic flow chart of a model training phase according to an embodiment of the present disclosure;
FIG. 3 is a schematic flow chart of a model prediction phase according to an embodiment of the present application;
fig. 4 is a block diagram of a smoking behavior recognition system according to an embodiment of the present application.
Detailed Description
The present invention is described in terms of particular embodiments, other advantages and features of the invention will become apparent to those skilled in the art from the following disclosure, and it is to be understood that the described embodiments are merely exemplary of the invention and that it is not intended to limit the invention to the particular embodiments disclosed. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
Fig. 1 shows a flow of a smoking behavior recognition method provided in an embodiment of the present application, and as shown in fig. 1, the method includes the following steps:
step 101: and detecting whether cigarettes exist in the video frames aiming at the collected videos to be analyzed, and labeling the cigarettes in the video frames with the cigarettes to form a cigarette data set.
Step 102: inputting the video to be analyzed into a human body posture estimation neural network to obtain key points of a human body in a frame sequence, and inputting the key points into a graph convolution neural network to obtain a behavior tag of the human body in each frame.
Step 103: and inputting the frame sequence of the cigarette data set into a target detection neural network to detect the cigarette position in the frame image and obtain the cigarette confidence of each frame.
Step 104: and judging whether smoking behaviors exist in each frame according to the behavior label of each frame and the cigarette confidence coefficient, and marking a smoking label in the video with the smoking behaviors.
In a possible implementation, before detecting whether a cigarette is present in the video frame in step 101, the method further includes: and aiming at the collected video to be analyzed, normalizing the frame rate and the resolution of the video so as to enable the time lengths in the graph convolution neural network to be consistent.
In a possible implementation manner, in step 102, inputting the video to be analyzed into a human body posture estimation neural network to obtain key points of a human body in a frame sequence, including: inputting the video to be analyzed into a human body posture estimation neural network, predicting and acquiring the whole body joint point connection of each person in each frame of video, taking a frame sequence with a set length for each section of video, connecting the frames and the same joint point to form a graph structure, and thus obtaining the key points of the human body in the frame sequence.
In one possible implementation, in step 102, the inputting the keypoints into a convolutional neural network to obtain a behavior tag of the person in each frame includes: constructing a skeleton space-time diagram G-V, E through each video frame; the node matrix set V ═ vti | T ═ 1., T, i ═ 1., N }, the frame number is T, the number of key points is N, E includes Es and Ef, Es refers to connectivity of key points in a single frame according to human body structures, edges are connected, and Ef refers to a single key point being connected to a node of the same type on consecutive video frames; automatically capturing the spatial configuration and the time dynamics among the key points by utilizing the graph convolution neural network, wherein in space, local features of adjacent key points in the same frame are learned by using convolution, and in time, features among corresponding key points among the convolution learning frames are also utilized to perform convolution alternation in space and time so as to extract a higher-level feature graph; and classifying the feature map into corresponding behavior categories by using a classifier to obtain the behavior label of the person in each frame.
In the method provided by the embodiment of the application, a human body posture estimation neural network is used for predicting and acquiring the joint point connection of the whole body of each person in each video frame, each video frame is a frame sequence (for example, a frame sequence corresponding to 10 seconds), and the same joint point is also connected between frames, so that a graph structure is formed, and the joint point connection is input as the node characteristic of the graph.
The spatial convolution requires a convolution kernel (sample space) and convolution weights, the sample space p (h, w) being defined over adjacent pixels with respect to the central position x, a sampling function is defined on the periphery of the node Vti of the graph set b (Vti) { Vti | D (Vtj, Vti) ≦ D }, where D (Vtj, Vti) denotes the minimum length of any path from node Vtj to node Vti, where D is set to 1 (i.e., the neighbors of x), the convolution weights are the set b (Vti) of nodes around node Vti divided into 3 subsets according to spatial configuration partitions, namely, a root node Ot exists in the joint points of the whole body, the length of d (Ot, Vti) is taken as a division standard, wherein Vti is divided into a group separately, the distance between the node and the root node Ot is greater than d, and divided into a centripetal group, otherwise, the centripetal group, 3 groups get different weight values, and the convolution also uses the same weight elsewhere on the frame.
The time convolution aims at the same type of joint points, is similar to the space convolution, but not only has D values but also F values (the number of frames) about the sampling function, namely F frames form a feature map, and the weight is directly modified by taking Vti as the root neighborhood (frame) because the time axis is ordered.
A human body joint point combination sequence in a group of video frame images within a period of time (such as 5 seconds) is learned from two directions of space and time by using a graph convolution neural network without the limitation of a topological structure, after space convolution and time convolution are carried out for each time, an Resnet mechanism is used, meanwhile, partial features of dropout are randomly generated according to a certain probability, overfitting is prevented, after multiple times of space-time convolution, a feature vector is finally sent to a SoftMax classifier, action label indexes of each person in each frame are predicted, and a graph convolution network pre-training model capable of identifying smoking behaviors is formed for real-time video analysis.
And directly analyzing the action sequence of the human body in continuous time and space in the real-time video stream by using the trained graph convolution neural network, and judging whether smoking action exists or not. Meanwhile, a target detection neural network is introduced to detect cigarettes in the video frames, and the graph volume analysis and the target detection aiming at the human key point combination sequence are combined for judgment, so that the accuracy rate of smoking behavior identification is high, and the robustness is strong.
Fig. 2 shows a model training phase flow provided by the embodiment of the present application, and fig. 3 shows a model prediction phase flow provided by the embodiment of the present application. The smoking behavior recognition method provided by the embodiment of the present application is further explained with reference to fig. 2 and fig. 3, and specifically may include the following steps:
step 1: preprocessing a data set: the frame rate and the resolution of the video in the data set are normalized, the frame number of the video is made to be equal, and the time length (the number of frames) of the graph convolution neural network is ensured to be the same. Meanwhile, in order to detect whether cigarettes exist in the video frame, if the cigarette position marked on the video frame exists, a new data set is formed.
Step 2: and (3) key point detection: and performing key point prediction on the processed video frame by using a human body posture estimation neural network. If the number of people in a single frame exceeds 5, the 5 people with the highest confidence coefficient are kept, and the human key point position sequences of all the frames of the video are stored into a json file. Each of which is 18 keypoints, each keypoint containing 3 features (coordinates x, y, confidence), and if there are some keypoints for a person that are not predicted, then such keypoints are set to (0, 0, 0).
And step 3: graph convolution neural network: and (2) constructing a bone space-time graph G- (V, E) through the json file of each video in the step 2, wherein a node matrix set V- (vti | T-1., T, i-1., N }, the frame number is T, the number of key points is N, E comprises Es and Ef, Es refers to the connectivity of key points according to the human body structure in a single frame, edges are connected, and Ef refers to the fact that a single key point is connected to the same type of node on the continuous video frames. The method comprises the steps of automatically capturing spatial configuration and time dynamics among key points by utilizing convolution, learning local features of adjacent key points in the same frame by utilizing convolution in space, learning features among corresponding key points among frames by utilizing convolution in time, and alternately performing spatial and time convolution to extract a higher-level feature map, and finally classifying the feature map into a corresponding behavior category by utilizing a classifier, namely giving a behavior label to a person detected in each frame. Target (cigarette) detection neural network: and (3) training a target detection neural network model by using the cigarette data set marked in the step (1) to detect the cigarette position in the image.
And 4, step 4: in fig. 3, a video which is actually acquired is mainly used as an input, a human body posture estimation neural network directly predicts key points of a human body in a frame sequence, then a result is transmitted into a graph convolution neural network, a behavior tag in each frame is predicted, meanwhile, the frame sequence is also transmitted into a target detection neural network, whether cigarettes exist in the frames is detected, finally, the two results are fused to judge whether smoking behaviors exist in each frame, and a wrinkling tag is pasted in the video.
And 5: a human-computer interaction interface: the method mainly realizes that an operator gives an Rtsp address or an original video to be analyzed, introduces a pre-trained deep neural network model, commands the model to execute smoking behavior detection operation, and graphically and simultaneously displays an original input video and a video marked with behavior labels and key points on an interface.
In practical application, if a person tends to smoke in a smoke forbidden area, namely when the person takes cigarettes, the graph convolution neural network and the target detection neural network can quickly and accurately judge whether potential smoking behaviors exist or not and timely send out warning prompts, so that the conditions of creating combustion conditions for combustible articles and enabling a warehouse to be in a smoldering state are avoided. Meanwhile, people can be traced according to the graphical display, so that the fire hazard is avoided.
The embodiment of the application has the following beneficial effects:
the accuracy is high, the smoking behavior is identified by adopting the graph convolution neural network and the target detection neural network together, and the accuracy rate reaches 95.52 percent. High efficiency, and the real-time video analysis speed can reach 26 frames/second. The edge embedded equipment can be deployed flexibly, namely the edge embedded equipment can be deployed quickly, the edge embedded equipment can also be deployed in a centralized manner at the cloud end, and the edge embedded equipment can be used in the field only by one common camera, so that the edge embedded equipment is suitable for popularization and large-scale deployment.
In summary, the embodiment of the present application provides a smoking behavior identification method, which detects whether a cigarette exists in a video frame for an acquired video to be analyzed, and marks the cigarette in the video frame where the cigarette exists to form a cigarette data set; inputting the video to be analyzed into a human body posture estimation neural network to obtain key points of a human body in a frame sequence, and inputting the key points into a graph convolution neural network to obtain a behavior label of the human body in each frame; inputting the frame sequence of the cigarette data set into a target detection neural network to detect the cigarette position in a frame image and obtain the cigarette confidence of each frame; and judging whether smoking behaviors exist in each frame according to the behavior label and the cigarette confidence coefficient of each frame, and marking a smoking label in the video with the smoking behaviors. The method for identifying smoking behaviors by using the key point sequence of the human body as input and using the graph volume and the target detection deep neural network algorithm together achieves real-time detection of the potential smoking behaviors while ensuring the precision, and effectively avoids fire.
Based on the same technical concept, an embodiment of the present application further provides a smoking behavior recognition system, as shown in fig. 4, the system includes:
the data preprocessing module 401 is configured to detect whether a cigarette exists in a video frame of the acquired video to be analyzed, and label the cigarette in the video frame where the cigarette exists to form a cigarette data set.
And the key point detection module 402 is configured to input the video to be analyzed into the human body posture estimation neural network to obtain key points of a human body in a frame sequence, and input the key points into the atlas convolution neural network to obtain a behavior tag of the person in each frame.
A cigarette detection module 403, configured to input the frame sequence of the cigarette data set into a target detection neural network to detect a cigarette position in a frame image, so as to obtain a cigarette confidence of each frame.
And a smoking action recognition module 404, configured to determine whether a smoking action exists in each frame according to the behavior tag of each frame and the cigarette confidence level, and print a smoking tag in a video in which the smoking action exists.
In a possible implementation, the key point detecting module 402 is specifically configured to: inputting the video to be analyzed into a human body posture estimation neural network, predicting and acquiring the whole body joint point connection of each person in each frame of video, taking a frame sequence with a set length for each section of video, connecting the frames and the same joint point to form a graph structure, and thus obtaining the key points of the human body in the frame sequence.
In a possible implementation manner, the key point detecting module 402 is specifically configured to: constructing a skeleton space-time diagram G-V, E through each video frame; the node matrix set V ═ vti | T ═ 1., T, i ═ 1., N }, the frame number is T, the number of key points is N, E includes Es and Ef, Es refers to connectivity of key points in a single frame according to human body structures, edges are connected, and Ef refers to a single key point being connected to a node of the same type on consecutive video frames; automatically capturing the spatial configuration and the time dynamics among the key points by utilizing the graph convolution neural network, wherein in space, local features of adjacent key points in the same frame are learned by using convolution, and in time, features among corresponding key points among the convolution learning frames are also utilized to perform convolution alternation in space and time so as to extract a higher-level feature graph; and classifying the feature map into corresponding behavior categories by using a classifier to obtain the behavior label of the person in each frame.
In a possible implementation, the data preprocessing module 401 is further configured to: and aiming at the collected video to be analyzed, normalizing the frame rate and the resolution of the video so as to enable the time lengths in the graph convolution neural network to be consistent.
Based on the same technical concept, the embodiment of the present application further provides an apparatus, including: the device comprises a data acquisition device, a processor and a memory; the data acquisition device is used for acquiring data; the memory is to store one or more program instructions; the processor is configured to execute one or more program instructions to perform the method.
Based on the same technical concept, the embodiment of the present application further provides a computer-readable storage medium, which contains one or more program instructions for executing the method.
In the present specification, each embodiment of the method is described in a progressive manner, and the same and similar parts among the embodiments are referred to each other, and each embodiment focuses on the differences from the other embodiments. Reference is made to the description of the method embodiments in part.
It should be noted that although the operations of the methods of the present invention are depicted in the drawings in a particular order, this does not require or imply that these operations must be performed in this particular order, or that all of the illustrated operations must be performed, to achieve desirable results. Additionally or alternatively, certain steps may be omitted, multiple steps combined into one step execution, and/or one step broken down into multiple step executions.
Although the present application provides method steps as in embodiments or flowcharts, additional or fewer steps may be included based on conventional or non-inventive approaches. The order of steps recited in the embodiments is merely one manner of performing the steps in a multitude of orders and does not represent the only order of execution. When an apparatus or client product in practice executes, it may execute sequentially or in parallel (e.g., in a parallel processor or multithreaded processing environment, or even in a distributed data processing environment) according to the embodiments or methods shown in the figures. The terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, the presence of additional identical or equivalent elements in processes, methods, articles, or apparatus that include the recited elements is not excluded.
The units, devices, modules, etc. set forth in the above embodiments may be implemented by a computer chip or an entity, or by a product with certain functions. For convenience of description, the above devices are described as being divided into various modules by functions, and are described separately. Of course, in implementing the present application, the functions of each module may be implemented in one or more software and/or hardware, or a module implementing the same function may be implemented by a combination of a plurality of sub-modules or sub-units, and the like. The above-described embodiments of the apparatus are merely illustrative, and for example, the division of the units is only one logical division, and other divisions may be realized in practice, for example, a plurality of units or components may be combined or integrated into another system, or some features may be omitted, or not executed. In addition, the shown or discussed mutual coupling or direct coupling or communication connection may be an indirect coupling or communication connection through some interfaces, devices or units, and may be in an electrical, mechanical or other form.
Those skilled in the art will also appreciate that, in addition to implementing the controller as pure computer readable program code, the same functionality can be implemented by logically programming method steps such that the controller is in the form of logic gates, switches, application specific integrated circuits, programmable logic controllers, embedded microcontrollers and the like. Such a controller may therefore be considered as a hardware component, and the means included therein for performing the various functions may also be considered as a structure within the hardware component. Or even means for performing the functions may be regarded as being both a software module for performing the method and a structure within a hardware component.
The application may be described in the general context of computer-executable instructions, such as program modules, being executed by a computer. Generally, program modules include routines, programs, objects, components, data structures, classes, etc. that perform particular tasks or implement particular abstract data types. The application may also be practiced in distributed computing environments where tasks are performed by remote processing devices that are linked through a communications network. In a distributed computing environment, program modules may be located in both local and remote computer storage media including memory storage devices.
From the above description of the embodiments, it is clear to those skilled in the art that the present application can be implemented by software plus necessary general hardware platform. Based on such understanding, the technical solutions of the present application may be embodied in the form of a software product, which may be stored in a storage medium, such as a ROM/RAM, a magnetic disk, an optical disk, or the like, and includes several instructions for enabling a computer device (which may be a personal computer, a mobile terminal, a server, or a network device) to execute the method according to the embodiments or some parts of the embodiments of the present application.
The embodiments in the present specification are described in a progressive manner, and the same or similar parts among the embodiments are referred to each other, and each embodiment focuses on the differences from the other embodiments. The application is operational with numerous general purpose or special purpose computing system environments or configurations. For example: personal computers, server computers, hand-held or portable devices, tablet-type devices, multiprocessor systems, microprocessor-based systems, set top boxes, programmable electronic devices, network PCs, minicomputers, mainframe computers, distributed computing environments that include any of the above systems or devices, and the like.
The above-mentioned embodiments are provided to further explain the objects, technical solutions and advantages of the present application in detail, and it should be understood that the above-mentioned embodiments are only examples of the present application and are not intended to limit the scope of the present application, and any modifications, equivalents, improvements and the like made within the spirit and principle of the present application should be included in the scope of the present application.
Claims (8)
1. A smoking behaviour recognition method, characterised in that the method comprises:
detecting whether cigarettes exist in the video frames aiming at the collected videos to be analyzed, and marking the cigarettes in the video frames with the cigarettes to form a cigarette data set;
inputting the video to be analyzed into a human body posture estimation neural network, predicting and acquiring the whole body joint point connection of each person in each frame of video, taking a frame sequence with a set length for each section of video, connecting the frames and the same joint point to form a graph structure so as to obtain key points of the human body in the frame sequence, and inputting the key points into the graph convolution neural network so as to obtain a behavior tag of the person in each frame;
inputting the frame sequence of the cigarette data set into a target detection neural network to detect the cigarette position in a frame image and obtain the cigarette confidence of each frame;
and judging whether smoking behaviors exist in each frame according to the behavior label and the cigarette confidence coefficient of each frame, and marking a smoking label in the video with the smoking behaviors.
2. The method of claim 1, wherein said inputting the keypoints into a convolutional neural network to obtain behavior labels of the persons in each frame, comprises:
constructing a skeleton space-time diagram G-V, E through each video frame; the node matrix set V ═ vti | T ═ 1., T, i ═ 1., N }, the frame number is T, the number of key points is N, E includes Es and Ef, Es refers to connectivity of key points in a single frame according to human body structures, edges are connected, and Ef refers to a single key point being connected to a node of the same type on consecutive video frames;
automatically capturing the spatial configuration and the time dynamics among the key points by utilizing the graph convolution neural network, wherein in space, local features of adjacent key points in the same frame are learned by using convolution, and in time, features among corresponding key points among the convolution learning frames are also utilized to perform convolution alternation in space and time so as to extract a higher-level feature graph;
and classifying the feature map into corresponding behavior categories by using a classifier to obtain behavior labels of people in each frame.
3. The method of claim 1, wherein prior to detecting whether a cigarette is present in the video frame, the method further comprises:
and aiming at the collected video to be analyzed, normalizing the frame rate and the resolution of the video so as to enable the time lengths in the graph convolution neural network to be consistent.
4. A smoking behaviour recognition system, characterised in that the system comprises:
the data preprocessing module is used for detecting whether cigarettes exist in the video frames aiming at the collected video to be analyzed and marking the cigarettes in the video frames with the cigarettes to form a cigarette data set;
the key point detection module is used for inputting the video to be analyzed into a human body posture estimation neural network, predicting and acquiring the connection of all body joint points of each person in each frame of video, taking a frame sequence with a set length for each section of video, connecting the frames and the same joint point to form a graph structure so as to obtain key points of the human body in the frame sequence, and inputting the key points into the graph convolution neural network so as to obtain a behavior label of the person in each frame;
the cigarette detection module is used for inputting the frame sequence of the cigarette data set into a target detection neural network so as to detect the cigarette position in a frame image and obtain the cigarette confidence of each frame;
and the smoking action recognition module is used for judging whether smoking actions exist in each frame according to the behavior label of each frame and the cigarette confidence coefficient, and printing a smoking label in the video with the smoking actions.
5. The system of claim 4, wherein the keypoint detection module is specifically configured to:
constructing a skeleton space-time diagram G-V, E through each video frame; the node matrix set V ═ vti | T ═ 1., T, i ═ 1., N }, the frame number is T, the number of key points is N, E includes Es and Ef, Es refers to connectivity of key points in a single frame according to human body structures, edges are connected, and Ef refers to a single key point being connected to a node of the same type on consecutive video frames;
automatically capturing the spatial configuration and the time dynamics among the key points by utilizing the graph convolution neural network, wherein in space, local features of adjacent key points in the same frame are learned by using convolution, and in time, features among corresponding key points among the convolution learning frames are also utilized to perform convolution alternation in space and time so as to extract a higher-level feature graph;
and classifying the feature map into corresponding behavior categories by using a classifier to obtain the behavior label of the person in each frame.
6. The system of claim 4, wherein the data pre-processing module is further to:
and aiming at the collected video to be analyzed, normalizing the frame rate and the resolution of the video so as to enable the time lengths in the graph convolution neural network to be consistent.
7. An apparatus, characterized in that the apparatus comprises: the device comprises a data acquisition device, a processor and a memory;
the data acquisition device is used for acquiring data; the memory is to store one or more program instructions; the processor, configured to execute one or more program instructions to perform the method of any of claims 1-3.
8. A computer-readable storage medium having one or more program instructions embodied therein for performing the method of any of claims 1-3.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010955978.9A CN112163469B (en) | 2020-09-11 | 2020-09-11 | Smoking behavior recognition method, system, equipment and readable storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010955978.9A CN112163469B (en) | 2020-09-11 | 2020-09-11 | Smoking behavior recognition method, system, equipment and readable storage medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN112163469A CN112163469A (en) | 2021-01-01 |
| CN112163469B true CN112163469B (en) | 2022-09-02 |
Family
ID=73858923
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010955978.9A Active CN112163469B (en) | 2020-09-11 | 2020-09-11 | Smoking behavior recognition method, system, equipment and readable storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112163469B (en) |
Families Citing this family (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113065474B (en) * | 2021-04-07 | 2023-06-27 | 泰豪软件股份有限公司 | Behavior recognition method and device and computer equipment |
| CN113392706A (en) * | 2021-05-13 | 2021-09-14 | 上海湃道智能科技有限公司 | Device and method for detecting smoking and using mobile phone behaviors |
| CN113688667A (en) * | 2021-07-08 | 2021-11-23 | 华中科技大学 | Deep learning-based luggage taking and placing action recognition method and system |
| CN113609963B (en) * | 2021-08-03 | 2022-10-11 | 北京睿芯高通量科技有限公司 | Real-time multi-human-body-angle smoking behavior detection method |
| CN116977883A (en) * | 2022-04-22 | 2023-10-31 | 中兴通讯股份有限公司 | Smoking behavior detection method and device and related equipment |
| CN114783061B (en) * | 2022-04-26 | 2023-04-18 | 南京积图网络科技有限公司 | Smoking behavior detection method, device, equipment and medium |
| CN115205767A (en) * | 2022-09-16 | 2022-10-18 | 浪潮通信信息系统有限公司 | Smoking behavior detection method, system and device |
Citations (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111144263A (en) * | 2019-12-20 | 2020-05-12 | 山东大学 | Construction worker high-fall accident early warning method and device |
Family Cites Families (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110472573A (en) * | 2019-08-14 | 2019-11-19 | 北京思图场景数据科技服务有限公司 | A kind of human body behavior analysis method, equipment and computer storage medium based on body key point |
| CN111274930B (en) * | 2020-04-02 | 2022-09-06 | 成都鼎安华智慧物联网股份有限公司 | Helmet wearing and smoking behavior identification method based on deep learning |
| CN111582333A (en) * | 2020-04-23 | 2020-08-25 | 浙江大学 | Lightning arrester picture carrying state identification method combining Yolo-v3 and Open-pos |
-
2020
- 2020-09-11 CN CN202010955978.9A patent/CN112163469B/en active Active
Patent Citations (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111144263A (en) * | 2019-12-20 | 2020-05-12 | 山东大学 | Construction worker high-fall accident early warning method and device |
Also Published As
| Publication number | Publication date |
|---|---|
| CN112163469A (en) | 2021-01-01 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN112163469B (en) | Smoking behavior recognition method, system, equipment and readable storage medium | |
| Lestari et al. | Fire hotspots detection system on CCTV videos using you only look once (YOLO) method and tiny YOLO model for high buildings evacuation | |
| CN108229407A (en) | A kind of behavioral value method and system in video analysis | |
| CN111325137B (en) | Violent sorting detection method, device, equipment and storage medium | |
| CN111126153B (en) | Safety monitoring method, system, server and storage medium based on deep learning | |
| CN111553326B (en) | Hand motion recognition method and device, electronic equipment and storage medium | |
| CN110569843B (en) | Intelligent detection and identification method for mine target | |
| Piciarelli et al. | Surveillance-oriented event detection in video streams | |
| CN111738218B (en) | Human body abnormal behavior recognition system and method | |
| CN110619324A (en) | Pedestrian and safety helmet detection method, device and system | |
| KR20190088087A (en) | method of providing categorized video processing for moving objects based on AI learning using moving information of objects | |
| Yandouzi et al. | Investigation of combining deep learning object recognition with drones for forest fire detection and monitoring | |
| US20230186634A1 (en) | Vision-based monitoring of site safety compliance based on worker re-identification and personal protective equipment classification | |
| CN111444850A (en) | Picture detection method and related device | |
| CN111931573A (en) | Helmet detection and early warning method based on YOLO evolution deep learning model | |
| CN116403162B (en) | Airport scene target behavior recognition method and system and electronic equipment | |
| CN115187884A (en) | High-altitude parabolic identification method and device, electronic equipment and storage medium | |
| CN116823884A (en) | Multi-target tracking method, system, computer equipment and storage medium | |
| CN111753601B (en) | Image processing method, device and storage medium | |
| CN115713806A (en) | Falling behavior identification method based on video classification and electronic equipment | |
| Tian et al. | Glance and stare: Trapping flying birds in aerial videos by adaptive deep spatio-temporal features | |
| Ali et al. | Real-time safety monitoring vision system for linemen in buckets using spatio-temporal inference | |
| CN116129523A (en) | Action recognition method, device, terminal and computer readable storage medium | |
| CN111191575B (en) | Naked flame detection method and system based on flame jumping modeling | |
| CN114663972A (en) | Target marking method and device based on motion difference |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |