CN115471916A - Smoking detection method, device, equipment and storage medium - Google Patents
Smoking detection method, device, equipment and storage medium Download PDFInfo
- Publication number
- CN115471916A CN115471916A CN202211148966.0A CN202211148966A CN115471916A CN 115471916 A CN115471916 A CN 115471916A CN 202211148966 A CN202211148966 A CN 202211148966A CN 115471916 A CN115471916 A CN 115471916A
- Authority
- CN
- China
- Prior art keywords
- change information
- determining
- position change
- smoking
- mouth
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
- G06V40/20—Movements or behaviour, e.g. gesture recognition
- G06V40/28—Recognition of hand or arm movements, e.g. recognition of deaf sign language
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V20/00—Scenes; Scene-specific elements
- G06V20/50—Context or environment of the image
- G06V20/52—Surveillance or monitoring of activities, e.g. for recognising suspicious objects
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
- G06V40/10—Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
- G06V40/16—Human faces, e.g. facial parts, sketches or expressions
- G06V40/161—Detection; Localisation; Normalisation
Landscapes
- Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Physics & Mathematics (AREA)
- Multimedia (AREA)
- Theoretical Computer Science (AREA)
- General Health & Medical Sciences (AREA)
- Human Computer Interaction (AREA)
- Health & Medical Sciences (AREA)
- Oral & Maxillofacial Surgery (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Psychiatry (AREA)
- Social Psychology (AREA)
- Image Analysis (AREA)
Abstract
The invention relates to the field of computer vision, in particular to a smoking detection method, a device, equipment and a storage medium, wherein the method comprises the following steps: acquiring a target image sequence; carrying out target detection on each target image in the target image sequence, and determining the position information of each target in the target image, wherein the target comprises a human body arm joint point, cigarettes and a mouth; determining whether the hand holds the cigarette or not based on the joint points of the human arms and the position of the cigarette; when the hand holds a cigarette, determining position change information between the hand and the mouth in the target image sequence based on the human arm joint points and the position information of the mouth; and determining the detection result of the smoking behavior according to the position change information. The method not only detects cigarettes, but also simultaneously detects whether smoking actions exist, judges whether smoking actions exist or not under the combination of the cigarettes and the smoking actions, and judges the smoking actions by adopting multiple modes, so that the accuracy of smoking action detection is improved.
Description
Technical Field
The invention relates to the field of computer vision, in particular to a smoking detection method, a smoking detection device, smoking detection equipment and a storage medium.
Background
In order to control fire hazards, maintain and improve the environment of the construction site, and ensure fire safety at the construction site, smoking behavior needs to be strictly avoided. In indoor scenarios, smoking behavior is typically detected by a device such as a smoke sensor, but smoke sensing devices are difficult to use in outdoor scenarios.
Whether take place the smoking incident of the most adoption ordinary camera or infrared camera among the prior art, catch the high temperature point through infrared camera, judge then whether there is the smoking action, but because infrared camera needs high resolution equipment, the price is expensive, and the cost that needs consume is great. If a common camera is adopted, a deep learning algorithm is combined to perform target detection on the picture, whether cigarettes exist in the picture is detected, and whether smoking behaviors exist is judged. Since the cigarette is a relatively small target, other objects of similar shape in the picture may be falsely detected as cigarettes by a general target detection method, and in addition, if someone takes out the cigarette but does not ignite and smoke, the cigarette may also be falsely detected by the general target detection method.
Disclosure of Invention
In view of this, embodiments of the present invention provide a smoking detection method, apparatus, device, and storage medium to solve the problem of low accuracy of smoking behavior detection.
According to a first aspect, an embodiment of the present invention provides a smoking detection method, including:
acquiring a target image sequence;
performing target detection on each target image in the target image sequence, and determining position information of each target in the target image, wherein the target comprises a human body arm joint point, a cigarette and a mouth;
determining whether a hand holds a cigarette or not based on the joint of the human arm and the position of the cigarette;
when the hand holds a cigarette, determining position change information between the hand and the mouth in the target image sequence based on the human arm joint points and the position information of the mouth;
and determining the detection result of the smoking behavior according to the position change information.
The smoking detection method provided by the embodiment of the invention obtains the target image sequence, detects the position information of the arm joint point, the cigarette and the mouth of the human body based on a plurality of target detection methods, judges whether the cigarette is held and judges whether smoking action or actions similar to smoking exist, and further determines whether smoking action exists. The method not only detects cigarettes, but also simultaneously detects whether smoking actions exist, judges whether smoking actions exist or not under the combination of the cigarettes and the smoking actions, and judges the smoking actions by adopting multiple modes, so that the accuracy of smoking action detection is improved.
In some embodiments, the determining the position change information between the hand and the mouth in the target image sequence based on the position information of the human arm joint point and the mouth includes:
for each of the target images, determining a distance between the hand and the mouth based on the human arm joint points and the position information of the mouth;
determining the position change information based on a temporal order of the target images and a distance between the hand and the mouth in each of the target images.
According to the smoking detection method provided by the embodiment of the invention, the distance between the hand and the mouth in each target image is determined, the position change information is determined based on the time sequence of the target images and the obtained distance between the hand and the mouth, and as the smoking action is dynamic, the information in a plurality of target images is combined for the subsequent judgment of the smoking action, so that the accuracy of smoking behavior detection is improved.
In some embodiments, said determining said position change information based on a temporal order of said target images and a distance between said hand and said mouth in each of said target images comprises:
determining first position change information based on the time sequence of the target images and the distance between the hand and the mouth in each of the target images;
for each of the target images, determining a bending angle of an elbow joint of an arm based on the human arm joint points;
determining second position change information based on the temporal sequence of the target images and the bend angle of the elbow joint.
In some embodiments, the determining a detection result of smoking behavior according to the location change information includes:
comparing first position change information in the position change information with a preset first threshold value, and comparing second position change information in the position change information with a preset first angle threshold value to determine a detection result of smoking action at a first stage;
comparing first position change information in the position change information with a preset second threshold value, and comparing second position change information in the position change information with a preset second angle threshold value to determine a detection result of second-stage smoking actions;
and determining the detection result of the smoking behavior according to the detection result of the first stage smoking action and the detection result of the second stage smoking action.
According to the smoking detection method provided by the embodiment of the invention, the first position change information and the second position change information in the position change information are respectively compared with the preset first threshold and the preset first angle threshold, so that whether the first stage smoking action exists is judged, and the first position change information and the second position change information are respectively compared with the preset second threshold and the preset second angle threshold, so that whether the second stage smoking action exists is judged. When judging whether smoking behavior exists, the smoking behavior is considered, so that comprehensive judgment needs to be carried out by combining the detection result of the first stage smoking behavior and the detection result of the second stage smoking behavior, and the accuracy of smoking behavior detection is improved.
In some embodiments, the comparing a first position change information of the position change information with a preset first threshold value and comparing a second position change information of the position change information with a preset first angle threshold value to determine a detection result of the smoking action in the first stage includes:
determining a preset first threshold according to the position information of the mouth;
and determining that a first stage smoking action exists when the first position change information is not greater than the preset first threshold and the second position change information is smaller than a preset first angle threshold within a preset time threshold.
In some embodiments, the comparing the first position change information in the position change information with a preset second threshold value, and comparing the second position change information in the position change information with a preset second angle threshold value to determine the detection result of the second stage smoking action includes:
determining a preset second threshold according to the human arm joint point;
and in a preset time threshold, when the first position change information is not smaller than the preset second threshold and the second position change information is smaller than a preset second angle threshold, determining that second-stage smoking actions exist.
In some embodiments, the performing object detection on each object image in the sequence of object images and determining position information of each object in the object images includes:
detecting the target image, and determining a human body and cigarettes in the target image;
extracting skeleton joint points of the human body, and determining arm joint points of the human body;
and detecting the human face of the human body and determining the position information of the mouth.
The smoking detection method provided by the embodiment of the invention simultaneously detects the human body and the cigarettes in the target image, and then detects the position information of the arm joint point and the mouth of the human body based on the detected human body, and simultaneously adopts various target detection methods, thereby improving the accuracy of target detection.
According to a second aspect, embodiments of the present invention provide a smoking detection apparatus, the apparatus comprising:
the image acquisition module is used for acquiring a target image sequence;
the first detection module is used for carrying out target detection on each target image in the target image sequence and determining the position information of each target in the target images, wherein the target comprises a human body arm joint point, a cigarette and a mouth;
the cigarette judging module is used for determining whether the hand holds a cigarette or not based on the joint points of the human arms and the positions of the cigarettes;
the second detection module is used for determining position change information between the hand and the mouth in the target image sequence based on the human body arm joint points and the position information of the mouth when the hand holds a cigarette;
and the result determining module is used for determining the detection result of the smoking behavior according to the position change information.
According to a third aspect, an embodiment of the present invention provides an electronic device, including: a memory and a processor, the memory and the processor being communicatively connected to each other, the memory storing therein computer instructions, and the processor executing the computer instructions to perform the smoking detection method according to the first aspect or any one of the embodiments of the first aspect.
According to a fourth aspect, the present invention provides a computer-readable storage medium storing computer instructions for causing a computer to perform the method of smoking detection as set forth in the first aspect or any one of the embodiments of the first aspect.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below, and it is obvious that the drawings in the following description are some embodiments of the present invention, and other drawings can be obtained by those skilled in the art without creative efforts.
FIG. 1 is a flow chart of a method of smoking detection according to an embodiment of the present invention;
FIG. 2 is a flow chart of a method of determining location change information according to an embodiment of the present invention;
FIG. 3 is a flow chart of a method of determining a detection result according to an embodiment of the invention;
FIG. 4 is a flow chart of a method of determining location information for targets according to an embodiment of the invention;
FIG. 5 is a schematic illustration of a method of smoking detection according to an embodiment of the present invention;
FIG. 6 is a flow chart of a method of smoking detection according to an embodiment of the present invention;
FIG. 7 is a schematic diagram of joint detection according to an embodiment of the present invention;
figure 8 is a schematic diagram of a smoking action according to an embodiment of the invention;
figure 9 is a schematic diagram of a smoking behaviour detection device according to an embodiment of the present invention;
fig. 10 is a schematic diagram of a hardware structure of an electronic device according to an embodiment of the present invention.
Detailed Description
In order to make the objects, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are some, but not all embodiments of the present invention. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
In accordance with an embodiment of the present invention, there is provided a smoking detection method embodiment, it is noted that the steps illustrated in the flowchart of the figure may be performed in a computer system, such as a set of computer-executable instructions, and that while a logical order is illustrated in the flowchart, in some cases the steps illustrated or described may be performed in an order different than here.
In an indoor scene, it is determined whether there is smoking behavior that can pass through the smoke detector device, however, in an outdoor scene, such as a worksite, where it is necessary to order prohibition of smoking, it is difficult to detect whether there is smoking behavior by the smoke detector device. The smoking detection method provided by the embodiment of the invention can be used in the scene that the smoking sensing equipment is inconvenient to use.
In this embodiment, a smoking detection method is provided, which can be used in devices such as a computer, a tablet computer, or a camera with artificial intelligence, and fig. 1 is a flowchart of the smoking detection method according to an embodiment of the present invention, and as shown in fig. 1, the flowchart includes the following steps:
and S11, acquiring a target image sequence.
The target image is an image of a place where whether smoking behavior exists or not needs to be detected, a video image can be acquired through a monitoring camera in the place, and the video data set comprises target images of continuous frames, namely a target image sequence.
And S12, carrying out target detection on each target image in the target image sequence, and determining the position information of each target in the target images, wherein the target comprises a human body arm joint point, a cigarette and a mouth.
The target image sequence comprises a plurality of target images, target detection is carried out on each target image, and targets needing to be confirmed comprise human arm joint points, cigarettes and mouths.
In order to determine the positions of the arm joint points and the mouth of the human body, the human body in the target image needs to be detected, joint point detection and face detection are further performed on the basis of the detected human body, and the coordinates of the arm joint points of the human body are determined through the joint point detection, wherein the arm joint points comprise a wrist joint, an elbow joint, a shoulder joint and the like. And determining the coordinates of the mouth through face detection.
The cigarette position in the target image is detected through a target detection algorithm, before that, whether a cigarette exists in the target image needs to be detected, specifically, a cigarette-shaped object in the target image can be detected based on a target detection model, and if the cigarette-shaped object is detected, the coordinates of the object are determined. Since there may be other cigarette-like objects, the present solution needs to determine whether there is smoking behavior in combination with whether there is smoking action, so the cigarettes in this step refer to all detected cigarette-like objects.
And S13, determining whether the hand holds the cigarette or not based on the joint points of the human arm and the position of the cigarette.
The positions of the human arm joints and the cigarettes are determined mainly by determining whether the hand holds the cigarette-shaped object according to the coordinates of the detected human arm joints and the coordinates of the detected cigarette-shaped object, and for example, the intersection ratio between the detection frame of the human arm joints and the cigarette detection frame may be set to be greater than a certain threshold. Alternatively, the distance between the cigarette and the hand may be calculated from the obtained coordinates of the human arm joint point and the coordinates of the cigarette, and if the calculated distance is less than a certain threshold value, it may be determined that the hand holds the cigarette.
And S14, when the hand holds the cigarette, determining position change information between the hand and the mouth in the target image sequence based on the human arm joint points and the position information of the mouth.
In S13, when it is determined that the hand does not hold a cigarette, it is basically determined that there is no smoking behavior, and it is possible to falsely detect an object having a shape similar to a cigarette as a cigarette.
When it is determined that the hand holds a cigarette, since it is not yet completely determined whether the detected cigarette is a real cigarette, there is a possibility that the hand holds an object having a shape similar to a cigarette, and therefore, it is not possible to easily determine whether there is a smoking behavior and it is also necessary to determine whether there is a smoking action. The determination of the smoking action requires a plurality of target images to be combined and a comprehensive determination to be made by combining the information of the position change between the hand and the mouth. Wherein the position change information includes a distance between the hand and the mouth, a bending angle of the elbow joint, and the like. Firstly, the distance relationship between the hand and the mouth needs to be determined, and the positions of the joint points passing through the arm of the human body comprise the positions of a wrist joint, an elbow joint, a shoulder joint, a hand joint and the like. The distance between the hand and the mouth can be calculated according to the coordinates of the hand joints and the coordinates of the mouth, and the bending angle of the elbow joint can be determined according to the arm joint points. Since the smoking action is dynamically continuous, it is necessary to combine the positional change information between the hand and the mouth in a plurality of target images. The position change information includes a change in the distance between the hand and the mouth and a change in the bending angle of the elbow joint.
And comparing the distance between the hand and the mouth, the elbow joint bending angle with a set distance threshold between the hand and the mouth and an elbow joint bending threshold, and judging whether the situations that the hand is far away from the mouth, the hand is close to the mouth, the elbow is bent within and outside a certain angle and the like exist in the target image sequence according to the time sequence.
And S15, determining the detection result of the smoking behavior according to the position change information.
The distance threshold between the hand and the mouth, the elbow joint bending angle threshold and the like can be set according to actual conditions, and when the distance between the hand and the mouth is smaller than a certain distance threshold and the elbow joint bending angle is within a certain elbow joint bending angle threshold, the hand is judged to move close to the mouth. When the distance between the hand and the mouth is larger than a certain distance threshold value and the elbow joint bending angle is not equal to or larger than a certain elbow joint bending angle, judging that the hand moves away from the mouth.
The smoking action includes that the hand is close to the mouth and gradually leaves away from the mouth, if the hand is judged to be close to the mouth and gradually leaves away from the mouth more than a certain number of times within a certain time, the smoking action or the action similar to smoking can be basically judged to exist. Since the presence of a hand holding a cigarette-like object has been detected in combination with the detection of the presence of smoking or a smoking-like action, it can basically be determined that smoking behavior is present. If there is no smoking action or action similar to smoking, it can be determined that there is no smoking activity within the range of the target image sequence. If the smoking behavior is judged to exist, an alarm prompt can be sent out, so that related personnel can process the smoking behavior in time, and the safety of a place is guaranteed.
The smoking detection method provided by the embodiment of the invention obtains the target image sequence, detects the position information of the joint point of the arm, the cigarette and the mouth of the human body based on a plurality of target detection methods, judges whether the cigarette is held and judges whether smoking actions or actions similar to smoking actions exist or not, and further determines whether smoking actions exist or not. The method not only detects cigarettes, but also simultaneously detects whether smoking actions exist, and judges the detection result of the smoking actions under the combination of the two, and adopts multiple modes to judge the smoking actions, so that the accuracy of smoking action detection is improved.
In this embodiment, a method for determining location change information is provided, corresponding to S14 in fig. 1, and may be used in a device such as a computer, a tablet computer, or a camera with artificial intelligence capability, and fig. 2 is a flowchart of a method for determining location change information according to an embodiment of the present invention, as shown in fig. 2, where the flowchart includes the following steps:
and S21, determining the distance between the hand and the mouth based on the joint points of the arms of the human body and the position information of the mouth for each target image.
Through various target detections on the target image, the position information of the human arm joint point and the mouth position information in the target image are determined, and the position information comprises coordinate information. The human arm joint points comprise wrist joints, elbow joints, shoulder joints, hand joints and the like, and the distance between the hand and the mouth is calculated according to the coordinates of the hand joints and the coordinates of the mouth. In addition, the bending angle of the elbow joint can be calculated by combining the joint points of each arm.
And S22, determining position change information based on the time sequence of the target images and the distance between the hand and the mouth in each target image.
The position change information includes the change of the distance between the hand and the mouth and the change of the elbow joint bending angle. Since the smoking action is dynamically changed, when determining whether or not there is a smoking action, it is necessary to determine whether or not the positional change information matches the change of the smoking action within a certain time by combining the detection results of the plurality of target images, that is, by combining the time sequence of the target images, for example, the distance between the hand and the mouth is within a certain preset threshold value, the bending angle of the elbow joint is smaller than a certain threshold value, the distance between the hand and the mouth is gradually increased with time, and the bending angle of the elbow joint is larger than a certain threshold value. Specifically, the step of determining the position change information is as follows:
s221, first position change information is determined based on the time sequence of the target images and the distance between the hand and the mouth in each target image.
The first position change information includes a distance change between the hand and the mouth, and the distance between the hand and the mouth is recorded in the time-series target image, so that the first position change information can be obtained.
S222, for each target image, determining the bending angle of the elbow joint of the arm based on the joint point of the human arm.
The human arm joint points comprise a wrist joint, an elbow joint, a shoulder joint and the like, and because the coordinates of all joints on the arm are determined, the bending angle of the elbow joint in each target image can be calculated according to the coordinates of all the joints.
S223, second position change information is determined based on the time sequence of the target images and the bending angle of the elbow joint of the arm.
The second position change information includes a change in a bending angle of the elbow joint, and the second position change information is obtained by recording the bending angle of the elbow joint in the time-series target images.
According to the smoking detection method provided by the embodiment of the invention, the distance between the hand and the mouth in each target image is determined, the position change information is determined based on the time sequence of the target images and the obtained distance between the hand and the mouth, and as the smoking action is dynamic, the information in a plurality of target images is combined for the subsequent judgment of the smoking action, so that the accuracy of smoking behavior detection is improved.
In this embodiment, a method for determining a detection result is provided, which corresponds to S15 in fig. 1, and can be used for devices such as a computer, a tablet computer, or a camera with artificial intelligence, and fig. 3 is a flowchart of a method for determining a detection result according to an embodiment of the present invention, as shown in fig. 3, where the flowchart includes the following steps:
and S31, comparing the first position change information in the position change information with a preset first threshold, and comparing the second position change information in the position change information with a preset first angle threshold, so as to determine the detection result of the smoking action in the first stage.
The first position change information comprises the change situation of the distance between the hand and the mouth in a certain time range, and the preset first threshold is a distance threshold which is set according to the actual situation and is used for judging whether the hand and the mouth are close to each other. The second position change information comprises the change condition of the bending angle of the elbow joint of the arm in a certain time range, and the preset first angle threshold is an angle threshold which is set according to the actual condition and is used for assisting in judging whether the hand is close to the mouth or not. By comparing the first position change information with a preset first threshold value and comparing the second position change information with a preset first angle threshold value, if the distance between the hand and the mouth is not greater than the preset first threshold value within a certain time and the bending angle of the elbow joint of the arm is less than the preset first angle threshold value, it can be determined that there is a first stage smoking action, i.e., there is an action that the hand is close to the mouth.
Specifically, S31 includes the steps of:
and S311, determining a preset first threshold according to the position information of the mouth.
The position information of the mouth includes the position of the left mouth corner and the position of the right mouth corner, and the distance from the left mouth corner to the right mouth corner can be obtained according to the position of the left mouth corner and the position of the right mouth corner. The preset first threshold is set according to an actual situation, and in consideration of individual differences, in this embodiment, the distance from the left mouth corner to the right mouth corner is used as the preset first threshold, and the preset first threshold may also be set in other manners in actual application, which is not limited specifically.
And S312, determining that the first stage smoking action exists when the first position change information is not greater than a preset first threshold and the second position change information is smaller than a preset first angle threshold within a preset time threshold.
When the distance between the hand and the mouth is not larger than a preset first threshold value and the bending angle of the elbow joint of the arm is smaller than a preset first angle threshold value, it is determined that a first stage smoking action exists, when the first stage smoking action is smoking, the hand is close to the mouth, and the arm is bent and within a preset time threshold value (for example, lasting for 3 seconds and more).
And S32, comparing the first position change information in the position change information with a preset second threshold, and comparing the second position change information in the position change information with a preset second angle threshold to determine the detection result of the second stage smoking action.
The first position change information comprises the change situation of the distance between the hand and the mouth in a certain time range, and the preset second threshold is a distance threshold which is set according to the actual situation and is used for judging whether the hand and the mouth are far away. The second position change information comprises the change condition of the bending angle of the elbow joint of the arm in a certain time range, and the preset second angle threshold is an angle threshold which is set according to the actual condition and is used for assisting in judging whether the hand is far away from the mouth or not. By comparing the first position change information with a preset second threshold value and comparing the second position change information with a preset second angle threshold value, if the distance between the hand and the mouth is not less than the preset second threshold value within a certain time and the bending angle of the elbow joint of the arm is greater than the preset first angle threshold value, it can be determined that the second stage smoking action, namely the action that the hand is far away from the mouth, exists.
Specifically, S32 includes the steps of:
and S321, determining a preset second threshold according to the human arm joint point.
The position information of the human arm joint point includes the position of the left shoulder and the position of the right shoulder, and the distance from the left shoulder to the right shoulder can be obtained according to the position of the left shoulder and the position of the right shoulder. The preset second threshold is set according to an actual situation, and in consideration of individual differences, in this embodiment, the distance from the left shoulder to the right shoulder is used as the preset second threshold, and the preset second threshold may also be set in other manners in actual application, which is not limited specifically.
And S322, determining that the second stage smoking action exists when the first position change information is not less than a preset second threshold and the second position change information is less than a preset second angle threshold within a preset time threshold.
When the distance between the hand and the mouth is not less than a preset second threshold value and the bending angle of the elbow joint of the arm is greater than a preset second angle threshold value, it is determined that a second stage smoking action exists, and after the second stage smoking action, namely smoking, the hand is far away from the mouth and within a preset time threshold value (for example, lasting for 3 seconds or more).
And S33, determining the detection result of the smoking behavior according to the detection result of the first stage smoking action and the detection result of the second stage smoking action.
When the detection result of the first stage smoking action is that the first stage smoking action is determined to exist, and the detection result of the second stage smoking action is that the second stage smoking action is determined to exist, namely that the smoking action that the hand part is close to the mouth and the action that the hand part is far away from the mouth are determined to exist in a certain time range, if the periodic first stage smoking action and the second stage smoking action which are more than a certain number of times are judged to exist, the existence of the smoking action can be determined by combining the detection of the cigarette-shaped object.
The detection result of the smoking behavior also comprises that no smoking behavior exists, if the first stage smoking action does not exist, the second stage smoking action does not exist, or both the first stage smoking action and the second stage smoking action do not exist, the smoking behavior does not exist, and when the smoking behavior does not exist, the smoking behavior does not exist.
According to the smoking detection method provided by the embodiment of the invention, the first position change information and the second position change information in the position change information are respectively compared with the preset first threshold and the preset first angle threshold, so that whether the first stage smoking action exists is judged, and the first position change information and the second position change information are respectively compared with the preset second threshold and the preset second angle threshold, so that whether the second stage smoking action exists is judged. When judging whether smoking behavior exists, the smoking behavior is considered, so that comprehensive judgment needs to be carried out by combining the detection result of the first stage smoking behavior and the detection result of the second stage smoking behavior, and the accuracy of smoking behavior detection is improved.
In the present embodiment, a method for determining location information of each object is provided, and corresponding to S12 in fig. 1, fig. 4 is a flowchart of the method for determining location information of each object according to the embodiment of the present invention, as shown in fig. 4, the flowchart includes the following steps:
s41, detecting the target image and determining the human body and the cigarettes in the target image.
In the process of detecting the target image, different targets can be detected by adopting a plurality of target detection models, for example, a human body in the target image can be detected by a human body detection model, namely, the human body is detected in a picture or a video stream by machine learning or deep learning, and the position of the human body is marked. The cigarette-shaped object in the target image is detected through the target detection model, in order to avoid false detection, the cigarette detected in the step is the cigarette-shaped object, whether the cigarette is not directly determined, and the comprehensive judgment needs to be carried out by combining the smoking action subsequently.
And S42, extracting the skeletal joint points of the human body, and determining the arm joint points of the human body.
After the human body is detected, the human body skeleton joint points can be extracted through the deep learning model so as to determine the joint points of the arm, and the joint points are connected according to the human body structure by using the connecting line.
S43, detecting the human face of the human body and determining the position information of the mouth.
After detecting the human body, the human face detection may be performed based on the detected human body, for example, the detection is performed through a human face key point model, and the position of the five sense organs is detected, that is, the position information of the mouth may be determined, and the position information of the mouth may include the coordinates of the left mouth corner, the coordinates of the right mouth corner, and the like.
The smoking detection method provided by the embodiment of the invention simultaneously detects the human body and the cigarettes in the target image, and then detects the position information of the arm joint point and the mouth of the human body based on the detected human body, and simultaneously adopts various target detection methods, thereby improving the accuracy of target detection.
In this embodiment, a smoking detection method is provided, which may be based on a hardware combination of a camera and an edge box with artificial intelligence computing power, as shown in fig. 5, or based on a camera with artificial intelligence computing power, and a flowchart of the method is shown in fig. 6. When a camera video stream is read into the device, it is broken down into a sequence of images. And determining each target in the target image by the image sequence through a plurality of target detection models. And detecting the human body in the target image through a human body detection model, such as a Yolov5 model, based on a machine learning or deep learning method, and marking the position of the human body. Based on the detected human body, human body joint points, such as openposition model, are extracted by using human body skeleton joint point detection model, and after the human body joint points are detected, the joint points are connected according to the human body structure by using connecting lines, as shown in fig. 7. The face detection is carried out based on the detected human body, or the face detection can be directly carried out, and a face key point model is adopted to detect the left eye, the right eye, the nose, the left mouth corner, the right mouth corner and the like, such as a Retinaface model. A cigarette-like object, such as a mobilesd or Yolo series (v 3, v4 or v 5) model, in the target image is detected using a target detection model. It should be noted that in this embodiment, several target detections may be used in parallel or in series. And judging whether smoking action exists or not through the detected left mouth corner, the detected right mouth corner and the detected human arm joint points. And calculating the distance between the midpoint of the left mouth corner and the right mouth corner in each target image and the hand, defining the distance from the left mouth corner to the right mouth corner as a preset first threshold value, and defining the distance from the left shoulder to the right shoulder as a preset second threshold value.
When the distance from the hand to the mouth is not more than the preset first threshold value and the bending angle of the elbow joint is less than the preset first angle threshold value during smoking, the motion is kept for a certain time (for example, more than 3 seconds), and then the existence of the first stage smoking motion can be judged. And then, gradually keeping the hand away from the mouth, and judging that the second stage smoking action exists when the distance between the hand and the mouth is not less than a preset second threshold value and the bending angle of the elbow joint is greater than a preset second angle threshold value and the action is kept for a certain time (for example, more than 3 seconds). Then, a periodic determination is made, and if the first stage smoking action and the second stage smoking action are repeated a set number of times within a certain period, it is determined that there is a smoking action, and the action diagram is shown in fig. 8. In this embodiment, the preset first angle threshold is set to 30 degrees, and the preset second angle threshold is set to 150 degrees. If a cigarette-shaped object is detected to exist in the target image and the intersection ratio of the detection frame of the cigarette-shaped object and the human body detection frame is greater than a certain value, in this embodiment, the threshold value of the intersection ratio is set to be 0.8, if a smoking action is determined to exist on the premise, and periodic determination is performed, that is, the smoking action is repeated for more than a certain number of times (for example, more than 3 times) within a set time period, after comprehensive determination, the smoking action is completely determined to exist, otherwise, the smoking action is not determined to exist. If the smoking behavior is judged to exist, an alarm prompt can be sent out so as to ensure the safety of the monitored place and avoid the occurrence of fire.
According to the smoking detection method based on the video or picture sequence, motion semantic analysis and target capture are performed through video frame and target detection, meanwhile, human body detection, human body joint point detection, target detection and human face key point detection are used for combining algorithm flows, an algorithm is constructed according to the characteristics of smoking motions to judge the smoking motions, and various judging mechanisms are adopted in combination with cigarette detection.
In this embodiment, a smoking behavior detection device is further provided, and the device is used to implement the above embodiments and implementation manners, which have already been described and will not be described again. As used below, the term "module" may be a combination of software and/or hardware that implements a predetermined function. Although the means described in the embodiments below are preferably implemented in software, an implementation in hardware, or a combination of software and hardware is also possible and contemplated.
The present embodiment provides a smoking behavior detection apparatus, as shown in fig. 9, including:
an image acquisition module 51 for acquiring a target image sequence;
a first detection module 52, configured to perform target detection on each target image in the target image sequence, and determine position information of each target in the target image, where the target includes a human arm joint point, a cigarette, and a mouth;
a cigarette judging module 53, configured to determine whether a hand holds a cigarette based on the joint points of the human arm and the position of the cigarette;
a second detecting module 54, configured to determine, when the hand holds a cigarette, position change information between the hand and the mouth in the target image sequence based on the human arm joint point and the position information of the mouth;
and the result determining module 55 is used for determining the detection result of the smoking behavior according to the position change information.
In some embodiments, the second detection module 54 includes:
a distance determination unit configured to determine, for each of the target images, a distance between the hand and the mouth based on the human arm joint point and the position information of the mouth;
a position change information determination unit configured to determine the position change information based on a time sequence of the target images and a distance between the hand and the mouth in each of the target images.
In some embodiments, the location change information determination unit includes:
a first position change information subunit configured to determine first position change information based on a time order of the target images and a distance between the hand and the mouth in each of the target images;
a bending angle determining subunit, configured to determine, for each of the target images, a bending angle of an elbow joint of an arm based on the human arm joint point;
a second position change information subunit operable to determine second position change information based on the time sequence of the target images and the bending angle of the elbow joint.
In some embodiments, the result determination module 55 includes:
the first result determining unit is used for comparing first position change information in the position change information with a preset first threshold value, comparing second position change information in the position change information with a preset first angle threshold value, and determining a detection result of smoking action in a first stage;
the second result determining unit is used for comparing first position change information in the position change information with a preset second threshold value, comparing second position change information in the position change information with a preset second angle threshold value, and determining a detection result of second-stage smoking actions;
and the detection result determining unit is used for determining the detection result of the smoking behavior according to the detection result of the first stage smoking action and the detection result of the second stage smoking action.
In some embodiments, the first result determination unit comprises:
a first threshold determining subunit, configured to determine a preset first threshold according to the position information of the mouth;
and the first action determining subunit is used for determining that a first-stage smoking action exists when the first position change information is not greater than the preset first threshold and the second position change information is smaller than a preset first angle threshold within a preset time threshold.
In some embodiments, the second result determination unit comprises:
the second threshold determining subunit is used for determining a preset second threshold according to the human arm joint point;
and the second action determining subunit is used for determining that second-stage smoking actions exist when the first position change information is not smaller than the preset second threshold and the second position change information is smaller than a preset second angle threshold within a preset time threshold.
In some embodiments, the first detection module 52 includes:
the target detection unit is used for detecting the target image and determining a human body and cigarettes in the target image;
the joint point determining unit is used for extracting skeleton joint points of the human body and determining arm joint points of the human body;
and the position information determining unit is used for detecting the human face of the human body and determining the position information of the mouth.
The smoking behaviour detection means in this embodiment is in the form of a functional unit, where the unit refers to an ASIC circuit, a processor and memory executing one or more software or fixed programs, and/or other devices that can provide the above-described functionality.
Further functional descriptions of the modules are the same as those of the corresponding embodiments, and are not repeated herein.
An embodiment of the present invention further provides an electronic device, which includes the smoking behavior detection apparatus shown in fig. 9.
Referring to fig. 10, fig. 10 is a schematic structural diagram of an electronic device according to an embodiment of the present invention, as shown in fig. 10, the electronic device may include: at least one processor 601, such as a CPU (Central Processing Unit), at least one communication interface 603, memory 604, and at least one communication bus 602. Wherein the communication bus 602 is used to enable connection communication between these components. The communication interface 603 may include a Display (Display) and a Keyboard (Keyboard), and the optional communication interface 603 may also include a standard wired interface and a standard wireless interface. The Memory 604 may be a high-speed RAM (Random Access Memory) or a non-volatile Memory (non-volatile Memory), such as at least one disk Memory. The memory 604 may optionally be at least one storage device located remotely from the processor 601. Wherein the processor 601 may be in connection with the apparatus described in fig. 9, an application program is stored in the memory 604, and the processor 601 calls the program code stored in the memory 604 for performing any of the above-mentioned method steps.
The communication bus 602 may be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus. The communication bus 602 may be divided into an address bus, a data bus, a control bus, etc. For ease of illustration, only one thick line is shown in FIG. 10, but this is not intended to represent only one bus or type of bus.
The memory 604 may include a volatile memory (RAM), such as a random-access memory (RAM); the memory may also include a non-volatile memory (e.g., flash memory), a hard disk (HDD) or a solid-state drive (SSD); the memory 604 may also comprise a combination of the above types of memory.
The processor 601 may be a Central Processing Unit (CPU), a Network Processor (NP), or a combination of a CPU and an NP.
The processor 601 may further include a hardware chip. The hardware chip may be an application-specific integrated circuit (ASIC), a Programmable Logic Device (PLD), or a combination thereof. The PLD may be a Complex Programmable Logic Device (CPLD), a field-programmable gate array (FPGA), a General Array Logic (GAL), or any combination thereof.
Optionally, memory 604 is also used to store program instructions. The processor 601 may call program instructions to implement the smoking detection method as shown in the embodiments of the present application.
Embodiments of the present invention further provide a non-transitory computer storage medium, where computer-executable instructions are stored, and the computer-executable instructions may execute the smoking detection method in any of the above method embodiments. The storage medium may be a magnetic Disk, an optical Disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), a Flash Memory (Flash Memory), a Hard Disk (Hard Disk Drive, abbreviated as HDD), a Solid State Drive (SSD), or the like; the storage medium may also comprise a combination of memories of the kind described above.
Although the embodiments of the present invention have been described in conjunction with the accompanying drawings, those skilled in the art may make various modifications and variations without departing from the spirit and scope of the invention, and such modifications and variations fall within the scope defined by the appended claims.
Claims (10)
1. A method of smoking detection, comprising:
acquiring a target image sequence;
performing target detection on each target image in the target image sequence, and determining position information of each target in the target image, wherein the target comprises a human body arm joint point, a cigarette and a mouth;
determining whether the hand holds the cigarette or not based on the joint points of the human arm and the position of the cigarette;
when the hand holds a cigarette, determining position change information between the hand and the mouth in the target image sequence based on the human arm joint points and the position information of the mouth;
and determining the detection result of the smoking behavior according to the position change information.
2. The method of claim 1, wherein the determining position change information between a hand and the mouth in the target image sequence based on the human arm joint point and the position information of the mouth comprises:
for each of the target images, determining a distance between the hand and the mouth based on the human arm joint points and the position information of the mouth;
determining the position change information based on a temporal order of the target images and a distance between the hand and the mouth in each of the target images.
3. The method of claim 2, wherein determining the position change information based on the temporal order of the target images and the distance between the hand and the mouth in each of the target images comprises:
determining first position change information based on the time sequence of the target images and the distance between the hand and the mouth in each of the target images;
for each of the target images, determining a bending angle of an elbow joint of an arm based on the human arm joint points;
determining second position change information based on the temporal sequence of the target images and the bend angle of the elbow joint.
4. The method of claim 1, wherein determining the detection result of smoking behavior according to the location change information comprises:
comparing first position change information in the position change information with a preset first threshold value, and comparing second position change information in the position change information with a preset first angle threshold value to determine a detection result of smoking action at a first stage;
comparing first position change information in the position change information with a preset second threshold value, and comparing second position change information in the position change information with a preset second angle threshold value to determine a detection result of second-stage smoking actions;
and determining the detection result of the smoking behavior according to the detection result of the first stage smoking action and the detection result of the second stage smoking action.
5. The method according to claim 4, wherein the comparing the first position change information in the position change information with a preset first threshold value and the comparing the second position change information in the position change information with a preset first angle threshold value to determine the detection result of the first stage smoking action comprises:
determining a preset first threshold according to the position information of the mouth;
and determining that a first stage smoking action exists when the first position change information is not greater than the preset first threshold and the second position change information is smaller than a preset first angle threshold within a preset time threshold.
6. The method according to claim 4, wherein the comparing the first position change information in the position change information with a preset second threshold value and the comparing the second position change information in the position change information with a preset second angle threshold value to determine the detection result of the second stage smoking action comprises:
determining a preset second threshold according to the human arm joint point;
and in a preset time threshold, when the first position change information is not smaller than the preset second threshold and the second position change information is smaller than a preset second angle threshold, determining that second-stage smoking actions exist.
7. The method according to any one of claims 1-6, wherein the performing object detection on each object image in the sequence of object images and determining position information of each object in the object images comprises:
detecting the target image, and determining a human body and cigarettes in the target image;
extracting skeleton joint points of the human body, and determining arm joint points of the human body;
and detecting the human face of the human body and determining the position information of the mouth.
8. A smoking detection device, the device comprising:
the image acquisition module is used for acquiring a target image sequence;
the first detection module is used for carrying out target detection on each target image in the target image sequence and determining the position information of each target in the target images, wherein the target comprises a human body arm joint point, a cigarette and a mouth;
the cigarette judging module is used for determining whether the hand holds a cigarette or not based on the joint points of the human arm and the position of the cigarette;
the second detection module is used for determining position change information between the hand and the mouth in the target image sequence based on the human body arm joint points and the position information of the mouth when the hand holds a cigarette;
and the result determining module is used for determining the detection result of the smoking behavior according to the position change information.
9. An electronic device, comprising:
a memory and a processor, the memory and the processor being communicatively connected to each other, the memory having stored therein computer instructions, the processor executing the computer instructions to perform the smoking detection method according to any one of claims 1 to 7.
10. A computer-readable storage medium having stored thereon computer instructions for causing a computer to perform the smoking detection method of any one of claims 1-7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202211148966.0A CN115471916A (en) | 2022-09-21 | 2022-09-21 | Smoking detection method, device, equipment and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202211148966.0A CN115471916A (en) | 2022-09-21 | 2022-09-21 | Smoking detection method, device, equipment and storage medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN115471916A true CN115471916A (en) | 2022-12-13 |
Family
ID=84334462
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202211148966.0A Pending CN115471916A (en) | 2022-09-21 | 2022-09-21 | Smoking detection method, device, equipment and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN115471916A (en) |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN116884034A (en) * | 2023-07-10 | 2023-10-13 | 中电金信软件有限公司 | Object identification method and device |
| CN117409484A (en) * | 2023-12-14 | 2024-01-16 | 四川汉唐云分布式存储技术有限公司 | Cloud-guard-based client offence detection method, device and storage medium |
-
2022
- 2022-09-21 CN CN202211148966.0A patent/CN115471916A/en active Pending
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN116884034A (en) * | 2023-07-10 | 2023-10-13 | 中电金信软件有限公司 | Object identification method and device |
| CN117409484A (en) * | 2023-12-14 | 2024-01-16 | 四川汉唐云分布式存储技术有限公司 | Cloud-guard-based client offence detection method, device and storage medium |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN109508688B (en) | Skeleton-based behavior detection method, terminal equipment and computer storage medium | |
| CN115471916A (en) | Smoking detection method, device, equipment and storage medium | |
| CN110659397B (en) | Behavior detection method and device, electronic equipment and storage medium | |
| CN108446669B (en) | Motion recognition method, motion recognition device and storage medium | |
| CN111126153B (en) | Safety monitoring method, system, server and storage medium based on deep learning | |
| CN109887234B (en) | Method and device for preventing children from getting lost, electronic equipment and storage medium | |
| CN112506340B (en) | Equipment control method, device, electronic equipment and storage medium | |
| CN110969045B (en) | Behavior detection method and device, electronic equipment and storage medium | |
| CN113792700A (en) | Storage battery car boxing detection method and device, computer equipment and storage medium | |
| CN112084882A (en) | Behavior detection method and device and computer readable storage medium | |
| CN112818796A (en) | Intelligent posture discrimination method and storage device suitable for online invigilation scene | |
| CN116524435A (en) | Online invigilation method based on electronic fence and related equipment | |
| WO2022091577A1 (en) | Information processing device and information processing method | |
| JP6991045B2 (en) | Image processing device, control method of image processing device | |
| JP6819689B2 (en) | Image processing equipment, stagnant object tracking system, image processing method and recording medium | |
| CN108108709B (en) | Identification method and device and computer storage medium | |
| CN113361469A (en) | Method, device and equipment for identifying coverage state and storage medium | |
| CN113314230A (en) | Intelligent epidemic prevention method, device, equipment and storage medium based on big data | |
| JP2021007055A (en) | Discriminator learning device, discriminator learning method, and computer program | |
| CN113660455B (en) | Method, system and terminal for fall detection based on DVS data | |
| CN116580432A (en) | Online examination monitoring method, system, computing device and storage medium | |
| CN112818929B (en) | Method and device for detecting people fighting, electronic equipment and storage medium | |
| CN114120368A (en) | Target detection method and detection equipment | |
| CN109523668B (en) | Intelligent access control method and device and computing equipment | |
| CN113705284A (en) | Climbing identification method and device and camera |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |