CN115471916A - Smoking detection method, device, equipment and storage medium - Google Patents

Abstract

The invention relates to the field of computer vision, in particular to a smoking detection method, a device, equipment and a storage medium, wherein the method comprises the following steps: acquiring a target image sequence; carrying out target detection on each target image in the target image sequence, and determining the position information of each target in the target image, wherein the target comprises a human body arm joint point, cigarettes and a mouth; determining whether the hand holds the cigarette or not based on the joint points of the human arms and the position of the cigarette; when the hand holds a cigarette, determining position change information between the hand and the mouth in the target image sequence based on the human arm joint points and the position information of the mouth; and determining the detection result of the smoking behavior according to the position change information. The method not only detects cigarettes, but also simultaneously detects whether smoking actions exist, judges whether smoking actions exist or not under the combination of the cigarettes and the smoking actions, and judges the smoking actions by adopting multiple modes, so that the accuracy of smoking action detection is improved.

Description

Smoking detection method, device, equipment and storage medium

technical field

The invention relates to the field of computer vision, in particular to a smoking detection method, device, equipment and storage medium.

Background technique

In order to control fire hazards, maintain and improve the environment at the construction site, and ensure fire safety at the construction site, it is necessary to strictly avoid smoking. In indoor scenes, smoking behavior is usually detected by devices such as smoke sensors, but smoke detection devices are difficult to use in outdoor scenes.

Most of the existing technologies use ordinary cameras or infrared cameras to judge whether smoking incidents occur, capture high-temperature points through infrared cameras, and then judge whether there is smoking behavior, but because infrared cameras require high-resolution equipment and are expensive, the cost required is relatively high. big. If an ordinary camera is used, it is necessary to combine the deep learning algorithm to detect the target of the picture, detect whether there is a cigarette in the picture, and then judge whether there is smoking behavior. Since cigarettes are relatively small objects, other objects with similar shapes in the picture may be misdetected as cigarettes by ordinary object detection methods. check.

Contents of the invention

In view of this, embodiments of the present invention provide a smoking detection method, device, device, and storage medium to solve the problem of low accuracy in smoking behavior detection.

According to the first aspect, an embodiment of the present invention provides a smoking detection method, including:

Obtain the target image sequence;

Perform target detection on each target image in the target image sequence, and determine the position information of each target in the target image, and the target includes human arm joints, cigarettes and mouth;

Based on the positions of the human arm joints and the cigarette, determine whether the hand holds the cigarette;

When the hand holds a cigarette, based on the position information of the human arm joints and the mouth, determine the position change information between the hand and the mouth in the target image sequence;

The detection result of the smoking behavior is determined according to the position change information.

The smoking detection method provided by the embodiment of the present invention obtains the target image sequence, detects the position information of the joints of the human arm, the cigarette and the mouth based on various target detection methods, judges whether a cigarette is held, and judges whether there is a smoking action or an action similar to smoking , and then determine whether there is smoking behavior. The method not only detects cigarettes, but also detects whether there is a smoking action at the same time, and then judges whether there is a smoking behavior after combining the two, and uses multi-modes to distinguish the smoking behavior, which improves the accuracy of the smoking behavior detection.

In some embodiments, the determining the position change information between the hand and the mouth in the target image sequence based on the joint points of the human arm and the position information of the mouth includes:

For each of the target images, based on the position information of the human arm joints and the mouth, determine the distance between the hand and the mouth;

The position change information is determined based on the time sequence of the target images and the distance between the hand and the mouth in each of the target images.

The smoking detection method provided by the embodiment of the present invention determines the distance between the hand and the mouth in each target image, and determines the position change information based on the time sequence of the target image and the obtained distance between the hand and the mouth. It is dynamic, so the information in multiple target images is combined for subsequent judgment of smoking actions, which improves the accuracy of smoking behavior detection.

In some implementation manners, the determining the position change information based on the time sequence of the target images and the distance between the hand and the mouth in each target image includes:

determining first position change information based on the time sequence of the target images and the distance between the hand and the mouth in each of the target images;

For each of the target images, determine the bending angle of the elbow joint of the arm based on the joint points of the human arm;

The second position change information is determined based on the time sequence of the target images and the bending angle of the elbow joint of the arm.

In some embodiments, the determination of the detection result of the smoking behavior according to the position change information includes:

comparing the first position change information in the position change information with a preset first threshold, and comparing the second position change information in the position change information with a preset first angle threshold to determine the first stage The detection result of smoking action;

comparing the first position change information in the position change information with a preset second threshold, and comparing the second position change information in the position change information with a preset second angle threshold to determine the second stage The detection result of smoking action;

The detection result of smoking behavior is determined according to the detection result of the first-stage smoking action and the detection result of the second-stage smoking action.

The smoking detection method provided by the embodiment of the present invention compares the first position change information and the second position change information in the position change information with the preset first threshold and the preset first angle threshold respectively, so as to determine whether there is a first For the staged smoking action, the first position change information and the second position change information are compared with the preset second threshold and the preset second angle threshold respectively, so as to determine whether there is a second-stage smoking action. When judging whether there is smoking behavior, since the smoking action is considered, it is necessary to combine the detection results of the first-stage smoking action and the detection results of the second-stage smoking action to make a comprehensive judgment, which improves the accuracy of smoking behavior detection.

In some embodiments, the first position change information in the position change information is compared with a preset first threshold, and the second position change information in the position change information is compared with a preset first angle Thresholds are compared to determine the detection results of the first-stage smoking action, including:

determining a preset first threshold according to the position information of the mouth;

Within the preset time threshold, when the first position change information is not greater than the preset first threshold and the second position change information is smaller than the preset first angle threshold, it is determined that there is a first-stage smoking action.

In some embodiments, the first position change information in the position change information is compared with a preset second threshold, and the second position change information in the position change information is compared with a preset second angle Thresholds are compared to determine the detection results of the second-stage smoking action, including:

determining a preset second threshold according to the joint points of the human arm;

Within the preset time threshold, when the first position change information is not less than the preset second threshold and the second position change information is smaller than the preset second angle threshold, it is determined that there is a second-stage smoking action.

In some implementation manners, the performing target detection on each target image in the target image sequence, and determining the position information of each target in the target image includes:

Detecting the target image to determine the human body and cigarettes in the target image;

Extracting bone joint points from the human body to determine the arm joint points of the human body;

The face of the human body is detected to determine the position information of the mouth.

The smoking detection method provided by the embodiment of the present invention detects the human body and the cigarette in the target image at the same time, and then detects the position information of the human arm joints and the mouth based on the detection of the human body. accuracy.

According to the second aspect, an embodiment of the present invention provides a smoking detection device, the device comprising:

An image acquisition module, configured to acquire a target image sequence;

The first detection module is used to perform target detection on each target image in the target image sequence, and determine the position information of each target in the target image, and the target includes human arm joints, cigarettes and mouths;

A cigarette judging module, configured to determine whether the hand holds a cigarette based on the positions of the human arm joints and the cigarette;

The second detection module is used to determine the position change between the hand and the mouth in the target image sequence based on the position information of the human arm joints and the mouth when the hand holds a cigarette information;

The result determination module is configured to determine the detection result of the smoking behavior according to the position change information.

According to a third aspect, an embodiment of the present invention provides an electronic device, including: a memory and a processor, the memory and the processor are connected to each other in communication, the memory stores computer instructions, and the processor By executing the computer instructions, the smoking detection method described in the first aspect or any implementation manner of the first aspect is implemented.

According to a fourth aspect, an embodiment of the present invention provides a computer-readable storage medium, the computer-readable storage medium stores computer instructions, and the computer instructions are used to enable the computer to execute any of the first aspect or the first aspect. A smoking detection method described in an embodiment.

Description of drawings

In order to more clearly illustrate the specific implementation of the present invention or the technical solutions in the prior art, the following will briefly introduce the accompanying drawings that need to be used in the specific implementation or description of the prior art. Obviously, the accompanying drawings in the following description The drawings show some implementations of the present invention, and those skilled in the art can obtain other drawings based on these drawings without any creative work.

Fig. 1 is the flowchart of the smoking detection method according to the embodiment of the present invention;

FIG. 2 is a flowchart of a method for determining location change information according to an embodiment of the present invention;

3 is a flowchart of a method for determining a detection result according to an embodiment of the present invention;

4 is a flow chart of a method for determining location information of each target according to an embodiment of the present invention;

5 is a schematic diagram of a smoking detection method according to an embodiment of the present invention;

6 is a flowchart of a smoking detection method according to an embodiment of the present invention;

Fig. 7 is a schematic diagram of joint point detection according to an embodiment of the present invention;

Fig. 8 is a schematic diagram of a smoking action according to an embodiment of the present invention;

Fig. 9 is a schematic diagram of a smoking behavior detection device according to an embodiment of the present invention;

FIG. 10 is a schematic diagram of a hardware structure of an electronic device provided by an embodiment of the present invention.

detailed description

In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments It is a part of embodiments of the present invention, but not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without making creative efforts belong to the protection scope of the present invention.

According to an embodiment of the present invention, an embodiment of a smoking detection method is provided. It should be noted that the steps shown in the flowcharts of the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and, although in The flowcharts show a logical order, but in some cases the steps shown or described may be performed in an order different from that shown or described herein.

In indoor scenes, it is possible to judge whether there is smoking behavior through smoke detection equipment. However, in outdoor scenes, such as construction sites and other places where smoking is prohibited, it is difficult to detect the existence of smoking behavior through smoke detection equipment. The smoking detection method provided by the embodiment of the present invention can be used in the above-mentioned scene where it is inconvenient to use a smoke detection device.

In this embodiment, a smoking detection method is provided, which can be used in devices such as computers, tablet computers, or cameras with artificial intelligence computing power. Fig. 1 is a flow chart of a smoking detection method according to an embodiment of the present invention, as shown in Fig. 1 As shown, the process includes the following steps:

S11. Acquire a target image sequence.

The target image is the image of the place where smoking behavior needs to be detected. The video image can be obtained through the surveillance camera in the place. The video data set contains the target image of continuous frames, that is, the target image sequence.

S12. Perform target detection on each target image in the target image sequence, and determine the position information of each target in the target image, where the targets include human arm joints, cigarettes, and mouths.

The target image sequence includes multiple target images, and target detection is performed on each target image, and the targets to be confirmed include human arm joints, cigarettes, and mouths.

In order to determine the position of the joint points of the human arm and the mouth, it is necessary to detect the human body in the target image, and then perform joint point detection and face detection based on the detected human body, and determine the coordinates of the human arm joint points through joint point detection. Joints include wrist joints, elbow joints and shoulder joints. Through face detection, determine the coordinates of the mouth.

The cigarette position in the target image is detected by the target detection algorithm. Before that, it is first necessary to detect whether there is a cigarette in the target image. Specifically, the cigarette-like object in the target image can be detected based on the target detection model. If a cigarette-like object is detected object, and then determine the coordinates of the object. Since there may be other cigarette-like objects, this solution needs to determine whether there is a smoking behavior together with whether there is a smoking action, so the cigarette in this step refers to all detected cigarette-like objects.

S13, based on the positions of the human arm joints and the cigarette, determine whether the hand holds the cigarette.

Judging the position of the joints of the human arm and the cigarette is mainly based on the detected coordinates of the joints of the human arm and the coordinates of the detected cigarette-like object to determine whether the hand holds the cigarette-like object. For example, the human body can be set The intersection ratio between the detection frame of the arm joint point and the cigarette detection frame needs to be greater than a certain threshold. In addition, the distance between the cigarette and the hand can also be calculated according to the obtained coordinates of the joint points of the human arm and the coordinates of the cigarette. If the calculated distance is less than a certain threshold, it can be determined that the hand holds the cigarette.

S14, when the hand holds the cigarette, based on the position information of the joint points of the human arm and the mouth, determine the position change information between the hand and the mouth in the target image sequence.

After S13, when it is determined that the hand does not hold a cigarette, it can basically be judged that there is no smoking behavior, and it may be that an object similar to a cigarette is misdetected as a cigarette.

When it is determined that the hand is holding a cigarette, since it is not completely sure whether the detected is a real cigarette, there may be a situation where the hand holds an object similar to a cigarette. Therefore, it is not possible to simply judge whether there is a smoking behavior, but also to judge Whether there is a smoking action. To judge the smoking action, it is necessary to combine multiple target images and the position change information between the hand and the mouth to make a comprehensive judgment. The position change information includes the distance between the hand and the mouth, the bending angle of the elbow joint, etc. First of all, it is necessary to determine the distance relationship between the hand and the mouth, through the position of the joint points of the human arm, including the positions of the wrist joint, elbow joint, shoulder joint and hand joint. The distance between the hand and the mouth can be calculated according to the coordinates of the hand joint and the mouth, and the bending angle of the elbow joint can be determined according to the arm joint points. Since the smoking action is dynamic and continuous, it is necessary to combine the position change information between the hand and the mouth in multiple target images. The position change information includes the change of the distance between the hand and the mouth and the change of the bending angle of the elbow joint.

Compare the distance between the hand and the mouth and the bending angle of the elbow joint with the set distance threshold between the hand and the mouth and the bending threshold of the elbow joint to determine whether there is a hand far away from the mouth or a hand in the target image sequence in chronological order. Proximity to the mouth, elbows bent in and out of a certain angle, etc.

S15. Determine the detection result of the smoking behavior according to the position change information.

The distance threshold between the hand and the mouth, the elbow bending angle threshold, etc. can be set according to the actual situation. When the distance between the hand and the mouth is less than a certain distance threshold, and the elbow bending angle is within a certain elbow bending angle threshold, then It is judged that there is a movement of the hand close to the mouth. When the distance between the hand and the mouth is greater than a certain distance threshold and the bending angle of the elbow joint is outside a certain bending angle of the elbow joint, it is determined that there is a movement of the hand moving away from the mouth.

Smoking actions include the hand approaching the mouth and the hand gradually moving away from the mouth. If it is judged that there are more than a certain number of hand approaching mouth movements and hand gradually moving away from the mouth within a certain period of time, it can basically be judged that there is a smoking action or an action similar to smoking. . Since the existence of a cigarette-like object held by the hand has been detected, combined with the detection of smoking or an action similar to smoking, it can basically be determined that there is a smoking behavior. If there is no smoking action or smoking-like action, it can be determined that there is no smoking action within the range of the target image sequence. If it is judged that there is a smoking behavior, an alarm prompt can be issued so that relevant personnel can deal with the smoking behavior in time to ensure the safety of the place.

The smoking detection method provided by the embodiment of the present invention obtains the target image sequence, detects the position information of the joints of the human arm, the cigarette and the mouth based on various target detection methods, judges whether a cigarette is held, and judges whether there is a smoking action or an action similar to smoking , and then determine whether there is smoking behavior. The method not only detects cigarettes, but also detects whether there is a smoking action at the same time, and then judges the detection result of the smoking behavior by combining the two, and uses multi-modes to distinguish the smoking behavior, which improves the accuracy of the smoking behavior detection.

In this embodiment, a method for determining position change information is provided, which corresponds to S14 in FIG. 1 and can be used for devices such as computers, tablet computers, or cameras with artificial intelligence computing power. FIG. 2 is a determination of position according to an embodiment of the present invention The flow chart of the method for changing information, as shown in Figure 2, the process includes the following steps:

S21. For each target image, determine the distance between the hand and the mouth based on the position information of the joint points of the human arm and the mouth.

After performing multiple target detections on the target image, the position information of the joint points of the human arm and the mouth in the target image have been determined, and the position information includes coordinate information. The human arm joints include wrist joints, elbow joints, shoulder joints, and hand joints. The distance between the hand and the mouth is calculated according to the coordinates of the hand joints and the coordinates of the mouth. In addition, the bending angle of the elbow joint can also be calculated in combination with each arm joint point.

S22. Determine position change information based on the time sequence of the target images and the distance between the hand and the mouth in each target image.

The position change information includes the change of the distance between the hand and the mouth, and the change of the bending angle of the elbow joint. Since the smoking action changes dynamically, it is necessary to combine the detection results of multiple target images when judging whether there is a smoking action, that is, combine the time sequence of the target image to judge whether the position change information within a certain period of time conforms to the change of the smoking action, such as hand The distance between the hand and the mouth will be within a certain preset threshold and the bending angle of the elbow joint will be smaller than a certain threshold. As time goes by, the distance between the hand and the mouth will gradually increase and the bending angle of the elbow joint will be greater than a certain threshold. Specifically, the steps of determining location change information are as follows:

S221. Determine first position change information based on the time sequence of the target images and the distance between the hand and the mouth in each target image.

The first position change information includes the change of the distance between the hand and the mouth, and the first position change information can be obtained by recording the distance between the hand and the mouth in the chronological target images.

S222. For each target image, determine the bending angle of the elbow joint of the arm based on the joint points of the human arm.

The human arm joint points include wrist joint, elbow joint, shoulder joint, etc. Since the coordinates of each joint on the arm have been determined, according to the coordinates of each joint, the bending angle of the elbow joint in each target image can be calculated.

S223. Determine second position change information based on the time sequence of the target images and the bending angle of the elbow joint of the arm.

The second position change information includes the change of the bending angle of the elbow joint of the arm, and the second position change information can be obtained by recording the bending angle of the elbow joint of the arm in the time sequence target images.

The smoking detection method provided by the embodiment of the present invention determines the distance between the hand and the mouth in each target image, and determines the position change information based on the time sequence of the target image and the obtained distance between the hand and the mouth. It is dynamic, so the information in multiple target images is combined for subsequent judgment of smoking actions, which improves the accuracy of smoking behavior detection.

In this embodiment, a method for determining the detection result is provided, which corresponds to S15 in FIG. 1 and can be used for devices such as computers, tablet computers, or cameras with artificial intelligence computing power. FIG. 3 is a determination of the detection result according to an embodiment of the present invention. The flow chart of the method, as shown in Figure 3, the process includes the following steps:

S31, comparing the first position change information in the position change information with a preset first threshold, and comparing the second position change information in the position change information with a preset first angle threshold, to determine the first-stage smoking action test results.

The first position change information includes the change of the distance between the hand and the mouth within a certain time range, and the preset first threshold is a distance threshold set according to the actual situation for judging whether the hand is close to the mouth. The second position change information includes the change of the bending angle of the elbow joint of the arm within a certain time range, and the preset first angle threshold is an angle threshold set according to the actual situation to assist in judging whether the hand is close to the mouth. By comparing the first position change information with the preset first threshold, and comparing the second position change information with the preset first angle threshold, if the distance between the hand and the mouth is not greater than the preset If the first threshold and the bending angle of the elbow joint of the arm is less than the preset first angle threshold, it can be determined that there is a first-stage smoking action, that is, there is an action of the hand approaching the mouth.

Specifically, S31 includes the following steps:

S311. Determine a preset first threshold according to the position information of the mouth.

The position information of the mouth includes the position of the left corner of the mouth and the position of the right corner of the mouth. According to the position of the left corner of the mouth and the position of the right corner of the mouth, the distance from the left corner of the mouth to the right corner of the mouth can be obtained. In this embodiment, the distance from the left corner of the mouth to the right corner of the mouth is determined is the preset first threshold. The preset first threshold is set according to the actual situation. In consideration of individual differences, in this embodiment, the distance from the left corner of the mouth to the right corner of the mouth is used as the preset first threshold, which can also be set in other ways in practical applications The first threshold is preset, which is not specifically limited.

S312. Within the preset time threshold, when the first position change information is not greater than the preset first threshold and the second position change information is smaller than the preset first angle threshold, determine that there is a first-stage smoking action.

When the distance between the hand and the mouth is not greater than the preset first threshold, and the bending angle of the elbow joint of the arm is less than the preset first angle threshold, it is determined that there is a first-stage smoking action, and the first-stage smoking action is when smoking, The hand is close to the mouth, the arm is bent and within a preset time threshold (for example, for 3 seconds or more).

S32, comparing the first position change information in the position change information with a preset second threshold, and comparing the second position change information in the position change information with a preset second angle threshold, to determine the second-stage smoking action test results.

The first position change information includes the change of the distance between the hand and the mouth within a certain time range, and the preset second threshold is a distance threshold set according to the actual situation for judging whether the hand is far away from the mouth. The second position change information includes the change of the bending angle of the elbow joint of the arm within a certain time range, and the preset second angle threshold is an angle threshold set according to the actual situation to assist in judging whether the hand is far away from the mouth. By comparing the first position change information with the preset second threshold, and comparing the second position change information with the preset second angle threshold, if the distance between the hand and the mouth is not less than the preset The second threshold and the bending angle of the elbow joint of the arm is greater than the preset first angle threshold, it can be determined that there is a second-stage smoking action, that is, there is an action where the hand moves away from the mouth.

Specifically, S32 includes the following steps:

S321. Determine a preset second threshold according to the joint points of the human arm.

The position information of the joint points of the human arm includes the position of the left shoulder and the position of the right shoulder. According to the position of the left shoulder and the position of the right shoulder, the distance from the left shoulder to the right shoulder can be obtained. In this embodiment, the distance from the left shoulder to the right shoulder is determined as Preset the second threshold. The preset second threshold is set according to the actual situation. Considering individual differences, in this embodiment, the distance from the left shoulder to the right shoulder is used as the preset second threshold. In actual application, other methods can also be used to set the preset threshold. A second threshold is set, which is not specifically limited.

S322. Within the preset time threshold, when the first position change information is not less than the preset second threshold and the second position change information is less than the preset second angle threshold, determine that there is a second-stage smoking action.

When the distance between the hand and the mouth is not less than the preset second threshold, and the bending angle of the elbow joint of the arm is greater than the preset second angle threshold, it is determined that there is a second-stage smoking action, and the second-stage smoking action is after smoking, The hand is kept away from the mouth and within a preset time threshold (for example, for 3 seconds or more).

S33. Determine the detection result of the smoking behavior according to the detection result of the smoking action in the first stage and the detection result of the smoking action in the second stage.

When the detection result of the first-stage smoking action is to determine the existence of the first-stage smoking action, and the detection result of the second-stage smoking action is to determine the existence of the second-stage smoking action, that is, it is determined that within a certain time range, there is a hand close to the mouth. Smoking action and the action of moving the hand away from the mouth. If it is judged that there are more than a certain number of periodic first-stage smoking actions and second-stage smoking actions, combined with the detection of a cigarette-like object, the existence of smoking behavior can be determined.

The detection result of smoking behavior also includes determining that there is no smoking behavior. If there is no first-stage smoking action, or there is no second-stage smoking action, or both do not exist, it can be determined that there is no smoking action. action, it can be determined that there is no smoking behavior.

The smoking detection method provided by the embodiment of the present invention compares the first position change information and the second position change information in the position change information with the preset first threshold and the preset first angle threshold respectively, so as to determine whether there is a first For the staged smoking action, the first position change information and the second position change information are compared with the preset second threshold and the preset second angle threshold respectively, so as to determine whether there is a second-stage smoking action. When judging whether there is smoking behavior, since the smoking action is considered, it is necessary to combine the detection results of the first-stage smoking action and the detection results of the second-stage smoking action to make a comprehensive judgment, which improves the accuracy of smoking behavior detection.

In this embodiment, a method for determining position information of each target is provided, corresponding to S12 in FIG. 1 , and FIG. 4 is a flow chart of a method for determining position information of each target according to an embodiment of the present invention. As shown in FIG. 4 , the flow Including the following steps:

S41. Detect the target image, and determine the human body and the cigarette in the target image.

In the process of detecting the target image, various target detection models can be used to detect different targets. For example, the human body in the target image can be detected through the human body detection model, that is, machine learning or deep learning can be used to detect in pictures or video streams Figure out the human body and mark its location. To detect the cigarette-like object in the target image through the target detection model, in order to avoid false detection, the cigarette detected in this step refers to the cigarette-like object, and it is not directly determined whether it is a cigarette, but it needs to be comprehensively judged in combination with the smoking action.

S42, extracting bone joint points from the human body, and determining human arm joint points.

After the human body is detected, the joint points of the human skeleton can be extracted through the deep learning model, so as to determine the joint points of the arm, and connect these joint points according to the human body structure with connecting lines.

S43. Detect the face of the human body, and determine the position information of the mouth.

After detecting the human body, face detection can be performed based on the detected human body. For example, through the detection of the key point model of the face, the position of the facial features can be detected, and the position information of the mouth can be determined. The position information of the mouth can include the left corner of the mouth. Coordinates, coordinates of the right mouth corner, etc.

The smoking detection method provided by the embodiment of the present invention detects the human body and the cigarette in the target image at the same time, and then detects the position information of the human arm joints and the mouth based on the detection of the human body. accuracy.

In this embodiment, a smoking detection method is provided, which can be based on the hardware combination of a camera plus an edge box with artificial intelligence computing power, as shown in Figure 5, or based on a camera with artificial intelligence computing power. The schematic flow diagram of the method is shown in the figure 6. When the camera video stream is read into the device, it is decomposed into a sequence of images. The image sequence passes through multiple object detection models to determine each object in the object image. Through the human detection model, such as the Yolov5 model, the human body is detected in the target image based on machine learning or deep learning methods, and its position is marked. Based on the detected human body, the joint points of the human body are extracted using a human skeleton joint point detection model, such as the openpose model. After detecting the joint points of the human body, connect these joint points according to the human body structure with connecting lines, as shown in Figure 7. Face detection is performed based on the detected human body, or face detection can be performed directly, and the left eye, right eye, nose, left mouth corner, and right mouth corner are detected by using the key point model of the face, such as the Retinaface model. Use a target detection model to detect cigarette-like objects in the target image, such as the MobileSSD or Yolo series (v3, v4 or v5) models. It should be noted that in this embodiment, several target detections can be used in parallel or serially. Judge whether there is a smoking action by detecting the left mouth corner, right mouth corner and human arm joints. Calculate the distance from the midpoint of the left and right mouth corners to the hand in each target image, define the distance from the left mouth corner to the right mouth corner as the preset first threshold, and define the distance from the left shoulder to the right shoulder as the preset second threshold.

When smoking, when the distance from the hand to the mouth is not greater than the preset first threshold and the bending angle of the elbow joint is smaller than the preset first angle threshold, and this action is kept for a certain period of time (for example, more than 3 seconds), it can be judged that there is a second threshold. One-stage smoking action. After that, the hand gradually moves away from the mouth. When the distance from the hand to the mouth is not less than the preset second threshold and the bending angle of the elbow joint is greater than the preset second angle threshold, and the action is kept for a certain period of time (for example, more than 3 seconds), it can be judged There is a second phase of smoking action. Periodic judgment is then carried out. If the first-stage smoking action and the second-stage smoking action are repeated for a set number of times within a certain period, it is determined that there is a smoking action. The schematic diagram of the action is shown in Figure 8. In this embodiment, the preset first angle threshold is set to 30 degrees, and the preset second angle threshold is set to 150 degrees. If it is detected that there is a cigarette-like object in the target image, and the intersection ratio between the detection frame of the cigarette-like object and the human body detection frame is greater than a certain value, in this embodiment, the threshold value of the intersection ratio is set to 0.8. There is a smoking action, and after periodic judgment, that is, the smoking action is repeated more than a certain number of times (for example, more than 3 times) within a set time period. After comprehensive judgment, it is completely determined that there is smoking behavior, otherwise it is still not judged that there is smoking behavior. If it is judged that there is a smoking behavior, an alarm prompt can be issued to ensure the safety of the monitored place and avoid the occurrence of fire.

The smoking detection method based on video or picture sequence provided in this embodiment uses video frame and target detection to perform action semantic analysis and target capture, and simultaneously uses human body detection, human body joint point detection, target detection and face key point detection to combine algorithms The process is based on the characteristics of the smoking action to construct an algorithm to distinguish the smoking action, and combined with cigarette detection, a variety of discrimination mechanisms are used. Although the analysis is based on video data in this method, video data can be extracted to solve the problem based on static pictures. detection leads to the problem of low detection accuracy.

In this embodiment, a device for detecting smoking behavior is also provided, and the device is used to implement the above embodiments and implementation modes, and what has been described will not be repeated. As used below, the term "module" may be a combination of software and/or hardware that realizes a predetermined function. Although the devices described in the following embodiments are preferably implemented in software, implementations in hardware, or a combination of software and hardware are also possible and contemplated.

This embodiment provides a smoking behavior detection device, as shown in Figure 9, including:

An image acquisition module 51, configured to acquire a target image sequence;

The first detection module 52 is configured to perform target detection on each target image in the target image sequence, and determine the position information of each target in the target image, and the target includes human arm joints, cigarettes and mouths;

A cigarette judging module 53, configured to determine whether the hand holds a cigarette based on the positions of the human arm joints and the cigarette;

The second detection module 54 is configured to determine the position between the hand and the mouth in the target image sequence based on the position information of the human arm joint points and the mouth when the hand holds a cigarette change information;

The result determination module 55 is configured to determine the detection result of the smoking behavior according to the position change information.

In some embodiments, the second detection module 54 includes:

a distance determination unit, configured to determine the distance between the hand and the mouth based on the position information of the joint points of the human arm and the mouth for each of the target images;

The position change information determining unit is configured to determine the position change information based on the time sequence of the target images and the distance between the hand and the mouth in each target image.

In some implementations, the position change information determining unit includes:

A first position change information subunit, configured to determine first position change information based on the time sequence of the target images and the distance between the hand and the mouth in each of the target images;

A bending angle determination subunit, configured to determine, for each of the target images, the bending angle of the elbow joint of the arm based on the joint points of the human arm;

The second position change information subunit is configured to determine second position change information based on the time sequence of the target images and the bending angle of the elbow joint of the arm.

In some embodiments, the outcome determination module 55 includes:

A first result determining unit, configured to compare the first position change information in the position change information with a preset first threshold, and compare the second position change information in the position change information with a preset first angle Threshold value is compared to determine the detection result of the smoking action in the first stage;

A second result determining unit, configured to compare the first position change information in the position change information with a preset second threshold, and compare the second position change information in the position change information with a preset second angle Threshold value is compared to determine the detection result of the second-stage smoking action;

The detection result determining unit is configured to determine the detection result of the smoking behavior according to the detection result of the first-stage smoking action and the detection result of the second-stage smoking action.

In some embodiments, the first result determination unit includes:

A first threshold determination subunit, configured to determine a preset first threshold according to the position information of the mouth;

The first action determination subunit is configured to, within a preset time threshold, when the first position change information is not greater than the preset first threshold and the second position change information is smaller than the preset first angle threshold , to determine the existence of the first-stage smoking action.

In some embodiments, the second result determination unit includes:

A second threshold determination subunit, configured to determine a preset second threshold according to the joint points of the human arm;

The second action determination subunit is configured to, within a preset time threshold, when the first position change information is not less than the preset second threshold and the second position change information is less than the preset second angle threshold , to determine the presence of a second-stage smoking action.

In some embodiments, the first detection module 52 includes:

a target detection unit, configured to detect the target image and determine the human body and cigarettes in the target image;

a joint point determination unit, configured to extract the skeleton joint points of the human body, and determine the joint points of the human arm;

The position information determination unit is used to detect the face of the human body and determine the position information of the mouth.

The smoking behavior detection device in this embodiment is presented in the form of a functional unit, where the unit refers to an ASIC circuit, a processor and memory that execute one or more software or fixed programs, and/or other devices that can provide the above functions device.

Further functional descriptions of the above-mentioned modules are the same as those in the above-mentioned corresponding embodiments, and will not be repeated here.

An embodiment of the present invention also provides an electronic device having the smoking behavior detection device shown in FIG. 9 above.

Please refer to FIG. 10. FIG. 10 is a schematic structural diagram of an electronic device provided by an embodiment of the present invention. As shown in FIG. 10, the electronic device may include: at least one processor 601, such as a CPU (Central Processing Unit, central processing unit ), at least one communication interface 603, memory 604, and at least one communication bus 602. Wherein, the communication bus 602 is used to realize connection and communication between these components. Wherein, the communication interface 603 may include a display screen (Display) and a keyboard (Keyboard), and the optional communication interface 603 may also include a standard wired interface and a wireless interface. The memory 604 may be a high-speed RAM memory (Random Access Memory, volatile random access memory), or a non-volatile memory (non-volatile memory), such as at least one disk memory. Optionally, the memory 604 may also be at least one storage device located away from the aforementioned processor 601 . Wherein the processor 601 may be combined with the apparatus described in FIG. 9 , the memory 604 stores an application program, and the processor 601 invokes the program code stored in the memory 604 to execute any of the above method steps.

Wherein, the communication bus 602 may be a peripheral component interconnect (PCI for short) bus or an extended industry standard architecture (EISA for short) bus or the like. The communication bus 602 can be divided into an address bus, a data bus, a control bus, and the like. For ease of representation, only one thick line is used in FIG. 10 , but it does not mean that there is only one bus or one type of bus.

Wherein, the memory 604 may include a volatile memory (English: volatile memory), such as a random-access memory (English: random-access memory, abbreviated as RAM); the memory may also include a non-volatile memory (English: non-volatile memory), such as flash memory (English: flash memory), hard disk (English: hard diskdrive, abbreviated: HDD) or solid-state hard drive (English: solid-state drive, abbreviated: SSD); the storage 604 can also include the above-mentioned types of storage The combination.

Wherein, the processor 601 may be a central processing unit (English: central processing unit, abbreviated: CPU), a network processor (English: network processor, abbreviated: NP) or a combination of CPU and NP.

Wherein, the processor 601 may further include a hardware chip. The aforementioned hardware chip may be an application-specific integrated circuit (English: application-specific integrated circuit, abbreviation: ASIC), a programmable logic device (English: programmable logic device, abbreviation: PLD) or a combination thereof. The above-mentioned PLD can be a complex programmable logic device (English: complex programmable logic device, abbreviation: CPLD), field-programmable logic gate array (English: field-programmable gate array, abbreviation: FPGA), general array logic (English: generic array logic , Abbreviation: GAL) or any combination thereof.

Optionally, the memory 604 is also used to store program instructions. The processor 601 may invoke program instructions to implement the smoking detection method shown in the embodiment of the present application.

The embodiment of the present invention also provides a non-transitory computer storage medium, the computer storage medium stores computer-executable instructions, and the computer-executable instructions can execute the smoking detection method in any method embodiment above. Wherein, the storage medium may be a magnetic disk, an optical disk, a read-only memory (Read-Only Memory, ROM), a random access memory (Random Access Memory, RAM), a flash memory (Flash Memory), a hard disk (Hard Disk) DiskDrive, abbreviation: HDD) or solid-state disk (Solid-State Drive, SSD), etc.; the storage medium may also include a combination of the above-mentioned types of memory.

Although the embodiments of the present invention have been described in conjunction with the accompanying drawings, those skilled in the art can make various modifications and variations without departing from the spirit and scope of the present invention. within the bounds of the requirements.

Claims (10)

1. A method of smoking detection, comprising:

acquiring a target image sequence;

performing target detection on each target image in the target image sequence, and determining position information of each target in the target image, wherein the target comprises a human body arm joint point, a cigarette and a mouth;

determining whether the hand holds the cigarette or not based on the joint points of the human arm and the position of the cigarette;

when the hand holds a cigarette, determining position change information between the hand and the mouth in the target image sequence based on the human arm joint points and the position information of the mouth;

and determining the detection result of the smoking behavior according to the position change information.

2. The method of claim 1, wherein the determining position change information between a hand and the mouth in the target image sequence based on the human arm joint point and the position information of the mouth comprises:

for each of the target images, determining a distance between the hand and the mouth based on the human arm joint points and the position information of the mouth;

determining the position change information based on a temporal order of the target images and a distance between the hand and the mouth in each of the target images.

3. The method of claim 2, wherein determining the position change information based on the temporal order of the target images and the distance between the hand and the mouth in each of the target images comprises:

determining first position change information based on the time sequence of the target images and the distance between the hand and the mouth in each of the target images;

for each of the target images, determining a bending angle of an elbow joint of an arm based on the human arm joint points;

determining second position change information based on the temporal sequence of the target images and the bend angle of the elbow joint.

4. The method of claim 1, wherein determining the detection result of smoking behavior according to the location change information comprises:

comparing first position change information in the position change information with a preset first threshold value, and comparing second position change information in the position change information with a preset first angle threshold value to determine a detection result of smoking action at a first stage;

comparing first position change information in the position change information with a preset second threshold value, and comparing second position change information in the position change information with a preset second angle threshold value to determine a detection result of second-stage smoking actions;

and determining the detection result of the smoking behavior according to the detection result of the first stage smoking action and the detection result of the second stage smoking action.

5. The method according to claim 4, wherein the comparing the first position change information in the position change information with a preset first threshold value and the comparing the second position change information in the position change information with a preset first angle threshold value to determine the detection result of the first stage smoking action comprises:

determining a preset first threshold according to the position information of the mouth;

and determining that a first stage smoking action exists when the first position change information is not greater than the preset first threshold and the second position change information is smaller than a preset first angle threshold within a preset time threshold.

6. The method according to claim 4, wherein the comparing the first position change information in the position change information with a preset second threshold value and the comparing the second position change information in the position change information with a preset second angle threshold value to determine the detection result of the second stage smoking action comprises:

determining a preset second threshold according to the human arm joint point;

and in a preset time threshold, when the first position change information is not smaller than the preset second threshold and the second position change information is smaller than a preset second angle threshold, determining that second-stage smoking actions exist.

7. The method according to any one of claims 1-6, wherein the performing object detection on each object image in the sequence of object images and determining position information of each object in the object images comprises:

detecting the target image, and determining a human body and cigarettes in the target image;

extracting skeleton joint points of the human body, and determining arm joint points of the human body;

and detecting the human face of the human body and determining the position information of the mouth.

8. A smoking detection device, the device comprising:

the image acquisition module is used for acquiring a target image sequence;

the first detection module is used for carrying out target detection on each target image in the target image sequence and determining the position information of each target in the target images, wherein the target comprises a human body arm joint point, a cigarette and a mouth;

the cigarette judging module is used for determining whether the hand holds a cigarette or not based on the joint points of the human arm and the position of the cigarette;

the second detection module is used for determining position change information between the hand and the mouth in the target image sequence based on the human body arm joint points and the position information of the mouth when the hand holds a cigarette;

and the result determining module is used for determining the detection result of the smoking behavior according to the position change information.

9. An electronic device, comprising:

a memory and a processor, the memory and the processor being communicatively connected to each other, the memory having stored therein computer instructions, the processor executing the computer instructions to perform the smoking detection method according to any one of claims 1 to 7.

10. A computer-readable storage medium having stored thereon computer instructions for causing a computer to perform the smoking detection method of any one of claims 1-7.

Images