CN116152928A

CN116152928A - Drowning prevention early warning method and system based on lightweight human body posture estimation model

Info

Publication number: CN116152928A
Application number: CN202310167330.9A
Authority: CN
Inventors: 于中阳; 杨文辉; 张萌飞
Original assignee: Shanghai Wangchain Information Technology Co ltd
Current assignee: Shanghai Wangchain Information Technology Co ltd
Priority date: 2023-02-27
Filing date: 2023-02-27
Publication date: 2023-05-23

Abstract

The invention provides a drowning prevention early warning method and a drowning prevention early warning system based on a lightweight human body posture estimation model, wherein the drowning prevention early warning method comprises the following steps: acquiring swimming video data of a swimmer shot by an underwater camera in real time; intercepting swimming video data to obtain multi-frame swimming images, sequentially inputting the multi-frame swimming images into a lightweight human body posture estimation model to mark key points of human bodies, and obtaining coordinates of a plurality of key points of the human bodies; calculating to obtain a key point feature vector according to the relation among coordinates of a plurality of human body key points; judging whether the drowning similarity between the key point feature vector in the multi-frame swimming image and the key point feature vector of any drowning state judging set is larger than or equal to the respective preset similarity threshold value; if yes, determining that the swimmer drowns, and sending a drowning early warning signal. The technical scheme of the invention can solve the problems that most of deep learning models in the prior art are overlarge in quantity and too many in parameters, and are difficult to deploy to embedded equipment for reasoning.

Description

Drowning prevention early warning method and system based on lightweight human body posture estimation model

Technical Field

The invention relates to the technical field of communication, in particular to an anti-drowning early warning method and system of a lightweight human body posture estimation model.

Background

Swimming is a very popular sport, however, swimming pool drowning accidents occur frequently. In order to reduce drowning accidents of swimming pools, related personnel research various drowning prevention technologies of swimming pools, such as wearable drowning prevention equipment, computer vision-based drowning prevention technologies of swimming pools and the like.

The wearable anti-drowning device is a device such as a bracelet or a swimming cap worn by a swimmer, detects indexes such as heart rate and blood oxygen of a human body in real time, and timely gives an alarm if abnormality occurs. Although the method can timely warn drowning of the swimming pool, the wearable anti-drowning device influences swimming experience and is easy to be interfered by environment, and frequent maintenance is needed.

For solving the defect of wearable drowning prevention equipment, prior art still provides the swimming pool and prevents drowning technique based on computer vision, and the swimming pool prevents drowning technique based on computer vision and is current research hotspot. The swimming pool drowning prevention technology based on computer vision utilizes computer vision to detect images of swimmers, and utilizes a deep learning model to carry out target detection on the images of the swimmers so as to identify drowning behaviors.

However, most of the current deep learning models are too large in quantity, too many in parameters, difficult to deploy to embedded equipment for real-time reasoning, and low in accuracy.

Disclosure of Invention

The invention provides an anti-drowning early warning scheme based on a lightweight human body posture estimation model, and aims to solve the problems that most of deep learning models in the prior art are too large in volume, too many in parameters, difficult to deploy to embedded equipment for real-time reasoning and low in accuracy.

In order to solve the above problems, according to a first aspect of the present invention, there is provided an anti-drowning early warning method based on a lightweight human body posture estimation model, including:

acquiring swimming video data of a swimmer shot by an underwater camera in real time;

intercepting swimming video data to obtain multi-frame swimming images, sequentially inputting the multi-frame swimming images into a lightweight human body posture estimation model to mark key points of human bodies, and obtaining coordinates of a plurality of key points of the human bodies;

according to the relation among coordinates of a plurality of human body key points, calculating to obtain key point feature vectors of the human body key points, wherein the key point feature vectors comprise key point distance vectors, key point included angle vectors and key point distance change speed vectors;

Judging whether the drowning similarity of the key point distance vector, the key point included angle vector and the key point distance change speed vector in the multi-frame swimming image and the key point distance vector, the key point included angle vector and the key point distance change speed vector of any drowning state judgment set in the multiple drowning state judgment sets is larger than or equal to the respective preset similarity threshold value;

if the drowning similarity of the key point distance vector, the key point included angle vector and the key point distance change speed vector is larger than or equal to the respective preset similarity threshold value, determining that the swimmer drowns, and sending a drowning early warning signal.

Preferably, in the drowning prevention early warning method, the step of sequentially inputting a plurality of frames of swimming images into a lightweight human body posture estimation model to mark key points of the human body to obtain coordinates of a plurality of key points of the human body includes:

dividing a training set, a verification set and a test set of the lightweight human body posture estimation model according to a preset quantity proportion;

respectively inputting the training set, the verification set and the test set into a lightweight human body posture estimation model according to a preset quantity proportion to train, verify and test to obtain a lightweight human body posture estimation model passing the test;

Inputting the swimming image into a MobileNet model of a lightweight human body posture estimation model for convolution to obtain a swimming feature map containing a plurality of human body key points;

inputting the swimming feature map into a single-branch convolution neural network of a lightweight human body posture estimation model for convolution;

predicting a position confidence map and an affinity domain of each human body key point in a plurality of human body key points by using a single-branch convolutional neural network;

and respectively matching the position confidence coefficient map and the affinity domain of each human body key point to obtain the coordinates of a plurality of human body key points.

Preferably, in the above drowning prevention early warning method, the step of calculating the key point feature vector of the key point of the human body according to the relation between coordinates of the key points of the human body includes:

selecting a first reference key point from a plurality of human body key points;

according to the coordinates of other human body key points and the coordinates of the first reference key points in the plurality of human body key points, respectively calculating the distances between all other human body key points and the first reference key points to obtain a key point distance vector;

selecting a plurality of second reference key points from the plurality of human body key points;

calculating included angles of the plurality of second reference key points according to the coordinates of the plurality of second reference key points to obtain a key point included angle vector;

Extracting the key point distance difference of the front frame swimming image and the rear frame swimming image from the multi-frame swimming image according to the coordinates of the key points of the human bodies;

calculating to obtain a key point distance change speed vector according to the key point distance difference of the front and rear frames of swimming images and the interval time of the front and rear frames of swimming images;

and constructing a key point feature vector by using the key point distance vector, the key point included angle vector and the key point distance change speed vector.

Preferably, in the above drowning prevention early warning method, the step of determining whether the drowning similarity of the key point distance vector, the key point included angle vector, and the key point distance change speed vector in the multi-frame swimming image and the key point distance vector, the key point included angle vector, and the key point distance change speed vector of any one of the multiple drowning state determination sets is greater than or equal to the respective preset similarity threshold value comprises:

according to a cosine similarity calculation formula:

respectively calculating the drowning similarity of the key point distance vector, the key point included angle vector and the key point distance change speed vector in the swimming image and the key point distance vector, the key point included angle vector and the key point distance change speed vector of any drowning state judgment set; wherein A is a key point feature vector in a swimming image, and B is a key point feature vector of a drowning state judgment set;

Judging whether the drowning similarity of the key point distance vector, the key point included angle vector and the key point distance change speed vector is larger than or equal to the respective preset similarity threshold value;

if the drowning similarity of the key point distance vector, the key point included angle vector and the key point distance change speed vector is larger than or equal to the respective preset similarity threshold value, determining that the swimmer drowns.

Preferably, before the step of determining whether the drowning similarity between the key point distance vector, the key point included angle vector and the key point distance change speed vector in the multi-frame swimming image and the key point distance vector, the key point included angle vector and the key point distance change speed vector of any one of the drowning state determination sets is greater than or equal to the respective preset similarity threshold, the method further includes:

selecting multiple drowning state data sets, wherein each drowning state data set in the multiple drowning state data sets comprises a plurality of continuous frames of drowning images;

inputting continuous multi-frame drowning images contained in each drowning state data set into a lightweight human body posture estimation model to obtain human body key point coordinates corresponding to each drowning state;

According to human body key point coordinates corresponding to each drowning state, key point feature vectors of human body key points corresponding to each drowning state are calculated respectively, wherein the key point feature vectors comprise key point distance vectors, key point included angle vectors and key point distance change speed vectors;

and constructing and obtaining a plurality of drowning state judgment sets by using the key point feature vectors of the human body key points corresponding to each drowning state.

Preferably, after the step of determining whether the drowning similarity of the key point distance vector, the key point included angle vector and the key point distance change speed vector is greater than or equal to the respective preset similarity threshold, the method further includes:

if the drowning similarity of any one of the key point distance vector, the key point included angle vector and the key point distance change speed vector is smaller than a preset similarity threshold value, recalculating the key point distance vector, the key point included angle vector and the key point distance change speed vector of the key points of the human body in the multi-frame swimming image;

and re-executing the step of judging whether the drowning similarity of the key point distance vector, the key point included angle vector and the key point distance change speed vector in the multi-frame swimming image and the key point distance vector, the key point included angle vector and the key point distance change speed vector of any drowning state judgment set in the multiple drowning state judgment sets is greater than or equal to the respective preset similarity threshold value or not until the fact that the swimmer drowns or the swimmer stops swimming is determined.

According to a second aspect of the present invention, the present invention further provides an anti-drowning early warning system based on a lightweight human body posture estimation model, comprising:

the swimming data acquisition module is used for acquiring swimming video data of a swimmer shot by the underwater camera in real time;

the swimming data intercepting module is used for intercepting swimming video data to obtain multi-frame swimming images;

the key point marking module is used for sequentially inputting a plurality of frames of swimming images into the lightweight human body posture estimation model to mark the key points of the human body so as to obtain coordinates of a plurality of key points of the human body;

the feature vector calculation module is used for calculating and obtaining key point feature vectors of the key points of the human body according to the relation among coordinates of the key points of the human body, wherein the key point feature vectors comprise key point distance vectors, key point included angle vectors and key point distance change speed vectors;

the similarity judging module is used for judging whether the drowning similarity of the key point distance vector, the key point included angle vector and the key point distance change speed vector in the multi-frame swimming image and the key point distance vector, the key point included angle vector and the key point distance change speed vector of any drowning state judging set in the multiple drowning state judging sets is larger than or equal to the respective preset similarity threshold value;

And the early warning signal sending module is used for determining that the swimmer is drowned and sending a drowned early warning signal if the similarity judging module judges that the drowned similarity of the key point distance vector, the key point included angle vector and the key point distance change speed vector is larger than or equal to the respective preset similarity threshold value.

Preferably, in the above drowning prevention early warning system, the key point marking module includes:

the data set dividing submodule is used for dividing a training set, a verification set and a test set of the lightweight human body posture estimation model according to a preset quantity proportion;

the network training test sub-module is used for respectively inputting a training set, a verification set and a test set into the lightweight human body posture estimation model according to a preset quantity proportion to train, verify and test to obtain a lightweight human body posture estimation model passing the test;

the model convolution sub-module is used for inputting the swimming image into a MobileNet model of the lightweight human body posture estimation model for convolution to obtain a swimming feature map containing a plurality of human body key points;

the neural network convolution sub-module is used for inputting the swimming feature map into a single-branch convolution neural network of the lightweight human body posture estimation model for convolution;

The key point prediction sub-module is used for predicting a position confidence coefficient map and an affinity domain of each human body key point in a plurality of human body key points by using a single-branch convolutional neural network;

and the key point matching sub-module is used for respectively matching the position confidence coefficient map and the affinity domain of each human key point to obtain the coordinates of a plurality of human key points.

Preferably, in the above drowning prevention early warning system, the feature vector calculation module includes:

the first key point selecting sub-module is used for selecting a first reference key point from a plurality of human body key points;

the key point distance calculation sub-module is used for respectively calculating the distances between all other human body key points and the first reference key points according to the coordinates of the other human body key points and the coordinates of the first reference key points in the plurality of human body key points to obtain key point distance vectors;

the second key point selecting sub-module is used for selecting a plurality of second reference key points from a plurality of human body key points;

the key point included angle calculation sub-module is used for calculating included angles of the plurality of second reference key points according to coordinates of the plurality of second reference key points to obtain a key point included angle vector;

the key point distance difference extraction sub-module is used for extracting the key point distance difference of the front frame swimming image and the rear frame swimming image from the multi-frame swimming image according to the coordinates of a plurality of human body key points;

The distance change speed vector calculation operator module is used for calculating and obtaining a key point distance change speed vector according to the key point distance difference of the front frame swimming image and the rear frame swimming image and the interval time of the front frame swimming image and the rear frame swimming image;

the key point feature vector construction submodule is used for constructing the key point feature vector by using the key point distance vector, the key point included angle vector and the key point distance change speed vector.

Preferably, in the above drowning prevention early warning system, the similarity judging module includes:

the drowning similarity calculation submodule is used for calculating a formula according to cosine similarity:

the drowning similarity judging sub-module is used for judging whether the drowning similarity of the key point distance vector, the key point included angle vector and the key point distance change speed vector is larger than or equal to the respective preset similarity threshold value.

In summary, according to the drowning prevention early warning scheme based on the lightweight human body posture estimation model, swimming video data of a swimmer photographed by an underwater camera in real time are obtained, the swimming video data are intercepted to obtain multi-frame swimming images, and then the multi-frame swimming images are sequentially input into the lightweight human body posture estimation model to perform human body key point marking, so that coordinates of a plurality of human body key points can be obtained. According to the technical scheme, the drowning detection is carried out by using the lightweight human body posture estimation model, and the model is small in size, high in speed and capable of meeting the precision requirement; in addition, the lightweight human body posture estimation model is used for marking to obtain a plurality of human body key points, the human body key points have strong correlation with swimming states, and drowning behaviors can be accurately detected. Calculating a key point feature vector of the key points of the human body by using the relation among coordinates of the key points of the human body, wherein the key point feature vector comprises a key point distance vector, a key point included angle vector and a key point distance change speed vector, and can accurately detect the swimming action of a swimmer to be used as a drowning judgment standard; and judging whether the drowning similarity of the key point distance vector, the key point included angle vector and the key point distance change speed vector in the multi-frame swimming image and the key point distance vector, the key point included angle vector and the key point distance change speed vector of any drowning state judgment set in the multiple drowning state judgment sets is larger than or equal to the respective preset similarity threshold value or not, so that the drowning behaviors of different swimming groups can be accurately detected. When the drowning similarity of the feature vectors of the key points is larger than or equal to the respective preset similarity threshold value, the drowning of the swimmer can be determined, so that a drowning early warning signal is sent, and the accuracy of the drowning judgment can be improved by using a simple deep learning model.

Drawings

In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings that are required in the embodiments or the description of the prior art will be briefly described, and it is obvious that the drawings in the following description are only some embodiments of the present invention, and other drawings may be obtained according to the structures shown in these drawings without inventive effort for a person skilled in the art.

FIG. 1 is a schematic diagram of an anti-drowning early warning system based on a lightweight human body posture estimation model according to an embodiment of the present invention;

fig. 2 is a schematic flow chart of a first drowning prevention early warning method based on a lightweight human body posture estimation model according to an embodiment of the present invention;

FIG. 3 is a flow chart of a method for marking key points of a human body by using a lightweight human body posture estimation model according to the embodiment shown in FIG. 2;

fig. 4 is a flowchart of a method for calculating a key point feature vector of a key point of a human body according to the embodiment shown in fig. 2;

FIG. 5 is a flow chart of a method for determining drowning similarity according to the embodiment shown in FIG. 2;

FIG. 6 is a flow chart of a method for generating key point feature vectors of a plurality of drowning state decision sets according to an embodiment of the present invention;

Fig. 7 is a schematic flow chart of a second drowning prevention early warning method based on a lightweight human body posture estimation model according to an embodiment of the present invention;

FIG. 8 is a schematic diagram of a structure of human body key points extracted by using a lightweight human body posture estimation model according to an embodiment of the present invention;

FIG. 9 is a schematic structural diagram of an anti-drowning early warning system based on a lightweight human body posture estimation model according to an embodiment of the present invention;

FIG. 10 is a schematic diagram of a swimming data intercepting module according to the embodiment shown in FIG. 9;

FIG. 11 is a schematic diagram of a feature vector calculation module according to the embodiment shown in FIG. 9;

fig. 12 is a schematic structural diagram of a similarity determining module according to the embodiment shown in fig. 9.

The achievement of the objects, functional features and advantages of the present invention will be further described with reference to the accompanying drawings, in conjunction with the embodiments.

Detailed Description

It should be understood that the specific embodiments described herein are for purposes of illustration only and are not intended to limit the scope of the invention.

The technical problems mainly solved by the embodiment of the invention are as follows:

the drowning technique is prevented to swimming pool based on computer vision that prior art provided, this drowning technique is prevented to swimming pool based on computer vision can utilize computer vision to detect swimmer image to utilize the deep learning model to carry out target detection to swimmer image, thereby discern drowning action. However, most of the current deep learning models are too large in quantity, too many in parameters, difficult to deploy to embedded equipment for real-time reasoning, and low in accuracy.

In order to solve the above problems, the following embodiments of the present invention provide an anti-drowning early warning scheme based on a lightweight human body posture estimation model, by using the lightweight human body posture estimation model, human body key point marks are performed on swimming video data of a swimmer to obtain coordinates of a plurality of human body key points, and then key point feature vectors of the human body key points are obtained by calculation according to the relationship between the coordinates, including a key point distance vector, a key point included angle vector and a key point distance change speed vector, so as to describe limb actions when the swimmer swims. The lightweight human body self-estimation model is small in size, few in parameters and poor in accuracy, the real-time performance and accuracy of drowning detection can be improved, multiple drowning state judgment sets are selected, any drowning state judgment set is provided with a key point distance vector, a key point included angle vector and a key point distance change speed vector corresponding to the drowning state, so that the purposes of accurately and timely drowning monitoring and early warning are achieved by judging whether the key point distance vector, the key point included angle vector and the key point distance change speed vector in the multi-frame swimming image are respectively greater than or equal to respective preset similarity threshold values with a drowning similarity algorithm of the key point distance vector, the key point included angle vector and the key point distance change speed vector of any drowning state judgment set.

In order to achieve the above objective, referring to fig. 1, fig. 1 is a schematic diagram of an anti-drowning early warning system based on a lightweight human body posture estimation model according to an embodiment of the present invention. As shown in fig. 1, the structure of the drowning prevention early warning system includes: a terminal layer, an edge layer (edge computing node) and a cloud layer; the structure of the drowning prevention early warning system utilizes cloud edge cooperative technology, a Lightweight OpenPose lightweight human body posture estimation model is deployed on the edge layer, and detection of drowning of a swimming pool can be guaranteed in real time. As shown in fig. 1, the structure of the drowning prevention early warning system includes:

terminal layer 300: and acquiring swimming video data of the swimmer through the underwater camera.

Edge layer (edge compute node) 200: preprocessing the video of the swimmer, and detecting drowning of the swimmer by using a trained lightweight human body posture estimation model Lightweight OpenPose.

Cloud layer 100: the method is used for managing the edge computing node 200, training Lightweight OpenPose a lightweight human body posture estimation model, updating model parameters and storing swimmer video data.

The structure of the drowning prevention early warning system utilizes a lightweight human body posture estimation model Lightweight OpenPose to mark key points of a swimming image. The common human body posture estimation model openPose is a posture estimation model widely applied, but the model body is too large, the parameters are too many, the requirement on hardware is high, and the instantaneity is to be improved. The Lightweight OpenPose model applied by the embodiment of the invention is based on the light improvement of the OpenPose model, compared with the 2-order OpenPose model, the parameter quantity is only 15%, but the precision is only reduced by 1%, and the method is suitable for a target detection scene with high real-time requirements.

The main improvement of the lightweight human body posture estimation model Lightweight OpenPose provided by the embodiment of the invention to the OpenPose model is as follows:

1) And replacing the VGG structure of the OpenPose model with the MobileNet structure with hole convolution.

2) Two branches of the openPose model (upper branch of the predicted human keypoint location confidence map (Part confidence maps, PCM) and lower branch of the affinity domain (Part Affinity Fields, PAF) of the predicted human keypoint) are merged into one branch.

3) The 7*7 convolution kernel of the openPose model is replaced with the convolution kernel of 3*3.

4) The Lightweight OpenPose model has only one refinement stage and the openelse model has 5 stages.

At cloud layer 100, the present invention uses the first fifty thousand samples of the COCO2017 dataset, according to the training set: verification set: test set = 8:1:1 ratio Lightweight OpenPose pose estimation model training, and deploying the trained Lightweight OpenPose model to edge computing nodes to perform real-time reasoning prediction on swimmer swimming data.

In order to achieve the above objective, referring to fig. 2, fig. 2 is a flow chart of a first drowning prevention early warning method based on a lightweight human body posture estimation model according to an embodiment of the present invention. As shown in FIG. 2, the drowning prevention early warning method based on the lightweight human body posture estimation model comprises the following steps:

S110: and acquiring swimming video data of the swimmer shot by the underwater camera in real time. As can be seen from the architecture shown in fig. 1, in the embodiment of the present invention, the terminal device (such as an underwater camera) is used to obtain the swimming video data of the swimmer in real time, and then the swimming video data is transmitted to the edge computing node through the local area network, and the neighborhood and training of the swimming video data are performed at the embedded device of the edge computing node, so as to mark the key points of the human body.

S120: intercepting swimming video data to obtain multi-frame swimming images, sequentially inputting the multi-frame swimming images into a lightweight human body posture estimation model to mark key points of human bodies, and obtaining coordinates of a plurality of key points of the human bodies.

The invention uses the lightweight human body posture estimation model to mark the key points of the human body, and the traditional human body posture estimation model OpenPose has the defects of too large model quantity, too many parameters, high requirement on hardware and high instantaneity to be improved. The lightweight human body posture estimation model Lightweight OpenPose is based on the lightweight improvement of the OpenPose model, and compared with the traditional 2-order OpenPose model, the lightweight human body posture estimation model Lightweight OpenPose has the advantages that the parameter quantity is only 15%, but the accuracy is only reduced by 1%, and the lightweight human body posture estimation model Lightweight OpenPose is suitable for a target detection scene with high real-time requirements. After the swimming video data are intercepted to obtain a plurality of frames of swimming images, the plurality of frames of swimming images are respectively input into the lightweight human body posture estimation model for marking, and each frame of swimming image is respectively marked to obtain coordinates of a plurality of human body key points. These human body key points include: human body key points such as nose, neck, right shoulder, right elbow, right wrist, left shoulder, left elbow, left wrist, right hip, right knee, right ankle, left hip, left knee, left ankle, right eye, left eye, right ear, left ear, and the like. By marking the key points of the human body, coordinates of the key points of different human bodies can be obtained, limb movements of the human body in a swimming state can be determined, and then the drowning state can be judged.

Specifically, as a preferred embodiment, as shown in fig. 3, step S120 described above: sequentially inputting a plurality of frames of swimming images into a lightweight human body posture estimation model for marking human body key points, and obtaining coordinates of a plurality of human body key points comprises the following steps:

s121: the training set, the verification set and the test set of the lightweight human body posture estimation model are divided according to a preset quantity proportion.

S122: and respectively inputting the training set, the verification set and the test set into the lightweight human body posture estimation model according to a preset quantity proportion to train, verify and test to obtain the lightweight human body posture estimation model passing the test.

Wherein, the first fifty thousand samples of the COCO2017 dataset can be used at the cloud layer according to the training set: verification set: test set = 8:1:1 ratio, training the lightweight human body posture estimation model Lightweight OpenPose, deploying the trained Lightweight OpenPose model to an edge computing node, and performing real-time reasoning and prediction on swimming images of a swimmer at the edge computing node to obtain coordinates of the plurality of human body key points.

S123: and inputting the swimming image into a MobileNet model of the lightweight human body posture estimation model for convolution to obtain a swimming feature map containing a plurality of human body key points. Because the lightweight human body posture estimation model uses the MobileNet structure with cavity convolution to replace the VGG structure of the OpenPose model, the lightweight human body posture estimation model can obtain swimming feature images of as many human body key points by less convolution, and the human body key points are marked in the swimming feature images.

S124: and inputting the swimming feature map into a single-branch convolution neural network of the lightweight human body posture estimation model for convolution.

S125: a single-branch convolutional neural network is used to predict a position confidence map and an affinity domain for each of a plurality of human keypoints. The lightweight human body posture estimation model combines two branches of a traditional OpenPose model into one branch, the traditional OpenPose model comprises an upper branch used for predicting a human body key point position degree map and a lower branch used for predicting an affinity domain of a human body key point, so that a single-branch convolutional neural network of the lightweight human body posture estimation model can combine the advantages of the two branches, and the position confidence degree map and the affinity domain of each human body key point in the plurality of human body key points are respectively predicted by using fewer parameters and structures.

S126: and matching the position confidence coefficient map of each human body key point with the affinity domain to obtain coordinates of a plurality of human body key points. The lightweight human body posture estimation model provided by the embodiment of the invention can replace 7*7 convolution kernels of the OpenPose model by using the convolution kernels of 3*3, so that fewer convolution kernels are used for operation, the position confidence coefficient map and the affinity domain are matched, and the coordinates of the plurality of human body key points are obtained. The multiple human body key points extracted by the lightweight human body posture estimation model are shown in fig. 8, wherein the figure comprises 18 human body key points, which are respectively: 0-nose, 1-neck, 2-right shoulder, 3-right elbow, 4-right wrist, 5-left shoulder, 6-left elbow, 7-left wrist, 8-right hip, 9-right knee, 10-right ankle, 11-left hip, 12-left knee, 13-left ankle, 14-right eye, 15-left eye, 16-right ear and 17-left ear.

According to the technical scheme provided by the embodiment of the invention, the lightweight human body posture estimation model passing the test is obtained by training, verifying and testing the lightweight human body posture estimation model, and the lightweight human body posture estimation model is used for convolving the swimming image to obtain the swimming feature images of a plurality of human body key points, so that the limb action features of a swimmer during swimming are determined. And then, a single-branch convolutional neural network model is used for convolution to obtain a position confidence coefficient map and an affinity domain, and then the position confidence coefficient map and the affinity domain are matched to obtain coordinates of a plurality of human body key points, so that the purpose of extracting the coordinates of a plurality of human body key points in a swimming state is achieved with a smaller structure, fewer parameters and higher precision of the model.

The drowning prevention early warning method provided in the embodiment shown in fig. 2 further includes the following steps after obtaining coordinates of a plurality of key points of the human body:

s130: according to the relation among coordinates of a plurality of human body key points, calculating to obtain key point feature vectors of the human body key points, wherein the key point feature vectors comprise key point distance vectors, key point included angle vectors and key point distance change speed vectors.

The trained lightweight human body posture estimation model LightvolOpenPose is utilized to estimate the human body posture of the swimmer, 18 obtained key point coordinates are shown in fig. 8, and a key point coordinate set is constructed as follows: p= { (x) ₀ ,y ₀ ),(x ₁ ,y ₁ ),...(x ₁₇ ,y ₁₇ ) And then calculating the key point distance vector, the key point included angle vector and the key point distance change speed vector according to the coordinates of the key points of the human body. The key point feature vector reflects the limb motion of the swimmer when swimming, and whether the swimmer is drowned or not can be accurately determined by analyzing the limb motion of the swimmer and comparing the limb motion with the limb motion in a drowned state.

Specifically, as a preferred embodiment, as shown in fig. 4, in the above-mentioned anti-drowning early warning method, step S130: according to the relation among coordinates of a plurality of human body key points, the step of calculating the key point feature vector of the human body key points comprises the following steps:

s131: a first reference keypoint is selected from a plurality of human body keypoints. The human body characteristics are combined, so that the movement amplitude of the neck of the swimmer is small when the swimmer swims, the neck key point is used as a first reference key point, and the actions of the swimmer can be described by using the distances between the other 17 key points and the neck key point.

S132: and respectively calculating the distances between all other human body key points and the first reference key point according to the coordinates of the other human body key points and the coordinates of the first reference key point in the plurality of human body key points to obtain a key point distance vector. Specifically, the neck key point is taken as a first reference key point, the distances between the other 17 human body key points and the neck key point are used for describing the action of the swimmer, and the distance between the other human body key points and the neck reference point is calculated according to the following formula:

(number 1 with the neck key removed)

The distance vector d= [ d ] of the other 17 key points and the neck key point can be obtained through the distance calculation formula ₁ ,d ₂ ,...d ₁₇ ]At this time d ₁ Represents the distance from the key point 0 to the neck key point d _i (i.gtoreq.2) represents the distance from the key point i to the neck key point.

S133: a plurality of second reference keypoints are selected from a plurality of human body keypoints. Many of the selected plurality of human body keypoints are bendable, such as left shoulder, left elbow, left hip, left knee, right shoulder, right elbow, right hip, and right knee, and these bendable keypoints are selected as the second reference keypoints, thereby constructing a keypoint angle vector using the angles of these second reference keypoints.

S134: and calculating the included angles of the plurality of second reference key points according to the coordinates of the plurality of second reference key points to obtain a key point included angle vector.

Specifically, calculating the included angles of key points at the left shoulder, the left elbow, the left hip, the left knee, the right shoulder, the right elbow, the right hip and the right knee, and constructing a key point included angle vector theta= [ theta ] ₅ ,θ ₆ ,θ ₁₁ ,θ ₁₂ ,θ ₂ ,θ ₃ ,θ ₈ ,θ ₉ ]. Taking the included angle at the right shoulder key point 2 as an example, the calculation formula of the key point included angle vector of the right shoulder key point is as follows:

through the key point included angle vector calculation formula, all included angle vectors of the second reference key can be obtained, so that data reference is provided for drowning judgment.

S135: and extracting the key point distance difference of the front and rear frames of swimming images from the multi-frame swimming images according to the coordinates of the key points of the human bodies.

S136: and calculating to obtain a key point distance change speed vector according to the key point distance difference of the front and rear frames of swimming images and the interval time of the front and rear frames of swimming images.

In the technical scheme provided by the embodiment of the invention, the swimming speed change of the swimmer is an important basis for drowning judgment, so that the key point distance difference of the front and rear frames of swimming images is extracted from the multi-frame swimming images, and the key point distance change speed of the front and rear frames of swimming images can be calculated according to the interval time of the front and rear frames of swimming images. Specifically, the calculation formula of the key point distance change speed vector is as follows:

Wherein deltas is the difference of the distances between key points in the front and rear frame images, and t is the interval time between the front and rear frame images. Constructing a key point distance change speed vector v= [ v ] ₁ ,v ₂ ,...v ₁₇ ]. The coordinate movement of the neck in the front frame image and the back frame image is not large, so that the key point distance difference is calculated by taking the neck key point as a reference, and the calculated key point distance difference is the distance from the other 17 key points to the neck key point. Similarly, the key point included angle vector and the key point distance change speed vector are not affected by the image scene change.

S137: and constructing a key point feature vector by using the key point distance vector, the key point included angle vector and the key point distance change speed vector.

The same swimming image comprises the key point distance vector, the key point included angle vector and the key point distance change speed vector, so that the vectors are packed into the same set, and the set f of key point feature vectors can be obtained _i ＝{d _i ，θ _i ，v _i }。

The drowning prevention early warning method provided in the embodiment shown in fig. 2 further includes the following steps after calculating the key point feature vector of the key point of the human body:

s140: judging whether the drowning similarity between the multi-frame swimming image and a key point distance vector, a key point included angle vector and a key point distance change speed vector of any drowning state judging set in the multiple drowning state judging sets is larger than or equal to a preset similarity threshold value;

The drowning similarity is the similarity between the key point distance vector, the key point included angle vector and the key point distance change speed vector in each frame of swimming image and the key point distance vector, the key point included angle vector and the key point distance change speed vector in the drowning state judging set, the higher the similarity is, the greater the drowning probability of the swimmer is indicated, and when the drowning similarity is greater than or equal to the respective preset similarity threshold value, the swimmer can be judged to be in the drowning state.

Specifically, as a preferred embodiment, as shown in fig. 5, the step of determining whether the drowning similarity of the key point distance vector, the key point included angle vector, and the key point distance change speed vector in the multi-frame swimming image and the key point distance vector, the key point included angle vector, and the key point distance change speed vector of any one of the multiple drowning state determination sets is greater than or equal to the respective preset similarity threshold value includes:

s141: according to the cosine similarity calculation formula,

respectively calculating the drowning similarity of the key point distance vector, the key point included angle vector and the key point distance change speed vector in the swimming image and the key point distance vector, the key point included angle vector and the key point distance change speed vector of any drowning state judgment set; wherein A is a key point feature vector in a swimming image, and B is a key point feature vector of a drowning state judgment set.

S142: judging whether the drowning similarity of the key point distance vector, the key point included angle vector and the key point distance change speed vector is larger than or equal to the respective preset similarity threshold value.

S143: if the drowning similarity of the key point distance vector, the key point included angle vector and the key point distance change speed vector is larger than or equal to the respective preset similarity threshold value, determining that the swimmer drowns.

According to the technical scheme provided by the embodiment of the invention, the cosine similarity is used as the drowning similarity, and the cosine similarity describes the similarity of the two vectors in the direction and is irrelevant to the length of the vectors, so that the drowning behaviors of different crowds can be accurately determined by using the cosine similarity. When the drowning similarity of the key point distance vector, the key point included angle vector and the key point distance change speed vector is larger than or equal to the respective preset similarity threshold (for example, the similarity of each vector is 0.6), the drowning of the swimmer can be determined, and at the moment, a drowning alarm signal is sent to implement rescue.

In addition, before calculating the drowning similarity, a plurality of drowning states are selected to form a drowning state judging set, and the key point distance vector, the key point included angle vector and the key point distance change speed vector corresponding to each drowning state are estimated by using the lightweight human body posture estimating model.

Specifically, as a preferred embodiment, as shown in fig. 6, before the step of determining whether the drowning similarity between the key point distance vector, the key point included angle vector, and the key point distance change speed vector in the multi-frame swimming image and the key point distance vector, the key point included angle vector, and the key point distance change speed vector of any one of the multiple drowning state determination sets is greater than or equal to the respective preset similarity threshold, the method further includes:

s210: and selecting multiple drowning state data sets, wherein each drowning state data set in the multiple drowning state data sets comprises a plurality of continuous frames of drowning images. According to the embodiment of the invention, 13 drowning states are selected, each drowning state data set comprises 20 frames of drowning images, human body key point coordinates corresponding to the drowning states are conveniently extracted from the 20 frames of drowning images, and key point feature vectors related to the drowning states are analyzed through the human body key point coordinates.

S220: and inputting continuous multi-frame drowning images contained in each drowning state data set into a lightweight human body posture estimation model to obtain human body key point coordinates corresponding to each drowning state. The lightweight human body posture estimation model can extract key points and coordinate marks of the multi-frame drowning image, so that coordinates of all human body key points corresponding to each drowning state in the 13 drowning states are obtained.

S230: and according to the coordinates of the human body key points corresponding to each drowning state, respectively calculating to obtain key point feature vectors of the human body key points corresponding to each drowning state, wherein the key point feature vectors comprise key point distance vectors, key point included angle vectors and key point distance change speed vectors.

The calculation formulas of the key point distance vector, the key point included angle vector and the key point distance change speed vector corresponding to each drowning state are the same as those of the key point distance vector, the key point included angle vector and the key point distance change speed vector of the human body key point corresponding to the swimming image, and are not repeated herein.

S240: and constructing and obtaining a plurality of drowning state judgment sets by using the key point feature vectors of the human body key points corresponding to each drowning state.

Calculation of the key point distance vector: specifically, since the variation range of the neck key point is small during the movement of the swimmer, the neck key point is used as a reference point for each drowning state, the movement of the swimmer is described by using the distances between the other 17 key points and the neck reference point, and the calculation formula of the distances between the other key points and the neck reference point is as follows:

(culling keypoints 1)

Distance vector d= [ d ] of other 17 key points and neck reference point ₁ ,d ₂ ,...d ₁₇ ]At this time d ₁ Represents the distance from the key point 0 to the neck reference point d _i (i.gtoreq.2) represents the distance from the key point i to the neck reference point.

Calculating the included angle vector of the key points: specifically, the key point included angles at the left shoulder, the left elbow, the left hip, the left knee, the right shoulder, the right elbow, the right hip and the right knee are calculated, and a key point included angle vector theta= [ theta ] is constructed ₅ ,θ ₆ ,θ ₁₁ ,θ ₁₂ ,θ ₂ ,θ ₃ ,θ ₈ ,θ ₉ ]Taking the included angle at the key point 2 of the right shoulder as an example, the calculation formula of the key point included angle vector is as follows:

calculation of a key point distance change speed vector:

the swimming speed change of the swimmer is also an important basis for drowning judgment, so that the distance change speed of the key points of the front and rear frame images of the drowned swimmer is calculated according to the following calculation formula of the distance change speed vector of the key points:

wherein deltas is the difference of the distances between key points of the two frames of drowning images, and t is the interval time of the two frames of drowning images. Constructing a key point distance change speed vector v= [ v ] ₁ ,v ₂ ,...v ₁₇ ]。

In order to describe the action change process in the drowning process of the swimmer, continuously calculating key point distance vectors, key point included angle vectors and front and back key point distance change speed vectors of 13 continuous 20 frames of images (capturing one image every 10 frames of video data) in a drowning state, and constructing 13 drowning judgment sets

F _i ＝{f ₁ ,f ₂ ,...f ₂₀ E {1,2,..13 }; wherein f _i The method is a set formed by a key point distance vector, a key point included angle vector and a key point distance change speed vector of a front frame and a rear frame of an ith picture.

The drowning prevention early warning method provided in the embodiment shown in fig. 2 further includes the following steps after determining that the drowning similarity algorithm is greater than or equal to the respective preset similarity threshold value:

s150: if the drowning similarity of the key point distance vector, the key point included angle vector and the key point distance change speed vector is larger than or equal to the respective preset similarity threshold value, determining that the swimmer drowns, and sending a drowning early warning signal.

In addition, if the drowning similarity of any one of the key point distance vector, the key point included angle vector and the key point distance change speed vector is smaller than a preset similarity threshold value, the key point distance vector, the key point included angle vector and the key point distance change speed vector of the key points of the human body in the multi-frame swimming image are recalculated. And re-executing the step of judging whether the drowning similarity of the key point distance vector, the key point included angle vector and the key point distance change speed vector in the multi-frame swimming image and the key point distance vector, the key point included angle vector and the key point distance change speed vector of any drowning state judgment set in the multiple drowning state judgment sets is greater than or equal to the respective preset similarity threshold value or not until the fact that the swimmer drowns or the swimmer stops swimming is determined.

In summary, according to the drowning prevention early warning method based on the lightweight human body posture estimation model provided by the embodiment of the invention, through acquiring swimming video data of a swimmer shot by an underwater camera in real time, intercepting the swimming video data to obtain a plurality of frames of swimming images, and then sequentially inputting the plurality of frames of swimming images into the lightweight human body posture estimation model for marking key points of a human body, the coordinates of a plurality of key points of the human body can be obtained. According to the technical scheme, the drowning detection is performed by using the lightweight human body posture estimation model, and the model is small in size and high in speed, and can meet the precision requirement; in addition, the lightweight human body posture estimation model is used for marking to obtain a plurality of human body key points, the human body key points have strong correlation with swimming states, and drowning behaviors can be accurately detected. Calculating a key point feature vector of the key points of the human body by using the relation among coordinates of the key points of the human body, wherein the key point feature vector comprises a key point distance vector, a key point included angle vector and a key point distance change speed vector, and can accurately detect the swimming action of a swimmer to be used as a drowning judgment standard; and judging whether the drowning similarity of the key point distance vector, the key point included angle vector and the key point distance change speed vector in the multi-frame swimming image and the key point distance vector, the key point included angle vector and the key point distance change speed vector of any drowning state judgment set in the multiple drowning state judgment sets is larger than or equal to the respective preset similarity threshold value or not, so that the drowning behaviors of different swimming groups can be accurately detected. When the drowning similarity of the feature vectors of the key points is larger than or equal to the respective preset similarity threshold value, the drowning of the swimmer can be determined, so that a drowning early warning signal is sent, and the accuracy of the drowning judgment can be improved by using a simple deep learning model.

In addition, referring to fig. 7, fig. 7 is a flow chart of a second drowning prevention early warning method based on a lightweight human body posture estimation model according to an embodiment of the present invention. As can be seen from the architectural diagram of the anti-drowning early warning system shown in fig. 1, the anti-drowning early warning method includes:

s310: the terminal layer (underwater camera) collects swim data of the swimmer.

S320: the edge computing node preprocesses the data, marks the key points of the human body on the intercepted video frames of the swimmer by using a trained Lightweight OpenPose model, and obtains 18 key point coordinates.

S330: and calculating the key point distance, the key point included angle and the key point distance change speed of the swimmer.

S340: and comparing the intercepted swimmer image with 13 drowned images frame by using the key point distance vector, the key point included angle vector and the key point distance change speed vector.

S350: judging whether the cosine similarity of the continuous 20 frames of images is more than or equal to 0.6; if yes, go to step S360; if not, the process returns to step S310.

S360: and judging that the vehicle is drowned, and carrying out rescue.

In addition, based on the same concept of the above method embodiment, the embodiment of the present invention further provides an anti-drowning early warning system based on a lightweight human body posture estimation model, which is used for implementing the above method of the present invention, and because the principle of solving the problem of the system embodiment is similar to that of the method, the system embodiment has at least all the beneficial effects brought by the technical solution of the above embodiment, and will not be described in detail herein.

Referring to fig. 9, fig. 9 is a schematic structural diagram of an anti-drowning early warning system based on a lightweight human body posture estimation model. As shown in fig. 9, the drowning prevention early warning system includes:

the swimming data acquisition module 110 is used for acquiring swimming video data of a swimmer shot by the underwater camera in real time;

the swimming data intercepting module 120 is used for intercepting swimming video data to obtain multi-frame swimming images;

the key point marking module 130 is configured to sequentially input multiple frames of swimming images to the lightweight human body posture estimation model for marking key points of the human body, so as to obtain coordinates of multiple key points of the human body;

the feature vector calculation module 140 is configured to calculate a key point feature vector of a key point of the human body according to a relationship between coordinates of a plurality of key points of the human body, where the key point feature vector includes a key point distance vector, a key point included angle vector, and a key point distance change speed vector;

the similarity judging module 150 is configured to judge whether the drowning similarity between the key point distance vector, the key point included angle vector, and the key point distance change speed vector in the multi-frame swimming image and the key point distance vector, the key point included angle vector, and the key point distance change speed vector of any one of the multiple drowning state judging sets is greater than or equal to the respective preset similarity threshold;

The early warning signal sending module 160 is configured to determine that the swimmer is drowned and send a drowned early warning signal if the similarity determining module determines that the drowned similarity of the key point distance vector, the key point included angle vector, and the key point distance change speed vector is greater than or equal to the respective preset similarity threshold.

In summary, in the drowning prevention early warning system based on the lightweight human body posture estimation model provided by the embodiment of the invention, the swimming data acquisition module 110 acquires the swimming video data of the swimmer shot by the underwater camera in real time, the swimming data interception module 120 intercepts the swimming video data to obtain a plurality of frames of swimming images, and the key point marking module 130 sequentially inputs the plurality of frames of swimming images into the lightweight human body posture estimation model to mark key points of the human body, so that coordinates of a plurality of key points of the human body can be obtained. According to the technical scheme, the drowning detection is performed by using the lightweight human body posture estimation model, and the model is small in size and high in speed, and can meet the precision requirement; in addition, the lightweight human body posture estimation model is used for marking to obtain a plurality of human body key points, the human body key points have strong correlation with swimming states, and drowning behaviors can be accurately detected. The feature vector calculation module 140 calculates a key point feature vector of the key point of the human body by using the relation among coordinates of the plurality of key points of the human body, wherein the key point feature vector comprises a key point distance vector, a key point included angle vector and a key point distance change speed vector, and can accurately detect the swimming action of a swimmer as a drowning judgment standard; the similarity judging module 150 judges whether the drowning similarity of the key point distance vector, the key point included angle vector and the key point distance change speed vector in the multi-frame swimming image and the key point distance vector, the key point included angle vector and the key point distance change speed vector of any drowning state judging set in the multiple drowning state judging sets is greater than or equal to the respective preset similarity threshold value, so that the drowning behaviors of different swimming populations can be accurately detected. The early warning signal sending module 160 can determine that the swimmer is drowned when determining that the drowned similarity of the feature vectors of the key points is greater than or equal to the respective preset similarity threshold value, thereby sending a drowned early warning signal, and improving the accuracy of the drowned determination by using a simple deep learning model.

Specifically, as a preferred embodiment, as shown in fig. 10, in the above-mentioned anti-drowning warning system, the key point marking module 120 includes:

a data set dividing submodule 121 for dividing a training set, a verification set and a test set of the lightweight human body posture estimation model according to a predetermined number proportion;

the network training test sub-module 122 is configured to input a training set, a verification set and a test set into the lightweight human body posture estimation model according to a predetermined number proportion, respectively, to perform training, verification and testing, so as to obtain a lightweight human body posture estimation model passing the test;

the model convolution sub-module 123 is configured to input a swimming image into a MobileNet model of the lightweight human body posture estimation model for convolution, so as to obtain a swimming feature map including a plurality of human body key points;

the neural network convolution sub-module 124 is used for inputting the swimming feature map into a single-branch convolution neural network of the lightweight human body posture estimation model for convolution;

a keypoint prediction sub-module 125 for predicting a position confidence map and an affinity domain for each of a plurality of human keypoints using a single-branch convolutional neural network;

the key point matching submodule 126 is configured to match the position confidence map and the affinity domain of each human key point, so as to obtain coordinates of a plurality of human key points.

Specifically, as a preferred embodiment, as shown in fig. 11, in the above-mentioned anti-drowning early warning system, the feature vector calculating module 140 includes:

a first key point selecting sub-module 141, configured to select a first reference key point from a plurality of human body key points;

the key point distance calculating sub-module 142 is configured to calculate distances between all other human body key points and the first reference key point according to the coordinates of the other human body key points and the coordinates of the first reference key point in the plurality of human body key points, respectively, to obtain a key point distance vector;

a second key point selecting sub-module 143, configured to select a plurality of second reference key points from a plurality of human body key points;

the key point included angle calculating sub-module 144 is configured to calculate included angles of the plurality of second reference key points according to coordinates of the plurality of second reference key points, so as to obtain a key point included angle vector;

the key point distance difference extracting sub-module 145 is configured to extract a key point distance difference of two frames of swimming images from the multiple frames of swimming images according to coordinates of a plurality of key points of the human body;

the distance change speed vector calculation operator module 146 is configured to calculate a distance change speed vector of the key point according to the distance difference of the key point of the swimming images of the front frame and the rear frame and the interval time of the swimming images of the front frame and the rear frame;

The key point feature vector construction submodule 147 is configured to construct a key point feature vector using a key point distance vector, a key point included angle vector, and a key point distance change speed vector.

Specifically, as a preferred embodiment, as shown in fig. 12, in the above-mentioned anti-drowning early warning system, the similarity determining module 150 includes:

the drowning similarity calculation submodule 151 is configured to calculate the formula according to the cosine similarity:

the drowning similarity determination sub-module 152 is configured to determine whether the drowning similarity of the key point distance vector, the key point included angle vector, and the key point distance change speed vector is greater than or equal to the respective preset similarity threshold.

In addition, as a preferred embodiment, in the above drowning prevention early warning system, the similarity determining module 150 is further configured to: selecting multiple drowning state data sets, wherein each drowning state data set in the multiple drowning state data sets comprises a plurality of continuous frames of drowning images; inputting continuous multi-frame drowning images contained in each drowning state data set into the lightweight human body posture estimation model to obtain human body key point coordinates corresponding to each drowning state; according to the human body key point coordinates corresponding to each drowning state, respectively calculating to obtain key point feature vectors of the human body key points corresponding to each drowning state, wherein the key point feature vectors comprise a key point distance vector, a key point included angle vector and a key point distance change speed vector; and constructing the multiple drowning state judging sets by using the key point feature vectors of the human body key points corresponding to each drowning state.

In summary, compared with the prior art, the drowning prevention early warning scheme based on the lightweight human body posture estimation model provided by the embodiment of the invention has the following advantages:

(1) Compared with other drowning detection models, the lightweight human body posture estimation model selected by the embodiment of the invention utilizes the position relationship among the head, the human body and the water surface of the swimmer to carry out drowning judgment, and the drowning behavior can be detected more accurately by utilizing the key point distance, the key point included angle and the key point distance change speed of the swimmer.

(2) According to the embodiment of the invention, the multi-frame swimming image and the drowning image of the swimmer and various drowning states are utilized, and particularly, the cosine similarity of the key point distance vector, the key point included angle vector and the key point distance change speed vector of the 20 frames of drowning images corresponding to each of 13 drowning states of the 20 frames of swimming images are taken as the drowning judgment standard, so that the drowning behaviors of different people can be effectively judged.

(3) Compared with other heavy swimming pool drowning detection deep learning models, the invention utilizes the lightweight human body posture estimation model Lightweight OpenPose model to carry out drowning detection, and the model has small volume and high speed and meets the precision requirement. In addition, the cloud edge cooperative technology is utilized to deploy the Lightweight OpenPose model, the heavy deep learning model training process is placed in the cloud, and the drowning detection of the swimming pool is only carried out on the edge layer, so that the real-time performance of the drowning detection model is further improved.

It will be appreciated by those skilled in the art that embodiments of the present invention may be provided as a method, system, or computer program product. Accordingly, the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present invention may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, and the like) having computer-usable program code embodied therein.

The present invention is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems) and computer program products according to embodiments of the invention. It will be understood that each flow and/or block of the flowchart illustrations and/or block diagrams, and combinations of flows and/or blocks in the flowchart illustrations and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.

These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means which implement the function specified in the flowchart flow or flows and/or block diagram block or blocks.

These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.

It should be noted that in the claims, any reference signs placed between parentheses shall not be construed as limiting the claim. The word "comprising" does not exclude the presence of elements or steps not listed in a claim. The word "a" or "an" preceding an element does not exclude the presence of a plurality of such elements. The invention may be implemented by means of hardware comprising several distinct elements, and by means of a suitably programmed computer. In the unit claims enumerating several means, several of these means may be embodied by one and the same item of hardware. The use of the words first, second, third, etc. do not denote any order. These words may be interpreted as names.

While preferred embodiments of the present invention have been described, additional variations and modifications in those embodiments may occur to those skilled in the art once they learn of the basic inventive concepts. It is therefore intended that the following claims be interpreted as including the preferred embodiments and all such alterations and modifications as fall within the scope of the invention.

It will be apparent to those skilled in the art that various modifications and variations can be made to the present invention without departing from the spirit or scope of the invention. Thus, it is intended that the present invention also include such modifications and alterations insofar as they come within the scope of the appended claims or the equivalents thereof.

Claims

1. The drowning prevention early warning method based on the lightweight human body posture estimation model is characterized by comprising the following steps of:

intercepting the swimming video data to obtain multi-frame swimming images, sequentially inputting the multi-frame swimming images into a lightweight human body posture estimation model to mark key points of human bodies, and obtaining coordinates of a plurality of key points of the human bodies;

according to the relation among the coordinates of the plurality of human body key points, calculating to obtain key point feature vectors of the human body key points, wherein the key point feature vectors comprise key point distance vectors, key point included angle vectors and key point distance change speed vectors;

Judging whether the drowning similarity of the key point distance vector, the key point included angle vector and the key point distance change speed vector in the multi-frame swimming image and the key point distance vector, the key point included angle vector and the key point distance change speed vector of any drowning state judgment set in a plurality of drowning state judgment sets is larger than or equal to the respective preset similarity threshold value;

and if the drowning similarity of the key point distance vector, the key point included angle vector and the key point distance change speed vector is larger than or equal to the respective preset similarity threshold value, determining that the swimmer drowns, and sending a drowning early warning signal.

2. The drowning prevention early warning method according to claim 1, wherein the step of sequentially inputting the multi-frame swimming images to a lightweight human body posture estimation model for human body key point marking to obtain coordinates of a plurality of human body key points comprises:

respectively inputting the training set, the verification set and the test set into the lightweight human body posture estimation model according to the preset quantity proportion to train, verify and test to obtain the lightweight human body posture estimation model which passes the test;

Inputting the swimming image into a MobileNet model of the lightweight human body posture estimation model for convolution to obtain a swimming feature map containing the plurality of human body key points;

inputting the swimming feature map to a single-branch convolution neural network of the lightweight human body posture estimation model for convolution;

predicting a position confidence map and an affinity domain for each human keypoint of the plurality of human keypoints using the single-branch convolutional neural network;

and matching the position confidence coefficient map of each human body key point with the affinity domain to obtain the coordinates of the plurality of human body key points.

3. The drowning prevention early warning method according to claim 1, wherein the step of calculating a key point feature vector of the key points of the human body according to the relation between coordinates of the key points of the human body comprises:

selecting a first reference key point from the plurality of human body key points;

according to the coordinates of other human body key points and the coordinates of the first reference key point in the plurality of human body key points, respectively calculating the distances between all other human body key points and the first reference key point to obtain the key point distance vector;

calculating included angles of the plurality of second reference key points according to the coordinates of the plurality of second reference key points to obtain included angle vectors of the key points;

calculating to obtain the key point distance change speed vector according to the key point distance difference of the front and rear swimming images and the interval time of the front and rear swimming images;

and constructing the key point feature vector by using the key point distance vector, the key point included angle vector and the key point distance change speed vector.

4. The drowning prevention early warning method according to claim 1 or 3, wherein the step of determining whether the drowning similarity of the key point distance vector, the key point included angle vector, and the key point distance change speed vector in the multi-frame swimming image to the key point distance vector, the key point included angle vector, and the key point distance change speed vector of any one of the plurality of drowning state determination sets is greater than or equal to respective preset similarity thresholds, respectively, comprises:

According to a cosine similarity calculation formula:

respectively calculating the distances between the key points, the included angles between the key points and the change speeds; wherein A is a key point feature vector in a swimming image, and B is a drowning state judgmentA set of keypoint feature vectors;

and if the drowning similarity of the key point distance vector, the key point included angle vector and the key point distance change speed vector is larger than or equal to the respective preset similarity threshold value, determining that the swimmer drowns.

5. The method of claim 4, wherein before the step of determining whether the drowning similarity of the key point distance vector, the key point included angle vector, and the key point distance change speed vector in the multi-frame swimming image to any one of the plurality of drowning state determination sets is greater than or equal to the respective preset similarity threshold, the method further comprises:

inputting continuous multi-frame drowning images contained in each drowning state data set into the lightweight human body posture estimation model to obtain human body key point coordinates corresponding to each drowning state;

according to the human body key point coordinates corresponding to each drowning state, respectively calculating to obtain key point feature vectors of the human body key points corresponding to each drowning state, wherein the key point feature vectors comprise a key point distance vector, a key point included angle vector and a key point distance change speed vector;

and constructing the multiple drowning state judging sets by using the key point feature vectors of the human body key points corresponding to each drowning state.

6. The drowning prevention warning method according to claim 1, wherein after the step of determining whether the drowning likeness of the key point distance vector, the key point included angle vector, and the key point distance change speed vector is greater than or equal to the respective preset likeness threshold, the method further comprises:

if the drowning similarity of any one of the key point distance vector, the key point included angle vector and the key point distance change speed vector is smaller than the preset similarity threshold value, recalculating the key point distance vector, the key point included angle vector and the key point distance change speed vector of the human key point in the multi-frame swimming image;

7. Anti-drowning early warning system based on lightweight human body posture estimation model, characterized by comprising:

the swimming data intercepting module is used for intercepting the swimming video data to obtain multi-frame swimming images;

the key point marking module is used for sequentially inputting the multi-frame swimming images into the lightweight human body posture estimation model to mark the key points of the human body so as to obtain coordinates of a plurality of key points of the human body;

the feature vector calculation module is used for calculating a key point feature vector of the human body key point according to the relation among coordinates of the plurality of human body key points, wherein the key point feature vector comprises a key point distance vector, a key point included angle vector and a key point distance change speed vector;

The similarity judging module is used for judging whether the drowning similarity between the key point distance vector, the key point included angle vector and the key point distance change speed vector in the multi-frame swimming image and the key point distance vector, the key point included angle vector and the key point distance change speed vector of any drowning state judging set in the multiple drowning state judging sets is larger than or equal to the respective preset similarity threshold value;

and the early warning signal sending module is used for determining that the swimmer drowns and sending a drowning early warning signal if the similarity judging module judges that the drowning similarity of the key point distance vector, the key point included angle vector and the key point distance change speed vector is larger than or equal to the respective preset similarity threshold value.

8. The anti-drowning warning system of claim 7, wherein the keypoint tagging module comprises:

the data set dividing submodule is used for dividing a training set, a verification set and a test set of the lightweight human body posture estimation model according to a preset number proportion;

the network training test sub-module is used for respectively inputting the training set, the verification set and the test set into a lightweight human body posture estimation model according to the preset quantity proportion to train, verify and test to obtain the lightweight human body posture estimation model which passes the test;

The model convolution sub-module is used for inputting the swimming image into a MobileNet model of the lightweight human body posture estimation model for convolution to obtain a swimming feature map containing the plurality of human body key points;

the neural network convolution sub-module is used for inputting the swimming feature map into the single-branch convolution neural network of the lightweight human body posture estimation model for convolution;

a key point prediction sub-module, configured to predict a position confidence map and an affinity domain of each human key point in the plurality of human key points using the single-branch convolutional neural network;

and the key point matching submodule is used for matching the position confidence coefficient map of each human key point with the affinity domain to obtain the coordinates of the plurality of human key points.

9. The anti-drowning warning system of claim 7, wherein the feature vector calculation module comprises:

the first key point selecting sub-module is used for selecting a first reference key point from the plurality of human body key points;

the key point distance calculation sub-module is used for respectively calculating the distances between all other human body key points and the first reference key point according to the coordinates of the other human body key points and the coordinates of the first reference key point in the plurality of human body key points to obtain the key point distance vector;

The second key point selecting sub-module is used for selecting a plurality of second reference key points from the plurality of human body key points;

the key point included angle calculation sub-module is used for calculating included angles of the plurality of second reference key points according to coordinates of the plurality of second reference key points to obtain the key point included angle vector;

the key point distance difference extraction sub-module is used for extracting the key point distance difference of the front frame swimming image and the rear frame swimming image from the multi-frame swimming image according to the coordinates of the plurality of human body key points;

the distance change speed vector calculation operator module is used for calculating the distance change speed vector of the key point according to the distance difference of the key point of the front frame swimming image and the rear frame swimming image and the interval time of the front frame swimming image and the rear frame swimming image;

and the key point feature vector construction submodule is used for constructing the key point feature vector by using the key point distance vector, the key point included angle vector and the key point distance change speed vector.

10. The anti-drowning warning system according to claim 7 or 9, wherein the similarity determination module includes:

drowning similarity calculation submodule for calculating similarity according to cosine The degree calculation formula:

respectively calculating the distances between the key points, the included angles between the key points and the change speeds; wherein A is a key point feature vector in a swimming image, and B is a key point feature vector of a drowning state judgment set;

and the drowning similarity judging sub-module is used for judging whether the drowning similarity of the key point distance vector, the key point included angle vector and the key point distance change speed vector is larger than or equal to the respective preset similarity threshold value.