CN116189311A - Protective clothing wears standardized flow monitoring system - Google Patents
Protective clothing wears standardized flow monitoring system Download PDFInfo
- Publication number
- CN116189311A CN116189311A CN202310464437.XA CN202310464437A CN116189311A CN 116189311 A CN116189311 A CN 116189311A CN 202310464437 A CN202310464437 A CN 202310464437A CN 116189311 A CN116189311 A CN 116189311A
- Authority
- CN
- China
- Prior art keywords
- action
- skeleton
- module
- protective clothing
- network
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
- G06V40/20—Movements or behaviour, e.g. gesture recognition
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
- G06V10/74—Image or video pattern matching; Proximity measures in feature spaces
- G06V10/761—Proximity, similarity or dissimilarity measures
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
- G06V10/764—Arrangements for image or video recognition or understanding using pattern recognition or machine learning using classification, e.g. of video objects
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
- G06V10/766—Arrangements for image or video recognition or understanding using pattern recognition or machine learning using regression, e.g. by projecting features on hyperplanes
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
- G06V10/82—Arrangements for image or video recognition or understanding using pattern recognition or machine learning using neural networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V20/00—Scenes; Scene-specific elements
- G06V20/40—Scenes; Scene-specific elements in video content
- G06V20/46—Extracting features or characteristics from the video content, e.g. video fingerprints, representative shots or key frames
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V20/00—Scenes; Scene-specific elements
- G06V20/50—Context or environment of the image
- G06V20/52—Surveillance or monitoring of activities, e.g. for recognising suspicious objects
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
- G06V40/10—Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02P—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN THE PRODUCTION OR PROCESSING OF GOODS
- Y02P90/00—Enabling technologies with a potential contribution to greenhouse gas [GHG] emissions mitigation
- Y02P90/30—Computing systems specially adapted for manufacturing
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Multimedia (AREA)
- General Health & Medical Sciences (AREA)
- Health & Medical Sciences (AREA)
- Evolutionary Computation (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Software Systems (AREA)
- Artificial Intelligence (AREA)
- Computing Systems (AREA)
- Medical Informatics (AREA)
- Databases & Information Systems (AREA)
- Human Computer Interaction (AREA)
- Social Psychology (AREA)
- Psychiatry (AREA)
- Life Sciences & Earth Sciences (AREA)
- Biomedical Technology (AREA)
- Biophysics (AREA)
- Computational Linguistics (AREA)
- Data Mining & Analysis (AREA)
- Molecular Biology (AREA)
- General Engineering & Computer Science (AREA)
- Mathematical Physics (AREA)
- Image Analysis (AREA)
- Measuring And Recording Apparatus For Diagnosis (AREA)
Abstract
The invention belongs to the technical field of deep learning and behavior recognition, and discloses a protective clothing wearing standardized flow monitoring system, which comprises: the boundary dividing module is used for dividing the video into a plurality of video clips, and each video clip comprises a staged action in the protective clothing wearing standardized process; the skeleton detection module is used for detecting skeleton key points of the single-frame image and generating a plurality of skeleton structure images; the skeleton association module is used for connecting a plurality of skeleton key points with action association in the skeleton structure diagram according to the characteristics of the staged actions to generate a skeleton structure association diagram; the action discriminating module is used for identifying each staged action in the video segment to be detected according to the spatial information of the skeleton architecture association diagram, and matching the identification result with the corresponding staged action in the protective clothing wearing standardized process, so as to realize real-time monitoring and prompting on whether the medical staff meets the standardized wearing process in the protective clothing wearing process.
Description
Technical Field
The invention relates to the technical field of deep learning and behavior recognition, in particular to a protective clothing wearing standardized flow monitoring system.
Background
PPE is a variety of barrier articles used to protect personnel from infectious agents, for respiratory protection, head-to-face protection, body protection, foot protection, and mainly includes masks, gloves, goggles, face masks, waterproof aprons, barrier gowns, protective apparel, and the like. PPE is an important barrier to protect healthcare workers themselves, and its proper use and wear is an effective measure to prevent healthcare workers from infection.
Currently, the standards for protective clothing wear are WHO recommendation and chinese CDC recommendation. With reference to the two recommended standards, the standard wearing process of the protective clothing is as follows: the medical staff wears the split clothes and the rubber shoes to enter the clothes changing room, and wears the protective surface screen, the disposable shoe covers, the latex gloves, the protective surface screen, the disposable isolation clothes, the disposable waterproof boot sleeves and the second latex gloves in the clothes changing room once. In order to ensure that medical staff wears protective clothing strictly according to standard processes, a specific wearing flow chart and wearing instructions are posted in a dressing room, the medical staff is prompted to wear protective clothing strictly according to standard processes, and meanwhile, supervision staff performs manual supervision. However, since the transmission process of the protective clothing is complicated and the attention items in the wearing process are more, it is difficult to fully supervise the action standard and the attention details in the wearing process of the protective clothing by means of human supervision.
Disclosure of Invention
The technical problems to be solved by the invention are as follows: it is difficult to fully process the action criteria and note details during the wear of protective clothing by means of human supervision. Aim at provides a protective clothing wears standardized flow monitoring system, relies on image acquisition, image processing and degree of depth study, realizes whether meeting the standardized flow of wearing and carrying out real-time supervision and suggestion to medical personnel in protective clothing wearing process.
The invention is realized by the following technical scheme:
the standardized process monitoring system for wearing the protective clothing comprises a video acquisition module, a video processing module and a video processing module, wherein the video acquisition module is used for acquiring videos of wearing the protective clothing; the boundary dividing module is used for dividing the video into a plurality of video clips, and each video clip comprises a staged action in the protective clothing wearing standardized process; the video splitting module is used for splitting the video clip into a plurality of Shan Zhen images, and each Shan Zhen image contains corresponding time information; the skeleton detection module is used for detecting skeleton key points of the single-frame image and generating a plurality of skeleton structure diagrams, wherein the skeleton structure diagrams contain the space information of each skeleton key point; the skeleton association module is used for connecting a plurality of skeleton key points with action association in the skeleton structure diagram according to the characteristics of the staged actions to generate a skeleton structure association diagram; the image splicing module is used for splicing the plurality of skeleton architecture association diagrams into video clips to be detected according to the corresponding time information sequence; the action discriminating module is used for discriminating each stepwise action in the video fragment to be tested according to the spatial information of the skeleton architecture association diagram to obtain a discrimination result of each stepwise action, and matching the discrimination result with the corresponding stepwise action in the protective clothing wearing standardized flow according to the time information of the skeleton architecture association diagram; and the error prompt module is used for outputting an error prompt message when the matching is unsuccessful.
Furthermore, the system also comprises a time sequence control module which is used for controlling the operation of the skeleton detection module, the skeleton association module, the image stitching module, the action judging module and the error prompting module according to the time information of the skeleton structure diagram.
Further, the boundary dividing module comprises a feature extraction unit, which is used for sending the video into a CNN network, and carrying out time sequence convolution on the video through the CNN network to generate a plurality of time sequence fragments containing feature information; the random sampling unit is used for randomly extracting a plurality of samples to be detected from a plurality of time sequence fragments; the similarity evaluation unit is used for sending the plurality of samples to be detected into an action classifier trained in advance, and performing action similarity evaluation on the characteristic information of each sample to be detected and the characteristic information of the standard action by using the action classifier to generate a time sequence-based action similarity waveform corresponding to each sample to be detected; a reference sample generation unit for inputting a plurality of time sequence-based action similarity waveforms into a time sequence-based classification network, and generating a plurality of action classification reference samples according to a preset time sequence-based classification network threshold by using the time sequence-based classification network; and the boundary dividing unit is used for inputting the plurality of action classification reference samples and the plurality of time sequence fragments into the action classifier, dividing the boundaries of the plurality of time sequence fragments according to the plurality of action classification reference samples by using the action classifier, and outputting the plurality of video fragments.
Further, the skeleton detection module comprises a feature map generation unit, which is used for inputting the single-frame image into a VGG network constructed in advance, and extracting the feature map of the single-frame image through the VGG network; the key point extraction unit is used for inputting the feature map into a double-branch convolutional neural network constructed in advance, and performing multistage regression on the feature map by utilizing the double-branch convolutional neural network to obtain a plurality of skeleton key points in the feature map; the skeleton diagram generating unit is used for generating a skeleton structure diagram by connecting a plurality of skeleton key points through a greedy algorithm according to the distribution characteristics of human skeleton; the spatial information assignment unit is used for mapping the skeleton architecture diagram into a spatial coordinate system constructed in advance and assigning corresponding three-dimensional coordinates to each skeleton key point in the skeleton architecture diagram.
Further, the bone detection module further comprises a data processing unit, which is used for carrying out normalization processing on the spatial information of the plurality of bone key points.
Further, the VGG network constructed in advance includes 10 3×3 convolutional layers and 3 2×2 pooling layers; 10 3 x 3 convolutional layers and 3 2 x 2 pooling layers are stacked in the following order: 2 convolutional layers, 1 pooled layer, 4 convolutional layers, 1 pooled layer, 2 convolutional layers.
Further, the double-branch convolutional neural network constructed in advance comprises a first branch convolutional neural network, a second branch convolutional neural network and an accumulator; the first branch convolutional neural network comprises a front-stage network, a front-stage loss calculator, a rear-stage network and a rear-stage loss calculator; the pre-network comprises 3 3×3 convolution layers and 2 1×1 convolution layers; 3X 3 convolution layers are sequentially stacked from bottom to top, 2 1X 1 convolution layers are sequentially stacked from bottom to top, the 1X 1 convolution layers are stacked above the 3X 3 convolution layers, and the output of the 1X 1 convolution layer positioned at the top layer is connected with the input of the front-stage loss calculator; the latter network comprises 5 7×7 convolutional layers and 2 1×1 convolutional layers; 5 7×7 convolution layers are sequentially stacked from bottom to top, 2 1×1 convolution layers are sequentially stacked from bottom to top, the 1×1 convolution layers are stacked above the 7×7 convolution layers, and the output of the 1×1 convolution layer positioned at the top layer is connected with the input of the subsequent-stage loss calculator; the structure of the first branch convolutional neural network is the same as that of the second branch convolutional neural network; the feature map is respectively input into a bottom layer of the first branch convolutional neural network, a bottom layer of the second branch neural network and an accumulator; the output of the front-stage loss calculator of the first branch convolutional neural network and the output of the front-stage loss calculator of the second branch convolutional neural network are connected into an accumulator; the output of the accumulator is respectively connected with the back-stage network of the first branch convolutional neural network and the back-stage network of the second branch convolutional neural network.
Further, the action discriminating module comprises a data input unit for inputting the space information of each skeleton key point; the motion recognition unit is used for outputting a recognition result of each stepwise motion according to the spatial information of each skeleton key point, wherein the recognition result is a stepwise motion image; the action matching unit is used for generating the identification result of the staged action image to classify, and matching the classification result with the staged action corresponding to the standardized process of wearing the protective clothing.
Further, the action recognition unit comprises a 4-layer overlapped full-connection layer and a Relu activation function; the action recognition unit comprises a softmax classifier.
Further, the motion matching unit comprises a matrix generation subunit, a motion matching unit and a motion matching unit, wherein the matrix generation subunit is used for generating an adjacency matrix of the staged motion image and an adjacency matrix of the corresponding standard motion image; and the similarity calculation subunit is used for calculating Euclidean distance between the adjacent matrix of the obtained staged action image and the adjacent matrix of the corresponding standard action image, judging that the staged action image is successfully matched with the corresponding standard action image when the Euclidean distance is smaller than a threshold value, and judging that the matching is failed otherwise.
Compared with the prior art, the invention has the following advantages and beneficial effects: the method integrates image acquisition, image processing and deep learning, and realizes real-time supervision of action standards and attention details of medical staff in the whole wearing process of the protective clothing. Specifically, on one hand, video images of wearing protective clothing by medical staff are collected, the video images are divided into a plurality of segments according to action characteristics in a protective clothing wearing standard flow, and human skeleton key points are detected for each video segment, so that the problems of low action recognition accuracy and poor continuous action distinction degree caused by the influence of light and shielding easily in the human body posture and natural man-machine interaction process can be solved; on the other hand, the collected bone key points are subjected to action association, and natural connection (inherent connection relation before the bone key points) and unnatural connection (action coordination relation between the bone key points) before the bone key points are established, so that the model is more in line with the human motion characteristics, and the actions in the wearing process of the protective clothing are accurately identified; on the one hand, the action recognition is completed by combining deep learning, meanwhile, the correlation matching algorithm is utilized, the Euclidean distance of the adjacent matrix is used as a judgment standard, the monitoring result is matched with the standard action, and finally, the full-flow real-time supervision on the standard wearing of the protective clothing is realized.
Drawings
In order to more clearly illustrate the technical solutions of the exemplary embodiments of the present invention, the drawings that are needed in the examples will be briefly described below, it being understood that the following drawings only illustrate some examples of the present invention and therefore should not be considered as limiting the scope, and that other related drawings may be obtained from these drawings without inventive effort for a person skilled in the art.
FIG. 1 is a diagram of a protective clothing wearing standardization process monitoring system architecture according to an embodiment of the present invention;
fig. 2 is a schematic workflow diagram of a boundary dividing module according to an embodiment of the present invention.
Description of the embodiments
For the purpose of making apparent the objects, technical solutions and advantages of the present invention, the present invention will be further described in detail with reference to the following examples and the accompanying drawings, wherein the exemplary embodiments of the present invention and the descriptions thereof are for illustrating the present invention only and are not to be construed as limiting the present invention.
Examples: as shown in fig. 1, the present embodiment provides a standardized flow monitoring system for wear of protective clothing, which includes a video acquisition module, a boundary dividing module, a video splitting module, a bone detection module, a bone association module, an image stitching module, an action discriminating module, an error prompting module and a timing control module. The function, composition and principle of each module are described in detail below.
1. Video acquisition module
The video acquisition module is used for acquiring videos of wearing protective clothing and sending the acquired videos to the boundary dividing module for image processing.
2. Boundary dividing module
The boundary dividing module is used for dividing the video into a plurality of video clips, and each video clip comprises a staged action in the protective clothing wearing standardization process. According to WHO recommendation and chinese CDC recommendation, the protective clothing wearing standardization process includes a plurality of wearing steps and corresponding standardized wearing actions. Therefore, it is necessary to define a start flag bit and an end flag bit of each standardized wearing action, that is, an accurate dividing boundary is provided for each standardized wearing action, specifically, accurate to a frame, so as to accurately identify each wearing action of a medical staff in a wearing process.
Specifically, the boundary dividing module includes, a feature extraction unit, a random sampling unit, a similarity evaluation unit, a reference sample generation unit, and a boundary dividing unit. The feature extraction unit is used for sending the video into a CNN network, carrying out time sequence convolution on the video through the CNN network, and generating a plurality of time sequence fragments containing feature information; the random sampling unit is used for randomly extracting a plurality of samples to be detected from a plurality of time sequence fragments; the similarity evaluation unit is used for sending a plurality of samples to be detected into an action classifier trained in advance, and performing action similarity evaluation on the characteristic information of each sample to be detected and the characteristic information of the standard action by using the action classifier to generate a time sequence-based action similarity waveform corresponding to each sample to be detected; the reference sample generation unit is used for inputting a plurality of action similarity waveforms based on time sequences into the classification network based on time sequences, and generating a plurality of action classification reference samples according to a preset classification network threshold based on time sequences by using the classification network based on time sequences; the boundary dividing unit is used for inputting a plurality of action classification reference samples and a plurality of time sequence fragments into the action classifier, dividing the boundaries of the plurality of time sequence fragments according to the plurality of action classification reference samples by using the action classifier, and outputting a plurality of video fragments.
As shown in fig. 2, which is a workflow of the boundary dividing module, the video enters the CNN network, a plurality of time sequence fragments containing different feature information are generated after image features are advanced, and then each time sequence fragment is sampled. The extracted samples enter an action classifier trained by a CNN network, similarity evaluation is carried out on the extracted samples and standard actions through the action classifier, one-dimensional action similarity waveforms are generated in a scoring mode, then the action similarity waveforms are sent into a time sequence-based classification network, and different time sequence-based classification network thresholds are set to meet the requirements of different division accuracy. And generating action classification parameter samples with different precision by a time sequence-based classification network, and finally inputting all the reference samples into the trained action classifier to perform action classification and boundary division and outputting a result.
The CNN network used for feature extraction is a deep neural network model trained by feature data of each standard action in the protective clothing wearing standardized process. The random sampling unit extracts typical characteristic information from the extracted video clips to form a plurality of samples to be detected. The motion classifier used in this embodiment is a clean labels retraining motion classifier trained in advance, and of course, other motion classifiers capable of achieving the same effect can be used.
3. Video splitting module
The video splitting module is used for splitting the video clip into a plurality of Shan Zhen images, and each Shan Zhen image contains corresponding time information.
4. Bone detection module
The skeleton detection module is used for detecting skeleton key points of the single-frame image and generating a plurality of skeleton structure diagrams, wherein the skeleton structure diagrams contain the spatial information of each skeleton key point.
Specifically, the skeleton detection module comprises a feature map generation unit, a key point extraction unit, a skeleton map generation unit, a spatial information assignment unit and a data processing unit. The feature map generation unit is used for inputting the single-frame image into a VGG network constructed in advance, and extracting the feature map of the single-frame image through the VGG network; the key point extraction unit is used for inputting the feature map into a double-branch convolutional neural network constructed in advance, and performing multistage regression on the feature map by utilizing the double-branch convolutional neural network to obtain a plurality of skeleton key points in the feature map; the skeleton diagram generating unit is used for generating a skeleton structure diagram by connecting a plurality of skeleton key points through a greedy algorithm according to the distribution characteristics of human skeleton; the space information assignment unit is used for mapping the skeleton architecture diagram into a space coordinate system constructed in advance and assigning corresponding three-dimensional coordinates to each skeleton key point in the skeleton architecture diagram; the data processing unit is used for carrying out normalization processing on the spatial information of the plurality of bone key points.
Workflow of bone detection module:
firstly, inputting a single frame image into a VGG network to extract a feature map of a picture. The VGG network used in this embodiment is the first 10 layers of VGG19, comprising 10 3×3 convolutional layers and 3 2×2 pooling layers; 10 3 x 3 convolutional layers and 3 2 x 2 pooling layers are stacked in the following order: 2 convolutional layers, 1 pooled layer, 4 convolutional layers, 1 pooled layer, 2 convolutional layers.
And then, detecting each bone key point in the human body by utilizing the double convolutional neural network according to the characteristic information to obtain three-dimensional coordinate information of each bone key point. In this embodiment, the dual branch convolutional neural network includes a first branch convolutional neural network, a second branch convolutional neural network, and an accumulator. The first branch convolutional neural network comprises a front-stage network, a front-stage loss calculator, a rear-stage network and a rear-stage loss calculator; the pre-network comprises 3 3×3 convolution layers and 2 1×1 convolution layers; 3X 3 convolution layers are sequentially stacked from bottom to top, 2 1X 1 convolution layers are sequentially stacked from bottom to top, the 1X 1 convolution layers are stacked above the 3X 3 convolution layers, and the output of the 1X 1 convolution layer positioned at the top layer is connected with the input of the front-stage loss calculator; the latter network comprises 5 7×7 convolutional layers and 2 1×1 convolutional layers; 5 7×7 convolution layers are sequentially stacked from bottom to top, 2 1×1 convolution layers are sequentially stacked from bottom to top, the 1×1 convolution layers are stacked above the 7×7 convolution layers, and the output of the 1×1 convolution layer positioned at the top layer is connected with the input of the subsequent-stage loss calculator; the structure of the first branch convolutional neural network is the same as that of the second branch convolutional neural network; the feature map is respectively input into a bottom layer of the first branch convolutional neural network, a bottom layer of the second branch neural network and an accumulator; the output of the front-stage loss calculator of the first branch convolutional neural network and the output of the front-stage loss calculator of the second branch convolutional neural network are connected into an accumulator; the output of the accumulator is respectively connected with the back-stage network of the first branch convolutional neural network and the back-stage network of the second branch convolutional neural network.
The feature map extracted through the VGG network is input into the front-stage networks of the first branch convolutional neural network and the second branch convolutional neural network, and three-dimensional coordinates of all key points in the picture and each key point are obtained through multi-order regression of the double branch convolutional neural network.
The key points of the human skeleton collected in this embodiment include: left and right eyebrows, left and right cheekbones, left and right shoulders, left and right elbow joints, left and right wrist joints, left and right hip joints, left and right knee joints, and left and right ankle joints.
Finally, all key points are connected through a greedy algorithm to form the whole skeleton diagram of the human body. In addition, the first branch convolution neural network outputs a 2D confidence map of key points of human bones, and the second branch convolution neural network outputs the trend of pixel points in the bones. In addition, in each branch convolutional neural network, after the multi-level regression of the front-level network and the multi-level regression of the rear-level network, the L2 paradigm is adopted to perform loss calculation.
When the VGG network is constructed in advance, in order to diversify data, people with different body types are selected as acquisition objects when the data set is acquired, and a large number of two-dimensional coordinates are complicated and time-consuming to train, so that the convergence of the network is accelerated during training, and normalization processing is required to be performed on the data.
5. Bone association module
The skeleton association module is used for connecting a plurality of skeleton key points with action association in the skeleton structure diagram according to the characteristics of the staged actions, and generating the skeleton structure association diagram.
In the human skeleton diagram, each node represents a key point in the human skeleton, and an edge in the diagram is the expression of connectivity between joints. Edges of the graph are divided into two classes: one type is a natural physical connection relationship existing between joints, namely, the inherent dependence among bones; the other is a connection relationship between nodes set manually, that is, in this embodiment, the extrinsic dependencies between the skeletal key points established by the advertising skeletal association model.
In addition to considering local natural connection of human joints, the embodiment focuses attention on interaction among all skeleton key points in the wearing action change process of the protective clothing, so that the expanded structure reflects linkage of all skeleton key points and influence on action type identification in human body activities, and accords with activity rules corresponding to human body action behaviours and daily experience. And the characteristics of the bone key points are expressed by three-dimensional coordinates, so that the linkage rule among the bone key points is reflected on the change of the relative positions of the bone key points. Therefore, the natural defects of the skeleton diagram can be effectively compensated by the action association, and the action logic of the skeleton action recognition is also met.
6. Image stitching module
The image splicing module is used for splicing the plurality of skeleton architecture association diagrams into video clips to be detected according to the corresponding time information sequence.
7. Action discriminating module
The action discriminating module is used for discriminating each stepwise action in the video fragment to be tested according to the spatial information of the skeleton architecture association diagram, obtaining the discrimination result of each stepwise action, and matching the discrimination result with the corresponding stepwise action in the protective clothing wearing standardized flow according to the time information of the skeleton architecture association diagram.
Specifically, the action discriminating module includes a data input unit, an action identifying unit and an action matching unit. The data input unit is used for inputting the space information of each skeleton key point; the action recognition unit comprises 4 layers of superimposed full-connection layers and a Relu activation function and is used for outputting a recognition result of each stepwise action according to the spatial information of each skeleton key point, wherein the recognition result is a stepwise action image; the action matching unit comprises a softmax classifier which is used for generating the identification result of the staged action image to classify, and matching the classification result with the staged action corresponding to the standardized process of wearing the protective clothing.
Further, the action matching unit comprises a matrix generation subunit and a similarity calculation subunit. The matrix generation subunit is used for generating an adjacent matrix of the staged action image and an adjacent matrix of the corresponding standard action image; the similarity calculation subunit is used for calculating Euclidean distance between the adjacent matrix of the stepped action image and the adjacent matrix of the corresponding standard action image, and judging that the stepped action image is successfully matched with the corresponding standard action image when the Euclidean distance is smaller than a threshold value, otherwise, the matching is failed.
8. Error prompting module
The error prompt module is used for outputting an error prompt message when the matching is unsuccessful.
9. Time sequence control module
The time sequence control module is used for controlling the operation of the skeleton detection module, the skeleton association module, the image splicing module, the action judging module and the error prompting module according to the time information of the skeleton structure diagram.
The foregoing description of the embodiments has been provided for the purpose of illustrating the general principles of the invention, and is not meant to limit the scope of the invention, but to limit the invention to the particular embodiments, and any modifications, equivalents, improvements, etc. that fall within the spirit and principles of the invention are intended to be included within the scope of the invention.
Claims (10)
1. A protective clothing dresses standardized flow monitoring system, characterized by including
The video acquisition module is used for acquiring videos of wearing protective clothing;
the boundary dividing module is used for dividing the video into a plurality of video clips, and each video clip comprises a staged action in the protective clothing wearing standardized process;
the video splitting module is used for splitting the video clip into a plurality of Shan Zhen images, and each Shan Zhen image contains corresponding time information;
the skeleton detection module is used for detecting skeleton key points of the single-frame image and generating a plurality of skeleton structure diagrams, wherein the skeleton structure diagrams contain the space information of each skeleton key point;
the skeleton association module is used for connecting a plurality of skeleton key points with action association in the skeleton structure diagram according to the characteristics of the staged actions to generate a skeleton structure association diagram;
the image splicing module is used for splicing the plurality of skeleton architecture association diagrams into video clips to be detected according to the corresponding time information sequence;
the action discriminating module is used for discriminating each stepwise action in the video fragment to be tested according to the spatial information of the skeleton architecture association diagram to obtain a discrimination result of each stepwise action, and matching the discrimination result with the corresponding stepwise action in the protective clothing wearing standardized flow according to the time information of the skeleton architecture association diagram;
and the error prompt module is used for outputting an error prompt message when the matching is unsuccessful.
2. The standardized flow monitoring system for wear of protective clothing of claim 1 further comprising a timing control module for controlling operation of the bone detection module, the bone correlation module, the image stitching module, the motion discrimination module and the error indication module according to time information of the bone architecture diagram.
3. The protective clothing wearing standardization process monitoring system of claim 1 or 2, wherein the boundary dividing module comprises
The feature extraction unit is used for sending the video into a CNN network, carrying out time sequence convolution on the video through the CNN network, and generating a plurality of time sequence fragments containing feature information;
the random sampling unit is used for randomly extracting a plurality of samples to be detected from a plurality of time sequence fragments;
the similarity evaluation unit is used for sending the plurality of samples to be detected into an action classifier trained in advance, and performing action similarity evaluation on the characteristic information of each sample to be detected and the characteristic information of the standard action by using the action classifier to generate a time sequence-based action similarity waveform corresponding to each sample to be detected;
a reference sample generation unit for inputting a plurality of time sequence-based action similarity waveforms into a time sequence-based classification network, and generating a plurality of action classification reference samples according to a preset time sequence-based classification network threshold by using the time sequence-based classification network;
and the boundary dividing unit is used for inputting the plurality of action classification reference samples and the plurality of time sequence fragments into the action classifier, dividing the boundaries of the plurality of time sequence fragments according to the plurality of action classification reference samples by using the action classifier, and outputting the plurality of video fragments.
4. The standardized procedure monitoring system for wear of protective clothing of claim 1 or 2 wherein the bone detection module comprises
The feature map generating unit is used for inputting the single-frame image into a VGG network constructed in advance, and extracting the feature map of the single-frame image through the VGG network;
the key point extraction unit is used for inputting the feature map into a double-branch convolutional neural network constructed in advance, and performing multistage regression on the feature map by utilizing the double-branch convolutional neural network to obtain a plurality of skeleton key points in the feature map;
the skeleton diagram generating unit is used for generating a skeleton structure diagram by connecting a plurality of skeleton key points through a greedy algorithm according to the distribution characteristics of human skeleton;
the spatial information assignment unit is used for mapping the skeleton architecture diagram into a spatial coordinate system constructed in advance and assigning corresponding three-dimensional coordinates to each skeleton key point in the skeleton architecture diagram.
5. The standardized procedure monitoring system for wear of protective clothing of claim 4 wherein the bone detection module further comprises a data processing unit for normalizing spatial information of a plurality of bone keypoints.
6. The protective clothing wearing standardized procedure monitoring system of claim 5 wherein the VGG network constructed in advance comprises 10 3 x 3 convolutional layers and 3 2 x 2 pooling layers; 10 3 x 3 convolutional layers and 3 2 x 2 pooling layers are stacked in the following order: 2 convolutional layers, 1 pooled layer, 4 convolutional layers, 1 pooled layer, 2 convolutional layers.
7. The standardized procedure monitoring system for protective clothing wear of claim 5 wherein,
the double-branch convolutional neural network constructed in advance comprises a first branch convolutional neural network, a second branch convolutional neural network and an accumulator;
the first branch convolutional neural network comprises a front-stage network, a front-stage loss calculator, a rear-stage network and a rear-stage loss calculator; the pre-network comprises 3 3×3 convolution layers and 2 1×1 convolution layers; 3X 3 convolution layers are sequentially stacked from bottom to top, 2 1X 1 convolution layers are sequentially stacked from bottom to top, the 1X 1 convolution layers are stacked above the 3X 3 convolution layers, and the output of the 1X 1 convolution layer positioned at the top layer is connected with the input of the front-stage loss calculator; the latter network comprises 5 7×7 convolutional layers and 2 1×1 convolutional layers; 5 7×7 convolution layers are sequentially stacked from bottom to top, 2 1×1 convolution layers are sequentially stacked from bottom to top, the 1×1 convolution layers are stacked above the 7×7 convolution layers, and the output of the 1×1 convolution layer positioned at the top layer is connected with the input of the subsequent-stage loss calculator;
the structure of the first branch convolutional neural network is the same as that of the second branch convolutional neural network;
the feature map is respectively input into a bottom layer of the first branch convolutional neural network, a bottom layer of the second branch neural network and an accumulator; the output of the front-stage loss calculator of the first branch convolutional neural network and the output of the front-stage loss calculator of the second branch convolutional neural network are connected into an accumulator; the output of the accumulator is respectively connected with the back-stage network of the first branch convolutional neural network and the back-stage network of the second branch convolutional neural network.
8. The standardized procedure monitoring system for wear of protective clothing according to claim 1 or 2, wherein the action discriminating module comprises
The data input unit is used for inputting the space information of each skeleton key point;
the motion recognition unit is used for outputting a recognition result of each stepwise motion according to the spatial information of each skeleton key point, wherein the recognition result is a stepwise motion image;
the action matching unit is used for generating the identification result of the staged action image to classify, and matching the classification result with the staged action corresponding to the standardized process of wearing the protective clothing.
9. The standardized procedure monitoring system for wear of protective clothing of claim 8 wherein the action recognition unit comprises 4 superimposed full connection layers and a Relu activation function;
the action recognition unit comprises a softmax classifier.
10. The protective clothing wearing standardization process monitoring system of claim 9, wherein the action matching unit comprises
A matrix generation subunit, configured to generate an adjacency matrix of the staged action image and an adjacency matrix of the corresponding standard action image;
and the similarity calculation subunit is used for calculating Euclidean distance between the adjacent matrix of the obtained staged action image and the adjacent matrix of the corresponding standard action image, judging that the staged action image is successfully matched with the corresponding standard action image when the Euclidean distance is smaller than a threshold value, and judging that the matching is failed otherwise.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310464437.XA CN116189311B (en) | 2023-04-27 | 2023-04-27 | Protective clothing wears standardized flow monitoring system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310464437.XA CN116189311B (en) | 2023-04-27 | 2023-04-27 | Protective clothing wears standardized flow monitoring system |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN116189311A true CN116189311A (en) | 2023-05-30 |
| CN116189311B CN116189311B (en) | 2023-07-25 |
Family
ID=86434839
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202310464437.XA Active CN116189311B (en) | 2023-04-27 | 2023-04-27 | Protective clothing wears standardized flow monitoring system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN116189311B (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN117115861A (en) * | 2023-10-19 | 2023-11-24 | 四川弘和数智集团有限公司 | Glove detection method and device, electronic equipment and storage medium |
Citations (13)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109034124A (en) * | 2018-08-30 | 2018-12-18 | 成都考拉悠然科技有限公司 | A kind of intelligent control method and system |
| CN110135319A (en) * | 2019-05-09 | 2019-08-16 | 广州大学 | A kind of anomaly detection method and its system |
| CN111444819A (en) * | 2020-03-24 | 2020-07-24 | 北京百度网讯科技有限公司 | Cutting frame determining method, network training method, device, equipment and storage medium |
| CN112466445A (en) * | 2020-11-11 | 2021-03-09 | 湖南德雅曼达科技有限公司 | Protective articles wears to take off monitoring facilities and keeps apart ward |
| US20210124914A1 (en) * | 2019-10-28 | 2021-04-29 | Jomoo Kitchen & Bath Co., Ltd | Training method of network, monitoring method, system, storage medium and computer device |
| WO2021082749A1 (en) * | 2019-10-30 | 2021-05-06 | 腾讯科技（深圳）有限公司 | Action identification method based on artificial intelligence and related apparatus |
| CN114140868A (en) * | 2021-06-04 | 2022-03-04 | 中山大学附属第三医院（中山大学肝脏病医院） | Method and device for identifying standard actions of putting-on and taking-off of protective appliance |
| CN114613013A (en) * | 2022-03-18 | 2022-06-10 | 长沙理工大学 | End-to-end human behavior recognition method and model based on skeleton nodes |
| CN114782874A (en) * | 2022-05-16 | 2022-07-22 | 中国计量大学 | Anti-epidemic protection article wearing behavior standard detection method based on human body posture |
| CN114937232A (en) * | 2022-07-25 | 2022-08-23 | 浙江大学 | Wearing detection method, system and equipment for medical waste treatment personnel protective appliance |
| CN115393944A (en) * | 2022-09-15 | 2022-11-25 | 重庆邮电大学 | Micro-expression identification method based on multi-dimensional feature fusion |
| US11538462B1 (en) * | 2022-03-15 | 2022-12-27 | My Job Matcher, Inc. | Apparatuses and methods for querying and transcribing video resumes |
| EP4125057A1 (en) * | 2021-07-28 | 2023-02-01 | SITA Information Networking Computing Canada Inc | A system for automated monitoring and actioning of health risk mitigation |
-
2023
- 2023-04-27 CN CN202310464437.XA patent/CN116189311B/en active Active
Patent Citations (13)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109034124A (en) * | 2018-08-30 | 2018-12-18 | 成都考拉悠然科技有限公司 | A kind of intelligent control method and system |
| CN110135319A (en) * | 2019-05-09 | 2019-08-16 | 广州大学 | A kind of anomaly detection method and its system |
| US20210124914A1 (en) * | 2019-10-28 | 2021-04-29 | Jomoo Kitchen & Bath Co., Ltd | Training method of network, monitoring method, system, storage medium and computer device |
| WO2021082749A1 (en) * | 2019-10-30 | 2021-05-06 | 腾讯科技（深圳）有限公司 | Action identification method based on artificial intelligence and related apparatus |
| CN111444819A (en) * | 2020-03-24 | 2020-07-24 | 北京百度网讯科技有限公司 | Cutting frame determining method, network training method, device, equipment and storage medium |
| CN112466445A (en) * | 2020-11-11 | 2021-03-09 | 湖南德雅曼达科技有限公司 | Protective articles wears to take off monitoring facilities and keeps apart ward |
| CN114140868A (en) * | 2021-06-04 | 2022-03-04 | 中山大学附属第三医院（中山大学肝脏病医院） | Method and device for identifying standard actions of putting-on and taking-off of protective appliance |
| EP4125057A1 (en) * | 2021-07-28 | 2023-02-01 | SITA Information Networking Computing Canada Inc | A system for automated monitoring and actioning of health risk mitigation |
| US11538462B1 (en) * | 2022-03-15 | 2022-12-27 | My Job Matcher, Inc. | Apparatuses and methods for querying and transcribing video resumes |
| CN114613013A (en) * | 2022-03-18 | 2022-06-10 | 长沙理工大学 | End-to-end human behavior recognition method and model based on skeleton nodes |
| CN114782874A (en) * | 2022-05-16 | 2022-07-22 | 中国计量大学 | Anti-epidemic protection article wearing behavior standard detection method based on human body posture |
| CN114937232A (en) * | 2022-07-25 | 2022-08-23 | 浙江大学 | Wearing detection method, system and equipment for medical waste treatment personnel protective appliance |
| CN115393944A (en) * | 2022-09-15 | 2022-11-25 | 重庆邮电大学 | Micro-expression identification method based on multi-dimensional feature fusion |
Non-Patent Citations (4)
| Title |
|---|
| ANIKET VERMA等: "Human Activity Classification Using Deep Convolutional Neural Network", 《RECENT ADVANCES IN ARTIFICIAL INTELLIGENCE AND DATA ENGINEERING 》, pages 41 - 50 * |
| 丁洪金等: "基于时序分析的人体活动状态识别与定位", 《计算机技术与发展》, vol. 29, no. 4, pages 82 - 86 * |
| 伏娜娜等: "面向嵌入式系统的人体行为识别", 《激光与光电子学进展》, vol. 59, no. 22, pages 212 - 219 * |
| 朱天逸: "基于深度学习的水务厂人员安全检测研究", 《中国优秀硕士学位论文全文数据库 (工程科技Ⅱ辑)》, no. 2, pages 038 - 3182 * |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN117115861A (en) * | 2023-10-19 | 2023-11-24 | 四川弘和数智集团有限公司 | Glove detection method and device, electronic equipment and storage medium |
| CN117115861B (en) * | 2023-10-19 | 2024-01-26 | 四川弘和数智集团有限公司 | Glove detection method and device, electronic equipment and storage medium |
Also Published As
| Publication number | Publication date |
|---|---|
| CN116189311B (en) | 2023-07-25 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN108052896B (en) | Human body behavior identification method based on convolutional neural network and support vector machine | |
| Gomez-Donoso et al. | Large-scale multiview 3d hand pose dataset | |
| CN103941866B (en) | Three-dimensional gesture recognizing method based on Kinect depth image | |
| CN103970271B (en) | The daily routines recognition methods of fusional movement and physiology sensing data | |
| CN112184705B (en) | Human body acupuncture point identification, positioning and application system based on computer vision technology | |
| CN102945366B (en) | A kind of method and device of recognition of face | |
| CN104463250B (en) | A kind of Sign Language Recognition interpretation method based on Davinci technology | |
| US9020250B2 (en) | Methods and systems for building a universal dress style learner | |
| Chaudhari et al. | Yog-guru: Real-time yoga pose correction system using deep learning methods | |
| US9183431B2 (en) | Apparatus and method for providing activity recognition based application service | |
| JP2001167110A (en) | Picture retrieving method and its device | |
| CN101493887B (en) | Eyebrow image segmentation method based on semi-supervision learning and Hash index | |
| CN114937232B (en) | Wearing detection method, system and equipment for medical waste treatment personnel protective appliance | |
| CN106845513B (en) | Manpower detector and method based on condition random forest | |
| CN110458059A (en) | A kind of gesture identification method based on computer vision and identification device | |
| CN116189311B (en) | Protective clothing wears standardized flow monitoring system | |
| CN104268514A (en) | Gesture detection method based on multi-feature fusion | |
| CN111950321B (en) | Gait recognition method, device, computer equipment and storage medium | |
| Rahmani et al. | Action classification with locality-constrained linear coding | |
| CN114758362A (en) | Clothing changing pedestrian re-identification method based on semantic perception attention and visual masking | |
| CN114782874A (en) | Anti-epidemic protection article wearing behavior standard detection method based on human body posture | |
| CN105373810A (en) | Method and system for building action recognition model | |
| Bhargavas et al. | Human identification using gait recognition | |
| Gawali et al. | 3d face recognition using geodesic facial curves to handle expression, occlusion and pose variations | |
| Ziaeefard et al. | Hierarchical human action recognition by normalized-polar histogram |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |