CN114037943A - Method and device for detecting falling-off prevention sleeping sentry - Google Patents
Method and device for detecting falling-off prevention sleeping sentry Download PDFInfo
- Publication number
- CN114037943A CN114037943A CN202111412259.3A CN202111412259A CN114037943A CN 114037943 A CN114037943 A CN 114037943A CN 202111412259 A CN202111412259 A CN 202111412259A CN 114037943 A CN114037943 A CN 114037943A
- Authority
- CN
- China
- Prior art keywords
- human body
- behavior
- sleep
- layer
- prediction model
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 30
- 230000002265 prevention Effects 0.000 title description 7
- 230000006399 behavior Effects 0.000 claims abstract description 155
- 230000007958 sleep Effects 0.000 claims abstract description 103
- 238000001514 detection method Methods 0.000 claims abstract description 95
- 210000000746 body region Anatomy 0.000 claims abstract description 74
- 238000010586 diagram Methods 0.000 claims abstract description 63
- 238000012549 training Methods 0.000 claims abstract description 29
- 238000011176 pooling Methods 0.000 claims description 21
- 238000002372 labelling Methods 0.000 claims description 15
- 230000008859 change Effects 0.000 claims description 14
- 238000004590 computer program Methods 0.000 claims description 13
- 238000000605 extraction Methods 0.000 claims description 9
- 230000007246 mechanism Effects 0.000 claims description 9
- 238000000354 decomposition reaction Methods 0.000 claims description 3
- 230000003542 behavioural effect Effects 0.000 claims 1
- 230000008569 process Effects 0.000 description 3
- 230000001360 synchronised effect Effects 0.000 description 2
- 238000004458 analytical method Methods 0.000 description 1
- 238000013473 artificial intelligence Methods 0.000 description 1
- 210000000988 bone and bone Anatomy 0.000 description 1
- 239000000284 extract Substances 0.000 description 1
- 230000010354 integration Effects 0.000 description 1
- 239000004973 liquid crystal related substance Substances 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000012544 monitoring process Methods 0.000 description 1
- 238000011160 research Methods 0.000 description 1
- 230000003068 static effect Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/045—Combinations of networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/049—Temporal neural networks, e.g. delay elements, oscillating neurons or pulsed inputs
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
Abstract
The application relates to a method and a device for detecting falling off of a sleeping station, computer equipment and a storage medium. The method comprises the following steps: training a YOLOv5x model by using a human body region detection sample, and outputting a human body region frame; extracting the coordinate positions of the human body key points from the human body region frame, and obtaining a sleep post skeleton space-time diagram according to the coordinate positions of the human body key points of each image frame in the image frame sequence; training a behavior prediction model through a sleep post framework space-time diagram to obtain a trained behavior prediction model; and (3) carrying out the off-duty detection of the human body by utilizing the trained YOLOv5x model, and if the YOLOv5x model outputs a human body region frame, carrying out the sleep-duty behavior detection on the human body according to the trained behavior prediction model. The method can be used for detecting the off-Shift sleeping behavior.
Description
Technical Field
The application relates to the technical field of intelligent security, in particular to a method and a device for detecting falling off of a sleeping sentry, computer equipment and a storage medium.
Background
Along with the deep application of internet and artificial intelligence, the work, study and life of camp gradually move towards intelligent integration, and the wisdom camp uses various application service system as the carrier, fully fuses teaching, scientific research, management and leisure life. The problem of how to detect and prohibit the staff from sleeping is always a problem for thinking of many managers, because the duty room often needs the staff to pay attention to and keep alert all the time, and once the staff is neglected due to sleeping, the problem is very likely to cause huge potential safety hazards.
However, the existing methods for performing sleep post warning by using video image data are few, generally, offline data are acquired by a sensor to perform sleep post classification and identification, and the methods for performing sleep post warning are few and have narrow range of alternative methods.
Disclosure of Invention
In view of the above, it is desirable to provide a method, an apparatus, a computer device, and a storage medium for off-duty sleep detection that can perform off-duty sleep detection.
A falling-off prevention sleeping post detection method comprises the following steps:
acquiring an off-Shift sleeping-on-Shift video shot by a camera;
decomposing the off-Shift sleeping Shift video to obtain an image frame sequence;
marking a human body region of an image frame in the image frame sequence to obtain a human body region detection sample, and marking a sleep post behavior in the image frame sequence to obtain a sleep post behavior sequence image frame sequence;
training a YOLOv5x model by using a human body region detection sample, and outputting a human body region frame;
extracting the coordinate positions of the key points of the human body from the human body region frame, and obtaining a sleep post skeleton space-time diagram according to the coordinate positions of the key points of the human body of each image frame in the sleep post behavior image frame sequence;
training a behavior prediction model through a sleep post framework space-time diagram to obtain a trained behavior prediction model; the trained behavior prediction model comprises the following steps: a graph convolution layer, a time convolution layer, a pooling layer, a full-link layer, and an output layer;
and (3) carrying out the human body off duty detection according to the trained YOLOv5x model, and if the YOLOv5x model outputs a human body region frame, carrying out the sleep duty behavior detection on the human body according to the trained behavior prediction model.
In one embodiment, the trained YOLOv5x model is used for human off-Shift detection, and if the YOLOv5x model does not output a human region box, the human is judged to be off-Shift.
In one embodiment, extracting the coordinate positions of the key points of the human body from the human body region frame comprises: inputting the human body region frame into a HigherHRNet network to obtain a thermodynamic diagram; the local maximum of the thermodynamic diagram is the coordinate position of the key point of the human body.
In one embodiment, the trained behavior prediction model further comprises an attention mechanism layer.
In one embodiment, the graph convolutional layer, the time convolutional layer and the attention mechanism layer constitute a sleep behavior detection network.
In one embodiment, the detection of the sleep-post behavior of the human body according to the trained behavior prediction model comprises: and performing space-time skeleton feature extraction on the sleep post skeleton space-time diagram through a diagram convolution layer in the trained behavior prediction model, and outputting a detection result of sleep post behavior detection after passing through a time convolution layer, a pooling layer, a full-connection layer and an output layer in sequence.
In one embodiment, the time-space skeleton feature extraction is performed on the sleep post skeleton space-time diagram through a graph convolution layer in a trained behavior prediction model, and a detection result of sleep post behavior detection is output after the time convolution layer, the pooling layer, the full-link layer and the output layer pass through in sequence, and the method comprises the following steps:
carrying out graph convolution on the sleep post skeleton space-time diagram through a graph convolution layer in a trained behavior prediction model to obtain local characteristics of adjacent key points of a human body in a space;
convolving the local features of the adjacent key points of the human body in the space through a time convolution layer to obtain the local features of the change of the key points of the human body in the time;
and (3) sequentially passing the local characteristics of the change of the key points of the human body in time through the pooling layer, the full-connection layer and the output layer, and outputting a detection result of the sleep post behavior detection.
An anti-falling off post sleeping detection device, the device comprising:
the video decomposition module is used for acquiring the off-Shift sleeping on-Shift video shot by the camera and decomposing the off-Shift sleeping on-Shift video to obtain an image frame sequence;
the labeling module is used for labeling the human body region of the image frame in the image frame sequence to obtain a human body region detection sample, and labeling the sleep post behavior in the image frame sequence to obtain a sleep post behavior sequence image frame sequence;
the skeleton space-time diagram obtaining module is used for training a YOLOv5x model by using a human body region detection sample and outputting a human body region frame; extracting the coordinate positions of the key points of the human body from the human body region frame, and obtaining a sleep post skeleton space-time diagram according to the coordinate positions of the key points of the human body of each image frame in the sleep post behavior image frame sequence;
the behavior prediction model training module is used for training a behavior prediction model through a sleep post framework space-time diagram to obtain a trained behavior prediction model; the trained behavior prediction model comprises the following steps: a graph convolution layer, a time convolution layer, a pooling layer, a full-link layer, and an output layer;
and the behavior detection module is used for detecting the off-duty of the human body according to the trained YOLOv5x model, and if the YOLOv5x model outputs a human body region frame, detecting the sleep-duty behavior of the human body according to the trained behavior prediction model.
A computer device comprising a memory and a processor, the memory storing a computer program, the processor implementing the following steps when executing the computer program:
acquiring an off-Shift sleeping-on-Shift video shot by a camera;
decomposing the off-Shift sleeping Shift video to obtain an image frame sequence;
marking a human body region of an image frame in the image frame sequence to obtain a human body region detection sample, and marking a sleep post behavior in the image frame sequence to obtain a sleep post behavior sequence image frame sequence;
training a YOLOv5x model by using a human body region detection sample, and outputting a human body region frame;
extracting the coordinate positions of the key points of the human body from the human body region frame, and obtaining a sleep post skeleton space-time diagram according to the coordinate positions of the key points of the human body of each image frame in the sleep post behavior image frame sequence;
training a behavior prediction model through a sleep post framework space-time diagram to obtain a trained behavior prediction model; the trained behavior prediction model comprises the following steps: a graph convolution layer, a time convolution layer, a pooling layer, a full-link layer, and an output layer;
and (3) carrying out the human body off duty detection according to the trained YOLOv5x model, and if the YOLOv5x model outputs a human body region frame, carrying out the sleep duty behavior detection on the human body according to the trained behavior prediction model.
A computer-readable storage medium, on which a computer program is stored which, when executed by a processor, carries out the steps of:
acquiring an off-Shift sleeping-on-Shift video shot by a camera;
decomposing the off-Shift sleeping Shift video to obtain an image frame sequence;
marking a human body region of an image frame in the image frame sequence to obtain a human body region detection sample, and marking a sleep post behavior in the image frame sequence to obtain a sleep post behavior sequence image frame sequence;
training a YOLOv5x model by using a human body region detection sample, and outputting a human body region frame;
extracting the coordinate positions of the key points of the human body from the human body region frame, and obtaining a sleep post skeleton space-time diagram according to the coordinate positions of the key points of the human body of each image frame in the sleep post behavior image frame sequence;
training a behavior prediction model through a sleep post framework space-time diagram to obtain a trained behavior prediction model; the trained behavior prediction model comprises the following steps: a graph convolution layer, a time convolution layer, a pooling layer, a full-link layer, and an output layer;
and (3) carrying out the human body off duty detection according to the trained YOLOv5x model, and if the YOLOv5x model outputs a human body region frame, carrying out the sleep duty behavior detection on the human body according to the trained behavior prediction model.
The method comprises the steps of firstly labeling a human body region of an image frame in an image frame sequence to obtain a human body region detection sample, labeling sleep post behaviors in the image frame sequence to obtain a sleep post behavior sequence image frame sequence, training a YOLOv5x model by using the human body region detection sample, training the YOLOv5x model, detecting a human body target by using the trained YOLOv5x model to judge whether the sleep post is off the guard or not, carrying out skeleton extraction on a judgment result, training a behavior prediction model by using a sleep post skeleton space-time diagram to obtain a trained behavior prediction model, and carrying out human body sleep post behavior detection according to the trained behavior prediction model.
Drawings
Fig. 1 is a schematic flow chart of a sleep-off prevention detection method in an embodiment;
FIG. 2 is a block diagram of an embodiment of an anti-falling sleep post detection apparatus;
FIG. 3 is a diagram of a network architecture for fall off sleep detection in one embodiment;
FIG. 4 is a diagram illustrating an internal structure of a computer device according to an embodiment.
Detailed Description
In order to make the objects, technical solutions and advantages of the present application more apparent, the present application is described in further detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the present application and are not intended to limit the present application.
In one embodiment, as shown in fig. 1, there is provided an off-post sleep post detection method, including the following steps:
and 102, acquiring the off Shift sleeping video shot by the camera, and decomposing the off Shift sleeping video to obtain an image frame sequence.
And decomposing the off-Shift sleeping video into an image frame sequence, so as to facilitate human body region labeling from a single frame image and construct a human body region detection sample.
And 104, marking the human body region of the image frames in the image frame sequence to obtain a human body region detection sample, and marking the sleep-off behavior in the image frame sequence to obtain a sleep-off behavior sequence image frame sequence.
And marking the human body region of the image frames in the image frame sequence to obtain a human body region detection sample, if the human body region detection sample is that the human body is detected, judging that the image frame sequence is off duty, wherein the sleep duty behavior is a continuous behavior, and the sleep duty segment in the image frame sequence needs to be marked to construct the sleep duty behavior sequence image frame sequence.
The YOLOv5x model is a target monitoring model used for finding some specific objects in image frames, the target detection can identify the types of the objects and mark the positions of the objects, after training the YOLOv5x model by using a human body region detection sample, the positions of human bodies in the image frames, namely human body region frames, can be obtained, and then the coordinate positions of key points of the human bodies are extracted from the human body region frames, so that the coordinate positions of the key points of the human bodies of each image frame in a sleep post behavior image frame sequence can be obtained, and the spatial relationship of the coordinate positions of the key points of the human bodies of different image frames forms a sleep post skeleton spatiotemporal diagram.
108, training a behavior prediction model through a sleep post framework space-time diagram to obtain a trained behavior prediction model; the trained behavior prediction model comprises the following steps: a graph convolution layer, a time convolution layer, a pooling layer, a full-link layer, and an output layer.
The behavior prediction model is a space-time graph convolution model, which is a model for modeling dynamic bones based on time series representation of human joint positions and capturing the space-time change relationship by expanding graph convolution into a space-time graph convolution network. The behavior prediction model trained by the sleep post skeleton space-time diagram can be used for detecting unknown sleep post behaviors.
And step 110, carrying out the off-duty detection of the human body according to the trained YOLOv5x model, and if the YOLOv5x model outputs a human body region frame, carrying out the sleep-duty behavior detection on the human body according to the trained behavior prediction model.
And if the YOLOv5x model outputs a human body region frame, judging that the human body is not off duty, and detecting the sleep duty behavior of the human body which is not off duty according to the trained behavior prediction model.
The method for detecting the sleep shift prevention of the human body comprises the steps of firstly labeling human body regions of image frames in an image frame sequence to obtain human body region detection samples, labeling sleep shift behaviors in the image frame sequence to obtain a sleep shift behavior sequence image frame sequence, training a YOLOv5x model by using the human body region detection samples, training the YOLOv5x model, detecting human body targets by using the trained YOLOv5x model to judge whether the human body is off shift, extracting a skeleton of a judgment result, training a behavior prediction model by using a sleep shift skeleton space-time diagram to obtain a trained behavior prediction model, and detecting the sleep shift behaviors of the human body according to the trained behavior prediction model.
In one embodiment, the trained YOLOv5x model is used for human off-Shift detection, and if the YOLOv5x model does not output a human region box, the human is judged to be off-Shift.
If the YOLOv5x model does not output the body region box, indicating that the target body is not detected, the person is judged to be off duty.
In one embodiment, extracting the coordinate positions of the key points of the human body from the human body region frame comprises: inputting the human body region frame into a HigherHRNet network to obtain a thermodynamic diagram; the local maximum of the thermodynamic diagram is the coordinate position of the key point of the human body.
The highherhrnet network is used for scale-aware representation learning of human body pose estimation from bottom to top. And extracting ROI areas from the human body area frames, inputting the frames of suspected sleeping behaviors such as lying sleep, side sleep, head support and the like into a HigherHRNet network for human body posture estimation, generating a thermodynamic diagram, confirming the sleeping behaviors by highlighting in the thermodynamic diagram, and taking a local maximum value in the thermodynamic diagram as the coordinate position of a human body key point.
In one embodiment, the behavior prediction model further comprises an attention mechanism layer-ATT.
Because the sleep behaviors are more concentrated in the upper half area of the human body, the attention mechanism is added in the behavior prediction model, so that the focus of the behavior prediction can be concentrated in the upper half area of the human body, and the accuracy of the behavior prediction model is improved.
In one embodiment, the trained behavior prediction model further comprises an attention mechanism layer.
In the trained behavior prediction model, sleep behavior detection is mainly performed by a graph convolution layer, a time convolution layer and an attention mechanism layer.
In one embodiment, the detection of the sleep-post behavior of the human body according to the trained behavior prediction model comprises: and performing space-time skeleton feature extraction on the sleep post skeleton space-time diagram through a diagram convolution layer in the trained behavior prediction model, and outputting a detection result of sleep post behavior detection after passing through a time convolution layer, a pooling layer, a full-connection layer and an output layer in sequence.
In one embodiment, the time-space skeleton feature extraction is performed on the sleep post skeleton space-time diagram through a graph convolution layer in a trained behavior prediction model, and a detection result of sleep post behavior detection is output after the time convolution layer, the pooling layer, the full-link layer and the output layer pass through in sequence, and the method comprises the following steps:
carrying out graph convolution on the sleep post skeleton space-time diagram through a graph convolution layer in a trained behavior prediction model to obtain local characteristics of adjacent key points of a human body in a space;
convolving the local features of the adjacent key points of the human body in the space through a time convolution layer to obtain the local features of the change of the key points of the human body in the time;
and (3) sequentially passing the local characteristics of the change of the key points of the human body in time through the pooling layer, the full-connection layer and the output layer, and outputting a detection result of the sleep post behavior detection.
As shown in fig. 3, performing space-time skeleton feature extraction on the sleep post skeleton space-time diagram through a graph convolution layer-GCN, and outputting a behavior prediction result of a behavior prediction model after sequentially passing through a time convolution layer-TCN, a pooling layer-POOL, a full-link layer-FC and an output layer-OUT, wherein the local features of adjacent key points of a human body in a space are obtained by performing graph convolution on the sleep post skeleton space-time diagram through the graph convolution layer; convolving the local features of the adjacent key points of the human body in the space through a time convolution layer to obtain the local features of the change of the key points of the human body in the time; and sequentially passing the local characteristics of the change of the key points of the human body in time through the pooling layer, the full-connection layer and the output layer, and outputting the behavior prediction result of the behavior prediction model.
The key points of the human body are human joints, and in the sleeping behavior process, the time sequence of human body skeletons forms a series of space-time diagrams, namely three-dimensional diagrams formed by the spatial relationship of skeletons of different frames. And (3) adopting the convolution of the space-time diagram to aggregate information on the time dimension and the space dimension to extract the characteristics. The graph convolution layer extracts the relation of the coordinate points of the skeleton on the space to obtain the local characteristics of the adjacent joints in the space, the time is continuous due to the fixed number of the joints, the joints are continuously changed in the continuous time, and the time convolution layer is used for carrying out Time Convolution (TCN) operation to obtain the local characteristics of the joint change in the time. And if the local characteristics of the adjacent key points of the human body in the space and the local characteristics of the change of the key points of the human body in the time are consistent with the behavior characteristics marked on the sleep post, judging that the human body behavior is the sleep post behavior.
In one embodiment, the behavior prediction result is the local features of the adjacent key points of the human body in the space and the local features of the change of the key points of the human body in the time.
The local features refer to extracted features and are used for representing behavior characteristics.
It should be understood that, although the various steps in the flowchart of fig. 1 are shown in order as indicated by the arrows, the steps are not necessarily performed in order as indicated by the arrows. The steps are not performed in the exact order shown and described, and may be performed in other orders, unless explicitly stated otherwise. Moreover, at least a portion of the steps in fig. 1 may include multiple sub-steps or multiple stages that are not necessarily performed at the same time, but may be performed at different times, and the order of performance of the sub-steps or stages is not necessarily sequential, but may be performed in turn or alternately with other steps or at least a portion of the sub-steps or stages of other steps.
In one embodiment, as shown in fig. 2, there is provided an off-post sleep detection device, including: the video analysis module 201, the labeling module 202, the skeleton space-time diagram obtaining module 203, the behavior prediction model training module 204 and the behavior detection module 205, wherein:
the video decomposition module 201 is configured to obtain an off-Shift sleeping video shot by a camera, and decompose the off-Shift sleeping video to obtain an image frame sequence;
the labeling module 202 is used for labeling the human body region of the image frame in the image frame sequence to obtain a human body region detection sample, and labeling the sleep post behavior in the image frame sequence to obtain a sleep post behavior sequence image frame sequence;
the skeleton space-time diagram obtaining module 203 is used for training a Yolov5x model by using a human body region detection sample and outputting a human body region frame; extracting the coordinate positions of the key points of the human body from the human body region frame, and obtaining a sleep post skeleton space-time diagram according to the coordinate positions of the key points of the human body of each image frame in the sleep post behavior image frame sequence;
the behavior prediction model training module 204 is used for training a behavior prediction model through a sleep post framework space-time diagram to obtain a trained behavior prediction model; the trained behavior prediction model comprises the following steps: a graph convolution layer, a time convolution layer, a pooling layer, a full-link layer, and an output layer;
and the behavior detection module 205 is configured to perform off-post detection on the human body according to the trained YOLOv5x model, and if the YOLOv5x model outputs a human body region frame, perform sleep-post behavior detection on the human body according to the trained behavior prediction model.
In one embodiment, the behavior detection module 205 is further configured to perform human off-Shift detection using the trained YOLOv5x model, and determine that the person is off-Shift if the YOLOv5x model does not output the human region box.
In one embodiment, the skeleton space-time diagram obtaining module 203 is further configured to input the human body region box into a highherhrnet network to obtain a thermodynamic diagram; the local maximum of the thermodynamic diagram is the coordinate position of the key point of the human body.
In one embodiment, the behavior prediction model training module 204 further comprises an attention mechanism layer.
In one embodiment, the behavior detection module 205 is further configured to perform sleep-post behavior detection on the human body according to the trained behavior prediction model, including: and performing space-time skeleton feature extraction on the sleep post skeleton space-time diagram through a diagram convolution layer in the trained behavior prediction model, and outputting a detection result of sleep post behavior detection after passing through a time convolution layer, a pooling layer, a full-connection layer and an output layer in sequence.
In one embodiment, the behavior detection module 205 is further configured to perform graph convolution on the sleep post skeleton space-time diagram through a graph convolution layer in the trained behavior prediction model to obtain local features of adjacent key points of the human body in the space; convolving the local features of the adjacent key points of the human body in the space through a time convolution layer to obtain the local features of the change of the key points of the human body in the time; and (3) sequentially passing the local characteristics of the change of the key points of the human body in time through the pooling layer, the full-connection layer and the output layer, and outputting a detection result of the sleep post behavior detection.
For the specific definition of the off-shift prevention sleeping position detection device, refer to the above definition of the off-shift prevention sleeping position detection method, and are not described herein again. All modules in the anti-falling sleep post detection device can be completely or partially realized through software, hardware and a combination thereof. The modules can be embedded in a hardware form or independent from a processor in the computer device, and can also be stored in a memory in the computer device in a software form, so that the processor can call and execute operations corresponding to the modules.
In one embodiment, a computer device is provided, which may be a terminal, and its internal structure diagram may be as shown in fig. 4. The computer device includes a processor, a memory, a network interface, a display screen, and an input device connected by a system bus. Wherein the processor of the computer device is configured to provide computing and control capabilities. The memory of the computer device comprises a nonvolatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of an operating system and computer programs in the non-volatile storage medium. The network interface of the computer device is used for communicating with an external terminal through a network connection. The computer program is executed by a processor to realize an anti-falling asleep post detection method. The display screen of the computer equipment can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer equipment can be a touch layer covered on the display screen, a key, a track ball or a touch pad arranged on the shell of the computer equipment, an external keyboard, a touch pad or a mouse and the like.
Those skilled in the art will appreciate that the architecture shown in fig. 4 is merely a block diagram of some of the structures associated with the disclosed aspects and is not intended to limit the computing devices to which the disclosed aspects apply, as particular computing devices may include more or less components than those shown, or may combine certain components, or have a different arrangement of components.
In an embodiment, a computer device is provided, comprising a memory storing a computer program and a processor implementing the steps of the method in the above embodiments when the processor executes the computer program.
In an embodiment, a computer storage medium is provided, on which a computer program is stored, which, when being executed by a processor, carries out the steps of the method in the above-mentioned embodiments.
It will be understood by those skilled in the art that all or part of the processes of the methods of the embodiments described above can be implemented by hardware instructions of a computer program, which can be stored in a non-volatile computer-readable storage medium, and when executed, can include the processes of the embodiments of the methods described above. Any reference to memory, storage, database, or other medium used in the embodiments provided herein may include non-volatile and/or volatile memory, among others. Non-volatile memory can include read-only memory (ROM), Programmable ROM (PROM), Electrically Programmable ROM (EPROM), Electrically Erasable Programmable ROM (EEPROM), or flash memory. Volatile memory can include Random Access Memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in a variety of forms such as Static RAM (SRAM), Dynamic RAM (DRAM), Synchronous DRAM (SDRAM), Double Data Rate SDRAM (DDRSDRAM), Enhanced SDRAM (ESDRAM), Synchronous Link DRAM (SLDRAM), Rambus Direct RAM (RDRAM), direct bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).
The technical features of the above embodiments can be arbitrarily combined, and for the sake of brevity, all possible combinations of the technical features in the above embodiments are not described, but should be considered as the scope of the present specification as long as there is no contradiction between the combinations of the technical features.
The above-mentioned embodiments only express several embodiments of the present application, and the description thereof is more specific and detailed, but not construed as limiting the scope of the invention. It should be noted that, for a person skilled in the art, several variations and modifications can be made without departing from the concept of the present application, which falls within the scope of protection of the present application. Therefore, the protection scope of the present patent shall be subject to the appended claims.
Claims (10)
1. A method for detecting falling off of a sleeping post is characterized by comprising the following steps:
acquiring an off-Shift sleeping-on-Shift video shot by a camera;
decomposing the off-Shift sleeping Shift video to obtain an image frame sequence;
marking a human body region of the image frames in the image frame sequence to obtain a human body region detection sample, and marking the sleep-off behavior in the image frame sequence to obtain a sleep-off behavior sequence image frame sequence;
training a YOLOv5x model by using the human body region detection sample, and outputting a human body region frame;
extracting the coordinate positions of the key points of the human body from the human body region frame, and obtaining a sleep post skeleton space-time diagram according to the coordinate positions of the key points of the human body of each image frame in the sleep post behavior image frame sequence;
training a behavior prediction model through the sleep post skeleton space-time diagram to obtain a trained behavior prediction model; the trained behavior prediction model comprises: a graph convolution layer, a time convolution layer, a pooling layer, a full-link layer, and an output layer;
and carrying out off-duty detection on the human body according to the trained YOLOv5x model, and if the YOLOv5x model outputs a human body region frame, carrying out sleep-duty behavior detection on the human body according to the trained behavior prediction model.
2. The method of claim 1, further comprising:
and (3) carrying out human off-Shift detection by using the trained YOLOv5x model, and if the YOLOv5x model does not output a human region frame, judging that the person is off-Shift.
3. The method of claim 1, wherein extracting coordinate locations of human keypoints from the human region box comprises:
inputting the human body region box into a HigherHRNet network to obtain a thermodynamic diagram; the local maximum of the thermodynamic diagram is the coordinate position of a key point of the human body.
4. The method of any of claims 1 to 3, wherein the trained behavioral prediction model further comprises an attention mechanism layer.
5. The method of claim 4, wherein the graph convolutional layer, the time convolutional layer, and the attention mechanism layer comprise a sleep behavior detection network.
6. The method of claim 1, wherein the detection of sleep behavior of the human body according to the trained behavior prediction model comprises:
and performing space-time skeleton feature extraction on the sleep post skeleton space-time diagram through a diagram convolution layer in the trained behavior prediction model, and outputting a detection result of sleep post behavior detection after passing through a time convolution layer, a pooling layer, a full-connection layer and an output layer in sequence.
7. The method of claim 6, wherein the time-space skeleton feature extraction is performed on the sleep post skeleton space-time diagram through a graph convolution layer in the trained behavior prediction model, and the detection result of the sleep post behavior detection is output after the time convolution layer, the pooling layer, the full-link layer and the output layer are sequentially passed through:
carrying out graph convolution on the sleep post skeleton space-time diagram through a graph convolution layer in a trained behavior prediction model to obtain local characteristics of adjacent key points of a human body in a space;
convolving the local features of the adjacent key points of the human body in the space through a time convolution layer to obtain the local features of the change of the key points of the human body in the time;
and sequentially enabling the local characteristics of the change of the key points of the human body in the time to pass through the pooling layer, the full-connection layer and the output layer, and outputting a detection result of the sleep post behavior detection.
8. The utility model provides an anticreep post detection device that sleeps which characterized in that, the device includes:
the video decomposition module is used for acquiring the off-Shift sleeping on-Shift video shot by the camera and decomposing the off-Shift sleeping on-Shift video to obtain an image frame sequence;
the labeling module is used for labeling the human body region of the image frames in the image frame sequence to obtain a human body region detection sample, and labeling the sleep post behavior in the image frame sequence to obtain a sleep post behavior sequence image frame sequence;
a skeleton space-time diagram obtaining module, configured to train a YOLOv5x model using the human body region detection sample, and output a human body region frame; extracting the coordinate positions of the key points of the human body from the human body region frame, and obtaining a sleep post skeleton space-time diagram according to the coordinate positions of the key points of the human body of each image frame in the sleep post behavior image frame sequence;
the behavior prediction model training module is used for training a behavior prediction model through the sleep post framework space-time diagram to obtain a trained behavior prediction model; the trained behavior prediction model comprises: a graph convolution layer, a time convolution layer, a pooling layer, a full-link layer, and an output layer;
and the behavior detection module is used for detecting the off-duty state of the human body according to the trained YOLOv5x model, and if the YOLOv5x model outputs a human body region frame, detecting the sleep-duty behavior of the human body according to the trained behavior prediction model.
9. A computer device comprising a memory and a processor, the memory storing a computer program, wherein the processor implements the steps of the method of any one of claims 1 to 7 when executing the computer program.
10. A computer-readable storage medium, on which a computer program is stored, which, when being executed by a processor, carries out the steps of the method of any one of claims 1 to 7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111412259.3A CN114037943A (en) | 2021-11-25 | 2021-11-25 | Method and device for detecting falling-off prevention sleeping sentry |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111412259.3A CN114037943A (en) | 2021-11-25 | 2021-11-25 | Method and device for detecting falling-off prevention sleeping sentry |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN114037943A true CN114037943A (en) | 2022-02-11 |
Family
ID=80138796
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202111412259.3A Pending CN114037943A (en) | 2021-11-25 | 2021-11-25 | Method and device for detecting falling-off prevention sleeping sentry |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN114037943A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115346169A (en) * | 2022-08-08 | 2022-11-15 | 航天神舟智慧系统技术有限公司 | Method and system for detecting sleep post behaviors |
-
2021
- 2021-11-25 CN CN202111412259.3A patent/CN114037943A/en active Pending
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115346169A (en) * | 2022-08-08 | 2022-11-15 | 航天神舟智慧系统技术有限公司 | Method and system for detecting sleep post behaviors |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| WO2021047232A1 (en) | Interaction behavior recognition method, apparatus, computer device, and storage medium | |
| CN108446585B (en) | Target tracking method and device, computer equipment and storage medium | |
| CN109241904B (en) | Character recognition model training, character recognition method, device, equipment and medium | |
| JP7447932B2 (en) | Image processing system, image processing method and program | |
| CN110490902B (en) | Target tracking method and device applied to smart city and computer equipment | |
| Ribaux et al. | Forensic intelligence | |
| CN113239874B (en) | Behavior gesture detection method, device, equipment and medium based on video image | |
| CN111080963A (en) | Construction site warning method and device, computer equipment and storage medium | |
| CN111275743B (en) | Target tracking method, device, computer readable storage medium and computer equipment | |
| CN112560796B (en) | Human body posture real-time detection method and device, computer equipment and storage medium | |
| Filtjens et al. | Skeleton-based action segmentation with multi-stage spatial-temporal graph convolutional neural networks | |
| CN112364715A (en) | Nuclear power operation abnormity monitoring method and device, computer equipment and storage medium | |
| CN110717449A (en) | Vehicle annual inspection personnel behavior detection method and device and computer equipment | |
| CN111881853A (en) | Method and device for identifying abnormal behaviors in oversized bridge and tunnel | |
| CN111144398A (en) | Target detection method, target detection device, computer equipment and storage medium | |
| CN114037943A (en) | Method and device for detecting falling-off prevention sleeping sentry | |
| CN116229560B (en) | Abnormal behavior recognition method and system based on human body posture | |
| CN113516778A (en) | Model training data acquisition method and device, computer equipment and storage medium | |
| CN110490079B (en) | Patrol data processing method and device, computer equipment and storage medium | |
| CN111144372A (en) | Vehicle detection method, device, computer equipment and storage medium | |
| CN113449586A (en) | Target detection method, target detection device, computer equipment and storage medium | |
| CN114119531A (en) | Fire detection method and device applied to campus smart platform and computer equipment | |
| CN111191706A (en) | Picture identification method, device, equipment and storage medium | |
| CN113963311B (en) | Safe production risk video monitoring method and system | |
| CN110516559A (en) | Suitable for precisely monitor method for tracking target and device, computer equipment |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| TA01 | Transfer of patent application right | ||
| TA01 | Transfer of patent application right |
Effective date of registration: 20231227 Address after: 410000 No. 96, tongzipo Road, Yuelu District, Changsha City, Hunan Province Applicant after: Hunan Zhongke Zhuying Intelligent Technology Research Institute Co.,Ltd. Applicant after: National University of Defense Technology Address before: 410000 No. 96, tongzipo Road, Yuelu District, Changsha City, Hunan Province Applicant before: Hunan Zhongke Zhuying Intelligent Technology Research Institute Co.,Ltd. |