CN113850229B - Personnel abnormal behavior early warning method and system based on video data machine learning and computer equipment - Google Patents
Personnel abnormal behavior early warning method and system based on video data machine learning and computer equipment Download PDFInfo
- Publication number
- CN113850229B CN113850229B CN202111211548.7A CN202111211548A CN113850229B CN 113850229 B CN113850229 B CN 113850229B CN 202111211548 A CN202111211548 A CN 202111211548A CN 113850229 B CN113850229 B CN 113850229B
- Authority
- CN
- China
- Prior art keywords
- data
- video
- person
- abnormal
- early warning
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 206010000117 Abnormal behaviour Diseases 0.000 title claims abstract description 71
- 238000000034 method Methods 0.000 title claims abstract description 54
- 238000010801 machine learning Methods 0.000 title claims description 34
- 230000005021 gait Effects 0.000 claims abstract description 19
- 238000013528 artificial neural network Methods 0.000 claims abstract description 14
- 230000033001 locomotion Effects 0.000 claims abstract description 10
- 238000010586 diagram Methods 0.000 claims abstract description 9
- 238000004364 calculation method Methods 0.000 claims abstract description 8
- 238000007781 pre-processing Methods 0.000 claims abstract description 7
- 238000004140 cleaning Methods 0.000 claims abstract description 6
- 230000006399 behavior Effects 0.000 claims description 42
- 238000012544 monitoring process Methods 0.000 claims description 35
- 230000002159 abnormal effect Effects 0.000 claims description 26
- 230000008569 process Effects 0.000 claims description 14
- 238000001514 detection method Methods 0.000 claims description 13
- 238000012806 monitoring device Methods 0.000 claims description 12
- 238000012545 processing Methods 0.000 claims description 12
- 230000005540 biological transmission Effects 0.000 claims description 10
- 230000006996 mental state Effects 0.000 claims description 8
- 230000008921 facial expression Effects 0.000 claims description 6
- 238000000926 separation method Methods 0.000 claims description 4
- 230000009466 transformation Effects 0.000 claims description 2
- 230000000994 depressogenic effect Effects 0.000 claims 1
- 238000013473 artificial intelligence Methods 0.000 abstract description 3
- 238000004422 calculation algorithm Methods 0.000 description 8
- 238000007726 management method Methods 0.000 description 7
- 238000004891 communication Methods 0.000 description 5
- 238000005516 engineering process Methods 0.000 description 5
- 230000006870 function Effects 0.000 description 5
- 230000005856 abnormality Effects 0.000 description 4
- 238000010276 construction Methods 0.000 description 4
- 238000000605 extraction Methods 0.000 description 4
- 238000004458 analytical method Methods 0.000 description 3
- 238000013461 design Methods 0.000 description 3
- 230000008451 emotion Effects 0.000 description 3
- 230000002787 reinforcement Effects 0.000 description 3
- 238000012549 training Methods 0.000 description 3
- 230000000007 visual effect Effects 0.000 description 3
- 230000008859 change Effects 0.000 description 2
- 238000011156 evaluation Methods 0.000 description 2
- 238000013401 experimental design Methods 0.000 description 2
- 230000014509 gene expression Effects 0.000 description 2
- 230000003993 interaction Effects 0.000 description 2
- 239000011159 matrix material Substances 0.000 description 2
- 230000003287 optical effect Effects 0.000 description 2
- 238000005457 optimization Methods 0.000 description 2
- 230000000630 rising effect Effects 0.000 description 2
- 230000033772 system development Effects 0.000 description 2
- 238000012795 verification Methods 0.000 description 2
- 206010003805 Autism Diseases 0.000 description 1
- 208000020706 Autistic disease Diseases 0.000 description 1
- 206010063659 Aversion Diseases 0.000 description 1
- 230000004075 alteration Effects 0.000 description 1
- 238000013145 classification model Methods 0.000 description 1
- 238000004590 computer program Methods 0.000 description 1
- 238000007405 data analysis Methods 0.000 description 1
- 238000013480 data collection Methods 0.000 description 1
- 238000013523 data management Methods 0.000 description 1
- 238000000586 desensitisation Methods 0.000 description 1
- 230000009429 distress Effects 0.000 description 1
- 230000009977 dual effect Effects 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 230000002996 emotional effect Effects 0.000 description 1
- 238000013210 evaluation model Methods 0.000 description 1
- 230000001815 facial effect Effects 0.000 description 1
- 239000008187 granular material Substances 0.000 description 1
- 235000019580 granularity Nutrition 0.000 description 1
- 230000003340 mental effect Effects 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000003909 pattern recognition Methods 0.000 description 1
- 230000001766 physiological effect Effects 0.000 description 1
- 230000002265 prevention Effects 0.000 description 1
- 238000005070 sampling Methods 0.000 description 1
- 230000008054 signal transmission Effects 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
- 230000008961 swelling Effects 0.000 description 1
- 238000012800 visualization Methods 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N20/00—Machine learning
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
Abstract
The invention relates to the field of artificial intelligence, in particular to a personnel abnormal behavior early warning method, a system and computer equipment, wherein the method comprises the steps of obtaining video image data and cleaning the video image data; preprocessing video image data; identifying the human body movement modes, setting classification labels for each type, and carrying out multi-mode feature coding; removing redundant characteristics of the data center and selecting characteristics through information gain; constructing a tensor neural network to track a person appearing in a video, detecting gait based on a gait energy diagram according to the tracked task outline, detecting the skin exposure of a task in the video, taking the exposure as an appearance clothing index of the person, calculating the correlation of images among frames of the video by integrating the indexes, and judging the abnormal behavior of the person according to the result of the correlation calculation; the invention can improve the field management and control work efficiency, reduce the work load and enhance the early warning error control.
Description
Technical Field
The invention relates to the field of artificial intelligence, in particular to a method, a system and computer equipment for early warning of abnormal behaviors of personnel based on machine learning of video data.
Background
Abnormal behavior in personnel in an intelligent park (hospitals, campuses, sites and the like) occurs, and sometimes even the life is at the cost of life (such as self-abandonment of a jump, murder of others and the like). The video image monitoring and other data are fully utilized in the intelligent park informatization construction, the abnormal behaviors of personnel in the park are analyzed, and decision support in the aspects of early warning and the like is provided in time. Based on relevant artificial intelligence technologies such as ontology knowledge, reinforcement learning, three-dimensional human body posture estimation and the like, new technologies are developed to improve decision support capability of monitoring video images of park personnel, and particularly, prevention and management capability of abnormal behavior risks (particularly life self-abandoning behaviors and the like) are improved.
Negative and abnormal rises are the abnormal behavior states of great concern for the present system. The face detection and the video image data reinforcement learning are carried out, and the characteristics extraction of the appearance clothes, the body state, the gait, the pace, the facial expression, the mental state and the like is carried out, so that three types of the emotional states (normal, sinking and abnormal rising) of the personnel are mainly judged. Since most life self-disclaimers will show their intention in the form of emotion, behavior, etc., these are non-negligible distress signals. The negative state is mainly represented by sad depression, autism, helplessness, destinkage and other negative characteristic information which are revealed for a long time. The abnormal rise is manifested as a sudden, unjustified full-of-passion. For example, by the trajectory extracted from the video image data, analysis results in that a person looks calm at ordinary times, suddenly a day shows full of passion, and the person gets through close contact with the like to know what is particularly important that the person has not happened recently. Then the person belongs to the abnormal behavior state, and warning information is given in the early warning decision support system.
Disclosure of Invention
In order to be able to conduct abnormal prediction on behaviors of personnel in a park in some specific parks and reduce special event occurrence, the invention provides a personnel abnormal behavior early warning method, a system and computer equipment based on video data machine learning, wherein the method comprises the following steps:
s1, acquiring video image data, and cleaning the video image data;
s2, preprocessing video image data, namely removing privacy-related information in the data, and improving video definition of a blurred video by adopting a frame quality enhancement method;
s3, recognizing human body movement modes, setting a classification label for each type, and performing multi-mode feature coding;
s4, performing feature engineering processing on the multi-mode codes, namely removing redundant features of the data center and selecting features through information gain;
s5, constructing a tensor neural network to track people appearing in the video, detecting gait based on a gait energy diagram according to the tracked task outline, detecting skin exposure of the task in the video, taking the exposure as an appearance clothing index of the people, calculating correlation of images among video frames according to the indexes, and judging abnormal behaviors of the people according to the correlation calculation result;
s6, constructing early warning decision support, namely if the dynamic information of the similarity and the difference between the acquired frame images shows that the behavior of the photographed person is abnormal, informing relevant persons to carry out the treatment.
Further, after the feature engineering processing is performed in the step S3, the data is detected, and if the non-random missing data exists and the data is made, the data is inferred by adopting a statistical machine learning method and double robustness, so that the complete data of the data is obtained.
Further, the multi-modal feature encoding specifically includes the following processes: using a 5-dimensional tensor structure for the acquired image, encoding video data by (samples, frames, height, width, channels) to represent 5-dimensional tensor data of samples, frames, height, width, and channel of the video data, respectively; the acquired video image is encoded with 4-dimensional tensor structure by (samples, height, width, channels) video data, and 4-dimensional tensor data representing samples, height, width, and channel of the video data, respectively.
Further, tracking the person appearing in the video includes:
taking the multi-mode coded image as input of a tensor neural network;
the tensor neural network comprises human body gesture recognition, pedestrian detection and separation of foreground and background;
acquiring a human body gesture key sequence point in an image through human body gesture recognition;
detecting a person through pedestrian detection, dividing a task from an image, and acquiring a gesture outline sequence of the person by combining separation of a foreground and a background;
and tracking and identifying the person through the acquired gesture key sequence points and gesture outline sequences.
Further, the process of detecting the skin exposure of the task in the video takes the exposure as an appearance clothing index of the person, namely, the skin area of the person in the image is obtained based on the elliptic skin model, and the proportion of the exposed skin area to the outline of the person is taken as the appearance clothing index.
Further, after the gait detection and appearance clothing indexes are carried out, a sliding window with the length of N frames and the step length of t is set, the gait detection and appearance clothing indexes are valued through the sliding window, and the average value of the data of each window is taken as the data of correlation calculation.
Further, in the case of performing the difference determination, the correlation between the two images is calculated by the euclidean distance.
Further, when the behavior abnormality of the person is judged, if the similarity of the two images extracted in the S frame is lower than a certain threshold value, the task abnormality is judged.
The invention also provides a personnel abnormal behavior early warning system, which comprises a front terminal system, a transmission network and a monitoring center, wherein the front terminal system comprises various monitoring devices, a front terminal subsystem transmits information acquired by the monitoring devices to the monitoring center through the transmission network, the monitoring center comprises an abnormal behavior prediction device, the device predicts whether a shot person is abnormal or not through the collected information, if so, the monitoring center gives an alarm and projects the monitoring device capable of shooting the person to a monitor center display for real-time monitoring.
The invention also provides personnel abnormal behavior early-warning computer equipment, which comprises a memory and a processor, wherein the memory is used for storing any of the personnel abnormal behavior early-warning methods, and the processor is used for running the method stored in the memory to predict whether the personnel in the monitor has abnormal behaviors.
The invention also provides a personnel abnormal behavior early warning system, which comprises a front terminal system, a transmission network and a monitoring center, wherein the front terminal system comprises various monitoring devices, the front terminal system transmits information acquired by the monitoring devices to the monitoring center through the transmission network, the monitoring center comprises an abnormal behavior prediction device, the device predicts whether a shot personnel has abnormality or not through the collected information, if the shot personnel has abnormality, the monitoring center gives an alarm and the monitoring device capable of shooting the personnel is projected onto a monitor center display for real-time monitoring.
The invention also provides personnel abnormal behavior early-warning computer equipment, which comprises a memory and a processor, wherein the memory is used for storing any one of the personnel abnormal behavior early-warning methods, and the processor is used for running the method stored in the memory to predict whether the personnel in the monitor has abnormal behaviors.
The invention constructs an abnormal behavior pattern classification model set based on machine learning, and a method for identifying abnormal behaviors according to different granularities of human behavior features, covers main requirements on a face data matching model, walking gait identification and mental state assessment, improves the intelligent level of an early warning decision support system, and plays an important role in monitoring video image data utilization and multiplexing; through processes such as video image sampling, preprocessing, encoding, feature extraction, data association and the like, an early warning database based on video images is constructed, technical support is provided for knowledge reasoning and data management of high-dimensional abnormal behavior identification data, on-site management and control working efficiency can be improved, workload is reduced, early warning error control is enhanced, service modes of an intelligent park information sharing platform are enriched, and application value of an early warning control panel is improved.
Drawings
FIG. 1 is a schematic diagram of a live video image monitoring network architecture as disclosed in an embodiment of the present invention;
fig. 2 is a schematic diagram of an abnormal behavior early warning flow based on monitoring video image machine learning according to an embodiment of the present invention;
FIG. 3 is a schematic diagram of another abnormal behavior early warning flow based on monitoring video image machine learning according to an embodiment of the present invention;
fig. 4 is a schematic information interaction diagram of abnormal behavior early warning based on monitoring video image machine learning according to the embodiment of the invention;
fig. 5 is a schematic structural diagram of an abnormal behavior early warning device for monitoring video image machine learning according to an embodiment of the present invention;
fig. 6 is a schematic structural diagram of another abnormal behavior early warning device for monitoring video image machine learning according to an embodiment of the present invention.
Detailed Description
The following description of the embodiments of the present invention will be made clearly and completely with reference to the accompanying drawings, in which it is apparent that the embodiments described are only some embodiments of the present invention, but not all embodiments. All other embodiments, which can be made by those skilled in the art based on the embodiments of the invention without making any inventive effort, are intended to be within the scope of the invention.
The invention provides a personnel abnormal behavior early warning method based on video data machine learning, which specifically comprises the following steps:
s1, acquiring video image data, and cleaning the video image data;
s2, preprocessing video image data, namely removing privacy-related information in the data, and improving video definition of a blurred video by adopting a frame quality enhancement method;
s3, recognizing human body movement modes, setting a classification label for each type, and performing multi-mode feature coding;
s4, performing feature engineering processing on the multi-mode codes, namely removing redundant features of the data center and selecting features through information gain;
s5, constructing a tensor neural network to perform correlation calculation on the inter-frame key semantic region by using the mental state evaluation model, the walking gait recognition model and the face data matching model;
s6, building early warning decision support, namely informing relevant personnel to carry out treatment if the behavior of the shot object is abnormal.
Example 1
The embodiment provides a personnel abnormal behavior early warning system based on video data machine learning, which comprises a front terminal system, a transmission network and a monitoring center, wherein the front terminal system comprises various monitoring equipment, the front terminal system transmits information collected by the monitoring equipment to the monitoring center through the transmission network, the monitoring center comprises an abnormal behavior prediction device, the device predicts whether a shot personnel is abnormal or not through the collected information, if so, the monitoring center gives an alarm and projects the monitoring equipment capable of shooting the personnel onto a monitor center display for real-time monitoring.
In the embodiment, the front terminal system is mainly responsible for field image acquisition, video storage, alarm receiving and sending, other sensor data acquisition and network transmission; the transmission network mainly realizes data uploading between the operation site and the monitoring center through a special line and the Internet; the monitoring center is a place for executing daily monitoring, system management and emergency early warning command. The functions of remote video monitoring on an operation site, remote cloud control of the rotation of the dome camera, remote receiving of site alarm, remote voice dialogue command with the site and the like are realized through the site monitoring system; the manager can know the early warning and emergency management and control process of the site in real time. And managing the real-time monitoring camera, and supporting the functions of backing up, inquiring and replaying the monitoring images according to the information such as the camera numbers, time, events and the like through the hard disk video recorder. Multiple pictures are supported for simultaneous playback on the same display, up to 25 pictures being simultaneously played back in synchronization. And remote real-time control of the camera holder and the lens is supported. The system and the device integrate a plurality of functions such as 5G wireless communication, equipment identification, data acquisition, personnel activity state detection and the like, and adopt an IEEE802.15.4 communication protocol, so that the technology is mature and stable. The method has the technical advantages of lower transmitting power, higher positioning precision (2-5 m), wider wireless coverage range (500 m), better communication confidentiality and the like.
In this embodiment, the on-site video monitoring network architecture for early warning of abnormal behaviors of personnel is shown in fig. 1, and on one side of a monitoring center, a ground central station host 102 is powered by an uninterruptible power supply UPS101, a machine learning and early warning management and control 103 is performed by a rack-mounted video server, the machine learning and early warning management and control 103 is connected to a bus by a switch 106, a matrix 104 is connected with a hard disk video recorder 105, and video image archive data of high-risk personnel is recorded; on the front subsystem side, a video/audio optical transceiver 109, a signal transmission device, a camera gun 110, a camera 111 and an audio speaker 12 are connected through a switch 108, and early warning information is controlled by an early warning network 113 and the result is transmitted to a sharing network. The user terminal may include a cell phone, a laptop, a palmtop, a smart stereo, a wearable device (e.g., a smart watch, a smart bracelet, etc.), etc.
Example 2
The embodiment provides a personnel abnormal behavior early warning method based on video data machine learning, as shown in fig. 2, specifically comprising the following steps:
201. personnel pattern recognition in surveillance video images
Feature extraction is carried out on the monitoring video image, and feature information of abnormal behaviors of a human body, including information of appearance clothes, body posture, gait, pace, facial expression, mental state and the like, is effectively captured. The system understands the behaviors and analyzes and identifies the motion modes of the human body, and describes the behaviors.
202. Ghost data collection and reasoning
Through the experimental design process, whether ghost data in the forms of non-random missing data, working data and the like exist is checked from the data. If so, verification is performed using a construction model and reasoning is performed using statistical machine learning methods and dual robustness.
The above reasoning process for the data can be performed by using an existing model, and will not be described here again.
203. Abnormal behavior reasoning based on machine learning
A method for analyzing and extracting key semantic regions from an image. And (3) analyzing and extracting key semantic information in the video image by using the design tensor neural network, and decomposing the complete image into a plurality of image groups according to different semantic information. Each image group expresses a semantic information, so that the human behavior space information can be expressed more finely and pertinently. Inter-frame critical semantic granule multi-frame joint enhancement is performed by using tensor convolution neural network.
204. Determination of whether behavior is abnormal
Through continuous optimization of machine learning, through modeling, system development and the like, whether the personnel behaviors are abnormal or not is judged. And the monitoring and early warning support system for the abnormal behaviors of the personnel of the video image data is realized through the judgment of the abnormal behaviors of the negative and abnormal swelling.
205. Early warning decision support control panel
If the abnormal behaviors of the personnel exist, information reminding is pushed, and abnormal behaviors of the personnel are monitored, early warning support and visualization are achieved.
Example 3
The embodiment proposes a more specific implementation manner based on embodiment 2, as shown in fig. 3, and the early warning method in the embodiment includes the following steps:
starting 301; the system will use the video image data intelligent collection flow 302 to upload the surveillance video as metadata 303 into the system;
the input data is subjected to data cleaning 304 such as extraction-Transform-Load (ETL), and this process mainly cleans low-quality video images in the source data to eliminate data which does not meet the specification. The method comprises the steps of realizing missing data processing, similar repeated object detection, abnormal data processing, logic error detection, inconsistent data processing and the like;
preprocessing 306 is performed on the video image data after cleaning, which includes two aspects:
the information 305 related to privacy in the data is removed, so that the data quality is improved;
for the blurred video image 307, the system pre-adopts a frame quality enhancement method to improve the video definition;
analyzing and identifying 308 the motion pattern of the human body by using an algorithm;
classifying the data after preprocessing, and assigning a classification label 309 to each type;
the multi-mode feature code 311 performs space-time desensitization data association 312 through personnel identification of the system; in combination with face recognition, using semi-structural data, such as video recorded by a sensor, corresponding marks, other life bodies, physiological activities and 3D behavior mark data, a plurality of abnormal behavior case libraries 310 of cascading personnel monitoring video images, three-dimensional posture human body moving images and the like, searching corresponding behavior specifications and monitoring information, realizing an early warning function module, and providing timely high-quality abnormal behavior risk management recommendation information;
these codes are subjected to feature engineering 314 to remove redundant features from the data, and the features are selected as inputs for use by models and algorithms by information gain. Mainly comprises two tasks: automatically selecting an algorithm; the expert participates in selecting a common feature set; the system understands behaviors, analyzes and identifies the motion modes of the human body, and describes the motion modes; simultaneously, feature extraction is carried out, so that feature information of abnormal behaviors of a human body, including information of appearance clothes, body states, gait, pace, facial expressions, mental states and the like, is effectively captured; dividing the data into three types of normal, depression and emotion rising, taking the characteristic data extracted from the video image as a conditional attribute, marking the category as a decision attribute, and storing the marked category into a medium to serve as a training case set;
the ghost data acquisition 313 is carried out through experimental design, and ghost data reasoning is carried out through methods such as Fourier transformation, hadamard matrix, hidden variables and the like;
the characteristics are mainly characterized in three aspects: basic information such as mental appearance, movement state, face data and the like, thus constructing three corresponding models: a mental state assessment model 317, a walking gait recognition model 318 and a face data matching model 319. Through knowledge discovery, optimization prediction, data analysis and other processes; analyzing and responding to the video image through related principles and methods such as image identification, pattern identification, tensor neural network learning and the like; the training learning algorithm can provide feedback information, improve parameter learning and model adjustment processes, analyze and extract key semantic areas from images, construct a neural network to perform correlation calculation on the inter-frame key semantic areas, and connect the areas with consistent inter-frame semantic information in series to obtain an inter-frame key semantic area chain, so that consistency of semantic information is ensured, time domain change information of human behaviors is expressed more pertinently, and the whole network utilizes global and local residual structures to reduce training difficulty. The system is continuously optimized through machine learning, and through modeling, system development and the like, a personnel abnormal behavior monitoring and early warning support system of video image data of an intelligent park is realized, and an evaluation index is provided by using a subject working characteristic curve (receiver operating characteristic curve, ROC) and an area under the curve (Area Under ROC Curve, AUC), so that the comprehensive evaluation is carried out on the model performance, and a prediction model and an algorithm are further feedback controlled;
judging whether the human body behaviors are abnormal 320, if the human body behaviors are abnormal, adding the information into an early warning database by the system and providing prompt information for early warning grades, verifying a human body behavior recognition method by using a related data set by using the abnormal human body behavior judgment accuracy, calculating the average value of various human body behavior recognition accuracy as the result of human body behavior recognition, wherein the human body behavior recognition accuracy is the percentage of the number of video images which can be correctly recognized in a certain type of behaviors to the total number of video images of the certain type of behaviors, and the certain type of behaviors alpha i Accuracy H of (2) i The calculation is as follows:
in the formula, the condition alpha i Representing the behavior of class i in practice, condition beta i Representing behavior identified by the algorithm as class i; operator I represents calculating the number of video frames meeting the condition;
the face data is matched 321 with the face database of the high risk group of table 2, and if the high risk group belongs to a high risk individual, the system associates personal information 324. The individual is classified into early warning grades by combining other abnormal information through the stages of lookup table processing, file association and dynamic matching, common sense logic judgment, field common sense judgment, data correlation verification and the like, and the early warning database 322 is added;
the system can set early warning parameters, when the real-time data exceeds the early warning range, the system can autonomously judge and send out warning signals, and according to actual requirements, the system interface popup window prompt information 323 is connected with an audible and visual alarm to realize audible and visual early warning;
remote early warning and linkage control 325, detecting multipath early warning signals, when early warning occurs, automatically starting various corresponding linkage equipment, switching video to a corresponding camera, triggering automatic video recording, alarming to a monitoring center through a network, and popping up an alarm information prompt from a client until the end 326;
the obtained abnormal data information is stored in an abnormal behavior case library, the video image realizes data reuse and feedback control 316 is carried out on the individual behaviors, the monitoring data of each monitoring point on high-risk personnel are displayed in real time, the real-time data are continuously mined, analyzed and displayed in a rich on-line analysis graph, the information is assisted in a graphical display form, the history monitoring data of the high-risk personnel are displayed in a graph form and the like, and a history data report is output and can be printed.
Table 1 example of feature data portion extracted by video image data and associated data
The person skilled in the art scores each item of data in table 1 according to actual conditions and assigns weights, when the weighted total score exceeds a threshold value, the person is judged to be abnormal, in the detection process, the correlation between two images is calculated through Euclidean distance, if the person is normal but the emotion, gait and the like are changed severely, for example, the facial expression is excited to aversion, the pace speed is very slow to very fast and the like, the data acquisition equipment is required to be enhanced for tracking when the situation occurs, and if the extreme change occurs, the person is listed as an abnormal person and the relevant person is notified.
Example 4
The embodiment proposes an implementation manner of a personnel abnormal behavior early warning system based on video data machine learning based on embodiment 1, as shown in fig. 4, the front terminal system corresponds to a video monitoring device, the server corresponds to an abnormal behavior prediction device in a monitoring center, the control panel terminal corresponds to a display in the monitoring system, and the interaction manner of the video monitoring device, the server and the control panel terminal comprises the following steps:
transmitting a field monitoring video 401 through video monitoring equipment, and then processing the acquired video image data 402;
obtaining a multi-mode feature coding frame 403, performing feature engineering processing 404, and extracting features therein;
constructing ghost data 405 for the feature designs, and carrying out abnormal behavior reasoning on the data by combining a machine learning reinforcement learning model 406;
judging whether the behavior in the video image data belongs to abnormal behavior 407;
classifying 408 the resulting abnormal behavior;
transmitting 409 these abnormal behavior information to the control panel terminal;
the pre-warning information 410 is issued by the control panel terminal.
As shown in fig. 5, in the abnormal behavior prediction apparatus of this embodiment, the video monitoring device obtains an image through the obtaining unit 501 and gives the image to the calculating unit for analysis, after the calculating unit 602 receives a signal, the controlling unit 603 performs design construction and control reasoning process on the image operation, and after the calculating unit finishes working, the result is transmitted to the output unit 504.
Example 5
The embodiment provides a personnel abnormal behavior early warning computer device based on video data machine learning, which comprises a memory and a processor, wherein the memory is used for storing a personnel abnormal behavior early warning method, and the processor is used for running the method stored in the memory to predict whether a person in a monitor has abnormal behaviors.
As shown in fig. 6, the processor 601 analyzes the image and transmits effective information, the memory 602 is responsible for storing data such as the image, the communication interface 603 refers to an interface between the central processing unit and the standard communication subsystem and the bus 604, and the display screen 605 displays images.
The invention provides a method, a device, computer equipment and a medium for personnel abnormal behavior early warning based on monitoring video images in an intelligent park, provides a highly integrated information system function with an open architecture, meets the early warning requirements of different levels by flexibly configuring monitoring parameters, can positively promote the intelligent of the fields of intelligent security, abnormal behavior prediction and the like for the human behavior recognition technology in the video images, and can acquire human behavior expression characteristics by using tensor neural networks to recognize human behaviors in the prior art. The human body behavior expression characteristics are designed and constructed by utilizing the image visual information, human body behavior recognition is carried out, and the tensor neural network is combined with the manual construction characteristics to carry out human body behavior recognition and the like. The human body characteristic recognition algorithm is adopted to detect, analyze, recognize and pre-warn the pedestrian target characteristic, meanwhile, the machine vision image sensing technology is adopted, human body movement and facial characteristics are collected in the machine vision image scene, and the automatic recognition of the human body target and the pre-warn information prompt are completed through the machine learning algorithm.
Those skilled in the art will appreciate that implementing all or part of the methods or embodiments of the present invention, the early warning process may be performed by hardware associated with computer program instructions, where the program is stored on a computer readable storage medium, and when the program is executed, the method described above may be performed. The storage medium may be an optical disc, a usb disk, a Read-Only Memory (ROM), a random access Memory (Random Access Memory, RAM), or the like.
Although embodiments of the present invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made therein without departing from the principles and spirit of the invention, the scope of which is defined in the appended claims and their equivalents.
Claims (9)
1. The personnel abnormal behavior early warning method based on video data machine learning is characterized by comprising the following steps of:
s1, acquiring video image data, and cleaning the video image data;
s2, preprocessing video image data, namely removing privacy-related information in the data, and improving video definition of a blurred video by adopting a frame quality enhancement method;
s3, recognizing human body movement modes, setting a classification label for each type, and performing multi-mode feature coding;
s4, performing feature engineering processing on the multi-mode codes, namely removing redundant features of the data center and selecting features through information gain;
s5, constructing a tensor neural network to track people appearing in the video, detecting gait based on a gait energy diagram according to the tracked task outline, detecting skin exposure of the task in the video, taking the exposure as an appearance clothing index of the person, calculating correlation of images between frames of the video according to the indexes, judging abnormal behaviors of the person according to the result of the correlation calculation, calculating correlation between two images through Euclidean distance, and if the person is normal but each item of data is intense in transformation, listing the person as an abnormal person, wherein each item of data comprises behavior type, sinking behavior, abnormal rise, associated information of a high-risk person in a management record, high-risk person, incident record of the high-risk person, appearance clothing, body state, gait, pace, facial expression, mental state and mark, wherein the behavior type data type is discrete, the critical semantic value range is { passive, abnormally high, normal }, the sinking behavior data type is discrete, the critical semantic value range is { sad depression, solitary, helplessness, hopeless }, the abnormally high data type is discrete, the critical semantic value range is { full of passion, fight is raised }, the high risk individual data type in the management record is boolean, the critical semantic value range is yes or no, the critical semantic of the high risk individual is a near three-day occurrence time record, the high risk individual is recorded as a text of no more than 100 words, the appearance clothing data type is discrete, the critical semantic value range is whether the body is more than 95% unwashed, the body state data type is discrete, the critical semantic value range is inclined by more than 10 degrees, the gait data type is discrete, the key semantic value range is whether the user walks vertically normally, the pace data type is of a digital type, the key semantic value range is normal, extremely fast and extremely slow, the facial expression data type is of a discrete type, the key semantic value range is { excited, liked, surprised, painful, fear, humidify, disgust, anger }, the mental state data type is of a discrete type, the key semantic is whether the user is depressed for a long time, the mark data type is of a discrete type, the key semantic value range is { misjudgment, missed judgment and correct };
tracking people appearing in the video includes: taking the multi-mode coded image as input of a tensor neural network; the tensor neural network comprises human body gesture recognition, pedestrian detection and separation of foreground and background; acquiring a human body gesture key sequence point in an image through human body gesture recognition; detecting a person through pedestrian detection, dividing a task from an image, and acquiring a gesture outline sequence of the person by combining separation of a foreground and a background; tracking and identifying the person through the acquired gesture key sequence points and gesture outline sequences;
s6, constructing early warning decision support, namely if the dynamic information of the similarity and the difference between the acquired frame images shows that the behavior of the photographed person is abnormal, informing relevant persons to carry out the treatment.
2. The method for early warning of abnormal behaviors of personnel based on machine learning of video data according to claim 1, wherein after the feature engineering processing is performed in the step S3, the data is detected, and if the non-random missing data exists and the data is made, the statistical machine learning method is adopted to infer the data, so that the complete data of the data is obtained.
3. The method for early warning of abnormal behaviors of personnel based on machine learning of video data according to claim 1, wherein the multi-modal feature coding specifically comprises the following processes: using a 5-dimensional tensor structure for the acquired image, encoding video data by (samples, frames, height, width, channels) to represent 5-dimensional tensor data of samples, frames, height, width, and channel of the video data, respectively; the acquired video image is encoded with 4-dimensional tensor structure by (samples, height, width, channels) video data, and 4-dimensional tensor data representing samples, height, width, and channel of the video data, respectively.
4. The method for early warning of abnormal behaviors of personnel based on machine learning of video data according to claim 1, wherein the process of detecting the skin exposure of a task in video takes the exposure as an appearance clothing index of a person, namely, the skin area of the person in an image is obtained based on an elliptic skin model, and the proportion of the exposed skin area to the outline of the person is taken as the appearance clothing index.
5. The method for early warning of abnormal behaviors of personnel based on machine learning of video data according to claim 1, wherein a sliding window with a length of N frames and a step length of t is set after gait detection and appearance clothes indexes are carried out, the gait detection and appearance clothes indexes are valued through the sliding window, and the average value of the data of each window is taken as the data of correlation calculation.
6. The method for early warning of abnormal behavior of a person based on machine learning of video data according to claim 4 or 5, wherein the correlation between two images is calculated by euclidean distance when the difference judgment is made.
7. The method for early warning of abnormal behavior of a person based on machine learning of video data according to claim 1, wherein when performing abnormal behavior judgment on the person, if the similarity of two images extracted in S frames is lower than a certain threshold, the task is judged to be abnormal.
8. The system is used for executing any one of the abnormal personnel behavior early warning methods based on video data machine learning according to the claims 1-7, and is characterized by comprising a front terminal system, a transmission network and a monitoring center, wherein the front terminal system comprises various monitoring devices, a front end subsystem transmits information collected by the monitoring devices to the monitoring center through the transmission network, the monitoring center comprises an abnormal behavior prediction device, the device predicts whether a shot personnel is abnormal or not through the collected information, and if the shot personnel is abnormal, the monitoring center gives an alarm and projects the monitoring devices capable of shooting the personnel to a monitor center display for real-time monitoring.
9. The personnel abnormal behavior early warning computer equipment based on video data machine learning is characterized by comprising a memory and a processor, wherein the memory is used for storing any one of the personnel abnormal behavior early warning methods described in claims 1-7, and the processor is used for running the method stored in the memory to predict whether the personnel in the monitor has abnormal behaviors.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111211548.7A CN113850229B (en) | 2021-10-18 | 2021-10-18 | Personnel abnormal behavior early warning method and system based on video data machine learning and computer equipment |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111211548.7A CN113850229B (en) | 2021-10-18 | 2021-10-18 | Personnel abnormal behavior early warning method and system based on video data machine learning and computer equipment |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN113850229A CN113850229A (en) | 2021-12-28 |
| CN113850229B true CN113850229B (en) | 2023-12-22 |
Family
ID=78978708
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202111211548.7A Active CN113850229B (en) | 2021-10-18 | 2021-10-18 | Personnel abnormal behavior early warning method and system based on video data machine learning and computer equipment |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113850229B (en) |
Families Citing this family (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114218992B (en) * | 2021-12-29 | 2023-09-08 | 重庆紫光华山智安科技有限公司 | Abnormal object detection method and related device |
| CN115883779B (en) * | 2022-10-13 | 2023-08-04 | 湖北公众信息产业有限责任公司 | Intelligent park information safety management system based on big data |
| CN115410324A (en) * | 2022-10-28 | 2022-11-29 | 山东世拓房车集团有限公司 | Car as a house night security system and method based on artificial intelligence |
| CN116935286B (en) * | 2023-08-03 | 2024-01-09 | 广州城市职业学院 | Short video identification system |
Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103761516A (en) * | 2014-02-14 | 2014-04-30 | 重庆科技学院 | ATM abnormal human face detection method based on video monitoring |
| CN106919921A (en) * | 2017-03-06 | 2017-07-04 | 重庆邮电大学 | With reference to sub-space learning and the gait recognition method and system of tensor neutral net |
| CN109255312A (en) * | 2018-08-30 | 2019-01-22 | 罗普特（厦门）科技集团有限公司 | A kind of abnormal dressing detection method and device based on appearance features |
| CN109819208A (en) * | 2019-01-02 | 2019-05-28 | 江苏警官学院 | A kind of dense population security monitoring management method based on artificial intelligence dynamic monitoring |
| CN110443109A (en) * | 2019-06-11 | 2019-11-12 | 万翼科技有限公司 | Abnormal behaviour monitor processing method, device, computer equipment and storage medium |
| CN111353343A (en) * | 2018-12-21 | 2020-06-30 | 国家电网有限公司客户服务中心 | Business hall service standard quality inspection method based on video monitoring |
| CN113239759A (en) * | 2021-04-29 | 2021-08-10 | 国网江苏省电力有限公司苏州供电分公司 | Power grid operation field violation identification method |
-
2021
- 2021-10-18 CN CN202111211548.7A patent/CN113850229B/en active Active
Patent Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103761516A (en) * | 2014-02-14 | 2014-04-30 | 重庆科技学院 | ATM abnormal human face detection method based on video monitoring |
| CN106919921A (en) * | 2017-03-06 | 2017-07-04 | 重庆邮电大学 | With reference to sub-space learning and the gait recognition method and system of tensor neutral net |
| CN109255312A (en) * | 2018-08-30 | 2019-01-22 | 罗普特（厦门）科技集团有限公司 | A kind of abnormal dressing detection method and device based on appearance features |
| CN111353343A (en) * | 2018-12-21 | 2020-06-30 | 国家电网有限公司客户服务中心 | Business hall service standard quality inspection method based on video monitoring |
| CN109819208A (en) * | 2019-01-02 | 2019-05-28 | 江苏警官学院 | A kind of dense population security monitoring management method based on artificial intelligence dynamic monitoring |
| CN110443109A (en) * | 2019-06-11 | 2019-11-12 | 万翼科技有限公司 | Abnormal behaviour monitor processing method, device, computer equipment and storage medium |
| CN113239759A (en) * | 2021-04-29 | 2021-08-10 | 国网江苏省电力有限公司苏州供电分公司 | Power grid operation field violation identification method |
Non-Patent Citations (1)
| Title |
|---|
| "Anomaly Detection with Tensor Networks";JinhuiWang等;《arXiv》;第1-12页 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN113850229A (en) | 2021-12-28 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN113850229B (en) | Personnel abnormal behavior early warning method and system based on video data machine learning and computer equipment | |
| Elharrouss et al. | A combined multiple action recognition and summarization for surveillance video sequences | |
| CN110674772B (en) | Intelligent safety control auxiliary system and method for electric power operation site | |
| CN110458101B (en) | Criminal personnel sign monitoring method and equipment based on combination of video and equipment | |
| Chackravarthy et al. | Intelligent crime anomaly detection in smart cities using deep learning | |
| Chen et al. | Towards automatic analysis of social interaction patterns in a nursing home environment from video | |
| Caruccio et al. | EDCAR: A knowledge representation framework to enhance automatic video surveillance | |
| CN112364696B (en) | Method and system for improving family safety by utilizing family monitoring video | |
| CN106355154B (en) | Method for detecting frequent passing of people in surveillance video | |
| Chang et al. | A hybrid CNN and LSTM-based deep learning model for abnormal behavior detection | |
| US11935297B2 (en) | Item monitoring for doorbell cameras | |
| CN103106394A (en) | Human body action recognition method in video surveillance | |
| Sun et al. | Real-time elderly monitoring for senior safety by lightweight human action recognition | |
| Kamoona et al. | Sparsity-based naive bayes approach for anomaly detection in real surveillance videos | |
| Girdhar et al. | Incept_LSTM: Accession for human activity concession in automatic surveillance | |
| CN111191498A (en) | Behavior recognition method and related product | |
| CN116862740A (en) | Intelligent prison management and control system based on Internet | |
| Abhishek et al. | Human Verification over Activity Analysis via Deep Data Mining | |
| CN114429677A (en) | Coal mine scene operation behavior safety identification and assessment method and system | |
| CN113486754A (en) | Event evolution prediction method and system based on video | |
| Horowitz et al. | Interruptions detection in video conferences | |
| Nithesh et al. | Anomaly Detection in Surveillance Videos Using Deep Learning | |
| Ma et al. | Research on Person Re-identification Method Based on Data Link Technology | |
| Kayani | Real-time Violence Detection using Deep Learning Techniques | |
| US20230360402A1 (en) | Video-based public safety incident prediction system and method therefor |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| TA01 | Transfer of patent application right |
Effective date of registration: 20231128 Address after: 518000 No. 1, west side of 5 / F, building e, tefa information port building, No. 2 Kefeng Road, Nanshan District, Shenzhen, Guangdong Applicant after: Shenzhen Chengyu Information Technology Co.,Ltd. Address before: Room 801, 85 Kefeng Road, Huangpu District, Guangzhou City, Guangdong Province Applicant before: Guangzhou Dayu Chuangfu Technology Co.,Ltd. Effective date of registration: 20231128 Address after: Room 801, 85 Kefeng Road, Huangpu District, Guangzhou City, Guangdong Province Applicant after: Guangzhou Dayu Chuangfu Technology Co.,Ltd. Address before: 400065 Chongwen Road, Nanshan Street, Nanan District, Chongqing Applicant before: CHONGQING University OF POSTS AND TELECOMMUNICATIONS |
|
| TA01 | Transfer of patent application right | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |