CN117133050A

CN117133050A - Method, device, equipment and storage medium for detecting abnormal behavior of car

Info

Publication number: CN117133050A
Application number: CN202311097588.2A
Authority: CN
Inventors: 曹晓磊; 邓伟
Original assignee: Shenzhen Inovance Technology Co Ltd
Current assignee: Shenzhen Inovance Technology Co Ltd
Priority date: 2023-08-28
Filing date: 2023-08-28
Publication date: 2023-11-28

Abstract

The invention discloses a method, a device, equipment and a storage medium for detecting abnormal behavior of a car, which relate to the technical field of elevator safety, and the method comprises the following steps: acquiring a video frame image of an elevator car; performing target detection and skeleton recognition on the video frame image by using a first recognition model to obtain a first behavior feature and a human skeleton feature; performing target detection and semantic segmentation on the video frame image by using a second recognition model to obtain a second behavior feature and a human edge feature; performing matching processing based on the first behavior feature and the second behavior feature to obtain an abnormal behavior recognition result; performing control analysis based on human skeleton characteristics and human edge characteristics to obtain an abnormal behavior prediction result; and obtaining a final detection result according to the abnormal behavior recognition result and the abnormal behavior prediction result. The method solves the problem of lower detection accuracy of the abnormal behavior in the related technology, and achieves the effect of improving the detection accuracy of the abnormal behavior and the generalization capability.

Description

Method, device, equipment and storage medium for detecting abnormal behavior of car

Technical Field

The invention relates to the technical field of elevator safety, in particular to a method, a device, equipment and a storage medium for detecting abnormal behaviors of a car.

Background

At present, in the detection of abnormal car behaviors, a vision technology is the most commonly used one, image information in the car is collected through a camera, and abnormal target behaviors are identified and analyzed by utilizing an image processing technology and a computer vision technology, so that the detection and alarm of the abnormal car behaviors are realized.

In the related art, the abnormal car behavior detection based on the visual technology is the target behavior identification based on the information of one dimension, and has great limitation, for example, when the door-pulling behavior is detected through the framework information, the door-pulling behavior and the head-holding behavior are difficult to distinguish, and the head-holding behavior is identified as the abnormal behavior due to the fact that the two behaviors are similar in whole but different in part and often have the false detection phenomenon.

Disclosure of Invention

The main purpose of the invention is that: the method, the device, the equipment and the storage medium for detecting the abnormal behavior of the car are provided, and the technical problem that the accuracy rate of detecting the abnormal behavior is low in the related technology is solved.

In order to achieve the above purpose, the invention adopts the following technical scheme:

in a first aspect, the present invention provides a method for detecting abnormal behavior of a car, the method comprising:

Acquiring a video frame image of an elevator car;

performing target detection and skeleton recognition on the video frame image by using a first recognition model to obtain a first behavior feature and a human skeleton feature;

performing target detection and semantic segmentation on the video frame image by using a second recognition model to obtain a second behavior feature and a human edge feature;

performing matching processing based on the first behavior feature and the second behavior feature to obtain an abnormal behavior recognition result;

performing control analysis based on human skeleton characteristics and human edge characteristics to obtain an abnormal behavior prediction result;

and obtaining a final detection result according to the abnormal behavior recognition result and the abnormal behavior prediction result.

Optionally, in the method for detecting abnormal behavior of a car, the step of performing matching processing based on the first behavior feature and the second behavior feature to obtain an abnormal behavior recognition result includes:

matching the first behavior feature with the second behavior feature;

when the matching is consistent, a first suspected abnormal behavior is obtained;

when the matching is inconsistent, determining suspected abnormal behaviors according to a preset first priority, and obtaining second suspected abnormal behaviors;

and obtaining an abnormal behavior identification result according to the first suspected abnormal behavior or the second suspected abnormal behavior.

Optionally, in the car abnormal behavior detection method, the step of obtaining an abnormal behavior prediction result based on a human skeleton feature and a human edge feature by performing a comparison analysis includes:

comparing the human skeleton characteristics with the human edge characteristics to obtain a human identification result;

and carrying out behavior state change recognition on human body recognition results corresponding to the plurality of video frame images to obtain abnormal behavior prediction results.

Optionally, in the car abnormal behavior detection method, the step of comparing the human skeleton feature and the human edge feature to obtain the human identification result includes:

matching human skeleton features and human edge features to obtain feature pairs with association relations;

when the number of the feature pairs exceeds the preset number, judging whether the prediction frames corresponding to the feature pairs are overlapped or not; the prediction frames comprise a first prediction frame corresponding to human skeleton characteristics and a second prediction frame corresponding to human edge characteristics, wherein the first prediction frame is generated when the target detection is carried out on the video frame image based on a first recognition model, and the second prediction frame is generated when the target detection is carried out on the video frame image based on a second recognition model;

If no overlap exists, determining a human body identification result according to the feature pairs;

if the overlap exists, carrying out de-duplication processing on the prediction frame to obtain a feature pair after de-duplication, and determining a human body recognition result according to the feature pair after de-duplication.

Optionally, in the car abnormal behavior detection method, the step of performing behavior state change recognition for the human body recognition results corresponding to the plurality of video frame images to obtain an abnormal behavior prediction result includes:

constructing human body feature vectors aiming at human body recognition results corresponding to a plurality of video frame images, wherein the human body feature vectors are multidimensional time domain feature vectors;

and inputting the human body characteristic vector into a human body behavior prediction model established by using a deep learning algorithm to obtain an abnormal behavior prediction result.

Optionally, in the method for detecting abnormal car behavior, the step of obtaining a final detection result according to the abnormal behavior recognition result and the abnormal behavior prediction result includes:

constructing an abnormal behavior feature vector aiming at abnormal behavior recognition results corresponding to a plurality of video frame images, wherein the abnormal behavior feature vector is a multidimensional time domain feature vector;

matching the abnormal behavior prediction result with the abnormal behavior feature vector;

When the matching is consistent, a first feature vector correction result is obtained;

when the matching is inconsistent, determining abnormal behaviors according to a preset second priority, and obtaining a second feature vector correction result;

and obtaining a final detection result according to the first feature vector correction result or the second feature vector correction result.

Optionally, in the method for detecting abnormal car behavior, the step of obtaining a final detection result according to the first feature vector correction result or the second feature vector correction result includes:

obtaining a corrected abnormal behavior feature vector according to the first feature vector correction result or the second feature vector correction result;

carrying out statistical analysis on the corrected abnormal behavior feature vector to obtain a statistical result of the abnormal behavior;

and obtaining a final detection result according to the statistical result and a preset judgment condition.

In a second aspect, the present invention provides a car abnormal behavior detection apparatus, comprising:

the image acquisition module is used for acquiring video frame images of the elevator car;

the first recognition module is used for carrying out target detection and skeleton recognition on the video frame image by utilizing the first recognition model to obtain a first behavior feature and a human skeleton feature;

The second recognition module is used for carrying out target detection and semantic segmentation on the video frame image by utilizing the second recognition model to obtain a second behavior characteristic and a human edge characteristic;

the first result module is used for carrying out matching processing based on the first behavior characteristic and the second behavior characteristic to obtain an abnormal behavior identification result;

the second result module is used for carrying out comparison analysis based on the human skeleton characteristics and the human edge characteristics to obtain an abnormal behavior prediction result;

and the result output module is used for obtaining a final detection result according to the abnormal behavior identification result and the abnormal behavior prediction result.

In a third aspect, the present invention provides a car abnormal behavior detection apparatus, the apparatus comprising a processor and a memory, the memory storing a car abnormal behavior detection program, the car abnormal behavior detection program, when executed by the processor, implementing the car abnormal behavior detection method as described above.

In a fourth aspect, the present invention provides a computer-readable storage medium having stored thereon a computer program which, when executed by one or more processors, implements a car abnormal behavior detection method as described above.

The one or more technical schemes provided by the invention can have the following advantages or at least realize the following technical effects:

According to the car abnormal behavior detection method, device, equipment and storage medium, the first recognition model is utilized to conduct target detection and skeleton recognition on the video frame image of the elevator car to obtain first behavior features and human skeleton features, the second recognition model is utilized to conduct target detection and semantic segmentation on the video frame image to obtain second behavior features and human edge features, skeleton information and edge information are respectively extracted, and feature information extraction of different dimensions is achieved; then, based on the first behavior feature and the second behavior feature, carrying out matching processing to obtain an abnormal behavior recognition result, firstly mining abnormal behaviors in the space environment in the car to realize feature extraction in the space dimension, and then matching the behavior features detected by different recognition models to unify the behavior features into a set of abnormal behavior recognition results to realize abnormal behavior recognition in the space dimension; meanwhile, based on human skeleton characteristics and human edge characteristics, comparison analysis is carried out to obtain an abnormal behavior prediction result, information extraction of different dimensions is carried out on targets in the car, comparison error correction is carried out, so that behavior prediction is carried out based on the determined targets, a set of abnormal behavior prediction result is obtained, and more accurate abnormal behavior prediction is realized; and finally, obtaining a final detection result according to the abnormal behavior recognition result and the abnormal behavior prediction result, and realizing the abnormal behavior detection by combining a plurality of dimensional information in the visual system, wherein the dimensional information is mutually corrected, so that the effect of improving the abnormal behavior detection accuracy and generalization capability is achieved.

Drawings

In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for the description of the embodiments will be briefly described below, and it is obvious that the drawings in the following description are only some embodiments of the present invention, and other drawings may be obtained according to the provided drawings without inventive effort for a person skilled in the art.

Fig. 1 is a flowchart of a first embodiment of a method for detecting abnormal behavior of a car according to the present invention;

fig. 2 is a schematic hardware configuration diagram of a car abnormal behavior detection apparatus according to the present invention;

fig. 3 is a flowchart of a second embodiment of the method for detecting abnormal behavior of a car according to the present invention;

fig. 4 is a flowchart of a third embodiment of the method for detecting abnormal behavior of a car according to the present invention;

fig. 5 is a schematic functional block diagram of a first embodiment of the abnormal car behavior detection device according to the present invention.

The achievement of the objects, functional features and advantages of the present invention will be further described with reference to the accompanying drawings, in conjunction with the embodiments.

Detailed Description

For the purpose of making the objects, technical solutions and advantages of the present invention more apparent, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention, and it is apparent that the described embodiments are only some embodiments of the present invention, but not all embodiments of the present invention. All other embodiments, which can be made by those skilled in the art based on the embodiments of the invention without making any inventive effort, are intended to be within the scope of the invention.

It should be noted that, in the present disclosure, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or system that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or system. Without further limitation, an element defined by the phrase "comprising … …" does not exclude the presence of other like elements in a process, method, article, or system that comprises the element. In addition, the meaning of "and/or" as it appears throughout includes three parallel schemes, for example "A and/or B", including the A scheme, or the B scheme, or the scheme where A and B are satisfied simultaneously. In the present invention, if there is a description referring to "first", "second", etc., the description of "first", "second", etc. is for descriptive purposes only and is not to be construed as indicating or implying a relative importance or implicitly indicating the number of technical features indicated. Thus, a feature defining "a first" or "a second" may explicitly or implicitly include at least one such feature. In the present invention, suffixes such as "module", "part" or "unit" used for representing elements are used only for facilitating the description of the present invention, and have no specific meaning per se. Thus, "module," "component," or "unit" may be used in combination. The specific meaning of the above terms in the present invention can be understood by those of ordinary skill in the art according to the specific circumstances. In addition, the technical solutions of the embodiments may be combined with each other, but it is based on the fact that those skilled in the art can implement the combination of the technical solutions, when the technical solutions contradict each other or cannot be implemented, the combination of the technical solutions should be considered as not existing and not falling within the protection scope of the present invention.

At present, the car abnormal behavior detection technology mainly adopts a visual camera and a radar sensor to collect data, and adopts a visual technology and a data processing algorithm to identify target behaviors. The visual technology is the most commonly used technology, the image information in the elevator car is collected through the camera, and the abnormal target behavior is identified and analyzed by utilizing the image processing technology and the computer visual technology, so that the detection and alarm of the abnormal behavior of the elevator car are realized.

Analysis of related technologies finds that, at present, a method for detecting abnormal car behaviors based on visual technology generally performs target behavior identification based on information of one dimension, and has great limitation, for example, when door-opening behaviors are detected through skeleton information, it is difficult to distinguish two behaviors which are similar in whole and different in part, so that the door-opening behaviors are identified as abnormal behaviors, and false detection phenomenon often occurs.

In view of the technical problem of low accuracy of abnormal behavior detection in the related art, the invention provides a method for detecting abnormal behavior of a car, which has the following overall thought:

acquiring a video frame image of an elevator car; performing target detection and skeleton recognition on the video frame image by using a first recognition model to obtain a first behavior feature and a human skeleton feature; performing target detection and semantic segmentation on the video frame image by using a second recognition model to obtain a second behavior feature and a human edge feature; based on the first behavior feature and the second behavior feature, obtaining an abnormal behavior recognition result; obtaining an abnormal behavior prediction result based on the human skeleton characteristics and the human edge characteristics; and obtaining a final detection result according to the abnormal behavior recognition result and the abnormal behavior prediction result.

Through the technical scheme, the skeleton information and the edge information are respectively extracted, so that the characteristic information extraction of different dimensions is realized; firstly, mining abnormal behaviors in a space environment in a car to realize feature extraction in a space dimension, then matching behavior features detected by different recognition models, unifying the behavior features into a set of abnormal behavior recognition results, and recognizing the abnormal behaviors in the space dimension; meanwhile, information extraction of different dimensions is carried out on targets in the lift car, then comparison and error correction are carried out, behavior prediction is carried out on the basis of the determined targets, a set of abnormal behavior prediction results are obtained, more accurate abnormal behavior prediction is carried out, and therefore a final detection result is obtained according to the abnormal behavior recognition result and the abnormal behavior prediction results, abnormal behavior detection is carried out by combining a plurality of dimensional information in a vision system, and the effect of improving the abnormal behavior detection accuracy and generalization capability is achieved by carrying out mutual error correction on each dimensional information.

The method, the device, the equipment and the storage medium for detecting abnormal car behaviors provided by the invention are described in detail below by specific examples and embodiments with reference to the accompanying drawings.

Example 1

Referring to fig. 1, a first embodiment of a car abnormal behavior detection method of the present invention is presented, which is applied to a car abnormal behavior detection apparatus.

The car abnormal behavior detection device refers to terminal equipment or network equipment capable of realizing network connection, and can be terminal equipment such as a mobile phone, a computer, a tablet personal computer, a portable computer, an embedded industrial personal computer and the like, or network equipment such as a server, a cloud platform and the like.

As shown in fig. 2, a schematic hardware configuration of the car abnormal behavior detection device is shown. The car abnormal behavior detection apparatus may include: a processor 1001, such as a CPU (Central Processing Unit ), a communication bus 1002, a user interface 1003, a network interface 1004, a memory 1005.

In particular, communication bus 1002 is configured to enable connective communication between these components; the user interface 1003 is used for connecting the client and communicating data with the client, and the user interface 1003 may include an output unit and an input unit; the network interface 1004 is used to connect to and communicate data with a background server, and the network interface 1004 may include an input/output interface; the memory 1005 is used for storing various types of data, which may include, for example, instructions of any application program or method in the car abnormal behavior detection apparatus, and application program-related data, and the memory 1005 may be a built-in memory; optionally, the memory 1005 may also be a storage device independent of the processor 1001, and with continued reference to fig. 2, the memory 1005 may include an operating system, a network communication module, a user interface module, and a car abnormal behavior detection program; the processor 1001 is configured to call a car abnormal behavior detection program stored in the memory 1005, and perform the following operations:

Acquiring a video frame image of an elevator car;

Based on the above-described car abnormal behavior detection apparatus, the following describes the car abnormal behavior detection method of the present embodiment in detail with reference to a flowchart shown in fig. 1. The method may comprise the steps of:

step S100: and acquiring video frame images of the elevator car.

Specifically, the car abnormal behavior detection device can perform video shooting on the elevator car through the camera device, and after shooting video is obtained, video frame images of the shooting video are continuously obtained. The video frame image may be an image corresponding to a video frame with a set frame number interval, or may be an image corresponding to a sampling frame obtained after sampling at a set sampling frequency.

In specific application, the video shooting of the elevator car can be carried out on the space environment in the car, and the video shooting comprises the shooting of objects such as people, animals, electric vehicles and the like in the car, and the corresponding objects can be included in video frame images obtained after video shooting. In this embodiment, the abnormal behavior detection is mainly performed on the person on the video frame image, so after the video frame image including a plurality of objects is acquired, the person on the video frame image can be identified first, the person is taken as the target, and then the abnormal behavior detection is performed on the target.

Step S200: and performing target detection and skeleton recognition on the video frame image by using the first recognition model to obtain a first behavior feature and a human skeleton feature.

Specifically, the first behavior features refer to some suspected abnormal behaviors performed by the target in the video frame image, and the target detection is performed only once, so that the recognized result cannot be determined to be the abnormal behavior, so the first behavior features are actually feature information of the suspected abnormal behaviors, wherein the abnormal behaviors refer to behaviors in which abnormal behaviors (such as falling, jumping and the like) or behaviors in which the behaviors can affect the safety of the elevator (such as door-moving and the like). The human skeleton feature refers to skeleton information of a human body, which is a target identified in a video frame image, and the skeleton information refers to positions of key points on the human skeleton, for example, 17 key points such as hip center, left and right hip, left and right knee, left and right ankle, chest, neck, chin, head, left and right shoulder, left and right elbow, left and right wrist, etc., and in practical application, the key points can be set as required.

After the car abnormal behavior detection equipment acquires the video frame image, the video frame image is processed by using a target recognition model, namely a first recognition model, and specifically target detection and skeleton recognition are carried out to obtain a first behavior feature and a human skeleton feature. The target detection herein refers to detecting whether a suspected abnormal behavior exists in the video frame image, and the skeleton recognition refers to recognizing skeleton information of all people in the video frame image. It can be understood that the algorithm used in the first recognition model may be two independent algorithms, that is, performing object detection processing and skeleton recognition processing on the video frame image, or two algorithms having association relationship, for example, performing skeleton recognition on the video frame image to obtain skeleton characteristics of a human body, and then performing object detection of suspected abnormal behaviors by using the skeleton characteristics of the human body to obtain first behavior characteristics. That is, the object detection and skeleton recognition of the video frame image using the first recognition model may be performed separately and simultaneously, or the skeleton recognition may be performed first, and then the object detection may be performed.

In the implementation process, the first recognition model can adopt an algorithm model with the functions of target detection and skeleton recognition, for example, a YOLO-else algorithm model, and the YOLO-else algorithm model can combine human body detection and key point estimation to realize human body gesture recognition. In practical application, the first recognition model can be formed by combining the target detection framework YOLO with other target detection algorithms so as to realize human body gesture recognition. The YOLO-else algorithm model is an end-to-end joint detection and multi-person gesture estimation framework based on YOLOv5, and can realize the leading multi-person gesture recognition performance under the condition of not adopting data enhancement. In specific application, the model parameter configuration can be carried out according to the needs, for example, the target detection part is modified to meet the multi-classification function.

Step S300: and performing target detection and semantic segmentation on the video frame image by using the second recognition model to obtain a second behavior characteristic and a human edge characteristic.

In the step S200, the behavior feature extraction and the human skeleton information extraction are performed on the video frame image, but only one abnormal behavior recognition is performed, the obtained result is often inaccurate, and the information extraction is performed only in one dimension, so that the actual situation cannot be truly reflected, and the false detection phenomenon is easy to occur, especially some similar abnormal behaviors, such as head holding and door pulling, are more prone to error. Therefore, in order to reflect the actual situation more accurately, the actual behavior of the human body in the video frame image is identified correctly, the situation of false detection or missing detection of abnormal behavior is avoided, and the behavior feature extraction can be carried out synchronously by using another identification model, so that the error correction of the abnormal behavior result is carried out based on the two behavior features. However, in order to better verify the human body target, the human body contour information extraction is performed without performing the identical human body skeleton feature extraction, and the human body contour and the human body skeleton can be used as reference bases for performing behavior recognition on the target human body, but the human body contour and the human body skeleton belong to different dimensions, so that when the first recognition model performs the human body skeleton information extraction, the second recognition model can perform the human body contour information extraction. Therefore, the first recognition model is utilized to perform target detection and skeleton recognition on the video frame image, and the second recognition model is utilized to perform target detection and semantic segmentation on the video frame image, so that feature information extraction of different dimensions is realized, error correction of related information is performed later, and accuracy of a subsequent result is ensured.

Specifically, the second behavior feature refers to some suspected abnormal behaviors performed by the target in the video frame image, and the target detection is performed only once, so that the identified result cannot be determined to be the abnormal behavior, so that the second behavior feature is actually feature information of the suspected abnormal behaviors, where the abnormal behaviors refer to behaviors in which the behavior abnormality (such as a fall, a jump, etc.) or behaviors in which the behavior may affect the safety of the elevator (such as a door-moving, etc.). The human body edge feature refers to contour information of a human body, which is a target identified in a video frame image, and the contour information refers to the position of a human body morphological edge contour.

The car abnormal behavior detection device may process the acquired video frame image by using another object recognition model, that is, the second recognition model, while executing the step S200 or after the step S200, specifically perform object detection and semantic segmentation, to obtain the second behavior feature and the human edge feature. The target detection herein refers to detecting whether a suspected abnormal behavior exists in the video frame image, and the semantic segmentation refers to identifying contour information, also called edge information, of all people in the video frame image. It can be understood that the algorithm used in the second recognition model may be two independent algorithms, that is, performing object detection processing and semantic segmentation processing on the video frame image, or two algorithms having association relationship, for example, performing semantic segmentation on the video frame image to obtain a human body edge feature, and then performing object detection of suspected abnormal behavior by using the human body edge feature to obtain a second behavior feature. That is, the object detection and the semantic segmentation of the video frame image using the second recognition model may be performed separately and simultaneously, or the semantic segmentation may be performed first, and then the object detection may be performed.

In the implementation process, the second recognition model can adopt an algorithm model with target detection and semantic segmentation functions, for example, an instance segmentation algorithm (Mask RCNN) model, wherein the Mask RCNN model is further refined on the basis of semantic detection, the foreground and the background of an object are separated, the object separation at the pixel level is realized, and objects of different instances in the same class can be segmented by the instance segmentation; the Mask RCNN model adopts a ResNet-FPN architecture to extract features, and a Mask prediction branch is added to predict a binary Mask, so that not only can targets in an image be detected, but also a high-quality segmentation result can be given to each target, and meanwhile, other tasks such as key point detection can be expanded, wherein targets, namely human bodies, can be detected from video frame images, and human body contour detection is carried out, so that human body behavior features and human body contour information are reflected specifically. The Mask RCNN model has the advantages of strong network feature extraction capability, excellent target detection effect, fine example segmentation effect and the like, and model parameter configuration can be carried out according to requirements in specific application, for example, a target detection part is modified to meet the multi-classification function.

The video frame image is subjected to target detection and identification by using two different identification models, human skeleton information and human edge information are respectively extracted, and feature information extraction of different dimensions is realized, so that the purpose of detecting abnormal behaviors by combining multiple dimensional information is realized.

Step S400: and carrying out matching processing based on the first behavior characteristic and the second behavior characteristic to obtain an abnormal behavior recognition result.

Specifically, after the car abnormal behavior detection device obtains the first behavior feature, the human skeleton feature, the second behavior feature and the human edge feature, the first behavior feature and the second behavior feature can be subjected to matching, de-duplication, screening, error correction and other processing, so that an integrated group of suspected abnormal behaviors is obtained and used as an abnormal behavior recognition result. The specific operation of processing the two groups of suspected abnormal behavior features can be realized through or and equal logic relations, priority setting can be performed on different abnormal behavior types, and further processing is performed by combining the priority relations on the basis of logic processing, so that an abnormal behavior recognition result which is more fit with reality, namely more accurate, is obtained.

The two groups of suspected abnormal behaviors output by the two different recognition models are utilized to integrate a more accurate group of suspected abnormal behaviors, firstly, the abnormal behaviors in the space environment in the car are mined, feature extraction in the space dimension is realized, then, the behavior features detected by the different recognition models are matched to form a set of abnormal behavior recognition results, the abnormal behavior recognition in the space dimension is realized, and the accuracy of the abnormal behavior recognition results is ensured.

Step S500: and performing control analysis based on the human skeleton characteristics and the human edge characteristics to obtain an abnormal behavior prediction result.

Specifically, after the car abnormal behavior detection device obtains the first behavior feature, the human skeleton feature, the second behavior feature and the human edge feature, the human skeleton feature and the human edge feature can be subjected to matching, de-duplication, screening, error correction and other processing, so as to determine how many specific persons exist in the elevator car, and then abnormal behavior prediction is performed based on the human skeleton feature and/or the human edge feature of the persons, so that an abnormal behavior prediction result based on human skeleton information and/or human edge information is obtained. It will be appreciated that the types of abnormal behavior referred to herein may be consistent with the types of abnormal behavior identified by the first and second identification models described above.

In the specific implementation process, when people are determined according to the human skeleton features and the human edge features, feature pairs can be obtained by matching the human skeleton features with the human edge features in a one-to-one correspondence manner, one person corresponds to one feature pair, and when a plurality of people exist in the elevator, the plurality of feature pairs are obtained correspondingly, so that all people in the elevator car are determined. For each person, the abnormal behavior prediction based on the human skeleton feature and the human edge feature may specifically be that the abnormal behavior prediction is performed based on the human skeleton feature of the person and the abnormal behavior prediction is performed based on the human edge feature of the person, and then the operations such as summarizing, matching, de-weighting and screening are performed on the respective predicted abnormal behaviors, so as to obtain a group of predicted abnormal behaviors as the abnormal behavior prediction result. The abnormal behavior prediction based on the human skeleton feature may be to predict a model constructed and trained by a machine learning algorithm such as a graph convolution neural network and a convolution countermeasure network, and the abnormal behavior prediction based on the human edge feature may be to predict a model constructed and trained by a deep learning algorithm such as an image semantic segmentation network and a convolution neural network. It can be understood that when the abnormal behavior prediction is performed based on the human skeleton feature and/or the human edge feature, the transient human skeleton information and the human edge information obtained for the video frame image at the current moment cannot predict an accurate result, so that the human skeleton feature and/or the human edge feature corresponding to the video frame image obtained by combining the history can be used for predicting, that is, the human skeleton feature and/or the human edge feature at a plurality of moments are utilized, and the information in the time dimension is considered.

The method has the advantages that the two dimensional information of the skeleton information and the edge information of the human body are utilized, a set of more accurate predicted abnormal behaviors are integrated, information extraction of different dimensions is firstly carried out on targets in the lift car, then comparison and error correction are carried out, behavior prediction is carried out on the basis of the determined targets and the skeleton information and the edge information extracted by the targets in a period of time, a set of abnormal behavior prediction results are obtained, more accurate abnormal behavior prediction in the time dimension is realized, multidimensional information is utilized, consideration is more comprehensive, the probability of misidentification of the human body can be reduced, and the accuracy of the abnormal behavior prediction results is guaranteed.

Step S600: and obtaining a final detection result according to the abnormal behavior recognition result and the abnormal behavior prediction result.

Specifically, the abnormal behavior recognition result is used as a detection result in the space dimension, the abnormal behavior prediction result is used as a detection result in the time dimension, and after the abnormal behavior recognition result and the abnormal behavior prediction result are obtained, the car abnormal behavior detection device can perform summarizing, statistics, screening, correction and other processing on the two groups of results to obtain a final abnormal behavior detection result aiming at the video frame image, and the final abnormal behavior detection result is used as a final detection result.

On the basis of the prior vision technology, the suspected abnormal behavior obtained by directly carrying out target detection by utilizing two recognition models is used as space recognition information, and the extracted multi-dimensional information such as human skeleton information, human edge information and the like is used for carrying out multi-dimensional abnormal behavior detection, including detection on a space domain and a time domain, so that a final detection result is obtained, and the accuracy of the final detection result is ensured.

According to the car abnormal behavior detection method provided by the embodiment, the first behavior characteristics and the human skeleton characteristics are obtained by utilizing the first recognition model to carry out target detection and skeleton recognition on the video frame image of the elevator car, the second behavior characteristics and the human edge characteristics are obtained by utilizing the second recognition model to carry out target detection and semantic segmentation on the video frame image, skeleton information and edge information are respectively extracted, and feature information extraction in different dimensions is achieved; then, based on the first behavior feature and the second behavior feature, carrying out matching processing to obtain an abnormal behavior recognition result, firstly mining abnormal behaviors in the space environment in the car to realize feature extraction in the space dimension, and then matching the behavior features detected by different recognition models to unify the behavior features into a set of abnormal behavior recognition results to realize abnormal behavior recognition in the space dimension; meanwhile, based on human skeleton characteristics and human edge characteristics, comparison analysis is carried out to obtain an abnormal behavior prediction result, information extraction of different dimensions is carried out on targets in a car, comparison error correction is carried out, so that behavior prediction is carried out based on the determined targets, a set of abnormal behavior prediction result is obtained, and more accurate abnormal behavior prediction in time dimension is realized; and finally, obtaining a final detection result according to the abnormal behavior recognition result and the abnormal behavior prediction result, and realizing the abnormal behavior detection by combining a plurality of dimensional information in the visual system, wherein the dimensional information is mutually corrected, so that the effect of improving the abnormal behavior detection accuracy and generalization capability is achieved.

Example two

Based on the same inventive concept, referring to fig. 3, a second embodiment of the abnormal car behavior detection method of the present invention is proposed, which is applied to the abnormal car behavior detection apparatus.

The following describes the method for detecting abnormal behavior of the car according to the present embodiment in detail with reference to the flowchart shown in fig. 3. The method may comprise the steps of:

step S100: and acquiring video frame images of the elevator car.

Specifically, step S100 may include:

step S110: acquiring video stream data of elevator car videos acquired by a camera;

step S120: and sampling the video stream data through a preset sampling frequency to obtain a video frame image.

In this embodiment, the camera may be used to capture an elevator car, collect an elevator car video, and send the captured elevator car video to the car abnormal behavior detection device in a video stream manner, where the car abnormal behavior detection device correspondingly receives video stream data of the elevator car video, so as to obtain a video frame image. The abnormal car behavior detection device may sample the video stream data with a preset sampling frequency, that is, instead of performing the method on each frame of the video, the method may be performed on sampling frames obtained by sampling at fixed intervals, where an image corresponding to each sampling frame is a video frame image obtained by the abnormal car behavior detection device. The car abnormal behavior detection device can call the first recognition model and the second recognition model for subsequent processing aiming at the video frame image.

In this embodiment, the first recognition model is an algorithm model having both the target detection and skeleton recognition functions, for example, a YOLO-else algorithm model. The structure of the YOLO-else algorithm model comprises a target detection head and a skeleton recognition head, wherein the target detection head can be a classified regression network structure of a target detection part in the model, and the skeleton recognition head can be a classified regression network structure of a skeleton recognition part. The target detection head of the YOLO-else algorithm model can directly detect people, identify all targets in video frame images, namely all people, classify normal people and abnormal behavior people, wherein the normal people can be people which stand, do not move greatly or do not perform abnormal actions, the abnormal behavior people can be people who do some move greatly or do abnormal actions, such as falling, door taking, smoking, jumping and the like, and particularly can set classification limits according to actual needs; the skeleton recognition head of the algorithm model can perform skeleton calculation on all targets, namely all people, in the video frame image, and extract skeleton characteristics of all people in the video frame image to obtain human skeleton characteristics. The skeleton generally comprises a plurality of skeleton points, and the number and the positions of the skeleton points can be set according to actual needs. The number of the first behavioral features output by the model may be identical to the number of the human skeleton features, or may be inconsistent.

In this embodiment, the second recognition model adopts an algorithm model having both the target detection and the semantic segmentation functions, for example, an example segmentation algorithm model. The structure of the example segmentation algorithm model comprises a target detection head and a semantic segmentation head, wherein the target detection head can be a classification regression network structure of a target detection part in the model, and the semantic segmentation head can be a classification regression network structure of a semantic segmentation part. The object detection head of the example segmentation algorithm model can directly detect people, identify all objects in video frame images, namely all people, classify normal people and abnormal behavior people, wherein the normal people can be people which stand, do not move greatly or do not perform abnormal actions, the abnormal behavior people can be people who do some move greatly or do abnormal actions such as falling, taking off a door, smoking, jumping and the like, and the classification limit can be set according to actual needs; the semantic segmentation head of the algorithm model can perform edge recognition on all targets, namely all people, in the video frame image, and extract edge mask features of all people in the video frame image to obtain human body edge features. The number of the second behavior features output by the model may be consistent with the number of the edge features of the human body or may not be consistent with the number of the edge features of the human body.

Specifically, as shown in fig. 3, step S400 may include:

step S410: matching the first behavior feature with the second behavior feature;

step S420: when the matching is consistent, a first suspected abnormal behavior is obtained;

step S430: when the matching is inconsistent, determining suspected abnormal behaviors according to a preset first priority, and obtaining second suspected abnormal behaviors;

step S440: and obtaining an abnormal behavior identification result according to the first suspected abnormal behavior or the second suspected abnormal behavior.

In this embodiment, the car abnormal behavior detection device may match the first behavior feature and the second behavior feature obtained in the foregoing steps, and when the first behavior feature, for example, A1, is consistent with the second behavior feature, for example, B1, for example, both of them are suspected to be abnormal behaviors of falling, then A1 and B1 may be combined into the abnormal behavior of falling, that is, a first suspected abnormal behavior; when the first behavior feature, for example A2, is inconsistent with the second behavior feature, for example B2, for example A2 is suspected to be a fall and B2 is suspected to be squat, determining that the abnormal behavior is a second suspected abnormal behavior when the fall is a fall according to a preset first priority, for example, the priority of the fall is greater than the priority of the squat. When a plurality of persons exist in the elevator car, a plurality of first behavior features and a plurality of second behavior features are correspondingly obtained, abnormal behaviors which are consistent in matching and abnormal behaviors which are inconsistent in matching may exist at the same time, or only abnormal behaviors which are consistent in matching or only abnormal behaviors which are inconsistent in matching may exist, so that according to different situations, finally, the abnormal behavior recognition results can be summarized according to the first suspected abnormal behaviors and/or the second suspected abnormal behaviors.

The first behavior feature and the second behavior feature are matched, so that the joint error correction of the suspected abnormal behavior of the current spatial domain is realized, and the suspected abnormal behavior of the spatial dimension after error correction, namely the abnormal behavior identification result of the embodiment, is obtained.

Specifically, as shown in fig. 3, step S500 may include:

step S510: comparing the human skeleton characteristics with the human edge characteristics to obtain a human identification result;

step S520: and carrying out behavior state change recognition on human body recognition results corresponding to the plurality of video frame images to obtain abnormal behavior prediction results.

In this embodiment, the car abnormal behavior detection device may match the human skeleton feature and the human edge feature acquired in the foregoing steps, specifically may match the human skeleton feature with the edge mask, correspondingly obtain a human recognition result, obtain human information on a video frame image at the current moment, and then combine the historical human information on a plurality of video frame images, for example, the latest preset number, to perform behavior state change recognition, so as to implement abnormal behavior prediction, and obtain an abnormal behavior prediction result.

In one embodiment, step S510 "performing a comparison process on the human skeleton feature and the human edge feature to obtain the human identification result" may include:

step S511: matching human skeleton features and human edge features to obtain feature pairs with association relations;

step S512: when the number of the feature pairs exceeds the preset number, judging whether the prediction frames corresponding to the feature pairs are overlapped or not; the prediction frames comprise a first prediction frame corresponding to human skeleton characteristics and a second prediction frame corresponding to human edge characteristics, wherein the first prediction frame is generated when the target detection is carried out on the video frame image based on a first recognition model, and the second prediction frame is generated when the target detection is carried out on the video frame image based on a second recognition model;

step S513: if no overlap exists, determining a human body identification result according to the feature pairs;

step S514: if the overlap exists, carrying out de-duplication processing on the prediction frame to obtain a feature pair after de-duplication, and determining a human body recognition result according to the feature pair after de-duplication.

Specifically, the feature pair means that when the human skeleton is matched with the human edge, the preset number of skeleton points on the human skeleton are all located in the outline of the human edge, so that the human skeleton and the human edge are very likely to belong to the same person, and then the human skeleton feature and the human edge feature can be regarded as a pair of features with association relation. It should be noted that, when the first recognition model detects a target in the video frame image, a prediction frame of the target, that is, a first prediction frame, is correspondingly generated, and the first prediction frame has a corresponding relationship with the recognized human skeleton feature; when the second recognition model detects the target of the video frame image, a second prediction frame which is a prediction frame of the target is correspondingly generated, and the second prediction frame has a corresponding relation with the recognized human edge characteristics.

In this embodiment, the car abnormal behavior detection device may match the human skeleton feature and the human edge feature, so that the features of different dimensions output by the two recognition models are associated with each other, and a feature pair with an association relationship is obtained; then, when the number of the feature pairs exceeds a preset number, for example, two or more feature pairs are obtained, whether the prediction frames corresponding to the feature pairs are overlapped or not is judged; if no overlap exists, the prediction frames of the human skeleton feature and the human edge feature identified by the video frame image are in a state that one person corresponds to one prediction frame, for example, two persons in a car stand separately, so that the situation that a plurality of prediction frames exist for one person to cause inaccurate detection results is avoided, and at the moment, one feature pair can be determined as one human body, so that a human body identification result is obtained; if overlapping exists, the situation that a plurality of prediction frames corresponding to one person possibly exist in the prediction frames aiming at the human skeleton features and the human edge features identified by the video frame images is indicated, and at the moment, the duplication elimination treatment can be carried out on the prediction frames, so that the feature pairs can be duplicated, one person corresponds to one feature pair, the feature pair obtained after duplication elimination is used for determining the one human body, and a human body identification result is obtained.

When matching the human skeleton feature and the human edge feature, a matching value can be calculated, for example, for a plurality of skeleton points of the human skeleton feature, such as 17 skeleton points, if the coordinate positions of the 17 skeleton points are all completely contained in the coordinate position of one human edge feature, the matching value can be 1, the matching value of different human skeletons and different human edges can be correspondingly calculated, the matching value is greater than or equal to zero and less than or equal to 1, and after the matching value is calculated according to the method, the matching value can be ranked or screened according to the size of the matching value, so that the human skeleton feature and the human edge feature with the association relationship can be obtained as a feature pair. The duplicate removal processing on the overlapped prediction frames can firstly extract the feature pairs corresponding to the two overlapped prediction frames, for example, the skeleton feature 1 and the edge feature 1 and the skeleton feature 2 and the edge feature 2 are obtained, whether the skeleton feature 1 is matched with the edge feature 2 is judged, whether the skeleton feature 2 is matched with the edge feature 1 is judged, or whether the skeleton feature 2 is matched with the edge feature 1 is judged at the same time, if the skeleton feature 2 is matched with the edge feature 1 is judged, the situation that one person has the two prediction frames at the moment is indicated, one of the two feature pairs can be deleted, duplicate removal is achieved, and if the skeleton feature 1 is not matched, the situation that two overlapped persons at the moment are likely to have the respective prediction frames is indicated, for example, the situation that two persons standing relatively close to each other need to be reserved at the moment.

In another embodiment, step S520 "performing behavior state change recognition on the human body recognition results corresponding to the plurality of video frame images" to obtain the abnormal behavior prediction result "may include:

step S521: constructing human body feature vectors aiming at human body recognition results corresponding to a plurality of video frame images, wherein the human body feature vectors are multidimensional time domain feature vectors;

step S522: and inputting the human body characteristic vector into a human body behavior prediction model established by using a deep learning algorithm to obtain an abnormal behavior prediction result.

Specifically, the deep learning algorithm may be an RNN (recurrent neural network ) algorithm, an LSTM (Long Short-Term Memory) algorithm, or the like, and may predict the character state transformation according to the human feature vector, identify the behavior state change condition of the task, and predict whether an abnormal behavior exists.

In this embodiment, after the car abnormal behavior detection device obtains the human body recognition result of the current video frame image according to step S510, the car abnormal behavior detection device may combine the human body recognition result with the human body recognition results of the plurality of video frame images sampled previously to form a feature vector on the N-dimensional time domain, i.e., a human body feature vector. Then, the abnormal behavior detection device of the car can calculate preset skeleton points of skeleton information in the feature vector, such as skeleton points related to abnormal behaviors, judge whether the behavior state changes according to the skeleton feature changes in the time domain, and correspondingly predict the abnormal behaviors, wherein the abnormal behavior prediction at the moment is to identify abnormal behaviors such as falling and door pulling of a person in the process of the behavior state changes, for example, the abnormal behaviors of the person can be predicted by finding out that the person falls from standing change based on the position changes of the skeleton points, and the abnormal behavior prediction in the time domain based on the feature changes is realized.

Specifically, as shown in fig. 3, step S600 may include:

step S610: constructing an abnormal behavior feature vector aiming at abnormal behavior recognition results corresponding to a plurality of video frame images, wherein the abnormal behavior feature vector is a multidimensional time domain feature vector;

step S620: matching the abnormal behavior prediction result with the abnormal behavior feature vector;

step S630: when the matching is consistent, a first feature vector correction result is obtained;

step S640: when the matching is inconsistent, determining abnormal behaviors according to a preset second priority, and obtaining a second feature vector correction result;

step S650: and obtaining a final detection result according to the first feature vector correction result or the second feature vector correction result.

Specifically, the setting rule of the second priority in step S640 may be inconsistent with the setting rule of the first priority in step S430, so as to consider specific situations of some specific abnormal behaviors, such as whether some abnormal behaviors need to be identified before in different detection stages, and specifically, the setting is set according to actual needs, which is not limited herein.

In this embodiment, after the abnormal behavior detection device of the car obtains the abnormal behavior recognition result of the current video frame image, the abnormal behavior detection device and the abnormal behavior recognition results of the previous sampled multiple video frame images may be combined together to form an N-dimensional feature vector, i.e. an abnormal behavior feature vector on the time domain. Then, the abnormal behavior prediction result and the abnormal behavior feature vector can be matched, and the abnormal behavior can be corrected, so that state error correction is realized. Specifically, based on the foregoing steps, an abnormal behavior prediction result can be predicted for a skeleton change of a human body in a certain stage, and at the same time, an abnormal behavior recognition result recognized by the human body at each moment in the stage can form an abnormal behavior feature vector, wherein one element of the feature vector is an abnormal behavior recognition result corresponding to a video frame image obtained at a certain moment, when the abnormal behavior prediction result is consistent with the abnormal behavior recognition result, it is indicated that the human body actually has an abnormal behavior, and at the moment, the abnormal behavior can be determined as an abnormal behavior of the human body, and an abnormal behavior correction result of the human body, namely a first feature vector correction result, is directly obtained; when the abnormal behavior prediction result is inconsistent with the abnormal behavior recognition result, it is indicated that the person may perform different abnormal behaviors at different moments, for example, the abnormal behavior feature vector includes standing and falling, and the abnormal behavior prediction result is only falling, at this time, two different abnormal behaviors can be determined to be the abnormal behaviors of the person according to a preset second priority, for example, the falling priority is greater than the standing priority, so that the abnormal behavior correction result of the human body, that is, the second feature vector correction result, is obtained.

When a plurality of persons exist in the elevator car, a plurality of first feature vector correction results and a plurality of second feature vector correction results are correspondingly obtained, abnormal behaviors which are consistent in matching and abnormal behaviors which are inconsistent in matching may exist at the same time, or only abnormal behaviors which are consistent in matching or only abnormal behaviors which are inconsistent in matching may exist, so that for different situations, finally, the abnormal behavior detection results of the video frame image at the current moment can be summarized according to the first feature vector correction results and/or the second feature vector correction results to obtain the final detection results.

In the process, the detection result of inconsistent abnormal behavior in the abnormal behavior feature vector in the time domain and the abnormal behavior predicted based on the state change process is corrected, so that error correction of the detection result is realized, and the accuracy of the final detection result is improved.

In one embodiment, step S650 "obtaining the final detection result according to the first feature vector correction result or the second feature vector correction result" may include:

step S651: obtaining a corrected abnormal behavior feature vector according to the first feature vector correction result or the second feature vector correction result;

Step S652: carrying out statistical analysis on the corrected abnormal behavior feature vector to obtain a statistical result of the abnormal behavior;

step S653: and obtaining a final detection result according to the statistical result and a preset judgment condition.

Further, for the corrected abnormal behavior, statistical analysis may be performed, specifically, for the abnormal behavior detection result obtained according to the first feature vector correction result or the second feature vector correction result, the abnormal behavior detection result is temporarily not used as a final detection result, but is defined as a corrected abnormal behavior feature vector, then statistical analysis is performed on each element in the corrected abnormal behavior feature vector, for example, step S610 obtains an abnormal behavior feature vector for the abnormal behavior recognition result corresponding to M video frame images, and the corrected abnormal behavior feature vector is obtained after correction, where M elements are provided, one element represents one abnormal behavior identified, whether N is a specified abnormal behavior in the M abnormal behaviors can be counted, M and N are both positive integers, N is less than or equal to M, for example, 8 abnormal behaviors in the 10 abnormal behaviors are all B behaviors, or the number of times that each type of abnormal behavior is identified in the detection process for the total M times can be counted, for example, the a behaviors are 2, and the B behaviors have 8 abnormal behaviors; then, according to the counted situation, whether the preset judging condition is met or not can be judged, for example, the abnormal behavior reaching 7 times in the current 10 times of detection can be used as the abnormal behavior detection result of the current video frame image, and then the B behaviors corresponding to the two statistical results can be used as the output of the current detection, namely the abnormal behavior detection result of the video frame image obtained in the step S100, namely the final detection result. Based on the above example, the preset determination condition is that the same type of abnormal behavior or a specified certain abnormal behavior is detected n=7 times in the detection of the consecutive M video frame images.

In the process, the abnormal behaviors in the corrected abnormal behavior feature vector in the time domain are further screened and determined, so that probability error correction is realized, and the accuracy of a final detection result can be further improved.

For more details of the above method steps, reference may be made to the description of the specific implementation in the first embodiment, and for brevity of description, a detailed description will not be repeated here.

According to the elevator car abnormal behavior detection method based on machine vision, a multi-dimensional deep learning algorithm is used for multi-dimensional mining of spatial features in a target scene, feature complementation is achieved, feature information of different dimensions is used for identification, and mutual error correction is performed according to different characterization forms in time dimensions including abnormal behavior identification results in time dimensions and abnormal behavior prediction results in time dimensions, so that the effect of improving the abnormal behavior detection accuracy and the floodability in the elevator car scene is achieved. In addition, the result correction is carried out based on the abnormal behavior, and the result correction is further carried out based on the detection times of the abnormal behavior, so that the accuracy of a final detection result is ensured through double correction.

Example III

Based on the same inventive concept, referring to fig. 4, a third embodiment of the abnormal car behavior detection method of the present invention is proposed, which is applied to the abnormal car behavior detection apparatus.

As shown in the flow chart of fig. 4, the method for detecting abnormal behavior of a car may specifically include the following steps:

step A1: sampling a car video;

the method comprises the steps that video in an elevator car is collected through a camera, and a video stream is sampled through a certain sampling frequency to obtain video frame images;

step A2: obtaining information by utilizing a YOLO-else algorithm;

performing target detection and skeleton recognition on the video frame image by using a YOLO-Pose algorithm model to obtain a first behavior feature and a human skeleton feature;

step A3: obtaining information by using an example segmentation algorithm;

performing target detection and semantic segmentation on the video frame image by using an example segmentation algorithm model to obtain a second behavior feature and a human edge feature;

step A4: performing joint error correction to obtain an abnormal behavior recognition result;

matching the first behavior feature and the second behavior feature to realize joint error correction and obtain an abnormal behavior identification result of the space dimension after error correction;

Step A5: target matching and duplication removal are carried out, and a human body identification result is obtained;

performing contrast analysis on human skeleton characteristics and human edge characteristics to realize target matching and de-duplication, correlating different dimension characteristics output by the two models, removing the phenomenon of a re-prediction frame in a model output result, obtaining a de-duplicated characteristic pair, and determining a human body recognition result according to the de-duplicated characteristic pair;

step A6: acquiring a human body characteristic vector in a time domain;

combining the human body identification result of the current frame obtained in the step A5 with the human body identification result obtained in the step A5 of the previous frame to form a human body feature vector in the N-dimensional time domain;

step A7: obtaining an abnormal behavior prediction result based on behavior state transition;

calculating skeleton points related to abnormal behaviors in the human body feature vector by using a deep learning algorithm, judging whether behavior state transition occurs according to the strength of time domain feature change, and predicting the abnormal behaviors to obtain an abnormal behavior prediction result;

step A8: obtaining an abnormal behavior feature vector in a time domain;

combining the abnormal behavior recognition result of the current frame obtained in the step A4 with the abnormal behavior recognition result obtained in the step A4 of the previous frame to form an N-dimensional abnormal behavior feature vector in the time domain;

Step A9: correcting the state to obtain a corrected abnormal behavior feature vector;

correcting the abnormal behavior which is inconsistent with the state transition rule in the abnormal behavior feature vector in the step A8, namely the abnormal behavior predicted result, so as to realize state error correction and obtain a corrected abnormal behavior feature vector;

step A10: probability error correction is carried out to obtain the finally determined abnormal behavior so as to output a final detection result;

and (3) carrying out statistical analysis on the corrected abnormal behavior feature vector in the step (A9) to realize probability error correction, taking N abnormal behaviors occurring in continuous M times of abnormal behavior detection as the finally determined abnormal behaviors, and outputting the finally determined abnormal behaviors by the final detection result of the current frame, wherein M, N is a positive integer, and N is smaller than or equal to M.

It should be noted that, for more details of implementation in the specific implementation of the above method steps, reference may be made to the description of the specific implementation in the first embodiment or the second embodiment, and for brevity of description, a detailed description is not repeated here.

The method for detecting the abnormal behavior of the car provided by the embodiment provides a more specific implementation mode, the spatial features in the target scene are excavated in multiple dimensions by using a multi-dimensional deep learning algorithm, and the different characterization forms of the feature information in different dimensions in the time dimension are used for mutual error correction, so that the aim of simultaneously improving the recognition accuracy and the generalization capability of the abnormal behavior in the car scene is fulfilled.

Example IV

Based on the same inventive concept, referring to fig. 5, a first embodiment of the abnormal car behavior detection apparatus of the present invention is presented, which may be a virtual apparatus applied to the abnormal car behavior detection device.

The following describes in detail the device for detecting abnormal behavior of a car provided in this embodiment with reference to a schematic diagram of functional modules shown in fig. 5, where the device may include:

Further, the first result module may include:

the first matching unit is used for matching the first behavior characteristic and the second behavior characteristic; when the matching is consistent, a first suspected abnormal behavior is obtained; when the matching is inconsistent, determining suspected abnormal behaviors according to a preset first priority, and obtaining second suspected abnormal behaviors;

the first result unit is used for obtaining an abnormal behavior identification result according to the first suspected abnormal behavior or the second suspected abnormal behavior.

Further, the second result module may include:

the second matching unit is used for carrying out contrast treatment on the human skeleton characteristics and the human edge characteristics to obtain a human identification result;

and the second result unit is used for carrying out behavior state change recognition on human body recognition results corresponding to the plurality of video frame images to obtain an abnormal behavior prediction result.

Further, the second matching unit is specifically configured to match the human skeleton feature and the human edge feature to obtain a feature pair with an association relationship; when the number of the feature pairs exceeds the preset number, judging whether the prediction frames corresponding to the feature pairs are overlapped or not; if no overlap exists, determining a human body identification result according to the feature pairs; if the overlap exists, carrying out de-duplication treatment on the prediction frame to obtain a feature pair after de-duplication, and determining a human body recognition result according to the feature pair after de-duplication; the prediction frames comprise a first prediction frame corresponding to human skeleton characteristics and a second prediction frame corresponding to human edge characteristics, the first prediction frame is generated when the target detection is carried out on the video frame image based on the first recognition model, and the second prediction frame is generated when the target detection is carried out on the video frame image based on the second recognition model.

Further, the second result unit is specifically configured to construct a human body feature vector according to human body recognition results corresponding to the plurality of video frame images, where the human body feature vector is a multidimensional time domain feature vector; and inputting the human body characteristic vector into a human body behavior prediction model established by using a deep learning algorithm to obtain an abnormal behavior prediction result.

Further, the result output module may include:

the time domain summarizing unit is used for constructing an abnormal behavior feature vector aiming at abnormal behavior recognition results corresponding to the plurality of video frame images, wherein the abnormal behavior feature vector is a multidimensional time domain feature vector;

the result matching unit is used for matching the abnormal behavior prediction result with the abnormal behavior feature vector; when the matching is consistent, a first feature vector correction result is obtained; when the matching is inconsistent, determining abnormal behaviors according to a preset second priority, and obtaining a second feature vector correction result;

and the result output unit is used for obtaining a final detection result according to the first characteristic vector correction result or the second characteristic vector correction result.

Further, the result output unit is specifically configured to obtain a corrected abnormal behavior feature vector according to the first feature vector correction result or the second feature vector correction result; carrying out statistical analysis on the corrected abnormal behavior feature vector to obtain a statistical result of the abnormal behavior; and obtaining a final detection result according to the statistical result and a preset judgment condition.

It should be noted that, the functions and the corresponding technical effects that can be achieved by each module in the device for detecting abnormal car behavior provided in this embodiment may refer to descriptions of specific embodiments in each embodiment of the method for detecting abnormal car behavior of the present invention, and for brevity of description, details are not repeated here.

Example five

Based on the same inventive concept, referring to the hardware structure schematic diagram of fig. 2, this embodiment provides a device for detecting abnormal car behavior, which may include a processor and a memory, where a program for detecting abnormal car behavior is stored, and when the program for detecting abnormal car behavior is executed by the processor, all or part of the steps of each embodiment of the method for detecting abnormal car behavior of the present invention are implemented.

Specifically, the car abnormal behavior detection device refers to terminal devices or network devices capable of realizing network connection, and may be terminal devices such as a mobile phone, a computer, a tablet computer, a portable computer, an embedded industrial personal computer, or network devices such as a server and a cloud platform.

It will be appreciated that the car abnormal behaviour detection apparatus may also comprise a communication bus, a user interface and a network interface. Wherein the communication bus is used for realizing connection communication among the components; the user interface is used for connecting the client and carrying out data communication with the client, and can comprise an output unit such as a display screen, a loudspeaker and the like, and an input unit such as a keyboard, a microphone and the like; the network interface is used for connecting with the background server and carrying out data communication with the background server, and can comprise an input/output interface, such as a standard wired interface and a wireless interface, such as a Wi-Fi interface; the Memory is used for storing various types of data, and the data can include, for example, instructions of any application program or method in the car abnormal behavior detection device, and application program related data, and the Memory can be implemented by any type of volatile or nonvolatile storage device or combination thereof, such as random access Memory (Random Access Memory, RAM), static random access Memory (Static Random Access Memory, SRAM), read-Only Memory (ROM), programmable Read-Only Memory (Programmable Read-Only Memory, PROM), erasable programmable Read-Only Memory (Erasable Programmable Read-Only Memory, EPROM), electrically erasable programmable Read-Only Memory (Electrically Erasable Programmable Read-Only Memory, EEPROM), magnetic Memory, flash Memory, magnetic or optical disk, and the like; optionally, the memory may also be a processor-independent storage device; the processor, which may be an application specific integrated circuit (Application Specific Integrated Circuit, ASIC), digital signal processor (Digital Signal Processor, DSP), programmable logic device (Programmable Logic Device, PLD), field programmable gate array (Field Programmable Gate Array, FPGA), controller, microcontroller, microprocessor or other electronic component, is used to invoke the car abnormal behavior detection program stored in the memory and to perform all or part of the steps of the various embodiments of the car abnormal behavior detection method described above.

It should be noted that the hardware configuration shown in fig. 2 does not constitute a limitation of the car abnormal behavior detection apparatus of the present invention, and may include more or fewer components than shown, or may combine some components, or may be arranged in different components.

Example six

Based on the same inventive concept, the present embodiment provides a computer readable storage medium, such as a flash memory, a hard disk, a multimedia card, a card memory (e.g., SD or DX memory, etc.), a Random Access Memory (RAM), a Static Random Access Memory (SRAM), a Read Only Memory (ROM), a Programmable Read Only Memory (PROM), an Erasable Programmable Read Only Memory (EPROM), an Electrically Erasable Programmable Read Only Memory (EEPROM), a magnetic memory, a magnetic disk, an optical disk, a server, etc., on which a computer program is stored, which computer program is executable by one or more processors, and which computer program, when executed by the processors, can implement all or part of the steps of the various embodiments of the car abnormal behavior detection method of the present invention.

It should be noted that, the foregoing reference numerals of the embodiments of the present invention are only for describing the embodiments, and do not represent the advantages and disadvantages of the embodiments. The above embodiments are only optional embodiments of the present invention, and not limiting the scope of the present invention, and all equivalent structures or equivalent processes using the descriptions of the present invention and the accompanying drawings or direct or indirect application in other related technical fields are included in the scope of the present invention.

Claims

1. A method for detecting abnormal behavior of a car, the method comprising:

acquiring a video frame image of an elevator car;

performing control analysis based on the human skeleton characteristics and the human edge characteristics to obtain an abnormal behavior prediction result;

and obtaining a final detection result according to the abnormal behavior identification result and the abnormal behavior prediction result.

2. The method for detecting abnormal behavior of a car according to claim 1, wherein the step of performing matching processing based on the first behavior feature and the second behavior feature to obtain an abnormal behavior recognition result includes:

matching the first behavior feature with the second behavior feature;

3. The method for detecting abnormal behavior of a car according to claim 1, wherein the step of obtaining an abnormal behavior prediction result based on the human skeleton feature and the human edge feature by performing a comparison analysis includes:

performing contrast treatment on the human skeleton characteristics and the human edge characteristics to obtain a human identification result;

and carrying out behavior state change recognition on the human body recognition results corresponding to the video frame images to obtain abnormal behavior prediction results.

4. The method for detecting abnormal car behavior according to claim 3, wherein said step of comparing said human skeleton feature with said human edge feature to obtain a human identification result comprises:

matching the human skeleton characteristics with the human edge characteristics to obtain characteristic pairs with association relations;

when the number of the feature pairs exceeds the preset number, judging whether the prediction frames corresponding to the feature pairs overlap or not; the prediction frames comprise a first prediction frame corresponding to the human skeleton characteristics and a second prediction frame corresponding to the human edge characteristics, the first prediction frame is generated when the target detection is carried out on the video frame image based on the first recognition model, and the second prediction frame is generated when the target detection is carried out on the video frame image based on the second recognition model;

If no overlap exists, determining a human body identification result according to the characteristic pair;

if the overlap exists, carrying out de-duplication processing on the prediction frame to obtain a de-duplicated feature pair, and determining a human body recognition result according to the de-duplicated feature pair.

5. The method for detecting abnormal behavior of a car according to claim 3, wherein the step of performing behavior state change recognition with respect to the human body recognition results corresponding to the plurality of video frame images to obtain an abnormal behavior prediction result comprises:

constructing human body feature vectors aiming at the human body recognition results corresponding to the video frame images, wherein the human body feature vectors are multidimensional time domain feature vectors;

6. The car abnormal behavior detection method according to claim 1, wherein the step of obtaining a final detection result based on the abnormal behavior recognition result and the abnormal behavior prediction result comprises:

constructing an abnormal behavior feature vector aiming at the abnormal behavior recognition results corresponding to a plurality of video frame images, wherein the abnormal behavior feature vector is a multidimensional time domain feature vector;

and obtaining a final detection result according to the first characteristic vector correction result or the second characteristic vector correction result.

7. The method of claim 6, wherein the step of obtaining a final detection result based on the first feature vector correction result or the second feature vector correction result comprises:

8. A car abnormal behavior detection device, characterized in that the device comprises:

The first recognition module is used for carrying out target detection and skeleton recognition on the video frame image by utilizing a first recognition model to obtain a first behavior feature and a human skeleton feature;

the second recognition module is used for carrying out target detection and semantic segmentation on the video frame image by utilizing a second recognition model to obtain a second behavior characteristic and a human edge characteristic;

9. A car abnormal behavior detection apparatus, characterized in that the apparatus comprises a processor and a memory, on which a car abnormal behavior detection program is stored, which when executed by the processor, implements the car abnormal behavior detection method according to any one of claims 1 to 7.

10. A computer-readable storage medium, wherein the storage medium has stored thereon a computer program which, when executed by one or more processors, implements the car abnormal behavior detection method according to any one of claims 1 to 7.