CN112200044B

CN112200044B - Abnormal behavior detection method and device and electronic equipment

Info

Publication number: CN112200044B
Application number: CN202011058838.8A
Authority: CN
Inventors: 蔚勇; 刘树明; 朱小龙
Original assignee: Navinfo Co Ltd
Current assignee: Navinfo Co Ltd
Priority date: 2020-09-30
Filing date: 2020-09-30
Publication date: 2024-04-30
Anticipated expiration: 2040-09-30
Also published as: CN112200044A

Abstract

The application provides a method and a device for detecting abnormal behaviors and electronic equipment, wherein the method comprises the following steps: acquiring data to be identified, wherein a target in the data to be identified comprises a vehicle; determining an identification frame of the vehicle through an identification frame detection module of a key point detection network model, and detecting key points of the vehicle in the identification frame of the vehicle by utilizing the key point detection module of the key point detection network model; performing key point tracking on the key points of the vehicle to determine the motion trail of the vehicle; and determining whether abnormal behaviors exist in the vehicle according to the movement track of the vehicle and the area where the vehicle is located. The method is suitable for various behavior scenes, various abnormal behaviors are detected, different surrounding environments can be adapted, and the robustness of abnormal behavior detection is good.

Description

Abnormal behavior detection method and device and electronic equipment

Technical Field

The present application relates to the field of abnormal behavior detection technologies, and in particular, to a method and an apparatus for detecting abnormal behavior, and an electronic device.

Background

At present, intelligent video monitoring has become one of the very active hot spot technologies in the field of artificial intelligence, and is an important application of computer vision technology in the field of security and protection. Traffic peaks or people flow peaks often exist in public areas such as traffic intersections, railway stations, subway stations, squares and the like, and the method is particularly important for detecting abnormal behaviors of vehicles and pedestrians, avoiding traffic accidents and timely finding the traffic accidents.

In the prior art, for detecting abnormal traffic behaviors, a traditional algorithm is generally adopted to detect abnormal traffic behaviors, and the method mainly comprises the steps of collecting traffic data, detecting the traffic data through a background difference method to determine a moving vehicle, comparing the detected traffic data with a predefined behavior to determine whether the behavior of the moving vehicle is the predefined behavior, so that the abnormal vehicle behavior is detected.

However, in the prior art, the detection of abnormal behaviors by adopting a traditional algorithm is limited by various behavior scenes, and the robustness of the detection of abnormal behaviors is poor.

Disclosure of Invention

The application provides a method, a device and electronic equipment for detecting abnormal behaviors, which are used for detecting the abnormal behaviors, are suitable for various behavior scenes and have good robustness.

In a first aspect, an embodiment of the present application provides a method for detecting abnormal behavior, including:

Acquiring data to be identified, wherein a target in the data to be identified comprises a vehicle; determining an identification frame of the vehicle through an identification frame detection module of the key point detection network model, and detecting key points of the vehicle in the identification frame of the vehicle by utilizing the key point detection module of the key point detection network model; performing key point tracking on key points of the vehicle to determine a motion trail of the vehicle; and determining whether the vehicle has abnormal behaviors according to the movement track of the vehicle and the area where the vehicle is located.

In the embodiment of the application, the key point detection is carried out on the vehicle in the data to be identified through the key point detection network model, the movement track of the vehicle is determined according to the key points of the vehicle, and whether the abnormal behavior exists in the vehicle is determined by utilizing the movement track of the vehicle and the area where the vehicle is positioned, so that the method and the device are suitable for various behavior scenes, can be used for detecting various abnormal behaviors, can be suitable for different surrounding environments, and have good robustness in abnormal behavior detection. Compared with the prior art that the background difference method is adopted to detect traffic data to determine the moving vehicle, and then the method is used for comparing the detected abnormal behavior with the predefined behavior, the accuracy and the flexibility of abnormal behavior detection can be improved.

In one possible embodiment, determining whether the vehicle has abnormal behavior according to the movement track of the vehicle and the region in which the vehicle is located includes:

If the area where the vehicle is located is a lane area and the movement track of the vehicle is different from the specified driving direction of the lane area, determining that the vehicle has a retrograde behavior; and/or if the area where the vehicle is located is a non-road area and the motion trail of the vehicle is unchanged within a first preset time, determining that the vehicle has a behavior of exiting the road surface; and/or if the area where the vehicle is located is a central intersection area and the motion trail of the vehicle is unchanged within a second preset time, determining that the vehicle has the behavior of stopping at the intersection; and/or if the first vehicle with the changed movement track exists in the area outside the current passable lane area and the vehicle track of the first vehicle passes through the central intersection area, determining that the first vehicle has the behavior of running the red light.

In the embodiment of the application, the judgment of the abnormal behavior of the vehicle is realized.

In a possible implementation manner, the object includes a pedestrian, and the abnormal behavior detection method provided by the embodiment of the application may further include:

Determining a pedestrian identification frame and key points of pedestrians through a key point detection network model; inputting the identification frame of the pedestrian and key points of the pedestrian into a behavior discrimination network model to determine the human body posture of the pedestrian; and determining whether the vehicle has abnormal behaviors according to the human body posture, the area where the pedestrian is located and the recognition frame of the vehicle.

In the embodiment of the application, the behavior of the vehicle is judged by utilizing the human body posture of the pedestrian, the area where the pedestrian is located and the identification frame of the vehicle, so that the applicability of detecting the abnormal behavior of the vehicle can be improved.

In one possible embodiment, determining whether the vehicle has abnormal behavior according to the human body posture, the area where the pedestrian is located, and the recognition frame of the vehicle includes:

If the overlapping time of the identification frames of at least two second vehicles is detected to exceed the third preset time and pedestrians in second preset postures are detected in a preset range outside the identification frames of the second vehicles, determining that collision behaviors exist in the second vehicles, wherein the second preset postures comprise any one or a combination of a plurality of upright postures, lying postures and squatting postures.

In the embodiment of the application, the judgment of whether the collision behavior exists in the vehicle is realized by combining the recognition frame of the vehicle and the human body gesture of the pedestrian, and the reliability of abnormal behavior detection is improved.

In one possible implementation manner, before acquiring the data to be identified, the method further includes:

acquiring data to be calibrated; and calibrating the picture area in the data to be calibrated so as to divide the picture area into different areas.

According to the method and the device for detecting the abnormal behavior, before the data to be identified are acquired, the image area in the data to be calibrated is accurately calibrated, and therefore the reliability of detecting the abnormal behavior can be further improved.

In one possible implementation manner, calibrating the picture area in the data to be calibrated includes:

Detecting lane lines, crosswalk lines and traffic lights in data to be calibrated; dividing a picture area into a lane area, a center crossing area, a crosswalk area and a non-road area according to the lane lines and the crosswalk lines; determining the running direction of a vehicle and the track of a pedestrian in data to be calibrated; determining the running direction of the vehicle exceeding a preset proportion in the lane area as the specified running direction of the lane area; when the pedestrian track changes in the pedestrian crosswalk area, the traffic signal lamp which is lighted by the green signal lamp is determined as the traffic signal lamp of the pedestrian crosswalk area.

acquiring a training data sample; constructing a network structure of a key point detection network model; training the training data sample by utilizing the network structure of the key point detection network model to generate a trained key point detection network model.

In one possible implementation, the network structure of the keypoint detection network model includes a two-stage detection network module, an identification frame detection module, and a keypoint detection module; the output of the two-stage detection network module is respectively connected with the input of the identification frame detection module and the input of the key point detection module, the output of the identification frame detection module is connected with the input of the key point detection module, and the identification frame detection module sequentially comprises a box-pooler module, a box-head module, a box-classes module and a box-boxes module; the key point detection module sequentially comprises keypoint-pooler module, keypoint-head module and keypoint-location module; the identification frame detection module is used for identifying an identification frame of the target, and the key point detection module is used for detecting key points of the target.

In a possible implementation manner, the abnormal behavior detection method provided by the embodiment of the present application further includes:

If the target has abnormal behaviors, alarm information is pushed to the user terminal, wherein the alarm information comprises the abnormal behaviors of the vehicle and the position of the vehicle.

In the embodiment of the application, the abnormal behavior of the vehicle and the position of the vehicle are pushed to the user terminal, so that the abnormal vehicle can be processed timely and accurately by the user.

The following describes an apparatus, an electronic device, a computer readable storage medium, and a computer program product provided by the embodiments of the present application, and the content and effects thereof may refer to the abnormal behavior detection method provided by the embodiments of the present application, which are not described in detail.

In a second aspect, an embodiment of the present application provides an abnormal behavior detection apparatus, including:

the acquisition module is used for acquiring data to be identified, and targets in the data to be identified comprise vehicles.

The detection module is used for determining the identification frame of the vehicle through the identification frame detection module of the key point detection network model, and detecting the key point of the vehicle in the identification frame of the vehicle by utilizing the key point detection module of the key point detection network model.

The determining module is used for carrying out key point tracking on key points of the vehicle and determining the motion trail of the vehicle; and determining whether the vehicle has abnormal behaviors according to the movement track of the vehicle and the area where the vehicle is located.

In a possible implementation manner, the determining module is specifically configured to:

In one possible embodiment, the target includes a pedestrian, and the determining module is further configured to:

If the traffic signal lamp of the pedestrian crosswalk area is a red lamp and the motion track of the pedestrian passes through the crosswalk area, determining that the pedestrian runs the red lamp.

In one possible implementation, the target includes a pedestrian, and the determining module further includes: inputting key points of the pedestrians and identification frames of the pedestrians into a behavior discrimination network model to determine the human body posture of the pedestrians; and determining whether the pedestrian has abnormal behaviors according to the human body posture and the area where the pedestrian is located.

If the area where the pedestrian is located is any one of a lane area, a central intersection area or a crosswalk area and the gesture of the pedestrian is a first preset gesture, determining that the pedestrian has abnormal behaviors, wherein the first preset gesture comprises a lying gesture or a squatting gesture.

In a possible implementation manner, the device for detecting abnormal behavior provided by the embodiment of the application further comprises a calibration module.

The calibration module is used for acquiring data to be calibrated and calibrating the picture area in the data to be calibrated so as to divide the picture area into different areas.

In one possible embodiment, the calibration module is specifically configured to:

In a possible implementation manner, the abnormal behavior detection device provided by the embodiment of the application further comprises a training module.

The training module is specifically used for acquiring training data samples; constructing a network structure of a key point detection network model; training the training data sample by utilizing the network structure of the key point detection network model to generate a trained key point detection network model.

In a possible implementation manner, the device for detecting abnormal behavior provided by the embodiment of the present application further includes:

The pushing module is used for pushing alarm information to the user terminal if the abnormal behavior exists in the vehicle, wherein the alarm information comprises the abnormal behavior of the vehicle and the position of the vehicle.

In a third aspect, an embodiment of the present application provides an electronic device, including:

at least one processor; and

A memory communicatively coupled to the at least one processor; wherein the method comprises the steps of

The memory stores instructions executable by the at least one processor to enable the at least one processor to perform a method as provided by the first aspect or an implementation of the first aspect.

In a fourth aspect, embodiments of the present application provide a non-transitory computer-readable storage medium storing computer instructions for causing a computer to perform a method as provided by the first aspect or an implementation of the first aspect.

In a fifth aspect, embodiments of the present application provide a computer program product comprising: executable instructions for implementing a method as provided in the first aspect or the alternative of the first aspect.

According to the abnormal behavior detection method, the abnormal behavior detection device and the electronic equipment, the data to be identified are obtained, the targets in the data to be identified comprise vehicles, then the identification frames of the vehicles are determined through the identification frame detection module of the key point detection network model, the key points of the vehicles in the identification frames of the vehicles are detected through the key point detection module of the key point detection network model, and finally the key points of the vehicles are tracked to determine the movement track of the vehicles; and determining whether the vehicle has abnormal behaviors according to the movement track of the vehicle and the area where the vehicle is located. In the embodiment of the application, the key point detection is carried out on the vehicle in the data to be identified through the key point detection network model, the motion trail of the vehicle is determined according to the key points of the vehicle, and whether the abnormal behavior exists in the vehicle is determined by utilizing the motion trail of the vehicle and the area where the vehicle is positioned, so that the method and the device are suitable for various behavior scenes, can detect various abnormal behaviors, can adapt to different surrounding environments, and have good robustness in abnormal behavior detection. Compared with the prior art that the background difference method is adopted to detect traffic data to determine the moving vehicle, and then the method is used for comparing the detected abnormal behavior with the predefined behavior, the accuracy and the flexibility of abnormal behavior detection can be improved.

Drawings

In order to more clearly illustrate the embodiments of the present application or the technical solutions of the prior art, the drawings that are needed in the embodiments or the description of the prior art will be briefly described below, it will be obvious that the drawings in the following description are some embodiments of the present application, and that other drawings can be obtained according to these drawings without inventive effort to a person skilled in the art.

FIG. 1 is an exemplary application scenario architecture diagram provided by an embodiment of the present application;

FIG. 2 is a flowchart illustrating a method for detecting abnormal behavior according to an embodiment of the present application;

FIG. 3 is a schematic illustration of key points of a vehicle according to an embodiment of the present application;

FIG. 4 is a schematic illustration of key points of a pedestrian provided by an embodiment of the application;

FIG. 5 is a schematic illustration of a traffic intersection provided in an embodiment of the present application;

FIG. 6 is a flowchart illustrating a method for detecting abnormal behavior according to still another embodiment of the present application;

FIG. 7 is a schematic diagram of a network structure of a network model for detecting key points according to an embodiment of the present application;

FIG. 8 is a schematic diagram of an abnormal behavior detection apparatus according to an embodiment of the present application;

Fig. 9 is a schematic structural diagram of an electronic device according to an embodiment of the present application.

Detailed Description

For the purpose of making the objects, technical solutions and advantages of the embodiments of the present application more apparent, the technical solutions of the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present application, and it is apparent that the described embodiments are some embodiments of the present application, but not all embodiments of the present application. All other embodiments, which can be made by those skilled in the art based on the embodiments of the application without making any inventive effort, are intended to be within the scope of the application.

The terms "first," "second," "third," "fourth" and the like in the description and in the claims and in the above drawings, if any, are used for distinguishing between similar objects and not necessarily for describing a particular sequential or chronological order. It is to be understood that the data so used may be interchanged where appropriate such that the embodiments of the application described herein may be implemented, for example, in sequences other than those illustrated or otherwise described herein. Furthermore, the terms "comprises," "comprising," and "having," and any variations thereof, are intended to cover a non-exclusive inclusion, such that a process, method, system, article, or apparatus that comprises a list of steps or elements is not necessarily limited to those steps or elements expressly listed but may include other steps or elements not expressly listed or inherent to such process, method, article, or apparatus.

At present, intelligent video monitoring has become one of the very active hot spot technologies in the field of artificial intelligence, and is an important application of computer vision technology in the field of security and protection. Traffic peaks or people flow peaks often exist in public areas such as traffic intersections, railway stations, subway stations, squares and the like, abnormal behavior detection on vehicles and pedestrians is particularly important for avoiding accidents and timely finding the accidents. In the prior art, for detecting abnormal traffic behaviors, a traditional algorithm is generally adopted to detect abnormal traffic behaviors, and the method mainly comprises the steps of collecting traffic data, detecting the traffic data through a background difference method to determine a moving vehicle, comparing the detected traffic data with a predefined behavior to determine whether the behavior of the moving vehicle is the predefined behavior, so that the abnormal vehicle behavior is detected. However, in the prior art, the detection of abnormal behaviors by adopting a traditional algorithm is limited by various behavior scenes, and the robustness of the detection of abnormal behaviors is poor.

The method and the device for detecting the abnormal behavior and the electronic equipment provided by the embodiment of the application have the advantages that the key point detection is carried out on the target in the data to be identified through the key point detection network model, whether the abnormal behavior exists in the target is determined according to the key point of the target and the area where the target is located, the method and the device for detecting the abnormal behavior are suitable for various behavior scenes, the various abnormal behaviors are detected, different surrounding environments can be met, and the robustness of detecting the abnormal behavior is good. In addition, by using the key point tracking and track calculation of the target, more detailed and accurate information of the position, the behavior and the like of the target can be obtained, the detection of the behavior of the target is more convenient and accurate, and the method is not limited to the acquisition weather of the data to be identified, such as rainy days, sunny days, cloudy days, daytime or night, and the like.

In the following, an exemplary application scenario of an embodiment of the present application is described.

The abnormal behavior detection method provided by the embodiment of the application can be executed by the abnormal behavior detection device provided by the embodiment of the application, and the abnormal behavior detection device provided by the embodiment of the application can be integrated on the terminal equipment or can be the terminal equipment. The embodiment of the application does not limit the specific type of the terminal equipment, and for example, the terminal equipment can be a video camera, a personal computer, a tablet personal computer, a wearable device, a vehicle-mounted terminal, a monitoring device and the like. Fig. 1 is a schematic diagram of an exemplary application scenario provided in an embodiment of the present application, as shown in fig. 1, where the architecture mainly includes: a terminal device 01 (personal computer) and a monitoring device 02. In a possible implementation manner, the abnormal behavior detection method provided by the embodiment of the application can be applied to the monitoring device 02, the data to be identified is obtained through the camera of the monitoring device 02, the data to be identified is processed through the graphic processor in the monitoring device, and then the processing result can be sent to the terminal device 01. In another possible implementation manner, the embodiment of the present application may be applied to the terminal device 01, by receiving the data to be identified captured by the monitoring device 02, and then processing the data to be identified. In still another possible implementation manner, the application scenario architecture diagram provided by the embodiment of the present application may further include a server 03, where the terminal device 01 obtains the data to be identified through the monitoring device 02, processes the data to be identified through the server 03, sends a processing result to the terminal device 01, and the terminal device 01 receives the processing result sent by the server 03. The embodiment of the present application is merely taken as an example, and is not limited thereto.

It should be noted that, the embodiment of the present application is only described by taking application to traffic intersections as an example, and the embodiment of the present application may also be applied to scenes such as railway stations, malls, hospitals, schools, subway openings, etc. to implement abnormal behavior detection on pedestrians or vehicles, and the abnormal behavior detection method in other scenes is similar to the abnormal behavior detection method in the traffic intersection scene in the embodiment of the present application, and will not be repeated.

Fig. 2 is a flow chart of an abnormal behavior detection method according to an embodiment of the present application, where the method may be executed by an abnormal behavior detection device, and the device may be implemented in a software and/or hardware manner, and the abnormal behavior detection method is described below with a terminal device as an execution body, and as shown in fig. 2, the abnormal behavior detection method according to the embodiment of the present application may include:

step S101: and acquiring data to be identified.

The data to be identified can comprise an image to be identified or a video to be identified, the image to be identified or the video to be identified can be obtained through a camera of the monitoring equipment, and the embodiment of the application does not limit the type, the type and the like of the camera. The data to be identified comprises a monitoring area, and the method can be used for detecting abnormal behaviors in the monitoring area, wherein the targets in the data to be identified can comprise vehicles, the abnormal behaviors can comprise abnormal behaviors of the vehicles, the targets in the data to be identified can also comprise pedestrians, and the abnormal behaviors can comprise abnormal behaviors of the pedestrians or abnormal behaviors of the vehicles, and the method is not limited in this respect.

Step S102: and determining an identification frame of the vehicle through an identification frame detection module of the key point detection network model, and detecting key points of the vehicle in the identification frame of the vehicle by utilizing the key point detection module of the key point detection network model.

And after the data to be identified is acquired, detecting the key points of the vehicles in the data to be identified through the key point detection network model. The embodiment of the application does not limit the network structure of the key point detection network model, so long as the key point detection of the target in the data to be identified can be realized. In one possible implementation manner, the keypoint detection network model may include an identification frame detection module and a keypoint detection module, and performing keypoint detection on the target in the data to be identified through the keypoint detection network model to obtain a keypoint of the target may include: and determining an identification frame of the vehicle through an identification frame detection module of the key point detection network model, and detecting key points of the vehicle in the identification frame of the vehicle by utilizing the key point detection module of the key point detection network model.

According to the embodiment of the application, the key point detection network module can obtain the key point of the target, the identification frame of the target can be obtained, the subsequent detection of the behavior of the target by combining the key point and the target frame is facilitated, and the reliability of abnormal behavior detection can be further improved.

The key point detection network model can directly adopt the key point detection network model trained by other terminal equipment or servers, and can also be trained by the terminal equipment. In a possible implementation manner, before acquiring the data to be identified, the method for detecting abnormal behavior provided by the embodiment of the application may further include:

The traffic data can be acquired through a camera or monitoring equipment, the traffic data can comprise training images or training videos, and then key point labeling is carried out on the traffic data to generate the traffic data sample. And inputting the training data sample into the network structure of the constructed key point detection network model, and finally training the training data sample by utilizing the network structure of the key point detection network model to generate a trained key point detection network model. The embodiment of the present application is merely taken as an example, and is not limited thereto.

For the key points of the targets, the number, positions, etc. of the key points required for the different targets may be different, and in a possible implementation, fig. 3 is a schematic diagram of the key points of the vehicle provided by an embodiment of the present application, and fig. 4 is a schematic diagram of the key points of the pedestrian provided by an embodiment of the present application. As shown in fig. 3, the key points of the vehicle may include a roof front left key point 1, a roof front right key point 2, a front lamp left key point 3, a front lamp right key point 4, a body front left key point 5, a body front right key point 6, a left side mirror key point 7, a right side mirror key point 8 (not shown), a roof rear left key point 9, a roof rear right key point 10, a rear lamp left key point 11 (not shown), a rear Fang Chedeng right key point 12 (not shown), a body rear left key point 13, and a body right key point 14 (not shown). The embodiment of the present application is merely taken as an example, and the number and the positions of the key points of the vehicle are not limited thereto, and for example, the key points may be provided at four wheels of the vehicle. As shown in fig. 4, key points of a pedestrian may include left eye 1, right eye 2, left ear 3, right ear 4, left shoulder 5, right shoulder 6, left elbow 7, right elbow 8, left hand 9, right hand 10, left hip 11, right hip 12, left knee 13, right knee 14, left foot 15, right foot 16, nose 17. The embodiment of the present application is merely taken as an example, and the number and the positions of the key points to the pedestrian are not limited thereto.

Step S103: and carrying out key point tracking on key points of the vehicle, and determining the motion trail of the vehicle.

Step S104: and determining whether the vehicle has abnormal behaviors according to the movement track of the vehicle and the area where the vehicle is located.

After determining the keypoints of the vehicle, in one possible implementation, it may be determined whether the vehicle has abnormal behavior based on the keypoints of the vehicle and the region in which the vehicle is located. The behavior, action, etc. of the vehicle can be determined by the key points of the vehicle. The embodiment of the application does not limit the specific implementation mode of determining whether the vehicle has abnormal behaviors according to the key points of the vehicle and the region where the vehicle is located. In different areas, normal behaviors of pedestrians or vehicles may be inconsistent, whether the current behavior of the vehicles is normal in the area where the vehicles are located can be judged through key points of the vehicles and the area where the vehicles are located, if not, abnormal behaviors of the vehicles are determined, and if not, abnormal behaviors of the vehicles are determined. For example, by acquiring the position of the camera of the data to be identified and the detected key points of the vehicle, determining the running direction of the vehicle and the area where the vehicle is located, if the running direction of the vehicle is inconsistent with the current running direction of the vehicle in the area, the vehicle may have a retrograde behavior.

In another possible embodiment, determining whether the vehicle has abnormal behavior according to the key points of the vehicle and the area where the vehicle is located may be implemented through step S103 and step S104.

After determining the key points of the vehicle, the key points of the vehicle can be tracked, and the specific implementation manner of the key point tracking of the key points of the vehicle is not limited as long as the movement track of the vehicle can be determined. For example, keypoint tracking may be implemented by recurrent neural networks, to name a few embodiments of the present application.

After determining the movement track of the vehicle, it may be determined whether there is abnormal behavior of the vehicle according to the movement track of the vehicle and the region in which the vehicle is located. For different vehicles and different application scenes, abnormal behaviors in the area where the vehicles are located may be different, and specifically, the abnormal behaviors can be set according to the different application scenes, so that judgment of the vehicles on the abnormal behaviors is determined according to the motion track of the vehicles and the area where the vehicles are located.

The following describes a determination of an abnormal behavior that may exist in the vehicle, which is merely taken as an example, and the embodiment of the present application is not limited thereto.

For convenience of description, fig. 5 is a schematic diagram of a traffic intersection provided by an embodiment of the present application, as shown in fig. 5, where the traffic intersection shown in fig. 5 includes four roads, and the embodiment of the present application only uses road 1 and road 2 as an example, where road 1 and road 2 include two lane areas, respectively, road 1 includes lane area 1 and lane area 4, road 2 includes lane area 2 and lane area 3, and an arrow in the lane area indicates a specified driving direction of the lane area. If the vehicle is in the lane area 1 and the movement track of the vehicle is different from the prescribed driving direction of the lane area 1, for example, the driving direction is opposite or the vehicle turns around, the vehicle is determined to have a reverse driving behavior.

The application is described by taking a traffic signal lamp as an example, the traffic signal lamp is used for indicating a current passable lane, for example, when the current traffic signal lamp indicates that a road 1 and a road 2 can pass, the current passable lane area is a lane area 1, a lane area 2, a lane area 3 and a lane area 4, if at the moment, when the movement track of a vehicle in other lanes changes and passes through a central crossing area, the behavior of the vehicle running a red light is determined, and the embodiment of the application takes only this example, wherein the central crossing area is a road area surrounded by four crosswalk areas in fig. 5.

The central intersection area is an important area of a road, vehicles without special conditions cannot stay for a long time, otherwise traffic jam can be caused, and therefore if the vehicles are detected to be in the central intersection area and the movement track is not changed within the second preset time, the behavior that the vehicles stay at the intersection is determined.

The vehicle is in a road area, a central intersection area and a crosswalk area, and a non-road area is also included, if the vehicle stays in the non-road area for a long time, the vehicle is likely to break down, so that the vehicle can be determined to have the behavior of exiting the road surface when the vehicle is in the non-road area and the movement track of the vehicle is unchanged within a first preset time. The embodiment of the present application is merely taken as an example, and is not limited thereto.

If abnormal behaviors of the target are detected, the target needs to be reminded or timely processed, for example, pedestrians or vehicles violate traffic rules, pedestrians can be reminded, and if traffic accidents or sudden unexpected accidents of the pedestrians occur, traffic departments, medical departments and the like can be informed of timely processing. In order to solve the above technical problem, in a possible implementation manner, the method for detecting abnormal behavior provided by the embodiment of the present application may further include:

if the abnormal behavior exists in the vehicle, alarm information is pushed to the user terminal, wherein the alarm information comprises the abnormal behavior of the vehicle and the position of the vehicle.

The embodiment of the application does not limit the specific implementation way of pushing the alarm information to the user terminal, for example, an audio player or a video player can be arranged at a traffic intersection, and the behavior violating the traffic rule can be played through the audio player or the behavior violating the traffic rule can be played through the video player. For another example, the alarm information can be pushed to a terminal device of a traffic department to monitor the traffic intersection. The embodiment of the application does not limit the specific information included in the alarm information, for example, the alarm information can include abnormal behavior of the vehicle and the position of the vehicle. For example, at the intersection of road a and road B, there is a collision between vehicle a and vehicle B, and the embodiment of the present application is merely an example and is not limited thereto. In the embodiment of the application, the abnormal behavior of the vehicle and the position of the vehicle are pushed to the user terminal, so that the vehicle can be processed timely and accurately by the user.

In the embodiment of the application, the key point detection is carried out on the vehicle in the data to be identified through the key point detection network model, the motion trail of the vehicle is determined according to the key points of the vehicle, and whether the abnormal behavior exists in the vehicle is determined by utilizing the motion trail of the vehicle and the area where the vehicle is positioned, so that the method and the device are suitable for various behavior scenes, can detect various abnormal behaviors, can adapt to different surrounding environments, and have good robustness in abnormal behavior detection. And by using the key point tracking and track calculation of the target, compared with the traditional method for detecting the target frame, the method can acquire more detailed and more accurate information such as the position, the behavior and the like of the target based on the key point detection, is more convenient and accurate for detecting the behavior of the target, not only improves the precision, but also simplifies the flow.

In a possible implementation manner, the target may also be a pedestrian, and the following description will take the target including the pedestrian as an example, and the determination of the abnormal behavior that may exist in the pedestrian is given only by way of example, but the embodiment of the present application is not limited to this.

Taking a traffic intersection as an example, for example, when a pedestrian lights up a red signal of a traffic signal corresponding to a pavement area, the pedestrian should stand on a non-lane area to wait until a green signal of the traffic signal lights up, and then pass through the pavement area. If the pedestrian is detected to pass through the pavement area or stay in the pavement area when the red signal lamp of the traffic signal lamp corresponding to the pavement area is on, the abnormal behavior of the pedestrian can be determined, and the abnormal behavior is red light running. For another example, the behavior of the pedestrian at the traffic intersection is usually standing and walking, and if the behavior of the pedestrian lying down, squatting and sitting up is detected in the pavement area, it can be determined that the pedestrian has abnormal behavior, and the abnormal behavior may be the situation of collision of the pedestrian, sudden diseases of the pedestrian and the like, and needs to be dealt with in time. The embodiment of the present application is merely taken as an example, and is not limited thereto.

In yet another possible embodiment, determining whether the pedestrian has abnormal behavior according to the motion trail of the pedestrian and the area where the pedestrian is located includes:

As shown in fig. 5, the traffic intersection comprises four sidewalks, in the embodiment of the application, only the sidewalk 1 and the sidewalk 2 are taken as examples, the area where the sidewalk 1 and the sidewalk 2 are located is a sidewalk area, the sidewalk area is provided with respective traffic signal lamps, the pedestrians pass through or wait by observing the traffic signal lamps corresponding to the sidewalk, and if the traffic signal lamps in the sidewalk area where the pedestrians are located are red lamps and the movement track of the pedestrians passes through the sidewalk area, the behavior that the pedestrians break the red lamps is determined.

On the basis of the foregoing embodiment, taking the example that the target includes pedestrians and vehicles as an example, in a possible implementation manner, fig. 6 is a schematic flow chart of an abnormal behavior detection method provided by another embodiment of the present application, as shown in fig. 6, the abnormal behavior detection method provided by the embodiment of the present application may further include:

step S201: and determining the identification frame of the pedestrian and the key points of the pedestrian through the key point detection network model.

Step S202: and inputting the identification frame of the pedestrian and the key points of the pedestrian into the behavior discrimination network model to determine the human body posture of the pedestrian.

Step S203: and determining whether the vehicle has abnormal behaviors according to the human body posture, the area where the pedestrian is located and the recognition frame of the vehicle.

The network model is detected through the key points, and can be used for determining a recognition frame of the pedestrian and the key points of the pedestrian, and the behavior discrimination network model is used for recognizing the human body posture of the pedestrian, wherein the human body posture comprises, but is not limited to, an upright posture, a bending posture, a lying posture, a squatting posture, a leaning posture and the like. Before the key points of the pedestrians and the identification boxes of the pedestrians are input into the behavior discrimination network model, the embodiment of the application can further comprise: the process of constructing and training the network structure of the behavior discrimination network model is not limited in the embodiment of the application, and is not repeated. The human body posture can be directly obtained through the behavior discrimination network model, and the database and the subsequent logic judgment are not required to be set in advance, so that the method is more accurate and rapid.

After the human body posture of the pedestrian is determined through the behavior discrimination network model, in one possible implementation manner, whether the pedestrian has abnormal behavior or not can be determined according to the human body posture and the area where the pedestrian is located, and in another possible implementation manner, whether the vehicle has abnormal behavior or not can also be determined according to the human body posture, the area where the pedestrian is located and the recognition frame of the vehicle.

The embodiment of the application does not limit the specific implementation mode of determining whether the pedestrian has abnormal behaviors according to the human body posture and the area where the pedestrian is located. For example, if a pedestrian is detected to have a behavior other than the corresponding behavior of the area, it may be determined that the pedestrian has an abnormal behavior, which is merely an example and not limited thereto.

In one possible embodiment, determining whether a pedestrian has abnormal behavior based on the body posture and the area in which the pedestrian is located includes: if the area where the pedestrian is located is any one of a lane area, a central intersection area or a crosswalk area and the gesture of the pedestrian is a first preset gesture, determining that the pedestrian has abnormal behaviors, wherein the first preset gesture comprises a lying gesture or a squatting gesture.

Referring to fig. 5, if the area where the pedestrian is located is a lane area, a central intersection area or a crosswalk area, the normal behavior of the pedestrian may be upright walking, static, low-head, etc., and if the posture of the pedestrian is abnormal in the lying posture or squatting posture, it may indicate that the pedestrian has an accident or accident, and needs to be dealt with in time. The first preset posture may include a lying posture or a squatting posture, and may be other postures.

In the embodiment of the application, whether the pedestrian has abnormal behaviors or not is judged through the human body posture of the pedestrian and the area where the pedestrian is located, so that the detection of the abnormal behaviors of the pedestrian is realized, and the detection accuracy is improved.

In another possible embodiment, determining whether the vehicle has abnormal behavior according to the human body posture, the area where the pedestrian is located, and the recognition frame of the vehicle may include:

If it is detected that the overlapping time of the identification frames of the plurality of second vehicles is long, traffic accidents such as friction and collision may occur in the plurality of second vehicles, and traffic jams may occur, so that the preceding vehicle is stopped. By combining a preset range outside the identification frame of the second vehicle, for example, the identification frame is expanded by 1.2 times, the size of the preset range is not limited in the embodiment of the present application. If a pedestrian of the second preset posture is detected within a preset range outside the recognition frame of the second vehicle, it is determined that a plurality of second vehicles have collision behaviors. The specific posture of the second preset posture is not limited, for example, the second preset posture includes any one or more of a combination of an upright posture, a lying posture, and a squatting posture, which is only taken as an example and is not limited thereto.

On the basis of the foregoing embodiment, in one possible implementation manner, before acquiring the data to be identified, the method further includes:

The data to be calibrated can be a video to be calibrated or an image to be calibrated, and the data to be calibrated is obtained through the camera after the camera is installed and fixed. The requirements for the data to be calibrated may be different for different application scenarios, for example, for traffic intersection scenarios, the data to be calibrated may need to include traffic data when the traffic signal is changed multiple times. The number of regions and the types of regions to be divided into the picture regions in the data to be calibrated may also be different for different application scenes, which is not limited in the embodiment of the present application.

Taking the data to be calibrated as traffic data as an example, in a possible implementation manner, calibrating the picture area in the data to be calibrated includes:

As shown in fig. 5, lane lines, crosswalk lines and traffic lights in the data to be calibrated are detected, and then each lane in the picture area can be divided into lane areas, each crosswalk is divided into crosswalk areas and a central intersection area and a non-road area are determined through corner points of the lane lines and recognition frames of the crosswalk lines. After the lane area and the crosswalk area are determined, the specified driving direction of the lane area and the traffic signal lamp of the crosswalk area can be determined according to the driving direction of the vehicle and the track of the pedestrian in the data to be calibrated.

In the embodiment of the application, the movement track of the target is determined by carrying out key point tracking on the key points of the target, and whether the target has abnormal behavior or not is determined according to the movement track of the target and the area where the target is positioned.

In order to realize the identification of the identification frame, the category and the key point of the target, in a possible implementation manner, fig. 7 is a schematic diagram of a network structure of a key point detection network model according to an embodiment of the present application, and as shown in fig. 7, the network structure of the key point detection network model includes a two-stage detection network module, an identification frame detection module and a key point detection module; the output of the two-stage detection network module is respectively connected with the input of the identification frame detection module and the input of the key point detection module, the output of the identification frame detection module is connected with the input of the key point detection module, and the identification frame detection module sequentially comprises a box-pooler module, a box-head module, a box-classes module and a box-boxes module; the key point detection module sequentially comprises keypoint-pooler module, keypoint-head module and keypoint-location module; the identification frame detection module is used for identifying the identification frame and the category of the target, and the key point detection module is used for detecting the key point of the target.

The training data sample is input into a two-stage detection network for feature recognition and feature extraction, then is input into an identification frame detection module, and classification of the target is realized through a box-pooler module, a box-head module and a box-classes module in the identification frame detection module so as to obtain the class of the target; and detecting the identification frame of the target through a box-pooler module, a box-head module and a box-boxes module in the identification frame detection module so as to obtain the identification frame of the target. The key points of the targets are obtained through processing of keypoint-pooler module, keypoint-head module and keypoint-location module in the key point detection module by inputting the identification frame of the targets, the types of the targets and the output results of the two-stage detection network to the key point detection module.

The following are examples of the apparatus of the present application that may be used to perform the method embodiments of the present application. For details not disclosed in the embodiments of the apparatus of the present application, please refer to the embodiments of the method of the present application.

Fig. 8 is a schematic structural diagram of an abnormal behavior detection apparatus according to an embodiment of the present application, where the apparatus may be implemented by software and/or hardware, for example, may be implemented by a terminal device, and as shown in fig. 8, the abnormal behavior detection apparatus according to an embodiment of the present application may include: an acquisition module 41, a detection module 42 and a determination module 43.

The acquiring module 41 is configured to acquire data to be identified, where the target in the data to be identified includes a vehicle.

The detection module 42 is configured to determine an identification frame of the vehicle through an identification frame detection module of the network model for detecting the key points, and detect the key points of the vehicle in the identification frame of the vehicle by using the key point detection module of the network model for detecting the key points.

A determining module 43, configured to perform key point tracking on key points of the vehicle, and determine a motion trail of the vehicle; and determining whether the vehicle has abnormal behaviors according to the movement track of the vehicle and the area where the vehicle is located.

In a possible implementation manner, the determining module 43 is specifically configured to:

In one possible implementation, the target includes a pedestrian, and the determining module 43 is further configured to:

In one possible implementation, the target includes a pedestrian, and the determining module 43 further includes: inputting key points of the pedestrians and identification frames of the pedestrians into a behavior discrimination network model to determine the human body posture of the pedestrians; and determining whether the pedestrian has abnormal behaviors according to the human body posture and the area where the pedestrian is located.

In a possible implementation manner, as shown in fig. 8, the device for detecting abnormal behavior provided in the embodiment of the present application further includes a calibration module 44.

The calibration module 44 is configured to obtain data to be calibrated, and calibrate a picture area in the data to be calibrated, so as to divide the picture area into different areas.

In one possible implementation, the calibration module 44 is specifically configured to:

In a possible implementation manner, as shown in fig. 8, the device for detecting abnormal behavior provided in the embodiment of the present application further includes a training module 45.

The training module 45 is specifically configured to obtain a training data sample; constructing a network structure of a key point detection network model; training the training data sample by utilizing the network structure of the key point detection network model to generate a trained key point detection network model.

The pushing module 46 is configured to push, to the user terminal, alarm information if the vehicle has abnormal behavior, where the alarm information includes the abnormal behavior of the vehicle and the position of the vehicle.

The embodiment of the apparatus provided by the present application is merely illustrative, and the module division in fig. 8 is merely a logic function division, and there may be other division manners in practical implementation. For example, multiple modules may be combined or may be integrated into another system. The coupling of the individual modules to each other may be achieved by means of interfaces which are typically electrical communication interfaces, but it is not excluded that they may be mechanical interfaces or other forms of interfaces. Thus, the modules illustrated as separate components may or may not be physically separate, may be located in one place, or may be distributed in different locations on the same or different devices.

Fig. 9 is a schematic structural diagram of an electronic device provided in an embodiment of the present application, where the electronic device may be a terminal device in the foregoing embodiment, and as shown in fig. 9, the electronic device includes:

a processor 61, a memory 62, a transceiver 63, and a computer program; wherein the transceiver 63 enables data transmission with other devices, a computer program is stored in the memory 62 and configured to be executed by the processor 61, the computer program comprising instructions for performing the above-mentioned abnormal behavior detection method, the content and effects of which refer to the method embodiments.

In addition, the embodiment of the application further provides a computer-readable storage medium, wherein computer-executable instructions are stored in the computer-readable storage medium, and when at least one processor of the user equipment executes the computer-executable instructions, the user equipment executes the various possible methods.

Among them, computer-readable media include computer storage media and communication media including any medium that facilitates transfer of a computer program from one place to another. A storage media may be any available media that can be accessed by a general purpose or special purpose computer. An exemplary storage medium is coupled to the processor such the processor can read information from, and write information to, the storage medium. In the alternative, the storage medium may be integral to the processor. The processor and the storage medium may reside in an ASIC. In addition, the ASIC may reside in a user device. The processor and the storage medium may reside as discrete components in a communication device.

Those of ordinary skill in the art will appreciate that: all or part of the steps for implementing the method embodiments described above may be performed by hardware associated with program instructions. The foregoing program may be stored in a computer readable storage medium. The program, when executed, performs steps including the method embodiments described above; and the aforementioned storage medium includes: various media that can store program code, such as ROM, RAM, magnetic or optical disks.

Finally, it should be noted that: the above embodiments are only for illustrating the technical solution of the present application, and not for limiting the same; although the application has been described in detail with reference to the foregoing embodiments, it will be understood by those of ordinary skill in the art that: the technical scheme described in the foregoing embodiments can be modified or some or all of the technical features thereof can be replaced by equivalents; such modifications and substitutions do not depart from the spirit of the application.

Claims

1. An abnormal behavior detection method, comprising:

Acquiring data to be identified, wherein a target in the data to be identified comprises a vehicle;

determining an identification frame of the vehicle through an identification frame detection module of a key point detection network model, and detecting key points of the vehicle in the identification frame of the vehicle by utilizing the key point detection module of the key point detection network model;

performing key point tracking on the key points of the vehicle to determine the motion trail of the vehicle;

determining whether abnormal behaviors exist in the vehicle according to the motion trail of the vehicle and the region where the vehicle is located;

The target comprises a pedestrian, the method further comprising:

Determining whether abnormal behaviors exist in the vehicle according to the human body posture, the area where the pedestrians are located and the recognition frame of the vehicle;

The determining whether the vehicle has abnormal behavior according to the human body posture, the area where the pedestrian is located and the recognition frame of the vehicle comprises:

2. The method according to claim 1, wherein before the determination of whether the vehicle has abnormal behavior based on the human body posture, the area in which the pedestrian is located, and the identification frame of the vehicle, the method further comprises:

Determining the identification frame of the pedestrian and the key points of the pedestrian through the key point detection network model;

and inputting the identification frame of the pedestrian and the key points of the pedestrian into a behavior discrimination network model to determine the human body posture of the pedestrian.

3. The method according to any one of claims 1-2, further comprising, prior to acquiring the data to be identified:

Acquiring data to be calibrated;

and calibrating the picture area in the data to be calibrated so as to divide the picture area into different areas.

4. A method according to claim 3, wherein said calibrating the picture area in the data to be calibrated comprises:

detecting lane lines, crosswalk lines and traffic signal lamps in the data to be calibrated;

Dividing the picture area into a lane area, a central intersection area, a crosswalk area and a non-road area according to the lane lines and the crosswalk lines;

Determining the running direction of the vehicle and the track of the pedestrian in the data to be calibrated;

determining the vehicle running direction exceeding a preset proportion in the lane area as the specified running direction of the lane area;

and when the pedestrian track changes in the pedestrian crossing area, determining a traffic signal lamp which is lighted by a green signal lamp as the traffic signal lamp of the pedestrian crossing area.

5. The method according to any one of claims 1-2, further comprising, prior to said obtaining the data to be identified:

Acquiring a training data sample;

Constructing a network structure of the key point detection network model;

and training the training data sample by utilizing the network structure of the key point detection network model to generate a trained key point detection network model.

6. The method of claim 5, wherein the network structure of the keypoint detection network model comprises a two-stage detection network module, an identification frame detection module, and a keypoint detection module;

The output of the two-stage detection network module is respectively connected with the input of the identification frame detection module and the input of the key point detection module, the output of the identification frame detection module is connected with the input of the key point detection module, and the identification frame detection module sequentially comprises a box-pooler module, a box-head module, a box-classes module and a box-boxes module; the key point detection module sequentially comprises keypoint-pooler module, keypoint-head module and keypoint-location module;

The identification frame detection module is used for identifying an identification frame of the target, and the key point detection module is used for detecting key points of the target.

7. The method according to any one of claims 1-2, further comprising:

If the abnormal behavior of the vehicle exists, pushing alarm information to a user terminal, wherein the alarm information comprises the abnormal behavior of the vehicle and the position of the vehicle.

8. An abnormal behavior detection apparatus, comprising:

the system comprises an acquisition module, a recognition module and a recognition module, wherein the acquisition module is used for acquiring data to be recognized, and targets in the data to be recognized comprise vehicles;

The detection module is used for determining the identification frame of the vehicle through the identification frame detection module of the key point detection network model, and detecting the key point of the vehicle in the identification frame of the vehicle by utilizing the key point detection module of the key point detection network model;

The determining module is used for carrying out key point tracking on key points of the vehicle and determining the motion trail of the vehicle; determining whether abnormal behaviors exist in the vehicle according to the motion trail of the vehicle and the region where the vehicle is located;

the target comprises a pedestrian, and the determining module is further used for determining whether the vehicle has abnormal behaviors according to the human body posture, the area where the pedestrian is located and the recognition frame of the vehicle;

The determining module is specifically configured to determine that a collision behavior exists in the second vehicle if it is detected that an overlapping time of identification frames of at least two second vehicles exceeds a third preset time and a pedestrian in a second preset posture is detected within a preset range outside the identification frames of the second vehicles, where the second preset posture includes any one or more combinations of an upright posture, a lying posture, or a squatting posture.

9. An electronic device, comprising:

at least one processor; and

The memory stores instructions executable by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-7.