CN112560547A

CN112560547A - Abnormal behavior judgment method and device, terminal and readable storage medium

Info

Publication number: CN112560547A
Application number: CN201910854597.9A
Authority: CN
Inventors: 尹力
Original assignee: ZTE Corp
Current assignee: ZTE Corp
Priority date: 2019-09-10
Filing date: 2019-09-10
Publication date: 2021-03-26
Also published as: WO2021047306A1

Abstract

The invention provides an abnormal behavior judging method, an abnormal behavior judging device, a terminal and a readable storage medium, wherein the abnormal behavior judging method is characterized in that behavior data in a monitoring video are obtained; determining at least one suspected abnormal behavior from the behavior data; acquiring a current state picture of an area where suspected abnormal behaviors are located; and judging whether the suspected abnormal behavior is the abnormal behavior according to the current state picture. The method provided by the embodiment of the invention comprises the steps of firstly preprocessing behavior data to determine suspected abnormal behaviors, then pointedly acquiring a current state picture aiming at an area where the suspected abnormal behaviors are located, more clearly and accurately shooting the suspected abnormal behaviors by the picture, and then judging whether the suspected abnormal behaviors are abnormal behaviors or not according to the current state picture. The invention also discloses an abnormal behavior judgment device, a terminal and a readable storage medium.

Description

Abnormal behavior judgment method and device, terminal and readable storage medium

Technical Field

The embodiment of the invention relates to the technical field of video monitoring, in particular to but not limited to a method, a device, a terminal and a readable storage medium for judging abnormal behaviors.

Background

With the rapid development of artificial intelligence technology in image processing and other aspects, the video monitoring technology based on artificial intelligence is widely used in many occasions. In many sensitive areas, such as meeting rooms, examination rooms, financial institutions, etc., more intelligent camera monitoring equipment is often needed to count personnel information and detect abnormal behavior. However, after acquiring image data from the front end, the current intelligent monitoring device directly transmits an original picture to the back-end GPU cluster for detection through the convolutional neural network, and a specific process thereof can be seen in fig. 1, as shown in fig. 1, in the current related art, a determination process thereof is as follows:

s101: the intelligent monitoring equipment acquires image data from the front end;

s102: the back-end GPU cluster receives the image data and detects the image data through a convolutional neural network;

s103: and outputting the result.

The method for judging has the following defects:

1, judging the abnormal behavior of the monitoring personnel inaccurately. Due to insufficient resolution of the surveillance camera; more people exist in the monitored area, and the pixels in the image are smaller and not clear enough; the abnormal behavior detection model based on deep learning often causes detection information errors and inaccurate judgment due to reasons such as poor single-step detection effect and the like.

2, the resource occupation is large, and the processing speed is low. In order to acquire clearer and more reliable data, a plurality of cameras need to be arranged; all the image data acquired by the plurality of cameras at the front end are sent to the high-definition image data at the back end continuously, and the back end is enabled to carry out information analysis and abnormal behavior detection. Therefore, not only is the complexity of field control increased, but also more space resources and hardware resources are needed, and the processing speed is low.

Disclosure of Invention

The abnormal behavior judgment method, the abnormal behavior judgment device, the terminal and the readable storage medium provided by the embodiment of the invention mainly solve the technical problems of inaccurate judgment on abnormal behaviors, large resource occupation and low processing speed in the prior art.

To solve the foregoing technical problem, an embodiment of the present invention provides an abnormal behavior determination method, including:

acquiring behavior data in a monitoring video;

determining at least one suspected abnormal behavior from the behavior data;

acquiring a current state image of an area where the suspected abnormal behavior is located;

and judging whether the suspected abnormal behavior is an abnormal behavior according to the current state image.

An embodiment of the present invention further provides an abnormal behavior determination apparatus, including: at least one video monitoring module and data processing module, wherein:

the video monitoring module includes:

the data acquisition module is used for acquiring behavior data in the monitoring video;

a data preliminary processing module for determining at least one suspected abnormal behavior from the behavior data;

the image acquisition module is used for acquiring a current state image of an area where the suspected abnormal behavior is located;

and the data processing module is used for judging whether the suspected abnormal behavior is an abnormal behavior according to the current state image.

The embodiment of the invention also provides an abnormal behavior judgment terminal, which comprises a processor, a memory and a communication bus;

the communication bus is used for realizing connection communication between the processor and the memory;

the processor is configured to execute one or more computer programs stored in the memory to implement the steps of any one of the above abnormal behavior determination methods.

Embodiments of the present invention also provide a readable storage medium, where one or more computer programs are stored, where the one or more computer programs are executable by one or more processors to implement the steps of any one of the above abnormal behavior determination methods.

The invention has the beneficial effects that:

the invention provides an abnormal behavior judging method, an abnormal behavior judging device, a terminal and a readable storage medium, wherein the abnormal behavior judging method is characterized in that behavior data in a monitoring video are obtained; determining at least one suspected abnormal behavior from the behavior data; acquiring a current state image of an area where suspected abnormal behaviors are located; and judging whether the suspected abnormal behavior is the abnormal behavior according to the current state image. The method provided by the embodiment of the invention determines the suspected abnormal behavior by preprocessing the behavior data, then pointedly acquires the current state image aiming at the area where the suspected abnormal behavior is located, the image can clearly and accurately shoot the suspected abnormal behavior, and then judges whether the suspected abnormal behavior is the abnormal behavior according to the current state image. By implementing the method and the device, the accuracy of judging the abnormal behavior is improved, the resource occupation is reduced, and the processing speed is improved.

Additional features and corresponding advantages of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention.

Drawings

Fig. 1 is a schematic flow chart of an abnormal behavior determination method in the background art of the present invention;

fig. 2 is a schematic flowchart of a method for determining abnormal behavior according to a first embodiment of the present invention;

fig. 3 is a schematic flowchart of an abnormal behavior determination method according to a first embodiment of the present invention;

fig. 4 is a structural diagram of an abnormal behavior determination apparatus according to a second embodiment of the present invention;

fig. 5 is a structural diagram of another abnormal behavior determination apparatus according to a second embodiment of the present invention;

fig. 6 is a structural diagram of another abnormal behavior determination apparatus according to a second embodiment of the present invention;

fig. 7 is a structural diagram of another abnormal behavior determination apparatus according to a second embodiment of the present invention;

fig. 8 is a schematic structural diagram of a terminal according to a third embodiment of the present invention.

Detailed Description

In order to make the objects, technical solutions and advantages of the present invention more apparent, embodiments of the present invention are described in detail below with reference to the accompanying drawings. It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.

The first embodiment is as follows:

referring to fig. 2, a method for determining an abnormal behavior according to the present embodiment includes:

s201: acquiring behavior data in a monitoring video;

s202: determining at least one suspected abnormal behavior from the behavior data;

s203: acquiring a current state image of an area where suspected abnormal behaviors are located;

s204: and judging whether the suspected abnormal behavior is the abnormal behavior according to the current state image.

It should be noted that the abnormal behavior in this embodiment may be some behaviors specific in some scenarios, for example: cheating in an examination room, taking a picture in a confidential meeting room, lighting fire or smoking in a dangerous area and the like; there may also be some specific actions such as: running, waving hands, clapping hands, etc.; but also facial micro-expressions of some characters, such as: frown, mouth, tongue, etc. The abnormal behavior can be determined by presetting rules, or can be determined by a user according to requirements.

In some embodiments, the monitoring area for monitoring the video may be a place such as a school, a hospital, a square, a conference room, a park, a scenic spot, etc., and the monitoring area is not limited in this embodiment.

In some embodiments, the behavior data in the surveillance video may be obtained in real time, or may be obtained at certain time intervals, for example, every 1 minute. The behavior data in the monitoring video can also be acquired in real time within a preset time period. For example, if the monitoring area is a conference room, it may be set that the behavior data in the real-time monitoring video is acquired in real time in the conference-on time period, and the acquisition is stopped after the conference is finished.

It should be noted that the behavior data may include the position of a specific subject, the appearance of a specific thing, motion data, light change, color change, and the like. For example, by acquiring a picture image of a certain frame or a certain number of frames or all frames in a monitoring video and analyzing the position of each preset object in the monitoring video as behavior data, for example, analyzing the position of a vehicle at a traffic intersection as behavior data, when the position of a certain vehicle exceeds a zebra crossing and a pedestrian exists on the zebra crossing, the behavior of the vehicle exceeding the zebra crossing can be determined as suspected abnormal behavior through the behavior data; acquiring and analyzing brightness data of a picture image in a monitoring video as behavior data, for example, acquiring brightness data of each area in the monitoring picture in a monitoring video of a kitchen, if the brightness of a certain non-cooking bench area exceeds a preset range, determining that the area has the possibility of suspected ignition, and determining that the behavior that the brightness of the non-cooking bench area exceeds the preset range is suspected abnormal behavior through the behavior data; for example, in a monitoring video of a square, if an image in which a plurality of persons move is captured, the state in which each person moves can be used as behavior data. For example, in a monitoring video screen of a smoke-forbidden place, objects existing in the screen are identified, each object is behavior data, and when suspected smoke or suspected lighter appears in the object, the suspected abnormal behavior exists in the screen. The specific behavior data acquisition may be set by those skilled in the art according to the characteristics of the abnormal behavior to be determined.

In some embodiments, after obtaining the current state image of the area where the suspected abnormal behavior is located, determining whether the suspected abnormal behavior is the abnormal behavior according to the current state image further includes: and carrying out normalization processing on the current state image.

It should be noted that, the normalizing process performed on the current state image includes, but is not limited to, adjusting the size of the current state image to a preset size, for example, the obtained current state image of the area where the abnormal behavior is located may be a picture, and the size of the obtained current state image may be 480 × 270 pixels, so that in order to make the subsequent processing on the current state image faster and occupy less resources, the size of the current state image may be adjusted to a uniform preset size of 240 × 135 pixels. In some embodiments, normalizing the current state image further includes unifying the format of the current state image, e.g., storing each as a JPG format, mp4 format, or the like. In some embodiments, normalizing the current-state image further includes adjusting parameters such as brightness and contrast of the current-state image.

In some embodiments, the current-state image is an image including a suspected abnormal behavior subject, and the image is a "close-up" of a suspected abnormal behavior, and in some embodiments, since acquiring the behavior data in the monitoring video may include a plurality of behavior execution subjects, after the behavior data of a certain behavior execution subject is determined as the suspected abnormal behavior, a single and clear shot of an area where the suspected abnormal behavior is located is performed to acquire the current-state image. In some embodiments, the current state image is a complete image including an abnormal behavior and an execution subject thereof, for example, in a conference room of a close conference, a suspected abnormal behavior that a participant uses a mobile phone to shoot a demonstration screen will be obtained as a current state image by using a certain frame or a certain number of frames or a certain section of video of a current monitoring video of an object shot by using the mobile phone; in some embodiments, the current state image is a complete image including an abnormal behavior, for example, after a suspected abnormal behavior of suspected smoking is found in a smoking ban, position information of a suspected smoking action is obtained, and the camera head and the camera device are adjusted according to the position information to perform "close-up" shooting on an area where the position information is located within a certain range, so as to obtain the current state image.

In some embodiments, determining whether the suspected abnormal behavior is an abnormal behavior according to the current state image further comprises:

if the suspected abnormal behavior is judged to be the abnormal behavior, outputting at least one of the following information:

abnormal behavior alert information;

and coordinate information corresponding to the abnormal behavior.

It should be noted that the abnormal behavior alert information may be an alert ring, or pop up a prompt box on some preset interface, or send the abnormal behavior information to a specified terminal or server, and the like, which is not limited in this embodiment.

It should be noted that the coordinate information corresponding to the abnormal behavior may be identification information with longitude and latitude as a unit, for example: north latitude 29.35 east longitude 106.33; information in units of places may be, for example: XX district XX road XX number XX city XX province XX; and coordinate rule information set by a user, such as an XX area, an XX meeting room X row X seats, an XX ward XX bed, an XX floor, an XX box and the like.

In some embodiments, determining whether the suspected abnormal behavior is an abnormal behavior from the current state image comprises:

acquiring a behavior category where the suspected abnormal behavior is;

acquiring a first convolution neural network corresponding to a behavior category where the suspected abnormal behavior is located;

and judging whether the suspected abnormal behavior is the abnormal behavior or not according to the current state image through a first convolutional neural network.

It should be noted that the corresponding relationship between the first convolutional neural network and the behavior category may be defined according to the requirement of the user, and is not limited in this embodiment. For example, the currently acquired suspected abnormal behavior runs in a classroom, the suspected abnormal behavior can be classified into a classroom class, a first convolutional neural network corresponding to the behavior class classroom class where the suspected abnormal behavior is located can be set as a classroom deep convolutional neural classification network, and then the current state image acquired corresponding to the suspected abnormal behavior is sent and transmitted to the classroom deep convolutional neural classification network, and the classroom deep convolutional neural classification network receives the current state image and then judges the current state image. In some embodiments, the first convolutional neural network comprises a deep convolutional neural classification network.

In some embodiments, the first convolutional neural network may include a plurality of behavior categories, each of which corresponds to a different one, and the plurality of behavior categories may also correspond to the same first convolutional neural network, where the number of the first convolutional neural networks is not limited herein, and a user may set the number as needed.

The first convolutional neural network in the embodiment of the invention is obtained by performing deep learning on the preset convolutional neural network on the abnormal behavior samples of the preset behavior types, which are manually marked. The first convolutional neural network can further judge whether the suspected abnormal behavior is the abnormal behavior through the acquired current state image.

In some embodiments, obtaining the current state image of the area in which the suspected abnormal behavior is located includes:

acquiring coordinate information of suspected abnormal behaviors;

and acquiring a current state image of an area corresponding to the coordinate information of the suspected abnormal behavior.

It should be noted that, at this time, the shooting area of the current state image may be the current state image obtained by taking the coordinate where the suspected abnormal behavior is located as the center and expanding a preset range to the periphery or the preset direction. For example: the coordinate information of the suspected abnormal behavior is obtained to be 1.2 meters high, 3 meters far away and 1 meter wide, and at the moment, the area targeted by the current state image can be in the range of 1-1.5 meters high, 3 meters far away and 0.8-1.3 meters wide.

In some embodiments, the current state image may be obtained by controlling and adjusting the camera pan and thus the camera angle of the camera device, and/or by controlling parameters of the camera device, such as focal length, horizontal resolution, signal-to-noise ratio, and the like. Sometimes, the coordinate position of the suspected abnormal behavior is exactly consistent with the angle of the current image pickup apparatus, and the current state image can be acquired only by adjusting the parameters of the image pickup apparatus. It should be noted that the image pickup apparatus includes, but is not limited to, an apparatus that can perform monitoring shooting, such as a video camera.

In some embodiments, after the coordinate information of the abnormal behavior is obtained, the camera pan-tilt can be controlled to adjust the elevation angle, the horizontal angle and the camera focal length by referring to the coordinate information, so that the camera is aligned to the area suspected of the abnormal behavior to obtain the targeted current state image. For example: and adjusting the elevation angle and the horizontal angle of the holder to align to the central point of the abnormal behavior area. And adjusting the focal length to enable the camera to receive the image taking the coordinate center point of the suspected abnormal behavior as the image center as the current state image.

In some embodiments, determining at least one suspected anomalous behavior from the behavior data comprises:

obtaining the confidence of the suspected abnormal behavior;

and if the confidence coefficient of the suspected abnormal behavior is larger than a preset threshold value, taking the suspected abnormal behavior as the target suspected abnormal behavior.

and acquiring a current state image of an area where the target suspected abnormal behavior is located.

The target suspected abnormal behaviors are screened out according to the confidence degrees in the suspected abnormal behaviors, and the current state image of the target suspected abnormal behaviors is obtained only by aiming at the target suspected abnormal behaviors, so that the efficiency of judging the abnormal behaviors can be further improved, and the resource occupation is reduced.

It should be noted that the rule of the confidence level calibration may be preset by a user according to a certain logic, or may be calibrated based on the confidence level rule specified by some calculation models.

In some embodiments, the confidence of the suspected abnormal behavior may be inferred by a general model algorithm such as logistic regression, decision tree, etc.

It should be noted that the preset threshold may be a uniform preset threshold set for all scenes, that is, for all behavior categories, or may be a corresponding preset threshold set according to different behavior categories, or may be a preset threshold adjusted by the user at any time as needed. For example: the confidence coefficient of a suspected abnormal behavior of a suspected dangerous article is 0.8, the suspected abnormal behavior is eliminated according to the initial preset threshold value of 0.85, the suspected abnormal behavior cannot enter the next step, and the suspected abnormal behavior is further analyzed. But the requirement of the current security level is increased, and the preset threshold value is set to be 0.79, then the suspected abnormal behavior is listed as the target suspected abnormal behavior.

It should be noted that, in some embodiments, a suspected abnormal behavior with a confidence level above a preset threshold is obtained as a target suspected abnormal behavior, that is, when the confidence level of the suspected abnormal behavior is equal to the preset threshold, the suspected abnormal behavior is also listed as the target suspected abnormal behavior. The setting of the target suspected abnormal behavior can be changed according to the needs of the user.

It should be noted that, when the step of determining the target suspected abnormal behavior from the suspected abnormal behaviors exists in the abnormal behavior determination method, the current state image of the area where the suspected abnormal behavior is obtained in the original method is immediately replaced with the current state image of the area where the target suspected abnormal behavior is obtained, and for those determined as the suspected abnormal behaviors but not the target suspected abnormal behavior, the current state image of the area where the target suspected abnormal behavior is not obtained is no longer obtained, so that the resource occupation of the system is reduced, and the processing speed is increased.

In some embodiments, when the number of target suspected abnormal behaviors exceeds 1,

the method for acquiring the current state image of the area where the target suspected abnormal behavior is located comprises the following steps:

obtaining a target suspected abnormal behavior sequence, wherein the target suspected abnormal behavior sequence is obtained by sequencing the confidence degrees of suspected abnormal behaviors corresponding to the target suspected abnormal behaviors according to a preset rule;

and acquiring a current state image of an area where the target suspected abnormal behaviors are located according to the sequence of the target suspected abnormal behavior sequence.

It should be noted that the confidence of the target suspected abnormal behavior is equal to the calibrated confidence of the suspected abnormal behavior corresponding to the target suspected abnormal behavior. For example, if the confidence level of the suspected abnormal behavior is 0.9 and the preset threshold is 0.8, if the confidence level of the suspected abnormal behavior is higher than the preset threshold, the suspected abnormal behavior is regarded as the target suspected abnormal behavior, and the confidence level of the target suspected abnormal behavior is equal to the confidence level of the suspected abnormal behavior and is equal to 0.9.

In some embodiments, the preset rule in the priority ranking of the target suspected abnormal behaviors according to the confidence corresponding to each target suspected abnormal behavior according to the preset rule may be that the target suspected abnormal behaviors are ranked from high to low according to the confidence, or the target suspected abnormal behaviors of each behavior category are ranked from high to low according to the confidence, and then the target suspected abnormal behaviors are ranked according to the priority of the behavior category, it should be noted that the order of the rank of the confidence and the rank of the category priority in the ranking of the target suspected abnormal behaviors including multiple behavior categories may be set according to the user requirement, and this embodiment is not limited. . For example: there are currently behavioral classes, class a: the confidence coefficient of the target suspected abnormal behavior A is 0.7, the confidence coefficient of the target suspected abnormal behavior B is 0.75, and the confidence coefficient of the target suspected abnormal behavior C is 0.79; behavior class b: the confidence of the target suspected abnormal behavior D is 0.7, the confidence of the target suspected abnormal behavior E is 0.75, and the confidence of the target suspected abnormal behavior F is 0.79. The preset rule is that the priority of the class A of the behavior class is higher than that of the class B of the behavior class, and the higher the confidence coefficient is, the higher the priority is. Then, the result of performing priority ranking on the above target suspected abnormal behaviors according to the preset rule is as follows: the target is suspected to be abnormal, C, B, A, F, E and D.

It should be noted that, in some embodiments, the preset rule may also perform priority ordering according to the confidence degree from low to high, so as to obtain the target suspected abnormal behavior sequence.

It should be noted that, according to the target suspected abnormal behavior sequence, the current state image of the area corresponding to the coordinate information of the target suspected abnormal behavior may be obtained by respectively controlling and adjusting at least one of the parameters of the camera pan-tilt and the camera device. It can be understood that, in some embodiments, at least one of the parameters of the camera pan-tilt and the camera device is controlled and adjusted from the coordinate information where the target suspected abnormal behavior with the higher priority is located according to the target suspected abnormal behavior sequence of each target suspected abnormal behavior to obtain the current state image of the area including the coordinate information where the corresponding target suspected abnormal behavior is located. For example, if the rank of the target suspected abnormal behavior a in the target suspected abnormal behavior sequence is higher than the priority of the target suspected abnormal behavior B, at least one of the parameters of the camera pan-tilt and the camera device is controlled and adjusted to obtain the current state image of the area including the coordinate information of the target suspected abnormal behavior a, and then at least one of the parameters of the camera pan-tilt and the camera device is controlled and adjusted to obtain the current state image of the area including the coordinate information of the target suspected abnormal behavior B.

In some embodiments, after the position information of the suspected abnormal behavior of the target is acquired, the position information is compared with the current position of the camera, and the elevation angle and the horizontal angle of the holder are adjusted and adjusted to be aligned with the central point of the abnormal behavior area. And adjusting the focal length to enable the camera to receive the image taking the coordinate center point of the suspected abnormal behavior as the image center to acquire the current state image. When a plurality of target suspected abnormal behaviors exist currently, the steps can be repeated according to the sequence of the target suspected abnormal behaviors of each target suspected abnormal behavior until the current state images of all the target suspected abnormal behaviors are obtained.

In some embodiments, determining at least one suspected anomalous behavior from the behavior data may be determined by a second convolutional neural network, and in some embodiments, the calculations may be performed directly on a camera-loaded GPU by a second convolutional neural network disposed on the front-end camera. In some embodiments, the second convolutional neural network is a small convolutional neural network.

It is noted that in some embodiments, the second convolutional neural network may determine suspected abnormal behavior for all of the behavior classes.

In some embodiments, the second convolutional neural network is obtained by establishing a full-class abnormal behavior sample after manual labeling, learning the sample by a preset full-class convolutional neural network, performing confidence calibration training, and taking the convolutional neural network obtained after learning and training as the second convolutional neural network. The second convolutional neural network can perform calculation analysis on the acquired behavior data, screen out suspected abnormal behaviors from the behavior data, and calibrate the confidence coefficient of the suspected abnormal behaviors to obtain the position information of the suspected abnormal behaviors.

The behavior data in the present embodiment may be human behavior data, animal behavior data, or something having an action such as some mechanical device. For example, behavior data in a target monitoring range in the protected area may be obtained, wherein the behavior data includes that one panda climbs a tree and another koala stays under the tree. At the moment, the fact that the panda climbs the tree and stays under the tree is determined to be a suspected abnormal behavior A through a small convolutional neural network, and the fact that the panda stays under the tree is determined to be a suspected abnormal behavior B through the koala. For example, when a certain conveyor belt in a plant is normally operated at a speed of 5 m/sec, but is stationary when the state of the conveyor belt is extracted from behavior data in a monitoring video, the conveyor belt can be determined to be stationary as a suspected abnormal behavior.

The abnormal behavior determination method in the above embodiment is further described below with reference to a specific embodiment. Referring to fig. 3, as shown in fig. 3, there is provided a method for determining an abnormal behavior of a photographing screen in a conference room, the method including:

s301: acquiring behavior data in a monitoring video aiming at a conference room;

s302: determining at least two suspected shooting screen behaviors from the behavior data through a second convolutional neural network;

s303: performing confidence degree calibration on the suspected shooting screen behavior through a second convolutional neural network to obtain the confidence degree of the suspected shooting screen behavior;

s304: acquiring at least two suspected shooting screen behaviors of a target;

s305: sequencing the suspected shooting screen behaviors of the targets according to the confidence degrees of the suspected shooting screen behaviors of the targets to obtain a suspected shooting screen behavior sequence of the targets;

s306: acquiring coordinate information of each suspected abnormal target behavior;

s307: controlling and adjusting parameters of a camera holder and/or a camera device to acquire a current state image of an area corresponding to coordinate information of the camera holder and/or the camera device according to the sequence of behavior sequences of a target suspected shooting screen;

s308, normalizing the current state image;

s309: acquiring a behavior category where a target suspected shooting screen behavior is located;

s310: acquiring a first convolutional neural network corresponding to the behavior category where the suspected shooting screen behavior is located, and judging whether the suspected shooting screen behavior is the shooting screen behavior or not according to the current state image through the first convolutional neural network;

s311: if yes, outputting shooting behavior alarm information;

s312: outputting coordinate information corresponding to the shooting behavior;

s313: if not, the flow is ended.

It should be noted that, in the above embodiment, there is no order limitation between step S311 and step S312, and at least one step may be selected to be executed.

It should be noted that the target suspected shooting screen behavior includes a suspected shooting screen behavior whose confidence level of the suspected shooting screen behavior is higher than a preset threshold, and the confidence level of the target suspected shooting screen behavior is equal to the confidence level of the corresponding suspected shooting screen behavior.

It should be noted that the behavior data may be each frame in the surveillance video. Determining suspected abnormal behavior may be performed by a small convolutional neural network.

In some embodiments, the behavior data in the surveillance video may be acquired through one set of the pan/tilt and the shooting device, or through multiple sets of the pan/tilt and the shooting device, when multiple sets of behavior data are acquired through multiple sets of the pan/tilt and the shooting device, and when suspected abnormal behavior data are acquired from multiple sets of behavior data, a current state image corresponding to the suspected abnormal behavior may be acquired through the shooting device corresponding to the suspected abnormal behavior, or a current state image corresponding to the suspected abnormal behavior may be acquired through other shooting devices capable of acquiring an area where the suspected abnormal behavior is located. For example, four photographing devices A, B, C, D exist in four directions of a classroom, behavior data of a first group of seat positions in the classroom is acquired through the photographing device a, behavior data of a second group of seat positions in the classroom is acquired through the photographing device B, behavior data of a third group of seat positions in the classroom is acquired through the photographing device C, behavior data of a fourth group of seat positions in the classroom is acquired through the photographing device D, the suspected abnormal behavior of the suspected mobile phone playing in the classroom is considered to be included in the behavior data of the second group of seat positions in the classroom after calculation and analysis by the second convolutional neural network, and at this time, a current state image of an area where the suspected abnormal behavior of the suspected mobile phone playing in the classroom is located needs to be acquired to further confirm whether the behavior of the mobile phone playing in the classroom is the suspected abnormal behavior. For the current state image, more accurate position information of the behavior of suspected to play the mobile phone in class can be obtained, and a second group of third rows is assumed, at this time, parameters such as the shooting angle of at least one shooting device in the four shooting devices can be selected to be adjusted to obtain a "close-up" of the second group of third rows, for example, only shooting the second group of third rows of areas to obtain at least one current state image.

According to the abnormal behavior judgment method provided by the embodiment of the invention, the behavior data in the monitoring video is obtained; determining at least one suspected abnormal behavior from the behavior data; acquiring a current state image of an area where suspected abnormal behaviors are located; and receiving the current state image and judging whether the suspected abnormal behavior is the abnormal behavior. The method provided by the embodiment of the invention has the advantages that the behavior data is preprocessed, the suspected abnormal behavior is determined, the current state image is obtained in a targeted manner aiming at the area where the suspected abnormal behavior is located, the image can be used for shooting the suspected abnormal behavior more clearly and accurately, and whether the suspected abnormal behavior is the abnormal behavior is judged according to the current state image. By implementing the method, the accuracy of judging the abnormal behavior can be improved, the resource occupation is reduced, and the processing speed is improved.

Example two:

the present embodiment further provides an abnormal behavior determination apparatus, as shown in fig. 4, the abnormal behavior determination apparatus 400 includes at least one video monitoring module 401, and a data processing module 402, where:

the video monitoring module 401 includes:

the data acquisition module 4011 is configured to acquire behavior data in the monitoring video;

a data preliminary processing module 4012 configured to determine at least one suspected abnormal behavior from the behavior data;

an image obtaining module 4013, configured to obtain a current state image of an area where a suspected abnormal behavior is located

And the data processing module 402 is configured to determine whether the suspected abnormal behavior is an abnormal behavior according to the current state image.

In some embodiments, the behavior data in the surveillance video may be obtained in real time, or may be obtained at certain time intervals, for example, every 1 minute. The behavior data in the monitoring video can also be acquired within a preset time period. For example, if the monitoring area is a conference room, it may be set that the behavior data in the monitoring video is acquired during the conference-on period, and the acquisition is stopped after the conference is finished.

In some embodiments, as shown in fig. 5, the abnormal behavior determination apparatus further includes a control data transmission module 501, where the control data transmission module 501 is configured to perform normalization processing on the current state image.

It should be noted that, the normalizing process performed on the current state image includes, but is not limited to, adjusting the size of the current state image to a preset size, for example, the current state image of the area where the abnormal behavior is located, which is acquired for the first time, may be a picture, and the size of the current state image may be 480 × 270 pixels, so that in order to make the subsequent processing of the current state image faster and occupy less resources, the size of the current state image may be adjusted to a uniform preset size of 240 × 135 pixels. In some embodiments, normalizing the current state image further includes unifying the format of the current state image, e.g., storing each as a JPG format, mp4 format, or the like. In some embodiments, normalizing the current-state image further includes adjusting parameters such as brightness and contrast of the current-state image.

In some embodiments, the current-state image is an image including a suspected abnormal behavior subject, and the image is a "close-up" of a suspected abnormal behavior, and in some embodiments, since acquiring the behavior data in the monitoring video may include a plurality of behavior execution subjects, after the behavior data of a certain behavior execution subject is determined as the suspected abnormal behavior, a single and clear shot of an area where the suspected abnormal behavior is located is performed to acquire the current-state image. For example, the behavior data acquired in the surveillance video of the classroom includes the motion data of 10 students in the classroom, wherein the congress on the left side of the third row of the second group is in a standing state, if the abnormal behavior data in the scene is a standing behavior, the behavior of the congress is judged to be a suspected abnormal behavior, and only the left area of the third row of the second group is shot to acquire the current state image of the left area of the third row of the second group.

In some embodiments, the current state image is a complete image including an abnormal behavior and an execution subject thereof, for example, in a conference room of a close conference, a suspected abnormal behavior that a participant uses a mobile phone to shoot a demonstration screen will be obtained as a current state image by using a certain frame or a certain number of frames or a certain section of video of a current monitoring video of an object shot by using the mobile phone; in some embodiments, the current state image is a complete image including abnormal behaviors, for example, position information for finding suspected smoking in a smoking ban, and the camera head and the camera device are adjusted to perform close-up shooting on a certain range of an area where the position information is located according to the position information to obtain the current state image.

In some embodiments, the data processing module is further configured to:

abnormal behavior alert information;

and coordinate information corresponding to the abnormal behavior.

In some embodiments, the data processing module further comprises at least one first convolutional neural network; the data preliminary processing module is also used for acquiring the behavior category of the suspected abnormal behavior;

the data processing module is further used for acquiring a first convolutional neural network corresponding to the behavior type where the suspected abnormal behavior is located, and judging whether the suspected abnormal behavior is the abnormal behavior according to the current state image through the first convolutional neural network.

It should be noted that the first convolutional neural network includes a deep convolutional neural classification network, and the classification of the first convolutional neural network may be defined according to the requirement of the user, which is not limited in this embodiment. For example, the currently acquired suspected abnormal behavior runs in a classroom, the suspected abnormal behavior can be classified into a classroom class, a deep convolutional neural classification network corresponding to the behavior class classroom class where the suspected abnormal behavior is located can be set as a classroom deep convolutional neural classification network, and then the current state image acquired corresponding to the suspected abnormal behavior is sent and transmitted to the classroom deep convolutional neural classification network, and the classroom deep convolutional neural classification network receives the current state image and then judges the current state image.

The first convolutional neural network in the embodiment of the invention is obtained by performing deep learning on the preset convolutional neural network on the abnormal behavior samples of the preset behavior types, which are manually marked. Whether the suspected abnormal behavior is the abnormal behavior can be further judged according to the obtained current state image through the first convolution neural network.

In some embodiments, the data preliminary processing module 4012 is further configured to obtain a confidence level of the suspected abnormal behavior;

as shown in fig. 6, the abnormal behavior determination apparatus 400 further includes a threshold determination module 601, where the threshold determination module 601 is further configured to take the suspected abnormal behavior as a target suspected abnormal behavior if the confidence of the suspected abnormal behavior is greater than a preset threshold.

In some embodiments, after the threshold determining module 601 determines the target suspected abnormal behavior, the image obtaining module 4013 is configured to obtain a current state image of an area where the target suspected abnormal behavior is located.

At this time, the current state image can be acquired only for the target suspected abnormal behavior with the confidence coefficient reaching a certain standard, and for the suspected abnormal behavior with the confidence coefficient not reaching the standard, subsequent judgment is not performed, so that the resource occupation is saved, and the judgment efficiency is improved.

In some embodiments, the current status image of the area including the coordinate information may be obtained by controlling the adjustment of the camera pan and thus the camera angle of the camera device, and/or by controlling parameters of the camera device, such as focal length, horizontal resolution, signal-to-noise ratio, etc. Sometimes, the coordinate position of the suspected abnormal behavior is exactly consistent with the angle of the current image pickup apparatus, and the current state image can be acquired only by adjusting the parameters of the image pickup apparatus. It should be noted that the image pickup apparatus includes, but is not limited to, an apparatus that can perform monitoring shooting, such as a video camera.

In some embodiments, as shown in fig. 6, the abnormal behavior determination apparatus further includes a sorting module 602, where the sorting module 602 is configured to, when the number of the target suspected abnormal behaviors exceeds 1, obtain a target suspected abnormal behavior sequence, where the target suspected abnormal behavior sequence is obtained by sorting confidence levels of suspected abnormal behaviors corresponding to the target suspected abnormal behaviors according to a preset rule; the image obtaining module 4013 obtains the current state image of the area where the target suspected abnormal behavior is located according to the sequence of the target suspected abnormal behavior.

It should be noted that the rule of the confidence level calibration may be preset by a user according to a certain logic, or may be calibrated based on the confidence level rule specified by some calculation models. Those skilled in the art can use available confidence calculation methods to calculate the confidence of the suspected abnormal behavior.

In some embodiments, after the position information of the suspected abnormal behavior of the target is acquired, the position information is compared with the current position of the camera, and the elevation angle and the horizontal angle of the holder are adjusted and adjusted to be aligned with the central point of the abnormal behavior area. And adjusting the focal length to enable the camera to receive the image taking the coordinate center point of the suspected abnormal behavior as the image center to acquire the current state image. When a plurality of target suspected abnormal behaviors exist currently, the steps can be repeated according to the sequence of the target suspected abnormal behaviors of each target until the current state images of all the target suspected abnormal behaviors are obtained.

In some embodiments, the data pre-processing module comprises a second convolutional neural network, which in some embodiments comprises a small convolutional neural network that can be computed directly on the camera-loaded GPU by a small convolutional neural network disposed on the front-end camera.

It should be noted that in some embodiments, the suspected abnormal behavior of all behavior classes may be preliminarily determined by the second convolutional neural network.

The abnormal behavior determination device provided by the present invention will be further described with reference to a specific embodiment.

Referring to fig. 7, as shown in fig. 7, an abnormal behavior determination apparatus 700 includes: at least one video monitoring module 701, a control data transmission module 702, a threshold determination module 703, a sorting module 704, and a data processing module 705, wherein:

the video monitoring module 701 includes a data obtaining module 7011, a data preliminary processing module 7012, and an image obtaining module 7013, where one of the data obtaining modules 7011 is mainly responsible for obtaining images and behavior data of the monitored area, where the behavior data includes, but is not limited to, images. The data acquisition module 7011 transmits the image to the data primary processing module 7012, the data primary processing module 7012 includes a small convolutional neural network and a GPU at the front end of the camera, and the received image is calculated by the small convolutional neural network and the GPU to output suspected abnormal behaviors and basic information of the person, where the basic information of the person includes, but is not limited to, information of the name, the position, the category, the sex, the age, the work number, and the like of the person. It should be noted that the small convolutional neural network has a high recall rate for detecting abnormal behaviors, and for a higher difference degree of suspected abnormal behaviors in a piece of behavior data, for example, 5 suspected abnormal behaviors exist in a certain behavior data, the small convolutional neural network can find all 5 suspected abnormal behaviors. The high recall rate is mainly due to the fact that the small convolutional neural network used in the embodiment of the invention is guaranteed by training and learning of a large number of abnormal behavior samples which are manually marked in the early stage. The image obtaining module 7013 may complete the adjustment of the camera pan-tilt angle, the camera focal length, and the like, mainly receive the coordinate information sent by the control module, and obtain the clear image information for the area where the coordinate information is located by adjusting the parameters of the camera pan-tilt angle, the focal length, and the like. It should be noted that the small convolutional neural network is further configured to obtain an execution degree of the suspected abnormal behavior, perform confidence calibration on the suspected abnormal behavior through the small convolutional neural network, and transmit the suspected abnormal behavior including the confidence to the threshold determination module for further processing.

The abnormal behavior determination apparatus 700 further includes a control data transmission module 702, a threshold determination module 703 and a sorting module 704, where the threshold determination module 703 receives the suspected abnormal behavior including the confidence level sent by the video monitoring module, sorts the suspected abnormal behavior according to the corresponding confidence level of the suspected abnormal behavior by the sorting module according to the information, and sends the suspected abnormal behavior larger than the preset threshold to the image acquisition module 7013. The threshold is a value between 0 and 1, and a better value can be obtained through a test set of different scenes, and generally for a particularly sensitive area, a lower value can be selected to extract a suspected area with a lower confidence of abnormal behavior, and the suspected area is sent to the image obtaining module 7013. The image obtaining module 7013 receives the coordinate information of the suspected abnormal behavior sent by the threshold determination module 703, and controls the camera pan-tilt to adjust the elevation angle, the horizontal angle, and the camera focal length, so that the camera is aligned to the area of the suspected abnormal behavior to obtain the targeted image data. The control data transmission module receives the image aiming at the suspected abnormal behavior area, reduces the image to 240 × 135 pixels, and then codes and transmits the image to the data processing module. Here, the images may be unified into a format such as PNG.

The data processing module mainly comprises a high-performance GPU cluster and a deep convolution neural classification network aiming at abnormal behaviors. The module receives the image sent by the control data transmission module and further confirms the image by using a deep convolutional neural network. And for the information for confirming the abnormal behavior, calculating the position of the original image, and finally outputting an alarm for confirming the abnormal behavior and corresponding coordinate information.

It should be noted that the corresponding coordinate information output by the data processing module may be coordinate information calculated by the data processing module, or the coordinate information where the suspected abnormal behavior sent by the threshold determination module included in the received image is located may be directly output by the data processing module.

It should be noted that the coordinate information corresponding to the suspected abnormal behavior may be calculated by the threshold determination module, or may be calculated by the small convolutional neural network.

It should be noted that the abnormal behavior determination apparatus may include a plurality of video monitoring modules, and the plurality of video monitoring modules may correspond to the same one or a plurality of data processing modules. In some embodiments, the plurality of video monitoring modules correspond to the plurality of threshold judgment modules, the control data transmission module and the sorting module, and each of the threshold judgment modules, the control data transmission module and the sorting module corresponds to the same data processing module.

It should be noted that the above modules are only an exemplary partition, and the data processing module, the threshold determining module, the control data transmission module, and the sorting module may belong to the same module.

In some embodiments, the data processing module is located at the back end of the entire abnormal behavior determination apparatus, and the video monitoring module and the control module are located at the front end of the abnormal behavior determination apparatus.

Through the use of the abnormal behavior determination device provided by the embodiment, only one data acquisition module, for example, one camera, needs to be arranged for one monitoring scene, for example, a conference room, a classroom and the like, so that the installation complexity is greatly reduced. The load of data processing module processing has greatly been alleviateed, information processing speed has been accelerated: the data processing module does not need to process the data acquired by each frame data acquisition module, and only processes partial frames with the suspected abnormal behavior confidence coefficient higher than a preset threshold value. And the processed partial frames are also normalized, so that the data volume is greatly reduced, the loads of information transmission and data processing are reduced, and the detection and judgment accuracy is improved. In addition, the accuracy of abnormal behavior detection is greatly improved: the behavior data are judged in two steps, the suspected abnormal behavior is selected after the first step of primary judgment of the video monitoring module through a second convolutional neural network, parameters of a pan-tilt and a camera are adjusted according to the suspected abnormal behavior to obtain a clear current state image of a suspected abnormal behavior area, the data transmission module is controlled to be normalized and then transmitted to the data processing module, and the second step of data processing module detects and judges the current state image extracted in the first step again by using a first convolutional neural network with pertinence, such as a deep convolutional neural classification network. The current state image obtained in the second step is clearer and more targeted. And the accuracy is obviously improved through the second step of judging the deep convolutional neural network.

The embodiment discloses an abnormal behavior determination device, including: at least one video monitoring module and data processing module, wherein: the video monitoring module includes: the data acquisition module is used for acquiring behavior data in the monitoring video; the data preliminary processing module is used for determining at least one suspected abnormal behavior from the behavior data; the image acquisition module is used for acquiring a current state image of an area where the suspected abnormal behavior is located; and the data processing module is used for judging whether the suspected abnormal behavior is the abnormal behavior according to the current state image. Through the implementation of the embodiment, not only can the computing resources be effectively utilized, but also the accuracy of judging the abnormal behavior can be improved, the resource occupation is reduced, and the processing speed is improved.

Example three:

the present embodiment further provides an abnormal behavior determination terminal, as shown in fig. 8, which includes a processor 81, a memory 83, and a communication bus 82, where:

the communication bus 82 is used for realizing connection communication between the processor 81 and the memory 83;

the processor 81 is configured to execute one or more computer programs stored in the memory 83 to implement at least one step of the abnormal behavior determination method in the embodiments described above.

Example four:

the present embodiments also provide a computer-readable storage medium including volatile or non-volatile, removable or non-removable media implemented in any method or technology for storage of information such as computer-readable instructions, data structures, computer program modules or other data. Computer-readable storage media include, but are not limited to, RAM (Random Access Memory), ROM (Read-Only Memory), EEPROM (Electrically Erasable Programmable Read-Only Memory), flash Memory or other Memory technology, CD-ROM (Compact disk Read-Only Memory), Digital Versatile Disks (DVD) or other optical disk storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other medium which can be used to store the desired information and which can be accessed by a computer.

The computer-readable storage medium in this embodiment may be used to store one or more computer programs, and the one or more computer programs stored in the computer-readable storage medium may be executed by a processor to implement at least one step of the abnormal behavior determination method in the embodiments described above.

The present embodiment further provides a computer program (or computer software), which can be distributed on a computer readable medium and executed by a computing device to implement at least one step of the abnormal behavior determination method in the foregoing embodiments; and in some cases at least one of the steps shown or described may be performed in an order different than that described in the embodiments above.

It should be understood that in some cases, at least one of the steps shown or described may be performed in a different order than described in the embodiments above.

The present embodiments also provide a computer program product comprising a computer readable means on which a computer program as shown above is stored. The computer readable means in this embodiment may include a computer readable storage medium as shown above.

It will be apparent to those skilled in the art that all or some of the steps of the methods, systems, functional modules/units in the devices disclosed above may be implemented as software (which may be implemented in computer program code executable by a computing device), firmware, hardware, and suitable combinations thereof. In a hardware implementation, the division between functional modules/units mentioned in the above description does not necessarily correspond to the division of physical components; for example, one physical component may have multiple functions, or one function or step may be performed by several physical components in cooperation. Some or all of the physical components may be implemented as software executed by a processor, such as a central processing unit, digital signal processor, or microprocessor, or as hardware, or as an integrated circuit, such as an application specific integrated circuit.

In addition, communication media typically embodies computer readable instructions, data structures, computer program modules or other data in a modulated data signal such as a carrier wave or other transport mechanism and includes any information delivery media as known to one of ordinary skill in the art. Thus, the present invention is not limited to any specific combination of hardware and software.

The foregoing is a more detailed description of embodiments of the present invention, and the present invention is not to be considered limited to such descriptions. For those skilled in the art to which the invention pertains, several simple deductions or substitutions can be made without departing from the spirit of the invention, and all shall be considered as belonging to the protection scope of the invention.

Claims

1. An abnormal behavior determination method, characterized by comprising:

acquiring behavior data in a monitoring video;

determining at least one suspected abnormal behavior from the behavior data;

2. The abnormal behavior determination method of claim 1, wherein the determining at least one suspected abnormal behavior from the behavior data comprises:

obtaining the confidence of the suspected abnormal behavior;

and if the confidence coefficient of the suspected abnormal behavior is larger than a preset threshold value, taking the suspected abnormal behavior as a target suspected abnormal behavior.

3. The abnormal behavior determination method according to claim 2, wherein the acquiring the current state image of the area where the suspected abnormal behavior exists comprises:

4. The abnormal behavior determination method according to claim 3,

when the number of the target suspected abnormal behaviors exceeds 1,

the acquiring of the current state image of the area where the target suspected abnormal behavior is located includes:

obtaining the target suspected abnormal behavior sequence, wherein the target suspected abnormal behavior sequence is obtained by sequencing the confidence degrees of the suspected abnormal behaviors corresponding to the target suspected abnormal behaviors according to a preset rule;

and acquiring a current state image of an area where the target suspected abnormal behavior is located according to the sequence of the target suspected abnormal behavior sequence.

5. The abnormal behavior determination method according to any one of claims 1 to 4, wherein after the obtaining of the current state image of the area where the suspected abnormal behavior is located, the determining whether the suspected abnormal behavior is an abnormal behavior according to the current state image further comprises:

and carrying out normalization processing on the current state image.

6. The abnormal behavior determination method according to any one of claims 1 to 4, wherein the determining whether the suspected abnormal behavior is an abnormal behavior based on the current-state image further comprises:

if the suspected abnormal behavior is judged to be abnormal behavior, outputting at least one of the following information:

abnormal behavior alert information;

and coordinate information corresponding to the abnormal behavior.

7. The abnormal behavior determination method according to any one of claims 1 to 4, wherein the determining whether the suspected abnormal behavior is an abnormal behavior based on the current-state image includes:

acquiring a behavior category where the suspected abnormal behavior is located;

acquiring a first convolution neural network corresponding to the behavior category where the suspected abnormal behavior is located;

and judging whether the suspected abnormal behavior is an abnormal behavior or not according to the current state image through the first convolutional neural network.

8. The abnormal behavior determination method according to any one of claims 1 to 4, wherein the determining at least one suspected abnormal behavior from the behavior data comprises:

determining at least one suspected abnormal behavior from the behavior data by a second convolutional neural network.

9. An abnormal behavior determination device characterized by comprising: at least one video monitoring module and data processing module, wherein:

the video monitoring module includes:

10. An abnormal behavior determination terminal, comprising: a processor, a memory, and a communication bus;

the processor is configured to execute one or more computer programs stored in the memory to implement the steps of the abnormal behavior determination method according to any one of claims 1 to 8.

11. A readable storage medium storing one or more computer programs, the one or more computer programs being executable by one or more processors to implement the steps of the abnormal behavior determination method according to any one of claims 1 to 8.