CN116206262A - Behavior recognition method, apparatus and storage medium - Google Patents
Behavior recognition method, apparatus and storage medium Download PDFInfo
- Publication number
- CN116206262A CN116206262A CN202310331466.9A CN202310331466A CN116206262A CN 116206262 A CN116206262 A CN 116206262A CN 202310331466 A CN202310331466 A CN 202310331466A CN 116206262 A CN116206262 A CN 116206262A
- Authority
- CN
- China
- Prior art keywords
- monitoring target
- preset
- behavior information
- determining
- optical flow
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V20/00—Scenes; Scene-specific elements
- G06V20/50—Context or environment of the image
- G06V20/52—Surveillance or monitoring of activities, e.g. for recognising suspicious objects
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/20—Analysis of motion
- G06T7/269—Analysis of motion using gradient-based methods
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02A—TECHNOLOGIES FOR ADAPTATION TO CLIMATE CHANGE
- Y02A10/00—TECHNOLOGIES FOR ADAPTATION TO CLIMATE CHANGE at coastal zones; at river basins
- Y02A10/40—Controlling or monitoring, e.g. of flood or hurricane; Forecasting, e.g. risk assessment or mapping
Landscapes
- Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Multimedia (AREA)
- Theoretical Computer Science (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Image Analysis (AREA)
Abstract
The invention provides a behavior recognition method, equipment and a storage medium, which comprise the steps of obtaining continuous first interval frame images in a current scene; based on the determined optical flow area, capturing a first interval frame image to obtain a first captured image corresponding to the first interval frame image; performing optical flow calculation on the first screenshot images of the front and rear frames to obtain a first optical flow graph corresponding to an optical flow region between the front and rear frames; based on the first optical flow graph, obtaining a first comprehensive light loss corresponding to the optical flow area; and determining first behavior information of the second monitoring target in the first interval frame image relative to the first monitoring target in the first interval frame image according to all the first comprehensive light loss. Therefore, the movement direction and the displacement amplitude of each pixel in the optical flow area can be accurately determined by using the obtained first comprehensive light loss quantity of the optical flow area under any view, so that the first behavior information of the second monitoring target relative to the first monitoring target can be accurately determined.
Description
Technical Field
The invention relates to the technical field of image processing, and particularly provides a behavior recognition method, behavior recognition equipment and a storage medium.
Background
The node records in various operation flows such as the airport apron flight operation flow and the like are mostly finished manually. For example, during the period from when any flight starts to enter the port until the flight leaves the port to take off, airport personnel determine and record the event that occurs through on-site observation or monitoring cameras, such as recording the aircraft in place when the aircraft is stationary at the port, recording the in place when the tanker arrives at the terminal ready for fueling, and recording the conveyor docking event when the conveyor starts to abut the aircraft ready for baggage transfer. Events such as these record up to several tens of hundreds of items. The labor investment is large, the personnel fatigue is easy to be caused, and the recording errors are easy to be caused. Thus, a visual AI-based workflow node automated identification system has arisen. Some existing node recognition algorithms can support the automatic recognition of the behavior of each monitoring target in the workflow, for example, the behavior of each monitoring target in dozens of node events such as airplane arrival and departure, refueling truck arrival and departure, baggage conveyor truck docking and evacuation airplane, corridor bridge head departure and reset.
In an actual airport operation scene, a left monitoring camera, a middle monitoring camera and a right monitoring camera are generally arranged to monitor the operation scene in an all-around mode, various operation node events are automatically identified based on a visual means, and the requirement trend of intelligent upgrading of airport equipment is met. However, due to different operation scenes and camera position angles, the node AI automatic analysis generalization is poor, part of scene effects are good, and part of scene effects cannot be used.
Taking the corridor bridge head as an example, the related events include four node events such as the corridor bridge head docking aircraft, the corridor bridge head withdrawing aircraft, the corridor bridge head leaving (i.e. leaving the berth in the non-working state), the corridor bridge head resetting (i.e. returning to and stabilizing the berth in the non-working state), and the like. The conventional corridor bridge head behavior recognition algorithm carries out target detection training by directly marking whether the corridor bridge head is in butt joint with the aircraft passenger door, so that the corridor bridge head is in butt joint with the aircraft behavior or the evacuation aircraft behavior is obtained. However, the left or right monitoring camera has a sight line overlapping with the movement direction of the corridor bridge, so that it is difficult to see whether the monitoring camera is in a docking state or an evacuation state, which often results in erroneous judgment. Some corridor bridge head behavior recognition algorithms use a corridor bridge head detection and tracking means to judge the movement state of the corridor bridge head, but the problem of the visual field of the left and right side angles is difficult to solve, and the main reason is that the pixel displacement change is small when the corridor bridge head moves, so that the algorithm misjudges.
Therefore, how to accurately monitor the behavior information of the target in the workflow is a technical problem to be solved by those skilled in the art.
Disclosure of Invention
The present invention has been made to overcome the above-mentioned drawbacks, and provides a behavior recognition method, apparatus, and storage medium that solve or at least partially solve the technical problem of low accuracy of behavior information of a monitoring target in a recognition workflow.
In a first aspect, the present invention provides a behavior recognition method, the behavior recognition method comprising:
acquiring continuous first interval frame images in a current scene; the first interval frame image comprises a first monitoring target and a second monitoring target;
based on the determined optical flow area, capturing the first interval frame image to obtain a first captured image corresponding to the first interval frame image;
performing optical flow calculation on the first screenshot images of the front and rear frames to obtain a first optical flow graph corresponding to the optical flow region between the front and rear frames;
based on the first optical flow graph, obtaining a first comprehensive light loss corresponding to the optical flow area;
and determining first behavior information of the second monitoring target relative to the first monitoring target according to all the first comprehensive light loss.
Further, in the behavior recognition method, obtaining a first integrated light loss amount corresponding to the light flow area based on the first light flow graph includes:
extracting a first preset number of first motion vectors from the first optical flow map;
selecting a first motion vector with a vector length larger than a first preset length as a first target motion vector;
If the ratio of the total number of the first target motion vectors to the first preset number is larger than a first preset proportion, averaging all the first target motion vectors to obtain a first average vector;
and if the vector length of the first average vector is greater than a second preset length, taking the first average vector as a first comprehensive light loss corresponding to the light flow area between the front frame and the rear frame.
Further, in the behavior recognition method, determining, according to all the first integrated light loss amounts, first behavior information of the second monitoring target relative to the first monitoring target includes:
if the ratio of the number of the first comprehensive light loss amounts to the total frame number of the images is larger than a second preset ratio, determining a first included angle between each first comprehensive light loss amount and the orthogonal direction of the first monitoring target; the orthogonal direction of the first monitoring target is a direction perpendicular to the direction of the first monitoring target;
if all the first included angles are smaller than a first preset included angle, determining that the first behavior information of the second monitoring target relative to the first monitoring target is in a close state;
if all the first included angles are larger than a second preset included angle, determining that the first behavior information of the second monitoring target relative to the first monitoring target is in an evacuation state; wherein the second preset included angle is larger than the first preset included angle;
And if part of the first included angles are larger than or equal to the first preset included angles, and/or part of the first included angles are smaller than or equal to the second preset included angles, determining that the first behavior information of the second monitoring target relative to the first monitoring target is in a static state.
Further, in the behavior recognition method, if the first behavior information is in a close state or an evacuation state, the method further includes:
acquiring continuous second interval frame images in the current scene; the second interval frame image comprises a first monitoring target and a second monitoring target; the first interval time corresponding to the first interval frame image is smaller than the interval time corresponding to the second interval frame image;
based on the determined optical flow area, capturing the second interval frame image to obtain a second captured image corresponding to the second interval frame image;
performing optical flow calculation on the second screenshot images of the front and rear frames to obtain a second optical flow graph corresponding to the optical flow region between the front and rear frames;
obtaining a second comprehensive light loss amount corresponding to the light flow area between the front frame and the rear frame based on the second light flow graph between the front frame and the rear frame;
determining second behavior information of the second monitoring target relative to the first monitoring target according to all the second comprehensive light loss;
And determining final behavior information of the second monitoring target relative to the first monitoring target according to the first behavior information and the second behavior information.
Further, in the behavior recognition method, obtaining a second integrated light loss amount corresponding to the light flow area between the front frame and the rear frame based on the second light flow graph between the front frame and the rear frame includes:
extracting a second preset number of second motion vectors from the second optical flow map;
selecting a second motion vector with the vector length being greater than a third preset length as a second target motion vector;
if the ratio of the total number of the second target motion vectors to the second preset number is larger than a third preset ratio, averaging all the second target motion vectors to obtain a second average vector;
and if the vector length of the second average vector is greater than a fourth preset length, taking the second average vector as a second comprehensive light loss amount corresponding to the light flow area between the front frame and the rear frame.
Further, in the behavior recognition method, determining second behavior information of the second monitoring target relative to the first monitoring target according to all the second integrated light loss amounts includes:
Determining a second included angle between each second comprehensive light loss and the orthogonal direction of the first monitoring target; the orthogonal direction of the first monitoring target is a direction perpendicular to the direction of the first monitoring target;
if all the second included angles are smaller than a third preset included angle, determining that second behavior information of the second monitoring target relative to the first monitoring target is in a close state;
if all the second included angles are larger than a fourth preset included angle, determining that second behavior information of the second monitoring target relative to the first monitoring target is in an evacuation state; wherein the fourth preset included angle is larger than the third preset included angle;
and if part of the second included angles are larger than or equal to the third preset included angles, and/or part of the second included angles are smaller than or equal to the fourth preset included angles, determining that the second behavior information of the second monitoring target relative to the first monitoring target is in a static state.
Further, in the behavior recognition method, determining final behavior information of the second monitoring target relative to the first monitoring target according to the first behavior information and the second behavior information includes:
if the first behavior information and the second behavior information are both in a close state, determining that final behavior information of the second monitoring target relative to the first monitoring target is in a close state;
If the first behavior information and the second behavior information are both in an evacuation state, determining that final behavior information of the second monitoring target relative to the first monitoring target is in the evacuation state;
and if the first behavior information and the second behavior information are inconsistent, determining that the final behavior information of the second monitoring target relative to the first monitoring target is in a static state.
Further, in the behavior recognition method, before acquiring the continuous first interval frame image in the current scene, the behavior recognition method further includes:
acquiring continuous third interval frame images in a current scene;
if the first monitoring target and the second monitoring target exist in the third interval frame image, generating a detection frame of the first monitoring target and a detection frame of the second monitoring target;
determining key points of the first monitoring target and a foreground area of a preset part in the first monitoring target based on a detection frame of the first monitoring target;
determining the direction of the first monitoring target according to the key points of the first monitoring target;
determining the overlapping degree of the detection frame of the second monitoring target and the foreground region of the preset part;
If the orientation of the first monitoring target is located in a first visual field range and the overlapping degree is greater than a first preset overlapping degree, or if the orientation of the first monitoring target is located in a second visual field range and the overlapping degree is greater than a second preset overlapping degree, taking a region corresponding to a detection frame of the second monitoring target as the optical flow region;
the first preset overlapping degree is smaller than the second preset overlapping degree.
In a second aspect, the present invention provides a behavior recognition apparatus comprising a processor and a storage device adapted to store a plurality of program code adapted to be loaded and executed by the processor to perform the behavior recognition method of any one of the preceding claims.
In a third aspect, a computer readable storage medium is provided, wherein the computer readable storage medium stores a plurality of program codes adapted to be loaded and executed by a processor to perform the behavior recognition method of any one of the above
The technical scheme provided by the invention has at least one or more of the following beneficial effects:
in the technical scheme of implementing the invention, after the continuous first interval frame image in the current scene is obtained, the first interval frame image can be subjected to screenshot based on the determined optical flow area, so as to obtain a first screenshot image corresponding to the first interval frame image; performing optical flow calculation on the first screenshot images of the front and rear frames to obtain a first optical flow graph corresponding to the optical flow region between the front and rear frames; based on the first optical flow graph, obtaining a first comprehensive light loss corresponding to the optical flow area; and determining first behavior information of the second monitoring target relative to the first monitoring target according to all the first comprehensive light loss. Therefore, the movement direction and the displacement amplitude of each pixel in the optical flow area can be accurately determined by using the obtained first comprehensive light loss quantity of the optical flow area under any view, so that the first behavior information of the second monitoring target relative to the first monitoring target can be accurately determined.
Drawings
The present disclosure will become more readily understood with reference to the accompanying drawings. As will be readily appreciated by those skilled in the art: the drawings are for illustrative purposes only and are not intended to limit the scope of the present invention. Moreover, like numerals in the figures are used to designate like parts, wherein:
FIG. 1 is a flow diagram of the main steps of a behavior recognition method according to one embodiment of the present invention;
FIG. 2 is a schematic flow diagram of determining an optical flow region;
FIG. 3 is a schematic diagram of the results of object detection on an image;
FIG. 4 is a schematic diagram of determining a keypoint of a first monitoring target and a foreground region of a preset location in the first monitoring target;
fig. 5 is a main structural block diagram of a behavior recognition apparatus according to an embodiment of the present invention.
Detailed Description
Some embodiments of the invention are described below with reference to the accompanying drawings. It should be understood by those skilled in the art that these embodiments are merely for explaining the technical principles of the present invention, and are not intended to limit the scope of the present invention.
In the description of the present invention, a "module," "processor" may include hardware, software, or a combination of both. A module may comprise hardware circuitry, various suitable sensors, communication ports, memory, or software components, such as program code, or a combination of software and hardware. The processor may be a central processor, a microprocessor, an image processor, a digital signal processor, or any other suitable processor. The processor has data and/or signal processing functions. The processor may be implemented in software, hardware, or a combination of both. Non-transitory computer readable storage media include any suitable medium that can store program code, such as magnetic disks, hard disks, optical disks, flash memory, read-only memory, random access memory, and the like. The term "a and/or B" means all possible combinations of a and B, such as a alone, B alone or a and B. The term "at least one A or B" or "at least one of A and B" has a meaning similar to "A and/or B" and may include A alone, B alone or A and B. The singular forms "a", "an" and "the" include plural referents.
In various operation flows such as airport parking apron flight operation flow, the front view camera can well observe behavior information of a monitoring target, such as docking and evacuation of a corridor bridge and an airplane, and resetting and dislocation of a corridor bridge head. However, the left side or right side view angle often has the condition that the movement direction of the monitoring target is the same as or opposite to the direction of the camera sight, or the pixel displacement change is very small when the corridor bridge head moves, so that the accuracy of the identification result of the behavior information of the monitoring target is lower.
Therefore, in order to solve the technical problems, the invention provides the following technical scheme:
referring to fig. 1, fig. 1 is a schematic flow chart of main steps of a behavior recognition method according to an embodiment of the present invention. As shown in fig. 1, the behavior recognition method in the embodiment of the present invention mainly includes the following steps 101 to 105.
in one implementation, a continuous first interval frame image in a current scene may be acquired using a monitoring camera disposed in the current scene. The first interval frame image comprises a first monitoring target and a second monitoring target. The first monitoring target may be a stationary target, such as an aircraft parked in an airport apron, and the second monitoring target may be a target to be maneuvered, such as a corridor bridge nose. The time interval corresponding to the first interval frame image may be 1s.
102, based on the determined optical flow area, performing screenshot on the first interval frame image to obtain a first screenshot image corresponding to the first interval frame image;
in a specific implementation process, an optical flow area may be predetermined, and the first interval frame image is captured by using the optical flow area, so as to obtain a first captured image corresponding to the first interval frame image. For example, the optical flow region may be a region surrounded by a rectangular frame, so that the obtained first screenshot image is a rectangular image.
In a specific implementation process, the optical flow area may be a preset area, or may be determined according to the following manner, and after the optical flow area is determined according to the following manner, the optical flow area is fixed and then directly used, so that the optical flow area does not need to be repeatedly determined. FIG. 2 is a schematic flow diagram of determining an optical flow region. As shown in fig. 2, in determining the optical flow region, it may be implemented as follows steps 201-206:
in one specific implementation, a continuous third interval frame image in the current scene may be acquired by using a monitoring camera disposed in the current scene. The time interval corresponding to the third interval image may be the same as or different from the time interval corresponding to the first interval image.
Here, the third interval frame images are acquired in order to determine the optical flow region using the third interval frame images. Since the subsequent optical flow analysis cannot be performed until the optical flow region is not determined, only the subsequent image needs to be subjected to the optical flow analysis after the optical flow region is determined, and the previous image does not need to be analyzed, that is, the first inter-frame image is an image located after the third inter-frame image.
Step 202, if the first monitoring target and the second monitoring target exist in the third interval frame image, generating a detection frame of the first monitoring target and a detection frame of the second monitoring target;
in a specific implementation process, a pre-trained YoloV5 detection model and the like can be used for detecting a third interval frame image, and if the first monitoring target and the second monitoring target exist in the third interval frame image, a detection frame for generating the first monitoring target and a detection frame for generating the second monitoring target can be used. Fig. 3 is a schematic diagram of the result of target detection on an image. As shown in fig. 3, taking the first monitoring target as an aircraft, the second monitoring target as a corridor bridge head as an example. And inputting a third interval frame image into a YoloV5 detection model, and if the aircraft is detected, indicating the aircraft stopping state, wherein the aircraft is framed by a first detection frame a and the corridor bridge head is framed by a second detection frame b.
If the first monitoring target is detected to be absent in the third interval frame image, the labeling of the subsequent detection frame may not be performed.
in a specific implementation process, a detection frame of a first monitoring target can be utilized to intercept a first monitoring target subgraph, pixels of the intercepted subgraph are adjusted to a preset pixel width and a preset pixel height, pixel value normalization is carried out, then a key point detection model is input, and key points of an airplane and a foreground area of a preset position in the first monitoring target are obtained in an inference mode. The preset pixel width may be 256, and the preset pixel height may be 256. Fig. 4 is a schematic diagram of determining a key point of a first monitoring target and a foreground region of a preset part in the first monitoring target. As shown in fig. 4, taking an aircraft as an example, the key points of the aircraft may include a nose point (represented by a nose in fig. 4), a main wing left end point (represented by a main wing left end in fig. 4), a main wing right end point (represented by a main wing right end in fig. 4), a nacelle midpoint, a tail front end point (represented by a tail in fig. 4), a tail end point (represented by a tail end in fig. 4), a tail right end point (represented by a tail right in fig. 4), a tail left end point (represented by a tail left in fig. 4), and the like. The left end and the right end of the main wing, the right end of the tail wing and the left end of the tail wing are divided relative to the direction of the machine head. The foreground region d of the fuselage can be obtained by masking techniques.
in a specific implementation process, after determining the key point of the first monitoring target, the key point of the first monitoring target may be used to determine the direction of the first monitoring target. As shown in fig. 4, the aircraft orientation may be determined to be Vp in fig. 4 by the nose point and the cabin center point or tail point. It should be noted that other key points may be used to determine the aircraft orientation.
Step 205, determining the overlapping degree of the detection frame of the second monitoring target and the foreground region of the preset part;
in a specific implementation process, the overlapping degree of the detection frame of the second monitoring target and the foreground region of the preset part may be obtained according to the calculation formula (1):
IOU ab ＝n p /(w b *h b ) (1)
wherein the IOU ab Is the overlapping degree, w b A pixel width of a detection frame for the second monitoring target, h b The pixel height of the detection frame for the second monitoring target is n p The total number of the pixel points in the detection frame of the second monitoring target is the foreground area of the preset part.
In one specific implementation procedure, for a scene where the three-sided monitoring camera is disposed, the field of view of the front monitoring camera may be set to a first field of view, specifically, an angle of ±35° between the direction Vp of the first monitoring target and the center line c in fig. 4 may be set, and the field of view of the monitoring cameras on the left and right sides may be set to a second field of view, in addition to the first field of view. When the orientation of the first monitoring target is within the first visual field range, the front camera can easily identify the second monitoring target, so that a smaller overlapping degree can be set as a first preset overlapping degree, then the obtained overlapping degree is compared with the first preset overlapping degree, when the obtained overlapping degree is larger than the first preset overlapping degree (0.15), the area corresponding to the detection frame of the second monitoring target can be used as the optical flow area, and otherwise, the obtained overlapping degree is continuously compared with the first preset overlapping degree. When the orientation of the first monitoring target is within the second visual field range, the left and right monitoring cameras are not easy to identify the second monitoring target, so that a larger overlapping degree can be set as a second preset overlapping degree (0.3), then the obtained overlapping degree is compared with the second preset overlapping degree, when the obtained overlapping degree is larger than the second preset overlapping degree, the area corresponding to the detection frame of the second monitoring target can be used as the optical flow area, otherwise, the obtained overlapping degree is continuously compared with the second preset overlapping degree.
Step 103, performing optical flow calculation on the first screenshot images of the front and rear frames to obtain a first optical flow graph corresponding to the optical flow region between the front and rear frames;
in a specific implementation process, a Lucas-Kanade algorithm, a pyramid Lucas-Kanade algorithm and the like can be adopted for performing optical flow calculation on first screenshot images of front and rear frames to obtain a first optical flow graph corresponding to the optical flow region between the front and rear frames. For dense optical flow, optical flow calculation can be performed on the first screenshot images of the front frame and the rear frame by adopting a Farnesback algorithm, an RLOF algorithm and the like, so as to obtain a first optical flow graph corresponding to the optical flow region between the front frame and the rear frame. And are not illustrated herein.
In order to reduce the amount of calculation and increase the calculation efficiency, the first screenshot image of the previous and subsequent frames may be scaled, and the first screenshot image may be scaled to an image with a pixel height of 160 and a pixel width of 160, converted into a gray scale, and then subjected to optical flow calculation.
104, obtaining a first comprehensive light loss amount corresponding to the light stream area based on the first light stream graph;
in a specific implementation process, the first integrated light loss amount corresponding to the light flow area may be calculated according to the following steps:
(1) Extracting a first preset number of first motion vectors from the first optical flow map;
in one implementation, the first preset number may be set according to a pixel width and a pixel height of the optical flow map, and the first preset number of first motion vectors may be extracted from the first optical flow map. For example, for an optical flow graph with a pixel height of 160 and a pixel width of 160, one motion vector can be extracted for every 4 pixels, and then the motion vector of the edge is removed, so that 39×39=1521 first motion vectors can be obtained.
(2) Selecting a first motion vector with a vector length larger than a first preset length as a first target motion vector;
in a specific implementation process, when the first light flow graph is obtained, the vector length of each first motion vector can be obtained according to the pixel coordinates of the previous frame and the pixel coordinates of the next frame of the pixels corresponding to each first motion vector in the first interval frame images of the previous frame and the next frame, so that the first motion vector with the vector length being greater than the first preset length can be selected as the first target motion vector, and the first motion vector with the displacement exceeding a certain amplitude can be selected. Wherein, the first preset length may be 1.5.
(3) If the ratio of the total number of the first target motion vectors to the first preset number is larger than a first preset proportion, averaging all the first target motion vectors to obtain a first average vector;
In a specific implementation process, the ratio of the total number of the first target motion vectors to the first preset number is greater than a first preset ratio, and all the first target motion vectors can be averaged to obtain a first average vector. Wherein the first preset ratio may be 0.2.
(4) And if the vector length of the first average vector is greater than a second preset length, taking the first average vector as a first comprehensive light loss corresponding to the light flow area between the front frame and the rear frame.
In a specific implementation process, the vector length of the first average vector may be obtained according to the pixel coordinates of the previous frame and the pixel coordinates of the next frame of the corresponding pixel of the pixel corresponding to the first average vector, and then the vector length of the first average vector is compared with the second preset length. If the vector length of the first average vector is greater than the second preset length, the first average vector may be used as a first integrated light loss corresponding to the light flow area between the front frame and the rear frame. The second preset length may be the same as the first preset length, and both the second preset length and the first preset length are 1.5.
It should be noted that, the reason why the vector length of the first average vector is compared again to be greater than the second preset length is that the vectors are directional, and although the vector length of the single first target motion vector is greater than the first preset length, the vector length of the first average vector may be shortened after all the first target motion vectors are averaged to obtain the first average vector, so if the vector length of the first average vector is greater than the second preset length, it is indicated that the displacement of the first average vector exceeds a certain amplitude, and thus the first integrated light loss corresponding to the light flow area may be obtained.
And 105, determining first behavior information of the second monitoring target relative to the first monitoring target according to all the first comprehensive light loss amounts.
In a specific implementation process, in the continuous N frames of first interval images, if the ratio of the number of the obtained first comprehensive light loss to the total frame number of the images is greater than a second preset proportion, the second monitoring target is indicated to be active in an optical flow area, and at the moment, a first included angle between each first comprehensive light loss and the orthogonal direction of the first monitoring target can be calculated; the orthogonal direction of the first monitoring target is a direction perpendicular to the direction of the first monitoring target. The direction shown as Vc in fig. 4 is the orthogonal direction of the first monitoring target.
In a specific implementation process, if all the first included angles are smaller than a first preset included angle, determining that the first behavior information of the second monitoring target relative to the first monitoring target is in a close state; if all the first included angles are larger than a second preset included angle, determining that the first behavior information of the second monitoring target relative to the first monitoring target is in an evacuation state; and if part of the first included angles are larger than or equal to the first preset included angles, and/or part of the first included angles are smaller than or equal to the second preset included angles, determining that the first behavior information of the second monitoring target relative to the first monitoring target is in a static state. Wherein the second preset included angle is larger than the first preset included angle. In some examples, the sum of the first preset angle and the second preset angle may be 180 °.
In this embodiment, by means of the accurate key point of the first monitoring target and the foreground region of the preset part in the first monitoring target, the reliable geometric information such as the orientation of the first monitoring target can be provided, so that the accuracy that the first behavior information of the second monitoring target relative to the first monitoring target is in a motion state is greatly improved. For example, whether the second monitoring target is in a stationary state or in a moving state can be accurately detected, and whether the moving direction of the second monitoring target is close to the first monitoring target or is evacuated from the first monitoring target can be accurately detected for the moving state. And the information such as the time from the start of the second monitoring target to the stop of the abutting, the time from the start of the evacuation to the end of the evacuation and the like can be accurately positioned according to the time stamp of the frame-free image.
According to the behavior recognition method, after continuous first interval frame images in a current scene are obtained, the first interval frame images can be subjected to screenshot based on the determined optical flow area, and first screenshot images corresponding to the first interval frame images are obtained; performing optical flow calculation on the first screenshot images of the front and rear frames to obtain a first optical flow graph corresponding to the optical flow region between the front and rear frames; based on the first optical flow graph, obtaining a first comprehensive light loss corresponding to the optical flow area; and determining first behavior information of the second monitoring target relative to the first monitoring target according to all the first comprehensive light loss. Therefore, the movement direction and the displacement amplitude of each pixel in the optical flow area can be accurately determined by using the obtained first comprehensive light loss quantity of the optical flow area under any view, so that the first behavior information of the second monitoring target relative to the first monitoring target can be accurately determined.
In a specific implementation process, under an outdoor scene, interference such as shake, instant shielding and the like caused by factors such as rain, snow, wind and the like on a monitoring camera are unavoidable, and the shake of the monitoring camera can cause optical flow distortion. Or, the monitoring target may repeatedly advance and retreat in the moving process, so that misjudgment occurs on the first behavior information of the second monitoring target relative to the first monitoring target obtained by the method.
Specifically, whether the monitoring camera shakes, is blocked instantaneously, or the second monitoring target is adjusted repeatedly, the second monitoring target is caused to repeatedly advance and retreat, and the final behavior information of the second monitoring target relative to the first monitoring target is judged to be in a motion state by the method. Therefore, if the first behavior information is in the approaching state or the evacuation state, the behavior recognition method of the present embodiment further includes the following steps:
(11) Acquiring continuous second interval frame images in the current scene;
in a specific implementation process, the second monitoring target can be obtained to be in a motion state relative to the first behavior information of the first monitoring target in a short time because of reasons such as jitter, instant shielding, repeated adjustment of the second monitoring target and the like of the monitoring camera, but if the time interval is longer, the final judgment result is inconsistent with the judgment result when the time interval is shorter. Therefore, in the present embodiment, continuous second-interval frame images in the current scene can be acquired. The second interval frame image may also include a first monitoring target and a second monitoring target. The first interval time corresponding to the first interval frame image is smaller than the interval time corresponding to the second interval frame image.
The first frame second interval frame image and the first frame first interval frame image are the same image.
(12) Based on the determined optical flow area, capturing the second interval frame image to obtain a second captured image corresponding to the second interval frame image;
(13) Performing optical flow calculation on the second screenshot images of the front and rear frames to obtain a second optical flow graph corresponding to the optical flow region between the front and rear frames;
the implementation process of the steps (12) to (13) may refer to the implementation process of the steps 102 to 103, which are not described herein.
(14) Obtaining a second comprehensive light loss amount corresponding to the light flow area between the front frame and the rear frame based on the second light flow graph between the front frame and the rear frame;
specifically, a second preset number of second motion vectors may be extracted from the second optical flow map; selecting a second motion vector with the vector length being greater than a third preset length as a second target motion vector; if the ratio of the total number of the second target motion vectors to the second preset number is larger than a third preset ratio, averaging all the second target motion vectors to obtain a second average vector; and if the vector length of the second average vector is greater than a fourth preset length, taking the second average vector as a second comprehensive light loss amount corresponding to the light flow area between the front frame and the rear frame.
It should be noted that the second preset number may be the same as the first preset number, the third preset length may be the same as the first preset length, the fourth preset length may be the same as the second preset length, and the third preset ratio may be the same as the first preset ratio.
(15) Determining second behavior information of the second monitoring target relative to the first monitoring target according to all the second comprehensive light loss;
in a specific implementation process, a second included angle between each second integrated light loss amount and the orthogonal direction of the first monitoring target can be determined; the orthogonal direction of the first monitoring target is a direction perpendicular to the direction of the first monitoring target; if all the second included angles are smaller than a third preset included angle, determining that second behavior information of the second monitoring target relative to the first monitoring target is in a close state; if all the second included angles are larger than a fourth preset included angle, determining that second behavior information of the second monitoring target relative to the first monitoring target is in an evacuation state; wherein the fourth preset included angle is larger than the third preset included angle; and if part of the second included angles are larger than or equal to the third preset included angles, and/or part of the second included angles are smaller than or equal to the fourth preset included angles, determining that the second behavior information of the second monitoring target relative to the first monitoring target is in a static state.
It should be noted that the third preset included angle may be the same as the first preset included angle. The fourth predetermined angle may be the same as the second predetermined angle.
(16) And determining final behavior information of the second monitoring target relative to the first monitoring target according to the first behavior information and the second behavior information.
In a specific implementation process, if the first behavior information and the second behavior information are both in a close state, determining that final behavior information of the second monitoring target relative to the first monitoring target is in a close state; if the first behavior information and the second behavior information are both in an evacuation state, determining that final behavior information of the second monitoring target relative to the first monitoring target is in the evacuation state; and if the first behavior information and the second behavior information are inconsistent, determining that the final behavior information of the second monitoring target relative to the first monitoring target is in a static state.
That is, if the first integrated light loss amount obtained based on the first interval image of the short time interval and the second integrated light loss amount obtained based on the second interval image of the longer time interval are the same in direction, and the included angles with the orthogonal vector Vc of the first monitoring target are both smaller than the respective corresponding preset included angles, determining that the final behavior information of the second monitoring target with respect to the first monitoring target is in a close state, otherwise, determining that the final behavior information of the second monitoring target with respect to the first monitoring target is in a stationary state; and if the first comprehensive light loss amount obtained based on the first interval image of the short time interval is the same as the second comprehensive light loss amount obtained based on the second interval image of the longer time interval, and the included angle between the first comprehensive light loss amount and the orthogonal vector Vc of the aircraft is larger than the preset corresponding to each of the first comprehensive light loss amount and the second comprehensive light loss amount, determining that the final behavior information of the second monitoring target relative to the first monitoring target is in an evacuation state, otherwise, determining that the final behavior information of the second monitoring target relative to the first monitoring target is in a static state.
According to the behavior recognition method, after a second comprehensive light loss amount is obtained based on a second interval image with a longer time interval, second behavior information of the second monitoring target relative to the first monitoring target is determined according to all the second comprehensive light loss amounts; and determining final behavior information of the second monitoring target relative to the first monitoring target according to the first behavior information and the second behavior information, so as to correct the first behavior information, thereby avoiding misjudgment caused by movement of the monitoring camera or by back and forth adjustment of the second monitoring target, and ensuring recognition result of the behavior information of the second monitoring target relative to the first monitoring target.
It should be noted that, although the foregoing embodiments describe the steps in a specific order, it will be understood by those skilled in the art that, in order to achieve the effects of the present invention, the steps are not necessarily performed in such an order, and may be performed simultaneously (in parallel) or in other orders, and these variations are within the scope of the present invention.
It will be appreciated by those skilled in the art that the present invention may implement all or part of the above-described methods according to the above-described embodiments, or may be implemented by means of a computer program for instructing relevant hardware, where the computer program may be stored in a computer readable storage medium, and where the computer program may implement the steps of the above-described embodiments of the method when executed by a processor. Wherein the computer program comprises computer program code which may be in source code form, object code form, executable file or some intermediate form etc. The computer readable storage medium may include: any entity or device, medium, usb disk, removable hard disk, magnetic disk, optical disk, computer memory, read-only memory, random access memory, electrical carrier wave signals, telecommunications signals, software distribution media, and the like capable of carrying the computer program code. It should be noted that the computer readable storage medium may include content that is subject to appropriate increases and decreases as required by jurisdictions and by jurisdictions in which such computer readable storage medium does not include electrical carrier signals and telecommunications signals.
Further, the invention also provides behavior recognition equipment.
Referring to fig. 5, fig. 5 is a main structural block diagram of a behavior recognition apparatus according to an embodiment of the present invention. As shown in fig. 5, the behavior recognition apparatus of the embodiment of the present invention may include a processor 51 and a storage device 52, the storage device 52 being adapted to store a plurality of program codes adapted to be loaded and executed by the processor 51 to perform the behavior recognition method of the above-described embodiment.
For convenience of explanation, only those portions of the embodiments of the present invention that are relevant to the embodiments of the present invention are shown, and specific technical details are not disclosed, please refer to the method portions of the embodiments of the present invention. The behavior recognition device may be a control device formed by including various electronic devices.
Further, the invention also provides a computer readable storage medium. In one computer-readable storage medium embodiment according to the present invention, the computer-readable storage medium may be configured to store a program that performs the behavior recognition method of the above-described method embodiment, the program being loadable and executable by a processor to implement the above-described behavior recognition method. For convenience of explanation, only those portions of the embodiments of the present invention that are relevant to the embodiments of the present invention are shown, and specific technical details are not disclosed, please refer to the method portions of the embodiments of the present invention. The computer readable storage medium may be a storage device including various electronic devices, and optionally, the computer readable storage medium in the embodiments of the present invention is a non-transitory computer readable storage medium.
Further, it should be understood that, since the respective modules are merely set to illustrate the functional units of the apparatus of the present invention, the physical devices corresponding to the modules may be the processor itself, or a part of software in the processor, a part of hardware, or a part of a combination of software and hardware. Accordingly, the number of individual modules in the figures is merely illustrative.
Those skilled in the art will appreciate that the various modules in the apparatus may be adaptively split or combined. Such splitting or combining of specific modules does not cause the technical solution to deviate from the principle of the present invention, and therefore, the technical solution after splitting or combining falls within the protection scope of the present invention.
Thus far, the technical solution of the present invention has been described in connection with the preferred embodiments shown in the drawings, but it is easily understood by those skilled in the art that the scope of protection of the present invention is not limited to these specific embodiments. Equivalent modifications and substitutions for related technical features may be made by those skilled in the art without departing from the principles of the present invention, and such modifications and substitutions will fall within the scope of the present invention.
Claims (10)
1. A method of behavior recognition, comprising:
acquiring continuous first interval frame images in a current scene; the first interval frame image comprises a first monitoring target and a second monitoring target;
based on the determined optical flow area, capturing the first interval frame image to obtain a first captured image corresponding to the first interval frame image;
performing optical flow calculation on the first screenshot images of the front and rear frames to obtain a first optical flow graph corresponding to the optical flow region between the front and rear frames;
based on the first optical flow graph, obtaining a first comprehensive light loss corresponding to the optical flow area;
and determining first behavior information of the second monitoring target relative to the first monitoring target according to all the first comprehensive light loss.
2. The behavior recognition method according to claim 1, wherein obtaining a first integrated light loss amount corresponding to the light flow area based on the first light flow graph includes:
extracting a first preset number of first motion vectors from the first optical flow map;
selecting a first motion vector with a vector length larger than a first preset length as a first target motion vector;
If the ratio of the total number of the first target motion vectors to the first preset number is larger than a first preset proportion, averaging all the first target motion vectors to obtain a first average vector;
and if the vector length of the first average vector is greater than a second preset length, taking the first average vector as a first comprehensive light loss corresponding to the light flow area between the front frame and the rear frame.
3. The behavior recognition method according to claim 1, wherein determining first behavior information of the second monitoring target with respect to the first monitoring target based on all of the first integrated light loss amounts includes:
if the ratio of the number of the first comprehensive light loss amounts to the total frame number of the images is larger than a second preset ratio, determining a first included angle between each first comprehensive light loss amount and the orthogonal direction of the first monitoring target; the orthogonal direction of the first monitoring target is a direction perpendicular to the direction of the first monitoring target;
if all the first included angles are smaller than a first preset included angle, determining that the first behavior information of the second monitoring target relative to the first monitoring target is in a close state;
if all the first included angles are larger than a second preset included angle, determining that the first behavior information of the second monitoring target relative to the first monitoring target is in an evacuation state; wherein the second preset included angle is larger than the first preset included angle;
And if part of the first included angles are larger than or equal to the first preset included angles, and/or part of the first included angles are smaller than or equal to the second preset included angles, determining that the first behavior information of the second monitoring target relative to the first monitoring target is in a static state.
4. A behavior recognition method according to any one of claims 1-3, wherein if the first behavior information is in an approaching state or an evacuation state, the method further comprises:
acquiring continuous second interval frame images in the current scene; the second interval frame image comprises a first monitoring target and a second monitoring target; the first interval time corresponding to the first interval frame image is smaller than the interval time corresponding to the second interval frame image;
based on the determined optical flow area, capturing the second interval frame image to obtain a second captured image corresponding to the second interval frame image;
performing optical flow calculation on the second screenshot images of the front and rear frames to obtain a second optical flow graph corresponding to the optical flow region between the front and rear frames;
obtaining a second comprehensive light loss amount corresponding to the light flow area between the front frame and the rear frame based on the second light flow graph between the front frame and the rear frame;
Determining second behavior information of the second monitoring target relative to the first monitoring target according to all the second comprehensive light loss;
and determining final behavior information of the second monitoring target relative to the first monitoring target according to the first behavior information and the second behavior information.
5. The behavior recognition method according to claim 4, wherein obtaining a second integrated light loss amount corresponding to the light flow area between the front and rear frames based on the second light flow graph between the front and rear frames, comprises:
extracting a second preset number of second motion vectors from the second optical flow map;
selecting a second motion vector with the vector length being greater than a third preset length as a second target motion vector;
if the ratio of the total number of the second target motion vectors to the second preset number is larger than a third preset ratio, averaging all the second target motion vectors to obtain a second average vector;
and if the vector length of the second average vector is greater than a fourth preset length, taking the second average vector as a second comprehensive light loss amount corresponding to the light flow area between the front frame and the rear frame.
6. The behavior recognition method according to claim 4, wherein determining second behavior information of the second monitoring target with respect to the first monitoring target based on all of the second integrated light loss amounts includes:
Determining a second included angle between each second comprehensive light loss and the orthogonal direction of the first monitoring target; the orthogonal direction of the first monitoring target is a direction perpendicular to the direction of the first monitoring target;
if all the second included angles are smaller than a third preset included angle, determining that second behavior information of the second monitoring target relative to the first monitoring target is in a close state;
if all the second included angles are larger than a fourth preset included angle, determining that second behavior information of the second monitoring target relative to the first monitoring target is in an evacuation state; wherein the fourth preset included angle is larger than the third preset included angle;
and if part of the second included angles are larger than or equal to the third preset included angles, and/or part of the second included angles are smaller than or equal to the fourth preset included angles, determining that the second behavior information of the second monitoring target relative to the first monitoring target is in a static state.
7. The behavior recognition method according to any one of claims 1 to 4, wherein determining final behavior information of the second monitoring target with respect to the first monitoring target based on the first behavior information and the second behavior information, comprises:
If the first behavior information and the second behavior information are both in a close state, determining that final behavior information of the second monitoring target relative to the first monitoring target is in a close state;
if the first behavior information and the second behavior information are both in an evacuation state, determining that final behavior information of the second monitoring target relative to the first monitoring target is in the evacuation state;
and if the first behavior information and the second behavior information are inconsistent, determining that the final behavior information of the second monitoring target relative to the first monitoring target is in a static state.
8. The behavior recognition method according to any one of claims 1-4, further comprising, before acquiring the consecutive first spacer frame images in the current scene:
acquiring continuous third interval frame images in a current scene;
if the first monitoring target and the second monitoring target exist in the third interval frame image, generating a detection frame of the first monitoring target and a detection frame of the second monitoring target;
determining key points of the first monitoring target and a foreground area of a preset part in the first monitoring target based on a detection frame of the first monitoring target;
Determining the direction of the first monitoring target according to the key points of the first monitoring target;
determining the overlapping degree of the detection frame of the second monitoring target and the foreground region of the preset part;
if the orientation of the first monitoring target is located in a first visual field range and the overlapping degree is greater than a first preset overlapping degree, or if the orientation of the first monitoring target is located in a second visual field range and the overlapping degree is greater than a second preset overlapping degree, taking a region corresponding to a detection frame of the second monitoring target as the optical flow region;
the first preset overlapping degree is smaller than the second preset overlapping degree.
9. A behavior recognition apparatus comprising a processor and a storage device adapted to store a plurality of program code adapted to be loaded and executed by the processor to perform the behavior recognition method of any one of claims 1-8.
10. A computer readable storage medium, characterized in that a plurality of program codes are stored, which are adapted to be loaded and executed by a processor to perform the behavior recognition method of any one of claims 1 to 8.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310331466.9A CN116206262A (en) | 2023-03-30 | 2023-03-30 | Behavior recognition method, apparatus and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310331466.9A CN116206262A (en) | 2023-03-30 | 2023-03-30 | Behavior recognition method, apparatus and storage medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN116206262A true CN116206262A (en) | 2023-06-02 |
Family
ID=86509640
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202310331466.9A Pending CN116206262A (en) | 2023-03-30 | 2023-03-30 | Behavior recognition method, apparatus and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN116206262A (en) |
-
2023
- 2023-03-30 CN CN202310331466.9A patent/CN116206262A/en active Pending
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN109607031B (en) | Intelligent warehousing system and method based on unmanned aerial vehicle panorama | |
| CN108364466B (en) | Traffic flow statistical method based on unmanned aerial vehicle traffic video | |
| CN114049624B (en) | Ship cabin intelligent detection method and system based on machine vision | |
| CN113947731B (en) | Foreign matter identification method and system based on contact net safety inspection | |
| CN107139666A (en) | Obstacle detouring identifying system and method | |
| CN112836683B (en) | License plate recognition method, device, equipment and medium for portable camera equipment | |
| CN112200911A (en) | Region overlapping type three-dimensional map construction method and device combined with markers | |
| CN112507760A (en) | Method, device and equipment for detecting violent sorting behavior | |
| CN115909223A (en) | Method and system for matching WIM system information with monitoring video data | |
| CN112802112B (en) | Visual positioning method, device, server and storage medium | |
| US20130093880A1 (en) | Height Measurement Apparatus And Method | |
| CN112489240B (en) | Commodity display inspection method, inspection robot and storage medium | |
| CN112686162B (en) | Method, device, equipment and storage medium for detecting clean state of warehouse environment | |
| CN113449688A (en) | Power transmission tree obstacle recognition system based on image and laser point cloud data fusion | |
| CN112802058A (en) | Method and device for tracking illegal moving target | |
| CN112802100A (en) | Intrusion detection method, device, equipment and computer readable storage medium | |
| CN116206262A (en) | Behavior recognition method, apparatus and storage medium | |
| CN111951328A (en) | Object position detection method, device, equipment and storage medium | |
| CN112016518A (en) | Crowd distribution form detection method based on unmanned aerial vehicle and artificial intelligence | |
| CN115984374A (en) | Fault positioning and tracking method and device for photovoltaic power station and electronic equipment | |
| CN115953759A (en) | Method and device for detecting parking space limiter, electronic equipment and storage medium | |
| CN110827348A (en) | Self-checking processing method, device and equipment for position of camera equipment and storage medium | |
| CN117132597B (en) | Image recognition target positioning method and device and electronic equipment | |
| CN105447496A (en) | Docking airplane model identification and verification method and system based on machine vision | |
| CN114998836B (en) | Panoramic monitoring method and device for airport runway |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |