WO2022227490A1 - 行为识别方法、装置、设备、存储介质、计算机程序及程序产品 - Google Patents

行为识别方法、装置、设备、存储介质、计算机程序及程序产品 Download PDF

Info

Publication number
WO2022227490A1
WO2022227490A1 PCT/CN2021/130036 CN2021130036W WO2022227490A1 WO 2022227490 A1 WO2022227490 A1 WO 2022227490A1 CN 2021130036 W CN2021130036 W CN 2021130036W WO 2022227490 A1 WO2022227490 A1 WO 2022227490A1
Authority
WO
WIPO (PCT)
Prior art keywords
frame
sequence
detection frame
detection
image
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Ceased
Application number
PCT/CN2021/130036
Other languages
English (en)
French (fr)
Inventor
苏海昇
苏婧
王栋梁
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Shanghai Sensetime Intelligent Technology Co Ltd
Original Assignee
Shanghai Sensetime Intelligent Technology Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Shanghai Sensetime Intelligent Technology Co Ltd filed Critical Shanghai Sensetime Intelligent Technology Co Ltd
Publication of WO2022227490A1 publication Critical patent/WO2022227490A1/zh
Anticipated expiration legal-status Critical
Ceased legal-status Critical Current

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V20/00—Scenes; Scene-specific elements
    • G06V20/40—Scenes; Scene-specific elements in video content
    • G06V20/41—Higher-level, semantic clustering, classification or understanding of video scenes, e.g. detection, labelling or Markovian modelling of sport events or news items
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F18/00—Pattern recognition
    • G06F18/20—Analysing
    • G06F18/24—Classification techniques
    • G06F18/241—Classification techniques relating to the classification model, e.g. parametric or non-parametric approaches
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N20/00—Machine learning
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/20—Image preprocessing
    • G06V10/25—Determination of region of interest [ROI] or a volume of interest [VOI]
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V20/00—Scenes; Scene-specific elements
    • G06V20/40—Scenes; Scene-specific elements in video content
    • G06V20/48—Matching video sequences

Definitions

  • the present disclosure relates to the field of computer vision, and relates to, but is not limited to, behavior recognition methods, apparatuses, devices, storage media, computer programs, and program products.
  • the behavior recognition method in the related art first performs full-image data enhancement or other preprocessing on the input video sequence, and then directly sends it to the classification model for prediction. That is to say, input the entire video sequence to the network model, and let the network model perceive the location of the executive body where the behavior occurs.
  • This method relies on the premise that the execution subject accounts for a large proportion of the screen, and is only suitable for video behavior recognition centered on the execution subject. This type of data is commonly found in public video academic datasets.
  • Embodiments of the present disclosure provide a behavior recognition method, apparatus, device, storage medium, computer program, and program product.
  • an embodiment of the present disclosure provides a method for behavior recognition, the method comprising: acquiring a detection result of each object in each frame of an image in a video sequence to be recognized; generating at least one detection result according to the detection result of each object A first sequence; wherein, each of the first sequences is a track sequence including the same object; performing behavior recognition on each of the first sequences to obtain a behavior recognition result of each of the objects.
  • the detection result of each object in each frame of image is obtained first to determine the exact trajectory sequence of a single object in the video sequence to be recognized, that is, the first sequence, and then behavior recognition is performed on at least one first sequence.
  • the single-frame behavior recognition algorithm is extended to a behavior recognition algorithm with time sequence relationship. Converting the video classification problem into the identification problem of trajectory sequences can reduce the recognition difficulty of the model and improve the acquisition of effective information.
  • the boundary of the video understanding model is expanded, the potential of the video understanding model is brought into play, and it is also the embodiment of the application of the existing video understanding model.
  • an embodiment of the present disclosure provides a behavior recognition device, including an acquisition module, a first determination module, and an identification module, wherein: the acquisition module is configured to acquire each object in each frame of image in the video sequence to be recognized The first determination module is configured to determine each first sequence in the to-be-identified video sequence according to the detection result of each of the objects; the identification module is configured to determine each of the first A sequence of behavior recognition is performed to obtain a behavior recognition result for each of the objects.
  • an embodiment of the present disclosure provides a behavior recognition device, including a memory and a processor, where the memory stores a computer program that can be run on the processor, and the processor implements the behavior recognition method when the processor executes the program steps in .
  • an embodiment of the present disclosure provides a computer-readable storage medium on which a computer program is stored, and when the computer program is executed by a processor, implements the steps in the above-mentioned behavior identification method.
  • the present disclosure further provides a computer program, comprising computer-readable code, when the computer-readable code is executed in an electronic device, the processor in the electronic device executes the program for implementing the above-mentioned first aspect, or steps in any possible implementation manner of the first aspect.
  • the present disclosure also provides a computer program product, the computer program product comprising one or more instructions, the one or more instructions being adapted to be loaded by a processor and execute the first aspect above, or the first aspect steps in any of the possible implementations.
  • FIG. 1A is a schematic diagram of a network architecture for behavior recognition provided by an embodiment of the present disclosure
  • FIG. 1B is a schematic flowchart of a behavior recognition method provided by an embodiment of the present disclosure
  • FIG. 2 is a schematic flowchart of determining a video sequence to be identified according to an embodiment of the present disclosure
  • FIG. 3 is a schematic flowchart of another behavior recognition method provided by an embodiment of the present disclosure.
  • FIG. 4 is a schematic flowchart of generating at least one first sequence according to an embodiment of the present disclosure
  • FIG. 5 is a schematic flowchart of still another behavior recognition method provided by an embodiment of the present disclosure.
  • 6A is a system block diagram of a single-person behavior recognition algorithm provided by an embodiment of the present disclosure.
  • 6B is a logical flow chart of a single-person behavior recognition algorithm provided by an embodiment of the present disclosure
  • 6C is a schematic diagram of a sample input trajectory sequence after preprocessing provided by an embodiment of the present disclosure.
  • 6D is a schematic diagram of the effect of spatial motion mask cropping provided by an embodiment of the present disclosure.
  • FIG. 7 is a schematic diagram of the composition and structure of a behavior recognition device according to an embodiment of the present disclosure.
  • FIG. 8 is a schematic diagram of a hardware entity of a behavior recognition device according to an embodiment of the present disclosure.
  • first ⁇ second ⁇ third involved in the embodiments of the present disclosure is only to distinguish similar objects, and does not represent a specific ordering of objects. It can be understood that “first ⁇ second ⁇ third” "Where permitted, the specific order or sequence may be interchanged to enable the embodiments of the disclosure described herein to be practiced in sequences other than those illustrated or described herein.
  • Behavior recognition in video is an important application in the field of computer vision, which has a wide range of applications in the field of smart cities.
  • video sources are becoming more and more abundant.
  • the utilization of these video sources only stays on the ability to view or check, and most of the decision-making ability still requires a lot of manpower.
  • computing power brought by cloud computing and the improvement of intelligence brought by deep learning the advantages of deep learning can be fully utilized to enable the equipment that collects video sources to have certain automated decision-making capabilities.
  • the embodiments of the present disclosure can be applied in dangerous areas to detect unsafe accidents in time, such as whether someone falls or a small animal falls into the water, so as to take protective measures in time.
  • FIG. 1A is a schematic diagram of a network architecture of behavior recognition provided by an embodiment of the present disclosure.
  • the network architecture includes: a camera 101, an object detection module 102, a preprocessing module 103, and a video recognition module 104: wherein, The object detection module 102 , the preprocessing module 103 and the video recognition module 104 may be provided in the server 100 .
  • the camera 101 establishes a communication connection with the server 100 through a network.
  • the camera 101 collects the video in a specific scene, and then samples the video sequence 11 to be recognized, that is, multi-frame images including the target object, and inputs the video sequence 11 to be recognized into the object detection module 102; the object detection module 102 can make full use of correlation detection Algorithms such as inter-frame difference method, background subtraction method, optical flow method, etc. realize the positioning and analysis of the target object in the video sequence 11 to be recognized, and obtain a multi-frame image 12 with the detection result (the detection frame and object identification of the target object).
  • correlation detection Algorithms such as inter-frame difference method, background subtraction method, optical flow method, etc.
  • the preprocessing module 103 the multi-frame image 12 with the detection result is processed, and based on the detection frame and the object identification of each object in each frame image, at least one first sequence 13 is generated, and the first sequence is a single The trajectory sequence of the object; input at least one first sequence 13 into the video recognition module 104; the video recognition module 104 can make full use of the relevant video understanding model to perform behavior recognition for each first sequence, and finally output the video-level recognition result.
  • a framework of behavior recognition method including object localization and behavior recognition can be designed.
  • the object localization stage can flexibly use all existing object detection algorithms, and the behavior recognition stage makes full use of the existing video understanding model, expands the boundaries of the video understanding model, exerts the potential of the video understanding model, and also implements the existing video understanding model. manifestation of the application.
  • FIG. 1B is a schematic flowchart of a behavior recognition method provided by an embodiment of the present disclosure. As shown in FIG. 1B , the method includes at least the following steps:
  • Step S110 acquiring the detection result of each object in each frame of image in the video sequence to be recognized.
  • the video sequence to be identified is obtained by sampling the video source.
  • Each frame of image in the video source usually covers a wide field of view and contains more information, such as pedestrians, vehicles, animals, buildings and other complex background information.
  • the video sequence to be recognized is a frame sequence composed of multiple frames of images, wherein each frame of image may contain at least one object or no object, and the objects contained in different frame images may not necessarily be the same.
  • the object here may be a pedestrian, or a moving vehicle, an animal, etc., which is determined according to the actual scene in the implementation, which is not limited in this embodiment of the present disclosure.
  • each object in the image may be an object that meets certain conditions.
  • the object meeting certain conditions may be: the proportion of the image area where the object is located in the image screen is greater than a certain proportion threshold. , or the type of the object belongs to a preset type, etc. For example, the image area where the object is located occupies more than 10% of the image, and the type of the object belongs to a moving object, such as an animal, and a freely movable machine such as Sweeper etc.
  • the objects in the image can be considered as the execution subjects of potential behaviors. Therefore, when the behavior recognition method adopts the classification model to recognize, the classification model will predict each object in the image.
  • the detection and localization analysis of objects in video images can be realized through relevant image or video processing technologies.
  • object detection algorithms preprocess the video sequence to be identified to obtain multiple frames of images with detection frames, and then extract the images in each frame.
  • the detection frame of the object may be implemented by an inter-frame difference method, a background subtraction method, an optical flow method, etc. for moving object detection in a video, which is not limited in this embodiment of the present disclosure.
  • the detection result of the object includes at least a detection frame of the object and an object identifier associated with the detection frame.
  • the detection result of each object and the unique object identifier associated with the detection frame will be obtained after object detection. That is, the detection frames of different objects are distinguished by the corresponding object identifiers, so that the subject of the behavior can be automatically identified and processed in time.
  • Step S120 generating at least one first sequence according to the detection result of each of the objects.
  • the first sequence is a sequence of trajectories including the same object, that is, a sequence of positions of a single behavior subject within the time period that the video sequence to be recognized lasts for.
  • the detection frames belonging to each single object are screened out, and then the detection frames of each single object are spliced in time sequence to generate a first sequence.
  • an accurate behavior occurrence interval that is, the location interval where the detection frame is located, can be obtained, thereby further generating each first sequence.
  • Step S130 Perform behavior recognition on each of the first sequences to obtain a behavior recognition result of each of the objects.
  • one or more first sequences obtained in the previous step are sent together into the behavior recognition model to obtain the discrimination scores of different behaviors. That is to say, perform independent behavior recognition on the behavioral trajectory sequences of multiple single objects in the video sequence to be recognized, and confirm the discriminant score of each first sequence after passing through the network model. If there is at least one discriminant score higher than a given The threshold value of , output the behavior of the single object corresponding to the first sequence, such as an old man falling, a child lying down, a puppy falling into the water, etc.; if there is no discriminant score higher than a given threshold, Then it is determined that there is no behavior of a single object in the video sequence to be recognized.
  • step S110 the relevant detection algorithm can be flexibly used to accurately locate and analyze the behavior subject, and in the behavior recognition process in step S130, the existing video understanding model can be fully utilized, and the two steps are combined to construct an object-based model.
  • the two-stage behavior recognition algorithm framework of the positioning stage and the action recognition stage can be adapted to more video information that covers a large field of view and contains more information, such as video sources obtained from outdoor urban street scenes and indoor rail transit scenes. .
  • the detection result of each object in each frame of image in the video sequence to be recognized is obtained; then, according to the detection result of each object, at least one first image in the video sequence to be recognized is generated. a sequence; finally, perform behavior recognition on each of the first sequences to obtain a behavior recognition result of each of the objects; in this way, by acquiring the detection results of each object in each frame of images, it is determined that a single object is waiting Identify the exact trajectory sequence in the video sequence, and then perform behavior recognition for each first sequence, and expand the single-frame behavior recognition algorithm into a behavior recognition algorithm with a time sequence relationship. Therefore, the video classification problem is converted into the identification problem of trajectory sequence, which can reduce the recognition difficulty of the model and improve the acquisition of effective information.
  • the boundary of the video understanding model is expanded, the potential of the video understanding model is brought into play, and it is also the embodiment of the application of the existing video understanding model.
  • FIG. 2 is a schematic flowchart of determining a video sequence to be identified according to an embodiment of the present disclosure. As shown in FIG. 2 , the process at least includes the following steps:
  • Step S210 select at least two frames of images from the video source.
  • the specific time interval is the sampling interval corresponding to the detection period/frequency allowed by the recognition model deployed online (for example, a detection frequency of once every 3 seconds) and the set number of sampling frames. For example, if 8 frames are sampled, the specific time interval The interval is 3/8 second. In the implementation, it may be set according to the actual situation, which is not limited in this embodiment of the present disclosure.
  • Step S220 storing the at least two frames of images into the frame sequence buffer.
  • the length of the frame sequence buffer is fixed, that is, the number of sampling frames of the video sequence to be detected is fixed.
  • Step S230 acquiring a frame image in the frame sequence buffer as the to-be-identified video sequence.
  • each frame of image is obtained through a specific time interval and stored in a fixed-length frame sequence buffer, and when the number of stored image frames exceeds the longest length of the frame sequence buffer, the oldest frame sequence buffer in the frame sequence buffer is stored.
  • a frame of image pops up, forming a sliding window of fixed length.
  • the embodiment of the present disclosure constructs a sliding-window frame sequence buffer by sampling multiple frames of images in a video sequence at specific time intervals. On the one hand, it can efficiently process offline long videos and online video streams, and process video frames in each window in parallel. sequence; on the other hand, using a fixed-size window as the basic processing unit can ensure the rationality of sampling and capture the global process of action execution.
  • the detection result of the object includes at least a detection frame of the object and an object identifier associated with the detection frame;
  • FIG. 3 is a schematic flowchart of still another behavior recognition method according to an embodiment of the present disclosure. 3, the above step S120 "generate at least one first sequence according to the detection result of each of the objects" can be realized by the following steps:
  • Step S310 from the detection frame of each object, select a detection frame that is independent in spatial position as a candidate detection frame.
  • the spatially independent detection frame may be a detection frame in a sparse area in each frame of image, or a detection frame with no overlapping portion with other detection frames.
  • each frame of image includes a detection frame, indicating that the object corresponding to the detection frame is a single object, and single object behavior is prone to occur.
  • each frame of image includes at least two detection frames, indicating that the to-be-recognized video sequence contains multiple objects, and the spatial positions of these multiple objects in each frame of image may be gathered near the same position, or may be relatively scattered. Distributed in different locations, it can also be partially aggregated and partially dispersed. The embodiment of the present disclosure expects to find out the detection frame of the scattered single object for subsequent processing.
  • the Intersection over Union (IoU) between each two detection boxes is the result of dividing the overlapping part of the two regions by the aggregate part of the two regions.
  • the value of (i, j) in the adjacency matrix represents the intersection ratio of detection frame i and detection frame j in the frame image. Take the number of values greater than 0 in the i-th row (except itself) as the number of matches for detection frame i.
  • each frame of image includes at least two detection frames
  • the detection frame of each object is expanded outward by a specific ratio to obtain the expanded detection frame; Screen at least two first detection frames from the expanded detection frame; wherein, the area of the first detection frame is larger than other detection frames in the expanded detection frame; then, determine the at least two first detection frames An intersection ratio between detection boxes.
  • the specific ratio is 1.5 times
  • the length and width of the detection frame of each of the objects are expanded by 1.5 times. This can increase the image resolution, so that the intersection ratio between overlapping detection frames can be better calculated.
  • Step S320 generating the at least one first sequence according to the candidate detection frame and the object identifier associated with the candidate detection frame.
  • the candidate detection frame may appear in each frame of the video sequence to be identified, or may only appear in a part of the frames.
  • Each first sequence can be generated by connecting all candidate detection frames associated with each object identifier appearing in the video sequences to be identified in chronological order.
  • the candidate detection frame is selected from the at least two first detection frames, and the candidate detection frame is first indented according to the specific ratio to obtain a candidate of the original size detecting a frame; and then generating the at least one first sequence based on the candidate detection frame of the original size and an object identifier associated with the candidate detection frame.
  • the candidate detection frames selected from the at least two first detection frames need to be indented according to the original scale before each first sequence is generated to ensure that the trajectory sequence that finally enters the behavior recognition stage is the original motion of a single object trajectories to avoid extra computation.
  • FIG. 4 is a schematic flowchart of generating at least one first sequence provided by an embodiment of the present disclosure. As shown in FIG. 4 , each of the first sequences may be determined by the following steps:
  • Step S3201 taking a union of the candidate detection frames associated with each of the object identifiers in the spatial position to obtain a minimum bounding box corresponding to each of the object identifiers.
  • the spatial position of the candidate detection frame on different frame images may be moved, and a larger area is merged into a larger area as the minimum bounding frame by taking the union.
  • the pedestrian detection frame belonging to the same ID (Identity document, identity) in the video frame sequence in the sliding window is merged in the spatial position to obtain the minimum bounding box of the pedestrian in the video frame sequence.
  • Step S3202 according to the minimum bounding box, intercept an area image corresponding to the minimum bounding box in each frame of image.
  • the regional image in each frame of image is intercepted with the smallest bounding box, which can avoid the loss of the relative position of the behavior subject, and has a good performance improvement for behavior recognition with similar space but different motion rhythms.
  • the candidate detection frames belonging to the specific object in other frame images in the video frame sequence can be used to perform the union operation to obtain the minimum bounding box belonging to the specific object, and the minimum bounding box can be used to simultaneously intercept the area image of the solid color background. .
  • the solid-color background is cropped by the minimum bounding box calculated by the pedestrian in the first 7 frames, so as to generate the final trajectory sequence of the pedestrian based on all cropped area images, which effectively solves the problem of target loss that is prone to target loss based on single-person continuous detection.
  • Step S3203 Determine the noise region in the region image except the candidate detection frame.
  • the area except the candidate detection frame in the clipped area image has little significance to the recognition result of behavior recognition, and can be used as a noise area.
  • Step S3204 filling the noise area with a solid color background to obtain an image of the target area.
  • the relative motion information is lost.
  • the surrounding noise area is filled with a solid color background, that is, a spatial motion mask is used to maintain the spatial relative position information of the behavior subject.
  • Step S3205 connect the target area images in sequence according to the time stamp of each frame of images to obtain a first sequence corresponding to each of the object identifiers.
  • the target area images belonging to each object identifier are connected according to the time stamp of each frame of images, that is, the first sequence corresponding to each object identifier is obtained.
  • the embodiment of the present disclosure obtains the minimum bounding box of each single object by taking the union of the selected candidate detection frames of a single object in the spatial position, so as to intercept the regional image of each frame image according to the minimum bounding box, and analyze the regional image The noise background is filled, and the processed region images are finally connected to obtain each first sequence.
  • the video sequence to be recognized is divided into the trajectory sequences of several target persons, and the video classification problem is converted into the recognition problem of the trajectory sequence, which can reduce the recognition difficulty of the model and improve the efficiency of effective information.
  • the spatial motion mask is used to solve the problem of losing relative motion information in previous algorithms, especially for running, walking and other actions that are similar in space but have different spatial motion rhythms.
  • FIG. 5 is a schematic flowchart of still another behavior recognition method provided by an embodiment of the present disclosure. As shown in FIG. 5 , the method includes the following steps:
  • Step S510 acquiring the detection result of each object in each frame of image in the video sequence to be recognized.
  • the detection result of each object in each frame of image that is, the detection frame of each object and the object identifier associated with the detection frame are obtained through the correlation detection algorithm.
  • Step S520 generating at least one first sequence according to the detection result of each of the objects.
  • the first sequence is a trajectory sequence including the same object.
  • the detection frames belonging to the same single object or the minimum enclosed area determined based on the detection frame are respectively connected to form the trajectory sequence of each single object.
  • Step S530 sending each of the first sequences into a behavior recognition model to obtain a discrimination score of each of the first sequences.
  • each first sequence is passed through the behavior recognition model to obtain its own classification result, that is, if the discriminant score is higher than a certain threshold, the output is abnormal; otherwise, it is normal.
  • Step S540 Determine the behavior recognition result of each of the objects according to the discrimination score of each of the first sequences.
  • any discrimination score is greater than a behavior threshold, it is determined that there is a single object behavior in the to-be-identified video sequence; or in the case that no discrimination score is greater than a specific threshold, it is determined that the to-be-identified video sequence There is no single object behavior.
  • Step S550 in the case that the discrimination score of the first sequence is greater than the behavior threshold, determine the spatial location and behavior category of the behavior of the single object corresponding to the corresponding first sequence.
  • the behavior of a single object in the video sequence to be recognized is identified through the previous step, and then according to the position coordinates of the detection frame corresponding to the single object in the frame image, the spatial position and behavior category of the behavior of the single object are marked.
  • the behavior category is the category that can be output by the trained behavior recognition model for a single object behavior, such as falling, climbing, and handing out flyers.
  • Step S560 Determine the alarm content according to the spatial location and the behavior category.
  • the marked spatial position and behavior category can be further used to generate alarm content.
  • Step S570 Send an alarm notification to the terminal device corresponding to the spatial location according to the alarm content.
  • the video sequence to be identified is divided into trajectory sequences of several objects, and the video classification problem is converted into the identification problem of the trajectory sequence, which can reduce the The recognition difficulty of the model is improved, and the acquisition of effective information is improved. After a life-threatening behavior occurs in outdoor urban street scenes and indoor rail transit scenes, the system will automatically identify the behavior subject and give an alarm, providing an efficient and convenient detection capability for those who need it.
  • the embodiment of the present disclosure takes the object as a pedestrian as an example for description.
  • the main task of single-person behavior recognition is to first detect a single pedestrian appearing in the video source, lock the behavior subject, and then continuously detect the behavior subject to identify specific behaviors. When the behavior subject performs behaviors that endanger life safety, the pedestrian needs to be warned.
  • Such a process enables the equipment that collects video sources to have autonomous and intelligent discrimination capabilities, which can greatly improve the efficiency of daily urban management.
  • the embodiment of the present disclosure utilizes a pedestrian detection algorithm to locate the behavior subject, and obtains an accurate behavior occurrence interval, thereby constructing a single-person behavior recognition algorithm.
  • FIG. 6A is a system block diagram of a single-person behavior recognition algorithm provided by an embodiment of the present disclosure. As shown in FIG. 6A , a video sequence 61 marked with a pedestrian detection frame is preprocessed to obtain the respective trajectory sequences of three single persons, that is, the trajectory sequence.
  • the trained behavior recognition network 62 respectively performs behavior recognition for each trajectory sequence, and obtains the discriminant score 621 corresponding to the trajectory sequence 611, the discriminant score 622 corresponding to the trajectory sequence 612, and the trajectory sequence. 613 corresponds to the discrimination score 623 , and finally, the discriminative scores of all the trajectory sequences in the video sequence 61 are combined to obtain the recognition result 63 of the video sequence 61 .
  • FIG. 6B is a logical flowchart of a single-person behavior recognition algorithm provided by an embodiment of the present disclosure. As shown in FIG. 6B , the flowchart includes the following steps:
  • Step S601 constructing a sliding-window video frame sequence buffer.
  • a frame (frame) image is acquired according to a preset sampling interval, and placed in a video frame sequence buffer (frame_buffer).
  • the sampling interval under a fixed number of sampling frames is specified according to the allowable detection period/frequency on the line (for example, a detection frequency of once every 3 seconds). For example, in the case of sampling 8 frames of images, the sampling interval is 3/8 seconds.
  • constructing a sliding-window video frame sequence buffer can efficiently process offline long video and online video streams, and process the video frame sequence in each window in parallel.
  • using a fixed-size window as the basic processing unit can ensure the rationality of sampling and capture the global process of action execution.
  • Step S602 define the number of pedestrian trajectory sequences.
  • the width and height of all pedestrian detection boxes are expanded by 1.5 times.
  • the adjacency matrix is obtained. For example, select the top 10 pedestrian detection frames by area, and construct a 10x10 adjacency matrix, where the value of the element (i, j) in the adjacency matrix represents the intersection ratio between detection frame i and detection frame j. If no pedestrian detection box exists, skip the step of generating the adjacency matrix.
  • the embodiment of the present disclosure analyzes and processes a pedestrian detection frame that does not overlap with other pedestrian detection frames, which can improve the efficiency of single-person behavior recognition.
  • Step S603 determining the trajectory sequence of the pedestrian.
  • the ID matching of the pedestrian detection frame between frames is performed on the sparse detection frame determined in the previous step. According to different IDs, the trajectory sequences of different pedestrians can be obtained.
  • FIG. 6C is a schematic diagram of a sample input trajectory sequence after preprocessing provided by an embodiment of the present disclosure. As shown in FIG. 6C , it is a trajectory sequence with a child as the execution subject. It can be seen that the trajectory sequence is based on 8 frames of images collected , and then use the minimum bounding box containing the child recognition result to connect the regional images obtained by intercepting 8 frames of images respectively.
  • step S604 the trajectory sequences of the plurality of pedestrians are sent into the behavior recognition model, and the recognition result is determined.
  • the trajectory sequences of multiple pedestrians determined in the previous step are sent into the behavior recognition model to obtain the discrimination scores of different behaviors. Observe the output score of each pedestrian's trajectory sequence after passing through the behavior recognition model. If it is higher than a given threshold, the corresponding single-person behavior is output. Otherwise, it means that there is no single-person behavior in the current video frame sequence.
  • the single-person behavior recognition algorithm provided by the embodiments of the present disclosure can be widely used in behavior recognition in intelligent video analysis. For example, after a life-threatening behavior occurs in outdoor urban street scenes and indoor rail transit scenarios, the system will automatically identify the behavior subject and alert the police, providing an efficient and convenient detection capability for those who need it.
  • a MaskAttention cropping method is designed to keep the pedestrian cropping without losing the relative motion information.
  • the cropped pedestrian area maintains its spatial position and is filled with a solid color background.
  • Figure 6D for the region image cropped with the minimum bounding box for each frame, the part of the region image except the pedestrian detection box is filled with a solid color background. Since the minimum bounding box is obtained based on the union of pedestrian detection boxes in multiple frames of images, the relative positions of the minimum bounding box and the pedestrian detection box are known, thereby maintaining the relative motion information of pedestrians.
  • the embodiment of the present disclosure makes full use of the existing pedestrian recognition algorithm and designs a two-stage single-person behavior recognition algorithm framework.
  • the pedestrian localization stage can flexibly use all existing pedestrian recognition algorithms, and the behavior recognition stage makes full use of the existing video understanding model. , which expands the boundaries of the video understanding model, exerts the potential of the video understanding model, and is also the embodiment of the application of the existing video understanding model.
  • Different from the previous single-frame behavior recognition algorithm by using the combination of video classification algorithm and detection algorithm, the single-frame behavior recognition algorithm is extended to a behavior recognition algorithm with a time sequence relationship.
  • the embodiments of the present disclosure simultaneously use the spatial motion mask cropping method to maintain the spatial relative position information of the behavior subject. Because the relative position of the behavior subject is lost due to pedestrian detection, the spatial motion mask is used to solve the problem of losing relative motion information in the previous algorithm. Especially for running, walking and other actions that are similar in space but different in spatial motion rhythm, it has a better performance improvement.
  • an embodiment of the present disclosure further provides a behavior recognition device, the recognition device includes each module included and each unit included in each module, which can be implemented by a processor in a behavior recognition device;
  • the processor can be a central processing unit (Central Processing Unit, CPU), a microprocessor (Micro Processing Unit, MPU), a digital signal processor (Digital Signal Processor) , DSP) or Field Programmable Gate Array (Field Programmable Gate Array, FPGA), etc.
  • CPU Central Processing Unit
  • MPU Micro Processing Unit
  • DSP Digital Signal Processor
  • FPGA Field Programmable Gate Array
  • FIG. 7 is a schematic diagram of the composition and structure of a behavior recognition device provided by an embodiment of the present disclosure.
  • the recognition device 700 includes an acquisition module 710, a first determination module 720, and an identification module 730, wherein:
  • the obtaining module 710 is configured to obtain the detection result of each object in each frame of image in the video sequence to be identified;
  • the first determining module 720 is configured to generate at least one first sequence according to the detection result of each of the objects; wherein each of the first sequences is a trajectory sequence including the same object;
  • the recognition module 730 is configured to perform behavior recognition on each of the first sequences to obtain a behavior recognition result of each of the objects.
  • the identification device 700 further includes a selection module, a storage module and a second determination module, wherein: the selection module is configured to select at least two frames of images from the video source according to a specific time interval;
  • the storage module is configured to store the at least two frames of images in a frame sequence buffer; wherein, the length of the frame sequence buffer is fixed;
  • the second determination module is configured to acquire the frame sequence buffer The frame image is used as the video sequence to be identified.
  • the detection result of the object includes at least a detection frame of the object and an object identifier associated with the detection frame;
  • the first determination module 720 includes a first determination sub-module and a second determination sub-module Sub-module, wherein: the first determination sub-module is configured to select a detection frame independent in spatial position from the detection frame of each object as a candidate detection frame; the second determination sub-module is configured as The at least one first sequence is generated based on the candidate detection frame and an object identification associated with the candidate detection frame.
  • each frame of image includes one detection frame
  • the first determination submodule is further configured to use one detection frame in each frame of image as the candidate detection frame.
  • each frame of image includes at least two detection frames
  • the first determination sub-module includes a first determination unit, a generation unit and a second determination unit, wherein: the first determination a unit, configured to determine the intersection ratio between at least two detection frames in each frame of image; the generating unit, configured to generate an adjacency matrix corresponding to each frame of image according to the intersection ratio ; the second determining unit is configured to use the detection frame with zero matching times in the adjacency matrix as the candidate detection frame.
  • the first determination sub-module further includes an expansion unit and a selection unit, wherein: the expansion unit is configured to expand the detection frame of each object outward by a specific proportion to obtain an expansion the expanded detection frame; the selection unit is configured to screen at least two first detection frames from the expanded detection frame; wherein, the area of the first detection frame is larger than other detection frames in the expanded detection frame detection frame; correspondingly, the first determining unit is further configured to determine the intersection ratio between the at least two first detection frames.
  • the second determining sub-module includes a third determining unit, an intercepting unit and a connecting unit, wherein: the third determining unit is configured to detect candidates associated with each of the object identifiers
  • the frame takes the union in the spatial position to obtain the minimum bounding box corresponding to each of the object identifiers; the intercepting unit is configured to intercept the minimum bounding box in each frame of the image and the minimum bounding box according to the minimum bounding box.
  • the area image corresponding to the frame; the connecting unit is configured to connect the area images in sequence according to the time stamp of each frame of the image to obtain a first sequence corresponding to each of the object identifiers.
  • the candidate detection frame is selected from the at least two first detection frames
  • the second determination sub-module further includes an indentation unit and a fourth determination unit, wherein: the an indentation unit, configured to indent the candidate detection frame according to the specific ratio to obtain a candidate detection frame of the original size; the fourth determination unit is configured to be based on the candidate detection frame of the original size and the The at least one first sequence is generated according to the object identifier associated with the candidate detection frame.
  • the second determination sub-module further includes a fifth determination unit and a filling unit, wherein: the fifth determination unit is configured to determine the region image except for the candidate detection frame the noise area; the filling unit is configured to fill the noise area with a solid color background to obtain an image of the target area; correspondingly, the connection unit is also configured to sequentially connect the time stamps of each frame of the image.
  • the target area image is obtained, and a first sequence corresponding to each of the object identifiers is obtained.
  • the identification module 730 includes an identification sub-module and a third determination sub-module, wherein: the identification sub-module is configured to input each of the first sequences into the behavior identification model, The discrimination score of each of the first sequences is obtained; the third determination sub-module is configured to determine the behavior recognition result of each of the objects according to the discrimination score of each of the first sequences.
  • the identification device 700 further includes a third determination module, a fourth determination module and a transmission module, wherein: the third determination module is configured to have a discrimination score greater than a behavior in the first sequence In the case of the threshold value, determine the spatial position and behavior category of the behavior of a single object corresponding to the corresponding first sequence; the fourth determining module is configured to determine the alarm content according to the spatial position and the behavior category; the sending module, It is configured to send an alarm notification to the terminal device corresponding to the spatial position according to the alarm content, so that the manager holding the terminal device can handle the behavior of the single object.
  • the third determination module is configured to have a discrimination score greater than a behavior in the first sequence In the case of the threshold value, determine the spatial position and behavior category of the behavior of a single object corresponding to the corresponding first sequence
  • the fourth determining module is configured to determine the alarm content according to the spatial position and the behavior category
  • the sending module It is configured to send an alarm notification to the terminal device corresponding to the spatial position according to the alarm content,
  • the above behavior recognition method is implemented in the form of a software function module and sold or used as an independent product, it may also be stored in a computer-readable storage medium.
  • the technical solutions of the embodiments of the present disclosure may be embodied in the form of software products that are essentially or contribute to related technologies.
  • the computer software products are stored in a storage medium and include several instructions to make
  • the behavior recognition device (which may be a smartphone with a camera, a tablet computer, etc.) performs all or part of the methods described in the various embodiments of the present disclosure.
  • the aforementioned storage medium includes: a U disk, a mobile hard disk, a read only memory (Read Only Memory, ROM), a magnetic disk or an optical disk and other media that can store program codes.
  • ROM Read Only Memory
  • an embodiment of the present disclosure provides a computer-readable storage medium on which a computer program is stored, and when the computer program is executed by a processor, implements the steps in any of the behavior recognition methods in the foregoing embodiments.
  • a chip is also provided, the chip includes a programmable logic circuit and/or program instructions, and when the chip is running, it is used to realize the behavior recognition in any of the above embodiments steps in the method.
  • a computer program product is also provided. When the computer program product is executed by the processor of the behavior recognition device, the computer program product is used to implement the behavior recognition method in any of the above embodiments. step.
  • An embodiment of the present disclosure further provides a computer program product, where the computer program product carries program codes, and the instructions included in the program codes can be used to execute steps in any of the behavior recognition methods in the foregoing method embodiments.
  • the above-mentioned computer program product can be specifically implemented by means of hardware, software or a combination thereof.
  • the computer program product is embodied as a computer storage medium, and in another optional embodiment, the computer program product is embodied as a software product, such as a software development kit (Software Development Kit, SDK), etc. Wait.
  • Embodiments of the present disclosure further provide a computer program, including computer-readable codes.
  • a processor in the electronic device executes any one of the foregoing method embodiments.
  • the behavior recognition method is not limited to:
  • FIG. 8 is a schematic diagram of a hardware entity of a behavior recognition device according to an embodiment of the present disclosure.
  • the recognition device 800 includes a memory 810 and a processor 820 , and the memory 810 stores data that can be stored on the processor 820 A running computer program, when the processor 820 executes the program, implements the steps in any of the behavior recognition methods described in the embodiments of the present disclosure.
  • the memory 810 is used to store instructions and applications executable by the processor 820, and can also cache data to be processed or processed by the processor 820 and each module in the recognition device 800 (eg, image data, audio data, voice communication data and Video communication data), which can be realized by flash memory (FLASH) or random access memory (Random Access Memory, RAM).
  • FLASH flash memory
  • RAM Random Access Memory
  • the processor 820 executes the program, the steps of any one of the above-mentioned behavior recognition methods are implemented.
  • the processor 820 generally controls the overall operation of the identification device 800 .
  • the above-mentioned processor can be a special purpose integrated circuit (Application Specific Integrated Circuit, ASIC), a digital signal processor (Digital Signal Processor, DSP), a digital signal processing device (Digital Signal Processing Device, DSPD), a programmable logic device (Programmable Logic At least one of Device, PLD), Field Programmable Gate Array (Field Programmable Gate Array, FPGA), Central Processing Unit (Central Processing Unit, CPU), controller, microcontroller, microprocessor.
  • ASIC Application Specific Integrated Circuit
  • DSP Digital Signal Processor
  • DSPD Digital Signal Processing Device
  • a programmable logic device Programmable Logic At least one of Device, PLD), Field Programmable Gate Array (Field Programmable Gate Array, FPGA), Central Processing Unit (Central Processing Unit, CPU), controller, microcontroller, microprocessor.
  • the electronic device implementing the function of the above-mentioned processor may also be other, which is not specifically limited in the embodiment of the present disclosure.
  • the above-mentioned computer storage medium/memory can be a read-only memory (Read Only Memory, ROM), a programmable read-only memory (Programmable Read-Only Memory, PROM), an erasable programmable read-only memory (Erasable Programmable Read-Only Memory, EPROM), Electrically Erasable Programmable Read-Only Memory (EEPROM), Magnetic Random Access Memory (FRAM), Flash Memory, Magnetic Surface Memory, CD-ROM, or CD-ROM (Compact Disc Read-Only Memory, CD-ROM) and other memories; it can also be a variety of behavior recognition devices including one or any combination of the above memories, such as mobile phones, computers, tablet devices, personal digital Assistant etc.
  • the disclosed apparatus and method may be implemented in other manners.
  • the device embodiments described above are only illustrative.
  • the division of the units is only a logical function division. In actual implementation, there may be other division methods.
  • multiple units or components may be combined, or Can be integrated into another system, or some features can be ignored, or not implemented.
  • the coupling, or direct coupling, or communication connection between the components shown or discussed may be through some interfaces, and the indirect coupling or communication connection of devices or units may be electrical, mechanical or other forms. of.
  • the unit described above as a separate component may or may not be physically separated, and the component displayed as a unit may or may not be a physical unit; it may be located in one place or distributed to multiple network units; Some or all of the units may be selected according to actual needs to achieve the purpose of the solutions of the embodiments of the present disclosure.
  • each functional unit in each embodiment of the present disclosure may be all integrated into one processing unit, or each unit may be separately used as a unit, or two or more units may be integrated into one unit; the above integration
  • the unit can be implemented either in the form of hardware or in the form of hardware plus software functional units.
  • the above-mentioned integrated units of the present disclosure are implemented in the form of software functional modules and sold or used as independent products, they may also be stored in a computer-readable storage medium.
  • the technical solutions of the embodiments of the present disclosure may be embodied in the form of software products that are essentially or contribute to related technologies.
  • the computer software products are stored in a storage medium and include several instructions to make The device automated test line performs all or part of the methods described in various embodiments of the present disclosure.
  • the aforementioned storage medium includes various media that can store program codes, such as a removable storage device, a ROM, a magnetic disk, or an optical disk.
  • the detection result of each object in each frame of image in the video sequence to be recognized is obtained; then, according to the detection result of each said object, at least one first sequence is generated; wherein, each said The first sequence is a sequence of trajectories including the same object; finally, behavior recognition is performed on each of the first sequences to obtain a behavior recognition result of each of the objects.
  • the single-frame behavior recognition algorithm is extended to a behavior recognition algorithm with time sequence relationship. Converting the video classification problem into the identification problem of trajectory sequences can reduce the recognition difficulty of the model and improve the acquisition of effective information.
  • the boundary of the video understanding model is expanded, the potential of the video understanding model is brought into play, and it is also the embodiment of the application of the existing video understanding model.

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Data Mining & Analysis (AREA)
  • Multimedia (AREA)
  • Software Systems (AREA)
  • General Engineering & Computer Science (AREA)
  • Evolutionary Computation (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Artificial Intelligence (AREA)
  • Evolutionary Biology (AREA)
  • Bioinformatics & Computational Biology (AREA)
  • Bioinformatics & Cheminformatics (AREA)
  • Computational Linguistics (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Medical Informatics (AREA)
  • Computing Systems (AREA)
  • Mathematical Physics (AREA)
  • Image Analysis (AREA)

Abstract

本公开实施例公开了一种行为识别方法、装置、设备、存储介质、计算机程序及程序产品，其中所述方法包括：获取待识别视频序列中每一帧图像中每一对象的检测结果；根据每一所述对象的检测结果，生成至少一个第一序列；其中，每一所述第一序列为包括同一对象的轨迹序列；对每一所述第一序列进行行为识别，得到每一所述对象的行为识别结果。

Description

行为识别方法、装置、设备、存储介质、计算机程序及程序产品
相关申请的交叉引用
本公开基于申请号为202110449968.2、申请日为2021年04月25日、申请名称为“行为识别方法及装置、设备和存储介质”的中国专利申请提出，并要求该中国专利申请的优先权，该中国专利申请的全部内容在此以全文引入的方式引入本公开。
技术领域
本公开涉及计算机视觉领域，涉及但不限定于行为识别方法、装置、设备、存储介质、计算机程序及程序产品。
背景技术
相关技术中的行为识别方法首先对输入的视频序列进行全图的数据增强或其他预处理，然后直接送入到分类模型中进行预测。也就是说，输入整个视频序列到网络模型，让网络模型去感知行为发生的执行主体的发生位置。这种方式依赖于执行主体占比画面很大的前提假设，只适用于以执行主体为中心的视频行为识别，该类数据常见于公开的视频学术数据集中。
然而在大规模视频数据下，行为发生的执行主体的数量不可预知，且视频中的每一帧图像往往包含更多的信息，覆盖的视野也更大，造成识别效果较低、计算量大。
发明内容
本公开实施例提供一种行为识别方法、装置、设备、存储介质、计算机程序及程序产品。
本公开实施例的技术方案是这样实现的：
第一方面，本公开实施例提供一种行为识别方法，所述方法包括：获取待识别视频序列中每一帧图像中每一对象的检测结果；根据每一所述对象的检测结果，生成至少一个第一序列；其中，每一所述第一序列为包括同一对象的轨迹序列；对每一所述第一序列进行行为识别，得到每一所述对象的行为识别结果。
这样，首先获取每一帧图像中每一对象的检测结果，以确定单个对象在待识别视频序列中准确的轨迹序列即第一序列，然后对至少一个第一序列进行行为识别。通过利用视频分类算法与检测算法的结合，将单帧行为识别算法拓展为一个具有时序关系的行为识别算法。将视频分类问题转换成轨迹序列的识别问题，能够减小模型的识别难度，并提升有效信息的获取。同时拓展了视频理解模型的边界，发挥了视频理解模型的潜力，也是对现有视频理解模型落地应用的体现。
第二方面，本公开实施例提供一种行为识别装置，包括获取模块、第一确定模块和识别模块，其中：所述获取模块，配置为获取待识别视频序列中每一帧图像中每一对象的检测结果；所述第一确定模块，配置为根据每一所述对象的检测结果，确定所述待识 别视频序列中每一第一序列；所述识别模块，配置为对每一所述第一序列进行行为识别，得到每一所述对象的行为识别结果。
第三方面，本公开实施例提供一种行为识别设备，包括存储器和处理器，所述存储器存储有可在处理器上运行的计算机程序，所述处理器执行所述程序时实现上述行为识别方法中的步骤。
第四方面，本公开实施例提供一种计算机可读存储介质，其上存储有计算机程序，该计算机程序被处理器执行时实现上述行为识别方法中的步骤。
第五方面，本公开还提供一种计算机程序，包括计算机可读代码，当所述计算机可读代码在电子设备中运行时，所述电子设备中的处理器执行用于实现上述第一方面，或第一方面中任一种可能的实施方式中的步骤。
第六方面，本公开还提供一种计算机程序产品，所述计算机程序产品包括一条或多条指令，所述一条或多条指令适于由处理器加载并执行上述第一方面，或第一方面中任一种可能的实施方式中的步骤。
附图说明
为了更清楚地说明本公开实施例中的技术方案，下面将对实施例描述中所需要使用的附图作简单地介绍，显而易见地，下面描述中的附图仅仅是本公开的一些实施例，对于本领域普通技术人员来讲，在不付出创造性劳动的前提下，还可以根据这些附图获得其它的附图，其中：
图1A为本公开实施例提供的行为识别的一种网络架构示意图；
图1B为本公开实施例提供的一种行为识别方法的流程示意图；
图2为本公开实施例提供的确定待识别视频序列的流程示意图；
图3为本公开实施例提供的又一种行为识别方法的流程示意图；
图4为本公开实施例提供的生成至少一个第一序列的流程示意图；
图5为本公开实施例提供的再一种行为识别方法的流程示意图；
图6A为本公开实施例提供的单人行为识别算法的系统框图；
图6B为本公开实施例提供的单人行为识别算法的逻辑流程图；
图6C为本公开实施例提供的预处理后的输入轨迹序列样例示意图；
图6D为本公开实施例提供的空间运动掩膜裁剪的效果示意图；
图7为本公开实施例提供的一种行为识别装置的组成结构示意图；
图8为本公开实施例提供的一种行为识别设备的硬件实体示意图。
具体实施方式
为使本公开实施例的目的、技术方案和优点更加清楚，下面将结合本公开实施例中的附图，对本公开实施例中的技术方案进行清楚、完整地描述，显然，所描述的实施例是本公开一部分实施例，而不是全部的实施例。以下实施例用于说明本公开，但不用来限制本公开的范围。基于本公开中的实施例，本领域普通技术人员在没有作出创造性劳动前提下所获得的所有其他实施例，都属于本公开保护的范围。
在以下的描述中，涉及到“一些实施例”，其描述了所有可能实施例的子集，但是可以理解，“一些实施例”可以是所有可能实施例的相同子集或不同子集，并且可以在不冲突的情况下相互结合。
需要指出，本公开实施例所涉及的术语“第一\第二\第三”仅仅是是区别类似的对象，不代表针对对象的特定排序，可以理解地，“第一\第二\第三”在允许的情况下可以互换特定的顺序或先后次序，以使这里描述的本公开实施例能够以除了在这里图示或描述的以外的顺序实施。
本技术领域技术人员可以理解，除非另外定义，这里使用的所有术语(包括技术术语和科学术语)，具有与本公开实施例所属领域中的普通技术人员的一般理解相同的意义。还应该理解的是，诸如通用字典中定义的那些术语，应该被理解为具有与现有技术的上下文中的意义一致的意义，并且除非像这里一样被特定定义，否则不会用理想化或过于正式的含义来解释。
视频中的行为识别是计算机视觉领域的一个重要应用，在智慧城市领域有着广泛的应用。随着现代通信能力的增强，在智慧城市的建设进程中，视频源越来越丰富。但是对这些视频源的利用仅仅停留在看或查的能力上，大部分决策能力还是需要大量的人力完成。随着云计算带来的算力提升，深度学习带来的智力的提升，可以充分利用深度学习的优势使得采集视频源的设备有一定的自动化决策能力。本公开实施例可应用于危险地带及时检测不安全事故如是否有人摔倒或者有小动物落水等行为，以便及时采取保护措施。
图1A为本公开实施例提供的行为识别的一种网络架构示意图，如图1A所示，该网络架构中包括：摄像头101、对象检测模块102、预处理模块103和视频识别模块104：其中，对象检测模块102、预处理模块103和视频识别模块104可以设置于服务器100中，为实现支撑一个示例性应用，摄像头101通过网络和服务器100建立通信连接。通过摄像头101采集特定场景下的视频，然后采样得到待识别视频序列11即包含目标对象的多帧图像，将待识别视频序列11输入到对象检测模块102中；对象检测模块102可以充分利用相关检测算法如帧间差分法、背景减除法、光流法等等实现对待识别视频序列11中目标对象的定位和分析，得到带检测结果(标注目标对象的检测框和对象标识)的多帧图像12；然后通过预处理模块103对带检测结果的多帧图像12进行处理，基于每一帧图像中每一对象的检测框和对象标识，生成至少一个第一序列13，该第一序列为的单个对象的轨迹序列；将至少一个第一序列13输入到视频识别模块104中；视频识别模块104可以充分利用相关视频理解模型对每一第一序列进行行为识别，最后输出视频层面的识别结果。基于该网络架构，可以设计一种包括对象定位和行为识别两个阶段的行为识别方法框架。其中对象定位阶段可以灵活的使用现有所有对象检测算法，行为识别阶段充分利用现有视频理解模型，拓展了视频理解模型的边界，发挥了视频理解模型的潜力，也是对现有视频理解模型落地应用的体现。
本公开实施例提供一种行为识别方法，应用于服务器、终端设备或其他设备。其中， 所述终端设备包括但不限于手机、笔记本电脑、平板电脑和掌上上网设备、多媒体设备、流媒体设备、移动互联网设备、可穿戴设备或其他类型的设备。图1B为本公开实施例提供的一种行为识别方法的流程示意图，如图1B所示，所述方法至少包括以下步骤：
步骤S110，获取待识别视频序列中每一帧图像中每一对象的检测结果。
这里，待识别视频序列为对视频源进行采样得到的。视频源中的每一帧图像，往往覆盖的视野比较广阔，包含较多的信息，如行人、车辆、动物、建筑物以及其他复杂的背景信息。
可以理解的，待识别视频序列为由多帧图像构成的帧序列，其中每一帧图像中可以能包含至少一个对象或者未包含对象，不同的帧图像包含的对象也不一定相同。这里的对象可以是行人，也可以是移动车辆、动物等，在实施中根据实际场景确定，本公开实施例对此不作限定。
可以理解的，图像中的每一对象可以是符合一定条件的对象，在实施的过程中，对象满足一定的条件可以是：对象所在的图像区域在图像画面中所占的比例大于一定的比例阈值，或者，对象的类型属于某一预设类型等的，例如对象所在的图像区域在图像画面中所占的比例大于10％，对象的类型属于移动对象，例如动物，可自由移动的机器设备如扫地机等。
图像中的对象可以认为是潜在的行为发生的执行主体，由此，行为识别方法采用分类模型识别时，分类模型将对图像中的每一对象进行预测。
可以通过相关图像或视频处理技术实现对视频图像中对象的检测和定位分析，例如对象检测算法对该待识别视频序列进行预处理，得到多帧带检测框的图像，然后提取每一帧图像中所述对象的检测框。其中，检测算法比如模板匹配，对于视频中运动目标检测，可以通过帧间差分法、背景减除法、光流法等等实现，本公开实施例对此不作限定。
这里，所述对象的检测结果至少包括对象的检测框和与所述检测框关联的对象标识。对于待识别视频序列中出现的不同对象，通过对象检测后会得到每一对象的检测结果和与检测框关联的唯一对象标识。即不同对象的检测框通过相应的对象标识区分，以便后续自动识别发生行为的主体并及时处理。
步骤S120，根据每一所述对象的检测结果，生成至少一个第一序列。
这里，所述第一序列为包括同一对象的轨迹序列，即单个行为主体在待识别视频序列所持续的时间段内的位置序列。
基于所述待识别视频序列中帧图像所包含对象的检测框，筛选出属于每一单个对象的检测框，然后将每一单个对象的检测框按时间顺序拼接即生成一个第一序列。
通过利用相关检测算法进行行为主体的定位，可以获得准确的行为发生区间即检测框所在的位置区间，从而进一步生成每一个第一序列。
步骤S130，对每一所述第一序列进行行为识别，得到每一所述对象的行为识别结果。
这里，将上一步骤得出的一个或多个第一序列一起送入行为识别模型中，获取不同行为的判别分数。也就是说，将待识别视频序列中的多个单个对象的行为轨迹序列进行 独立行为识别，确认每一第一序列经过网络模型后的判别分数，如果存在至少一个判别分数高于某个给定的阈值，则输出该第一序列对应的单个对象的行为，例如一个老人跌倒、一个小孩躺卧、一只小狗落水等行为；如果不存在任一个判别分数高于某个给定的阈值，则确定该待识别视频序列中不存在单个对象的行为。
需要说明的是，步骤S110中可以灵活的使用相关检测算法进行行为主体的准确定位和分析，步骤S130中的行为识别过程中可以充分利用现有视频理解模型，结合这两个步骤构建了基于对象定位阶段和行为识别阶段两阶段的行为识别的算法框架，可以适应于更多覆盖视野大、包含更多信息的视频信息，如室外的城市街道场景以及室内的轨道交通等场景下获取的视频源。
在本公开实施例中，首先，获取待识别视频序列中每一帧图像中每一对象的检测结果；然后，根据每一所述对象的检测结果，生成所述待识别视频序列中至少一个第一序列；最后，对每一所述第一序列进行行为识别，得到每一所述对象的行为识别结果；如此，通过获取每一帧图像中每一对象的检测结果，以确定单个对象在待识别视频序列中准确的轨迹序列，然后对每一第一序列进行行为识别，将单帧行为识别算法拓展为一个具有时序关系的行为识别算法。从而将视频分类问题转换成轨迹序列的识别问题，能够减小模型的识别难度，并提升有效信息的获取。同时拓展了视频理解模型的边界，发挥了视频理解模型的潜力，也是对现有视频理解模型落地应用的体现。
图2为本公开实施例提供的确定待识别视频序列的流程示意图，如图2所述，所述流程至少包括以下步骤：
步骤S210，按照特定时间间隔，从视频源中选取至少两帧图像。
这里，所述特定时间间隔为根据线上部署的识别模型所允许的检测周期/频率(例如3秒一次检测频率)和设定的采样帧数对应的采样间隔，例如采样8帧，则特定时间间隔为3/8秒。在实施中可以根据实际情况设定，本公开实施例对此不作限定。
步骤S220，将所述至少两帧图像存入帧序列缓冲区。
这里，所述帧序列缓冲区的长度固定，也就是固定需要检测的视频序列的采样帧数。
步骤S230，获取所述帧序列缓冲区中的帧图像作为所述待识别视频序列。
这里，通过特定时间间隔获取每一帧图像存入固定长度的帧序列缓冲区，并当存入的图像帧数超过帧序列缓冲区的最长长度时，将帧序列缓冲区中时间最早的第一帧图像弹出，从而形成长度固定的滑动窗。
本公开实施例通过间隔特定时间间隔采样视频序列中的多帧图像，构造滑窗式帧序列缓冲区，一方面是能高效处理离线长视频和在线视频流，并行处理每个窗口内的视频帧序列；另一方面，以固定大小的窗口作为基本处理单元可以保证采样的合理性，捕捉动作执行的全局过程。
在一些其他实施例中，所述对象的检测结果至少包括所述对象的检测框和与所述检测框关联的对象标识；图3为本公开实施例提供的又一种行为识别方法的流程示意图，如图3所述，上述步骤S120“根据每一所述对象的检测结果，生成至少一个第一序列” 可以通过以下步骤实现：
步骤S310，从每一所述对象的检测框中，选择在空间位置上独立的检测框作为候选检测框。
这里，所述在空间位置上独立的检测框可以为每一帧图像中稀疏区域的检测框，或者与其他检测框之间无重叠部分的检测框。
通常认为独立的对象容易发生单个对象行为，因此从每一帧图像包含的所有对象的检测框中，筛选出每一单个对象的候选检测框。
可以通过以下方式实现“从每一所述对象的检测框中，选择在空间位置上独立的检测框作为候选检测框”的过程：
方式一：在所述每一帧图像中包括一个检测框的情况下，将所述每一帧对象中的一个检测框作为所述候选检测框。
这里，每一帧图像中包括一个检测框说明该检测框对应的对象为单个对象，容易发生单个对象行为。
方式二：在所述每一帧图像中包括至少两个检测框的情况下，确定所述每一帧图像中的至少两个检测框之间的交并比；根据所述交并比，生成所述每一帧图像对应的邻接矩阵；将所述邻接矩阵中匹配次数为零的检测框作为所述候选检测框。
这里，每一帧图像中包括至少两个检测框说明该待识别视频序列中包含多个对象，这多个对象在每一帧图像中的空间位置可以聚集在同一位置附近，也可以相对分散的分布在不同位置，也可以部分聚集部分分散分布。本公开实施例期望找出其中分散的单个对象的检测框进行后续处理。
每两个检测框之间的交并比(Intersection over Union，IoU)是两个区域重叠的部分除以两个区域的集合部分得出的结果。邻接矩阵中(i,j)的值代表帧图像中检测框i和检测框j的交并比。将第i行中数值大于0的个数(除了自身)作为检测框i的匹配次数。
需要说明的是，如果邻接矩阵中不存在与检测框i之间的交并比大于0的其他检测框，那么检测框i的匹配次数为0，说明该检测框i与所在帧图像中其他检测框之间没有重叠区域，可以作为单个对象的候选检测框。
在一些其他的实施例中，在所述每一帧图像中包括至少两个检测框的情况下，首先对每一所述对象的检测框，向外扩展特定比例，得到扩展后的检测框；从所述扩展后的检测框中筛选至少两个第一检测框；其中，所述第一检测框的面积大于所述扩展后的检测框中其他检测框；然后，确定所述至少两个第一检测框之间的交并比。例如，在所述特定比例为1.5倍的情况下，将每一所述对象的检测框的长和宽都扩展1.5倍。这样能够增大图像分辨率，从而能更好地计算相互重叠的检测框之间的交并比。
步骤S320，根据所述候选检测框和与所述候选检测框关联的对象标识，生成所述至少一个第一序列。
这里，所述候选检测框可以在待识别视频序列中的每一帧图像中出现，也可以只在一部分帧中出现。将所有待识别视频序列中出现的与每一对象标识关联的候选检测框按 照时间顺序连接起来，即可生成每一第一序列。
在一些其他的实施例中，所述候选检测框为从所述至少两个第一检测框中选择的，首先对所述候选检测框，按照所述特定比例进行内缩，得到原始尺寸的候选检测框；然后根据所述原始尺寸的候选检测框和与所述候选检测框关联的对象标识，生成所述至少一个第一序列。这样，对于从所述至少两个第一检测框中选择的候选检测框，在生成每一第一序列之前需要按照原比例内缩，确保最终进入行为识别阶段的轨迹序列为单个对象原始的运动轨迹，避免产生额外的计算量。
在实施中，图4为本公开实施例提供的生成至少一个第一序列的流程示意图，如图4所示，可以通过以下步骤确定每一所述第一序列：
步骤S3201，对与每一所述对象标识关联的候选检测框在空间位置上取并集，得到与每一所述对象标识对应的最小包围框。
这里，对于所述候选检测框在不同帧图像上的空间位置可能是移动的，通过取并集合并成一个较大的区域作为最小包围框。例如，将滑窗内的视频帧序列中属于同一个ID(Identity document，身份标识)的行人检测框在空间位置上取并集，得到这个行人在视频帧序列中的最小包围框。
步骤S3202，根据所述最小包围框，截取所述每一帧图像中与所述最小包围框对应的区域图像。
这样，以最小包围框截取每一帧图像中的区域图像，能够避免行为主体的相对位置丢失，对于空间相似但运动节奏不同的行为识别有较好的性能提升。
值得注意的是，在截取区域图像的时候，可能存在某些帧图像中并未出现某个特定对象。这种情况下，可以用视频帧序列中其他帧图像中属于该特定对象的候选检测框进行取并集操作，得到属于该特定对象的最小包围框，并用最小包围框同时截取纯色背景的区域图像。
示例地，采样8帧图像，属于同一个ID的行人可能只出现在8帧中的前7帧，对于不出现该行人的第8帧用-1进行补全。即用该行人在前7帧中计算出的最小包围框对纯色背景进行裁剪，从而根据所有裁剪的区域图像生成该行人最终的轨迹序列，有效解决了基于单人持续检测容易出现目标丢失问题。
步骤S3203，确定所述区域图像中除去所述候选检测框之外的噪声区域。
这里，截取后的区域图像中除去候选检测框之外的区域，对行为识别的识别结果意义不大，可以作为噪声区域。
步骤S3204，利用纯色背景填充所述噪声区域，得到目标区域图像。
这里，由于在检测定位阶段只将单个对象的候选检测框提取出来，丢失了相对运动信息，例如对于跑步、走路这类单人行为动作，由于其动作形式和动作幅度相似，很难见其区分。本公开实施例对提取的检测框保持空间位置的同时，将周围的噪声区域用纯色背景填充，即利用空间运动掩膜(mask)保持行为主体的空间相对位置信息。
步骤S3205，按照所述每一帧图像的时间戳，依次连接所述目标区域图像，得到与 每一所述对象标识对应的第一序列。
这里，将属于每一对象标识的目标区域图像按照所述每一帧图像的时间戳连接起来，即得到与每一所述对象标识对应的第一序列。
本公开实施例通过对筛选的单个对象的候选检测框在空间位置上取并集，得到每个单个对象的最小包围框，从而根据最小包围框截取每一帧图像的区域图像，并对区域图像的噪声背景进行填充，最终连接处理后的区域图像得到每一第一序列。借助待识别视频序列的检测结果，将待识别视频序列拆分成若干个对象人的轨迹序列，将视频分类问题转换成轨迹序列的识别问题，能够减小模型的识别难度，并提升有效信息的获取。同时，利用空间运动掩膜解决以往算法丢失相对运动信息问题，特别是对于跑步、走路等空间相似但空间运动节奏不同的动作有较好的性能提升。
图5为本公开实施例提供的再一种行为识别方法的流程示意图，如图5所述，所述方法包括以下步骤：
步骤S510，获取待识别视频序列中每一帧图像中每一对象的检测结果。
这里，对于滑动窗口内的视频帧序列，通过相关检测算法获取每一帧图像中每一对象的检测结果即每一对象的检测框和与检测框关联的对象标识。
步骤S520，根据每一所述对象的检测结果，生成至少一个第一序列。
这里，所述第一序列为包括同一对象的轨迹序列。根据上一步骤获得的每一帧图像中每一对象的检测结果，将属于同一单个对象的检测框(或者基于检测框确定的最小包围区域)分别连接起来，形成每一单个对象的轨迹序列。
步骤S530，将每一所述第一序列送入到行为识别模型中，得到每一所述第一序列的判别分数。
这里，将每一第一序列经过行为识别模型，得到各自的分类结果，即如果判别分数高于某个特定阈值，则输出异常；否则为正常。
步骤S540，根据每一所述第一序列的判别分数，确定每一所述对象的行为识别结果。
这里，在存在任一判别分数大于行为阈值的情况下，确定所述待识别视频序列中存在单个对象行为；或者在不存在任一判别分数大于特定阈值的情况下，确定所述待识别视频序列为不存在单个对象行为。
步骤S550，在所述第一序列的判别分数大于行为阈值的情况下，确定相应第一序列对应的单个对象行为发生的空间位置和行为类别。
这里，通过上一步骤识别出待识别视频序列中的单个对象行为，然后根据该单个对象对应检测框在帧图像中的位置坐标，标记单个对象行为发生的空间位置和行为类别。行为类别为针对单个对象行为，经过训练好的行为识别模型能输出的类别，如跌倒、攀爬、发传单等行为。
步骤S560，根据所述空间位置和行为类别，确定告警内容。
这里，通过对单个对象行为的执行主体进行标记，可以进一步将标记的空间位置和行为类别生成告警内容。
步骤S570，根据告警内容向所述空间位置对应的终端设备发送告警通知。
可以理解的，不同的位置区域有对应的持有终端设备的管理人员，通过终端设备接收告警系统的通知，可以快速定位和处理该位置区域内发生的单个对象行为。
在本公开实施例中，借助待识别视频序列的每一对象的检测结果，将待识别视频序列拆分成若干个对象的轨迹序列，将视频分类问题转换成轨迹序列的识别问题，能够减小模型的识别难度，同时提升有效信息的获取。在室外的城市街道场景以及室内的轨道交通等场景下发生危害生命安全的行为后，系统就会自动识别出行为主体并告警，给相关需求的人员提供了一个高效而便捷的检测能力。
下面结合一个具体实施例对上述行为识别方法进行说明，然而值得注意的是，该具体实施例仅是为了更好地说明本公开，并不构成对本公开的不当限定。
本公开实施例以对象为行人为例进行说明。对于单人行为识别的主要任务是首先对出现在视频源中的单个行人进行检测，锁定行为主体，然后对该行为主体进行持续检测，识别出具体行为。当行为主体执行危害生命安全的行为时需要对该行人预警。这样的流程使得采集视频源的设备具有自主化和智能化的判别能力，可以给日常城市管理带来极大的效率提升。
传统的行为识别方法通常对输入的视频序列进行全图的数据增强或其他预处理后送入到分类模型中进行预测，然而这种方式只适配置为以人为中心的视频行为识别，该类数据常见于公开的视频学术数据集中。对于其他视频源中的图像来说，每帧图像中往往包含更多的信息，覆盖的视野也更大。为此，本公开实施例利用行人检测算法来进行行为主体的定位，并获得准确的行为发生区间，以此构建单人行为识别算法。
基于单人行为识别问题，本公开实施例通过调用上游检测组件获取视频图像中的行人检测框，基于行人检测框来生成多个行人检测框轨迹序列，对每个行人的轨迹序列进行行为识别。图6A为本公开实施例提供的单人行为识别算法的系统框图，如图6A所示，将标有行人检测框的视频序列61经过预处理，得到三个单人各自的轨迹序列即轨迹序列611、轨迹序列612和轨迹序列613，再经过已训练的行为识别网络62分别对每个轨迹序列进行行为识别，得到轨迹序列611对应的判别分数621、轨迹序列612对应的判别分数622和轨迹序列613对应的判别分数623，最后，再将视频序列61中所有的轨迹序列的判别分数进行合并，得到该视频序列61的识别结果63。
图6B为本公开实施例提供的单人行为识别算法的逻辑流程图，如图6B所示，该流程包括以下步骤：
步骤S601，构造滑窗式的视频帧序列缓冲区。
对于获取的一段待识别视频，根据预设的采样间隔获取一帧(frame)图像，并放入视频帧序列缓冲区(frame_buffer)。在实施中，根据线上允许的检测周期/频率(例如3秒一次检测频率)，明确固定采样帧数下的采样间隔，例如采样8帧图像的情况下，采样间隔为3/8秒。当该视频帧序列缓冲区中的图像帧数超过最大缓冲区范围(max_buffer_size)时，将该视频帧序列缓冲区中的第一帧图像弹出。
构造滑窗式视频帧序列缓冲区一方面能高效处理离线长视频和在线视频流，并行处理每个窗口内的视频帧序列。另一方面以固定大小的窗口作为基本处理单元可以保证采样的合理性，捕捉动作执行的全局过程。
步骤S602，定义行人轨迹序列的数量。
对于当前视频帧序列缓冲区内的每一帧图像，首先对所有的行人检测框的宽和高均扩展1.5倍。然后根据所有的行人检测框，得到邻接矩阵。例如按面积选取前10个行人检测框，构建10x10的邻接矩阵，其中邻接矩阵中元素(i,j)的值代表检测框i和检测框j之间的交并比。如果不存在行人检测框，则跳过生成邻接矩阵的步骤。
计算检测框i的匹配次数时，可以将第i行中邻接矩阵中(i,j)的值大于0的个数(忽略与自身的交并比)作为匹配次数。根据邻接矩阵，从每一帧图像中筛选出匹配次数为0的行人检测框作为稀疏检测框，从而每一帧图像中稀疏检测框的数量即为行人轨迹序列的数量。通常认为独立的行人容易发生单人行为，因此本公开实施例对与其他行人检测框之间无重叠的行人检测框进行分析处理，能够提高单人行为识别的效率。
步骤S603，确定行人的轨迹序列。
对上一步确定的稀疏检测框进行帧间行人检测框的ID匹配，根据不同的ID，可以得到不同行人的轨迹序列。
首先将确定的稀疏检测框内缩1.5倍来恢复分辨率。然后对于每个ID在当前视频帧序列缓冲区内所有帧图像的稀疏检测框，计算最小包围框，也就是将视频帧序列缓冲区中属于同一个行人的稀疏检测框在空间位置上取并集，得到该行人在当前视频帧序列中的最小包围框。最终将最小包围框裁剪(crop)每一帧图像出来对应的区域图像连接，生成每个行人的轨迹序列。图6C为本公开实施例提供的预处理后的输入轨迹序列样例示意图，如图6C所示，为以小孩为执行主体的轨迹序列，可以看出，该轨迹序列是基于采集的8帧图像，然后利用包含小孩识别结果的最小包围框，分别截取8帧图像得到的区域图像连接起来形成的。
对于该视频帧序列缓冲区中的所有帧图像，可能存在一些帧缺失同一ID的行人的情况，例如采样8帧图像，属于同一个ID的行人可能只出现在8帧中的前7帧，对于不出现该行人的第8帧用-1进行补全。即用该行人在前7帧中计算出的最小包围框对纯色背景进行裁剪，从而根据所有裁剪的区域图像生成该行人最终的轨迹序列。有效解决了基于单人持续检测容易出现目标丢失问题。
步骤S604，将多个行人的轨迹序列送入行为识别模型，确定识别结果。
将上一步确定的多个行人的轨迹序列送入行为识别模型中获取不同行为的判别分数。观察每个行人的轨迹序列经过行为识别模型后的输出分数，如果高于某个给定的阈值，则输出对应的单人行为，否则表示当前视频帧序列不存在单人行为。
本公开实施例提供的单人行为识别算法可以广泛应用于智能视频分析中的行为识别中。例如，在室外的城市街道场景以及室内的轨道交通等场景下发生危害生命安全的行为后，系统就会自动识别出行为主体并报警，给相关需求的人员提供了一个高效而便捷 的检测能力。
对于跑步，走路这类单人行为动作。由于其动作形式和动作幅度相似，很难见其区分。在行人检测框提取阶段，由于将行人裁剪出来，丢失了相对运动信息。为了解决这种相对运动信息丢失问题，设计了一种MaskAttention裁剪法来保持行人裁剪同时又不丢失相对运动信息，将裁减的行人区域保持其空间位置同时周围用纯色背景填充。如图6D所示，对于每帧用最小包围框裁剪的区域图像，该区域图像中除行人检测框外的部分都用纯色背景填充。由于最小包围框是根据多帧图像里的行人检测框取并集得到的，因此，最小包围框和行人检测框的相对位置已知，从而保持了行人的相对运动信息。
本公开实施例充分利用现有行人识别算法并设计了一套两阶段的单人行为识别算法框架，行人定位阶段可以灵活的使用现有所有行人识别算法，行为识别阶段充分利用现有视频理解模型，拓展了视频理解模型的边界，发挥了视频理解模型的潜力，也是对现有视频理解模型落地应用的体现。不同于以往的单帧行为识别算法，通过利用视频分类算法与检测算法的结合，将单帧行为识别算法拓展为一个具有时序关系的行为识别算法。
本公开实施例同时使用了空间运动掩膜裁剪法来保持行为主体的空间相对位置信息。由于行人检测导致行为主体的相对位置丢失，利用空间运动掩膜解决以往算法丢失相对运动信息问题。特别是对于跑步、走路等空间相似但空间运动节奏不同的动作有较好的性能提升。
基于前述的实施例，本公开实施例再提供一种行为识别装置，所述识别装置包括所包括的各模块、以及各模块所包括的各单元，可以通过行为识别设备中的处理器来实现；当然也可通过具体的逻辑电路实现；在实施的过程中，处理器可以为中央处理器(Central Processing Unit，CPU)、微处理器(Micro Processing Unit，MPU)、数字信号处理器(Digital Signal Processor，DSP)或现场可编程门阵列(Field Programmable Gate Array，FPGA)等。
图7为本公开实施例提供的一种行为识别装置的组成结构示意图，如图7所示，所述识别装置700包括获取模块710、第一确定模块720和识别模块730，其中：
所述获取模块710，配置为获取待识别视频序列中每一帧图像中每一对象的检测结果；
所述第一确定模块720，配置为根据每一所述对象的检测结果，生成至少一个第一序列；其中，每一所述第一序列为包括同一对象的轨迹序列；
所述识别模块730，配置为对每一所述第一序列进行行为识别，得到每一所述对象的行为识别结果。
在一些可能的实施例中，所述识别装置700还包括选取模块、存储模块和第二确定模块，其中：所述选取模块，配置为按照特定时间间隔，从视频源中选取至少两帧图像；所述存储模块，配置为将所述至少两帧图像存入帧序列缓冲区；其中，所述帧序列缓冲区的长度固定；所述第二确定模块，配置为获取所述帧序列缓冲区中的帧图像作为所述待识别视频序列。
在一些可能的实施例中，所述对象的检测结果至少包括所述对象的检测框和与所述检测框关联的对象标识；所述第一确定模块720包括第一确定子模块和第二确定子模块，其中：所述第一确定子模块，配置为从每一所述对象的检测框中，选择在空间位置上独立的检测框作为候选检测框；所述第二确定子模块，配置为根据所述候选检测框和与所述候选检测框关联的对象标识，生成所述至少一个第一序列。
在一些可能的实施例中，所述每一帧图像中包括一个检测框，所述第一确定子模块还配置为将所述每一帧图像中的一个检测框作为所述候选检测框。
在一些可能的实施例中，所述每一帧图像中包括至少两个检测框，所述第一确定子模块包括第一确定单元、生成单元和第二确定单元，其中：所述第一确定单元，配置为确定所述每一帧图像中的至少两个检测框之间的交并比；所述生成单元，配置为根据所述交并比，生成所述每一帧图像对应的邻接矩阵；所述第二确定单元，配置为将所述邻接矩阵中匹配次数为零的检测框作为所述候选检测框。
在一些可能的实施例中，所述第一确定子模块还包括扩展单元和选取单元，其中：所述扩展单元，配置为对每一所述对象的检测框，向外扩展特定比例，得到扩展后的检测框；所述选取单元，配置为从所述扩展后的检测框中筛选至少两个第一检测框；其中，所述第一检测框的面积大于所述扩展后的检测框中其他检测框；相应地，所述第一确定单元，还配置为确定所述至少两个第一检测框之间的交并比。
在一些可能的实施例中，所述第二确定子模块包括第三确定单元、截取单元和连接单元，其中：所述第三确定单元，配置为对与每一所述对象标识关联的候选检测框在空间位置上取并集，得到与每一所述对象标识对应的最小包围框；所述截取单元，配置为根据所述最小包围框，截取所述每一帧图像中与所述最小包围框对应的区域图像；所述连接单元，配置为按照所述每一帧图像的时间戳，依次连接所述区域图像，得到与每一所述对象标识对应的第一序列。
在一些可能的实施例中，所述候选检测框为从所述至少两个第一检测框中选择的，所述第二确定子模块还包括内缩单元和第四确定单元，其中：所述内缩单元，配置为对所述候选检测框，按照所述特定比例进行内缩，得到原始尺寸的候选检测框；所述第四确定单元配置为根据所述原始尺寸的候选检测框和与所述候选检测框关联的对象标识，生成所述至少一个第一序列。
在一些可能的实施例中，所述第二确定子模块还包括第五确定单元和填充单元，其中：所述第五确定单元，配置为确定所述区域图像中除去所述候选检测框之外的噪声区域；所述填充单元，配置为利用纯色背景填充所述噪声区域，得到目标区域图像；相应地，所述连接单元，还配置为按照所述每一帧图像的时间戳，依次连接所述目标区域图像，得到与每一所述对象标识对应的第一序列。
在一些可能的实施例中，所述识别模块730包括识别子模块和第三确定子模块，其中：所述识别子模块，配置为将每一所述第一序列送入到行为识别模型中，得到每一所述第一序列的判别分数；所述第三确定子模块，配置为根据每一所述第一序列的判别分 数，确定所述每一所述对象的行为识别结果。
在一些可能的实施例中，所述识别装置700还包括第三确定模块、第四确定模块和发送模块，其中：所述第三确定模块，配置为在所述第一序列的判别分数大于行为阈值的情况下，确定相应第一序列对应的单个对象行为发生的空间位置和行为类别；所述第四确定模块，配置为根据所述空间位置和行为类别，确定告警内容；所述发送模块，配置为根据所述告警内容向所述空间位置对应的终端设备发送告警通知，以使持有所述终端设备的管理人员处理所述单个对象行为。
这里需要指出的是：以上装置实施例的描述，与上述方法实施例的描述是类似的，具有同方法实施例相似的有益效果。对于本公开装置实施例中未披露的技术细节，请参照本公开方法实施例的描述而理解。
需要说明的是，本公开实施例中，如果以软件功能模块的形式实现上述行为识别方法，并作为独立的产品销售或使用时，也可以存储在一个计算机可读取存储介质中。基于这样的理解，本公开实施例的技术方案本质上或者说对相关技术做出贡献的部分可以以软件产品的形式体现出来，该计算机软件产品存储在一个存储介质中，包括若干指令用以使得行为识别设备(可以是具有摄像头的智能手机、平板电脑等)执行本公开各个实施例所述方法的全部或部分。而前述的存储介质包括：U盘、移动硬盘、只读存储器(Read Only Memory，ROM)、磁碟或者光盘等各种可以存储程序代码的介质。这样，本公开实施例不限制于任何特定的硬件和软件结合。
对应地，本公开实施例提供一种计算机可读存储介质，其上存储有计算机程序，该计算机程序被处理器执行时实现上述实施例中任一所述行为识别方法中的步骤。对应地，本公开实施例中，还提供了一种芯片，所述芯片包括可编程逻辑电路和/或程序指令，当所述芯片运行时，用于实现上述实施例中任一所述行为识别方法中的步骤。对应地，本公开实施例中，还提供了一种计算机程序产品，当该计算机程序产品被行为识别设备的处理器执行时，其用于实现上述实施例中任一所述行为识别方法中的步骤。
本公开实施例还提供一种计算机程序产品，该计算机程序产品承载有程序代码，所述程序代码包括的指令可用于执行上述方法实施例中任一所述行为识别方法中的步骤。其中，上述计算机程序产品可以具体通过硬件、软件或其结合的方式实现。在一个可选实施例中，所述计算机程序产品具体体现为计算机存储介质，在另一个可选实施例中，计算机程序产品具体体现为软件产品，例如软件开发包(Software Development Kit，SDK)等等。
本公开实施例还提供一种计算机程序，包括计算机可读代码，当所述计算机可读代码在电子设备中运行时，所述电子设备中的处理器执行用于实现上述方法实施例中任一所述行为识别方法。
基于同一技术构思，本公开实施例提供一种行为识别设备，用于实施上述方法实施例记载的行为识别方法。图8为本公开实施例提供的一种行为识别设备的硬件实体示意图，如图8所示，所述识别设备800包括存储器810和处理器820，所述存储器810存储 有可在处理器820上运行的计算机程序，所述处理器820执行所述程序时实现本公开实施例任一所述行为识别方法中的步骤。
存储器810用于存储由处理器820可执行的指令和应用，还可以缓存待处理器820以及识别设备800中各模块待处理或已经处理的数据(例如，图像数据、音频数据、语音通信数据和视频通信数据)，可以通过闪存(FLASH)或随机访问存储器(Random Access Memory，RAM)实现。
处理器820执行程序时实现上述任一项的行为识别方法的步骤。处理器820通常控制识别设备800的总体操作。
上述处理器可以为特定用途集成电路(Application Specific Integrated Circuit，ASIC)、数字信号处理器(Digital Signal Processor，DSP)、数字信号处理装置(Digital Signal Processing Device，DSPD)、可编程逻辑装置(Programmable Logic Device，PLD)、现场可编程门阵列(Field Programmable Gate Array，FPGA)、中央处理器(Central Processing Unit，CPU)、控制器、微控制器、微处理器中的至少一种。可以理解地，实现上述处理器功能的电子器件还可以为其它，本公开实施例不作具体限定。
上述计算机存储介质/存储器可以是只读存储器(Read Only Memory，ROM)、可编程只读存储器(Programmable Read-Only Memory，PROM)、可擦除可编程只读存储器(Erasable Programmable Read-Only Memory，EPROM)、电可擦除可编程只读存储器(Electrically Erasable Programmable Read-Only Memory，EEPROM)、磁性随机存取存储器(Ferromagnetic Random Access Memory，FRAM)、快闪存储器(Flash Memory)、磁表面存储器、光盘、或只读光盘(Compact Disc Read-Only Memory，CD-ROM)等存储器；也可以是包括上述存储器之一或任意组合的各种行为识别设备，如移动电话、计算机、平板设备、个人数字助理等。
这里需要指出的是：以上存储介质和设备实施例的描述，与上述方法实施例的描述是类似的，具有同方法实施例相似的有益效果。对于本公开存储介质和设备实施例中未披露的技术细节，请参照本公开方法实施例的描述而理解。
应理解，说明书通篇中提到的“一个实施例”或“一实施例”意味着与实施例有关的特定特征、结构或特性包括在本公开的至少一个实施例中。因此，在整个说明书各处出现的“在一个实施例中”或“在一实施例中”未必一定指相同的实施例。此外，这些特定的特征、结构或特性可以任意适合的方式结合在一个或多个实施例中。应理解，在本公开的各种实施例中，上述各过程的序号的大小并不意味着执行顺序的先后，各过程的执行顺序应以其功能和内在逻辑确定，而不应对本公开实施例的实施过程构成任何限定。上述本公开实施例序号仅仅为了描述，不代表实施例的优劣。
需要说明的是，在本文中，术语“包括”、“包含”或者其任何其他变体意在涵盖非排他性的包含，从而使得包括一系列要素的过程、方法、物品或者装置不仅包括那些要素，而且还包括没有明确列出的其他要素，或者是还包括为这种过程、方法、物品或者装置所固有的要素。在没有更多限制的情况下，由语句“包括一个……”限定的要素， 并不排除在包括该要素的过程、方法、物品或者装置中还存在另外的相同要素。
在本公开所提供的几个实施例中，应该理解到，所揭露的设备和方法，可以通过其它的方式实现。以上所描述的设备实施例仅仅是示意性的，例如，所述单元的划分，仅仅为一种逻辑功能划分，实际实现时可以有另外的划分方式，如：多个单元或组件可以结合，或可以集成到另一个系统，或一些特征可以忽略，或不执行。另外，所显示或讨论的各组成部分相互之间的耦合、或直接耦合、或通信连接可以是通过一些接口，设备或单元的间接耦合或通信连接，可以是电性的、机械的或其它形式的。
上述作为分离部件说明的单元可以是、或也可以不是物理上分开的，作为单元显示的部件可以是、或也可以不是物理单元；既可以位于一个地方，也可以分布到多个网络单元上；可以根据实际的需要选择其中的部分或全部单元来实现本公开实施例方案的目的。
另外，在本公开各实施例中的各功能单元可以全部集成在一个处理单元中，也可以是各单元分别单独作为一个单元，也可以两个或两个以上单元集成在一个单元中；上述集成的单元既可以采用硬件的形式实现，也可以采用硬件加软件功能单元的形式实现。
或者，本公开上述集成的单元如果以软件功能模块的形式实现并作为独立的产品销售或使用时，也可以存储在一个计算机可读取存储介质中。基于这样的理解，本公开实施例的技术方案本质上或者说对相关技术做出贡献的部分可以以软件产品的形式体现出来，该计算机软件产品存储在一个存储介质中，包括若干指令用以使得设备自动测试线执行本公开各个实施例所述方法的全部或部分。而前述的存储介质包括：移动存储设备、ROM、磁碟或者光盘等各种可以存储程序代码的介质。
本公开所提供的几个方法实施例中所揭露的方法，在不冲突的情况下可以任意组合，得到新的方法实施例。
本公开所提供的几个方法或设备实施例中所揭露的特征，在不冲突的情况下可以任意组合，得到新的方法实施例或设备实施例。
以上所述，仅为本公开的实施方式，但本公开的保护范围并不局限于此，任何熟悉本技术领域的技术人员在本公开揭露的技术范围内，可轻易想到变化或替换，都应涵盖在本公开的保护范围之内。因此，本公开的保护范围应以所述权利要求的保护范围为准。
工业实用性
在本公开实施例中，首先获取待识别视频序列中每一帧图像中每一对象的检测结果；然后根据每一所述对象的检测结果，生成至少一个第一序列；其中，每一所述第一序列为包括同一对象的轨迹序列；最后对每一所述第一序列进行行为识别，得到每一所述对象的行为识别结果。通过利用视频分类算法与检测算法的结合，将单帧行为识别算法拓展为一个具有时序关系的行为识别算法。将视频分类问题转换成轨迹序列的识别问题，能够减小模型的识别难度，并提升有效信息的获取。同时拓展了视频理解模型的边界，发挥了视频理解模型的潜力，也是对现有视频理解模型落地应用的体现。

Claims (26)

  1. 一种行为识别方法，所述方法包括：
    获取待识别视频序列中每一帧图像中每一对象的检测结果；
    根据每一所述对象的检测结果，生成至少一个第一序列；其中，每一所述第一序列为包括同一对象的轨迹序列；
    对每一所述第一序列进行行为识别，得到每一所述对象的行为识别结果。
  2. 如权利要求1所述的方法，其中，所述待识别视频序列是通过以下过程得到的：
    按照特定时间间隔，从视频源中选取至少两帧图像；
    将所述至少两帧图像存入帧序列缓冲区；其中，所述帧序列缓冲区的长度固定；
    获取所述帧序列缓冲区中的帧图像作为所述待识别视频序列。
  3. 如权利要求1或2所述的方法，其中，所述对象的检测结果至少包括所述对象的检测框和与所述检测框关联的对象标识；
    所述根据每一所述对象的检测结果，生成至少一个第一序列，包括：
    从每一所述对象的检测框中，选择在空间位置上独立的检测框作为候选检测框；
    根据所述候选检测框和与所述候选检测框关联的对象标识，生成所述至少一个第一序列。
  4. 如权利要求3所述的方法，其中，所述每一帧图像中包括一个检测框，所述从每一所述对象的检测框中，选择在空间位置上独立的检测框作为候选检测框，包括：
    将所述每一帧图像中的一个检测框作为所述候选检测框。
  5. 如权利要求3所述的方法，其中，所述每一帧图像中包括至少两个检测框，所述从每一所述对象的检测框中，选择在空间位置上独立的检测框作为候选检测框，包括：
    确定所述每一帧图像中的至少两个检测框之间的交并比；
    根据所述交并比，生成所述每一帧图像对应的邻接矩阵；
    将所述邻接矩阵中匹配次数为零的检测框作为所述候选检测框。
  6. 如权利要求3至5任一项所述的方法，其中，所述根据所述候选检测框和与所述候选检测框关联的对象标识，生成所述至少一个第一序列，包括：
    对与每一所述对象标识关联的候选检测框在空间位置上取并集，得到与每一所述对象标识对应的最小包围框；
    根据所述最小包围框，截取所述每一帧图像中与所述最小包围框对应的区域图像；
    按照所述每一帧图像的时间戳，依次连接所述区域图像，得到与每一所述对象标识对应的第一序列。
  7. 如权利要求5所述的方法，其中，所述方法还包括：
    对每一所述对象的检测框，向外扩展特定比例，得到扩展后的检测框；
    从所述扩展后的检测框中筛选至少两个第一检测框；其中，所述第一检测框的面积 大于所述扩展后的检测框中的其他检测框；
    所述确定所述每一帧图像中的至少两个检测框之间的交并比，包括：
    确定所述至少两个第一检测框之间的交并比。
  8. 如权利要求7所述的方法，其中，所述候选检测框为从所述至少两个第一检测框中选择的，所述根据所述候选检测框和与所述候选检测框关联的对象标识，生成所述至少一个第一序列，包括：
    对所述候选检测框，按照所述特定比例进行内缩，得到原始尺寸的候选检测框；
    根据所述原始尺寸的候选检测框和与所述候选检测框关联的对象标识，生成所述至少一个第一序列。
  9. 如权利要求6所述的方法，其中，所述方法还包括：
    确定所述区域图像中除去所述候选检测框之外的噪声区域；
    利用纯色背景填充所述噪声区域，得到目标区域图像；
    按照所述每一帧图像的时间戳，依次连接所述目标区域图像，得到与每一所述对象标识对应的第一序列。
  10. 如权利要求3至9任一项所述的方法，其中，所述对每一所述第一序列进行行为识别，得到每一所述对象的行为识别结果，包括：
    将每一所述第一序列送入到行为识别模型中，得到每一所述第一序列的判别分数；
    根据每一所述第一序列的判别分数，确定每一所述对象的行为识别结果。
  11. 如权利要求10所述的方法，其中，所述方法还包括：
    在所述第一序列的判别分数大于行为阈值的情况下，确定相应第一序列对应的单个对象行为发生的空间位置和行为类别；
    根据所述空间位置和行为类别，确定告警内容；
    根据所述告警内容向所述空间位置对应的终端设备发送告警通知。
  12. 一种行为识别装置，其中，所述装置包括获取模块、第一确定模块和识别模块，其中：
    所述获取模块，配置为获取待识别视频序列中每一帧图像中每一对象的检测结果；
    所述第一确定模块，配置为根据每一所述对象的检测结果，生成至少一个第一序列；其中，每一所述第一序列为包括同一对象的轨迹序列；
    所述识别模块，配置为对每一所述第一序列进行行为识别，得到每一所述对象的行为识别结果。
  13. 如权利要求12所述的装置，其中，所述识别装置还包括：
    选取模块，配置为按照特定时间间隔，从视频源中选取至少两帧图像；
    存储模块，配置为将所述至少两帧图像存入帧序列缓冲区；其中，所述帧序列缓冲区的长度固定；
    第二确定模块，配置为获取所述帧序列缓冲区中的帧图像作为所述待识别视频序列。
  14. 如权利要求12或13所述的装置，其中，所述对象的检测结果至少包括所述对 象的检测框和与所述检测框关联的对象标识；所述第一确定模块包括：
    第一确定子模块，配置为从每一所述对象的检测框中，选择在空间位置上独立的检测框作为候选检测框；
    第二确定子模块，配置为根据所述候选检测框和与所述候选检测框关联的对象标识，生成所述至少一个第一序列。
  15. 如权利要求14所述的装置，其中，所述每一帧图像中包括一个检测框，所述第一确定子模块还配置为将所述每一帧图像中的一个检测框作为所述候选检测框。
  16. 如权利要求14所述的装置，其中，所述每一帧图像中包括至少两个检测框，所述第一确定子模块包括：
    第一确定单元，配置为确定所述每一帧图像中的至少两个检测框之间的交并比；
    生成单元，配置为根据所述交并比，生成所述每一帧图像对应的邻接矩阵；
    第二确定单元，配置为将所述邻接矩阵中匹配次数为零的检测框作为所述候选检测框。
  17. 如权利要求14至16任一项所述的装置，其中，所述第二确定子模块包括：
    第三确定单元，配置为对与每一所述对象标识关联的候选检测框在空间位置上取并集，得到与每一所述对象标识对应的最小包围框；
    截取单元，配置为根据所述最小包围框，截取所述每一帧图像中与所述最小包围框对应的区域图像；
    连接单元，配置为按照所述每一帧图像的时间戳，依次连接所述区域图像，得到与每一所述对象标识对应的第一序列。
  18. 如权利要求16所述的装置，所述第一确定子模块还包括：
    扩展单元，配置为对每一所述对象的检测框，向外扩展特定比例，得到扩展后的检测框；
    选取单元，配置为从所述扩展后的检测框中筛选至少两个第一检测框；其中，所述第一检测框的面积大于所述扩展后的检测框中其他检测框；
    所述第一确定单元，还配置为确定所述至少两个第一检测框之间的交并比。
  19. 如权利要求18所述的装置，其中，所述候选检测框为从所述至少两个第一检测框中选择的，所述第二确定子模块还包括：
    内缩单元，配置为对所述候选检测框，按照所述特定比例进行内缩，得到原始尺寸的候选检测框；
    第四确定单元，配置为根据所述原始尺寸的候选检测框和与所述候选检测框关联的对象标识，生成所述至少一个第一序列。
  20. 如权利要求17所述的装置，其中，所述第二确定子模块还包括：
    第五确定单元，配置为确定所述区域图像中除去所述候选检测框之外的噪声区域；
    填充单元，配置为利用纯色背景填充所述噪声区域，得到目标区域图像；
    所述连接单元，还配置为按照所述每一帧图像的时间戳，依次连接所述目标区域图 像，得到与每一所述对象标识对应的第一序列。
  21. 如权利要求14至20任一项所述的装置，其中，所述识别模块包括：
    识别子模块，配置为将每一所述第一序列送入到行为识别模型中，得到每一所述第一序列的判别分数；
    第三确定子模块，配置为根据每一所述第一序列的判别分数，确定所述每一所述对象的行为识别结果。
  22. 如权利要求21所述的装置，其中，所述识别装置还包括：
    第三确定模块，配置为在所述第一序列的判别分数大于行为阈值的情况下，确定相应第一序列对应的单个对象行为发生的空间位置和行为类别；
    第四确定模块，配置为根据所述空间位置和行为类别，确定告警内容；
    发送模块，配置为根据所述告警内容向所述空间位置对应的终端设备发送告警通知，以使持有所述终端设备的管理人员处理所述单个对象行为。
  23. 一种行为识别设备，包括存储器和处理器，所述存储器存储有可在处理器上运行的计算机程序，所述处理器执行所述程序时实现权利要求1至11任一项所述方法中的步骤。
  24. 一种计算机可读存储介质，其上存储有计算机程序，该计算机程序被处理器执行时实现权利要求1至11任一项所述方法中的步骤。
  25. 一种计算机程序，包括计算机可读代码，当所述计算机可读代码在电子设备中运行时，所述电子设备中的处理器执行用于权利要求1至11中任一项所述的方法。
  26. 一种计算机程序产品，所述计算机程序产品包括一条或多条指令，所述一条或多条指令适于由处理器加载并执行如权利要求1至11任一项所述方法中的步骤。
PCT/CN2021/130036 2021-04-25 2021-11-11 行为识别方法、装置、设备、存储介质、计算机程序及程序产品 Ceased WO2022227490A1 (zh)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN202110449968.2A CN113111839A (zh) 2021-04-25 2021-04-25 行为识别方法及装置、设备和存储介质
CN202110449968.2 2021-04-25

Publications (1)

Publication Number Publication Date
WO2022227490A1 true WO2022227490A1 (zh) 2022-11-03

Family

ID=76720025

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2021/130036 Ceased WO2022227490A1 (zh) 2021-04-25 2021-11-11 行为识别方法、装置、设备、存储介质、计算机程序及程序产品

Country Status (2)

Country Link
CN (1) CN113111839A (zh)
WO (1) WO2022227490A1 (zh)

Cited By (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN115830500A (zh) * 2022-11-28 2023-03-21 西安电子科技大学广州研究院 基于本体库的撕纸行为识别方法、装置、设备及存储介质
CN116342537A (zh) * 2023-03-27 2023-06-27 重庆中科云从科技有限公司 轮档状态识别方法、控制装置及可读存储介质
CN117079346A (zh) * 2023-08-01 2023-11-17 上海云从企业发展有限公司 行为识别方法、系统、电子设备及存储介质
CN117612069A (zh) * 2024-01-19 2024-02-27 福思(杭州)智能科技有限公司 真值数据的构建方法和装置、存储介质
CN118097788A (zh) * 2024-03-20 2024-05-28 北京积加科技有限公司 应用于婴幼儿的行为识别方法、装置和电子设备
CN118433357A (zh) * 2024-04-19 2024-08-02 中国建筑第七工程局有限公司 基于智能巡检安全帽的施工现场巡检方法、设备与介质
CN119229391A (zh) * 2024-11-29 2024-12-31 齐鲁空天信息研究院 基于监控视频流的交通事故识别处理方法、装置及设备

Families Citing this family (11)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN113111839A (zh) * 2021-04-25 2021-07-13 上海商汤智能科技有限公司 行为识别方法及装置、设备和存储介质
CN113553950B (zh) * 2021-07-23 2025-03-25 上海商汤智能科技有限公司 一种异常事件检测方法、装置、电子设备及存储介质
CN113837066A (zh) * 2021-09-22 2021-12-24 深圳市商汤科技有限公司 行为识别方法、装置、电子设备及计算机存储介质
CN113837084B (zh) * 2021-09-24 2025-01-17 上海商汤智能科技有限公司 识别方法、装置、电子设备及计算机可读存储介质
CN113963438B (zh) * 2021-10-20 2025-09-16 上海商汤智能科技有限公司 行为识别方法及装置、设备和存储介质
CN114419480A (zh) * 2021-12-15 2022-04-29 厦门智小金智能科技有限公司 一种多人身份和动作关联识别方法、装置及可读介质
CN114419580A (zh) * 2021-12-27 2022-04-29 北京百度网讯科技有限公司 障碍物关联方法、装置、电子设备及存储介质
CN114444895B (zh) * 2021-12-31 2025-04-01 深圳云天励飞技术股份有限公司 清洁质量评估方法及相关设备
CN114758415A (zh) * 2022-03-31 2022-07-15 深圳市商汤科技有限公司 模型控制方法、装置、设备及存储介质
CN116434338A (zh) * 2023-04-06 2023-07-14 深圳市威富视界有限公司 摔倒检测方法、装置、电子设备及存储介质
CN120339918A (zh) * 2025-06-16 2025-07-18 北京城建智控科技股份有限公司 身份识别方法以及装置

Citations (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20160132754A1 (en) * 2012-05-25 2016-05-12 The Johns Hopkins University Integrated real-time tracking system for normal and anomaly tracking and the methods therefor
CN108446669A (zh) * 2018-04-10 2018-08-24 腾讯科技（深圳）有限公司 运动识别方法、装置及存储介质
CN111652043A (zh) * 2020-04-15 2020-09-11 北京三快在线科技有限公司 对象状态识别方法、装置、图像采集设备及存储介质
CN112016413A (zh) * 2020-08-13 2020-12-01 南京领行科技股份有限公司 对象间异常行为的检测方法及装置
CN112150514A (zh) * 2020-09-29 2020-12-29 上海眼控科技股份有限公司 视频的行人轨迹追踪方法、装置、设备及存储介质
CN112287741A (zh) * 2020-06-19 2021-01-29 北京京东尚科信息技术有限公司 基于图像处理的农事操作管理方法及装置
CN113111839A (zh) * 2021-04-25 2021-07-13 上海商汤智能科技有限公司 行为识别方法及装置、设备和存储介质

Family Cites Families (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN105913416A (zh) * 2016-04-06 2016-08-31 中南大学 一种自动分割三维人脸模型区域的方法
CN109214322A (zh) * 2018-08-27 2019-01-15 厦门哲林软件科技有限公司 一种文档图像视觉效果的优化方法及系统
CN109977949B (zh) * 2019-03-20 2024-01-26 深圳华付技术股份有限公司 边框微调的文本定位方法、装置、计算机设备及存储介质
CN112507760B (zh) * 2019-09-16 2024-05-31 杭州海康威视数字技术股份有限公司 暴力分拣行为的检测方法、装置及设备
CN111291704A (zh) * 2020-02-24 2020-06-16 上海眼控科技股份有限公司 干扰去除方法、装置、计算机设备和存储介质
CN111985385B (zh) * 2020-08-14 2023-08-29 杭州海康威视数字技术股份有限公司 一种行为检测方法、装置及设备

Patent Citations (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20160132754A1 (en) * 2012-05-25 2016-05-12 The Johns Hopkins University Integrated real-time tracking system for normal and anomaly tracking and the methods therefor
CN108446669A (zh) * 2018-04-10 2018-08-24 腾讯科技（深圳）有限公司 运动识别方法、装置及存储介质
CN111652043A (zh) * 2020-04-15 2020-09-11 北京三快在线科技有限公司 对象状态识别方法、装置、图像采集设备及存储介质
CN112287741A (zh) * 2020-06-19 2021-01-29 北京京东尚科信息技术有限公司 基于图像处理的农事操作管理方法及装置
CN112016413A (zh) * 2020-08-13 2020-12-01 南京领行科技股份有限公司 对象间异常行为的检测方法及装置
CN112150514A (zh) * 2020-09-29 2020-12-29 上海眼控科技股份有限公司 视频的行人轨迹追踪方法、装置、设备及存储介质
CN113111839A (zh) * 2021-04-25 2021-07-13 上海商汤智能科技有限公司 行为识别方法及装置、设备和存储介质

Cited By (8)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN115830500A (zh) * 2022-11-28 2023-03-21 西安电子科技大学广州研究院 基于本体库的撕纸行为识别方法、装置、设备及存储介质
CN116342537A (zh) * 2023-03-27 2023-06-27 重庆中科云从科技有限公司 轮档状态识别方法、控制装置及可读存储介质
CN117079346A (zh) * 2023-08-01 2023-11-17 上海云从企业发展有限公司 行为识别方法、系统、电子设备及存储介质
CN117612069A (zh) * 2024-01-19 2024-02-27 福思(杭州)智能科技有限公司 真值数据的构建方法和装置、存储介质
CN117612069B (zh) * 2024-01-19 2024-04-26 福思(杭州)智能科技有限公司 真值数据的构建方法和装置、存储介质
CN118097788A (zh) * 2024-03-20 2024-05-28 北京积加科技有限公司 应用于婴幼儿的行为识别方法、装置和电子设备
CN118433357A (zh) * 2024-04-19 2024-08-02 中国建筑第七工程局有限公司 基于智能巡检安全帽的施工现场巡检方法、设备与介质
CN119229391A (zh) * 2024-11-29 2024-12-31 齐鲁空天信息研究院 基于监控视频流的交通事故识别处理方法、装置及设备

Also Published As

Publication number Publication date
CN113111839A (zh) 2021-07-13

Similar Documents

Publication Publication Date Title
CN113111839A (zh) 行为识别方法及装置、设备和存储介质
WO2022227480A1 (zh) 行为识别方法、装置、设备、存储介质、计算机程序及程序产品
JP7036863B2 (ja) ビデオデータを用いた活動モニタリングのためのシステム及び方法
CN112052815B (zh) 一种行为检测方法、装置及电子设备
WO2021051601A1 (zh) 利用Mask R-CNN选择检测框的方法及系统、电子装置及存储介质
CN114943936B (zh) 目标行为识别方法、装置、电子设备及存储介质
CN108052859A (zh) 一种基于聚类光流特征的异常行为检测方法、系统及装置
CN112507860A (zh) 一种视频标注方法、装置、设备及存储介质
CN108062349A (zh) 基于视频结构化数据及深度学习的视频监控方法和系统
CN111241343A (zh) 一种道路信息监控与分析检测方法、智慧交通控制系统
CN111429726A (zh) 一种监控视频违停车辆检测、管理方法及对应的系统
CN110795595A (zh) 基于边缘计算的视频结构化存储方法、装置、设备及介质
CN110826447A (zh) 一种基于注意力机制的餐厅后厨人员行为识别方法
CN112905824A (zh) 目标车辆追踪方法、装置、计算机设备及存储介质
JP2023505864A (ja) ターゲット移動軌跡の構築方法、機器及びコンピュータ記憶媒体
WO2022228325A1 (zh) 行为检测方法、电子设备以及计算机可读存储介质
WO2020094088A1 (zh) 一种图像抓拍方法、监控相机及监控系统
CN113963438B (zh) 行为识别方法及装置、设备和存储介质
CN102411703A (zh) 检测图像序列中的特定对象的装置、方法和视频摄像设备
CN113139403A (zh) 违规行为识别方法、装置、计算机设备和存储介质
CN111291587A (zh) 一种基于密集人群的行人检测方法、存储介质及处理器
CN110189355A (zh) 安全疏散通道占用检测方法、装置、电子设备及存储介质
CN104463232A (zh) 一种基于hog特征和颜色直方图特征的密度人群计数的方法
CN110855932B (zh) 基于视频数据的报警方法、装置、电子设备及存储介质
CN113449563B (zh) 一种人员跟踪标记方法、装置、电子设备及存储介质

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 21938950

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 21938950

Country of ref document: EP

Kind code of ref document: A1

122 Ep: pct application non-entry in european phase

Ref document number: 21938950

Country of ref document: EP

Kind code of ref document: A1

32PN Ep: public notification in the ep bulletin as address of the adressee cannot be established

Free format text: NOTING OF LOSS OF RIGHTS PURSUANT TO RULE 112(1) EPC (EPO FORM 1205A DATED 16/05/2024)

122 Ep: pct application non-entry in european phase

Ref document number: 21938950

Country of ref document: EP

Kind code of ref document: A1