WO2019152177A3 - System and method for neuromorphic visual activity classification based on foveated detection and contextual filtering - Google Patents

System and method for neuromorphic visual activity classification based on foveated detection and contextual filtering Download PDF

Info

Publication number
WO2019152177A3
WO2019152177A3 PCT/US2019/013513 US2019013513W WO2019152177A3 WO 2019152177 A3 WO2019152177 A3 WO 2019152177A3 US 2019013513 W US2019013513 W US 2019013513W WO 2019152177 A3 WO2019152177 A3 WO 2019152177A3
Authority
WO
WIPO (PCT)
Prior art keywords
foveated
activity
detection
classification
activity classification
Prior art date
Application number
PCT/US2019/013513
Other languages
French (fr)
Other versions
WO2019152177A2 (en
Inventor
Deepak Khosla
Ryan M. UHLENBROCK
Yang Chen
Huapeng SU
Original Assignee
Hrl Laboratories, Llc
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Priority claimed from US15/883,822 external-priority patent/US11055872B1/en
Priority claimed from US15/947,032 external-priority patent/US10997421B2/en
Application filed by Hrl Laboratories, Llc filed Critical Hrl Laboratories, Llc
Priority to EP19748018.9A priority Critical patent/EP3746938A4/en
Priority to CN201980006835.9A priority patent/CN111566661B/en
Publication of WO2019152177A2 publication Critical patent/WO2019152177A2/en
Publication of WO2019152177A3 publication Critical patent/WO2019152177A3/en

Links

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V20/00—Scenes; Scene-specific elements
    • G06V20/50—Context or environment of the image
    • G06V20/52—Surveillance or monitoring of activities, e.g. for recognising suspicious objects
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N20/00—Machine learning
    • G06N20/10—Machine learning using kernel methods, e.g. support vector machines [SVM]
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/044—Recurrent networks, e.g. Hopfield networks
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/045—Combinations of networks
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/20—Image preprocessing
    • G06V10/255—Detecting or recognising potential candidate objects based on visual cues, e.g. shapes
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V2201/00—Indexing scheme relating to image or video recognition or understanding
    • G06V2201/08—Detecting or categorising vehicles
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
    • G06V40/20—Movements or behaviour, e.g. gesture recognition

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Software Systems (AREA)
  • Artificial Intelligence (AREA)
  • Computing Systems (AREA)
  • Data Mining & Analysis (AREA)
  • Evolutionary Computation (AREA)
  • Mathematical Physics (AREA)
  • General Engineering & Computer Science (AREA)
  • Computational Linguistics (AREA)
  • Health & Medical Sciences (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Biomedical Technology (AREA)
  • Biophysics (AREA)
  • General Health & Medical Sciences (AREA)
  • Molecular Biology (AREA)
  • Multimedia (AREA)
  • Medical Informatics (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Image Analysis (AREA)

Abstract

Described is a system for visual activity recognition. In operation, the system detects a set of objects of interest (OI) in video data and determines an object classification for each object in the set of OI, the set including at least one OI. A corresponding activity track is formed for each object in the set of OI by tracking each object across frames. Using a feature extractor, the system determines a corresponding feature in the video data for each OI, which is then used to determine a corresponding initial activity classification for each OI. One or more OI are then detected in each activity track via foveation, with the initial object detection and foveated object detection thereafter being appended into a new detected-objects list. Finally, a final classification is provided for each activity track using the new detected-objects list and filtering the initial activity classification results using contextual logic.
PCT/US2019/013513 2018-01-30 2019-01-14 System and method for neuromorphic visual activity classification based on foveated detection and contextual filtering WO2019152177A2 (en)

Priority Applications (2)

Application Number Priority Date Filing Date Title
EP19748018.9A EP3746938A4 (en) 2018-01-30 2019-01-14 System and method for neuromorphic visual activity classification based on foveated detection and contextual filtering
CN201980006835.9A CN111566661B (en) 2018-01-30 2019-01-14 Systems, methods, computer-readable media for visual activity classification

Applications Claiming Priority (6)

Application Number Priority Date Filing Date Title
US15/883,822 2018-01-30
US15/883,822 US11055872B1 (en) 2017-03-30 2018-01-30 Real-time object recognition using cascaded features, deep learning and multi-target tracking
US201862642959P 2018-03-14 2018-03-14
US62/642,959 2018-03-14
US15/947,032 US10997421B2 (en) 2017-03-30 2018-04-06 Neuromorphic system for real-time visual activity recognition
US15/947,032 2018-04-06

Publications (2)

Publication Number Publication Date
WO2019152177A2 WO2019152177A2 (en) 2019-08-08
WO2019152177A3 true WO2019152177A3 (en) 2019-10-10

Family

ID=67479383

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/US2019/013513 WO2019152177A2 (en) 2018-01-30 2019-01-14 System and method for neuromorphic visual activity classification based on foveated detection and contextual filtering

Country Status (3)

Country Link
EP (1) EP3746938A4 (en)
CN (1) CN111566661B (en)
WO (1) WO2019152177A2 (en)

Families Citing this family (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN111652128B (en) * 2020-06-02 2023-09-01 浙江大华技术股份有限公司 High-altitude power operation safety monitoring method, system and storage device
US20230206615A1 (en) * 2021-12-29 2023-06-29 Halliburton Energy Services, Inc. Systems and methods to determine an activity associated with an object of interest
US11776247B2 (en) 2022-01-07 2023-10-03 Tomahawk Robotics Classification parallelization architecture
KR20230134846A (en) * 2022-03-15 2023-09-22 연세대학교 산학협력단 Multiscale object detection device and method

Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20110081043A1 (en) * 2009-10-07 2011-04-07 Sabol Bruce M Using video-based imagery for automated detection, tracking, and counting of moving objects, in particular those objects having image characteristics similar to background
US20140218468A1 (en) * 2012-04-05 2014-08-07 Augmented Vision Inc. Wide-field of view (fov) imaging devices with active foveation capability
US9230302B1 (en) * 2013-03-13 2016-01-05 Hrl Laboratories, Llc Foveated compressive sensing system
US20170132468A1 (en) * 2015-11-06 2017-05-11 The Boeing Company Systems and methods for object tracking and classification

Family Cites Families (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
BRPI0806968B1 (en) * 2007-02-08 2018-09-18 Behavioral Recognition Sys Inc method for processing video frame stream and associated system
US9008366B1 (en) 2012-01-23 2015-04-14 Hrl Laboratories, Llc Bio-inspired method of ground object cueing in airborne motion imagery
US9147255B1 (en) 2013-03-14 2015-09-29 Hrl Laboratories, Llc Rapid object detection by combining structural information from image segmentation with bio-inspired attentional mechanisms

Patent Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20110081043A1 (en) * 2009-10-07 2011-04-07 Sabol Bruce M Using video-based imagery for automated detection, tracking, and counting of moving objects, in particular those objects having image characteristics similar to background
US20140218468A1 (en) * 2012-04-05 2014-08-07 Augmented Vision Inc. Wide-field of view (fov) imaging devices with active foveation capability
US9230302B1 (en) * 2013-03-13 2016-01-05 Hrl Laboratories, Llc Foveated compressive sensing system
US20170132468A1 (en) * 2015-11-06 2017-05-11 The Boeing Company Systems and methods for object tracking and classification

Non-Patent Citations (1)

* Cited by examiner, † Cited by third party
Title
JEFF DONAHUE ET AL.: "Long-term Recurrent Convolutional Networks for Visual Recognition and Description", THE IEEE CONFERENCE ON COMPUTER VISION AND PATTERN RECOGNITION (CVPR, 2015, pages 2625 - 2634, XP032793708, DOI: 10.1109/CVPR.2015.7298878 *

Also Published As

Publication number Publication date
WO2019152177A2 (en) 2019-08-08
CN111566661B (en) 2023-11-17
EP3746938A2 (en) 2020-12-09
CN111566661A (en) 2020-08-21
EP3746938A4 (en) 2021-10-06

Similar Documents

Publication Publication Date Title
WO2019152177A3 (en) System and method for neuromorphic visual activity classification based on foveated detection and contextual filtering
US10102421B2 (en) Method and device for face recognition in video
US9008365B2 (en) Systems and methods for pedestrian detection in images
US20190205620A1 (en) High-quality training data preparation for high-performance face recognition systems
EP2905724B1 (en) Object detection system and method
WO2016107103A1 (en) Method and device for recognizing main region of image
US9619753B2 (en) Data analysis system and method
US20120148160A1 (en) Landmark localization for facial imagery
EP3376431B1 (en) Method and apparatus for identifying pupil in image
CN104834916A (en) Multi-face detecting and tracking method
US20180173939A1 (en) Recognition of objects within a video
US20210097692A1 (en) Data filtering of image stacks and video streams
US20220172378A1 (en) Image processing apparatus, image processing method and non-transitory computer readable medium
CN103324953B (en) Video monitoring multi-target detection and tracking
KR102195940B1 (en) System and Method for Detecting Deep Learning based Human Object using Adaptive Thresholding Method of Non Maximum Suppression
US20200394802A1 (en) Real-time object detection method for multiple camera images using frame segmentation and intelligent detection pool
CN108446581A (en) A kind of unmanned plane detection method in adverse circumstances
US20220076005A1 (en) Person authentication apparatus, control method, and non-transitory storage medium
US20190340452A1 (en) Image processing apparatus, image processing method, and computer-readable recording medium recording image processing program
Sumi et al. Frame level difference (FLD) features to detect partially occluded pedestrian for ADAS
CN107403192B (en) Multi-classifier-based rapid target detection method and system
Chang et al. Robust abandoned object detection and analysis based on online learning
KR102646430B1 (en) Method for learning classifier and prediction classification apparatus using the same
AU2021100154A4 (en) Automated location identification system using text and image data
CN106919926A (en) It is applied to the detection identification module of video brand identity system

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 19748018

Country of ref document: EP

Kind code of ref document: A2

NENP Non-entry into the national phase

Ref country code: DE

ENP Entry into the national phase

Ref document number: 2019748018

Country of ref document: EP

Effective date: 20200831