WO2019005257A3 - A neuromorphic system for real-time visual activity recognition - Google Patents

A neuromorphic system for real-time visual activity recognition

Info

Publication number
WO2019005257A3
WO2019005257A3 PCT/US2018/026432 US2018026432W WO2019005257A3 WO 2019005257 A3 WO2019005257 A3 WO 2019005257A3 US 2018026432 W US2018026432 W US 2018026432W WO 2019005257 A3 WO2019005257 A3 WO 2019005257A3
Authority
WO
WIPO (PCT)
Prior art keywords
interest
objects
processors
perform operations
activity recognition
Prior art date
Application number
PCT/US2018/026432
Other languages
French (fr)
Other versions
WO2019005257A2 (en
Inventor
Deepak Khosla
Ryan M. UHLENBROCK
Yang Chen
Original Assignee
Hrl Laboratories, Llc
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Priority claimed from US15/883,822 external-priority patent/US11055872B1/en
Application filed by Hrl Laboratories, Llc filed Critical Hrl Laboratories, Llc
Priority to EP18823688.9A priority Critical patent/EP3635628A4/en
Priority to CN201880030086.9A priority patent/CN110603542B/en
Publication of WO2019005257A2 publication Critical patent/WO2019005257A2/en
Publication of WO2019005257A3 publication Critical patent/WO2019005257A3/en

Links

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/06—Physical realisation, i.e. hardware implementation of neural networks, neurons or parts of neurons
    • G06N3/063—Physical realisation, i.e. hardware implementation of neural networks, neurons or parts of neurons using electronic means
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F18/00—Pattern recognition
    • G06F18/20—Analysing
    • G06F18/24—Classification techniques
    • G06F18/241—Classification techniques relating to the classification model, e.g. parametric or non-parametric approaches
    • G06F18/2413—Classification techniques relating to the classification model, e.g. parametric or non-parametric approaches based on distances to training or reference patterns
    • G06F18/24133—Distances to prototypes
    • G06F18/24143—Distances to neighbourhood prototypes, e.g. restricted Coulomb energy networks [RCEN]
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F18/00—Pattern recognition
    • G06F18/20—Analysing
    • G06F18/285—Selection of pattern recognition techniques, e.g. of classifiers in a multi-classifier system
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N20/00—Machine learning
    • G06N20/10—Machine learning using kernel methods, e.g. support vector machines [SVM]
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/044—Recurrent networks, e.g. Hopfield networks
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/045—Combinations of networks
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
    • G06V10/82—Arrangements for image or video recognition or understanding using pattern recognition or machine learning using neural networks
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V20/00—Scenes; Scene-specific elements
    • G06V20/50—Context or environment of the image
    • G06V20/52—Surveillance or monitoring of activities, e.g. for recognising suspicious objects
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
    • G06V40/20—Movements or behaviour, e.g. gesture recognition

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Evolutionary Computation (AREA)
  • Health & Medical Sciences (AREA)
  • Artificial Intelligence (AREA)
  • Data Mining & Analysis (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Software Systems (AREA)
  • General Health & Medical Sciences (AREA)
  • General Engineering & Computer Science (AREA)
  • Computing Systems (AREA)
  • Multimedia (AREA)
  • Biophysics (AREA)
  • Biomedical Technology (AREA)
  • Mathematical Physics (AREA)
  • Molecular Biology (AREA)
  • Computational Linguistics (AREA)
  • Medical Informatics (AREA)
  • Neurology (AREA)
  • Human Computer Interaction (AREA)
  • Social Psychology (AREA)
  • Databases & Information Systems (AREA)
  • Psychiatry (AREA)
  • Bioinformatics & Cheminformatics (AREA)
  • Bioinformatics & Computational Biology (AREA)
  • Evolutionary Biology (AREA)
  • Image Analysis (AREA)
  • Closed-Circuit Television Systems (AREA)

Abstract

Described is a system for visual activity recognition that includes one or more processors and a memory, the memory being a non-transitory computer-readable medium having executable instructions encoded thereon, such that upon execution of the instructions, the one or more processors perform operations including detecting a set of objects of interest in video data and determining an object classification for each object in the set of objects of interest, the set including at least one object of interest. The one or more processors further perform operations including forming a corresponding activity track for each object in the set of objects of interest by tracking each object across frames. The one or more processors further perform operations including, for each object of interest and using a feature extractor, determining a corresponding feature in the video data. The system may provide a report to a user's cell phone or central monitoring facility.
PCT/US2018/026432 2017-06-07 2018-04-06 A neuromorphic system for real-time visual activity recognition WO2019005257A2 (en)

Priority Applications (2)

Application Number Priority Date Filing Date Title
EP18823688.9A EP3635628A4 (en) 2017-06-07 2018-04-06 A neuromorphic system for real-time visual activity recognition
CN201880030086.9A CN110603542B (en) 2017-06-07 2018-04-06 Systems, methods, and computer-readable media for visual activity recognition

Applications Claiming Priority (4)

Application Number Priority Date Filing Date Title
US201762516217P 2017-06-07 2017-06-07
US62/516,217 2017-06-07
US15/883,822 2018-01-30
US15/883,822 US11055872B1 (en) 2017-03-30 2018-01-30 Real-time object recognition using cascaded features, deep learning and multi-target tracking

Publications (2)

Publication Number Publication Date
WO2019005257A2 WO2019005257A2 (en) 2019-01-03
WO2019005257A3 true WO2019005257A3 (en) 2019-05-02

Family

ID=64741843

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/US2018/026432 WO2019005257A2 (en) 2017-06-07 2018-04-06 A neuromorphic system for real-time visual activity recognition

Country Status (3)

Country Link
EP (1) EP3635628A4 (en)
CN (1) CN110603542B (en)
WO (1) WO2019005257A2 (en)

Citations (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US9216737B1 (en) * 2012-04-13 2015-12-22 Google Inc. System and method for automatically detecting key behaviors by vehicles
US20160148054A1 (en) * 2014-11-26 2016-05-26 Zepp Labs, Inc. Fast Object Tracking Framework For Sports Video Recognition

Family Cites Families (10)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
MX2009008376A (en) * 2007-02-08 2009-12-14 Behavioral Recognition Systems Behavioral recognition system.
US8175333B2 (en) * 2007-09-27 2012-05-08 Behavioral Recognition Systems, Inc. Estimator identifier component for behavioral recognition system
US8345984B2 (en) * 2010-01-28 2013-01-01 Nec Laboratories America, Inc. 3D convolutional neural networks for automatic human action recognition
TWI430212B (en) * 2010-06-08 2014-03-11 Gorilla Technology Inc Abnormal behavior detection system and method using automatic classification of multiple features
US9008366B1 (en) * 2012-01-23 2015-04-14 Hrl Laboratories, Llc Bio-inspired method of ground object cueing in airborne motion imagery
US9576214B1 (en) * 2012-01-23 2017-02-21 Hrl Laboratories, Llc Robust object recognition from moving platforms by combining form and motion detection with bio-inspired classification
CA2884096C (en) * 2012-09-05 2021-01-26 Element, Inc. System and method for biometric authentication in connection with camera-equipped devices
EP2720172A1 (en) * 2012-10-12 2014-04-16 Nederlandse Organisatie voor toegepast -natuurwetenschappelijk onderzoek TNO Video access system and method based on action type detection
WO2014098783A1 (en) * 2012-12-21 2014-06-26 Echostar Ukraine, LLC Identification and tracking onscreen
GB201512283D0 (en) * 2015-07-14 2015-08-19 Apical Ltd Track behaviour events

Patent Citations (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US9216737B1 (en) * 2012-04-13 2015-12-22 Google Inc. System and method for automatically detecting key behaviors by vehicles
US20160148054A1 (en) * 2014-11-26 2016-05-26 Zepp Labs, Inc. Fast Object Tracking Framework For Sports Video Recognition

Non-Patent Citations (3)

* Cited by examiner, † Cited by third party
Title
DONAHUE ET AL.: "Long-term Recurrent Convolutional Networks for Visual Recognition and Description", IN: THE IEEE CONFERENCE ON COMPUTER VISION AND PATTERN RECOGNITION (CVPR, 2015, pages 2625 - 2634, XP032793708, DOI: doi:10.1109/CVPR.2015.7298878 *
KARPATHY ET AL.: "Large-scale Video Classification with Convolutional Neural N etworks", IN: THE IEEE CONFERENCE ON COMPUTER VISION AND PATTERN RECOGNITION (CVPR, 2014, pages 1725 - 1732, XP055560536, DOI: doi:10.1109/CVPR.2014.223 *
SIMONYAN ET AL., TWO-STREAM CONVOLUTIONAL NETWORKS FOR ACTION RECOGNITION IN VIDEOS. IN: PROCEEDING NIPS'14 PROCEEDINGS OF THE 27TH INTERNATIONAL CONFERENCE ON NEURAL INFORMATION PROCESSING SYSTEMS (NIPS, December 2014 (2014-12-01), pages 1 - 9, XP055324674 *

Also Published As

Publication number Publication date
WO2019005257A2 (en) 2019-01-03
EP3635628A2 (en) 2020-04-15
CN110603542A (en) 2019-12-20
EP3635628A4 (en) 2021-03-10
CN110603542B (en) 2023-04-25

Similar Documents

Publication Publication Date Title
SG10201807330XA (en) Tracking and/or analyzing facility-related activities
WO2017091730A3 (en) Automated health data acquisition, processing and communication system and method
AU2018337654A1 (en) Method and apparatus for determining health status
WO2017151241A3 (en) Video processing
EP4287590A3 (en) Autonomous vehicle advanced sensing and response
SG11201909068XA (en) Methods and apparatuses for dynamically adding facial images into database, electronic devices, media and programs
AU2018289552A1 (en) Linking observed human activity on video to a user account
PH12018500287A1 (en) Key frame acquisition method for human image video system
WO2019010136A3 (en) Determining and utilizing corrections to robot actions
EP4254285A3 (en) Refrigerator, server and method of controlling thereof
WO2017019732A8 (en) Systems and methods for tracking data using user provided data tags
MX2016014071A (en) Method and apparatus for analyzing media content.
MX2020009382A (en) Method for identifying an object within an image and mobile device for executing the method.
MX2020014325A (en) Systems and methods for determining potential malicious event.
KR20180084576A (en) Artificial agents and method for human intention understanding based on perception-action connected learning, recording medium for performing the method
EP3229177A3 (en) Methods and systems for authenticating users
MX2017012839A (en) Image processing method and apparatus.
WO2012154832A3 (en) Object tracking
PH12018550213A1 (en) System and method for learning-based group tagging
EP2966644A3 (en) Methods and systems for managing speech recognition in a multi-speech system environment
PH12018502661A1 (en) Epicentral distance estimating device, epicentral distance estimating method, and computer-readable recording medium
WO2019152177A3 (en) System and method for neuromorphic visual activity classification based on foveated detection and contextual filtering
GB2553926A (en) Automatic detection of panoramic gestures
MX2021009164A (en) Pet food recommendation devices and methods.
SG11201909105TA (en) Information processing apparatus, control method, and program

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 18823688

Country of ref document: EP

Kind code of ref document: A2

NENP Non-entry into the national phase

Ref country code: DE

ENP Entry into the national phase

Ref document number: 2018823688

Country of ref document: EP

Effective date: 20200107