EP3782075A1 - System for real-time object detection and recognition using both image and size features - Google Patents

System for real-time object detection and recognition using both image and size features

Info

Publication number
EP3782075A1
EP3782075A1 EP19789101.3A EP19789101A EP3782075A1 EP 3782075 A1 EP3782075 A1 EP 3782075A1 EP 19789101 A EP19789101 A EP 19789101A EP 3782075 A1 EP3782075 A1 EP 3782075A1
Authority
EP
European Patent Office
Prior art keywords
cnn
confidence score
target
modified
input image
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
EP19789101.3A
Other languages
German (de)
French (fr)
Other versions
EP3782075A4 (en
Inventor
Yang Chen
Deepak Khosla
Ryan M. UHLENBROCK
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
HRL Laboratories LLC
Original Assignee
HRL Laboratories LLC
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by HRL Laboratories LLC filed Critical HRL Laboratories LLC
Publication of EP3782075A1 publication Critical patent/EP3782075A1/en
Publication of EP3782075A4 publication Critical patent/EP3782075A4/en
Pending legal-status Critical Current

Links

Classifications

    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F18/00—Pattern recognition
    • G06F18/20—Analysing
    • G06F18/24—Classification techniques
    • G06F18/241—Classification techniques relating to the classification model, e.g. parametric or non-parametric approaches
    • G06F18/2413—Classification techniques relating to the classification model, e.g. parametric or non-parametric approaches based on distances to training or reference patterns
    • G06F18/24133—Distances to prototypes
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F18/00—Pattern recognition
    • G06F18/20—Analysing
    • G06F18/24—Classification techniques
    • G06F18/241—Classification techniques relating to the classification model, e.g. parametric or non-parametric approaches
    • G06F18/2413—Classification techniques relating to the classification model, e.g. parametric or non-parametric approaches based on distances to training or reference patterns
    • G06F18/24133—Distances to prototypes
    • G06F18/24143—Distances to neighbourhood prototypes, e.g. restricted Coulomb energy networks [RCEN]
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F18/00—Pattern recognition
    • G06F18/20—Analysing
    • G06F18/24—Classification techniques
    • G06F18/243—Classification techniques relating to the number of classes
    • G06F18/24323—Tree-organised classifiers
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F18/00—Pattern recognition
    • G06F18/20—Analysing
    • G06F18/25—Fusion techniques
    • G06F18/254—Fusion techniques of classification results, e.g. of results related to same input data
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/045—Combinations of networks
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/0464—Convolutional networks [CNN, ConvNet]
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/047—Probabilistic or stochastic networks
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods
    • G06N3/09—Supervised learning
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/10—Image acquisition
    • G06V10/12—Details of acquisition arrangements; Constructional details thereof
    • G06V10/14—Optical characteristics of the device performing the acquisition or on the illumination arrangements
    • G06V10/143—Sensing or illuminating at different wavelengths
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/10—Image acquisition
    • G06V10/12—Details of acquisition arrangements; Constructional details thereof
    • G06V10/14—Optical characteristics of the device performing the acquisition or on the illumination arrangements
    • G06V10/145—Illumination specially adapted for pattern recognition, e.g. using gratings
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/20—Image preprocessing
    • G06V10/25—Determination of region of interest [ROI] or a volume of interest [VOI]
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/40—Extraction of image or video features
    • G06V10/44—Local feature extraction by analysis of parts of the pattern, e.g. by detecting edges, contours, loops, corners, strokes or intersections; Connectivity analysis, e.g. of connected components
    • G06V10/443—Local feature extraction by analysis of parts of the pattern, e.g. by detecting edges, contours, loops, corners, strokes or intersections; Connectivity analysis, e.g. of connected components by matching or filtering
    • G06V10/449—Biologically inspired filters, e.g. difference of Gaussians [DoG] or Gabor filters
    • G06V10/451—Biologically inspired filters, e.g. difference of Gaussians [DoG] or Gabor filters with interaction between the filter responses, e.g. cortical complex cells
    • G06V10/454—Integrating the filters into a hierarchical structure, e.g. convolutional neural networks [CNN]
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/40—Extraction of image or video features
    • G06V10/50—Extraction of image or video features by performing operations within image blocks; by using histograms, e.g. histogram of oriented gradients [HoG]; by summing image-intensity values; Projection analysis
    • G06V10/507—Summing image-intensity values; Histogram projection analysis
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
    • G06V10/82—Arrangements for image or video recognition or understanding using pattern recognition or machine learning using neural networks
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V20/00—Scenes; Scene-specific elements
    • G06V20/50—Context or environment of the image
    • G06V20/56—Context or environment of the image exterior to a vehicle by using sensors mounted on the vehicle
    • G06V20/58—Recognition of moving objects or obstacles, e.g. vehicles or pedestrians; Recognition of traffic objects, e.g. traffic signs, traffic lights or roads
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V20/00—Scenes; Scene-specific elements
    • G06V20/10—Terrestrial scenes
    • G06V20/13—Satellite images
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V20/00—Scenes; Scene-specific elements
    • G06V20/50—Context or environment of the image
    • G06V20/52—Surveillance or monitoring of activities, e.g. for recognising suspicious objects

Definitions

  • the present invention relates to an object detection system and, more specifically, to an object detection and recognition system using both image and size features.
  • the system includes one or more processors and a memory.
  • the memory includes executable instructions encoded thereon, such that upon execution of the instructions, the one or more processors perform the operations as described herein.
  • ICF integral channel features
  • the system extracts a candidate target region (having an associated original confidence score representing a candidate object) from an input image of a scene surrounding a platform.
  • a modified confidence score is generated based on a location and height of detection of the candidate object.
  • the candidate target regions are classified based on the modified confidence score using a trained convolutional neural network (CNN) classifier, resulting in classified objects.
  • CNN convolutional neural network
  • classification of each classified object as a target or non-target. If the classified object is a target, a device can be controlled based on the target.
  • generating the modified confidence score uses a
  • modified confidence score original confidence score * wf , wherein h denotes a height of the candidate object in the input image, m and s denote mean and standard deviation, respectively, of object height distribution in the input image and bin, N is a multiplier and * denotes multiplication.
  • the present invention also includes a computer program product and a computer implemented method.
  • the computer program product includes computer-readable instructions stored on a non-transitory computer-readable medium that are executable by a computer having one or more processors, such that upon execution of the instructions, the one or more processors perform the operations listed herein.
  • the computer implemented method includes an act of causing a computer to execute such instructions and perform the resulting operations.
  • FIG. 5 is an image illustrating an image frame as being divided into N
  • FIG. 8 is a graph illustrating post-CNN (Stage 2) receiver operating
  • FIG. 9 A is an illustration depicting example height distribution of ground truth objects (e.g., a person, a dismount) in 88 training sequences for the side- facing sensors;
  • ground truth objects e.g., a person, a dismount
  • the computer system 100 is configured to perform calculations, processes, operations, and/or functions associated with a program or algorithm.
  • certain processes and steps discussed herein are realized as a series of instructions (e.g., software program) that reside within computer readable memory units and are executed by one or more processors of the computer system 100. When executed, the instructions cause the computer system 100 to perform specific actions and exhibit specific behavior, such as described herein.
  • processors configured to communicate information.
  • one or more data processing units such as a processor 104 (or processors) are coupled with the address/data bus 102.
  • the processor 104 is configured to process information and instructions.
  • the processor 104 is a microprocessor.
  • the processor 104 may be a different type of processor such as a parallel processor, application-specific integrated circuit (ASIC), programmable logic array (PLA), complex programmable logic device (CPLD), or a field
  • the computer system 100 is configured to utilize one or more data storage units.
  • the computer system 100 may include a volatile memory unit 106 (e.g., random access memory (“RAM”), static RAM, dynamic RAM, etc.) coupled with the address/data bus 102, wherein a volatile memory unit 106 is configured to store information and instructions for the processor 104.
  • the computer system 100 further may include a non-volatile memory unit 108 (e.g., read-only memory (“ROM”), programmable ROM (“PROM”), erasable programmable ROM
  • the communication interfaces implemented by the one or more interfaces may include wireline (e.g., serial cables, modems, network adaptors, etc.) and/or wireless (e.g., wireless modems, wireless network adaptors, etc.) communication technology.
  • wireline e.g., serial cables, modems, network adaptors, etc.
  • wireless e.g., wireless modems, wireless network adaptors, etc.
  • the computer system 100 may include an input device 112
  • the input device 112 may be an input device other than an alphanumeric input device.
  • the computer system 100 may include a cursor control device 114 coupled with the address/data bus 102, wherein the cursor control device 114 is configured to communicate user input information and/or command selections to the processor 100.
  • the cursor control device 114 is implemented using a device such as a mouse, a track-ball, a track pad, an optical tracking device, or a touch screen.
  • the cursor control device 114 is directed and/or activated via input from the input device 112, such as in response to the use of special keys and key sequence commands associated with the input device 112.
  • the cursor control device 114 is configured to be directed or guided by voice commands.
  • the computer system 100 further may include one or more
  • a storage device 116 coupled with the address/data bus 102.
  • the storage device 116 is configured to store information and/or computer executable instructions.
  • the storage device 116 is a storage device such as a magnetic or optical disk drive (e.g., hard disk drive (“HDD”), floppy diskette, compact disk read only memory (“CD-ROM”), digital versatile disk (“DVD”)).
  • a display device 118 is coupled with the address/data bus 102, wherein the display device 118 is configured to display video and/or graphics.
  • the display device 118 may include a cathode ray tube (“CRT”), liquid crystal display (“LCD”), field emission display (“FED”), plasma display, or any other display device suitable for displaying video and/or graphic images and alphanumeric characters recognizable to a user.
  • CTR cathode ray tube
  • LCD liquid crystal display
  • FED field emission display
  • plasma display or any other display device suitable for displaying video and/or graphic images and alphanumeric characters recognizable to a user.
  • the computer system 100 presented herein is an example computing
  • the non-limiting example of the computer system 100 is not strictly limited to being a computer system.
  • the computer system 100 represents a type of data processing analysis that may be used in accordance with various aspects described herein.
  • other computing systems may also be implemented.
  • the spirit and scope of the present technology is not limited to any single data processing environment.
  • one or more operations of various aspects of the present technology are controlled or implemented using computer-executable instructions, such as program modules, being executed by a computer.
  • program modules include routines, programs, objects, components and/or data structures that are configured to perform particular tasks or implement particular abstract data types.
  • an aspect provides that one or more aspects of the present technology are implemented by utilizing one or more distributed computing environments, such as where tasks are performed by remote processing devices that are linked through a communications network, or such as where various program modules are located in both local and remote computer- storage media including memory- storage devices.
  • FIG. 2 An illustrative diagram of a computer program product (i.e., storage device) embodying the present invention is depicted in FIG. 2.
  • the computer program product is depicted as floppy disk 200 or an optical disk 202 such as a CD or DVD.
  • the computer program product generally represents computer-readable instructions stored on any compatible non-transitory computer-readable medium.
  • the term“instructions” as used with respect to this invention generally indicates a set of operations to be performed on a computer, and may represent pieces of a whole program or individual, separable, software modules.
  • Non-limiting examples of“instruction” include computer program code (source or object code) and“hard-coded” electronics (i.e. computer operations coded into a computer chip).
  • The“instruction” is stored on any non-transitory computer-readable medium, such as in the memory of a computer or on a floppy disk, a CD-ROM, and a flash drive. In either event, the instructions are encoded on a non-transitory computer-readable medium.
  • the present disclosure provides an object detection and recognition system that uses both image and size/location features.
  • the system extends upon the disclosure of U.S. Application No. 15/883,822, which only used image features.
  • the system of the present disclosure is operable for: 1) learning from image and location data to accurately detect and recognize targets; 2) confidence tuning of detection results based on location data; and 3) combining all of the above into integrated system as a single pipeline.
  • this disclosure provides a marked technological improvement to the field and technologies as used for object detection and recognition.
  • the system of the present disclosure improves upon a three- stage cascaded classifier for target recognition in EO and IR videos from static or moving platforms.
  • the first stage is an Integral Channel Features (ICF) detector 300 that takes in video and runs a fast detection (e.g., greater than 15 frames per second) to provide high-confidence candidate target regions as bounding boxes and scores (e.g.,“MAN” or person target (or other object of interest)) in the video.
  • ICF Integral Channel Features
  • the ICF is based on agglomeration of“channel features” and the training of small decision trees using these features.
  • Basic features can be thought of as maps from raw pixel values (RGB/IR) to more informative features such as oriented gradients, Haar features, difference of regions or simply color-space transformations.
  • the output of the ICF detector 300 are detected target box locations and associated confidences.
  • the ICF detector 300 computes channel feature vectors for image frames of the video, and each image frame, an ICF classifier is applied at multiple image scales and across the entire image frame.
  • the system of the present disclosure adds a target size filter 302 to the system.
  • the target size filter 302 is applied to the output of the first stage to affect the confidence scores based on expected versus detected target size box.
  • the candidate bounding boxes with the modified confidence scores are then fed through a second stage, which is a convolutional neural net (CNN) classifier 304 that outputs target class, location and confidences.
  • CNN convolutional neural net
  • the CNN classifier 304 is implemented as interacting software modules comprising a CNN interface and a CNN server (e.g., one or more processors and corresponding memory), wherein the CNN interface displays results received from the CNN server.
  • the CNN interface takes the candidate target box information from the ICF detector 300 and extracts the image regions from the input video, and hands that off to the CNN server for classification.
  • the CNN interface can display it live and also log the results to disk files and provide the output target boxes for further processing.
  • the third stage is a multi-target tracker (MTT) 306 that tracks the target boxes from the CNN stage (i.e., the CNN classifier 304) for final target classification, locations and confidence scores.
  • the tracker results are fed to a comparator for further processing by the CNN stage.
  • the system empirically estimates the bottom of object (e.g., feet location) versus object height (i.e., top of object) in the image rather than constructing an analytical formula.
  • the image frame 500 e.g., 640x480 pixels
  • N e.g. 16
  • the ground truth objects are grouped into N bins 504 according to where the bottom (or feet) of the ground truth box is located in the images.
  • the system can estimate their height distribution. For example, a normal (Gaussian) distribution with mean and standard deviation (m, s) is used to represent this distribution. However, it is often convenient to illustrate the distribution via histogram plots, as described and illustrated.
  • the system learns to predict object class by combining image and location/size information to train an alternative neural network to produce classification results.
  • the top row processing employs the same deep convolution network (CNN-l) classifier 304 as in Stage 2, whereas the second row employs a modified convolution network (CNN-2) classifier 400.
  • the modified convolution network classifier 400 outputs target class, location and confidences, which are fused 402 with that from the CNN classifier 304 and provided to the MTT 306 that tracks the target boxes for final target classification, locations and confidence scores.
  • the 1024- D (dimensional) features 408 from the final convolution layer i.e., deep convolution layers 406 are padded with the target size and location 410 and fed to the fully connected (FC) layer 412 before the classifier layer.
  • the modified CNN-2 400 can be used alongside of the original CNN-l 304 and the results of CNN-l 304 and CNN-2 400 can be fused 402 to arrive at a final decision.
  • CNN-l 304 can be replaced with CNN-2 400 while maintaining the same processing flow.
  • Fusion 402 can be carried out by combining the probability distributions of CNN-l and CNN-2 over the set of classes to be classified; for example, a simple average of the weights of the classes from the two CNN’s and the renormalize to make the sum of weights to 1.0.
  • the first method of modifying the confidence score uses a weighted Gaussian according to the following equation:
  • FIGs. 6A and 6B show plots 604 depicting the target height 600 distributions of 88 training sequences of the IR data set from side-facing (shown in FIG. 6A) and front-facing (shown in FIG. 6B) sensors, respectively, based on ground truth (GT) information from human annotation.
  • GT ground truth
  • the height distribution 600 is collected in 16 horizontal bands (depicted as 16 plots in each of FIGs. 6 A and 6B) across image height of 480 rows, and plotted as 25-bin histograms with their Gaussian approximation (mean and standard deviation).
  • the histograms for each band are labeled with the image rows it covers. Where there are no histograms plotted means there is not sufficient GT target samples to support the histogram estimation for corresponding bands.
  • FIGs. 6A and 6B illustrate two things. First, when a histogram plot is missing in the corresponding bin, targets appearing in those bins are unlikely regardless of target height. Second, for the bins with histogram plots, the histograms are Gaussian-like.

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Evolutionary Computation (AREA)
  • Artificial Intelligence (AREA)
  • Data Mining & Analysis (AREA)
  • Multimedia (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • General Health & Medical Sciences (AREA)
  • General Engineering & Computer Science (AREA)
  • Health & Medical Sciences (AREA)
  • Biomedical Technology (AREA)
  • Molecular Biology (AREA)
  • Computing Systems (AREA)
  • Software Systems (AREA)
  • Computational Linguistics (AREA)
  • Mathematical Physics (AREA)
  • Biophysics (AREA)
  • Bioinformatics & Cheminformatics (AREA)
  • Bioinformatics & Computational Biology (AREA)
  • Evolutionary Biology (AREA)
  • Databases & Information Systems (AREA)
  • Medical Informatics (AREA)
  • Biodiversity & Conservation Biology (AREA)
  • Probability & Statistics with Applications (AREA)
  • Image Analysis (AREA)

Abstract

Described is an object recognition system. Using an integral channel features (ICF) detector, the system extracts a candidate target region (having an associated original confidence score representing a candidate object) from an input image of a scene surrounding a platform. A modified confidence score is generated based on a location and height of detection of the candidate object. The candidate target regions are classified based on the modified confidence score using a trained convolutional neural network (CNN) classifier, resulting in classified objects. The classified objects are tracked using a multi-target tracker for final classification of each classified object as a target or non-target. If the classified object is a target, a device can be controlled based on the target.

Description

[0001] SYSTEM FOR REAL-TIME OBJECT DETECTION AND RECOGNITION USING BOTH IMAGE AND SIZE FEATURES
[0002] GOVERNMENT RIGHTS
[0003] This invention was made with government support under U.S. Government
Contract Number W15P7T-10-D-D413. The government has certain rights in the invention.
[0004] CROSS-REFERENCE TO RELATED APPLICATIONS
[0005] This application is a Continuation-in-Part application of U.S. Application No.
15/883,822, filed on January 30, 2018, which is a non-provisional application of U.S. Provisional Application No. 62/479,204, filed March 30, 2017, the entirety of which are hereby incorporated by reference.
[0006] This application is ALSO a non-provisional patent application of U.S. provisional application Serial No. 62/659,100, filed on April 17, 2018, the entirety of which is hereby incorporated by reference.
[0007] BACKGROUND OF INVENTION
[0008] (1) Field of Invention
[0009] The present invention relates to an object detection system and, more specifically, to an object detection and recognition system using both image and size features.
[00010] (2) Description of Related Art
[00011] Object detection and recognition systems are often employed in autonomous vehicles and reconnaissance systems to quickly and automatically detect and recognize objects within a field-of-view. Traditional object detection and recognition systems attempt to identify the object based on image features of the object. While functional, such systems are limited in their inability to validate the recognition based on size features. [00012] Other attempts to determine size have been made using estimates of a ground- plane. See, for example, Dragon R., Van Gool L., "Ground plane estimation using a hidden Markov model", published at the 27th IEEE conference on computer vision and pattern recognition - CVPR 2014, pp. 4026-4033, June 23- 28, 2014, Columbus, Ohio, ETSA, the entirety of which is incorporated herein by reference. However, this approach can often fail when there are occlusions, shadows or other issues not providing a clear view of open space (e.g., vehicle moving in a wooded area).
[00013] Other prior art uses camera calibration to determine object height from an image. See, for example, G. Fiihr, C. R. Jung and M. B. d. Paula, "On the ETse of Calibration for Pedestrian Detection in On-Board Vehicular Cameras," published at the 2016 29th SIBGRAPI Conference on Graphics, Patterns and Images (SIBGRAPI), Sao Paulo, 2016, pp. 80-87, the entirety of which is incorporated herein by reference. Such a calibration approach fails when no camera calibration is available.
[00014] Thus, a continuing need exists for an object detection and recognition system that simultaneously learns from image and location data, that is robust to data type and accurate under a variety of target sizes and locations.
[00015] SUMMARY OF INVENTION
[00016] This disclosure provides an object recognition system. In various
embodiments, the system includes one or more processors and a memory. The memory includes executable instructions encoded thereon, such that upon execution of the instructions, the one or more processors perform the operations as described herein. For example, using an integral channel features (ICF) detector, the system extracts a candidate target region (having an associated original confidence score representing a candidate object) from an input image of a scene surrounding a platform. A modified confidence score is generated based on a location and height of detection of the candidate object. The candidate target regions are classified based on the modified confidence score using a trained convolutional neural network (CNN) classifier, resulting in classified objects.
The classified objects are tracked using a multi-target tracker for final
classification of each classified object as a target or non-target. If the classified object is a target, a device can be controlled based on the target.
[00017] In another aspect, the ICF detector computes channel feature vectors for
image frames of a video, and wherein, for each image frame, an ICF classifier is applied at multiple image scales and across the entire image frame.
[00018] In yet another aspect, the CNN classifier is implemented as interacting
software modules comprising a CNN interface and a CNN server, wherein the CNN interface displays results received from the CNN server.
[00019] In another aspect, the trained CNN is used for both electro-optical (EO) and infrared (IR) image classification.
[00020] In yet another aspect, the input image is divided into a plurality of horizontal bands and ground truth objects are put into a same number of groups based on whether a location of the ground truth objects in the input image is in the band, with the objects in each group being used to estimate the mean and standard deviation of object height distribution in the input image.
[00021] In another aspect, generating the modified confidence score uses a weighted Gaussian according to the following equation:
modified confidence score = original confidence score * wf , wherein h denotes a height of the candidate object in the input image, m and s denote mean and standard deviation, respectively, of object height distribution in the input image and bin, exp(.) denotes an exponential function, N is a multiplier and * denotes multiplication.
[00022] In yet another aspect, generating the modified confidence score uses a
weighted gate according to the following equation:
-. fl, if \\h— m\\ < Ns
, and
J (0, other ise
modified confidence score = original confidence score * wf , wherein h denotes a height of the candidate object in the input image, m and s denote mean and standard deviation, respectively, of object height distribution in the input image and bin, N is a multiplier and * denotes multiplication.
[00023] In yet another aspect, the one or more processors perform operations of
classifying the candidate target regions based on the modified confidence score using a modified convolution network (CNN-2) classifier, resulting in modified classified objects; and fusing the modified classified objects with the classified objects from the trained CNN classifier for processing by the multi-target tracker.
[00024] Finally, the present invention also includes a computer program product and a computer implemented method. The computer program product includes computer-readable instructions stored on a non-transitory computer-readable medium that are executable by a computer having one or more processors, such that upon execution of the instructions, the one or more processors perform the operations listed herein. Alternatively, the computer implemented method includes an act of causing a computer to execute such instructions and perform the resulting operations.
[00025] BRIEF DESCRIPTION OF THE DRAWINGS
[00026] The objects, features and advantages of the present invention will be apparent from the following detailed descriptions of the various aspects of the invention in conjunction with reference to the following drawings, where:
[00027] FIG. 1 is a block diagram depicting the components of a system according to various embodiments of the present invention;
[00028] FIG. 2 is an illustration of a computer program product embodying an aspect of the present invention;
[00029] FIG. 3 is a system block diagram according to various embodiments of the present invention;
[00030] FIG. 4 A is a system block diagram according to various embodiments of the present invention;
[00031] FIG. 4B is a system block diagram depicting a modified convolution network classifier according to various embodiments of the present invention;
[00032] FIG. 5 is an image illustrating an image frame as being divided into N
horizontal bands, with the ground truth objects being grouped into N bins;
[00033] FIG. 6 A is an illustration depicting example height distribution of objects in
88 training sequences for a side-facing sensor;
[00034] FIG. 6B is an illustration depicting example height distribution of objects in 88 training sequences for a front-facing sensor; [00035] FIG. 7 is a graph illustrating a comparison of detection scores weighted and unweighted for 30 test sequences;
[00036] FIG. 8 is a graph illustrating post-CNN (Stage 2) receiver operating
characteristics (ROCs) for the Gaussian and gated detection score weight approaches compared with the baseline ROC of unweighted scores;
[00037] FIG. 9 A is an illustration depicting example height distribution of ground truth objects (e.g., a person, a dismount) in 88 training sequences for the side- facing sensors;
[00038] FIG. 9B is an illustration depicting example height distribution of ground truth MAN objects in 88 training sequences for the front facing sensors;
[00039] FIG. 10 is a graph illustrating the result comparing detection score weighted versus unweights ROCs for 30 EO test sequences in both the Pre-CNN (Stage 1 + size filtering) and Post-CNN (Stage 2) results; and [00040] FIG. 11 is a block diagram depicting control of a device according to various embodiments.
[00041 ] DETAILED DESCRIPTION
[00042] The present invention relates to an object detection system and, more
specifically, to an object detection and recognition system using both image and size features. The following description is presented to enable one of ordinary skill in the art to make and use the invention and to incorporate it in the context of particular applications. Various modifications, as well as a variety of uses in different applications will be readily apparent to those skilled in the art, and the general principles defined herein may be applied to a wide range of aspects.
Thus, the present invention is not intended to be limited to the aspects presented, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.
[00043] In the following detailed description, numerous specific details are set forth in order to provide a more thorough understanding of the present invention.
However, it will be apparent to one skilled in the art that the present invention may be practiced without necessarily being limited to these specific details. In other instances, well-known structures and devices are shown in block diagram form, rather than in detail, in order to avoid obscuring the present invention.
[00044] The reader’s attention is directed to all papers and documents which are filed concurrently with this specification and which are open to public inspection with this specification, and the contents of all such papers and documents are incorporated herein by reference. All the features disclosed in this specification, (including any accompanying claims, abstract, and drawings) may be replaced by alternative features serving the same, equivalent or similar purpose, unless expressly stated otherwise. Thus, unless expressly stated otherwise, each feature disclosed is one example only of a generic series of equivalent or similar features. [00045] Furthermore, any element in a claim that does not explicitly state“means for” performing a specified function, or“step for” performing a specific function, is not to be interpreted as a“means” or“step” clause as specified in 35 U.S.C.
Section 112, Paragraph 6. In particular, the use of“step of’ or“act of’ in the claims herein is not intended to invoke the provisions of 35 U.S.C. 112, Paragraph 6.
[00046] Before describing the invention in detail, first a description of the various principal aspects of the present invention is provided. Subsequently, an introduction provides the reader with a general understanding of the present invention. Next, specific details of various embodiment of the present invention are provided to give an understanding of the specific aspects. Fourth, an example implementation with experimental results are provided. Finally, example implementations illustrating practical applications of the system are described.
[00047] (1) Principal Aspects
[00048] Various embodiments of the invention include three“principal” aspects. The first is a system for object detection and recognition. The system is typically in the form of a computer system operating software or in the form of a“hard- coded” instruction set. This system may be incorporated into a wide variety of devices that provide different functionalities. The second principal aspect is a method, typically in the form of software, operated using a data processing system (computer). The third principal aspect is a computer program product. The computer program product generally represents computer-readable instructions stored on a non-transitory computer-readable medium such as an optical storage device, e.g., a compact disc (CD) or digital versatile disc (DVD), or a magnetic storage device such as a floppy disk or magnetic tape. Other, non-limiting examples of computer-readable media include hard disks, read-only memory (ROM), and flash-type memories. These aspects will be described in more detail below.
[00049] A block diagram depicting an example of a system (i.e., computer system
100) of the present invention is provided in FIG. 1. The computer system 100 is configured to perform calculations, processes, operations, and/or functions associated with a program or algorithm. In one aspect, certain processes and steps discussed herein are realized as a series of instructions (e.g., software program) that reside within computer readable memory units and are executed by one or more processors of the computer system 100. When executed, the instructions cause the computer system 100 to perform specific actions and exhibit specific behavior, such as described herein.
[00050] The computer system 100 may include an address/data bus 102 that is
configured to communicate information. Additionally, one or more data processing units, such as a processor 104 (or processors), are coupled with the address/data bus 102. The processor 104 is configured to process information and instructions. In an aspect, the processor 104 is a microprocessor. Alternatively, the processor 104 may be a different type of processor such as a parallel processor, application-specific integrated circuit (ASIC), programmable logic array (PLA), complex programmable logic device (CPLD), or a field
programmable gate array (FPGA).
[00051] The computer system 100 is configured to utilize one or more data storage units. The computer system 100 may include a volatile memory unit 106 (e.g., random access memory ("RAM"), static RAM, dynamic RAM, etc.) coupled with the address/data bus 102, wherein a volatile memory unit 106 is configured to store information and instructions for the processor 104. The computer system 100 further may include a non-volatile memory unit 108 (e.g., read-only memory ("ROM"), programmable ROM ("PROM"), erasable programmable ROM
("EPROM"), electrically erasable programmable ROM "EEPROM"), flash memory, etc.) coupled with the address/data bus 102, wherein the non-volatile memory unit 108 is configured to store static information and instructions for the processor 104. Alternatively, the computer system 100 may execute instructions retrieved from an online data storage unit such as in“Cloud” computing. In an aspect, the computer system 100 also may include one or more interfaces, such as an interface 110, coupled with the address/data bus 102. The one or more interfaces are configured to enable the computer system 100 to interface with other electronic devices and computer systems. The communication interfaces implemented by the one or more interfaces may include wireline (e.g., serial cables, modems, network adaptors, etc.) and/or wireless (e.g., wireless modems, wireless network adaptors, etc.) communication technology.
[00052] In one aspect, the computer system 100 may include an input device 112
coupled with the address/data bus 102, wherein the input device 112 is configured to communicate information and command selections to the processor 100. In accordance with one aspect, the input device 112 is an alphanumeric input device, such as a keyboard, that may include alphanumeric and/or function keys.
Alternatively, the input device 112 may be an input device other than an alphanumeric input device. In an aspect, the computer system 100 may include a cursor control device 114 coupled with the address/data bus 102, wherein the cursor control device 114 is configured to communicate user input information and/or command selections to the processor 100. In an aspect, the cursor control device 114 is implemented using a device such as a mouse, a track-ball, a track pad, an optical tracking device, or a touch screen. The foregoing notwithstanding, in an aspect, the cursor control device 114 is directed and/or activated via input from the input device 112, such as in response to the use of special keys and key sequence commands associated with the input device 112. In an alternative aspect, the cursor control device 114 is configured to be directed or guided by voice commands.
[00053] In an aspect, the computer system 100 further may include one or more
optional computer usable data storage devices, such as a storage device 116, coupled with the address/data bus 102. The storage device 116 is configured to store information and/or computer executable instructions. In one aspect, the storage device 116 is a storage device such as a magnetic or optical disk drive (e.g., hard disk drive ("HDD"), floppy diskette, compact disk read only memory ("CD-ROM"), digital versatile disk ("DVD")). Pursuant to one aspect, a display device 118 is coupled with the address/data bus 102, wherein the display device 118 is configured to display video and/or graphics. In an aspect, the display device 118 may include a cathode ray tube ("CRT"), liquid crystal display ("LCD"), field emission display ("FED"), plasma display, or any other display device suitable for displaying video and/or graphic images and alphanumeric characters recognizable to a user.
[00054] The computer system 100 presented herein is an example computing
environment in accordance with an aspect. However, the non-limiting example of the computer system 100 is not strictly limited to being a computer system. For example, an aspect provides that the computer system 100 represents a type of data processing analysis that may be used in accordance with various aspects described herein. Moreover, other computing systems may also be implemented. Indeed, the spirit and scope of the present technology is not limited to any single data processing environment. Thus, in an aspect, one or more operations of various aspects of the present technology are controlled or implemented using computer-executable instructions, such as program modules, being executed by a computer. In one implementation, such program modules include routines, programs, objects, components and/or data structures that are configured to perform particular tasks or implement particular abstract data types. In addition, an aspect provides that one or more aspects of the present technology are implemented by utilizing one or more distributed computing environments, such as where tasks are performed by remote processing devices that are linked through a communications network, or such as where various program modules are located in both local and remote computer- storage media including memory- storage devices.
[00055] An illustrative diagram of a computer program product (i.e., storage device) embodying the present invention is depicted in FIG. 2. The computer program product is depicted as floppy disk 200 or an optical disk 202 such as a CD or DVD. However, as mentioned previously, the computer program product generally represents computer-readable instructions stored on any compatible non-transitory computer-readable medium. The term“instructions” as used with respect to this invention generally indicates a set of operations to be performed on a computer, and may represent pieces of a whole program or individual, separable, software modules. Non-limiting examples of“instruction” include computer program code (source or object code) and“hard-coded” electronics (i.e. computer operations coded into a computer chip). The“instruction” is stored on any non-transitory computer-readable medium, such as in the memory of a computer or on a floppy disk, a CD-ROM, and a flash drive. In either event, the instructions are encoded on a non-transitory computer-readable medium.
[00056] (2) Introduction
[00057] The present disclosure provides an object detection and recognition system that uses both image and size/location features. The system extends upon the disclosure of U.S. Application No. 15/883,822, which only used image features. The system of the present disclosure is operable for: 1) learning from image and location data to accurately detect and recognize targets; 2) confidence tuning of detection results based on location data; and 3) combining all of the above into integrated system as a single pipeline. Upon review of the system described below and corresponding performance evaluations, it is evident that this disclosure provides a marked technological improvement to the field and technologies as used for object detection and recognition. [00058] (3) Specific Details of Various Embodiments
As shown in FIG. 3, the system of the present disclosure improves upon a three- stage cascaded classifier for target recognition in EO and IR videos from static or moving platforms. Specific details regarding the three-staged classifier can be found in U.S. Application No. 15/883,822. The first stage is an Integral Channel Features (ICF) detector 300 that takes in video and runs a fast detection (e.g., greater than 15 frames per second) to provide high-confidence candidate target regions as bounding boxes and scores (e.g.,“MAN” or person target (or other object of interest)) in the video. The ICF is based on agglomeration of“channel features” and the training of small decision trees using these features. Basic features can be thought of as maps from raw pixel values (RGB/IR) to more informative features such as oriented gradients, Haar features, difference of regions or simply color-space transformations. The output of the ICF detector 300 are detected target box locations and associated confidences. The ICF detector 300 computes channel feature vectors for image frames of the video, and each image frame, an ICF classifier is applied at multiple image scales and across the entire image frame.
[00059] To be contrasted with the three-stage classifier as described in U.S.
Application No. 15/883,822, the system of the present disclosure adds a target size filter 302 to the system. The target size filter 302 is applied to the output of the first stage to affect the confidence scores based on expected versus detected target size box. The candidate bounding boxes with the modified confidence scores are then fed through a second stage, which is a convolutional neural net (CNN) classifier 304 that outputs target class, location and confidences.
[00060] In various embodiments, the CNN classifier 304 is implemented as interacting software modules comprising a CNN interface and a CNN server (e.g., one or more processors and corresponding memory), wherein the CNN interface displays results received from the CNN server. The CNN interface takes the candidate target box information from the ICF detector 300 and extracts the image regions from the input video, and hands that off to the CNN server for classification. When the results are received from the CNN server, the CNN interface can display it live and also log the results to disk files and provide the output target boxes for further processing. [00061] The third stage is a multi-target tracker (MTT) 306 that tracks the target boxes from the CNN stage (i.e., the CNN classifier 304) for final target classification, locations and confidence scores. In an alternative embodiment, the tracker results are fed to a comparator for further processing by the CNN stage.
[00062] The idea of imposing object size constraints is as follows. When a camera senses a dismount (e.g.,“MAN” or person (or other object of interest)) on a flat surface, the height of the subject in the image is directly related to the distance of the subject to the camera, which is reflected in the location of subject’s feet in the image. Given the image row where the subject’s feet are located in the image, the system can calculate (or predict) the subject’s height in the image precisely given the height of the subject, the camera intrinsic, camera height above ground and the camera tilt. By comparing the predicted height and the height of detection boxes in image, the system can provide a confidence of the detection according to how well the two match.
[00063] The system empirically estimates the bottom of object (e.g., feet location) versus object height (i.e., top of object) in the image rather than constructing an analytical formula. In doing so and as shown in FIG. 5, the image frame 500 (e.g., 640x480 pixels) is divided into N (e.g., 16) horizontal bands 502, and the ground truth objects are grouped into N bins 504 according to where the bottom (or feet) of the ground truth box is located in the images. From the group of objects in each bin, the system can estimate their height distribution. For example, a normal (Gaussian) distribution with mean and standard deviation (m, s) is used to represent this distribution. However, it is often convenient to illustrate the distribution via histogram plots, as described and illustrated.
[00064] To use the above empirical height distribution to affect confidence in object detection and thereby improve the system performance, the detector or confidence score coming out of the ICF detector 300 is modified. Post-processing is used to evaluate this approach in terms of its impact on system receiver operating characteristics (ROC) so that no real detector or classification processing is required. This evaluation approach only involves the first two stages (i.e., ICF detector 300 and CNN Classifier 304) along with the target size filter 302, and does not involve the third stage (i.e., MTT 306). To modify the confidence score, the system computes and applies a multiplicative weight factor (wj) on the detection confidence score based on the location and height of the detection. Two methods to compute wf are described further below.
[00065] In a second embodiment and as shown in FIG. 4A, the system learns to predict object class by combining image and location/size information to train an alternative neural network to produce classification results. In this embodiment, the top row processing employs the same deep convolution network (CNN-l) classifier 304 as in Stage 2, whereas the second row employs a modified convolution network (CNN-2) classifier 400. The modified convolution network classifier 400 outputs target class, location and confidences, which are fused 402 with that from the CNN classifier 304 and provided to the MTT 306 that tracks the target boxes for final target classification, locations and confidence scores.
[00066] The modified convolution network classifier is expanded and further
illustrated in FIG. 4B. After receiving the input 404 from the ICF 300, the 1024- D (dimensional) features 408 from the final convolution layer (i.e., deep convolution layers 406) are padded with the target size and location 410 and fed to the fully connected (FC) layer 412 before the classifier layer. The modified CNN-2 400 can be used alongside of the original CNN-l 304 and the results of CNN-l 304 and CNN-2 400 can be fused 402 to arrive at a final decision.
Alternatively, CNN-l 304 can be replaced with CNN-2 400 while maintaining the same processing flow. Fusion 402 can be carried out by combining the probability distributions of CNN-l and CNN-2 over the set of classes to be classified; for example, a simple average of the weights of the classes from the two CNN’s and the renormalize to make the sum of weights to 1.0.
[00067] (3.1) Method 1 : Weighted Gaussian
[00068] The first method of modifying the confidence score uses a weighted Gaussian according to the following equation:
new score = original score * wf ,
wherein h denotes a height of a detected object in the image whose detection confidence score is to be modified m and s denote mean and standard deviation, respectively, of object height distribution in the corresponding image and bin. exp(.) denotes the exponential function. In the case where the histogram corresponding to an image band is missing (i.e., no estimation of m and s), the multiplicative weight factor (wf) is set to zero (i.e., wf=0.0). Further, N = { 1, 2, 3, 4 ... } is a multiplier to relax the detection size constraints. The multiplicative weight factor (wf) is then multiplied with the original score to derive the new and modified confidence score (i.e., new score).
[00069] It was determined through experiments that N = 1 is too restrictive and does not result in appreciative improvement in system performance, while N=4 provides most of the benefits. FIGs. 6A and 6B show plots 604 depicting the target height 600 distributions of 88 training sequences of the IR data set from side-facing (shown in FIG. 6A) and front-facing (shown in FIG. 6B) sensors, respectively, based on ground truth (GT) information from human annotation.
The height distribution 600 is collected in 16 horizontal bands (depicted as 16 plots in each of FIGs. 6 A and 6B) across image height of 480 rows, and plotted as 25-bin histograms with their Gaussian approximation (mean and standard deviation). The histograms for each band are labeled with the image rows it covers. Where there are no histograms plotted means there is not sufficient GT target samples to support the histogram estimation for corresponding bands. [00070] For further understanding, FIGs. 6A and 6B illustrate two things. First, when a histogram plot is missing in the corresponding bin, targets appearing in those bins are unlikely regardless of target height. Second, for the bins with histogram plots, the histograms are Gaussian-like. That is, the underlying target heights can be modeled well by Gaussian distributions. As such, once an estimated target height for a particular bin is estimated, the likelihood of the target detection to be true can be estimated based on such distributions. These observations support the approach laid out here for Method 1.
[00071] (3.2) Method 2: Weighted Gate
[00072] The second method of modifying the confidence score uses a weighted gate according to the following equation:
-. fl, if \\h— m\\ < Ns
J (0, other ise
[00073] This basically gates the detection box height ( h ) around the mean (m) of the GT box height for the corresponding image band. If the detection height is within the gate, the original score is maintained. Otherwise, the original score is zeroed out, effectively eliminating that detection from further processing. This approach is referred to as“weighted gated”. Similar to Method 1, in the case where the histogram corresponding to an image band is missing (i.e., no estimation of m and s), wf=0. As was the case above, once the multiplicative weight factor (wf) is determined, it is then multiplied with the original score to derive the new and modified confidence score. [00074] In experiments, it was found that the“weighted gate” approach was similarly effective in improving the detection performance in terms of ROC when N is >3, while N=l achieved poor performance. For example, FIG. 8 is a graph illustrating post-CNN (Stage 2) ROCs for the Gaussian and gated detection score weight approaches as compared with the baseline ROC of unweighted scores. As can be seen, both Gaussian and gated approaches performed about the same (for N=4), and are about 5% better than the baseline.
[00075] (4) Experimental Results
[00076] The embodiment as illustrated in FIG. 3 has been implemented and tested for the task of recognizing dismounts (“MAN” target) and their activities in both EO and IR videos from stationary and moving ground vehicles.
[00077] (4.1) IR video data
[00078] As shown in FIGs. 6 A and 6B, a set of IR video sequences was collected from a moving vehicle and the histogram distributions of the height 600 of the ground- truth (GT) boxes for objects 602 were plotted to form a collection of collection of height histogram plots 604. Since the sensors used for side-facing IR sensor and front-facing IR sensor sequences have different tilt angles, the histogram gathering was performed separately for the hlco and h2co sequences. These are shown in FIGs. 6A and 6B discussed before. As can be seen from the histogram plots in FIGs. 6A and 6B, the GT height 600 decreases monotonically from the bottom of the image (rows 450-479) 606 towards the top, and ends in rows 30-59 608 with a mean of 23.9 pixels for hlco, and rows 180-209 610 with a mean of 17.2 for h2co. For both hlco and h2co, the greatest concentration of GT MAN-* objects are located in the 3-4 bands just below the bands with shortest targets listed above.
[00079] The weight factor wf as described above was computed, and the detection scores were modified using wf. The rest of the pipeline shown in FIG. 3 was performed, with the performances evaluated. Experiments demonstrated how the weighted Gaussian (Method 1) improved the effectiveness of the confidence scores. Specifically, FIG. 7 is a graph illustrating a comparison of detection scores weighted and unweighted for 30 IRtest sequences in both Pre-CNN (Stage 1) and Post-CNN (Stage 2) receiver operating characteristic (ROC) curves. Only the Gaussian window approach is shown since the Gated approach (Method 2) achieves almost identical results. As can be seen from these plots in FIG.7, we get up to 7% (0.07) higher Pd (probability of correct detection) for any given false positive per image (FPPI) or close to 50% reduction in FPPI (e.g., at a little above 75% Pd level).
[00080] (4.2) EO video data
[00081] The same experiments and analysis were conducted on visible band (EO) video data. The results are shown in FIGs. 9A and 9B, respectively. Specifically, FIGs. 9A and 9B are graphs (i.e., collection of height histogram plots 900) illustrating height distribution of GT“MAN*” objects in 88 training sequences for side-facing color (or EO) sensors, FIG. 9A) and front-facing EO sensors, FIG. 9B). The height distribution is collected in 16 horizontal bands across image height of 768 rows, and plotted as 25-bin histograms with the Gaussian approximation (from mean and stand deviation) overlaid. The histograms for each band are labeled with the image rows it covers. Where there are no histogram plotted, it means there is not sufficient GT entries to support the histogram estimation. [00082] These histograms do not look much different than those from IR sequences as presented in the previous section (see FIGs. 6A and 6B). The trend is the same though the specific numbers differ: as the targets move closer to the sensor (the target box’s bottom edges move closer to the bottom of the image) the height of target appears larger as represented by the mean of the plots. The estimated distributions shown in FIGs. 9A and 8B were used to weight the MAN object detection scores in the same way as discussed for IR images. FIG. 10 shows the result comparing detection score weighted versus unweights ROCs for 30 EO test sequences in both the Pre-CNN (Stage 1 + size filtering) and Post-CNN (Stage 2) results. As can be seen, the detection ROC (pre-CNN) from weighted scores has
~4% performance advantage higher than the unweighted ROC, but the benefit almost disappears for Post-CNN ROC. It is hypothesized that this happens because the CNN trained in EO is performing very well already, and there is little the improved detection can help further improve the situation. This is contrary to the IR case. The CNN in IR domain is not as good as the one in EO, because the target objects tend to be smaller and contain less distinguishing texture.
[00083] Thus, described above is an object detection and recognition system that improves upon the work described in ET.S. Application No. 15/883,822 with the addition of detection size filtering. The experiments showed positive
improvements on overall detection performance (5~7% in Pd, and close to 50% reduction in FPPI) both before (i.e. with Stage 1 only) and after CNN recognition (with Stages 1 and 2) compared with the baseline according to system ROCs. [00084] (5) Control of a Device
[00085] The invention described herein allows EO or IR vision-based object/target recognition in real-time even on small, low power, low cost platform (such ETAVs and ETGVs). This approach is also amenable for implementation on emerging spiking neuromorphic hardware, for example, a neuromorphic chip. The system according to embodiments of the present disclosure can be used in intelligence, surveillance, and reconnaissance (ISR) operations, border security, and mission safety, such as for UAV based surveillance, human activity detection, threat detection, and distributed mobile operations. For example, for a military application, the classified object output can be used to alert (via audible, tactile, and/or visual alert, etc.) the driver/team that there is a high-confidence“MAN” target and its location. The vehicle can then take evasive action by causing the vehicle to change route, etc., or attack that target after manual confirmation of its danger. For a remotely operated vehicle, it can also provide a similar alert.
[00086] Additionally, the system can be embedded in autonomous robotic vehicles, such as UAVs and UGVs, and self-driving vehicles. For instance, in a self- driving vehicle, the system can be used for collision avoidance. In this example, if the system detects an object (target) in its path (e.g., a pedestrian, another vehicle), an alert is sent to the vehicle operating system to cause the vehicle to perform a braking operation. Alternatively, the alert may signal that the vehicle operating system should perform a swerving motion around the object (target), involving steering and accelerating operations or any other operations as required to provide for collision avoidance. Further, the object detected may be a road sign, such as a stop sign. Upon classification of the stop sign, an alert can be sent to the vehicle operating system causing the vehicle to brake or otherwise adhere to the message as conveyed by the road sign. Therefore and as noted above, the system and process described herein can be used to control a variety of devices, such as causing said device to perform an operation or physical maneuver.
[00087] FIG. 11 is a flow diagram illustrating using a processor 1100 to control a device 1102 based on classification of an object as a target. Non-limiting examples of devices 1102 that can be controlled via the processor 1100 and the classification of the target object include a vehicle or a vehicle component, such as a brake, acceleration/deceleration controls, a steering mechanism, suspension, or safety device (e.g., airbags, seatbelt tensioners, etc.), or any combination thereof. Further, the vehicle could be an unmanned aerial vehicle (UAV), an autonomous ground vehicle, or a human operated vehicle controlled either by a driver or by a remote operator. As can be appreciated by one skilled in the art, control of other device types is also possible given classification of an object as a target and the corresponding circumstances in which the system is employed.
[00088] Finally, while this invention has been described in terms of several
embodiments, one of ordinary skill in the art will readily recognize that the invention may have other applications in other environments. It should be noted that many embodiments and implementations are possible. Further, the following claims are in no way intended to limit the scope of the present invention to the specific embodiments described above. In addition, any recitation of“means for” is intended to evoke a means-plus-function reading of an element and a claim, whereas, any elements that do not specifically use the recitation“means for”, are not intended to be read as means-plus-function elements, even if the claim otherwise includes the word“means”. Further, while particular method steps have been recited in a particular order, the method steps may occur in any desired order and fall within the scope of the present invention.

Claims

CLAIMS What is claimed is:
1. An object recognition system, the system comprising:
one or more processors and a non-transitory computer-readable medium having executable instructions encoded thereon such that when executed, the one or more processors perform operations of:
using an integral channel features (ICF) detector, extracting a candidate target region from an input image of a scene surrounding a platform, wherein the candidate target region has an associated original confidence score representing a candidate object;
generating a modified confidence score based on a location and height of detection of the candidate object;
classifying the candidate target regions based on the modified confidence score using a trained convolutional neural network (CNN) classifier, resulting in classified objects;
using a multi-target tracker, tracking the classified objects for final classification of each classified object as a target or non-target; and
if the classified object is a target, then controlling a device based on the target.
2. The system as set forth in Claim 1, wherein the ICF detector computes channel feature vectors for image frames of a video, and wherein, for each image frame, an ICF classifier is applied at multiple image scales and across the entire image frame.
3. The system as set forth in Claim 1, wherein the CNN classifier is implemented as interacting software modules comprising a CNN interface and a CNN server, wherein the CNN interface displays results received from the CNN server.
4. The system as set forth in Claim 1, wherein the trained CNN is used for both electro- optical (EO) and infrared (IR) image classification.
The system as set forth in Claim 1, wherein the input image is divided into a plurality of horizontal bands and ground truth objects are put into a same number of groups based on whether a location of the ground truth objects in the input image is in the band, with the objects in each group being used to estimate the mean and standard deviation of object height distribution in the input image.
6. The system as set forth in Claim 1, wherein generating the modified confidence score uses a weighted Gaussian according to the following equation:
modified confidence score = original confidence score * wf , wherein h denotes a height of the candidate object in the input image, m and s denote mean and standard deviation, respectively, of object height distribution in the input image and bin, exp(.) denotes an exponential function, N is a multiplier and * denotes multiplication.
7. The system as set forth in Claim 1, wherein generating the modified confidence score uses a weighted gate according to the following equation:
-. fl, if \\h— m|| < Ns
, and
J (0, other ise
modified confidence score = original confidence score * wf , wherein h denotes a height of the candidate object in the input image, m and s denote mean and standard deviation, respectively, of object height distribution in the input image and bin, N is a multiplier and * denotes multiplication.
8. The system as set forth in Claim 1, further comprising operations of:
classifying the candidate target regions based on the modified confidence score using a modified convolution network (CNN-2) classifier, resulting in modified classified objects; and
fusing the modified classified objects with the classified objects from the trained CNN classifier for processing by the multi-target tracker.
9. A computer program product for object recognition the computer program product comprising:
a non-transitory computer-readable medium having executable instructions encoded thereon, such that upon execution of the instructions by one or more processors, the one or more processors perform operations of:
using an integral channel features (ICF) detector, extracting a candidate target region from an input image of a scene surrounding a platform, wherein the candidate target region has an associated original confidence score representing a candidate object;
generating a modified confidence score based on a location and height of detection of the candidate object;
classifying the candidate target regions based on the modified confidence score using a trained convolutional neural network (CNN) classifier, resulting in classified objects;
using a multi-target tracker, tracking the classified objects for final classification of each classified object as a target or non-target; and
if the classified object is a target, then controlling a device based on the target.
10. The computer program product as set forth in Claim 9, wherein the ICF detector computes channel feature vectors for image frames of a video, and wherein, for each image frame, an ICF classifier is applied at multiple image scales and across the entire image frame.
11. The computer program product as set forth in Claim 9, wherein the CNN classifier is implemented as interacting software modules comprising a CNN interface and a CNN server, wherein the CNN interface displays results received from the CNN server.
12. The computer program product as set forth in Claim 9, wherein the trained CNN is used for both electro-optical (EO) and infrared (IR) image classification.
13. The computer program product as set forth in Claim 9, wherein the input image is divided into a plurality of horizontal bands and ground truth objects are put into a same number of groups based on whether a location of the ground truth objects in the input image is in the band, with the objects in each group being used to estimate the mean and standard deviation of object height distribution in the input image.
14. The computer program product as set forth in Claim 9, wherein generating the
modified confidence score uses a weighted Gaussian according to the following equation:
modified confidence score = original confidence score * wf , wherein h denotes a height of the candidate object in the input image, m and s denote mean and standard deviation, respectively, of object height distribution in the input image and bin, exp(.) denotes an exponential function, N is a multiplier and * denotes multiplication.
15. The computer program product as set forth in Claim 9, wherein generating the modified confidence score uses a weighted gate according to the following equation:
-. fl, if \\h— m\\ < Ns
, and
J (0, other ise
modified confidence score = original confidence score * wf , wherein h denotes a height of the candidate object in the input image, m and s denote mean and standard deviation, respectively, of object height distribution in the input image and bin, N is a multiplier and * denotes multiplication.
16. The computer program product as set forth in Claim 9, further comprising operations of:
classifying the candidate target regions based on the modified confidence score using a modified convolution network (CNN-2) classifier, resulting in modified classified objects; and
fusing the modified classified objects with the classified objects from the trained CNN classifier for processing by the multi-target tracker.
17. A computer implemented method for object recognition, the method comprising an act of:
using an integral channel features (ICF) detector, extracting a candidate target region from an input image of a scene surrounding a platform, wherein the candidate target region has an associated original confidence score representing a candidate object;
generating a modified confidence score based on a location and height of detection of the candidate object;
classifying the candidate target regions based on the modified confidence score using a trained convolutional neural network (CNN) classifier, resulting in classified objects; using a multi-target tracker, tracking the classified objects for final classification of each classified object as a target or non-target; and
if the classified object is a target, then controlling a device based on the target.
18. The method as set forth in Claim 17, wherein the ICF detector computes channel feature vectors for image frames of a video, and wherein, for each image frame, an ICF classifier is applied at multiple image scales and across the entire image frame.
19. The method as set forth in Claim 17, wherein the CNN classifier is implemented as interacting software modules comprising a CNN interface and a CNN server, wherein the CNN interface displays results received from the CNN server.
20. The method as set forth in Claim 17, wherein the trained CNN is used for both electro-optical (EO) and infrared (IR) image classification.
21. The method as set forth in Claim 17, wherein the input image is divided into a plurality of horizontal bands and ground truth objects are put into a same number of groups based on whether a location of the ground truth objects in the input image is in the band, with the objects in each group being used to estimate the mean and standard deviation of object height distribution in the input image.
22. The method as set forth in Claim 17, wherein generating the modified confidence score uses a weighted Gaussian according to the following equation:
modified confidence score = original confidence score * wf , wherein h denotes a height of the candidate object in the input image, m and s denote mean and standard deviation, respectively, of object height distribution in the input image and bin, exp(.) denotes an exponential function, N is a multiplier and * denotes multiplication.
23. The method as set forth in Claim 17, wherein generating the modified confidence score uses a weighted gate according to the following equation:
-. fl, if \\h— m\\ < Ns
, and
J (.0, other ise
modified confidence score = original confidence score * wf , wherein h denotes a height of the candidate object in the input image, m and s denote mean and standard deviation, respectively, of object height distribution in the input image and bin, N is a multiplier and * denotes multiplication.
24. The method as set forth in Claim 17, further comprising acts of:
classifying the candidate target regions based on the modified confidence score using a modified convolution network (CNN-2) classifier, resulting in modified classified objects; and
fusing the modified classified objects with the classified objects from the trained CNN classifier for processing by the multi-target tracker.
EP19789101.3A 2018-04-17 2019-02-14 System for real-time object detection and recognition using both image and size features Pending EP3782075A4 (en)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
US201862659100P 2018-04-17 2018-04-17
PCT/US2019/018119 WO2019203921A1 (en) 2018-04-17 2019-02-14 System for real-time object detection and recognition using both image and size features

Publications (2)

Publication Number Publication Date
EP3782075A1 true EP3782075A1 (en) 2021-02-24
EP3782075A4 EP3782075A4 (en) 2021-12-29

Family

ID=68239189

Family Applications (1)

Application Number Title Priority Date Filing Date
EP19789101.3A Pending EP3782075A4 (en) 2018-04-17 2019-02-14 System for real-time object detection and recognition using both image and size features

Country Status (3)

Country Link
EP (1) EP3782075A4 (en)
CN (1) CN111801689B (en)
WO (1) WO2019203921A1 (en)

Families Citing this family (12)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN111275054B (en) * 2020-01-16 2023-10-31 北京迈格威科技有限公司 Image processing methods, devices, electronic equipment and storage media
US12573190B2 (en) 2020-10-27 2026-03-10 Telefonaktiebolaget Lm Ericsson (Publ) Computer software module arrangement, a circuitry arrangement, an arrangement and a method for improved object detection
US12602905B2 (en) 2020-10-27 2026-04-14 Telefonaktiebolaget Lm Ericsson (Publ) Computer software module arrangement, a circuitry arrangement, an arrangement and a method for improved object detection adapting the detection through shifting the image
CN116529727A (en) * 2020-10-27 2023-08-01 瑞典爱立信有限公司 Computer software module device, circuit module device, device and method for improving object detection by compensating confidence determination of detected object
CN112633323B (en) * 2020-11-26 2024-04-30 成都佳发安泰教育科技股份有限公司 A posture detection method and system for classroom
CN112560726B (en) * 2020-12-22 2023-08-29 阿波罗智联(北京)科技有限公司 Target detection confidence determination method, roadside equipment and cloud control platform
US11900652B2 (en) * 2021-03-05 2024-02-13 Mujin, Inc. Method and computing system for generating a safety volume list for object detection
KR20220131646A (en) * 2021-03-22 2022-09-29 현대자동차주식회사 Method and apparatus for tracking an object, and recording medium for recording program performing the method
CN117075130B (en) * 2023-07-07 2024-06-25 中国电子科技集团公司第三十八研究所 Low-speed, slow, small-target laser tracking device and its working method
CN119648992B (en) * 2024-11-25 2025-09-16 北京理工大学 Remote sensing image road extraction method
CN120823566B (en) * 2025-09-18 2025-11-21 中南信息科技(深圳)有限公司 AI-based video analytics-based safety production monitoring methods and systems
CN121170276B (en) * 2025-11-21 2026-02-03 浙江工业大学 A target detection method and system based on scene prior and hierarchical detection mechanism

Family Cites Families (14)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
GB2341231A (en) * 1998-09-05 2000-03-08 Sharp Kk Face detection in an image
CN101615252B (en) * 2008-06-25 2012-07-04 中国科学院自动化研究所 Method for extracting text information from adaptive images
US8050459B2 (en) * 2008-07-25 2011-11-01 GM Global Technology Operations LLC System and method for detecting pedestrians
EP2863359A4 (en) * 2012-06-14 2015-11-11 Toyota Motor Co Ltd DISCRIMINATION CONTAINER GENERATION DEVICE AND PATTERN DETECTION DEVICE
US9613273B2 (en) * 2015-05-19 2017-04-04 Toyota Motor Engineering & Manufacturing North America, Inc. Apparatus and method for object tracking
US9582895B2 (en) * 2015-05-22 2017-02-28 International Business Machines Corporation Real-time object analysis with occlusion handling
CN105760858A (en) * 2016-03-21 2016-07-13 东南大学 Pedestrian detection method and apparatus based on Haar-like intermediate layer filtering features
CN105913003B (en) * 2016-04-07 2019-06-07 国家电网公司 A kind of pedestrian detection method of multiple features multi-model
US20190150764A1 (en) * 2016-05-02 2019-05-23 The Regents Of The University Of California System and Method for Estimating Perfusion Parameters Using Medical Imaging
EP3261017A1 (en) * 2016-06-20 2017-12-27 Delphi Technologies, Inc. Image processing system to detect objects of interest
US10083369B2 (en) * 2016-07-01 2018-09-25 Ricoh Company, Ltd. Active view planning by deep learning
CN107092883A (en) * 2017-04-20 2017-08-25 上海极链网络科技有限公司 Object identification method for tracing
CN107273832B (en) * 2017-06-06 2020-09-22 青海省交通科学研究院 License plate recognition method and system based on integral channel characteristics and convolutional neural network
CN107679525B (en) * 2017-11-01 2022-11-29 腾讯科技（深圳）有限公司 Image classification method, device and computer-readable storage medium

Also Published As

Publication number Publication date
CN111801689B (en) 2024-07-16
CN111801689A (en) 2020-10-20
EP3782075A4 (en) 2021-12-29
WO2019203921A1 (en) 2019-10-24

Similar Documents

Publication Publication Date Title
US10699139B2 (en) System for real-time object detection and recognition using both image and size features
EP3782075A1 (en) System for real-time object detection and recognition using both image and size features
US11055872B1 (en) Real-time object recognition using cascaded features, deep learning and multi-target tracking
Li et al. Fast and robust UAV to UAV detection and tracking from video
Wang et al. Pedestrian recognition and tracking using 3D LiDAR for autonomous vehicle
US10289934B2 (en) Landmark localization on objects in images using convolutional neural networks
US8948501B1 (en) Three-dimensional (3D) object detection and multi-agent behavior recognition using 3D motion data
US10289910B1 (en) System and method for performing real-time video object recognition utilizing convolutional neural networks
US20210110180A1 (en) Method and apparatus for traffic sign detection, electronic device and computer storage medium
US9569531B2 (en) System and method for multi-agent event detection and recognition
WO2020020472A1 (en) A computer-implemented method and system for detecting small objects on an image using convolutional neural networks
Zhang et al. Autonomous long-range drone detection system for critical infrastructure safety
James et al. Learning to detect aircraft for long-range vision-based sense-and-avoid systems
Liu et al. Trajectory and image-based detection and identification of UAV
US10691972B2 (en) Machine-vision system for discriminant localization of objects
Premebida et al. A multi-target tracking and GMM-classifier for intelligent vehicles
US10332265B1 (en) Robust recognition on degraded imagery by exploiting known image transformation under motion
Zhang et al. An intruder detection algorithm for vision based sense and avoid system
US11727575B2 (en) 3-D object detection and classification from imagery
Chandana et al. Autonomous drones based forest surveillance using Faster R-CNN
Talbi et al. An overview on computer vision analysis in the airport applications
Sezgin et al. Advancements in object detection for unmanned aerial vehicles: Applications, challenges, and future perspectives
Imad et al. Navigation system for autonomous vehicle: A survey
Demars et al. Multispectral detection and tracking of multiple moving targets in cluttered urban environments
Bhusal Object detection and tracking in wide area surveillance using thermal imagery

Legal Events

Date Code Title Description
STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: THE INTERNATIONAL PUBLICATION HAS BEEN MADE

PUAI Public reference made under article 153(3) epc to a published international application that has entered the european phase

Free format text: ORIGINAL CODE: 0009012

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: REQUEST FOR EXAMINATION WAS MADE

17P Request for examination filed

Effective date: 20200727

AK Designated contracting states

Kind code of ref document: A1

Designated state(s): AL AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HR HU IE IS IT LI LT LU LV MC MK MT NL NO PL PT RO RS SE SI SK SM TR

AX Request for extension of the european patent

Extension state: BA ME

DAV Request for validation of the european patent (deleted)
DAX Request for extension of the european patent (deleted)
A4 Supplementary search report drawn up and despatched

Effective date: 20211126

RIC1 Information provided on ipc code assigned before grant

Ipc: G06N 3/04 20060101ALI20211122BHEP

Ipc: G06K 9/32 20060101ALI20211122BHEP

Ipc: G06K 9/46 20060101ALI20211122BHEP

Ipc: G06T 7/292 20170101ALI20211122BHEP

Ipc: G06T 7/11 20170101ALI20211122BHEP

Ipc: G06N 3/08 20060101ALI20211122BHEP

Ipc: G06K 9/62 20060101ALI20211122BHEP

Ipc: G06K 9/00 20060101ALI20211122BHEP

Ipc: G06K 9/20 20060101AFI20211122BHEP

P01 Opt-out of the competence of the unified patent court (upc) registered

Effective date: 20230525

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: EXAMINATION IS IN PROGRESS

17Q First examination report despatched

Effective date: 20240313