WO2020077525A1 - Systems and methods for model for incremental and adaptive object recognition using hierarchical representations - Google Patents
Systems and methods for model for incremental and adaptive object recognition using hierarchical representations Download PDFInfo
- Publication number
- WO2020077525A1 WO2020077525A1 PCT/CN2018/110403 CN2018110403W WO2020077525A1 WO 2020077525 A1 WO2020077525 A1 WO 2020077525A1 CN 2018110403 W CN2018110403 W CN 2018110403W WO 2020077525 A1 WO2020077525 A1 WO 2020077525A1
- Authority
- WO
- WIPO (PCT)
- Prior art keywords
- model
- glimpses
- glimpse
- image
- processors
- Prior art date
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
- G06V10/82—Arrangements for image or video recognition or understanding using pattern recognition or machine learning using neural networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/21—Design or setup of recognition systems or techniques; Extraction of features in feature space; Blind source separation
- G06F18/217—Validation; Performance evaluation; Active pattern learning techniques
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/24—Classification techniques
- G06F18/241—Classification techniques relating to the classification model, e.g. parametric or non-parametric approaches
- G06F18/2413—Classification techniques relating to the classification model, e.g. parametric or non-parametric approaches based on distances to training or reference patterns
- G06F18/24133—Distances to prototypes
- G06F18/24137—Distances to cluster centroïds
- G06F18/2414—Smoothing the distance, e.g. radial basis function networks [RBFN]
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V30/00—Character recognition; Recognising digital ink; Document-oriented image-based pattern recognition
- G06V30/10—Character recognition
- G06V30/18—Extraction of features or characteristics of the image
- G06V30/1801—Detecting partial patterns, e.g. edges or contours, or configurations, e.g. loops, corners, strokes or intersections
- G06V30/18019—Detecting partial patterns, e.g. edges or contours, or configurations, e.g. loops, corners, strokes or intersections by matching or filtering
- G06V30/18038—Biologically-inspired filters, e.g. difference of Gaussians [DoG], Gabor filters
- G06V30/18048—Biologically-inspired filters, e.g. difference of Gaussians [DoG], Gabor filters with interaction between the responses of different filters, e.g. cortical complex cells
- G06V30/18057—Integrating the filters into a hierarchical structure, e.g. convolutional neural networks [CNN]
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V30/00—Character recognition; Recognising digital ink; Document-oriented image-based pattern recognition
- G06V30/10—Character recognition
- G06V30/18—Extraction of features or characteristics of the image
- G06V30/18086—Extraction of features or characteristics of the image by performing operations within image blocks or by using histograms
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V30/00—Character recognition; Recognising digital ink; Document-oriented image-based pattern recognition
- G06V30/10—Character recognition
- G06V30/18—Extraction of features or characteristics of the image
- G06V30/18162—Extraction of features or characteristics of the image related to a structural representation of the pattern
- G06V30/18181—Graphical representation, e.g. directed attributed graph
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V30/00—Character recognition; Recognising digital ink; Document-oriented image-based pattern recognition
- G06V30/10—Character recognition
- G06V30/19—Recognition using electronic means
- G06V30/192—Recognition using electronic means using simultaneous comparisons or correlations of the image signals with a plurality of references
- G06V30/194—References adjustable by an adaptive method, e.g. learning
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V30/00—Character recognition; Recognising digital ink; Document-oriented image-based pattern recognition
- G06V30/10—Character recognition
Definitions
- the present disclosure relates generally to the field of computer vision. More particularly, the present disclosure relates to systems and methods for a model for incremental and adaptive object recognition using hierarchical representation.
- Computer vision can implement convolutional neural networks ( “CNNs” ) and related solutions, including architectures based CNNs and using the availability of large amount of data and high performing GPU resources.
- CNNs convolutional neural networks
- Existing networks typically go deep, with neurons at higher layers extracting more complex features of a larger receptive fields. However, as such networks go deeper, they may lose finer details and errors become increasingly harder to propagate to earlier layers.
- the present disclosure relates to systems and methods that can execute an adaptive object recognition model inspired by the foveation mechanism of human visual cortex.
- the model can tightly integrate the resolution of “where” and “what” in a series of glimpses, and output incrementally better predictions.
- the model can adapt to data complexity and trade-off resource consumption and performance on-demand.
- Systems and methods as disclosed herein can explore the graphical hierarchy of visual features, deriving a clean and effective architecture that is fast, efficient and robust. In some embodiments, glimpses at different levels of the feature hierarchy are processed with convolutional feature extractors with the same capacity but do not share parameters. As such, systems and methods as disclosed herein can attend to statistics of different granularity, but their limited capacities act as an information bottleneck, leading to automatic discovery of structures.
- an object recognition system includes one or more processors and a memory storing computer-readable instructions that cause the one or more processors to provide, to an image recognition model, an input image; generate, by the image recognition model, a plurality of patches of the image using a plurality of corresponding glimpses of the image; extract, from each patch, a plurality of features of the patch; and predict, by the image recognition model, an object represented by the input image based on a sequence of the glimpses and each corresponding feature of each glimpse.
- FIG. 1 is a block diagram of an object recognition system, according to an embodiment of the present disclosure.
- FIGS. 2A-2B are schematic diagrams of computation of a binary tree including per-node computations (FIG. 2A) and an overall tree structure and prediction (FIG. 2B) , according to an embodiment of the present disclosure.
- FIG. 3 illustrates charts of training a model for incremental and adaptive object recognition using a single digit with structured background noise dataset, according to an embodiment of the present disclosure.
- FIG. 4 illustrates training curves of the training ofFIG. 3.
- FIG. 5 illustrates charts of training a model for incremental and adaptive object recognition using a MNIST-Lego dataset, according to an embodiment of the present disclosure.
- FIG. 6 illustrates training curves of the training ofFIG. 5.
- FIGS. 7A-7B illustrates examples of bounding box visualizes of testing a model for incremental and adaptive object recognition using a CIFAR-10 dataset.
- FIGS. 8A-8C illustrates examples of testing a model for incremental and adaptive object recognition using a CUB-200-2011 dataset.
- the present solution can improve upon existing systems, including existing multi-glimpse systems, by fully exploring the graphical hierarchy of visual features, deriving a clean and effective architecture that is also fast in speed, and can automatically discover structures of the data by a constant capacity feature extractor that acts as an information bottleneck.
- Existing systems may be operate in a manner contrary to everyday experience.
- the human visual cortex employs a foveation mechanism, takes a series of glimpses, engages the resolution of “where” and “what” simultaneously, and is robust against adversarial examples.
- the serial nature of the process does not necessarily mean that it is slower: processing 10 glimpses each with a 10-layer CNN is equivalent to a one-pass processing through a 100-layer CNN.
- a glimpse restricts feature extraction at local region of the input canvas, possibly down-sampled for a coarse representation if a larger field is required, it can be substantially cheaper.
- Some existing systems attempt to add skip connections to address performance issues; however, such attempts may not allow for networks to go deeper.
- the present solution performs object recognition by learning a parameterized model p ⁇ (y
- H and W are image size in pixels
- ⁇ is the model parameters
- is number of candidate classes.
- the class with the largest probability is taken (e.g., ) if only the most probable class is needed. If describes the difference between ground truth y * and class prediction then the learning objective of the model can be to find the model parameters ⁇ that minimize the expected loss
- existing models may lack adaptivity, efficiency, and robustness.
- existing models may lack adaptivity due to having a fixed cost rather than adapting its computation cost against complexity of the data and/or resolution of the image.
- Efficiency may be related to adaptivity; some context information is useful in identifying the object, but computation over regions of unrelated background context can be wasteful.
- robustness an assumption of typical systems is that the objects lie in a manifold distribution with an inherently much lower dimension, despite the fact that the input dimension is much higher (e.g., 3-by-1024-by-1024, roughly 3 million) . By consuming all pixels, the model approximates an manifold distribution that is much more brittle than focusing only on the most information-carrying regions.
- the present solution can improve over such existing models by incorporating design principles based on how the human visual cortex works.
- the entire human vision system can be adaptive in that a series of glimpses, the receptive fields of which are limited, can be induced from the input.
- Each glimpse can have a fixed computational cost, and can choose to either extract features of a down-sampled and large region to build context, or zoom in to focus on smaller regions for finer details.
- the present solution can generate an understanding of an image, such as by recognition what features are located where, by inducing a dynamic graph of glimpses.
- the present solution can be bio-inspired but does not constrain itself to be bio-plausible.
- the present solution can decompose each glimpse into finding where to look next and what to extract.
- the present solution can leverage existing models by taking their feature extraction pipelines as off-the-shelf components that we can freely plug-in.
- the present solution does not require location labels which are expensive to obtain.
- the present solution can require only images of single object with correct label only.
- the present solution can identify multiple objects essentially by layering another graph atop, with branches that focus on one object only.
- an object recognition system 100 in accordance with the present disclosure can include a processing circuit.
- the processing circuit 104 can be used to execute various functions described herein, including training and operating models for incremental and adaptive object recognition using hierarchical representations.
- the processing circuit 104 includes a processor 108 and memory 112.
- the processor 108 may be implemented as a specific purpose processor, an application specific integrated circuit ( “ASIC” ) , one or more field programmable gate arrays ( “FPGAs” ) , a group of processing components, or other suitable electronic processing components.
- the memory 112 is one or more devices (e.g., RAM, ROM, flash memory, hard disk storage) for storing data and computer code for completing and facilitating the various user or client processes, layers, and modules described in the present disclosure.
- the memory 112 may be or include volatile memory or non-volatile memory and may include database components, object code components, script components, or any other type of information structure for supporting the various activities and information structures of the inventive concepts disclosed herein.
- the memory 112 is communicably connected to the processor 108 and includes computer code or instruction modules for executing one or more processes described herein.
- the memory 112 includes various circuits, software engines, and/or modules that cause the processor 108 to execute the systems and methods described herein.
- the object recognition system 100 includes a model 116 and a machine learning engine 120.
- the model 116 can be a model for incremental and adaptive object recognition using hierarchical representations.
- the machine learning engine 120 can train the model 116, such as by providing inputs of training data to the model 116, executing the model 116 using the inputs, comparing an output of the model 116 to outputs of the training data, and modifying the model 116 based on the comparison.
- the model 116 can execute a prediction process as a sequential decision process.
- the model 116 can observe a patch of the image with the glimpse g t , which is a small set of dynamically generated parameters (e.g., four in the case of a bounding box) .
- a feature extractor e.g., a tower of CNN
- the feature extractor may have different parameters at different time step, but has a constant capacity irrespective of the glimpse parameters. Therefore, a large patch may need to be pooled (e.g., downsized) before feeding into the extractor.
- the sequence of the glimpses and the associated features allows the model to make a prediction sequence.
- the frequencies and boundaries of the predictions of the model 116 can be freely determined.
- the model 116 can have dependency assumptions for glimpse parameters generation; the next glimpse parametersg i+1 may only condition on a subset of the historical glimpses and glimpse parameters.
- Such dependencies can be drawn as a directed graph where each node u corresponds to a glimpse. This can enable the model 116 to induce hierarchical representations, each processed with a “bottleneck” feature extractor. can be a tree.
- the tree is K-ary with depth L.
- a glimpse of tree may be a natural representation of an object, where deeper nodes focusing increasingly on more local regions and finer details, other topologies are possible.
- K and L can be set as hyper-parameters, or can be random variables that are stochastically determined.
- a top-down model can be considered, and all K children generated at once from a given node (unless it is a leaf node) .
- Bottom-up or sequential generation of the nodes can be used.
- a parent node needs to decide the locations of its children’s glimpses.
- This decision can be stochastic (e.g., sampled from a predicted parameterized distribution) , or deterministic. The latter can be considered as a stochastic decision with zero variance. While deterministic decisions allows a fully-differentiable model to be trained with stochastic gradient descent, stochastic models can actively explore different strategies and potentially utilize non-differentiable policies. The model in this paper uses deterministic policy.
- levelwise-prediction a new prediction is made only when a level of the tree has grown.
- FIGS. 2A-2B computation of a full binary tree with depth three is shown.
- FIG. 2A shows how the node representations and the glimpse parameters of its children are computed, given its own glimpse parameters and the image x.
- FIG. 2B shows how levelwise-prediction is computed by averaging the node representations of all levels generated so far.
- the model 116 can incorporate features analogous to the cognitive process of the human visual system.
- the model 116 can induce a dynamic tree of glimpses from a still image; the root has a glimpse over the entire canvas with coarse resolution, whereas later glimpses behave the opposite. Every patch can be pooled to feed into a CNN tower of a fixed receptive field and constant capacity. By doing so, top-level patches can examine the larger contexts, and lower-level patches attend to details, which can enable the “where” and “what” pathways to be integrated.
- each node first computes its glimpse and then extracts the associated features.
- the present solution executes a different decomposition.
- a node u takes two inputs, the image x and the glimpse g u to look at. Its task is three-folds.
- the node extracts the visual features of the patch, by pooling the patch and feeding into a CNN tower with a n ⁇ n receptive size.
- the next two actions can happen in parallel: computing the glimpses of the children, unless the node is itself a leaf node, and forming the hidden representation h as another output of the node.
- Per node computational flow and a cartoon example of a two-level tree are shown as discussed above in FIGS. 2A-2B.
- a glimpse ⁇ (g u , x) is a function which takes in a set of glimpse parameters g u , as well as the original imagex, and returns a patch of the image x u within the reception field parameterized by g u ; the patch has a constant size n ⁇ n.
- a glimpse may be implemented in various ways. In some embodiments, a glimpse is implemented by adopting a bank of Gaussian kernels applied over the image into a fixed-size output patch.
- the model 116 can generate two filterbanks and on x-and y-axis respectively.
- the filterbanks can be normalied so that the rows sum up to one, and the filters are applied with matrix multiplication.
- the visual feature of the patch is obtained from the function such as by using a CNN as discussed further below.
- the final hidden representation h u of node u can then be a function of the glimpse parameters g u , and the parent node representation h Par (u) , computed with the function f H ( ⁇ ) .
- Various such functions are discussed below.
- the glimpse parameters for children can be a function of g u , h u .
- the glimpse parameter prediction can be treated as sequential modeling, such that the glimpse parameters of a specific child can depend on the glimpse parameters of all previously computed children
- f. are functions parameterized by a (learnable) neural network.
- the readout stage can be executed, which can take in all the hidden representations into account and produces the final task-specific output.
- Various CNNs can be used as a feature extractor. As compared to existing systems, which may employ a global average pooling layer which reduces any feature map size to 1x1 (which may cause loss of spatial information) , the present solution can replace the last layer with an adaptive average pooling to a predefined size (e.g., 2x2) . In addition to the option of having a single CNN producing a set of feature maps, the present solution can use two separate CNNs and obtain two feature map sets having different responsibilities
- the first set may be used.
- the second set and optionally the first set can take part in the readout stage for downstream tasks.
- Another option is to induce from using a simple transformation, by normalizing each channel:
- the rational being that contains enough information, but needs to normalize actions across features of different images in order to make the parameterization stable. We found this to be effective on real-world images.
- the present solution can address various factors to select the hidden representation extractor, including the form of h u , and how dependent it is with its parent hidden representation h Par (u) .
- h u needs to integrate the image patch feature the glimpse parameters g u , and optionally the hidden representation of parent h Par (u) .
- h u can be vector-based. For example, can be flattened, into a 1-dimensional vector, concatenated with g u , and projected non-linearly into h u .
- non-linear projections of and g u can be added before the concatenation. As such, the model 116 to learn the spatial association of the glimpsed feature and the location.
- g u is a very small set of parameters and the model 116 may have difficulty mapping it to a proper association with
- the Gaussian kernels can be reversed and mapped to a 2-D zero-initialized matrix where bits corresponding to g u are turned on.
- h u can be feature map-based.
- the features can be explicitly embedded spatially, such as starting with an empty canvas in feature-map space (e.g., a 3D tensor of zeros) , finding the relative location of the patch using g u and setting them with
- feature-map space e.g., a 3D tensor of zeros
- embodiments can allow the hidden representations from different nodes to be integrated similar to alpha-channel composite in photo editing.
- the dependencies of h u with its parent can be either weak or strong.
- h Par (u) is essentially ignored, and the only dependency to the parent is the glimpse parameters.
- the parents can influence the children more strongly, such as by when computing h u , including h Par (u) in the nonlinear projection, which can either be an MLP, or a recurrent network such as GRU or LSTM taking h u as recurrent state.
- the predictor could be an MLP that generates all simultaneously.
- the predictor could be a recurrent network that generates one by one.
- h i RNNCell ( ⁇ g′ i-1 , h i-1 )
- level-wise image classification is performed.
- An additional benefit is at test time, the model can stop predicting when the prediction rises above certain confidence level, thereby exhibiting adaptive behaviour.
- the model 116 can use use levelwise-prediction, predicting once per tree level. In total, the same number of predictions as the tree depth can be made.
- the summary function includes an average of all h u :
- the summary function includes an average of all h u :
- the attention weight can be learned.
- the contribution of a glimpse can be weighed according to how detailed it is (e.g., ⁇ i ⁇ Area (g i ) -1 ) .
- the summary is computed by first computing a weight for the nodes in each level. Then we compute a weighted average of hidden representations for each level, and perform a max-pooling across the levels.
- all hidden representations can have set their features at the corresponding positions on a feature-space canvas to be painted together.
- For each can be overlaid on top of the canvas at the location dictated by g u sequentially:
- the process includes as alpha compositing, such as to compose a photo with segments of different toplogies by manipulating a sequence of opacity maps.
- model 116 builds more nodes and levels, it can accumulate more features with finer details. As such, the model 116 can spend more computational resources to extract finer details progressively.
- the behavior of diminishing returns is by design, and the underpinning of the adaptive nature of the model 116.
- the decomposition into tree nodes and the isomorphic sub-tasks within a node yields a variety of ways to construct model parameters from are sharing across all nodes or not to share at all. It has been found that it may be crucial to have different CNN parameters at different levels, as they can deal with features at different scale and resolutions. In some embodiments, these parameters can be simple transformation from a base set.
- Equations (1) - (4) becomes the following for node u at level l:
- the same f pred can be used for all levelwise predictions.
- a prediction loss can ensure all predictions are relevant:
- a sibling glimpse overlap penalty can encourage children glimpses to look at different regions:
- a parent-children containment penalty can encourage children to stay within the parent’s region so as to induce the “zoom-in” effect:
- a resolution penalty can force the glimpses on the last level to have the resolution as close to the original image as possible.
- the L2 weight decay and also optionally dropout in f pred can be used.
- the model 116 can be trained by minimizing where D * is the training set.
- the loss is end-to-end differentiable, and the model 116 can be trained with gradient descent. Note that the actual coefficients depend on the model variant being used.
- the entire model can be trained end-to-end. However, the flow of the gradients can be stopped for various embodiments, including to implement monotonic prediction improvement and a single source of influence on glimpse. With respect to monotonic prediction improvement, the model 116 can make forward progress at any prediction boundary, in a greedy fashion. A parent prediction should not sacrifice itself so that its children glimpses make better predictions. This is important since at test time the model might quit at any level (e.g., for energy saving) . Thus, to emulate this behavior, when the summary function is computed at leveli, error gradient may not flow back to earlier levels.
- h u can encode both g u and and is dependent on g u .
- the gradient can be stopped from flowing through h u to g u , thus forcing g u to optimize for the purpose of getting the maximum information.
- “what” can precede “where. ”
- a given CNN would take C FLOPS to compute over an M ⁇ M image (considering only a single channel (e.g., greyscale) ) .
- M the number of nodes
- M the ratio between image size and glimpse size.
- residual networks often operate on 224 ⁇ 224 images on ImageNet dataset, so a glimpse size of 50 x 50 and a binary tree with depth-3 would take of the original computation over the entire image considering the CNN alone.
- the analysis above did not count the FLOPS of glimpses.
- the glimpse computation can include matrix multiplication among a filter bank with size M g ⁇ M, the image sized M ⁇ M, and another filter bank M ⁇ M g . This can take about FLOPS counting both multiplications and additions, which can usually be significantly less than C.
- the glimpse computation can include Gaussian filter bank generation, which can include elementwise computations of M g ⁇ M matrices, which may also be much less than C. For numerical stability, a small constant ⁇ can be added on both the numerator and denominator for the two fractions in our implementation. The same goes for the Parent-children containment penalty below.
- the number of FLOPS to execute the model 116 should be much less than that from the CNN over the entire image.
- the present solution can learn from the human visual cortex and adopt its design principles instead of implementations.
- the present solution can mimic how the human fovea system works by looking at the most information-carrying regions progressively, deriving a hierarchy of glimpses with increasingly finer details, and process them with a constant cost. It can adapt to data complexity and trade-off resource consumption and performance on-demand.
- the present solution can be complementary to all existing CNN models in the sense that they can be used as an off-the-shelf plug-in module to extract features.
- the model 116 may not be deep, but rather adjust to scales and resolutions adaptively and dynamically, by zooming in (or out) at different nodes. As such, the model 116 can be decoupled with input image’s size and resolution.
- the model 116 can predict not only the class but also a progressively refined location map of the object. As such, the model 1166 can inherently integrate “what” and “where. ”
- the model 116 can be adaptive, making predictions with increasingly higher accuracy. This trade-off can be important when the model is under time or resource pressure, as it can quit earlier or extend the search of new evidences as needed.
- the model 116 can operate such that per-step visual feature extraction is constant. This allows the system 100 to control both computational resources and memory footprint. In some embodiments, this improvement of efficiency can be analogous to imposing sparsity at a higher granularity (e.g., regions of input) , rather than doing so at neuron or weight level. In some embodiments, the constant capacity of the feature extractor can force the discovery of structures within the data. By progressively focusing on the most information-carrying regions, the model 116 can ignore irrelevant background noises and therefore be less sensitive to adversarial attacks. The model 116 can rely on images containing one object, without requiring bounding box labels which are both error-prone and expensive to acquire, and can therefore be easier to improve.
- the model 116 can be similar to neuroscience findings.
- the model 116 can be analogous to a visual cortex that is not deep, but instead has abundant top-to-bottom and lateral connections. During the process, steps to resolve “where” and “what” are tightly intertwined, instead of separate.
- the model 116 can fully explore the hierarchical and graphical view of feature extractions. Instead of relying on a serial recurrent network to compute subsequent glimpses, the model 116 can explicitly model the parent-to-children relationship in a top-down, coarse-to-finer manner. Predictions can be made by reading out the summary from a subset of graph nodes. Organizing glimpses into an hierarchy can significantly improve speed: for N glimpses the processing time is O (log N) instead of O (N) . Working out a hierarchy of features with an extractor with constant capacity also can lead to discovery of structures.
- each level adopts a different set of parameters. This can enable each feature extractor, while sharing the same architecture, to focus on capturing different scales.
- the present solution can integrate predicting glimpses of different sizes and shapes, applying Gaussian kernels to smoothly pool into a fixed input for feature extraction, and using deterministic actions instead of stochastic ones.
- An embodiment of the model 116 was evaluated with MNIST-cluttered dataset where a digit is placed at a random location of a n ⁇ n canvas, spread with k random 8 ⁇ 8 subpatches from other random MNIST digits.
- the sizes of training, valid and test set are set to 50000, 10000 and 10000 respectively.
- the CNN the model 116 is a simple 5-layer with 12-by-12 input size.
- the model 116 can also give levelwise predictions.
- the model 116 can outperform both RAM and DRAM, even with fewer number of glimpses. Furthermore, levelwise predictions of the model 116 can have progressively better accuracies.
- FIG. 3 presents several visualizations of 100 ⁇ 100 cases, with 2-level (1 branch per node, 3 glimpses in total) and 3-level (2 branches per node, 7 glimpses in total) on the left half and right half, receptively.
- Level-2 and level-3 glimpses are colored with yellow and red boxes.
- Each sample is paired with an image reconstructed by overlaying appropriate up-sampling contents in the glimpses; this image thus shows the accumulated result of all the glimpses.
- the glimpses focus on the object while ignoring the background noises.
- 3 shows original image and glimpse boxes (3 levels, 3 glimpses) in the first column, glimpse content (3 levels, 3 glimpses) in the second column, original image and glimpse boxes (3 levels, 7 glimpses) in the third column, and glimpse content (3 levels, 7 glimpses) in the fourth column.
- the training curve of the 3-level tree model with 3 glimpses on 100 ⁇ 100 Cluttered Translated MNIST dataset is shown in FIG. 4 (level 1: 62.99; level 2: 95.85; level 3: 97.07) . As training progresses, predictions from all three levels can rise in tandem.
- a 3-level full binary tree structure with glimpse size 12 ⁇ 12 was picked, and the model 116 trained on two datasets with different background noise types described above.
- the model 116 was optimized with RMSProp optimizer and the learning rate set to 10 -4 . Early stopping was adopted to avoid over-fitting.
- the model 116 achieved an accuracy of 96.24%and 96.20%, with cluttered and Gaussian noise, respectively.
- FIG. 6 Training curves are shown in FIG. 6 (level 1: 46.04; level 2: 94.23; level 3: 96.20) .
- FIG. 5 presents several visualization cases on the MNIST Lego dataset (Left: cluttered noise, Right: Gaussian noise) .
- substructures were shown to be automatically discovered.
- more levels may be needed to localize parts (in this case each digit is one part of the overall class) , even though the end performances do not differ much. As such, focusing on prediction alone may not tell the whole story.
- Table 2 Classification result on CUB-200-2011 dataset.
- the CNN baselines resize the input image to a fixed size (e.g., 200x200) , whereas our model does so adaptively and dynamically, the 100x100 model thus takes roughly half of the computational cost to achieve the same level of performance.
- the model 116 was trained on the CIFAR-10 dataset.
- the dataset consists of 60,000 32 ⁇ 32 RGB images, 50,000 for training and 10,000 for testing.
- the object takes up a large portion of the image, and the glimpse size is large compared to that of the images, such that requiring the children to cover different image parts may not be effective.
- f ⁇ was initialized using a pretrained ResNet-18.92%accuracy was achieved on the test set. Some of the examples are visualized in FIGS. 7A-7B.
- FIGS. 7A-7B Some of the examples are visualized in FIGS. 7A-7B.
- FIG. 7A-7B show glimpse bounding box visualizations (FIG. 7A) in which the two yellow boxes correspond to the glimpses on the second level, while the four red ones are for the glimpses on the third (last) level;
- FIG. 7B shows glimpse content visualizations on the same image; this figure is obtained by applying the generated Gaussian filter banks as masks on the original image.
- FIGS. 8A-C The model 116 was tested on fine-grained classification task as shown in FIGS. 8A-C using the CUB-200-2011 dataset.
- FIG. 8A shows glimpse bounding box of structure 1-1
- FIG. 8B shows glimpse bounding box of structure 1-2
- FIG. 8C shows glimpse bounding box of structure 1-1-1.
- This figure was obtained by applying the generated Gaussian filter banks as masks on the original image.
- the dataset consists of 11, 788 images of birds with 200 types. We resized all images to 500x500 for both training and evaluating.
- the f ⁇ module in our model was initialized with pretrained ResNet-50, is channel-se normalized from We again selected vector-based with levelwise attention and max-pooling as our readout module, and used sequential model as the choice of predictor for next glimpses.
- the model 116 can incorporate static topology with fine-grained control. For example, in some embodiments, having redundant nodes and levels hurt generalization. Some of the future works include adaptively growing one node at a time during the training procedure by looking at the best validation metric, in similar spirit to, or imposing sparsity constraints on gate activations or attention masks.
- the model 116 can incorporate dynamic topology.
- the branch factor K and the depth of the tree L are both hyper-parameters.
- a flexible model shall zoom-in where needed, while stop growing when statistics of the current glimpse is already sufficient. However, such stochastic decisions can make the model hard to learn.
- the model 116 can have metrics established such that an overgrown tree can be truncated at test time without much loss of performance.
- the model 116 can incorporate dynamic CNN parameters. Sharing feature extractor parameters within a level may not be very principled, since there is no guarantee that glimpses at a given level are inspecting the same level of details. This assumption may not be true across images, and parameters may be refined on-demand.
- the model 116 can incorporate multiple iterations.
- the model 116 may offer a plausible explanation to why there are abundant top-down connection in our visual cortex. At least partially, it is a feedback signal to direct next focus. Lateral connection can be explained by introducing multiple iterations over the same graph. This way, node representation can become increasingly richer. This may be implemented using Message-Passing Neural Network ( “MPNN” ) .
- MPNN Message-Passing Neural Network
- the model 116 can integrate with multi-object detection.
- the tree of the model 116 is for one object.
- the model 116 can impose one more level on top.
- the model 116 can leverage the localization part of other models, such as YOLO, to propose the top-level branches.
- the model 116 can predict a multiset to learn from scratch.
- the model 116 can be extended to image sequences.
- the multiple-glimpses induced by the model 116 can map out the most salient points about an object. As such, the locations and features can be more informative than a single bounding box, as many the video tracking systems do.
- a Gaussian glimpse function can have the following form:
- the corresponding inverse Gaussian glimpse function can be defined as
- the transpose of the Gaussian filter banks can be taken, normalized by column, and the resulting filters applied on the glimpse x u to recover to the size of original canvas.
- Alpha-channel overlaying can be performed as follows:
- the additional channel of 1’s can be referred to as an alpha channel.
- the overlay is obtained by
- Coupled means the joining of two members directly or indirectly to one another. Such joining may be stationary (e.g., permanent or fixed) or moveable (e.g., removable or releasable) . Such joining may be achieved with the two members coupled directly to each other, with the two members coupled to each other using a separate intervening member and any additional intermediate members coupled with one another, or with the two members coupled to each other using an intervening member that is integrally formed as a single unitary body with one of the two members. Such members may be coupled mechanically, electrically, and/or fluidly.
- DSP digital signal processor
- a general purpose processor may be a microprocessor, or, any conventional processor, controller, microcontroller, or state machine.
- a processor also may be implemented as a combination of computing devices, such as a combination of a DSP and a microprocessor, a plurality of microprocessors, one or more microprocessors in conjunction with a DSP core, or any other such configuration.
- particular processes and methods may be performed by circuitry that is specific to a given function.
- the memory e.g., memory, memory unit, storage device, etc.
- the memory may include one or more devices (e.g., RAM, ROM, Flash memory, hard disk storage, etc. ) for storing data and/or computer code for completing or facilitating the various processes, layers and modules described in the present disclosure.
- the memory may be or include volatile memory or non-volatile memory, and may include database components, object code components, script components, or any other type of information structure for supporting the various activities and information structures described in the present disclosure.
- the memory is communicably connected to the processor via a processing circuit and includes computer code for executing (e.g., by the processing circuit and/or the processor ) the one or more processes described herein.
- the present disclosure contemplates methods, systems and program products on any machine-readable media for accomplishing various operations.
- the embodiments of the present disclosure may be implemented using existing computer processors, or by a special purpose computer processor for an appropriate system, incorporated for this or another purpose, or by a hardwired system.
- Embodiments within the scope of the present disclosure include program products comprising machine-readable media for carrying or having machine-executable instructions or data structures stored thereon.
- Such machine-readable media can be any available media that can be accessed by a general purpose or special purpose computer or other machine with a processor.
- machine-readable media can comprise RAM, ROM, EPROM, EEPROM, or other optical disk storage, magnetic disk storage or other magnetic storage devices, or any other medium which can be used to carry or store desired program code in the form of machine-executable instructions or data structures and which can be accessed by a general purpose or special purpose computer or other machine with a processor. Combinations of the above are also included within the scope of machine-readable media.
- Machine-executable instructions include, for example, instructions and data which cause a general purpose computer, special purpose computer, or special purpose processing machines to perform a certain function or group of functions.
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Multimedia (AREA)
- Evolutionary Computation (AREA)
- Artificial Intelligence (AREA)
- Data Mining & Analysis (AREA)
- General Health & Medical Sciences (AREA)
- Life Sciences & Earth Sciences (AREA)
- Databases & Information Systems (AREA)
- Health & Medical Sciences (AREA)
- Biomedical Technology (AREA)
- Molecular Biology (AREA)
- Computing Systems (AREA)
- Medical Informatics (AREA)
- Software Systems (AREA)
- Biodiversity & Conservation Biology (AREA)
- Bioinformatics & Cheminformatics (AREA)
- Bioinformatics & Computational Biology (AREA)
- Evolutionary Biology (AREA)
- General Engineering & Computer Science (AREA)
- Image Analysis (AREA)
Abstract
Systems and methods are disclosed that can execute an adaptive object recognition model inspired by the foveation mechanism of human visual cortex. The model can tightly integrates the resolution of "where" and "what" in a series of glimpses, and output incrementally better predictions. The model can adapt to data complexity and trade-off resource consumption and performance on-demand. Systems and methods as disclosed herein can explore the graphical hierarchy of visual features, deriving a clean and effective architecture that is fast, efficient and robust. Glimpses at different levels of the feature hierarchy are processed with convolutional feature extractors with the same capacity but do not share parameters. As such, systems and methods as disclosed herein can attend to statistics of different granularity, but their limited capacities act as an information bottleneck, leading to automatic discovery of structures.
Description
The present disclosure relates generally to the field of computer vision. More particularly, the present disclosure relates to systems and methods for a model for incremental and adaptive object recognition using hierarchical representation.
One of the more matured domain applying deep learning technologies is computer vision. Computer vision can implement convolutional neural networks ( “CNNs” ) and related solutions, including architectures based CNNs and using the availability of large amount of data and high performing GPU resources. Existing networks typically go deep, with neurons at higher layers extracting more complex features of a larger receptive fields. However, as such networks go deeper, they may lose finer details and errors become increasingly harder to propagate to earlier layers.
SUMMARY
The present disclosure relates to systems and methods that can execute an adaptive object recognition model inspired by the foveation mechanism of human visual cortex. The model can tightly integrate the resolution of “where” and “what” in a series of glimpses, and output incrementally better predictions. The model can adapt to data complexity and trade-off resource consumption and performance on-demand. Systems and methods as disclosed herein can explore the graphical hierarchy of visual features, deriving a clean and effective architecture that is fast, efficient and robust. In some embodiments, glimpses at different levels of the feature hierarchy are processed with convolutional feature extractors with the same capacity but do not share parameters. As such, systems and methods as disclosed herein can attend to statistics of different granularity, but their limited capacities act as an information bottleneck, leading to automatic discovery of structures.
In some embodiments, an object recognition system includes one or more processors and a memory storing computer-readable instructions that cause the one or more processors to provide, to an image recognition model, an input image; generate, by the image recognition model, a plurality of patches of the image using a plurality of corresponding glimpses of the image; extract, from each patch, a plurality of features of the patch; and predict, by the image recognition model, an object represented by the input image based on a sequence of the glimpses and each corresponding feature of each glimpse.
This summary is illustrative only and is not intended to be in any way limiting. Other aspects, inventive features, and advantages of the devices and/or processes described herein, will become apparent in the detailed description set forth herein, taken in conjunction with the accompanying figures, wherein like reference numerals refer to like elements.
FIG. 1 is a block diagram of an object recognition system, according to an embodiment of the present disclosure.
FIGS. 2A-2B are schematic diagrams of computation of a binary tree including per-node computations (FIG. 2A) and an overall tree structure and prediction (FIG. 2B) , according to an embodiment of the present disclosure.
FIG. 3 illustrates charts of training a model for incremental and adaptive object recognition using a single digit with structured background noise dataset, according to an embodiment of the present disclosure.
FIG. 4 illustrates training curves of the training ofFIG. 3.
FIG. 5 illustrates charts of training a model for incremental and adaptive object recognition using a MNIST-Lego dataset, according to an embodiment of the present disclosure.
FIG. 6 illustrates training curves of the training ofFIG. 5.
FIGS. 7A-7B illustrates examples of bounding box visualizes of testing a model for incremental and adaptive object recognition using a CIFAR-10 dataset.
FIGS. 8A-8C illustrates examples of testing a model for incremental and adaptive object recognition using a CUB-200-2011 dataset.
Before turning to the figures, which illustrate the exemplary embodiments in detail, it should be understood that the present disclosure is not limited to the details or methodology set forth in the description or illustrated in the figures. It should also be understood that the terminology used herein is for the purpose of description only and should not be regarded as limiting.
The present solution can improve upon existing systems, including existing multi-glimpse systems, by fully exploring the graphical hierarchy of visual features, deriving a clean and effective architecture that is also fast in speed, and can automatically discover structures of the data by a constant capacity feature extractor that acts as an information bottleneck.
Existing systems, including existing one-pass models, may be operate in a manner contrary to everyday experience. The human visual cortex employs a foveation mechanism, takes a series of glimpses, engages the resolution of “where” and “what” simultaneously, and is robust against adversarial examples. The serial nature of the process does not necessarily mean that it is slower: processing 10 glimpses each with a 10-layer CNN is equivalent to a one-pass processing through a 100-layer CNN. Furthermore, since a glimpse restricts feature extraction at local region of the input canvas, possibly down-sampled for a coarse representation if a larger field is required, it can be substantially cheaper. Some existing systems attempt to add skip connections to address performance issues; however, such attempts may not allow for networks to go deeper.
The present solution performs object recognition by learning a parameterized model p
θ (y|x) to map an input image
to a probability distribution y~ [0, 1]
|y|. Here, H and W are image size in pixels, θ is the model parameters, and |y| is number of candidate classes. In some embodiments, the class with the largest probability is taken (e.g.,
) if only the most probable class is needed. If
describes the difference between ground truth y
* and class prediction
then the learning objective of the model can be to find the model parametersθ that minimize the expected loss
Existing models typically transform the image through many layers, which typically consist of linear or non-linear functions such as convolution, max-pooling, and/or batch- normalization. Each layer can extract richer features with larger receptive fields as it goes deeper. Finally, the features are mapped into a vector of scores for the candidate classes.
As such, existing models may lack adaptivity, efficiency, and robustness. For example, existing models may lack adaptivity due to having a fixed cost rather than adapting its computation cost against complexity of the data and/or resolution of the image. Efficiency may be related to adaptivity; some context information is useful in identifying the object, but computation over regions of unrelated background context can be wasteful. With respect to robustness, an assumption of typical systems is that the objects lie in a manifold distribution with an inherently much lower dimension, despite the fact that the input dimension is much higher (e.g., 3-by-1024-by-1024, roughly 3 million) . By consuming all pixels, the model approximates an manifold distribution that is much more brittle than focusing only on the most information-carrying regions.
The present solution can improve over such existing models by incorporating design principles based on how the human visual cortex works. The entire human vision system can be adaptive in that a series of glimpses, the receptive fields of which are limited, can be induced from the input. Each glimpse can have a fixed computational cost, and can choose to either extract features of a down-sampled and large region to build context, or zoom in to focus on smaller regions for finer details. By computing at where matters more, the present solution simultaneously more efficient and robust.
The present solution can generate an understanding of an image, such as by recognition what features are located where, by inducing a dynamic graph of glimpses. The present solution can be bio-inspired but does not constrain itself to be bio-plausible. The present solution can decompose each glimpse into finding where to look next and what to extract. The present solution can leverage existing models by taking their feature extraction pipelines as off-the-shelf components that we can freely plug-in.
In some embodiments, the present solution does not require location labels which are expensive to obtain. The present solution can require only images of single object with correct label only. The present solution can identify multiple objects essentially by layering another graph atop, with branches that focus on one object only.
Referring now to FIG. 1, an object recognition system 100 in accordance with the present disclosure can include a processing circuit. The processing circuit 104 can be used to execute various functions described herein, including training and operating models for incremental and adaptive object recognition using hierarchical representations.
The processing circuit 104 includes a processor 108 and memory 112. The processor 108 may be implemented as a specific purpose processor, an application specific integrated circuit ( “ASIC” ) , one or more field programmable gate arrays ( “FPGAs” ) , a group of processing components, or other suitable electronic processing components. The memory 112 is one or more devices (e.g., RAM, ROM, flash memory, hard disk storage) for storing data and computer code for completing and facilitating the various user or client processes, layers, and modules described in the present disclosure. The memory 112 may be or include volatile memory or non-volatile memory and may include database components, object code components, script components, or any other type of information structure for supporting the various activities and information structures of the inventive concepts disclosed herein. The memory 112 is communicably connected to the processor 108 and includes computer code or instruction modules for executing one or more processes described herein. The memory 112 includes various circuits, software engines, and/or modules that cause the processor 108 to execute the systems and methods described herein.
The object recognition system 100 includes a model 116 and a machine learning engine 120. The model 116 can be a model for incremental and adaptive object recognition using hierarchical representations. The machine learning engine 120 can train the model 116, such as by providing inputs of training data to the model 116, executing the model 116 using the inputs, comparing an output of the model 116 to outputs of the training data, and modifying the model 116 based on the comparison.
The model 116 can execute a prediction process as a sequential decision process. At each step, the model 116 can observe a patch of the image
with the glimpse g
t, which is a small set of dynamically generated parameters (e.g., four in the case of a bounding box) . A feature extractor (e.g., a tower of CNN) can extract the associated features
The feature extractor may have different parameters at different time step, but has a constant capacity irrespective of the glimpse parameters. Therefore, a large patch may need to be pooled (e.g., downsized) before feeding into the extractor. The sequence of the glimpses and the associated features
allows the model to make a prediction sequence. The frequencies and boundaries of the predictions of the model 116 can be freely determined.
The model 116 can have dependency assumptions for glimpse parameters generation; the next glimpse parametersg
i+1 may only condition on a subset of the historical glimpses and glimpse parameters. Such dependencies can be drawn as a directed graph
where each node u corresponds to a glimpse. This can enable the model 116 to induce hierarchical representations, each processed with a “bottleneck” feature extractor.
can be a tree.
With respect to topology, in some embodiments, the tree is K-ary with depth L. A glimpse of tree may be a natural representation of an object, where deeper nodes focusing increasingly on more local regions and finer details, other topologies are possible. Likewise, K and L can be set as hyper-parameters, or can be random variables that are stochastically determined.
With respect to traversal and generation order, in some embodiments, a top-down model can be considered, and all K children generated at once from a given node (unless it is a leaf node) . Bottom-up or sequential generation of the nodes can be used.
With respect to deterministic locations, in some embodiments, a parent node needs to decide the locations of its children’s glimpses. This decision can be stochastic (e.g., sampled from a predicted parameterized distribution) , or deterministic. The latter can be considered as a stochastic decision with zero variance. While deterministic decisions allows a fully-differentiable model to be trained with stochastic gradient descent, stochastic models can actively explore different strategies and potentially utilize non-differentiable policies. The model in this paper uses deterministic policy.
With respect to prediction sequence: at any t, all past history of glimpses can be gathered to make a prediction. Among numerous options, levelwise-prediction was chosen: a new prediction is made only when a level of the tree has grown.
Referring to FIGS. 2A-2B, computation of a full binary tree with depth three is shown. FIG. 2A shows how the node representations and the glimpse parameters of its children are computed, given its own glimpse parameters and the image x. FIG. 2B shows how levelwise-prediction is computed by averaging the node representations of all levels generated so far.
The model 116 can incorporate features analogous to the cognitive process of the human visual system. The model 116 can induce a dynamic tree of glimpses from a still image; the root has a glimpse over the entire canvas with coarse resolution, whereas later glimpses behave the opposite. Every patch can be pooled to feed into a CNN tower of a fixed receptive field and constant capacity. By doing so, top-level patches can examine the larger contexts, and lower-level patches attend to details, which can enable the “where” and “what” pathways to be integrated.
To emulate the “where” and “what” behavior, an approach may be that each node first computes its glimpse and then extracts the associated features. However, given that the root node has the default glimpse of the entire canvas already, the present solution executes a different decomposition. A node u takes two inputs, the image x and the glimpse g
u to look at. Its task is three-folds. First, the node extracts the visual features of the patch, by pooling the patch and feeding into a CNN tower with a n×n receptive size. The next two actions can happen in parallel: computing the glimpses
of the children, unless the node is itself a leaf node, and forming the hidden representation h as another output of the node. Per node computational flow and a cartoon example of a two-level tree are shown as discussed above in FIGS. 2A-2B.
In some embodiments, a glimpse Γ (g
u, x) is a function which takes in a set of glimpse parameters g
u, as well as the original imagex, and returns a patch of the image x
u within the reception field parameterized by g
u; the patch has a constant size n×n. A glimpse may be implemented in various ways. In some embodiments, a glimpse is implemented by adopting a bank of Gaussian kernels applied over the image into a fixed-size output patch.
Where
includes six parameters corresponding to the center, the stride and the standard deviation along the x and y dimension, respectively. That is, the model 116 can generate two filterbanks
and
on x-and y-axis respectively. The filterbanks can be normalied so that the rows sum up to one, and the filters are applied with matrix multiplication.
The visual feature of the patch
is obtained from the function
such as by using a CNN as discussed further below.
The final hidden representation h
u of node u can then be a function of
the glimpse parameters g
u, and the parent node representation h
Par (u) , computed with the function f
H (·) . Various such functions are discussed below.
The glimpse parameters for children
can be a function of g
u, h
u. In some embodiments, the glimpse parameter prediction can be treated as sequential modeling, such that the glimpse parameters of a specific child
can depend on the glimpse parameters of all previously computed children
In some embodiments, the node u emits the representations h
u and the glimpse parameters
for all K children v
1, ..., v
K, given the image x and its own glimpse parameters g
u:x
u = Γ (g
u, x) (1)
where f. are functions parameterized by a (learnable) neural network.
After obtaining the hidden representations for all nodes, the readout stage can be executed, which can take in all the hidden representations into account and produces the final task-specific output.
Various CNNs can be used as a feature extractor. As compared to existing systems, which may employ a global average pooling layer which reduces any feature map size to 1x1 (which may cause loss of spatial information) , the present solution can replace the last layer with an adaptive average pooling to a predefined size (e.g., 2x2) . In addition to the option of having a single CNN producing a set of feature maps, the present solution can use two separate CNNs and obtain two feature map sets having different responsibilities
When predicting the children’s glimpse parameters, only the first set
may be used. The second set
and optionally the first set
can take part in the readout stage for downstream tasks.
Another option is to induce
from
using a simple transformation, by normalizing each channel:
The rational being that
contains enough information, but needs to normalize actions across features of different images in order to make the parameterization stable. We found this to be effective on real-world images.
Choices of hidden representation extractor f
H
The present solution can address various factors to select the hidden representation extractor, including the form of h
u, and how dependent it is with its parent hidden representation h
Par (u) .
In some embodiments, h
u needs to integrate the image patch feature
the glimpse parameters g
u, and optionally the hidden representation of parent h
Par (u) . In some embodiments, h
u can be vector-based. For example,
can be flattened, into a 1-dimensional vector, concatenated with g
u, and projected non-linearly into h
u. In some embodiments, non-linear projections of
and g
u can be added before the concatenation. As such, the model 116 to learn the spatial association of the glimpsed feature and the location. In some embodiments, g
u is a very small set of parameters and the model 116 may have difficulty mapping it to a proper association with
In some embodiments, the Gaussian kernels can be reversed and mapped to a 2-D zero-initialized matrix where bits corresponding to g
u are turned on.
In some embodiments, h
u can be feature map-based. For example, the features can be explicitly embedded spatially, such as starting with an empty canvas in feature-map space (e.g., a 3D tensor of zeros) , finding the relative location of the patch using g
u and setting them with
In addition to removing the need of learning the spatial association, such embodiments can allow the hidden representations from different nodes to be integrated similar to alpha-channel composite in photo editing.
In various embodiments, the dependencies of h
u with its parent can be either weak or strong. In an isolated approach, h
Par (u) is essentially ignored, and the only dependency to the parent is the glimpse parameters. In a message-passing approach, the parents can influence the children more strongly, such as by when computing h
u, including h
Par (u) in the nonlinear projection, which can either be an MLP, or a recurrent network such as GRU or LSTM taking h
u as recurrent state.
h
0 = MLP (h
u)
g′
0 = (0, 0, 0, 0, 0, 0)
h
i = RNNCell (Δg′
i-1, h
i-1)
g′
i = MLP (h
i)
Each
is expressed by a relative translation and scaling to the current glimpse parameters g
u (taking MLP as an example) :
Choices of readout stage
In some embodiments, level-wise image classification is performed. The sequence h
<=t = (h
1, ..., h
t) can include visual features of various patches up to time t. To guarantee that progresses will be made, some or all of the h
<=t, t∈ [1, N] can drive predictions and subject them to classification losses. An additional benefit is at test time, the model can stop predicting when the prediction rises above certain confidence level, thereby exhibiting adaptive behaviour. The model 116 can use use levelwise-prediction, predicting once per tree level. In total, the same number of predictions as the tree depth can be made.
y
(l) = f
pred (Summary (h
1, ..., h
m) )
Various summary functions can be used depending on whether the representations are vector-based or feature-map-based. In vector-based embodiments, the summary function includes an average of all h
u:
In vector-based with global attention embodiments, the summary function includes an average of all h
u:
The attention weight can be learned. In some embodiments, the contribution of a glimpse can be weighed according to how detailed it is (e.g., α
i ∝ Area (g
i)
-1) . A normalization constraint can be imposed on the attention weight so that ∑
iα
i = 1.
In vector-based with levelwise attention and max-pooling, the summary is computed by first computing a weight for the nodes in each level. Then we compute a weighted average of hidden representations for each level, and perform a max-pooling across the levels.
Mathematically, we define a scalar γ
v for each node v as follows:
· The root of the tree always has a γ
root = 1.
· For all children of u, γ
v = γ
usoftmax (MLP (h
v) ) .
In feature-map-based embodiments, all hidden representations can have set their features at the corresponding positions on a feature-space canvas to be painted together. Thus, for each
can be overlaid on top of the canvas at the location dictated by g
u sequentially:
η
0 = 0
Summary (h
1, ..., h
m) = η
m
The mathematical formulation of function Overlay (·, ·, ·) is described further below. In some embodiments, the process includes as alpha compositing, such as to compose a photo with segments of different toplogies by manipulating a sequence of opacity maps.
As the model 116 builds more nodes and levels, it can accumulate more features with finer details. As such, the model 116 can spend more computational resources to extract finer details progressively. The behavior of diminishing returns is by design, and the underpinning of the adaptive nature of the model 116.
Parameter sharing
The decomposition into tree nodes and the isomorphic sub-tasks within a node yields a variety of ways to construct model parameters from are sharing across all nodes or not to share at all. It has been found that it may be crucial to have different CNN parameters at different levels, as they can deal with features at different scale and resolutions. In some embodiments, these parameters can be simple transformation from a base set.
Therefore, the model 116 can share the same
at the same level, while having different f’s across levels, meaning that Equations (1) - (4) becomes the following for node u at level l:
x
u = Γ (g
u, x) (5)
The same f
pred can be used for all levelwise predictions.
Learning
Loss components
Aside of the cross-entropy loss for prediction error, a number of regularizations can be used to restrict model 116 behavior, as follows:
A prediction loss can ensure all predictions are relevant:
A sibling glimpse overlap penalty can encourage children glimpses to look at different regions:
A parent-children containment penalty can encourage children to stay within the parent’s region so as to induce the “zoom-in” effect:
A resolution penalty can force the glimpses on the last level to have the resolution as close to the original image as possible.
can be used to regularize the parameters of the entire model; the L2 weight decay and also optionally dropout in f
pred can be used.
Each of these regularization has its own coefficient, leading to the final loss of the following form:
The model 116 can be trained by minimizing
where D
*is the training set. The loss is end-to-end differentiable, and the model 116 can be trained with gradient descent. Note that the actual coefficients depend on the model variant being used.
Controlling the gradient propagation
The entire model can be trained end-to-end. However, the flow of the gradients can be stopped for various embodiments, including to implement monotonic prediction improvement and a single source of influence on glimpse. With respect to monotonic prediction improvement, the model 116 can make forward progress at any prediction boundary, in a greedy fashion. A parent prediction should not sacrifice itself so that its children glimpses make better predictions. This is important since at test time the model might quit at any level (e.g., for energy saving) . Thus, to emulate this behavior, when the summary function is computed at leveli, error gradient may not flow back to earlier levels.
With respect to single source of influence on glimpse, h
u can encode both g
u and
and
is dependent on g
u. The gradient can be stopped from flowing through h
u to g
u, thus forcing g
u to optimize
for the purpose of getting the maximum information. As such, “what” can precede “where. ”
Computation cost analysis
Suppose that a given CNN would take C FLOPS to compute over an M×M image (considering only a single channel (e.g., greyscale) ) . If the same CNN is used as the feature extractor
since the CNN computes over a glimpse whose resolution M
g×M
g is usually much lower than the original image, the CNN would take
FLOPS, wherein where N is the number of nodes and M is the ratio between image size and glimpse size. For example, residual networks often operate on 224 × 224 images on ImageNet dataset, so a glimpse size of 50 x 50 and a binary tree with depth-3 would take
of the original computation over the entire image considering the CNN alone.
The analysis above did not count the FLOPS of glimpses. The glimpse computation can include matrix multiplication among a filter bank with size M
g×M, the image sized M×M, and another filter bank M × M
g. This can take about
FLOPS counting both multiplications and additions, which can usually be significantly less than C. The glimpse computation can include Gaussian filter bank generation, which can include elementwise computations of M
g×M matrices, which may also be much less than C. For numerical stability, a small constant∈ can be added on both the numerator and denominator for the two fractions in our implementation. The same goes for the Parent-children containment penalty below.
From the discussion above, the number of FLOPS to execute the model 116 should be much less than that from the CNN over the entire image.
As discussed herein, the present solution can learn from the human visual cortex and adopt its design principles instead of implementations. The present solution can mimic how the human fovea system works by looking at the most information-carrying regions progressively, deriving a hierarchy of glimpses with increasingly finer details, and process them with a constant cost. It can adapt to data complexity and trade-off resource consumption and performance on-demand.
The present solution can be complementary to all existing CNN models in the sense that they can be used as an off-the-shelf plug-in module to extract features. The model 116 may not be deep, but rather adjust to scales and resolutions adaptively and dynamically, by zooming in (or out) at different nodes. As such, the model 116 can be decoupled with input image’s size and resolution. The model 116 can predict not only the class but also a progressively refined location map of the object. As such, the model 1166 can inherently integrate “what” and “where. ” The model 116 can be adaptive, making predictions with increasingly higher accuracy. This trade-off can be important when the model is under time or resource pressure, as it can quit earlier or extend the search of new evidences as needed. The model 116 can operate such that per-step visual feature extraction is constant. This allows the system 100 to control both computational resources and memory footprint. In some embodiments, this improvement of efficiency can be analogous to imposing sparsity at a higher granularity (e.g., regions of input) , rather than doing so at neuron or weight level. In some embodiments, the constant capacity of the feature extractor can force the discovery of structures within the data. By progressively focusing on the most information-carrying regions, the model 116 can ignore irrelevant background noises and therefore be less sensitive to adversarial attacks. The model 116 can rely on images containing one object, without requiring bounding box labels which are both error-prone and expensive to acquire, and can therefore be easier to improve.
As compared to one-pass models, the model 116 can be similar to neuroscience findings. For example, the model 116 can be analogous to a visual cortex that is not deep, but instead has abundant top-to-bottom and lateral connections. During the process, steps to resolve “where” and “what” are tightly intertwined, instead of separate.
The model 116 can fully explore the hierarchical and graphical view of feature extractions. Instead of relying on a serial recurrent network to compute subsequent glimpses, the model 116 can explicitly model the parent-to-children relationship in a top-down, coarse-to-finer manner. Predictions can be made by reading out the summary from a subset of graph nodes. Organizing glimpses into an hierarchy can significantly improve speed: for N glimpses the processing time is O (log N) instead of O (N) . Working out a hierarchy of features with an extractor with constant capacity also can lead to discovery of structures.
In some embodiments, each level adopts a different set of parameters. This can enable each feature extractor, while sharing the same architecture, to focus on capturing different scales. The present solution can integrate predicting glimpses of different sizes and shapes, applying Gaussian kernels to smoothly pool into a fixed input for feature extraction, and using deterministic actions instead of stochastic ones.
Single digit with structured background noises
An embodiment of the model 116 was evaluated with MNIST-cluttered dataset where a digit is placed at a random location of a n×n canvas, spread with k random 8×8 subpatches from other random MNIST digits.
The model 116 results were compared against RAM and DRAW with the same setting: for experiments on n=60, we set the k to 4, and for experiments on n=100, we set k to 8. The sizes of training, valid and test set are set to 50000, 10000 and 10000 respectively. The CNN the model 116 is a simple 5-layer with 12-by-12 input size.
Two different structures were selected, both are with 3 levels, in which one is a chain having 3 glimpses and the other is a full binary tree having 7 glimpses. The model with vector-based with levelwise attention and max-pooling was the readout module, and sequential module as the choice of predictor for the next glimpses, performed the best. For that model, sibling glimpse overlap penalty was disabled (i.e., λ
cc = 0) as well as parent-children containment penalty λ
pc was set to 0. The model 116 was optimized with RMSProp optimizer with learning rate 10
-4. Early stopping was adopted to avoid over-fitting.
The model 116 can also give levelwise predictions.
Table 1: Experiment results on Cluttered Translated MNIST task; shown with percentage of errors.
As shown in Table 1, the model 116 can outperform both RAM and DRAM, even with fewer number of glimpses. Furthermore, levelwise predictions of the model 116 can have progressively better accuracies.
Note that in the case of a single digit, having 3 glimpses can generalize better than having 7. We conjecture that with a fixed tree structure, the model is unable to selectively discard individual nodes or tree layers. The model will instead try to utilize these redundant nodes and layers to fit the training set. The simplest way to do so is to focus on background noises for these nodes to memorize the training examples better, which will result in overfitting. We note that naive gating and attention mechanisms could not appropriately mitigate this issue, since the model is in theory still able to look into background noises. We leave the resolution of node and level redundancy to future work.
FIG. 3 presents several visualizations of 100 × 100 cases, with 2-level (1 branch per node, 3 glimpses in total) and 3-level (2 branches per node, 7 glimpses in total) on the left half and right half, receptively. Level-2 and level-3 glimpses are colored with yellow and red boxes. Each sample is paired with an image reconstructed by overlaying appropriate up-sampling contents in the glimpses; this image thus shows the accumulated result of all the glimpses. As shown in FIG. 3, the glimpses focus on the object while ignoring the background noises. FIG. 3 shows original image and glimpse boxes (3 levels, 3 glimpses) in the first column, glimpse content (3 levels, 3 glimpses) in the second column, original image and glimpse boxes (3 levels, 7 glimpses) in the third column, and glimpse content (3 levels, 7 glimpses) in the fourth column.
The training curve of the 3-level tree model with 3 glimpses on 100 × 100 Cluttered Translated MNIST dataset is shown in FIG. 4 (level 1: 62.99; level 2: 95.85; level 3: 97.07) . As training progresses, predictions from all three levels can rise in tandem.
MNIST-Lego
The previous experiment discussed above can measure the model 116’s ability to detect “where” and “what” (e.g., to both classify and localize) . To test the ability of the model 116 to leverage and extract hierarchical features, a new dataset was made where two digits float freely in the upper and bottom half of the image, each with variable size. This makes up a total of 100 classes; class ab means digits a and b are at the top and the bottom half, respectively.
Experiments were performed with background with no noise, random Gaussian noise with various intensity, and clutters as described above.
A 3-level full binary tree structure with glimpse size 12 × 12 was picked, and the model 116 trained on two datasets with different background noise types described above. A vector-based with levelwise attention and max-pooling was selected as the readout module, and sequential model was the choice of predictor for the next glimpses. All penalties were disabled by setting their coefficients to zero (i.e., λ
pc = 0, λ
cc = 0, λ
res = 0) . The model 116 was optimized with RMSProp optimizer and the learning rate set to 10
-4. Early stopping was adopted to avoid over-fitting. The model 116 achieved an accuracy of 96.24%and 96.20%, with cluttered and Gaussian noise, respectively.
Training curves are shown in FIG. 6 (level 1: 46.04; level 2: 94.23; level 3: 96.20) . By extracting feature hierarchy, predictions made by two-and three-level improve consistently over time. FIG. 5 presents several visualization cases on the MNIST Lego dataset (Left: cluttered noise, Right: Gaussian noise) . In both cases, substructures were shown to be automatically discovered. To fight against more structured noises, more levels may be needed to localize parts (in this case each digit is one part of the overall class) , even though the end performances do not differ much. As such, focusing on prediction alone may not tell the whole story.
Table 2: Classification result on CUB-200-2011 dataset. The CNN baselines resize the input image to a fixed size (e.g., 200x200) , whereas our model does so adaptively and dynamically, the 100x100 model thus takes roughly half of the computational cost to achieve the same level of performance.
Real Image
The model 116 was trained on the CIFAR-10 dataset. The dataset consists of 60,000 32 ×32 RGB images, 50,000 for training and 10,000 for testing. We chose the 3-level full binary tree with glimpse size 15 × 15. The object takes up a large portion of the image, and the glimpse size is large compared to that of the images, such that requiring the children to cover different image parts may not be effective. Disabled parent-children containment penalty was disabled as well as sibling glimpse overlap penalty (i.e., λ
pc = λ
cc = 0) as a result. f
φ was initialized using a pretrained ResNet-18.92%accuracy was achieved on the test set. Some of the examples are visualized in FIGS. 7A-7B. FIGS. 7A-7B show glimpse bounding box visualizations (FIG. 7A) in which the two yellow boxes correspond to the glimpses on the second level, while the four red ones are for the glimpses on the third (last) level; FIG. 7B shows glimpse content visualizations on the same image; this figure is obtained by applying the generated Gaussian filter banks as masks on the original image.
The model 116 was tested on fine-grained classification task as shown in FIGS. 8A-C using the CUB-200-2011 dataset. FIG. 8A shows glimpse bounding box of structure 1-1; FIG. 8B shows glimpse bounding box of structure 1-2; FIG. 8C shows glimpse bounding box of structure 1-1-1. This figure was obtained by applying the generated Gaussian filter banks as masks on the original image. The dataset consists of 11, 788 images of birds with 200 types. We resized all images to 500x500 for both training and evaluating. We chose several structures: 2-level (1-1) , 2-level (1-2) , 3-level (1-1-1) trees for this task. The f
φ module in our model was initialized with pretrained ResNet-50,
is channel-se normalized from
We again selected vector-based with levelwise attention and max-pooling as our readout module, and used sequential model as the choice of predictor for next glimpses. We achieved comparable results on test set compared to directly applying the same CNN on the image with less FLOPs, details are listed in Table 2.
The model 116 can incorporate static topology with fine-grained control. For example, in some embodiments, having redundant nodes and levels hurt generalization. Some of the future works include adaptively growing one node at a time during the training procedure by looking at the best validation metric, in similar spirit to, or imposing sparsity constraints on gate activations or attention masks.
The model 116 can incorporate dynamic topology. For example, in some embodiments, the branch factor K and the depth of the tree L are both hyper-parameters. In some embodiments, a flexible model shall zoom-in where needed, while stop growing when statistics of the current glimpse is already sufficient. However, such stochastic decisions can make the model hard to learn. The model 116 can have metrics established such that an overgrown tree can be truncated at test time without much loss of performance.
The model 116 can incorporate dynamic CNN parameters. Sharing feature extractor parameters within a level may not be very principled, since there is no guarantee that glimpses at a given level are inspecting the same level of details. This assumption may not be true across images, and parameters may be refined on-demand.
The model 116 can incorporate multiple iterations. The model 116 may offer a plausible explanation to why there are abundant top-down connection in our visual cortex. At least partially, it is a feedback signal to direct next focus. Lateral connection can be explained by introducing multiple iterations over the same graph. This way, node representation can become increasingly richer. This may be implemented using Message-Passing Neural Network ( “MPNN” ) . The model 116 can offer a new perspective to explain how brain works.
The model 116 can integrate with multi-object detection. The tree of the model 116 is for one object. To parse a scene with multiple objects, the model 116 can impose one more level on top. In some embodiments, since “objectiveness” is far easier to detect than classification, the model 116 can leverage the localization part of other models, such as YOLO, to propose the top-level branches. In some embodiments, the model 116 can predict a multiset to learn from scratch.
The model 116 can be extended to image sequences. The multiple-glimpses induced by the model 116 can map out the most salient points about an object. As such, the locations and features can be more informative than a single bounding box, as many the video tracking systems do.
Overlay function
A Gaussian glimpse function can have the following form:
The corresponding inverse Gaussian glimpse function can be defined as
Equivalently, the transpose of the Gaussian filter banks can be taken, normalized by column, and the resulting filters applied on the glimpse x
u to recover to the size of original canvas.
The formulation of overlaying a new C-channel image x
u on top of an old C-channel image x at the canvas location g
u can be given. Alpha-channel overlaying can be performed as follows:
Concatenating with the new image x
u another feature map filled with 1’s:
The additional channel of 1’s can be referred to as an alpha channel.
Compute the feature map to overlay:
where c = 1, ..., C+ 1.
Separate the alpha channel:
The overlay is obtained by
Definitions
As utilized herein, the terms “approximately, ” “about, ” “substantially, ” and similar terms are intended to have a broad meaning in harmony with the common and accepted usage by those of ordinary skill in the art to which the subject matter of this disclosure pertains. It should be understood by those of skill in the art who review this disclosure that these terms are intended to allow a description of certain features described and claimed without restricting the scope of these features to the precise numerical ranges provided. Accordingly, these terms should be interpreted as indicating that insubstantial or inconsequential modifications or alterations of the subject matter described and claimed are considered to be within the scope of the disclosure as recited in the appended claims.
It should be noted that the term “exemplary” and variations thereof, as used herein to describe various embodiments, are intended to indicate that such embodiments are possible examples, representations, and/or illustrations of possible embodiments (and such terms are not intended to connote that such embodiments are necessarily extraordinary or superlative examples) .
The term “coupled, ” as used herein, means the joining of two members directly or indirectly to one another. Such joining may be stationary (e.g., permanent or fixed) or moveable (e.g., removable or releasable) . Such joining may be achieved with the two members coupled directly to each other, with the two members coupled to each other using a separate intervening member and any additional intermediate members coupled with one another, or with the two members coupled to each other using an intervening member that is integrally formed as a single unitary body with one of the two members. Such members may be coupled mechanically, electrically, and/or fluidly.
The term “or, ” as used herein, is used in its inclusive sense (and not in its exclusive sense) so that when used to connect a list of elements, the term “or” means one, some, or all of the elements in the list. Conjunctive language such as the phrase “at least one of X, Y, and Z, ” unless specifically stated otherwise, is understood to convey that an element may be either X, Y, Z; X and Y; X and Z; Y and Z; or X, Y, and Z (i.e., any combination of X, Y, and Z) . Thus, such conjunctive language is not generally intended to imply that certain embodiments require at least one of X, at least one of Y, and at least one of Z to each be present, unless otherwise indicated.
References herein to the positions of elements (e.g., “top, ” “bottom, ” “above, ” “below, ” etc. ) are merely used to describe the orientation of various elements in the FIGURES. It should be noted that the orientation of various elements may differ according to other exemplary embodiments, and that such variations are intended to be encompassed by the present disclosure.
The hardware and data processing components used to implement the various processes, operations, illustrative logics, logical blocks, modules and circuits described in connection with the embodiments disclosed herein may be implemented or performed with a general purpose single-or multi-chip processor, a digital signal processor ( “DSP” ) , an ASIC, an FPGA, or other programmable logic device, discrete gate or transistor logic, discrete hardware components, or any combination thereof designed to perform the functions described herein. A general purpose processor may be a microprocessor, or, any conventional processor, controller, microcontroller, or state machine. A processor also may be implemented as a combination of computing devices, such as a combination of a DSP and a microprocessor, a plurality of microprocessors, one or more microprocessors in conjunction with a DSP core, or any other such configuration. In some embodiments, particular processes and methods may be performed by circuitry that is specific to a given function. The memory (e.g., memory, memory unit, storage device, etc. ) may include one or more devices (e.g., RAM, ROM, Flash memory, hard disk storage, etc. ) for storing data and/or computer code for completing or facilitating the various processes, layers and modules described in the present disclosure. The memory may be or include volatile memory or non-volatile memory, and may include database components, object code components, script components, or any other type of information structure for supporting the various activities and information structures described in the present disclosure. According to an exemplary embodiment, the memory is communicably connected to the processor via a processing circuit and includes computer code for executing (e.g., by the processing circuit and/or the processor ) the one or more processes described herein.
The present disclosure contemplates methods, systems and program products on any machine-readable media for accomplishing various operations. The embodiments of the present disclosure may be implemented using existing computer processors, or by a special purpose computer processor for an appropriate system, incorporated for this or another purpose, or by a hardwired system. Embodiments within the scope of the present disclosure include program products comprising machine-readable media for carrying or having machine-executable instructions or data structures stored thereon. Such machine-readable media can be any available media that can be accessed by a general purpose or special purpose computer or other machine with a processor. By way of example, such machine-readable media can comprise RAM, ROM, EPROM, EEPROM, or other optical disk storage, magnetic disk storage or other magnetic storage devices, or any other medium which can be used to carry or store desired program code in the form of machine-executable instructions or data structures and which can be accessed by a general purpose or special purpose computer or other machine with a processor. Combinations of the above are also included within the scope of machine-readable media. Machine-executable instructions include, for example, instructions and data which cause a general purpose computer, special purpose computer, or special purpose processing machines to perform a certain function or group of functions.
Although the figures and description may illustrate a specific order of method steps, the order of such steps may differ from what is depicted and described, unless specified differently above. Also, two or more steps may be performed concurrently or with partial concurrence, unless specified differently above. Such variation may depend, for example, on the software and hardware systems chosen and on designer choice. All such variations are within the scope of the disclosure. Likewise, software implementations of the described methods could be accomplished with standard programming techniques with rule-based logic and other logic to accomplish the various connection steps, processing steps, comparison steps, and decision steps.
Claims (10)
- An object recognition system, comprising:one or more processors; anda memory storing computer-readable instructions that when executed by the one or more processors, cause the one or more processors to:provide, to an image recognition model, an input image;generate, by the image recognition model, a plurality of patches of the image using a plurality of corresponding glimpses of the image;extract, from each patch, a plurality of features of the patch; andpredict, by the image recognition model, an object represented by the input image based on a sequence of the glimpses and each corresponding feature of each glimpse.
- The object recognition system of claim 1, comprising instructions that cause the one or more processors to train the image recognition model by evaluating an expected loss function based on the predicted object and a ground truth corresponding to the input image.
- The object recognition system of claim 1, comprising instructions that cause the one or more processors to extract the plurality of features using a convolutional neural network (CNN) .
- The object recognition system of claim 3, wherein the CNN has a fixed receptive field and constant capacity.
- The object recognition system of claim 1, comprising instructions that cause the one or more processors to generate a plurality of dependencies corresponding to glimpses at different time steps as a directed graph comprising a plurality of nodes, each node corresponding to a respective glimpse of the plurality of glimpses.
- The object recognition system of claim 1, comprising instructions that cause the one or more processors to predict the object using levelwise prediction.
- The object recognition system of claim 1, comprising instructions that cause the one or more processors to generate a tree of the plurality of glimpses, a root node of the tree having a glimpse over the entire image.
- The object recognition system of claim 1, comprising instructions that cause the one or more processors to generate each glimpse using a Gaussian kernel.
- The object recognition system of claim 1, comprising instructions that cause the one or more processors to generate a plurality of predictions of the object at a plurality of time steps, each prediction using data of each previous time step.
- The object recognition system of claim 1, comprising instructions that cause the one or more processors to use a recurrent network to predict the object.
Priority Applications (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
PCT/CN2018/110403 WO2020077525A1 (en) | 2018-10-16 | 2018-10-16 | Systems and methods for model for incremental and adaptive object recognition using hierarchical representations |
Applications Claiming Priority (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
PCT/CN2018/110403 WO2020077525A1 (en) | 2018-10-16 | 2018-10-16 | Systems and methods for model for incremental and adaptive object recognition using hierarchical representations |
Publications (1)
Publication Number | Publication Date |
---|---|
WO2020077525A1 true WO2020077525A1 (en) | 2020-04-23 |
Family
ID=70283698
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
PCT/CN2018/110403 WO2020077525A1 (en) | 2018-10-16 | 2018-10-16 | Systems and methods for model for incremental and adaptive object recognition using hierarchical representations |
Country Status (1)
Country | Link |
---|---|
WO (1) | WO2020077525A1 (en) |
Cited By (3)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN112906718A (en) * | 2021-03-09 | 2021-06-04 | 西安电子科技大学 | Multi-target detection method based on convolutional neural network |
CN113688723A (en) * | 2021-08-21 | 2021-11-23 | 河南大学 | Infrared image pedestrian target detection method based on improved YOLOv5 |
US20220058427A1 (en) * | 2018-11-30 | 2022-02-24 | Huawei Technologies Co., Ltd. | Image Parsing Method and Apparatus |
Citations (4)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN106778860A (en) * | 2016-12-12 | 2017-05-31 | 中国矿业大学 | Image position method based on Histogram Matching |
US20170287137A1 (en) * | 2016-03-31 | 2017-10-05 | Adobe Systems Incorporated | Utilizing deep learning for boundary-aware image segmentation |
CN107527053A (en) * | 2017-08-31 | 2017-12-29 | 北京小米移动软件有限公司 | Object detection method and device |
CN108292369A (en) * | 2015-12-10 | 2018-07-17 | 英特尔公司 | Visual identity is carried out using deep learning attribute |
-
2018
- 2018-10-16 WO PCT/CN2018/110403 patent/WO2020077525A1/en active Application Filing
Patent Citations (4)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN108292369A (en) * | 2015-12-10 | 2018-07-17 | 英特尔公司 | Visual identity is carried out using deep learning attribute |
US20170287137A1 (en) * | 2016-03-31 | 2017-10-05 | Adobe Systems Incorporated | Utilizing deep learning for boundary-aware image segmentation |
CN106778860A (en) * | 2016-12-12 | 2017-05-31 | 中国矿业大学 | Image position method based on Histogram Matching |
CN107527053A (en) * | 2017-08-31 | 2017-12-29 | 北京小米移动软件有限公司 | Object detection method and device |
Cited By (6)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20220058427A1 (en) * | 2018-11-30 | 2022-02-24 | Huawei Technologies Co., Ltd. | Image Parsing Method and Apparatus |
US11734953B2 (en) * | 2018-11-30 | 2023-08-22 | Huawei Technologies Co., Ltd. | Image parsing method and apparatus |
CN112906718A (en) * | 2021-03-09 | 2021-06-04 | 西安电子科技大学 | Multi-target detection method based on convolutional neural network |
CN112906718B (en) * | 2021-03-09 | 2023-08-22 | 西安电子科技大学 | Multi-target detection method based on convolutional neural network |
CN113688723A (en) * | 2021-08-21 | 2021-11-23 | 河南大学 | Infrared image pedestrian target detection method based on improved YOLOv5 |
CN113688723B (en) * | 2021-08-21 | 2024-03-19 | 河南大学 | Infrared image pedestrian target detection method based on improved YOLOv5 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
Li et al. | Zoom out-and-in network with map attention decision for region proposal and object detection | |
Gonzalez et al. | Deep convolutional recurrent autoencoders for learning low-dimensional feature dynamics of fluid systems | |
Baldassarre et al. | Explainability techniques for graph convolutional networks | |
Ghiasi et al. | Laplacian pyramid reconstruction and refinement for semantic segmentation | |
Ayachi et al. | Strided convolution instead of max pooling for memory efficiency of convolutional neural networks | |
Moniz et al. | Nested lstms | |
Zhou et al. | Context-reinforced semantic segmentation | |
Calderero et al. | Region merging techniques using information theory statistical measures | |
US8504493B2 (en) | Self-organizing sequential memory pattern machine and reinforcement learning method | |
WO2020077525A1 (en) | Systems and methods for model for incremental and adaptive object recognition using hierarchical representations | |
Fu et al. | Continual image deraining with hypergraph convolutional networks | |
Yuan et al. | Polyphonicformer: Unified query learning for depth-aware video panoptic segmentation | |
Nithin et al. | Generic feature learning in computer vision | |
Tesfaldet et al. | Attention-based neural cellular automata | |
Govathoti et al. | Data augmentation techniques on chilly plants to classify healthy and bacterial blight disease leaves | |
Qin et al. | Deep Learning Basics | |
Deck et al. | Easing color shifts in score-based diffusion models | |
Rajalakshmi et al. | Hyperspectral Remote Sensing Image Classification Using Improved Metaheuristic with Deep Learning. | |
Fakhari et al. | A new restricted boltzmann machine training algorithm for image restoration | |
Yuan et al. | Sin: Superpixel interpolation network | |
Du et al. | AViT: Adapting Vision Transformers for Small Skin Lesion Segmentation Datasets | |
Zhang et al. | Fast contour detection with supervised attention learning | |
Xu et al. | Infusing Spatial Knowledge into Deep Learning for Earth Science: A Hydrological Application | |
Nuzzo | Sanity checks for explanations of deep neural networks predictions | |
Bakurov et al. | Semantic segmentation network stacking with genetic programming |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
121 | Ep: the epo has been informed by wipo that ep was designated in this application |
Ref document number: 18937259 Country of ref document: EP Kind code of ref document: A1 |
|
NENP | Non-entry into the national phase |
Ref country code: DE |
|
122 | Ep: pct application non-entry in european phase |
Ref document number: 18937259 Country of ref document: EP Kind code of ref document: A1 |