WO2017132830A1 - Methods and systems for cnn network adaption and object online tracking - Google Patents

Methods and systems for cnn network adaption and object online tracking Download PDF

Info

Publication number
WO2017132830A1
WO2017132830A1 PCT/CN2016/073184 CN2016073184W WO2017132830A1 WO 2017132830 A1 WO2017132830 A1 WO 2017132830A1 CN 2016073184 W CN2016073184 W CN 2016073184W WO 2017132830 A1 WO2017132830 A1 WO 2017132830A1
Authority
WO
WIPO (PCT)
Prior art keywords
adaptive
cnn
frame
training
trained
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Ceased
Application number
PCT/CN2016/073184
Other languages
French (fr)
Inventor
Xiaogang Wang
Lijun Wang
Wanli OUYANG
Huchuan Lu
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Individual
Original Assignee
Individual
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Individual filed Critical Individual
Priority to CN201680080165.1A priority Critical patent/CN108701210B/en
Priority to PCT/CN2016/073184 priority patent/WO2017132830A1/en
Publication of WO2017132830A1 publication Critical patent/WO2017132830A1/en
Priority to US16/052,500 priority patent/US11521095B2/en
Anticipated expiration legal-status Critical
Ceased legal-status Critical Current

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
    • G06V10/82—Arrangements for image or video recognition or understanding using pattern recognition or machine learning using neural networks
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F18/00—Pattern recognition
    • G06F18/20—Analysing
    • G06F18/21—Design or setup of recognition systems or techniques; Extraction of features in feature space; Blind source separation
    • G06F18/217—Validation; Performance evaluation; Active pattern learning techniques
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F18/00—Pattern recognition
    • G06F18/20—Analysing
    • G06F18/24—Classification techniques
    • G06F18/241—Classification techniques relating to the classification model, e.g. parametric or non-parametric approaches
    • G06F18/2413—Classification techniques relating to the classification model, e.g. parametric or non-parametric approaches based on distances to training or reference patterns
    • G06F18/24133—Distances to prototypes
    • G06F18/24143—Distances to neighbourhood prototypes, e.g. restricted Coulomb energy networks [RCEN]
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/045—Combinations of networks
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/0464—Convolutional networks [CNN, ConvNet]
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods
    • G06N3/084—Backpropagation, e.g. using gradient descent
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods
    • G06N3/09—Supervised learning
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods
    • G06N3/096—Transfer learning
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N5/00—Computing arrangements using knowledge-based models
    • G06N5/04—Inference or reasoning models
    • G06N5/046—Forward inferencing; Production systems
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/10—Segmentation; Edge detection
    • G06T7/11—Region-based segmentation
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/20—Analysis of motion
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/70—Determining position or orientation of objects or cameras
    • G06T7/73—Determining position or orientation of objects or cameras using feature-based methods
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/40—Extraction of image or video features
    • G06V10/44—Local feature extraction by analysis of parts of the pattern, e.g. by detecting edges, contours, loops, corners, strokes or intersections; Connectivity analysis, e.g. of connected components
    • G06V10/443—Local feature extraction by analysis of parts of the pattern, e.g. by detecting edges, contours, loops, corners, strokes or intersections; Connectivity analysis, e.g. of connected components by matching or filtering
    • G06V10/449—Biologically inspired filters, e.g. difference of Gaussians [DoG] or Gabor filters
    • G06V10/451—Biologically inspired filters, e.g. difference of Gaussians [DoG] or Gabor filters with interaction between the filter responses, e.g. cortical complex cells
    • G06V10/454—Integrating the filters into a hierarchical structure, e.g. convolutional neural networks [CNN]
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
    • G06V10/764—Arrangements for image or video recognition or understanding using pattern recognition or machine learning using classification, e.g. of video objects
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
    • G06V10/77—Processing image or video features in feature spaces; using data integration or data reduction, e.g. principal component analysis [PCA] or independent component analysis [ICA] or self-organising maps [SOM]; Blind source separation
    • G06V10/776—Validation; Performance evaluation
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V20/00—Scenes; Scene-specific elements
    • G06V20/10—Terrestrial scenes
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20081—Training; Learning

Definitions

  • pre-trained deep features can be effectively transferred for online application with a reduced over-fitting.
  • the proposed object online tracking method, apparatus and system are constructed based on the proposed CNN adaption method, apparatus and system. Due to the reduced over-fitting, the proposed object online tracking method and system can perform an improved object online tracking.
  • Figs. 2a-2c illustrate an adaptive CNN for adapting a pre-trained CNN to a target video, according to an embodiment of the present application.
  • Fig. 4 is a flowchart illustrating the process of object online tracking, according to an embodiment of the present application.
  • Fig. 5 is a schematic diagram illustrating a system architecture for the object online tracking, according to an embodiment of the present application.
  • Fig. 1 illustrates such a conventional method 1000 for adapting a pre-trained CNN 1100 to a target image.
  • an adaptive CNN 1200 is disposed sequence to the pre-trained CNN 1100 and configured for refining the pre-trained features to a final adapted feature 1300 for supervision.
  • an adaptive CNN 2200 is proposed for better transferring pre-trained deep features, as shown in Fig. 2a to Fig. 2c.
  • the pre-trained CNN 2100 is denoted as CNN-E, and an RGB image is taken as input and a convolution feature map X is output.
  • the online adaptive CNN 2200 denoted as CNN-A, is randomly initialized and consists of two convolution layers interleaved with a ReLU layer as the nonlinear activation unit.
  • the first convolution layer is linked to the pre-trained CNN 2100 and configured to transform the first feature map into the plurality of sub-feature maps.
  • the sub-feature map in the second layer is obtained by convolving the kernel with the sub-feature map in the first layer as:
  • C 1 denotes the number of channels of the sub-feature map output by the first convolution layer
  • b c is the bias
  • symbol "*" denotes a convolution operation. The summation is conducted over all the channels.
  • the parameter indicates the corresponding kernel weights and bias in both the first and second convolution layers of CNN-A.
  • a plurality of parameters are trained for each of the adaptive convolution kernels, respectively, after the initialization.
  • the parameter ⁇ * with the smallest training error is selected and grouped into an ensemble set ⁇ and the rest of trained parameters are grouped into a candidate set c.
  • the parameters in the candidate set will be optimized in following frames of the target video.
  • two or more parameters with smallest training errors may be selected into the ensemble set.
  • a mask layer may be contained in the adaptive CNN and linked to the second convolution layer to further reduce a correlation among the sub-feature maps. Specifically, each channel of the output sub-feature map from the second convolution layer is associated with an individual binary mask which has the same spatial size with the sub-feature map. All the masks are initialized in a random manner and then fixed throughout the online training process. The forward propagation of the convolution layer at the training stage is then conducted as
  • a method for an object online tracking.
  • Fig. 4 schematically illustrates a general flowchart for an object online tracking.
  • a target object is manually selected with a target boundary box in the first frame of the video, and accordingly, a region of interest (ROI) is determined.
  • the ROI is centered at the target object.
  • the ROI may have twice the size of the target boundary box, as an example.
  • the ROI is fed forward a pre-trained CNN to extract an initial feature map for initialization.
  • the initial feature map comprises the information of the location and the scale of the target object in the first frame.
  • both the adaptive CNN and the scale estimation network have better adaptabilities with respect to the relating frame.
  • the location and scale of target object in the subsequent frames of the target video will be predicted and estimated based on the updated adaptive CNN (step S407 and S408) .
  • the result of the prediction and estimation further updates, in turn, the adaptive CNN and the scale estimation network.
  • a feature extraction unit 501 comprising a pre-trained CNN, i.e., CNN-E, is configured to determine a ROI and to extract the feature of any frame.
  • the extracted feature is transmitted to a location prediction unit 502 comprising the adaptive CNN and a scale estimation unit 503 comprising the scale estimation network.
  • the extracted feature is also transmitted to an initialization and update unit 504 for initializing at the first frame of the target video.
  • the extracted feature map of the current frame and the output from the location prediction unit 502 and scale estimation unit 503 are transmitted to the initialization and update unit 504 to update the adaptive CNN and the scale estimation network.
  • the adaptive CNN is specifically used to transform, by performing a heat map regression, the feature map extracted from the pre-trained CNN into a target heat map.
  • the location of the target object is then determined by the location on the heat map with the maximum value and the corresponding maximum heat map value serves as the confidence of this prediction.
  • the updating of the adaptive CNN and the scale estimation network are conducted only if the confidence is higher than a pre-defined threshold, in order to avoid updating using contaminated training sample.
  • the present application may be embodied as a system, a method or a computer program product. Accordingly, the present application may take the form of an entirely hardware embodiment and hardware aspects that may all generally be referred to herein as a "unit” , “circuit” , “module” , or “system” .
  • ICs integrated circuits

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Evolutionary Computation (AREA)
  • Artificial Intelligence (AREA)
  • General Health & Medical Sciences (AREA)
  • Health & Medical Sciences (AREA)
  • Computing Systems (AREA)
  • Software Systems (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Data Mining & Analysis (AREA)
  • Multimedia (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • General Engineering & Computer Science (AREA)
  • Biomedical Technology (AREA)
  • Molecular Biology (AREA)
  • Mathematical Physics (AREA)
  • Computational Linguistics (AREA)
  • Biophysics (AREA)
  • Databases & Information Systems (AREA)
  • Medical Informatics (AREA)
  • Biodiversity & Conservation Biology (AREA)
  • Bioinformatics & Computational Biology (AREA)
  • Evolutionary Biology (AREA)
  • Bioinformatics & Cheminformatics (AREA)
  • Image Analysis (AREA)

Abstract

Disclosed are methods, apparatuses and systems for CNN network adaption and object online tracking. The CNN network adaption method comprises: transforming a first feature map into a plurality of sub-feature maps, wherein the first feature map is generated by the pre-trained CNN according to a frame of the target video; convolving each of the sub-feature maps with one of a plurality of adaptive convolution kernels, respectively, to output a plurality of second feature maps with improved adaptability; training, frame by frame, the adaptive convolution kernels.

Description

METHODS AND SYSTEMS FOR CNN NETWORK ADAPTION AND OBJECT ONLINE TRACKING Technical Field
The disclosures relate to methods, apparatuses and systems for Convolutional Neural Network (CNN) network adaption and object online tracking.
Background
Visual tracking is a fundamental problem in computer vision that has been received a rapidly growing attention. For a model-free object tracking problem, a category agnostic target is indicated by a bounding box in the first frame, and the tracker aims at locating the target in each of the following frames. Due to significant target appearance changes caused by abrupt motion, deformation, occlusion and illumination variation, visual tracking is still a challenging problem. Prior approaches rely on hand-crafted features to describe the target and have addressed the above challenging factors to a certain extend.
Recently, deep CNNs trained on large scale image classification data sets have demonstrated great success. These semantic representations discovered by the learning process are shown to be very effective at distinguishing objects of various categories. However, supervised training of deep CNNs with millions of parameters entails a large number of annotated training samples. To apply deep CNNs for tasks with a limited amount of training samples, previous approaches adopt a transfer learning method by first pre-training a deep CNN on a source task with a large scale training data set and then fine-tuning the learned feature on the target task. Due to the good generalization capability of CNN features across different data sets, this transfer learning approach is effective and has shown state-of-the-art performance in many applications.
However, for online visual tracking, the lack of training samples becomes even more severe, since the only training sample with ground truth label is provided in the first frame,  and the tracking results used for updating the tracker are also obtained in a sequential manner. Thus, directly fine-tuning a pre-trained deep CNN online is prone to over-fitting, which will degrade the tracker and gradually leads to tracking drift
Summary
The following presents a simplified summary of the disclosure in order to provide a basic understanding of some aspects of the disclosure. This summary is not an extensive overview of the disclosure. It is intended to neither identify key or critical elements of the disclosure nor delineate any scope of particular embodiments of the disclosure, or any scope of the claims. Its sole purpose is to present some concepts of the disclosure in a simplified form as a prelude to the more detailed description that is presented later.
In order to address, at least partially, one of the above issues, a CNN network adaption method is proposed for adapting a pre-trained CNN to a target video, in one aspect of the present application. The method comprises: transforming a first feature map into a plurality of sub-feature maps, wherein the first feature map is generated by the pre-trained CNN according to a frame of the target video； convolving each of the sub-feature maps with one of a plurality of adaptive convolution kernels, respectively, to output a plurality of second feature maps with improved adaptability； and training, frame by frame, the adaptive convolution kernels.
In one embodiment of the present application, the transforming and the convolving are implemented in an adaptive CNN comprising: a first convolution layer, linked to the pre-trained CNN and configured to transform the first feature map into the plurality of sub-feature maps； and a second convolution layer, linked to the first convolution layer and configured to convolve each of the sub-feature maps with one of the adaptive convolution kernels, respectively.
In one embodiment of the present application, the training comprises: feeding a first  training sample forward through the pre-trained CNN and the adaptive CNN to generate a first output image, wherein the first training sample is obtained according to a first frame of the target video； comparing the generated first output image with a first ground truth derived from the first frame to obtain a plurality of first training errors for the adaptive convolution kernels, respectively； back-propagating repeatedly the first training errors through the pre-trained CNN and the adaptive CNN to train the adaptive convolution kernels until the first training errors converge, wherein a plurality of parameters are obtained for the trained adaptive convolution kernels, respectively； grouping a parameter of the obtained parameters, which has a smallest first training error, and a rest of the obtained parameters into an ensemble set and a candidate set, respectively； and optimizing, according to a subsequent frame of the target video, the parameters grouped in the candidate set.
In one embodiment of the present application, the optimizing comprises: feeding the second training sample forward through the pre-trained CNN and the adaptive CNN to generate a second output image, wherein the second training sample is obtained according to a second frame of the target video and the second frame is subsequent to the first frame； comparing the second output image with a second ground truth derived from the second frame to obtain a plurality of second training errors for the plurality of adaptive convolution kernels； and if any of the second training errors is higher than a threshold, back-propagating the second training errors through the pre-trained CNN and the adaptive CNN to further refine the parameters in the candidate set and transferring at least one of the further refined parameters to the ensemble set.
In one embodiment of the present application, each of the adaptive convolution kernels is trained under a different loss criterion.
In one embodiment of the present application, the method further comprises further reducing, by a mask layer, a correlation among the sub-feature maps, wherein the mask layer is linked to the second convolution layer of the adaptive CNN.
In one embodiment of the present application, the mask layer comprises a plurality of binary masks, each of which is convolved with one of the sub-feature maps and has a same spatial size with the convolved sub-feature map.
In another aspect, a method is proposed for an object online tracking, comprising: determining a region of interest (ROI) in a first frame of a target video； feeding the determined ROI forward through a pre-trained CNN to extract an initial feature map thereof； initializing, with the initial feature map, an adaptive CNN used for detecting a location of the object and a scale estimation network used for defining a scale of the object； predicting, with the initialized adaptive CNN, a second location of the object in a second frame of the target video, wherein the second frame is subsequent to the first frame； estimating, with the initialized scale estimation network, a second scale of the object in the second frame of the target video； updating, with optimized network parameters acquired in the predicting and the estimating, the adaptive CNN and the scale estimation network, respectively； predicting, with the updated adaptive CNN, a third location of the object in a third frame of the target video, wherein the third frame is subsequent to the second frame； and estimating, with the updated scale estimation network, a third scale of the object in the third frame of the target video.
In one embodiment of the present application, the adaptive CNN comprises: a first convolution layer, linked to the pre-trained CNN and configured to transform a first feature map into a plurality of sub-feature maps, wherein the first feature map is generated by the pre-trained CNN according to any frame of the target video； and a second convolution layer, linked to the first convolution layer and configured to convolve each of the sub-feature maps with one of a plurality of adaptive convolution kernels , respectively, to output a plurality of second feature maps with improved adaptability.
In one embodiment of the present application, the adaptive CNN is initialized by: feeding a first training sample forward through the pre-trained CNN and the adaptive CNN to generate a first output image, wherein the first training sample is obtained according to a first frame of the target video； comparing the generated first output image with a first ground truth  derived from the first frame to obtain a plurality of first training errors for the adaptive convolution kernels, respectively； back-propagating repeatedly the first training errors through the pre-trained CNN and the adaptive CNN to train the adaptive convolution kernels until the first training errors converges, wherein a plurality of parameters are obtained for the trained adaptive convolution kernels, respectively； grouping a parameter of the obtained parameters, which has a smallest first training error, and a rest of the obtained parameters into an ensemble set and a candidate set, respectively.
In one embodiment of the present application, the adaptive CNN is updated by: feeding a second training sample forward through the pre-trained CNN and the adaptive CNN to generate a second output image, wherein the second training sample is obtained according to a second frame of the target video and the second frame is subsequent to the first frame； comparing the second output image with a second ground truth derived from the second frame to obtain a plurality of second training errors for the plurality of adaptive convolution kernels, respectively； and if any of the second training errors is higher than a threshold, back-propagating the second training errors through the pre-trained CNN and the adaptive CNN to further refine the parameters in the candidate set and transferring at least one of the further refined parameters to the ensemble set.
In one embodiment of the present application, each of the adaptive convolution kernels is trained under a different loss criterion.
In one embodiment of the present application, the adaptive CNN further comprises a mask layer linked to the second convolution layer to further reduce a correlation among the sub-feature maps.
In one embodiment of the present application, the mask layer comprises a plurality of binary masks, each of which is convolved with one of the sub-feature maps and has a same spatial size with the convolved sub-feature map.
In one embodiment of the present application, the location of the object is predicted by a heat map generated by the adaptive CNN, wherein a location with a maximum value is predicted to be the location of the object and the maximum value is sampled as a confidence.
In one embodiment of the present application, the updating is performed only if the confidence is higher than a pre-defined threshold.
In one embodiment of the present application, the ROI is centered at an object to be tracked.
In another aspect, a system is proposed for adapting a pre-trained CNN to a target video, comprising: a memory that stores executable components； and a processor electrically coupled to the memory to execute the executable components. The executable components are executed for: transforming a first feature map into a plurality of sub-feature maps, wherein the first feature map is generated by the pre-trained CNN according to a frame of the target video； convolving each of the sub-feature maps with one of a plurality of adaptive convolution kernels, respectively, to output a plurality of second feature maps with improved adaptability； training, frame by frame, the adaptive convolution kernels.
In another aspect, a system is proposed for an object online tracking, comprising: a memory that stores executable components； and a processor electrically coupled to the memory to execute the executable components. The executable components are executed for: determining a region of interest (ROI) in a first frame of a target video； feeding the determined ROI forward through a pre-trained CNN to extract an initial feature map thereof； initializing, with the initial feature map, an adaptive CNN used for detecting a location of the object and a scale estimation network used for defining a scale of the object； predicting, with the initialized adaptive CNN, a second location of the object in a second frame of the target video, wherein the second frame is subsequent to the first frame； estimating, with the initialized scale estimation network, a second scale of the object in the second frame of the target video； updating, with optimized network parameters acquired in the predicting and the  estimating, the adaptive CNN and the scale estimation network, respectively； predicting, with the updated adaptive CNN, a third location of the object in a third frame of the target video, wherein the third frame is subsequent to the second frame； and estimating, with the updated scale estimation network, a third scale of the object in the third frame of the target video.
In another aspect, an apparatus is proposed for adapting a pre-trained CNN to a target video, comprising: means for transforming a first feature map into a plurality of sub-feature maps, wherein the first feature map is generated by the pre-trained CNN according to a frame of the target video； means for convolving each of the sub-feature maps with one of a plurality of adaptive convolution kernels, respectively, to output a plurality of second feature maps with improved adaptability； and means for training, frame by frame, the adaptive convolution kernels.
In another aspect, an apparatus is proposed for an object online tracking. The apparatus comprises a feature extraction unit, configured for: determining a region of interest (ROI) in a first frame of a target video； and feeding the determined ROI forward through a pre-trained CNN to extract an initial feature map thereof. The apparatus further comprises: an initialization and update unit, configured for initializing, with the initial feature map, an adaptive CNN used for detecting a location of the object and a scale estimation network used for defining a scale of the object； a location prediction unit, configured for predicting, with the initialized adaptive CNN, a second location of the object in a second frame of the target video, wherein the second frame is subsequent to the first frame； and a scale estimation unit, configured for estimating, with the initialized scale estimation network, a second scale of the object in the second frame of the target video. In addition, the initialization and update unit is further configured for updating, with optimized network parameters acquired in the predicting and the estimating, the adaptive CNN and the scale estimation network, respectively； the location prediction unit is further configured for predicting, with the updated adaptive CNN, a third location of the object in a third frame of the target video, wherein the third frame is subsequent to the second frame； and the scale estimation unit is further configured for  estimating, with the updated scale estimation network, a third scale of the object in the third frame of the target video.
Based on the proposed CNN adaption method and system, pre-trained deep features can be effectively transferred for online application with a reduced over-fitting. The proposed object online tracking method, apparatus and system are constructed based on the proposed CNN adaption method, apparatus and system. Due to the reduced over-fitting, the proposed object online tracking method and system can perform an improved object online tracking.
Brief Description of the Drawing
Exemplary non-limiting embodiments of the present application are described below with reference to the attached drawings. The drawings are illustrative and generally not to an exact scale. The same or similar elements on different figures are referenced with the same reference numbers.
Fig. 1 illustrates a conventional method for adapting a pre-trained CNN to a target image.
Figs. 2a-2c illustrate an adaptive CNN for adapting a pre-trained CNN to a target video, according to an embodiment of the present application.
Fig. 3 is a flowchart illustrating the initialization of the adaptive CNN, according to an embodiment of the present application.
Fig. 4 is a flowchart illustrating the process of object online tracking, according to an embodiment of the present application.
Fig. 5 is a schematic diagram illustrating a system architecture for the object online tracking, according to an embodiment of the present application.
Detailed Description
Reference will now be made in detail to some specific embodiments of the invention including the best modes contemplated by the inventors for carrying out the invention. Examples of these specific embodiments are illustrated in the accompanying drawings. While the invention is described in conjunction with these specific embodiments, it will be appreciated by one skilled in the art that it is not intended to limit the invention to the described embodiments. On the contrary, it is intended to cover alternatives, modifications, and equivalents as may be included within the spirit and scope of the invention as defined by the appended claims. In the following description, numerous specific details are set forth in order to provide a thorough understanding of the present application. The present application may be practiced without some or all of these specific details. In other instances, well-known process operations have not been described in detail in order not to unnecessarily obscure the present application.
The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention. As used herein, the singular forms "a" , "an" and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will be further understood that the terms "comprises" and/or "comprising" when used in this specification, specify the presence of stated features, integers, steps, operations, elements, and/or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and/or groups thereof.
For online applications, one simple approach to transfer offline pre-trained CNN features is to add one or more randomly initialized CNN layers, i.e., adaptive CNN, sequence to the pre-trained CNN model. Then the parameters, i.e. convolutional kernels and bias, of the pre-trained CNN is fixed while only the parameters of the adaptative CNN is trained online to fit the current task, for example, a target video or a target image. Fig. 1 illustrates such a conventional method 1000 for adapting a pre-trained CNN 1100 to a target image. As can be  seen from Fig. 1, an adaptive CNN 1200 is disposed sequence to the pre-trained CNN 1100 and configured for refining the pre-trained features to a final adapted feature 1300 for supervision. However, as the parameters of the adaptive CNN 1200 are jointly learned during a training process, this transfer learning method suffers from severe over-fitting for an online application, where training sample with ground truth label is only provided in the first frame. The online learned parameters mainly focus on recent training samples and are less likely to well generalize to both historical and future samples. This phenomenon can be fatal to online visual tracking where the target often undergoes significant appearance changes or heavy occlusion.
To tackle the above issue, an adaptive CNN 2200 is proposed for better transferring pre-trained deep features, as shown in Fig. 2a to Fig. 2c. The pre-trained CNN 2100 is denoted as CNN-E, and an RGB image is taken as input and a convolution feature map X is output. The online adaptive CNN 2200, denoted as CNN-A, is randomly initialized and consists of two convolution layers interleaved with a ReLU layer as the nonlinear activation unit. The first convolution layer is linked to the pre-trained CNN 2100 and configured to transform the first feature map into the plurality of sub-feature maps. The second convolution layer is linked to the first convolution layer and configured to convolve each of the sub-feature maps with one of the adaptive convolution kernels, respectively. The online adaptive CNN 2200 takes the feature map X as input and generates the final feature map 2300, which is formulated by:
Figure PCTCN2016073184-appb-000001
where
Figure PCTCN2016073184-appb-000002
indicates the c-th channel of the sub-feature map generated by the second convolution layer with spatial size of m×n. The sub-feature map in the second layer is obtained by convolving the kernel with the sub-feature map in the first layer as:
Figure PCTCN2016073184-appb-000003
where C1 denotes the number of channels of the sub-feature map output by the first convolution layer； 
Figure PCTCN2016073184-appb-000004
denotes the convolution kernel connecting the k-th channel of the first convolution layer sub-feature map with the c-th channel of the second convolution layer sub-feature map； bc is the bias and the symbol "*" denotes a convolution operation. The summation is conducted over all the channels.
In order to introduce randomness into the parameter learning process, the output sub-feature map is regarded as an set of base learners, formulated by:
Figure PCTCN2016073184-appb-000005
where each base leaner is defined as:
Figure PCTCN2016073184-appb-000006
and the parameter
Figure PCTCN2016073184-appb-000007
indicates the corresponding kernel weights and bias in both the first and second convolution layers of CNN-A.
The online training of the CNN-Anetwork is then equivalent to online training each base learner and to sequentially sample a well-trained parameter for each of base learners into an ensemble set. Since the proposed online training method is conducted independently in each channel of the output sub-feature map, in the following discussion, only one output channel will be discussed as an example to describe the training method. For notification simplicity, the superscript channel number is omitted and the notation {γk|k＝1， 2， ...， C1} is used to denote the parameters of the base learners for any one output sub-feature map channel.
Fig. 3 shows how the online training process is initialized. At the beginning of the online training process, a first training sample is prepared according to a first frame of the target video at step S301. The strategy for preparing a training sample from a given image, such as a stochastic gradient descent (SGD) , is well-known to one skilled in the art and thus will not be discussed in detail hereinafter. The first training sample is then fed forward, at step S302, through the pre-trained CNN and the adaptive CNN to generate a first output image. Then at step S303, the first output image is compared with the first ground truth, which is derived from the first frame, to obtain a plurality of first training errors for the plurality of adaptive convolution kernels, respectively. The first training errors are back-propagated, through the pre-trained CNN and the adaptive CNN to train the adaptive convolution kernels in an iterative manner until the first training errors converge, as shown at step S304.
In one implement, a plurality of parameters are trained for each of the adaptive convolution kernels, respectively, after the initialization. The parameter γ* with the smallest training error is selected and grouped into an ensemble set ε and the rest of trained parameters are grouped into a candidate set c. The parameters in the candidate set will be optimized in following frames of the target video. However, in an alternative implement, two or more parameters with smallest training errors may be selected into the ensemble set.
In the following training process, i.e., an optimizing process, the parameters in the candidate set are sequentially added to the ensemble set in a similar manner. Since the optimizing process is similar to that of the initialization, such as the preparing, the feeding, the comparing and the back-propagating, only the difference will be discussed hereinafter.
In the optimizing process, all the parameters in the ensemble set are used to form an ensemble with output
Figure PCTCN2016073184-appb-000008
for online testing. At the t-th step,  a newly training sample Xt with target output Yt is obtained. The parameters in the ensemble set ε is jointly refined, for example by SGD with the loss function Lε＝L (Yt， F (Xt； ε) ) .Meanwhile, each parameter γj∈c is refined independently, for example by SGD using with following loss function:
Lc (Yt， f (Xt； γj) ) ＝L (Yt， f (Xt； γj) +ηF (Xt； ε) )     (5)
where F (Xt； ε) is fixed and the parameter η is used to balance the impact of the ensemble on the candidate parameters, such that the refining of the parameter γj∈c considers both the target output Yt and the output of the ensemble F (Xt； ε) . If the training error Lε is higher than a predefined threshold and the candidate set C is not empty, a refined parameter is sampled from the candidate set C, for example, according to the following sampling probability density
p (γ) ＝q (Lc (Yt， f (Xt； γ) ) ) ， γ∈c     (6)
where q (·) is a monotonically decreasing function. Thus the sampled parameter is removed from the candidate set C and added into the ensemble set ε. The above online training approach is conducted sequentially in each time step. When all the parameters are sampled from the candidate set to the ensemble set, the ensemble F (X； ε) evolves into a well-trained CNN model. In an alternate implement, the parameters incorporated in the well-trained CNN model would still be jointly updated with a further training process during the period of the subsequent frames. The proposed adaptive CNN demonstrate a moderate diversity since the parameters thereof are trained independently, especially by using different loss criterions.
In one embodiment, a mask layer may be contained in the adaptive CNN and linked  to the second convolution layer to further reduce a correlation among the sub-feature maps. Specifically, each channel of the output sub-feature map from the second convolution layer is associated with an individual binary mask which has the same spatial size with the sub-feature map. All the masks are initialized in a random manner and then fixed throughout the online training process. The forward propagation of the convolution layer at the training stage is then conducted as
Figure PCTCN2016073184-appb-000009
where Xk indicates the k-th channel of the sub-feature map； Mc denotes the binary mask associated with the c-th channel of the output feature map Fc； and the symbol "⊙" denotes a Hadamard product. Accordingly, the backward propagation is also conducted by considering the binary masks. Trained in this way, the learned convolution kernels are enforced to focus on different part of the input feature maps through the binary masks.
In another aspect, a method is proposed for an object online tracking. Fig. 4 schematically illustrates a general flowchart for an object online tracking. At the beginning of the tracking process, as shown at step S401, a target object is manually selected with a target boundary box in the first frame of the video, and accordingly, a region of interest (ROI) is determined. In one implement, the ROI is centered at the target object. In a further implement, the ROI may have twice the size of the target boundary box, as an example. At step S402, the ROI is fed forward a pre-trained CNN to extract an initial feature map for initialization. The initial feature map comprises the information of the location and the scale of the target object in the first frame. At step S403, an adaptive CNN and a scale estimation network is initialized with the initial feature map, wherein adaptive CNN is utilized to predict the location of the target object and the scale estimation network is utilized to estimate the scale of the target object. The scale estimation process will be further described in the following paragraphs. With the initialized adaptive CNN and the scale estimation network, the location and scale of the target object is predicted and estimated in a following  frame, for example, a second frame of the target video, as shown at step S404 and S405. Besides, the process of location prediction and scale estimation is also a training process, which gives a plurality of optimized parameters for the adaptive CNN and the scale estimation network, respectively. With the optimized parameters, the adaptive CNN and the scale estimation network are jointly updated at step S406. After updating process, both the adaptive CNN and the scale estimation network have better adaptabilities with respect to the relating frame. The location and scale of target object in the subsequent frames of the target video will be predicted and estimated based on the updated adaptive CNN (step S407 and S408) . The result of the prediction and estimation further updates, in turn, the adaptive CNN and the scale estimation network.
The above-mentioned object online tracking process may be implemented in a system illustrated in Fig. 5. A feature extraction unit 501 comprising a pre-trained CNN, i.e., CNN-E, is configured to determine a ROI and to extract the feature of any frame. The extracted feature is transmitted to a location prediction unit 502 comprising the adaptive CNN and a scale estimation unit 503 comprising the scale estimation network. In addition, the extracted feature is also transmitted to an initialization and update unit 504 for initializing at the first frame of the target video. In subsequent frames, the extracted feature map of the current frame and the output from the location prediction unit 502 and scale estimation unit 503 are transmitted to the initialization and update unit 504 to update the adaptive CNN and the scale estimation network.
The structure, the initializing and the optimizing of the adaptive CNN is previously discussed and will not be further detailed hereinafter. In the case of object online tracking, the adaptive CNN is specifically used to transform, by performing a heat map regression, the feature map extracted from the pre-trained CNN into a target heat map. The location of the target object is then determined by the location on the heat map with the maximum value and the corresponding maximum heat map value serves as the confidence of this prediction. In one embodiment, the updating of the adaptive CNN and the scale estimation network are  conducted only if the confidence is higher than a pre-defined threshold, in order to avoid updating using contaminated training sample.
The scale estimation network has a conventional deep learning network structure, such as a CNN or a fully-connected network, and thus would not be described in detail herein. In the case of object online tracking, the scale prediction unit receives the feature map extracted from the pre-trained network and applies a set of pre-defined scale transformations to obtain the corresponding scale-transformed feature maps. The scale-transformed feature maps are fed forward through the scale estimation network, which assigns a score for each scale transformation. The scale with the highest score is then predicted as the current scale of the target. With the location and the scale resulted from the tracking system, the target object will be tracked at an improved precision.
As will be appreciated by one skilled in the art, the present application may be embodied as a system, a method or a computer program product. Accordingly, the present application may take the form of an entirely hardware embodiment and hardware aspects that may all generally be referred to herein as a "unit" , "circuit" , "module" , or "system" . Much of the inventive functionality and many of the inventive principles when implemented, are best supported with or integrated circuits (ICs) , such as a digital signal processor and software therefore or application specific ICs. It is expected that one of ordinary skill, notwithstanding possibly significant effort and many design choices motivated by, for example, available time, current technology, and economic considerations, when guided by the concepts and principles disclosed herein will be readily capable of generating ICs with minimal experimentation. Therefore, in the interest of brevity and minimization of any risk of obscuring the principles and concepts according to the present application, further discussion of such software and ICs, if any, will be limited to the essentials with respect to the principles and concepts used by the preferred embodiments. In addition, the present invention may take the form of an entirely software embodiment (including firmware, resident software, micro-code, etc. ) or an embodiment combining software. For example, the system may comprise a memory that  stores executable components and a processor, electrically coupled to the memory to execute the executable components to perform operations of the system, as discussed in reference to Figs. 1-6. Furthermore, the present invention may take the form of a computer program product embodied in any tangible medium of expression having computer-usable program code embodied in the medium.
Although the preferred examples of the present application have been described, those skilled in the art can make variations or modifications to these examples upon knowing the basic inventive concept. The appended claims are intended to be considered as comprising the preferred examples and all the variations or modifications fell into the scope of the present application.
Obviously, those skilled in the art can make variations or modifications to the present application without departing the spirit and scope of the present application. As such, if these variations or modifications belong to the scope of the claims and equivalent technique, they may also fall into the scope of the present application.

Claims (51)

  1. A method for adapting a pre-trained CNN to a target video, comprising:
    transforming a first feature map into a plurality of sub-feature maps, wherein the first feature map is generated by the pre-trained CNN according to a frame of the target video；
    convolving each of the sub-feature maps with one of a plurality of adaptive convolution kernels, respectively, to output a plurality of second feature maps with improved adaptability； and
    training, frame by frame, the adaptive convolution kernels.
  2. The method of claim 1, wherein the transforming and the convolving are implemented in an adaptive CNN comprising:
    a first convolution layer, linked to the pre-trained CNN and configured to transform the first feature map into the plurality of sub-feature maps； and
    a second convolution layer, linked to the first convolution layer and configured to convolve each of the sub-feature maps with one of the adaptive convolution kernels, respectively.
  3. The method of claim 2, wherein the training comprises:
    feeding a first training sample forward through the pre-trained CNN and the adaptive CNN to generate a first output image, wherein the first training sample is obtained according to a first frame of the target video；
    comparing the generated first output image with a first ground truth derived from the first frame to obtain a plurality of first training errors for the adaptive convolution kernels, respectively；
    back-propagating repeatedly the first training errors through the pre-trained CNN and the adaptive CNN to train the adaptive convolution kernels until the first training errors converge,  wherein a plurality of parameters are obtained for the trained adaptive convolution kernels, respectively；
    grouping a parameter of the obtained parameters, which has a smallest first training error, and a rest of the obtained parameters into an ensemble set and a candidate set, respectively； and
    optimizing, according to a subsequent frame of the target video, the parameters grouped in the candidate set.
  4. The method of claim 3, wherein the optimizing comprises:
    feeding a second training sample forward through the pre-trained CNN and the adaptive CNN to generate a second output image, wherein the second training sample is obtained according to a second frame of the target video and the second frame is subsequent to the firs frame；
    comparing the second output image with a second ground truth derived from the second frame to obtain a plurality of second training errors for the plurality of adaptive convolution kernels； and
    if any of the second training errors is higher than a threshold,
    back-propagating the second training errors through the pre-trained CNN and the adaptive CNN to further refine the parameters in the candidate set； and
    transferring at least one of the further refined parameters to the ensemble set.
  5. The method of claim 1, wherein each of the adaptive convolution kernels is trained under a different loss criterion.
  6. The method of claim 2, wherein the method further comprises:
    reducing, by a mask layer, a correlation among the sub-feature maps, wherein the mask layer is linked to the second convolution layer of the adaptive CNN.
  7. The method of claim 6, wherein the mask layer comprises a plurality of binary masks, each of which is convolved with one of the sub-feature maps and has a same spatial size with the convolved sub-feature map.
  8. A method for an object online tracking, comprising:
    determining a region of interest (ROI) in a first frame of a target video；
    feeding the determined ROI forward through a pre-trained CNN to extract an initial feature map thereof；
    initializing, with the initial feature map, an adaptive CNN used for detecting a location of the object and a scale estimation network used for defining a scale of the object；
    predicting, with the initialized adaptive CNN, a second location of the object in a second frame of the target video, wherein the second frame is subsequent to the first frame；
    estimating, with the initialized scale estimation network, a second scale of the object in the second frame of the target video；
    updating, with optimized network parameters acquired in the predicting and the estimating, the adaptive CNN and the scale estimation network, respectively；
    predicting, with the updated adaptive CNN, a third location of the object in a third frame of the target video, wherein the third frame is subsequent to the second frame； and
    estimating, with the updated scale estimation network, a third scale of the object in the third frame of the target video.
  9. The method of claim 8, wherein the adaptive CNN comprises:
    a first convolution layer, linked to the pre-trained CNN and configured to transform a first feature map into a plurality of sub-feature maps, wherein the first feature map is generated by the pre-trained CNN according to any frame of the target video； and
    a second convolution layer, linked to the first convolution layer and configured to convolve each of the sub-feature maps with one of a plurality of adaptive convolution kernels , respectively, to output a plurality of second feature maps with improved adaptability.
  10. The method of claim 9, wherein the adaptive CNN is initialized by:
    feeding a first training sample forward through the pre-trained CNN and the adaptive CNN to generate a first output image, wherein the first training sample is obtained according to a first frame of the target video；
    comparing the generated first output image with a first ground truth derived from the first frame to obtain a plurality of first training errors for the adaptive convolution kernels, respectively；
    back-propagating repeatedly the first training errors through the pre-trained CNN and the adaptive CNN to train the adaptive convolution kernels until the first training errors converges, wherein a plurality of parameters are obtained for the trained adaptive convolution kernels, respectively；
    grouping a parameter of the obtained parameters, which has a smallest first training error, and a rest of the obtained parameters into an ensemble set and a candidate set, respectively.
  11. The method of claim 9, wherein the adaptive CNN is updated by:
    feeding the second training sample forward through the pre-trained CNN and the adaptive CNN to generate a second output image, wherein the second training sample is obtained according to a second frame of the target video and the second frame is subsequent to the first frame；
    comparing the second output image with a second ground truth derived from the second frame to obtain a plurality of second training errors for the plurality of adaptive convolution kernels, respectively； and
    if any of the second training errors is higher than a threshold,
    back-propagating the second training errors through the pre-trained CNN and the adaptive CNN to further refine the parameters in the candidate set； and
    transferring at least one of the further refined parameters to the ensemble set.
  12. The method of claim 9, wherein each of the adaptive convolution kernels is trained  under a different loss criterion.
  13. The method of claim 9, wherein the adaptive CNN further comprises a mask layer linked to the second convolution layer to further reduce a correlation among the sub-feature maps.
  14. The method of claim 13, wherein the mask layer comprises a plurality of binary masks, each of which is convolved with one of the sub-feature maps and has a same spatial size with the convolved sub-feature map.
  15. The method of claim 9, wherein the location of the object is predicted by a heat map generated by the adaptive CNN, wherein a location with a maximum value is predicted to be the location of the object and the maximum value is sampled as a confidence.
  16. The method of claim 15, wherein the updating is performed only if the confidence is higher than a pre-defined threshold.
  17. The method of claim 8, wherein the ROI is centered at an object to be tracked.
  18. A system for adapting a pre-trained CNN to a target video, comprising:
    a memory that stores executable components； and
    a processor electrically coupled to the memory to execute the executable components for:
    transforming a first feature map into a plurality of sub-feature maps, wherein the first feature map is generated by the pre-trained CNN according to a frame of the target video；
    convolving each of the sub-feature maps with one of a plurality of adaptive convolution kernels, respectively, to output a plurality of second feature maps with improved adaptability； and
    training, frame by frame, the adaptive convolution kernels.
  19. The system of claim 18, wherein the executable components comprises an adaptive CNN comprising:
    a first convolution layer, linked to the pre-trained CNN and configured to transform the first feature map into the plurality of sub-feature maps； and
    a second convolution layer, linked to the first convolution layer and configured to convolve each of the sub-feature maps with one of the adaptive convolution kernels, respectively.
  20. The system of claim 19, wherein the training comprises:
    feeding a first training sample forward through the pre-trained CNN and the adaptive CNN to generate a first output image, wherein the first training sample is obtained according to a first frame of the target video；
    comparing the generated first output image with a first ground truth derived from the first frame to obtain a plurality of first training errors for the adaptive convolution kernels, respectively；
    back-propagating repeatedly the first training errors through the pre-trained CNN and the adaptive CNN to train the adaptive convolution kernels until the first training errors converge, wherein a plurality of parameters are obtained for the trained adaptive convolution kernels, respectively；
    grouping a parameter of the obtained parameters, which has a smallest first training error, and a rest of the obtained parameters into an ensemble set and a candidate set, respectively； and
    optimizing, according to a subsequent frame of the target video, the parameters grouped in the candidate set.
  21. The system of claim 20, wherein the optimizing comprises:
    feeding a second training sample forward through the pre-trained CNN and the adaptive  CNN to generate a second output image, wherein the second training sample is obtained according to a second frame of the target video and the second frame is subsequent to the first frame；
    comparing the second output image with a second ground truth derived from the second frame to obtain a plurality of second training errors for the plurality of adaptive convolution kernels； and
    if any of the second training errors is higher than a threshold,
    back-propagating the second training errors through the pre-trained CNN and the adaptive CNN to further refine the parameters in the candidate set； and
    transferring at least one of the further refined parameters to the ensemble set.
  22. The system of claim 18, wherein each of the adaptive convolution kernels is trained under a different loss criterion.
  23. The system of claim 19, wherein the adaptive CNN further comprises a mask layer linked to the second convolution layer to further reduce a correlation among the sub-feature maps.
  24. The system of claim 23, wherein the mask layer comprises a plurality of binary masks, each of which is convolved with one of the sub-feature maps and has a same spatial size with the convolved sub-feature map.
  25. A system for an object online tracking, comprising:
    a memory that stores executable components； and
    a processor electrically coupled to the memory to execute the executable components for:
    determining a region of interest (ROI) in a first frame of a target video；
    feeding the determined ROI forward through a pre-trained CNN to extract an initial feature map thereof；
    initializing, with the initial feature map, an adaptive CNN used for detecting a location of the object and a scale estimation network used for defining a scale of the object；
    predicting, with the initialized adaptive CNN, a second location of the object in a second frame of the target video, wherein the second frame is subsequent to the first frame；
    estimating, with the initialized scale estimation network, a second scale of the object in the second frame of the target video；
    updating, with optimized network parameters acquired in the predicting and the estimating, the adaptive CNN and the scale estimation network, respectively；
    predicting, with the updated adaptive CNN, a third location of the object in a third frame of the target video, wherein the third frame is subsequent to the second frame； and
    estimating, with the updated scale estimation network, a third scale of the object in the third frame of the target video.
  26. The system of claim 25, wherein the adaptive CNN comprises:
    a first convolution layer, linked to the pre-trained CNN and configured to transform a first feature map into a plurality of sub-feature maps, wherein the first feature map is generated by the pre-trained CNN according to any frame of the target video； and
    a second convolution layer, linked to the first convolution layer and configured to convolve each of the sub-feature maps with one of a plurality of adaptive convolution kernels , respectively, to output a plurality of second feature maps with improved adaptability.
  27. The system of claim 26, wherein the adaptive CNN is initialized by:
    feeding a first training sample forward through the pre-trained CNN and the adaptive CNN to generate a first output image, wherein the first training sample is obtained according to a first frame of the target video；
    comparing the first output image with a first ground truth derived from the first frame to  obtain a plurality of first training errors for the plurality of adaptive convolution kernels, respectively；
    back-propagating repeatedly the first training errors through the pre-trained CNN and the adaptive CNN to train the adaptive convolution kernels until the first training errors converges, wherein a plurality of parameters are obtained for trained adaptive convolution kernels, respectively；
    grouping a parameter of the obtained parameters, which has a smallest first training error, and a rest of the obtained parameters into an ensemble set and a candidate set, respectively.
  28. The system of claim 26, wherein the adaptive CNN is updated by:
    feeding the second training sample forward through the pre-trained CNN and the adaptive CNN to generate a second output image, wherein the second training sample is obtained according to a second frame of the target video and the second frame is subsequent to the first frame；
    comparing the second output image with a second ground truth derived from the second frame to obtain a plurality of second training errors for the plurality of adaptive convolution kernels, respectively； and
    if any of the second training errors is higher than a threshold,
    back-propagating the second training errors through the pre-trained CNN and the adaptive CNN to further refine the parameters in the candidate set； and
    transferring at least one of the further refined parameters to the ensemble set.
  29. The system of claim 26, wherein each of the adaptive convolution kernels is trained under a different loss criterion.
  30. The system of claim 26, wherein the adaptive CNN further comprises a mask layer linked to the second convolution layer to further reduce a correlation among the sub-feature maps.
  31. The system of claim 30, wherein the mask layer comprises a plurality of binary masks, each of which is convolved with one of the sub-feature maps and has a same spatial size with the convolved sub-feature map.
  32. The system of claim 26, wherein the location of the object is predicted by a heat map generated by the adaptive CNN, wherein a location with a maximum value is predicted to be the location of the object and the maximum value is sampled as a confidence.
  33. The system of claim 32, wherein the updating is performed only if the confidence is higher than a pre-defined threshold.
  34. The system of claim 25, wherein the ROI is centered at an object to be tracked.
  35. An apparatus for adapting a pre-trained CNN to a target video, comprising:
    means for transforming a first feature map into a plurality of sub-feature maps, wherein the first feature map is generated by the pre-trained CNN according to a frame of the target video；
    means for convolving each of the sub-feature maps with one of a plurality of adaptive convolution kernels, respectively, to output a plurality of second feature maps with improved adaptability； and
    means for training, frame by frame, the adaptive convolution kernels.
  36. The apparatus of claim 35, wherein the means for transforming and the means for convolving are organized in an adaptive CNN comprising:
    a first convolution layer, linked to the pre-trained CNN and configured to transform the first feature map into the plurality of sub-feature maps； and
    a second convolution layer, linked to the first convolution layer and configured to convolve each of the sub-feature maps with one of the adaptive convolution kernels,  respectively.
  37. The apparatus of claim 36, wherein the training comprises:
    feeding a first training sample forward through the pre-trained CNN and the adaptive CNN to generate a first output image, wherein the first training sample is obtained according to a first frame of the target video；
    comparing the generated first output image with a first ground truth derived from the first frame to obtain a plurality of first training errors for the adaptive convolution kernels, respectively；
    back-propagating repeatedly the first training errors through the pre-trained CNN and the adaptive CNN to train the adaptive convolution kernels until the first training errors converge, wherein a plurality of parameters are obtained for the trained adaptive convolution kernels, respectively；
    grouping a parameter of the obtained parameters, which has a smallest first training error, and a rest of the obtained parameters into an ensemble set and a candidate set, respectively； and
    optimizing, according to a subsequent frame of the target video, the parameters grouped in the candidate set.
  38. The apparatus of claim 37, wherein the optimizing comprises:
    feeding a second training sample forward through the pre-trained CNN and the adaptive CNN to generate a second output image, wherein the second training sample is obtained according to a second frame of the target video and the second frame is subsequent to the first frame；
    comparing the second output image with a second ground truth derived from the second frame to obtain a plurality of second training errors for the plurality of adaptive convolution kernels； and
    if any of the second training errors is higher than a threshold,
    back-propagating the second training errors through the pre-trained CNN  and the adaptive CNN to further refine the parameters in the candidate set； and
    transferring at least one of the further refined parameters to the ensemble set.
  39. The apparatus of claim 35, wherein each of the adaptive convolution kernels is trained under a different loss criterion.
  40. The apparatus of claim 36, wherein the adaptive CNN further comprises a mask layer linked to the second convolution layer to further reduce a correlation among the sub-feature maps.
  41. The apparatus of claim 40, wherein the mask layer comprises a plurality of binary masks, each of which is convolved with one of the sub-feature maps and has a same spatial size with the convolved sub-feature map.
  42. An apparatus for an object online tracking, comprising:
    a feature extraction unit configured for:
    determining a region of interest (ROI) in a first frame of a target video； and
    feeding the determined ROI forward through a pre-trained CNN to extract an initial feature map thereof；
    an initialization and update unit configured for initializing, with the initial feature map, an adaptive CNN used for detecting a location of the object and a scale estimation network used for defining a scale of the object；
    a location prediction unit configured for predicting, with the initialized adaptive CNN, a second location of the object in a second frame of the target video, wherein the second frame is subsequent to the first frame； and
    a scale estimation unit configured for estimating, with the initialized scale estimation network, a second scale of the object in the second frame of the target video；
    wherein, the initialization and update unit is further configured for updating, with  optimized network parameters acquired in the predicting and the estimating, the adaptive CNN and the scale estimation network, respectively；
    the location prediction unit is further configured for predicting, with the updated adaptive CNN, a third location of the object in a third frame of the target video, wherein the third frame is subsequent to the second frame； and
    the scale estimation unit is further configured for estimating, with the updated scale estimation network, a third scale of the object in the third frame of the target video.
  43. The apparatus of claim 42, wherein the adaptive CNN comprises:
    a first convolution layer linked to the pre-trained CNN and configured to transform a first feature map into a plurality of sub-feature maps, wherein the first feature map is generated by the pre-trained CNN according to any frame of the target video； and
    a second convolution layer linked to the first convolution layer and configured to convolve each of the sub-feature maps with one of a plurality of adaptive convolution kernels , respectively, to output a plurality of second feature maps with improved adaptability.
  44. The apparatus of claim 43, wherein the adaptive CNN is initialized by:
    feeding a first training sample forward through the pre-trained CNN and the adaptive CNN to generate a first output image, wherein the first training sample is obtained according to a first frame of the target video；
    comparing the generated first output image with a first ground truth derived from the first frame to obtain a plurality of first training errors for the adaptive convolution kernels, respectively；
    back-propagating repeatedly the first training errors through the pre-trained CNN and the adaptive CNN to train the adaptive convolution kernels until the first training errors converges, wherein a plurality of parameters are obtained for the trained adaptive convolution kernels, respectively；
    grouping a parameter of the obtained parameters, which has a smallest first training error, and a rest of the obtained parameters into an ensemble set and a candidate set, respectively.
  45. The apparatus of claim 43, wherein the adaptive CNN is updated by:
    feeding a second training sample forward through the pre-trained CNN and the adaptive CNN to generate a second output image, wherein the second training sample is obtained according to a second frame of the target video and the second frame is subsequent to the first frame；
    comparing the second output image with a second ground truth derived from the second frame to obtain a plurality of second training errors for the plurality of adaptive convolution kernels, respectively； and
    if any of the second training errors is higher than a threshold,
    back-propagating the second training errors through the pre-trained CNN and the adaptive CNN to further refine the parameters in the candidate set； and
    transferring at least one of the further refined parameters to the ensemble set.
  46. The apparatus of claim 42, wherein each of the adaptive convolution kernels is trained under a different loss criterion.
  47. The apparatus of claim 43, wherein the adaptive CNN further comprises a mask layer linked to the second convolution layer to further reduce a correlation among the sub-feature maps.
  48. The apparatus of claim 47, wherein the mask layer comprises a plurality of binary masks, each of which is convolved with one of the sub-feature maps and has a same spatial size with the convolved sub-feature map.
  49. The apparatus of claim 43, wherein the location of the object is predicted by a heat map generated by the adaptive CNN, wherein a location with a maximum value is predicted to be the location of the object and the maximum value is sampled as a confidence.
  50. The apparatus of claim 49, wherein the updating is performed only if the confidence is higher than a pre-defined threshold.
  51. The apparatus of claim 42, wherein the ROI is centered at an object to be tracked.
PCT/CN2016/073184 2016-02-02 2016-02-02 Methods and systems for cnn network adaption and object online tracking Ceased WO2017132830A1 (en)

Priority Applications (3)

Application Number Priority Date Filing Date Title
CN201680080165.1A CN108701210B (en) 2016-02-02 2016-02-02 Method and system for CNN network adaptation and object online tracking
PCT/CN2016/073184 WO2017132830A1 (en) 2016-02-02 2016-02-02 Methods and systems for cnn network adaption and object online tracking
US16/052,500 US11521095B2 (en) 2016-02-02 2018-08-01 Methods and systems for CNN network adaption and object online tracking

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
PCT/CN2016/073184 WO2017132830A1 (en) 2016-02-02 2016-02-02 Methods and systems for cnn network adaption and object online tracking

Related Child Applications (1)

Application Number Title Priority Date Filing Date
US16/052,500 Continuation US11521095B2 (en) 2016-02-02 2018-08-01 Methods and systems for CNN network adaption and object online tracking

Publications (1)

Publication Number Publication Date
WO2017132830A1 true WO2017132830A1 (en) 2017-08-10

Family

ID=59499227

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2016/073184 Ceased WO2017132830A1 (en) 2016-02-02 2016-02-02 Methods and systems for cnn network adaption and object online tracking

Country Status (3)

Country Link
US (1) US11521095B2 (en)
CN (1) CN108701210B (en)
WO (1) WO2017132830A1 (en)

Cited By (10)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN108288282A (en) * 2017-12-26 2018-07-17 浙江工业大学 A kind of adaptive features select method for tracking target based on convolutional neural networks
US10147019B2 (en) * 2017-03-20 2018-12-04 Sap Se Small object detection
CN109754412A (en) * 2017-11-07 2019-05-14 北京京东尚科信息技术有限公司 Method for tracking target, target tracker and computer readable storage medium
CN109816689A (en) * 2018-12-18 2019-05-28 昆明理工大学 A kind of motion target tracking method that multilayer convolution feature adaptively merges
WO2019116291A1 (en) * 2017-12-14 2019-06-20 International Business Machines Corporation Fusing sparse kernels to approximate a full kernel of a convolutional neural network
CN109961034A (en) * 2019-03-18 2019-07-02 西安电子科技大学 Video object detection method based on convolutional gated recurrent neural unit
CN109993775A (en) * 2019-04-01 2019-07-09 云南大学 Monotrack method based on feature compensation
CN111684470A (en) * 2018-02-08 2020-09-18 三星电子株式会社 Dynamic Memory Map for Neural Networks
CN113302649A (en) * 2018-10-16 2021-08-24 香港中文大学 Method, device and system for automatic diagnosis
CN113424203A (en) * 2019-05-16 2021-09-21 三星电子株式会社 Image processing apparatus and method of operating the same

Families Citing this family (39)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US10769453B2 (en) * 2017-05-16 2020-09-08 Samsung Electronics Co., Ltd. Electronic device and method of controlling operation of vehicle
EP3311364B1 (en) * 2015-09-30 2018-12-26 FotoNation Limited A method and system for tracking an object
WO2018033137A1 (en) * 2016-08-19 2018-02-22 北京市商汤科技开发有限公司 Method, apparatus, and electronic device for displaying service object in video image
KR102824640B1 (en) * 2016-09-07 2025-06-25 삼성전자주식회사 Recognition apparatus based on neural network and training method of neural network
US10699184B2 (en) * 2016-12-29 2020-06-30 Facebook, Inc. Updating predictions for a deep-learning model
WO2018191555A1 (en) * 2017-04-14 2018-10-18 Drishti Technologies. Inc Deep learning system for real time analysis of manufacturing operations
US11164071B2 (en) * 2017-04-18 2021-11-02 Samsung Electronics Co., Ltd. Method and apparatus for reducing computational complexity of convolutional neural networks
US10803323B2 (en) * 2017-05-16 2020-10-13 Samsung Electronics Co., Ltd. Electronic device and method of detecting driving event of vehicle
US10783393B2 (en) 2017-06-20 2020-09-22 Nvidia Corporation Semi-supervised learning for landmark localization
US10977755B2 (en) * 2017-11-21 2021-04-13 International Business Machines Corporation Cognitive screening for prohibited items across multiple checkpoints by using context aware spatio-temporal analysis
US10762662B2 (en) * 2018-03-14 2020-09-01 Tata Consultancy Services Limited Context based position estimation of target of interest in videos
US11019355B2 (en) * 2018-04-03 2021-05-25 Electronics And Telecommunications Research Institute Inter-prediction method and apparatus using reference frame generated based on deep learning
US10565475B2 (en) * 2018-04-24 2020-02-18 Accenture Global Solutions Limited Generating a machine learning model for objects based on augmenting the objects with physical properties
US20190354850A1 (en) * 2018-05-17 2019-11-21 International Business Machines Corporation Identifying transfer models for machine learning tasks
US10853726B2 (en) * 2018-05-29 2020-12-01 Google Llc Neural architecture search for dense image prediction tasks
US10769485B2 (en) * 2018-06-19 2020-09-08 Himax Technologies Limited Framebuffer-less system and method of convolutional neural network
CN108898086B (en) * 2018-06-20 2023-05-26 腾讯科技（深圳）有限公司 Video image processing method and device, computer readable medium and electronic equipment
KR102695519B1 (en) * 2018-07-02 2024-08-14 삼성전자주식회사 Method and device to build image model
US11055854B2 (en) * 2018-08-23 2021-07-06 Seoul National University R&Db Foundation Method and system for real-time target tracking based on deep learning
KR102813368B1 (en) * 2018-12-03 2025-05-27 삼성전자주식회사 Method and apparatus for performing dilated convolution operation in neural network
KR102697346B1 (en) * 2018-12-20 2024-08-21 삼성전자주식회사 Electronic device and operating method for recognizing an object in a image
US10346693B1 (en) * 2019-01-22 2019-07-09 StradVision, Inc. Method and device for attention-based lane detection without post-processing by using lane mask and testing method and testing device using the same
CN110110599B (en) * 2019-04-03 2023-05-09 天津大学 Remote sensing image target detection method based on multi-scale feature fusion
CN111797855B (en) * 2019-04-09 2024-08-13 腾讯科技（深圳）有限公司 Image processing, model training methods, devices, media and equipment
CN110163188B (en) * 2019-06-10 2023-08-08 腾讯科技（深圳）有限公司 Video processing and method, device and device for embedding target object in video
US12271800B2 (en) * 2019-11-14 2025-04-08 Qualcomm Incorporated Conditional computation for continual learning
US12437190B2 (en) 2019-12-05 2025-10-07 International Business Machines Corporation Automated fine-tuning of a pre-trained neural network for transfer learning
CN111294512A (en) * 2020-02-10 2020-06-16 深圳市铂岩科技有限公司 Image processing method, device, storage medium, and imaging device
US11551445B2 (en) 2020-08-14 2023-01-10 Sony Corporation Heatmap visualization of object detections
CN112053386B (en) * 2020-08-31 2023-04-18 西安电子科技大学 Target tracking method based on depth convolution characteristic self-adaptive integration
CN112084952B (en) * 2020-09-10 2023-08-15 湖南大学 Video point location tracking method based on self-supervision training
CN112085779B (en) * 2020-09-21 2024-03-22 上海海事大学 Wave parameter estimation method and device
US12061991B2 (en) * 2020-09-23 2024-08-13 International Business Machines Corporation Transfer learning with machine learning systems
US11602132B2 (en) 2020-10-06 2023-03-14 Sixgill, LLC System and method of counting livestock
CN112966811B (en) * 2021-02-04 2023-04-14 北京邮电大学 The solution method and network of task conflict in MTL convolutional neural network
CN113762316B (en) * 2021-02-05 2025-10-28 北京沃东天骏信息技术有限公司 Attribute value prediction method, device, computer system and readable storage medium
US11568543B2 (en) 2021-03-10 2023-01-31 Western Digital Technologies, Inc. Attention masks in neural network video processing
CN113674844A (en) * 2021-08-19 2021-11-19 浙江远图互联科技股份有限公司 Hospital outpatient flow prediction and triage system based on multi-head CNN network
CN118194924B (en) * 2023-12-18 2025-03-21 贵州民族大学 An adaptive fine-tuning method and system based on contribution measurement

Citations (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN102054170A (en) * 2011-01-19 2011-05-11 中国科学院自动化研究所 Visual tracking method based on minimized upper bound error
US8582807B2 (en) * 2010-03-15 2013-11-12 Nec Laboratories America, Inc. Systems and methods for determining personal characteristics

Family Cites Families (16)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US7747070B2 (en) * 2005-08-31 2010-06-29 Microsoft Corporation Training convolutional neural networks on graphics processing units
US9025868B2 (en) * 2013-02-27 2015-05-05 Sony Corporation Method and system for image processing to determine a region of interest
US9400925B2 (en) * 2013-11-15 2016-07-26 Facebook, Inc. Pose-aligned networks for deep attribute modeling
US9430829B2 (en) * 2014-01-30 2016-08-30 Case Western Reserve University Automatic detection of mitosis using handcrafted and convolutional neural network features
CN104200224A (en) * 2014-08-28 2014-12-10 西北工业大学 Valueless image removing method based on deep convolutional neural networks
CN104574445B (en) * 2015-01-23 2015-10-14 北京航空航天大学 A kind of method for tracking target
US10860837B2 (en) * 2015-07-20 2020-12-08 University Of Maryland, College Park Deep multi-task learning framework for face detection, landmark localization, pose estimation, and gender recognition
CN105160310A (en) * 2015-08-25 2015-12-16 西安电子科技大学 3D (three-dimensional) convolutional neural network based human body behavior recognition method
US10019631B2 (en) * 2015-11-05 2018-07-10 Qualcomm Incorporated Adapting to appearance variations when tracking a target object in video sequence
US10181195B2 (en) * 2015-12-28 2019-01-15 Facebook, Inc. Systems and methods for determining optical flow
US10049279B2 (en) * 2016-03-11 2018-08-14 Qualcomm Incorporated Recurrent networks with motion-based attention for video understanding
US9836853B1 (en) * 2016-09-06 2017-12-05 Gopro, Inc. Three-dimensional convolutional neural networks for video highlight detection
CN108229455B (en) * 2017-02-23 2020-10-16 北京市商汤科技开发有限公司 Object detection method, neural network training method and device and electronic equipment
US10679351B2 (en) * 2017-08-18 2020-06-09 Samsung Electronics Co., Ltd. System and method for semantic segmentation of images
US10402978B1 (en) * 2019-01-25 2019-09-03 StradVision, Inc. Method for detecting pseudo-3D bounding box based on CNN capable of converting modes according to poses of objects using instance segmentation and device using the same
CN111967305B (en) * 2020-07-01 2022-03-18 华南理工大学 Real-time multi-scale target detection method based on lightweight convolutional neural network

Patent Citations (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US8582807B2 (en) * 2010-03-15 2013-11-12 Nec Laboratories America, Inc. Systems and methods for determining personal characteristics
CN102054170A (en) * 2011-01-19 2011-05-11 中国科学院自动化研究所 Visual tracking method based on minimized upper bound error

Non-Patent Citations (2)

* Cited by examiner, † Cited by third party
Title
LI HUANYU ET AL.: "Research on Visual Tracking Algorithm Based on Deep Feature Expression and Learning", JOURNAL OF ELECTRONICS & INFORMATION TECHNOLOGY, vol. 37, no. 9, 30 September 2015 (2015-09-30), XP055404234, DOI: doi:10.11999/JEIT150031 *
WANG LIJUN ET AL.: "Visual Tracking with Fully Convolutional Networks", IEEE INTERNATIONAL CONFERENCE ON COMPUTER VISION, 31 December 2015 (2015-12-31), XP032866662 *

Cited By (17)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US10147019B2 (en) * 2017-03-20 2018-12-04 Sap Se Small object detection
CN109754412B (en) * 2017-11-07 2021-10-01 北京京东乾石科技有限公司 Target tracking method, target tracking device, and computer-readable storage medium
CN109754412A (en) * 2017-11-07 2019-05-14 北京京东尚科信息技术有限公司 Method for tracking target, target tracker and computer readable storage medium
WO2019116291A1 (en) * 2017-12-14 2019-06-20 International Business Machines Corporation Fusing sparse kernels to approximate a full kernel of a convolutional neural network
US10740659B2 (en) 2017-12-14 2020-08-11 International Business Machines Corporation Fusing sparse kernels to approximate a full kernel of a convolutional neural network
GB2583623A (en) * 2017-12-14 2020-11-04 Ibm Fusing sparse kernels to approximate a full kernel of a convolutional neural network
CN108288282B (en) * 2017-12-26 2022-04-08 浙江工业大学 An adaptive feature selection target tracking method based on convolutional neural network
CN108288282A (en) * 2017-12-26 2018-07-17 浙江工业大学 A kind of adaptive features select method for tracking target based on convolutional neural networks
CN111684470A (en) * 2018-02-08 2020-09-18 三星电子株式会社 Dynamic Memory Map for Neural Networks
CN111684470B (en) * 2018-02-08 2023-10-24 三星电子株式会社 Dynamic memory mapping for neural networks
CN113302649A (en) * 2018-10-16 2021-08-24 香港中文大学 Method, device and system for automatic diagnosis
CN109816689A (en) * 2018-12-18 2019-05-28 昆明理工大学 A kind of motion target tracking method that multilayer convolution feature adaptively merges
CN109961034A (en) * 2019-03-18 2019-07-02 西安电子科技大学 Video object detection method based on convolutional gated recurrent neural unit
CN109961034B (en) * 2019-03-18 2022-12-06 西安电子科技大学 Video Object Detection Method Based on Convolution Gated Recurrent Neural Unit
CN109993775A (en) * 2019-04-01 2019-07-09 云南大学 Monotrack method based on feature compensation
CN109993775B (en) * 2019-04-01 2023-03-21 云南大学 Single target tracking method based on characteristic compensation
CN113424203A (en) * 2019-05-16 2021-09-21 三星电子株式会社 Image processing apparatus and method of operating the same

Also Published As

Publication number Publication date
US11521095B2 (en) 2022-12-06
US20180341872A1 (en) 2018-11-29
CN108701210A (en) 2018-10-23
CN108701210B (en) 2021-08-17

Similar Documents

Publication Publication Date Title
US11521095B2 (en) Methods and systems for CNN network adaption and object online tracking
US11776236B2 (en) Unsupervised representation learning with contrastive prototypes
US10958869B1 (en) System, device and method for video frame interpolation using a structured neural network
US12175347B2 (en) Neural trees
WO2020140422A1 (en) Neural network for automatically tagging input image, computer-implemented method for automatically tagging input image, apparatus for automatically tagging input image, and computer-program product
EP4244762A1 (en) A temporal bottleneck attention architecture for video action recognition
CN110956185A (en) A method for detecting salient objects in images
CN116670687A (en) Method and system for adapting a trained object detection model to domain shift
CN110942142B (en) Neural network training and face detection method, device, equipment and storage medium
US12217391B2 (en) Spatially adaptive image filtering
CN114298179A (en) Data processing method, device and equipment
Mostayed et al. Content-adaptive u-net architecture for medical image segmentation
CN113780331B (en) Training method, classification method and system implemented by computer and computer readable recording medium
Li et al. Integrating global context contrast and local sensitivity for blind image quality assessment
CN117636051A (en) Unsupervised domain adaptive image classification method based on self-training contrastive learning
Schanz et al. Robust detection of marine life with label-free image feature learning and probability calibration
US20240152787A1 (en) Method and system for learning models for a mixture of domains (mod)
US20230252651A1 (en) Synergistic Object Tracking and Pattern Recognition for Event Representation
EP4695778A1 (en) Performing image processing tasks based on demonstration examples
Dias et al. Probabilistic semantic segmentation refinement by monte carlo region growing
CN114648094B (en) Machine learning devices and methods
Huynh et al. Online training for single hidden-layer feedforward neural networks using RLS-ELM
CN113591913B (en) Picture classification method and device supporting incremental learning
De Alvis et al. Online learning for scene segmentation with laser-constrained CRFs
Chen et al. Efficient convolution-based model pruning with learn gates & BN and channel interdependence learning

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 16888655

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 16888655

Country of ref document: EP

Kind code of ref document: A1