WO2017173605A1 - Method and system for person recognition - Google Patents

Method and system for person recognition Download PDF

Info

Publication number
WO2017173605A1
WO2017173605A1 PCT/CN2016/078549 CN2016078549W WO2017173605A1 WO 2017173605 A1 WO2017173605 A1 WO 2017173605A1 CN 2016078549 W CN2016078549 W CN 2016078549W WO 2017173605 A1 WO2017173605 A1 WO 2017173605A1
Authority
WO
WIPO (PCT)
Prior art keywords
person
layer
roi
identifying
training
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Ceased
Application number
PCT/CN2016/078549
Other languages
French (fr)
Inventor
Xiaogang Wang
Tong Xiao
Shuang Li
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Individual
Original Assignee
Individual
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Individual filed Critical Individual
Priority to PCT/CN2016/078549 priority Critical patent/WO2017173605A1/en
Priority to CN201680084297.1A priority patent/CN109074472B/en
Publication of WO2017173605A1 publication Critical patent/WO2017173605A1/en
Anticipated expiration legal-status Critical
Ceased legal-status Critical Current

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
    • G06V40/10—Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
    • G06V40/103—Static body considered as a whole, e.g. static pedestrian or occupant recognition
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F18/00—Pattern recognition
    • G06F18/20—Analysing
    • G06F18/24—Classification techniques
    • G06F18/241—Classification techniques relating to the classification model, e.g. parametric or non-parametric approaches
    • G06F18/2413—Classification techniques relating to the classification model, e.g. parametric or non-parametric approaches based on distances to training or reference patterns
    • G06F18/24133—Distances to prototypes
    • G06F18/24137—Distances to cluster centroïds
    • G06F18/2414—Smoothing the distance, e.g. radial basis function networks [RBFN]
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/20—Image preprocessing
    • G06V10/25—Determination of region of interest [ROI] or a volume of interest [VOI]
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/40—Extraction of image or video features
    • G06V10/44—Local feature extraction by analysis of parts of the pattern, e.g. by detecting edges, contours, loops, corners, strokes or intersections; Connectivity analysis, e.g. of connected components
    • G06V10/443—Local feature extraction by analysis of parts of the pattern, e.g. by detecting edges, contours, loops, corners, strokes or intersections; Connectivity analysis, e.g. of connected components by matching or filtering
    • G06V10/449—Biologically inspired filters, e.g. difference of Gaussians [DoG] or Gabor filters
    • G06V10/451—Biologically inspired filters, e.g. difference of Gaussians [DoG] or Gabor filters with interaction between the filter responses, e.g. cortical complex cells
    • G06V10/454—Integrating the filters into a hierarchical structure, e.g. convolutional neural networks [CNN]
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
    • G06V10/82—Arrangements for image or video recognition or understanding using pattern recognition or machine learning using neural networks

Definitions

  • the disclosures relate to a method and system for person recognition.
  • Person recognition targets on recognizing a person in images It is a fast growing technical area and has many practical applications in video surveillance and multimedia, for example but not limited to person retrieval, cross-camera visual tracking, and activity analysis. Person recognition is particularly challenging because of complex variations of viewpoints, poses, lighting conditions, occlusions, resolutions, backgrounds and camera settings etc.
  • an end-to-end deep learning method for recognizing a person in an image comprises: extracting, via a feature extracting layer of a Convolutional Neural Network (CNN) , a feature map from an image; cropping, via a ROI proposal layer coupled to the feature extracting layer, a ROI containing a candidate person, from the extracted feature map; and identifying, via a person identifying layer coupled to the ROI proposal layer, the candidate person contained in the cropped ROI, according to a preset image of a target person; wherein the ROI proposal layer and the person identifying layer are trained synchronously.
  • CNN Convolutional Neural Network
  • the cropping comprises pooling the cropped ROI into a feature vector with a fixed length.
  • the person identifying layer further comprises a bounding box regression layer
  • the cropped ROI includes a bounding box overlapping the candidate person
  • the identifying comprises moving the boundary via the bounding box regression layer to enhance the overlapping with the candidate person.
  • the identifying further comprises downsampling the cropped ROI to accelerate the identifying.
  • the person identifying layer comprises a random sampling softmax (RSS) loss layer and the training of the person identifying layer comprises: scoring, according to the preset image of the target person, a plurality of training person images and a background for the training person images in a first training set; selecting, according to scores of the training person images and the background, a subset from the first training set, wherein the subset comprises the target person and the background; constituting a second training set comprising the scores of the person images and the background in the subset; determining, by the RSS loss layer, a loss and a gradient based on the second training set; and back-propagating the determined loss and gradient to adjust parameters of the person identifying layer until the loss and the gradient converge.
  • RSS random sampling softmax
  • a system for recognizing a person in an image comprises: a memory that stores executable components; and a processor electrically coupled to the memory to execute the executable components for: extracting, via a feature extracting layer of a CNN, a feature map from an image; cropping, via a ROI proposal layer coupled to the feature extracting layer, a ROI containing a candidate person, from the extracted feature map; and identifying, via a person identifying layer coupled to the ROI proposal layer, the candidate person contained in the cropped ROI, according to a preset image of a target person; wherein the ROI proposal layer and the person identifying layer are trained synchronously.
  • a system for recognizing a person in an image comprises: a feature extractor configured for extracting a feature map from an image; a ROI generator coupled to the feature extractor and configured for cropping a ROI containing a candidate person, out from the extracted feature map; and a recognizer coupled to the ROI generator and configured for identifying the candidate person contained in the cropped ROI according to a preset image of a target person; wherein the feature extractor is implemented in a feature extracting layer of a CNN, the ROI generator is implemented in a ROI proposal layer of the CNN, and the recognizer is implemented in a person identifying layer of the CNN, and the ROI proposal layer and the person identifying layer are trained synchronously.
  • Fig. 1 illustrates an exemplary CNN for recognizing a person in an image according to one embodiment of the present application
  • Fig. 2 is a flow chart illustrating the process for recognizing a person in an image according to one embodiment of the present application
  • Fig. 3 illustrates an exemplary structure of the ROI proposal layer according to one embodiment of the present application
  • Fig. 4 illustrates an exemplary structure of the person identifying layer according to one embodiment of the present application.
  • Fig. 5 illustrates an exemplary system for recognizing a person in an image according to one embodiment of the present application.
  • An exemplary CNN 1000 for recognizing a person in an image will be described with reference to Fig. 1.
  • An image 1100 comprising a candidate person and a random background is input to a feature extracting layer 1200 of the CNN 1000.
  • the feature extracting layer 1200 constitutes the lowest part of the CNN 1000 and may be implemented in various forms and multiple layers, for example but not limited to, convolutional layers 1-5 of a VGG16 model.
  • a feature map containing semantically meaningful representations is extracted from the image 1100 and transmitted to a ROI proposal layer 1300.
  • the ROI proposal layer 1300 analyzes the extracted feature map to crop a ROI containing a candidate person from the extracted feature map.
  • a feature map or a feature vector representing the ROI generated from the ROI proposal layer 1300 is subsequently transmitted to a person identifying layer 1400, which determines whether the candidate person contained in the copped ROI is the target person preset manually.
  • the ROI proposal layer 1300 cooperates with the person identifying layer 1400 in a high level since they are integrated in a single CNN and may be trained synchronously. As a result, the efficiency and the accuracy of the person recognition are enhanced significantly.
  • a feature map is extracted from an input image at step S2020.
  • the extracted feature map is analyzed by a person detector (i.e., the ROI proposal layer 1300 of Fig. 1) and a ROI containing a candidate person is thus cropped from the extracted feature map at step S2040.
  • the candidate person contained in the cropped ROI is compared, via a recognizer (i.e., the person identifying layer 1300 of Fig. 1) , with a preset target person at step S2060 to accomplish person recognition.
  • a ROI proposal layer 3000 may comprise a convolutional layer 3200, a loss layer 3400, and a ROI pool 3500.
  • the convolutional layer 3200 may perform a nonlinear transformation to a first feature map 3100 supposed to contain a candidate person to generate a second feature map 3300 more suitable to distinguish the candidate person from a background.
  • the second feature map 3300 may be divided into a plurality of anchors indicating a plurality of regions of the second feature map 3300.
  • the loss layer 3400 may comprise an anchor classification layer 3410 and an anchor regression layer 3420.
  • the anchor classification layer 3410 may score each anchor contained in the second feature map 3300 according to their similarity to a human classification, while the anchor regression layer 3420 may calculate, according to the score obtained by the anchor classification layer 3410, a location and a size of a bounding box to enable the calculated bounding box overlapping the candidate person.
  • the calculated bounding box is utilized to crop the feature map 3100 into a ROI.
  • the feature map of the cropped ROI is pooled, by a ROI pooling technique, in a ROI pool 3500, resulting in a feature vector with a fixed length. In such a case, a back-propagating process performed in a training phase may bypass the ROI pool 3500.
  • a person identifying layer 4000 may comprise a first fully connected layer 4200, a second fully connected layer 4300, and a random sampling softmax (RSS) loss layer 4400.
  • the fully connected layer 4200 may comprise FC6 and FC7 as provided by the VGG16 model.
  • a feature vector 4100 may be transformed via the fully connected layer 4200 to enhance the feature representation.
  • the second fully connected layer 4300 implemented in another fully connected layer with a smaller dimension, may perform another nonlinear transformation to the feature vector generated by the fully connected layer 4200 to further enhance the feature representation thereof.
  • the feature vector generated by the fully connected layer 4200 may also be downsampled by the second fully connected layer 4300 to facilitate a subsequent process.
  • the RSS loss layer 4400 may perform a nonlinear operation to the downsampled feature vector from the second fully connected layer 4300 and can be trained in a training phase to optimize the parameters thereof.
  • the person identifying layer 4000 may further comprise a bounding box regression layer 4500 to fine-tune the location of the bounding box generated by the ROI proposal layer in order to enhance the overlapping between the bounding box and the candidate person.
  • the CNN needs to be trained before it is applied in person recognition.
  • all of the loss layer, both in the feature extracting layer and in the ROI proposal layer may be trained synchronously.
  • a classifier (not shown) , for example but not limited to a softmax classifier, may be interposed between the second fully connected layer 4300 and the RSS loss layer 4400. Therefore, in a training phase, a plurality of training person images and a background comprised in a first training set may be scored, via the classifier, according to a preset target person image. For each training sample (i.e., a target person) , a subset may be selected from the first training set to reduce the burden of computing in order to accelerate the convergence of the training. The subset may be selected according to the scores of the plurality of training person images and the background comprised in the first training set to ensure the target person and the background being encompassed by the subset.
  • a second training set comprising the scores of each item in the subset may be constituted to train the random sampling softmax (RSS) loss layer mentioned above.
  • RSS random sampling softmax
  • a loss and a gradient may be calculated based on the scores in the second training set and the target person.
  • the loss and gradient may be back-propagated through the person identifying layer to adjust the parameters thereof until the loss and gradient converge.
  • Fig. 5 illustrates an exemplary system 5000 for recognizing a person in an input image 5100 according to one embodiment of the present application.
  • the system 5000 may comprise a feature extractor 5200 to extract a feature map from the input image 5100.
  • a ROI generator 5300 may receive the extracted feature map and crops a ROI containing a candidate person from the extracted feature map.
  • the cropped ROI may be transmitted to the recognizer 5400, which identifies whether the candidate person is a preset target person.
  • the system 5000 may be constructed in a single CNN.
  • the feature extractor 5200 may be implemented in the feature extracting layer 1200 of the CNN 1000
  • the ROI generator 5300 may be implemented in the ROI proposal layer 1300 of the CNN 1000
  • the recognizer 5400 may be implemented in the person identifying layer 1400 of the CNN 1000.
  • the ROI proposal layer 1300 and the person identifying layer 1400 may be trained synchronously.
  • the present application may be embodied as a system, a method or a computer program product. Accordingly, the present application may take the form of an entirely hardware embodiment and hardware aspects that may all generally be referred to herein as a "unit” , “circuit” , “module” , or “system” . Much of the functionality and many of the principles when implemented, are best supported with or integrated circuits (ICs) , for example but not limited to, a digital signal processor and software therefore or application specific ICs.
  • ICs integrated circuits
  • the system may comprise a memory that stores executable components and a processor, electrically coupled to the memory to execute the executable components to perform operations of the system, as discussed in reference to Figs. 1-5.
  • the present application may take the form of a computer program product embodied in any tangible medium of expression having computer-usable program code embodied in the medium.

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Multimedia (AREA)
  • Evolutionary Computation (AREA)
  • Artificial Intelligence (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • General Health & Medical Sciences (AREA)
  • Health & Medical Sciences (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Biomedical Technology (AREA)
  • Molecular Biology (AREA)
  • Medical Informatics (AREA)
  • Software Systems (AREA)
  • Databases & Information Systems (AREA)
  • Biodiversity & Conservation Biology (AREA)
  • Computing Systems (AREA)
  • Human Computer Interaction (AREA)
  • Data Mining & Analysis (AREA)
  • Bioinformatics & Cheminformatics (AREA)
  • Bioinformatics & Computational Biology (AREA)
  • Evolutionary Biology (AREA)
  • General Engineering & Computer Science (AREA)
  • Image Analysis (AREA)
  • Character Discrimination (AREA)

Abstract

A method and systems for person recognition, wherein the method comprises: providing a Convolutional Neural Network (CNN) having a feature extracting layer (1200), a region of interest (ROI) proposal layer (1300) coupled to the feature extracting layer (1200), and a person identifying layer (1400) coupled to the ROI proposal layer (1300), the ROI proposal layer (1300) and the person identifying layer (1400) being trained synchronously; extracting, via the feature extracting layer (1200), a feature map from the image; cropping, via the ROI proposal layer (1300), a ROI containing a candidate person, from the extracted feature map; and identifying, via the person identifying layer (1400), the candidate person contained in the cropped ROI, according to a preset target person image.

Description

METHOD AND SYSTEM FOR PERSON RECOGNITION Technical Field
The disclosures relate to a method and system for person recognition.
Background
Person recognition targets on recognizing a person in images. It is a fast growing technical area and has many practical applications in video surveillance and multimedia, for example but not limited to person retrieval, cross-camera visual tracking, and activity analysis. Person recognition is particularly challenging because of complex variations of viewpoints, poses, lighting conditions, occlusions, resolutions, backgrounds and camera settings etc.
Although person recognition frameworks have been proposed in recent years and the performances on these frameworks have been improved, there is still a substantial gap between the proposed frameworks and the practical applications. In most prior frameworks, the person recognition is implemented by comparing manually cropped persons in a candidate person set with a preset target person instead of searching for the target person over a whole image. Using protocols implementing these frameworks, the developed person recognition methods assume that persons are perfectly singled out from backgrounds. For example, a candidate person may be selected by a manually preset bounding box. However, the annotations of the bounding boxes are unavailable in real-world scenarios.
Some false alarms, misdetections, and misalignments can be hardly removed by prior person recognition methods and harm the overall person recognition result. Under such circumstances, the efficiencies and the accuracies of these person recognition methods are relatively low. Thus, there is a need for a more effective method for person recognition.
Summary
The following presents a simplified summary of the disclosure in order to provide a basic understanding of some aspects of the disclosure. This summary is not an extensive overview of the disclosure. This summary neither identifies key or critical elements of the disclosure nor delineates any scope of particular embodiments of the disclosure, or any scope of the claims. Its sole purpose is to present some concepts of the disclosure in a simplified form as a prelude to the more detailed description that is presented later.
In order to address, at least partially, one of the above issues, an end-to-end deep learning method for recognizing a person in an image is proposed in one aspect of the present application. The method comprises: extracting, via a feature extracting layer of a Convolutional Neural Network (CNN) , a feature map from an image； cropping, via a ROI proposal layer coupled to the feature extracting layer, a ROI containing a candidate person, from the extracted feature map； and identifying, via a person identifying layer coupled to the ROI proposal layer, the candidate person contained in the cropped ROI, according to a preset image of a target person； wherein the ROI proposal layer and the person identifying layer are trained synchronously. It brings several benefits by training the ROI proposal layer and the person identifying layer synchronously. On one hand, the learned ROI proposal layer allows some false alarms that can be easily handled by the person identifying layer. On the other hand, the ROI proposal layer and the person identifying layer fit each other better to output a more accurate result.
In one embodiment of the present application, the cropping comprises pooling the cropped ROI into a feature vector with a fixed length.
In one embodiment of the present application, the person identifying layer further comprises a bounding box regression layer, the cropped ROI includes a bounding box overlapping the candidate person, and the identifying comprises moving the boundary via the bounding box regression layer to enhance the overlapping with the candidate person.
In one embodiment of the present application, the identifying further comprises  downsampling the cropped ROI to accelerate the identifying.
In one embodiment of the present application, the person identifying layer comprises a random sampling softmax (RSS) loss layer and the training of the person identifying layer comprises: scoring, according to the preset image of the target person, a plurality of training person images and a background for the training person images in a first training set； selecting, according to scores of the training person images and the background, a subset from the first training set, wherein the subset comprises the target person and the background； constituting a second training set comprising the scores of the person images and the background in the subset； determining, by the RSS loss layer, a loss and a gradient based on the second training set； and back-propagating the determined loss and gradient to adjust parameters of the person identifying layer until the loss and the gradient converge.
In another aspect, a system for recognizing a person in an image is provided. The system comprises: a memory that stores executable components； and a processor electrically coupled to the memory to execute the executable components for: extracting, via a feature extracting layer of a CNN, a feature map from an image； cropping, via a ROI proposal layer coupled to the feature extracting layer, a ROI containing a candidate person, from the extracted feature map； and identifying, via a person identifying layer coupled to the ROI proposal layer, the candidate person contained in the cropped ROI, according to a preset image of a target person； wherein the ROI proposal layer and the person identifying layer are trained synchronously.
In yet another aspect, a system for recognizing a person in an image is provided. The system comprises: a feature extractor configured for extracting a feature map from an image； a ROI generator coupled to the feature extractor and configured for cropping a ROI containing a candidate person, out from the extracted feature map； and a recognizer coupled to the ROI generator and configured for identifying the candidate person contained in the cropped ROI according to a preset image of a target person； wherein the feature extractor is implemented in a feature extracting layer of a CNN, the ROI generator is implemented in a  ROI proposal layer of the CNN, and the recognizer is implemented in a person identifying layer of the CNN, and the ROI proposal layer and the person identifying layer are trained synchronously.
Brief Description of the Drawing
Exemplary non-limiting embodiments of the present application are described below with reference to the attached drawings. The drawings are illustrative and generally not to an exact scale. The same or similar elements on different figures are referenced with the same reference numbers.
Fig. 1 illustrates an exemplary CNN for recognizing a person in an image according to one embodiment of the present application；
Fig. 2 is a flow chart illustrating the process for recognizing a person in an image according to one embodiment of the present application；
Fig. 3 illustrates an exemplary structure of the ROI proposal layer according to one embodiment of the present application；
Fig. 4 illustrates an exemplary structure of the person identifying layer according to one embodiment of the present application； and
Fig. 5 illustrates an exemplary system for recognizing a person in an image according to one embodiment of the present application.
Detailed Description
Reference will now be made in detail to some specific embodiments of the present application contemplated by the inventors for carrying out the present application. Examples of these specific embodiments are illustrated in the accompanying drawings. While the present application is described in conjunction with these specific embodiments, it will be  appreciated by one skilled in the art that it is not intended to limit the present application to the described embodiments. In the following description, numerous specific details are set forth in order to provide a thorough understanding of the present application. The present application may be practiced without some or all of these specific details. In other instances, well-known process operations have not been described in detail in order not to unnecessarily obscure the present application.
The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the present application. As used herein, the singular forms "a" , "an" and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will be further understood that the terms "comprises" and/or "comprising" when used in this specification, specify the presence of stated features, integers, steps, operations, elements, and/or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and/or groups thereof.
An exemplary CNN 1000 for recognizing a person in an image will be described with reference to Fig. 1. An image 1100 comprising a candidate person and a random background is input to a feature extracting layer 1200 of the CNN 1000. The feature extracting layer 1200 constitutes the lowest part of the CNN 1000 and may be implemented in various forms and multiple layers, for example but not limited to, convolutional layers 1-5 of a VGG16 model. A feature map containing semantically meaningful representations is extracted from the image 1100 and transmitted to a ROI proposal layer 1300. The ROI proposal layer 1300 analyzes the extracted feature map to crop a ROI containing a candidate person from the extracted feature map. A feature map or a feature vector representing the ROI generated from the ROI proposal layer 1300 is subsequently transmitted to a person identifying layer 1400, which determines whether the candidate person contained in the copped ROI is the target person preset manually. The ROI proposal layer 1300 cooperates with the person identifying layer 1400 in a high level since they are integrated in a single  CNN and may be trained synchronously. As a result, the efficiency and the accuracy of the person recognition are enhanced significantly.
The person recognition process stated above is shown by Fig. 2. A feature map is extracted from an input image at step S2020. The extracted feature map is analyzed by a person detector (i.e., the ROI proposal layer 1300 of Fig. 1) and a ROI containing a candidate person is thus cropped from the extracted feature map at step S2040. At last, the candidate person contained in the cropped ROI is compared, via a recognizer (i.e., the person identifying layer 1300 of Fig. 1) , with a preset target person at step S2060 to accomplish person recognition.
Fig. 3 schematically illustrates a ROI proposal layer structure. A ROI proposal layer 3000 may comprise a convolutional layer 3200, a loss layer 3400, and a ROI pool 3500. The convolutional layer 3200 may perform a nonlinear transformation to a first feature map 3100 supposed to contain a candidate person to generate a second feature map 3300 more suitable to distinguish the candidate person from a background. The second feature map 3300 may be divided into a plurality of anchors indicating a plurality of regions of the second feature map 3300. The loss layer 3400 may comprise an anchor classification layer 3410 and an anchor regression layer 3420. The anchor classification layer 3410 may score each anchor contained in the second feature map 3300 according to their similarity to a human classification, while the anchor regression layer 3420 may calculate, according to the score obtained by the anchor classification layer 3410, a location and a size of a bounding box to enable the calculated bounding box overlapping the candidate person. The calculated bounding box is utilized to crop the feature map 3100 into a ROI. In a preferred implement, the feature map of the cropped ROI is pooled, by a ROI pooling technique, in a ROI pool 3500, resulting in a feature vector with a fixed length. In such a case, a back-propagating process performed in a training phase may bypass the ROI pool 3500.
Fig. 4 schematically illustrates a person identifying layer structure. A person identifying layer 4000 may comprise a first fully connected layer 4200, a second fully  connected layer 4300, and a random sampling softmax (RSS) loss layer 4400. The fully connected layer 4200 may comprise FC6 and FC7 as provided by the VGG16 model. A feature vector 4100 may be transformed via the fully connected layer 4200 to enhance the feature representation. In a preferred implement, the second fully connected layer 4300, implemented in another fully connected layer with a smaller dimension, may perform another nonlinear transformation to the feature vector generated by the fully connected layer 4200 to further enhance the feature representation thereof. Besides, the feature vector generated by the fully connected layer 4200 may also be downsampled by the second fully connected layer 4300 to facilitate a subsequent process. The RSS loss layer 4400 may perform a nonlinear operation to the downsampled feature vector from the second fully connected layer 4300 and can be trained in a training phase to optimize the parameters thereof. In a preferred implement, the person identifying layer 4000 may further comprise a bounding box regression layer 4500 to fine-tune the location of the bounding box generated by the ROI proposal layer in order to enhance the overlapping between the bounding box and the candidate person. As would be appreciated by one skilled in the art, the CNN needs to be trained before it is applied in person recognition. In the CNN proposed by the present application, all of the loss layer, both in the feature extracting layer and in the ROI proposal layer, may be trained synchronously.
In a preferred implement, a classifier (not shown) , for example but not limited to a softmax classifier, may be interposed between the second fully connected layer 4300 and the RSS loss layer 4400. Therefore, in a training phase, a plurality of training person images and a background comprised in a first training set may be scored, via the classifier, according to a preset target person image. For each training sample (i.e., a target person) , a subset may be selected from the first training set to reduce the burden of computing in order to accelerate the convergence of the training. The subset may be selected according to the scores of the plurality of training person images and the background comprised in the first training set to ensure the target person and the background being encompassed by the subset. Subsequently, a second training set comprising the scores of each item in the subset may be constituted to train the random sampling softmax (RSS) loss layer mentioned above. For example, a loss  and a gradient may be calculated based on the scores in the second training set and the target person. The loss and gradient may be back-propagated through the person identifying layer to adjust the parameters thereof until the loss and gradient converge.
Fig. 5 illustrates an exemplary system 5000 for recognizing a person in an input image 5100 according to one embodiment of the present application. The system 5000 may comprise a feature extractor 5200 to extract a feature map from the input image 5100. A ROI generator 5300 may receive the extracted feature map and crops a ROI containing a candidate person from the extracted feature map. The cropped ROI may be transmitted to the recognizer 5400, which identifies whether the candidate person is a preset target person. The system 5000 may be constructed in a single CNN. For example, the feature extractor 5200 may be implemented in the feature extracting layer 1200 of the CNN 1000, the ROI generator 5300 may be implemented in the ROI proposal layer 1300 of the CNN 1000, and the recognizer 5400 may be implemented in the person identifying layer 1400 of the CNN 1000. As has been discussed with reference to Fig. 1, the ROI proposal layer 1300 and the person identifying layer 1400 may be trained synchronously.
As will be appreciated by one skilled in the art, the present application may be embodied as a system, a method or a computer program product. Accordingly, the present application may take the form of an entirely hardware embodiment and hardware aspects that may all generally be referred to herein as a "unit" , "circuit" , "module" , or "system" . Much of the functionality and many of the principles when implemented, are best supported with or integrated circuits (ICs) , for example but not limited to, a digital signal processor and software therefore or application specific ICs. It is expected that one of ordinary skill, notwithstanding possibly significant effort and many design choices motivated by, for example, available time, current technology, and economic considerations, when guided by the concepts and principles disclosed herein will be readily capable of generating ICs with minimal experimentation. Therefore, in the interest of brevity and minimization of any risk of obscuring the principles and concepts according to the present application, further discussion  of such software and ICs, if any, will be limited to the essentials with respect to the principles and concepts used by the preferred embodiments. In addition, the present application may take the form of an entirely software embodiment (including firmware, resident software, micro-code, etc. ) or an embodiment combining software. For example, the system may comprise a memory that stores executable components and a processor, electrically coupled to the memory to execute the executable components to perform operations of the system, as discussed in reference to Figs. 1-5. Furthermore, the present application may take the form of a computer program product embodied in any tangible medium of expression having computer-usable program code embodied in the medium.

Claims (15)

  1. A method for recognizing a person in an image, comprising:
    providing a Convolutional Neural Network (CNN) comprising:
    a feature extracting layer；
    a region of interest (ROI) proposal layer coupled to the feature extracting layer； and
    a person identifying layer coupled to the ROI proposal layer, the ROI proposal layer and the person identifying layer being trained synchronously；
    extracting, via the feature extracting layer, a feature map from the image；
    cropping, via the ROI proposal layer, a ROI containing a candidate person, from the extracted feature map； and
    identifying, via the person identifying layer, the candidate person contained in the cropped ROI, according to a preset image of a target person.
  2. The method of claim 1, wherein the cropping comprises pooling the cropped ROI into a feature vector with a fixed length.
  3. The method of claim 1, wherein the person identifying layer further comprises a bounding box regression layer, wherein the cropped ROI includes a bounding box overlapping the candidate person, and wherein the identifying comprises moving the boundary via the bounding box regression layer to enhance the overlapping with the candidate person.
  4. The method of claim 1, wherein the identifying further comprises downsampling the cropped ROI to accelerate the identifying.
  5. The method of claim 1, wherein the person identifying layer comprises a random sampling softmax (RSS) loss layer and the training of the person identifying layer comprises:
    scoring, according to the preset image of the target person, a plurality of training person images and a background for the training person images in a first training set；
    selecting, according to scores of the training person images and the background, a subset from the first training set, wherein the subset comprises the target person and the background；
    constituting a second training set comprising the scores of the person images and the background in the subset；
    determining, by the RSS loss layer, a loss and a gradient based on the second training set； and
    back-propagating the determined loss and gradient to adjust parameters of the person identifying layer until the loss and the gradient converge.
  6. A system for recognizing a person in an image, comprising:
    a memory that stores executable components； and
    a processor electrically coupled to the memory to execute the executable components for:
    providing a CNN comprising:
    a feature extracting layer；
    a ROI proposal layer, coupled to the feature extracting layer； and
    a person identifying layer coupled to the ROI proposal layer, the ROI proposal layer and the person identifying layer being trained synchronously；
    extracting, via the feature extracting layer, a feature map from the image；
    cropping, via the ROI proposal layer, a ROI containing a candidate person, from the extracted feature map； and
    identifying, via the person identifying layer, the candidate person contained in the cropped ROI, according to a preset image of a target person.
  7. The system of claim 6, wherein the cropping comprises pooling the cropped ROI into  a feature vector with a fixed length.
  8. The system of claim 6, wherein the person identifying layer further comprises a bounding box regression layer, wherein the cropped ROI includes a bounding box overlapping the candidate person, and wherein the identifying comprises moving the boundary via the bounding box regression layer to enhance the overlapping with the candidate person.
  9. The system of claim 6, wherein the identifying further comprises downsampling the cropped ROI to accelerate the identifying.
  10. The system of claim 6, wherein the person identifying layer comprises a random sampling softmax (RSS) loss layer and the training of the person identifying layer comprises:
    scoring, according to the preset image of the target person, a plurality of training person images and a background for the training person images in a first training set；
    selecting, according to scores of the training person images and the background, a subset from the first training set, wherein the subset comprises the target person and the background；
    constituting a second training set comprising the scores of the person images and the background in the subset；
    determining, by the RSS loss layer, a loss and a gradient based on the second training set； and
    back-propagating the determined loss and gradient to adjust parameters of the person identifying layer until the loss and the gradient converge.
  11. A system for recognizing a person in an image, comprising:
    a feature extractor configured for extracting a feature map from the image；
    a ROI generator coupled to the feature extractor and configured for cropping a ROI containing a candidate person, out from the extracted feature map； and
    a recognizer coupled to the ROI generator and configured for identifying the candidate  person contained in the cropped ROI according to a preset image of a target person；
    wherein the feature extractor is implemented in a feature extracting layer of a CNN, the ROI generator is implemented in a ROI proposal layer of the CNN, and the recognizer is implemented in a person identifying layer of the CNN, and the ROI proposal layer and the person identifying layer are trained synchronously.
  12. The system of claim 11, wherein the ROI generator comprises a ROI pool to pool the cropped ROI into a feature vector with a fixed length.
  13. The system of claim 11, wherein the person identifying layer further comprises a bounding box regression layer, wherein the cropped ROI includes a bounding box overlapping the candidate person, and wherein the identifying comprises moving the boundary via the bounding box regression layer to enhance the overlapping with the candidate person.
  14. The system of claim 11, wherein the recognizer comprises a downsampler to downsample the cropped ROI in order to accelerate the identifying.
  15. The system of claim 11, wherein the recognizer comprises:
    a scoring device, scoring, according to the preset image of the target person, a plurality of training person images and a background for the training person images in a first training set；
    a selector, selecting, according to scores of the training person images and the background, a subset from the first training set, wherein the subset comprises the target person and the background；
    a training set generator, constituting a second training set comprising the scores of the person images and the background in the subset； and
    a RSS loss calculator, determining a loss and a gradient based on the second training set；
    wherein the determined loss and gradient are back-propagated to adjust parameters of the  person identifying layer until the loss and the gradient converge.
PCT/CN2016/078549 2016-04-06 2016-04-06 Method and system for person recognition Ceased WO2017173605A1 (en)

Priority Applications (2)

Application Number Priority Date Filing Date Title
PCT/CN2016/078549 WO2017173605A1 (en) 2016-04-06 2016-04-06 Method and system for person recognition
CN201680084297.1A CN109074472B (en) 2016-04-06 2016-04-06 Method and system for person recognition

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
PCT/CN2016/078549 WO2017173605A1 (en) 2016-04-06 2016-04-06 Method and system for person recognition

Publications (1)

Publication Number Publication Date
WO2017173605A1 true WO2017173605A1 (en) 2017-10-12

Family

ID=60000199

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2016/078549 Ceased WO2017173605A1 (en) 2016-04-06 2016-04-06 Method and system for person recognition

Country Status (2)

Country Link
CN (1) CN109074472B (en)
WO (1) WO2017173605A1 (en)

Cited By (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
KR20190041925A (en) * 2017-10-13 2019-04-23 주식회사 스트라드비젼 Learning method and learning device for adjusting parameters of cnn by using augmented loss and testing method and testing device using the same
WO2019148362A1 (en) * 2018-01-31 2019-08-08 富士通株式会社 Object detection method and apparatus
CN114972791A (en) * 2022-06-02 2022-08-30 电子科技大学长三角研究院(衢州) Image classification model training method, image classification method and related device

Families Citing this family (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN110414344B (en) * 2019-06-25 2023-06-06 深圳大学 Character classification method based on video, intelligent terminal and storage medium
CN111695491B (en) * 2020-06-10 2023-12-19 阿波罗智联(北京)科技有限公司 Method and device for detecting pedestrians
CN115238124A (en) * 2022-08-09 2022-10-25 平安科技（深圳）有限公司 Video character retrieval method, device, equipment and storage medium

Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN103955718A (en) * 2014-05-15 2014-07-30 厦门美图之家科技有限公司 Image subject recognition method
CN104102919A (en) * 2014-07-14 2014-10-15 同济大学 Image classification method capable of effectively preventing convolutional neural network from being overfit
CN104268521A (en) * 2014-09-23 2015-01-07 朱毅 Image recognition method based on convolutional neural network in non-finite category
US20150363634A1 (en) * 2014-06-17 2015-12-17 Beijing Kuangshi Technology Co.,Ltd. Face Hallucination Using Convolutional Neural Networks

Family Cites Families (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN100458832C (en) * 2007-06-21 2009-02-04 中国科学院合肥物质科学研究院 Palmprint Recognition Method Based on Orientation Feature
CN101916379A (en) * 2010-09-03 2010-12-15 华中科技大学 An Object Search and Recognition Method Based on Object Accumulation Visual Attention Mechanism
CN103903223B (en) * 2014-04-24 2017-03-01 厦门美图之家科技有限公司 A kind of method of the acquisition thumbnail based on saliency detection

Patent Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN103955718A (en) * 2014-05-15 2014-07-30 厦门美图之家科技有限公司 Image subject recognition method
US20150363634A1 (en) * 2014-06-17 2015-12-17 Beijing Kuangshi Technology Co.,Ltd. Face Hallucination Using Convolutional Neural Networks
CN104102919A (en) * 2014-07-14 2014-10-15 同济大学 Image classification method capable of effectively preventing convolutional neural network from being overfit
CN104268521A (en) * 2014-09-23 2015-01-07 朱毅 Image recognition method based on convolutional neural network in non-finite category

Cited By (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
KR20190041925A (en) * 2017-10-13 2019-04-23 주식회사 스트라드비젼 Learning method and learning device for adjusting parameters of cnn by using augmented loss and testing method and testing device using the same
KR102229328B1 (en) * 2017-10-13 2021-03-19 주식회사 스트라드비젼 Learning method and learning device for adjusting parameters of cnn by using augmented loss and testing method and testing device using the same
WO2019148362A1 (en) * 2018-01-31 2019-08-08 富士通株式会社 Object detection method and apparatus
CN111095295A (en) * 2018-01-31 2020-05-01 富士通株式会社 Object detection method and device
CN111095295B (en) * 2018-01-31 2021-09-03 富士通株式会社 Object detection method and device
CN114972791A (en) * 2022-06-02 2022-08-30 电子科技大学长三角研究院(衢州) Image classification model training method, image classification method and related device

Also Published As

Publication number Publication date
CN109074472B (en) 2020-12-18
CN109074472A (en) 2018-12-21

Similar Documents

Publication Publication Date Title
CN114787865B (en) Light Tracking: Systems and Methods for Online Top-Down Human Pose Tracking
US11527000B2 (en) System and method for re-identifying target object based on location information of CCTV and movement information of object
US11205274B2 (en) High-performance visual object tracking for embedded vision systems
US11494938B2 (en) Multi-person pose estimation using skeleton prediction
WO2016026371A1 (en) Fast object detection method based on deformable part model (dpm)
WO2016183766A1 (en) Method and apparatus for generating predictive models
US11507784B2 (en) Device and method of digital image content recognition, training of the same
CN109074472B (en) Method and system for person recognition
US8718362B2 (en) Appearance and context based object classification in images
CN111429476A (en) Method and device for determining action track of target person
Iwamura et al. ICDAR2017 robust reading challenge on omnidirectional video
Si et al. Compact triplet loss for person re-identification in camera sensor networks
Pan et al. Multi-object tracking hierarchically in visual data taken from drones
Sehgal Human activity recognition using BPNN classifier on HOG features
US20210049351A1 (en) Action recognition apparatus, action recognition method, and computer-readable recording medium
Chang et al. A vision-based human action recognition system for moving cameras through deep learning
Huo et al. Emergency obstacle avoidance system of sugarcane basecutter based on improved YOLOv5s
EP2998928B1 (en) Apparatus and method for extracting high watermark image from continuously photographed images
CN110543817A (en) Pedestrian re-identification method based on posture guidance feature learning
Cao et al. A yolov8-based lightweight detection model for different perspectives infrared images
Shikha et al. Kannada word detection in heterogeneous scene images
CN115083016A (en) Monocular camera-based small-target-oriented hand space interaction method and device
Gupta et al. HandyPose: Multi-level framework for hand pose estimation
CN118803391A (en) Target detection method based on continuously variable focal plane combined with infrared-visible light camera
CN116912763A (en) A multi-line person re-identification method integrating gait and face modalities

Legal Events

Date Code Title Description
NENP Non-entry into the national phase

Ref country code: DE

121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 16897540

Country of ref document: EP

Kind code of ref document: A1

122 Ep: pct application non-entry in european phase

Ref document number: 16897540

Country of ref document: EP

Kind code of ref document: A1