WO2020000096A1 - Système et procédé d'analyse de pose chez un être humain - Google Patents

Système et procédé d'analyse de pose chez un être humain Download PDF

Info

Publication number
WO2020000096A1
WO2020000096A1 PCT/CA2019/050887 CA2019050887W WO2020000096A1 WO 2020000096 A1 WO2020000096 A1 WO 2020000096A1 CA 2019050887 W CA2019050887 W CA 2019050887W WO 2020000096 A1 WO2020000096 A1 WO 2020000096A1
Authority
WO
WIPO (PCT)
Prior art keywords
detector
human
hand
related image
features
Prior art date
Application number
PCT/CA2019/050887
Other languages
English (en)
Inventor
Dongwook Cho
Maggie Zhang
Paul Kruszewski
Original Assignee
Wrnch Inc.
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Wrnch Inc. filed Critical Wrnch Inc.
Priority to KR1020217000622A priority Critical patent/KR20210028185A/ko
Priority to EP19825430.2A priority patent/EP3813661A4/fr
Priority to JP2020572672A priority patent/JP7417555B2/ja
Priority to US17/256,307 priority patent/US20210264144A1/en
Priority to CA3105272A priority patent/CA3105272A1/fr
Publication of WO2020000096A1 publication Critical patent/WO2020000096A1/fr

Links

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/10—Segmentation; Edge detection
    • G06T7/11—Region-based segmentation
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F18/00—Pattern recognition
    • G06F18/20—Analysing
    • G06F18/21—Design or setup of recognition systems or techniques; Extraction of features in feature space; Blind source separation
    • G06F18/214—Generating training patterns; Bootstrap methods, e.g. bagging or boosting
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/70—Determining position or orientation of objects or cameras
    • G06T7/73—Determining position or orientation of objects or cameras using feature-based methods
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/40—Extraction of image or video features
    • G06V10/44—Local feature extraction by analysis of parts of the pattern, e.g. by detecting edges, contours, loops, corners, strokes or intersections; Connectivity analysis, e.g. of connected components
    • G06V10/443—Local feature extraction by analysis of parts of the pattern, e.g. by detecting edges, contours, loops, corners, strokes or intersections; Connectivity analysis, e.g. of connected components by matching or filtering
    • G06V10/449—Biologically inspired filters, e.g. difference of Gaussians [DoG] or Gabor filters
    • G06V10/451—Biologically inspired filters, e.g. difference of Gaussians [DoG] or Gabor filters with interaction between the filter responses, e.g. cortical complex cells
    • G06V10/454—Integrating the filters into a hierarchical structure, e.g. convolutional neural networks [CNN]
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
    • G06V10/764—Arrangements for image or video recognition or understanding using pattern recognition or machine learning using classification, e.g. of video objects
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
    • G06V10/82—Arrangements for image or video recognition or understanding using pattern recognition or machine learning using neural networks
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
    • G06V40/10—Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
    • G06V40/10—Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
    • G06V40/103—Static body considered as a whole, e.g. static pedestrian or occupant recognition
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
    • G06V40/10—Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
    • G06V40/16—Human faces, e.g. facial parts, sketches or expressions
    • G06V40/168—Feature extraction; Face representation
    • G06V40/171—Local features and components; Facial parts ; Occluding parts, e.g. glasses; Geometrical relationships
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/045—Combinations of networks
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods
    • G06N3/084—Backpropagation, e.g. using gradient descent
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20081—Training; Learning
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20084—Artificial neural networks [ANN]
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/30—Subject of image; Context of image processing
    • G06T2207/30196—Human being; Person

Definitions

  • the present invention relates to the field of human pose analysis, and more particularly to human pose analysis systems and methods using lightweight convolutional neural networks (CNNs).
  • CNNs convolutional neural networks
  • a system for extracting human pose information from an image comprising: a feature extractor for extracting human-related image features from the image, the feature extractor being connectable to a database comprising a dataset of reference images and provided with a first convolutional neural network (CNN) architecture including a first plurality of CNN layers, each convolutional layer applies convolutional operation to its input data using trained kernel weights; and at least one of the following modules: a 2D body skeleton detector for determining 2D body skeleton information from the human-related image features; a body silhouette detector for determining body silhouette information from the human-related image features; a hand silhouette detector for determining hand silhouette detector from the human-related image features; a hand skeleton detector for determining hand skeleton from the human-related image features; a 3D body skeleton detector for determining 3D body skeleton from the human-related image features; and a facial keypoints detector for determining facial keypoints from the human-related
  • CNN convolutional neural network
  • the feature extractor comprises: a low- level feature extractor for extracting low-level features from the image; and an intermediate feature extractor for extracting intermediate features, the low-level features and the intermediate features forming together the human-related image features.
  • At least one of the first and second architecture comprises a deep CNN architecture.
  • one of the first and second CNN layers comprise lightweight layers.
  • a method for extracting human pose information from an image comprising: receiving an image; extracting human- related image features from the image using a feature extractor, the feature extractor being connectable to a database comprising a dataset of reference images and provided with a first convolutional neural network (CNN) architecture including a first plurality of CNN layers, each convolutional layer applies convolutional operation to its input data using trained kernel weights; and determining the human pose information using at least one of the following modules: a 2D body skeleton detector for determining 2D body skeleton information from the human-related image features; a body silhouette detector for determining body silhouette information from the human-related image features; a hand silhouette detector for determining hand silhouette detector from the human-related image features; a hand skeleton detector for determining hand skeleton from the human-related image features; a 3D body skeleton detector for determining 3D
  • the feature extractor comprises: a low- level feature extractor for extracting low-level features from the image; and an intermediate feature extractor for extracting intermediate features, the low-level features and the intermediate features forming together the human-related image features.
  • At least one of the first and second architecture comprises a deep CNN architecture.
  • one of the first and second CNN layers comprise lightweight layers.
  • Figure 1 is a block diagram illustrating a system for extracting human pose information from an image, the system comprising a feature extractor, a 2D body skeleton detector, a body silhouette detector, a hand silhouette detector and 3D body skeleton detector, a facial keypoint detector and a hand skeleton detector, in accordance with an embodiment;
  • Figure 2 is a block diagram illustrating the feature extractor of Figure 1, in accordance with an embodiment
  • Figure 3 is a block diagram illustrating the 2D body skeleton detector of
  • Figure 4 is a block diagram illustrating the body silhouette detector of
  • Figure 5 is a block diagram illustrating the hand silhouette detector of
  • Figure 6 is a block diagram illustrating the hand skeleton detector of
  • Figure 7 is a block diagram illustrating the 3D body skeleton detector of
  • Figure 8 is a block diagram illustrating the facial keypoint detector of
  • Figure 1 in accordance with an embodiment; and [0024]
  • Figure 9 is a block diagram of a processing module adapted to execute at least some of the steps of the extraction of human pose information, in accordance with an embodiment.
  • Figure 1 illustrates one embodiment of a system 10 for extracting human pose information from an image.
  • the system 10 is configured for receiving images, localize humans within the received images and automatically infer human pose information from each image.
  • the human pose information comprises geometric information of human skeletons and body part shapes.
  • Human skeletons may be expressed by bone joint locations and/or bone orientations with lengths and body part shapes can be expressed as silhouettes and/or surface meshes with locations.
  • human pose information may include information such as 2D and/or 3D body skeletons with human joints, body shapes or silhouettes, and/or skeletons and silhouettes of body part like hand, etc.
  • the system 10 is configured to first extract from the images human-related image features that are learned by an image dataset and determine the human pose information from the extracted human-related image features.
  • the human-related image features comprise primitive information related to human bodies and human body parts obtained from an image, such as points, edges, lines, contours, intensities, gradients, contrasts of small to large objects in an image, relations of those objects, etc.
  • the data set comprises a set of reference images with and without human beings and ground-truth labels related to human body geometry.
  • Labels may include 2D body joint locations (x,y) and visibilities (such as not available, visible, existing in image but occluded) in image; 2D hand joint locations and visibilities in image; 2D face keypoints locations and visibilities in image, a silhouette of a human body, a silhouette of a hand, 3D body joint locations, etc. It should be understood that not all reference images contained in the data set have all the labels associated thereto.
  • the data set of reference images comprises at least tens of thousands of images for training and may be qualified as being large scale.
  • CNNs are composed of convolutional neural network layers, hereinafter referred to as convolutional layers.
  • Each convolutional layer receives input data or processed data from previous convolutional layer(s) and sends its output data to the following layer(s) after applying a convolutional operation to its input data.
  • the output of a convolutional layer is in the form of a tensor or a multi-dimensional array.
  • Each convolutional layer applies convolutional operation to its input data using trained kernel weights.
  • the training of the weights of convolutional layers is performed by backpropagation technique using the dataset of reference images.
  • each convolutional layer is configured for applying a nonlinear activation function such as a rectified linear unit (ReLU) to the input data to allow for more robust decision of CNNs.
  • ReLU rectified linear unit
  • the system 10 uses deep CNNs. In one embodiment, the
  • CNNs comprise at least three convolutional layers.
  • a deep CNN architecture preserves more neurons or weights and possibly accommodates a variety of input data and analyzes them robustly without being influenced by noise or clutter.
  • the CNN architecture comprises lightweight convolutional layers.
  • each convolutional layer is made “computationally light” by reducing the number of kernels and/or their size and/or by applying down-sampling.
  • the architecture may be adequate for real-time human pose analysis performed on a low-end device.
  • pooling layers that do not significantly influence the accuracy of estimated results may be eliminated. For example, pooling layers that also perform down- sampling may be removed and a neighboring convolution layer located before a pooling layer may perform down-sampling during its convolutional operation.
  • a minimal input image resolution may be chosen by considering common person size in an image. For example, a person of 80x80 pixels in an image can be robustly analyzed without losing much human-related image features. A lower resolution image may present a lack of details, but it may be sufficient for a good approximation of body pose.
  • the resolution of the image is 48x48.
  • the resolution of the image is 96x96.
  • the resolution of the image is 256x144.
  • the resolution of the image is 400x320.
  • Receptive fields of a person may be considered in order to decide the number of convolutional layers and their kernel sizes by limiting the maximum resolution to be analyzed. For instance, a region with 84x84 pixels can be covered by two convolution layers with 11x11 kernels after down-sampling an input image by 4. Ten 3x3 convolution layers can cover the same region with more layers yet less computational cost.
  • the output depth size defined in each convolutional layer may be reduced as long as the resulting accuracy is higher than a minimum target accuracy chosen by the user.
  • the computational cost is proportional to the kernel size in each dimension (kernel width, height, and depth) as well as output depth size. Size of weights may be decided by the multiplied sum of kernel width, height, and depth in addition to the number of biases.
  • a CNN model is a collection of weights and biases learned by a machine given a dataset and designed architecture. The CNN model may be chosen empirically to provide the highest accuracy.
  • the system comprises a feature extractor 30 in communication with a database 20.
  • the system also comprises a 2D body skeleton detector 40, a body silhouette detector 60, a hand silhouette detector 70 and 3D body skeleton detector 90, which are all in communication with the feature extractor 30.
  • the system 10 further comprises a facial keypoint detector 50 in communication with the 2D body skeleton detector 40 and a hand skeleton detector 80 in communication with the hand silhouette detector 70.
  • the database 20 comprises the data set of reference images stored therein.
  • the database 20 is stored in a memory that is comprised in the system 10.
  • the database 20 is stored in a memory that is not included in the system 10.
  • the feature extractor 30 comprises a low-level feature extractor 110 and at least one intermediate feature extractor 120.
  • the feature extractor 30 is composed of a low- level feature extractor 110 and one or more intermediate feature extractor 120.
  • the low- level feature extractor 110 is configured for receiving an image and extracting from the image low-level features that represent elemental characteristics of local regions in an image such as intensities, edges, gradients, curvatures, points, object shapes and/or the like.
  • the intermediate feature extractor 120 is (are) configured for receiving the low-level features and determining intermediate features which correspond to high-level features obtained by correlating the low-level features extracted by the low-level feature extractor 110 and are related to human pose information such as shapes and/or relations of body parts.
  • the low-level features and the intermediate features form together the human- related image features outputted by the feature extractor 30.
  • the low-level extractor 110 comprises repeated blocks in different image scales and each block comprises a series of convolutional layers with ReLU activations.
  • the low-level feature extractor 110 preserves generic image features such as edges, contours, blobs, their orientations, or some other observations learned from large scale image dataset.
  • a proven CNN architecture such as Inception, VGG, and ResNet can be considered for backbone networks.
  • Lightweight backbone networks can be designed for reduced computational cost while preserving the minimum human pose-related features as mentioned above.
  • the intermediate feature extractor 120 is configured for intermediate supervision when a CNN model is trained. Intermediate supervision allows for the training of a CNN model by adding loss layers in the middle layers (or output layers of intermediate feature extractors) in addition to the last output layer. In neural networks, a loss layer compares difference between the output layer and ground-truth data and propagate backward to train weights and biases in each layer.
  • the number of convolutional layers present in the intermediate feature extractors 120 and their parameters for each intermediate stage are tailored by the size of input image and target objects, i.e. humans, as described above.
  • Each intermediate stage is trained using the dataset of reference images. For example, a stack of 2D joint heat map in which the human joints in an image are marked in the same location may be generated using 2D joint locations. The exact joint location on the heat map has high response value while the location has low or no response value if the distance from the joint location is farther.
  • the ground-truth heat maps that are generated from the dataset using the annotated 2D joint locations are compared to the estimated heat maps that are inferred from the training model during the model training.
  • the model is trained by adjusting weight and bias values by repeating forward and backward propagation process throughout the connected layers in the neural networks. [0051] In one embodiment, by training multiple stages of the convolutional layers of the intermediate feature extractors 120, the features related to human poses are refined through deeper network layers and therefore more robust results may be obtained. In addition, the model training becomes more efficient.
  • the output of each layer in the low-level feature extractor 110 and the intermediate feature extractors 120 form the human-related image features which can be presented as human-related image feature tensors. Depending on the purpose, a subset of the human-related image feature tensors can be used for detailed human pose analysis.
  • Figure 3 illustrates one embodiment of a 2D body skeleton detector 40 which comprises a 2D body joint estimation network 210 and a post-processing module 220.
  • the 2D body skeleton detector 40 receives as input a subset of the human- related image features generated by the feature extractor 30 and generates 2D joint heat maps as output.
  • the subset of the human-related image features comprises a combination of output feature tensors of different convolution layers of the feature extractor 30 that preserve distinctive features related to human joints and shapes.
  • the convolution layers that are close to the end of the low-level feature extractor 110 and/or the intermediate feature extractor 120 can be considered since they are normally refined throughout the convolution layers.
  • the output feature tensors of the last convolution layers in the low-level feature extractor 110 and N-th intermediate feature extractor 110 can be chosen to provide data to the 2D body skeleton detector 40.
  • the 2D body skeleton detector 40 infers the estimated heat maps, which are used to decide the candidates of joint locations that are local maxima in the heat maps and a heat map response value is over a manually defined threshold.
  • Figure 4 illustrates one embodiment of the body silhouette detector 60 which comprises a body silhouette segmentation module 310 comprising convolutional layers and a post-processing module 320.
  • the body silhouette detector 60 is configured for segmenting all the human bodies in an image and generating a mask image for human bodies.
  • the convolutional layers of the body silhouette segmentation 310 receive the human-related image feature tensors from the feature extractor 30 and construct a body mask image with human body silhouettes.
  • Masks are used to segment different objects in an image by applying bitwise masking to each pixel.
  • a body mask image is a binary image where a mask value is 1 if a pixel belongs to a human and non-human pixel is 0. Since the scale of the human-related image feature tensors is reduced normally by factor of 2 to 16 compared to an input image width and height, upscaling can be performed during the convolutional operations to increase the body mask image resolution and preserve more details.
  • the post-processing module 320 takes the inferred mask image from the body silhouette segmentation module 310 and resizes the mask image to the same resolution as the source input image.
  • the body mask image can then be used for identifying the location and shape of a person in an image.
  • Figure 5 illustrates one embodiment for the hand silhouette detector 70 which comprises a hand silhouette segmentation module 410 formed of convolutional layers and a post-processing module 420.
  • the hand silhouette detector module 410 is configured for segmenting the hands of the human bodies present in an input image and generates mask images for left and/or right hand similarly to the body silhouette detector 60.
  • the convolutional layers of the hand silhouette segmentation module 410 receives the human-related image feature tensors from the feature extractor 30 and constructs hand mask images with human body silhouettes.
  • the post-processing module 420 is configured for resizing the inferred hand mask images.
  • the hand mask images may then be used for identifying the location and shape of visible hands in an image. This information can be used for further analysis of each hand pose.
  • the hand silhouette detector 70 can be merged with the body silhouette detector 60 and the merged detectors 60 and 70 can be trained together.
  • the neural network layers in these merged detectors may be shared for more efficient computations.
  • Figure 6 illustrates one embodiment for the hand skeleton detector 80 which comprises a hand joint estimation module 510 comprising convolutional layers and a post- processing module 520.
  • the hand skeleton detector 80 receives an image of a hand, i.e. a hand image, and estimates hand joints in the hand image.
  • the hand image may be any image of a hand such as an image not specified in the system.
  • the hand image may be a hand image cropped from an input image data 20 using a hand region (or bounding box) detected by the hand silhouette detector 70.
  • the hand joint estimation module 510 can be designed with a similar architecture that combines the architecture of the feature extraction networks 110 and 120 and the architecture of the 2D body joint estimation networks 210.
  • the hand skeleton detector 80 can be designed to directly receive the human-related image feature tensors from the feature extractor 30.
  • the post-processing module 520 for hand pose estimation takes the estimated heat maps and decides the candidates of joint locations and constructs a hand skeleton.
  • Figure 7 illustrates one embodiment for the 3D body skeleton detector 90 which comprises a 3D body joint estimation module 610 comprising convolutional layers and a post-processing module 620.
  • the 3D body skeleton detector 90 is configured for estimating 3D coordinates of human body joints from a single image.
  • the 3D body skeleton detector 90 receives human-related image feature tensors and estimates normalized 3D coordinates of a human body detected in an image.
  • the post-processing module 620 is configured for mapping the normalized 3D locations into image and real-world spaces.
  • FIG. 8 illustrates one embodiment for the facial keypoints detector 50 which comprises a facial keypoints estimation module 710 comprising convolutional layers, and a post-processing module 720.
  • the facial keypoints detector 50 receives a cropped facial image decided by the 2D body skeleton detector 40 which estimates rough location of facial keypoints such as eyes, ears, nose, and/or the like.
  • the locations of more detailed keypoints such as contour points of eyes, upper and lower lips, chin, eyebrows, nose, etc. are estimated by the convolutional layers of the facial keypoints estimation module 710. Alignment of detected facial keypoints and/or outlier filtering may be performed by the post-processing module 720.
  • the same human-related image features determined by the feature extractor 30 are shared by at least some of the detectors 40-90 to infer the human pose information.
  • the feature extractor 30 determines all the human-related image features that can be obtained from an image and stores them at each neural network layer in a tensor form.
  • the feature extractor 30 can be designed by explicitly defining feature descriptors such as scale-invariant feature transform (SIFT) and histogram of oriented gradients (HOG). Such a feature extractor pre-defmes image features regardless of the dataset.
  • SIFT scale-invariant feature transform
  • HOG histogram of oriented gradients
  • the extractor 30 and the detectors 40-90 are each provided with at least one respective processor or processing unit, a respective communication unit and a respective memory. In another embodiment, at least two of the group consisting of the extractor 30 and the detectors 40-90 share a same processor, a same communication and/or a same memory. For example, the extractor 30 and the detectors 40- 90 may share the same processor, the same communication unit and the same memory. In this case, the extractor 30 and the detectors 40-90 may correspond to different modules executed by the processor of a computer machine such as a personal computer, a laptop, a tablet, a smart phone, etc.
  • a computer machine such as a personal computer, a laptop, a tablet, a smart phone, etc.
  • system 10 comprising the detectors 40-90
  • system 10 may comprise only one of the detectors 40-90.
  • the system 10 may comprise at least two of the detectors 40- 90.
  • the sharing of the same human-related image features between a plurality of detectors makes the analysis consistent and fast by minimizing computations for each detector.
  • FIG. 9 is a block diagram illustrating an exemplary processing module 800 for executing the above described pose information extraction from an image, in accordance with some embodiments.
  • the processing module 800 typically includes one or more Computer Processing Units (CPUs) and/or Graphic Processing Units (GPUs) 802 for executing modules or programs and/or instructions stored in memory 804 and thereby performing processing operations, memory 804, and one or more communication buses 806 for interconnecting these components.
  • the communication buses 806 optionally include circuitry (sometimes called a chipset) that interconnects and controls communications between system components.
  • the memory 804 includes high-speed random access memory, such as DRAM, SRAM, DDR RAM or other random access solid state memory devices, and may include non-volatile memory, such as one or more magnetic disk storage devices, optical disk storage devices, flash memory devices, or other non-volatile solid state storage devices.
  • the memory 804 optionally includes one or more storage devices remotely located from the CPU(s) 802.
  • the memory 804, or alternately the non-volatile memory device(s) within the memory 804, comprises a non-transitory computer readable storage medium.
  • the memory 804, or the computer readable storage medium of the memory 804 stores the following programs, modules, and data structures, or a subset thereof: a feature extraction module 810 for extracting human-related image features from an image; a 2D body skeleton detection module 812 for estimating 2D body joint positions; a body silhouette detection module 814 for identifying and segmenting body silhouettes; a hand silhouette detection module 816 for identifying and segmenting hand silhouettes; a 3D body skeleton detection module 818 for estimating 3D body joint positions; a facial keypoints detection module 820 for estimating facial keypoint positions: and a hand skeleton detection module 822 for estimating hand joint positions.
  • a feature extraction module 810 for extracting human-related image features from an image
  • a 2D body skeleton detection module 812 for estimating 2D body joint positions
  • a body silhouette detection module 814 for identifying and segmenting body silhouettes
  • a hand silhouette detection module 816 for identifying and segmenting hand silhouettes
  • Each of the above identified elements may be stored in one or more of the previously mentioned memory devices, and corresponds to a set of instructions for performing a function described above.
  • the above identified modules or programs i.e., sets of instructions
  • the memory 804 may store a subset of the modules and data structures identified above.
  • the memory 804 may store additional modules and data structures not described above.
  • FIG. 9 is intended more as functional description of the various features which may be present in a management module than as a structural schematic of the embodiments described herein. In practice, and as recognized by those of ordinary skill in the art, items shown separately could be combined and some items could be separated. [0079]
  • the embodiments of the invention described above are intended to be exemplary only. The scope of the invention is therefore intended to be limited solely by the scope of the appended claims.

Abstract

L'invention concerne un système et un procédé d'extraction d'informations de pose chez un être humain à partir d'une image, comprenant un extracteur de caractéristiques relié à une base de données, et un réseau neuronal convolutif (CNN) comprenant une pluralité de couches de CNN. Ledit système/procédé comprend en outre au moins l'un des modules suivants : un détecteur de squelette de corps 2D pour déterminer des informations de squelette de corps 2D à partir des caractéristiques d'image associées à l'être humain ; un détecteur de silhouette de corps pour déterminer des informations de silhouette de corps à partir des caractéristiques d'image associées à l'être humain ; un détecteur de silhouette de main pour déterminer une silhouette de main à partir des caractéristiques d'image associées à l'être humain ; un détecteur de squelette de main pour déterminer un squelette de main à partir des caractéristiques d'image associées à l'être humain ; un détecteur de squelette de corps 3D pour déterminer un squelette de corps 3D à partir des caractéristiques d'image associées à l'être humain ; et un détecteur de points clés faciaux pour déterminer des points clés faciaux à partir des caractéristiques d'image associées à l'être humain.
PCT/CA2019/050887 2018-06-29 2019-06-27 Système et procédé d'analyse de pose chez un être humain WO2020000096A1 (fr)

Priority Applications (5)

Application Number Priority Date Filing Date Title
KR1020217000622A KR20210028185A (ko) 2018-06-29 2019-06-27 사람 자세 분석 시스템 및 방법
EP19825430.2A EP3813661A4 (fr) 2018-06-29 2019-06-27 Système et procédé d'analyse de pose chez un être humain
JP2020572672A JP7417555B2 (ja) 2018-06-29 2019-06-27 人体ポーズ分析システム及び方法
US17/256,307 US20210264144A1 (en) 2018-06-29 2019-06-27 Human pose analysis system and method
CA3105272A CA3105272A1 (fr) 2018-06-29 2019-06-27 Systeme et procede d'analyse de pose chez un etre humain

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
US201862691818P 2018-06-29 2018-06-29
US62/691,818 2018-06-29

Publications (1)

Publication Number Publication Date
WO2020000096A1 true WO2020000096A1 (fr) 2020-01-02

Family

ID=68985280

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CA2019/050887 WO2020000096A1 (fr) 2018-06-29 2019-06-27 Système et procédé d'analyse de pose chez un être humain

Country Status (6)

Country Link
US (1) US20210264144A1 (fr)
EP (1) EP3813661A4 (fr)
JP (1) JP7417555B2 (fr)
KR (1) KR20210028185A (fr)
CA (1) CA3105272A1 (fr)
WO (1) WO2020000096A1 (fr)

Cited By (8)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN112131965A (zh) * 2020-08-31 2020-12-25 深圳云天励飞技术股份有限公司 一种人体姿态估计方法、装置、电子设备及存储介质
CN112336342A (zh) * 2020-10-29 2021-02-09 深圳市优必选科技股份有限公司 手部关键点检测方法、装置及终端设备
KR20210036879A (ko) 2020-06-08 2021-04-05 베이징 바이두 넷컴 사이언스 테크놀로지 컴퍼니 리미티드 가상 오브젝트 구동 방법, 장치, 전자기기 및 판독 가능 저장매체
WO2021212411A1 (fr) * 2020-04-23 2021-10-28 Intel Corporation Système d'interaction cinématique à suivi de posture amélioré
US20210366127A1 (en) * 2019-05-07 2021-11-25 Tencent Technology (Shenzhen) Company Limited Image processing method and apparatus, computer-readable storage medium
US20220210334A1 (en) * 2020-12-29 2022-06-30 Industrial Technology Research Institute Movable photographing system and photography composition control method
WO2022181252A1 (fr) * 2021-02-26 2022-09-01 日本電気株式会社 Dispositif de détection de joint, dispositif de génération de modèle d'apprentissage, procédé de détection de joint, procédé de génération de modèle d'apprentissage et support d'enregistrement lisible par ordinateur
WO2022181251A1 (fr) * 2021-02-26 2022-09-01 日本電気株式会社 Dispositif de détection de point d'articulation, procédé de détection de point d'articulation et support d'enregistrement lisible par ordinateur

Families Citing this family (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2020084667A1 (fr) * 2018-10-22 2020-04-30 富士通株式会社 Procédé de reconnaissance, programme de reconnaissance, dispositif de reconnaissance, procédé d'apprentissage, programme d'apprentissage et dispositif d'apprentissage
CN111435432B (zh) * 2019-01-15 2023-05-26 北京市商汤科技开发有限公司 网络优化方法及装置、图像处理方法及装置、存储介质
US20210312236A1 (en) * 2020-03-30 2021-10-07 Cherry Labs, Inc. System and method for efficient machine learning model training
CN111985414B (zh) * 2020-08-21 2024-02-23 成都数字天空科技有限公司 一种关节点位置确定方法及装置
CN114496263B (zh) * 2022-04-13 2022-07-12 杭州研极微电子有限公司 用于体重指数估计的神经网络模型建立方法及存储介质
WO2024015620A1 (fr) * 2022-07-15 2024-01-18 Omnimed Suivi de réalisation d'interventions médicales

Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US8437506B2 (en) * 2010-09-07 2013-05-07 Microsoft Corporation System for fast, probabilistic skeletal tracking
CN104346607A (zh) * 2014-11-06 2015-02-11 上海电机学院 基于卷积神经网络的人脸识别方法
CN105069423A (zh) * 2015-07-29 2015-11-18 北京格灵深瞳信息技术有限公司 一种人体姿态检测方法及装置

Family Cites Families (9)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP4931218B2 (ja) 2007-02-22 2012-05-16 キヤノン株式会社 撮像装置、物体検出方法及び姿勢パラメータの算出方法
WO2015180100A1 (fr) * 2014-05-29 2015-12-03 Beijing Kuangshi Technology Co., Ltd. Localisation de point de repère facial à l'aide de réseaux neuronaux en cascade grossiers à fins
JP6873600B2 (ja) 2016-03-04 2021-05-19 キヤノン株式会社 画像認識装置、画像認識方法及びプログラム
CA3015492C (fr) * 2016-03-21 2021-11-23 The Procter & Gamble Company Systemes et procedes pour fournir des recommandations personnalisees pour des produits
WO2018017399A1 (fr) * 2016-07-20 2018-01-25 Usens, Inc. Procédé et système de poursuite 3d du squelette de la main
US10582907B2 (en) * 2016-10-31 2020-03-10 Siemens Healthcare Gmbh Deep learning based bone removal in computed tomography angiography
US10360494B2 (en) 2016-11-30 2019-07-23 Altumview Systems Inc. Convolutional neural network (CNN) system based on resolution-limited small-scale CNN modules
US11010595B2 (en) * 2017-03-23 2021-05-18 Samsung Electronics Co., Ltd. Facial verification method and apparatus
CA2995242A1 (fr) * 2018-02-15 2019-08-15 Wrnch Inc. Methode et systeme de classification d'activite

Patent Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US8437506B2 (en) * 2010-09-07 2013-05-07 Microsoft Corporation System for fast, probabilistic skeletal tracking
CN104346607A (zh) * 2014-11-06 2015-02-11 上海电机学院 基于卷积神经网络的人脸识别方法
CN105069423A (zh) * 2015-07-29 2015-11-18 北京格灵深瞳信息技术有限公司 一种人体姿态检测方法及装置

Non-Patent Citations (1)

* Cited by examiner, † Cited by third party
Title
See also references of EP3813661A4 *

Cited By (16)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US11869194B2 (en) * 2019-05-07 2024-01-09 Tencent Technology (Shenzhen) Company Limited Image processing method and apparatus, computer-readable storage medium
US20210366127A1 (en) * 2019-05-07 2021-11-25 Tencent Technology (Shenzhen) Company Limited Image processing method and apparatus, computer-readable storage medium
WO2021212411A1 (fr) * 2020-04-23 2021-10-28 Intel Corporation Système d'interaction cinématique à suivi de posture amélioré
US11532127B2 (en) 2020-06-08 2022-12-20 Beijing Baidu Netcom Science Technology Co., Ltd. Virtual object driving method, apparatus, electronic device, and readable storage medium
KR20210036879A (ko) 2020-06-08 2021-04-05 베이징 바이두 넷컴 사이언스 테크놀로지 컴퍼니 리미티드 가상 오브젝트 구동 방법, 장치, 전자기기 및 판독 가능 저장매체
JP2021103564A (ja) * 2020-06-08 2021-07-15 ベイジン バイドゥ ネットコム サイエンス テクノロジー カンパニー リミテッドＢｅｉｊｉｎｇ Ｂａｉｄｕ Ｎｅｔｃｏｍ Ｓｃｉｅｎｃｅ Ｔｅｃｈｎｏｌｏｇｙ Ｃｏ．， Ｌｔｄ． 仮想オブジェクト駆動方法、装置、電子機器及び可読記憶媒体
KR102590841B1 (ko) * 2020-06-08 2023-10-19 베이징 바이두 넷컴 사이언스 테크놀로지 컴퍼니 리미티드 가상 오브젝트 구동 방법, 장치, 전자기기 및 판독 가능 저장매체
JP7273880B2 (ja) 2020-06-08 2023-05-15 ベイジン バイドゥ ネットコム サイエンス テクノロジー カンパニー リミテッド 仮想オブジェクト駆動方法、装置、電子機器及び可読記憶媒体
CN112131965B (zh) * 2020-08-31 2023-10-13 深圳云天励飞技术股份有限公司 一种人体姿态估计方法、装置、电子设备及存储介质
CN112131965A (zh) * 2020-08-31 2020-12-25 深圳云天励飞技术股份有限公司 一种人体姿态估计方法、装置、电子设备及存储介质
CN112336342B (zh) * 2020-10-29 2023-10-24 深圳市优必选科技股份有限公司 手部关键点检测方法、装置及终端设备
CN112336342A (zh) * 2020-10-29 2021-02-09 深圳市优必选科技股份有限公司 手部关键点检测方法、装置及终端设备
US11445121B2 (en) * 2020-12-29 2022-09-13 Industrial Technology Research Institute Movable photographing system and photography composition control method
US20220210334A1 (en) * 2020-12-29 2022-06-30 Industrial Technology Research Institute Movable photographing system and photography composition control method
WO2022181251A1 (fr) * 2021-02-26 2022-09-01 日本電気株式会社 Dispositif de détection de point d'articulation, procédé de détection de point d'articulation et support d'enregistrement lisible par ordinateur
WO2022181252A1 (fr) * 2021-02-26 2022-09-01 日本電気株式会社 Dispositif de détection de joint, dispositif de génération de modèle d'apprentissage, procédé de détection de joint, procédé de génération de modèle d'apprentissage et support d'enregistrement lisible par ordinateur

Also Published As

Publication number Publication date
JP2021529389A (ja) 2021-10-28
EP3813661A4 (fr) 2022-04-06
EP3813661A1 (fr) 2021-05-05
CA3105272A1 (fr) 2020-01-02
JP7417555B2 (ja) 2024-01-18
US20210264144A1 (en) 2021-08-26
KR20210028185A (ko) 2021-03-11

Similar Documents

Publication Publication Date Title
US20210264144A1 (en) Human pose analysis system and method
Pitaloka et al. Enhancing CNN with preprocessing stage in automatic emotion recognition
Kruthiventi et al. Saliency unified: A deep architecture for simultaneous eye fixation prediction and salient object segmentation
US9275273B2 (en) Method and system for localizing parts of an object in an image for computer vision applications
CN112446270A (zh) 行人再识别网络的训练方法、行人再识别方法和装置
CN111178251B (zh) 一种行人属性识别方法及系统、存储介质及终端
CN111052126A (zh) 行人属性识别与定位方法以及卷积神经网络系统
Aich et al. Improving object counting with heatmap regulation
CN111476806B (zh) 图像处理方法、装置、计算机设备和存储介质
JP6756406B2 (ja) 画像処理装置、画像処理方法および画像処理プログラム
CN113361495A (zh) 人脸图像相似度的计算方法、装置、设备及存储介质
CN113989890A (zh) 基于多通道融合和轻量级神经网络的人脸表情识别方法
Vinoth Kumar et al. A decennary survey on artificial intelligence methods for image segmentation
Wu et al. Privacy leakage of sift features via deep generative model based image reconstruction
CN115063526A (zh) 二维图像的三维重建方法、系统、终端设备及存储介质
Kalam et al. Gender classification using geometric facial features
Akbas et al. Low-level hierarchical multiscale segmentation statistics of natural images
Marban et al. Estimating position & velocity in 3d space from monocular video sequences using a deep neural network
Jacques et al. Improved head-shoulder human contour estimation through clusters of learned shape models
Choudhury et al. Human detection using orientation shape histogram and coocurrence textures
Beebe A Complete Bibliography of Publications in International Journal of Image and Graphics
Pemula et al. Generation of Random Fields for Image Segmentation Techniques: A Review
Ingale et al. Deep Learning for Crowd Image Classification for Images Captured Under Varying Climatic and Lighting Condition
O’Brien et al. Computer Vision Concepts and Applications
Deshapriya et al. Centroid-UNet: Detecting centroids in aerial images

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 19825430

Country of ref document: EP

Kind code of ref document: A1

ENP Entry into the national phase

Ref document number: 2020572672

Country of ref document: JP

Kind code of ref document: A

ENP Entry into the national phase

Ref document number: 3105272

Country of ref document: CA

NENP Non-entry into the national phase

Ref country code: DE

ENP Entry into the national phase

Ref document number: 2019825430

Country of ref document: EP

Effective date: 20210129