EP4292059A1 - Prédiction humaine neuronale multivue à l?aide d?un moteur de rendu différentiable implicite pour l?expression faciale, la forme et la pose du corps, et la capture de performance de vêtements - Google Patents

Prédiction humaine neuronale multivue à l?aide d?un moteur de rendu différentiable implicite pour l?expression faciale, la forme et la pose du corps, et la capture de performance de vêtements

Info

Publication number
EP4292059A1
EP4292059A1 EP22715732.8A EP22715732A EP4292059A1 EP 4292059 A1 EP4292059 A1 EP 4292059A1 EP 22715732 A EP22715732 A EP 22715732A EP 4292059 A1 EP4292059 A1 EP 4292059A1
Authority
EP
European Patent Office
Prior art keywords
human
images
neural network
image
mesh
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
EP22715732.8A
Other languages
German (de)
English (en)
Inventor
Qing Zhang
Hanyuan XIAO
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Sony Group Corp
Sony Corp of America
Original Assignee
Sony Group Corp
Sony Corp of America
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Priority claimed from US17/701,991 external-priority patent/US11961266B2/en
Application filed by Sony Group Corp, Sony Corp of America filed Critical Sony Group Corp
Publication of EP4292059A1 publication Critical patent/EP4292059A1/fr
Pending legal-status Critical Current

Links

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
    • G06V40/10—Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
    • G06V40/103—Static body considered as a whole, e.g. static pedestrian or occupant recognition
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T13/00—Animation
    • G06T13/20—3D [Three Dimensional] animation
    • G06T13/40—3D [Three Dimensional] animation of characters, e.g. humans, animals or virtual beings
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T15/00—3D [Three Dimensional] image rendering
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T17/00—Three dimensional [3D] modelling, e.g. data description of 3D objects
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T17/00—Three dimensional [3D] modelling, e.g. data description of 3D objects
    • G06T17/20—Finite element generation, e.g. wire-frame surface description, tesselation
    • G06T17/205—Re-meshing
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/50—Depth or shape recovery
    • G06T7/55—Depth or shape recovery from multiple images
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/70—Determining position or orientation of objects or cameras
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/40—Extraction of image or video features
    • G06V10/42—Global feature extraction by analysis of the whole pattern, e.g. using frequency domain transformations or autocorrelation
    • G06V10/422—Global feature extraction by analysis of the whole pattern, e.g. using frequency domain transformations or autocorrelation for representing the structure of the pattern or shape of an object therefor
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/40—Extraction of image or video features
    • G06V10/46—Descriptors for shape, contour or point-related descriptors, e.g. scale invariant feature transform [SIFT] or bags of words [BoW]; Salient regional features
    • G06V10/467—Encoded features or binary features, e.g. local binary patterns [LBP]
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
    • G06V10/766—Arrangements for image or video recognition or understanding using pattern recognition or machine learning using regression, e.g. by projecting features on hyperplanes
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
    • G06V10/82—Arrangements for image or video recognition or understanding using pattern recognition or machine learning using neural networks
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
    • G06V40/10—Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
    • G06V40/16—Human faces, e.g. facial parts, sketches or expressions
    • G06V40/174—Facial expression recognition
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20084—Artificial neural networks [ANN]
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/30—Subject of image; Context of image processing
    • G06T2207/30196—Human being; Person
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2210/00—Indexing scheme for image generation or computer graphics
    • G06T2210/12—Bounding box
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2210/00—Indexing scheme for image generation or computer graphics
    • G06T2210/16—Cloth

Definitions

  • the present invention relates to three dimensional computer vision and graphics for the entertainment industry. More specifically, the present invention relates to acquiring and processing three dimensional computer vision and graphics for film, TV, music and game content creation.
  • the output model contains three layers from inner to outer: a skeleton at a predicted pose; a naked 3D body of a predicted shape with facial expression (e.g., SMPL-X model parameterized by blendshapes and joint rotations); and a 3D field of clothes displacement and the appearance RGB color inferred from the input images.
  • a clothed body mesh is obtained by deforming the naked 3D body mesh according to the clothes displacement field.
  • MVS-3DCNN takes the multiview image set as input, chooses the frontal view as the reference view and extracts a feature volume.
  • HMR MLP regresses all the feature volumes to the human pose, shape, facial expression parameters.
  • SMPL-X model generates the human naked body mesh according to the parameters. And then the naked body mesh is converted into an occupancy field in its bounding box.
  • Figure 2 illustrates the workflow of a forward prediction represented by the tensor notations, in which the weight of all the networks MVS 3DCNN, HMR MLP and NeRF MLP are known, according to some embodiments.
  • Figure 3 illustrates the workflow of training the network using supervision according to some embodiments.
  • Figure 5 illustrates the alignment of the MVS 3DCNN of each view to the NeRF MLP according to some embodiments.
  • Neural human prediction includes predicting a 3D human model including a pose of a skeleton, body shape and clothes displacement and appearance from a set of images (a single image or multiview images).
  • Embodiments of the neural human prediction describe methods for using a neural network.
  • Multiview neural human prediction outperforms the single image -based mocap and human lifting in quality and robustness, simplifies the architecture of the body clothes prediction network such as Implicit Part Network, which takes a sparse point cloud as input with heavy memory cost and performs slowly, and avoids the resolution limitation of latent-code -based network, such as Neural Body, which encodes the entire 3D volume.
  • Figure 1 illustrates a flowchart of neural human prediction according to some embodiments.
  • an input set I of images a single image or multiview images, e.g., a set of pictures taken around a subject, are acquired as input.
  • the input I is denoted as a 4D tensor of size N x w x h x c, N for number of views, w, h, c for image width, height and channel, respectively.
  • the cameras are already calibrated, so all of the camera information (e.g., camera parameters) is known.
  • An image preprocess extracts the subject’s bounding box and foreground mask using existing approaches such as Detectron2 and image Grab-Cut. Images are cropped by the bounding box and zoomed to size of w x h with the same aspect ratio. Image borders are filled in black.
  • the neural network (MVS-PERF) 102 is comprised of three components: a multiview stereo 3D convolutional neural network (MVS-3DCNN) 104, which encodes an input set of images to features; a human mesh recovery multilayer perceptron (HMR MLP) 106, which regresses the features to human parameters; and a neural radiance field multilayer perceptron (NeRF MLP) 108, which fine-tunes the MVS-3DCNN and decodes a query 3D ray (3D location and direction) to an RGB color and a clothes-to-body displacement.
  • a multiview stereo 3D convolutional neural network (MVS-3DCNN) 104, which encodes an input set of images to features
  • HMR MLP human mesh recovery multilayer perceptron
  • NeRF MLP neural radiance field multilayer perceptron
  • a deep 2D CNN extracts image features from each view.
  • Each convolutional layer is followed by a batch-normalization (BN) layer and a rectified linear unit (ReLU) except for the last layer.
  • BN batch-normalization
  • ReLU rectified linear unit
  • Two downsampling layers are also placed.
  • the output of the 2D CNN are a feature map of size w/4 x h/4 x 32.
  • a view is first chosen as a reference view and its view frustum is set according to perspective projection and near far planes to cover the entire working space of the subject. From near to far, the frustrum is sampled by d depth planes which are parallel to both near and far planes. All the feature maps are transformed and blended to each depth plane.
  • K [R, /] stand for the camera intrinsic and extrinsic parameters
  • z is the distance from a depth plane to the camera center of the reference view
  • n is the normal direction of the depth plane.
  • the human mesh recovery multilayer perceptron includes three layers of linear regression separated by flatten and dropout layers. It regresses the feature volume from MVS 3DCNN to the human body parameter
  • Human body parameter is able to manipulate a human parametric model, e.g., SMPL- X, to a 3D naked body mesh 202.
  • a SMPL-X representation of contains the skeletal poses (the 3D rotation angles of each joint), the body blendshape parameter to control the body shape, e.g., height, weight, and others, and the facial blendshape parameter to control the expression of the face. It builds a T-pose mesh using blendshape parameters and deforms it to a posed mesh by the skeletal pose of a linear skinning model.
  • the cost volume is sent to a differentiable rendering MLP, such as neural radiance field (NeRF).
  • NeRF MLP is formularized as a functional M that maps a query ray, represented by a 3D position x and a direction f, to a 4-channel color is the feature map from the cost volume of the frustum MVS
  • 3DCNN 104 to the NeRF volume
  • G is the weight of the NeRF MLP network
  • s denotes the occupancy density of a probability if the 3D point is inside a mesh.
  • the occupancy density field of a naked body can be directly obtained by converting the mesh 202 (Fig. 2) in the frustum 104. Then the density field s of clothed body can be represented as a function of a 3D displacement vector field D and the feature map
  • the 3D displacement vector field D 116 represents how a point on the clothed body surface 204 is related to a point on the naked body surface. When NeRF MLP is trained, the displacement vector field D is also optimized.
  • Figure 2 illustrates the workflow of a forward prediction represented by the tensor notations, in which the weight of all the networks MVS 3DCNN, HMR MLP and NeRF MLP are trained and fixed, according to some embodiments.
  • the appearance image 112 is rendered.
  • 3D human prediction 110 is implemented.
  • the displacement field D 116 is obtained.
  • the naked body mesh V b 202 can be deformed to a clothed body mesh V c 204 by adding an interpolated displacement vector to each vertex.
  • Figure 3 illustrates the workflow of training the network using supervision according to some embodiments.
  • a supervised training dataset e.g., Human3.6M
  • a shape loss 304 is directly obtained by summing the difference of the predicted naked body and the ground truth.
  • J are the joints of the naked body
  • P denotes the perspective projection of a 3D point for each camera view.
  • rays 306 are sampled from the input image set 100, typically using an uneven sampling strategy proportional to the image saliency. More rays are sampled in high salient regions and fewer rays are from plain or background regions. These rays are sent together with the feature map from MVS 3DCNN 104 into the NeRF MLP 106, which renders the samples appearance RGB colors 308. A color loss 310 is computed by summing all the difference of sampled color in the input image and the rendered colors 308.
  • Figure 4 illustrates the workflow of training the network in a self-improving strategy according to some embodiments.
  • the training dataset only provides human images without any annotation or human ground truth parameters.
  • an optimization-based prediction 400 e.g., SMPLifyX algorithm
  • the optimization-based prediction detects human 2D key points on each image first and applies a nonlinear optimization to fit the 3D human.
  • K denotes the detected 2D location of a key point, and the sum takes over all the corresponding key points and all the views.
  • the neural human prediction is able to be directly applied in both commercial and/or personal markerless performance capture applications, for example, a markerless motion capture in game studio, or human 3D surface reconstruction RGB camera setup.
  • Other applications of embodiments of the multiview neural human prediction are able to be as a real-time backbone technique able to be combined with any extension, for example, combining the input of depth sensing, 3D modeling, or using the output for creating novel animation.
  • Multiview neural human prediction is also able to be applied in gaming, VR/AR and any real-time human interactive applications.
  • suitable computing devices include a personal computer, a laptop computer, a computer workstation, a server, a mainframe computer, a handheld computer, a personal digital assistant, a cellular/mobile telephone, a smart appliance, a gaming console, a digital camera, a digital camcorder, a camera phone, a smart phone, a portable music player, a tablet computer, a mobile device, a video player, a video disc writer/player (e.g., DVD writer/player, high definition disc writer/player, ultra high definition disc writer/player), a television, a home entertainment system, an augmented reality device, a virtual reality device, smart jewelry (e.g., smart watch), a vehicle (e.g., a self-driving vehicle) or any other suitable computing device.
  • An apparatus comprising: a non-transitory memory configured for storing an application, the application configured for: acquiring a set of images as input; and processing, with a neural network, the set of images, wherein processing includes: encoding the set of images to one or more features; regressing the features to human parameters; fine-tuning the neural network; and decoding a query 3D ray to an RGB color and a clothes-to-body displacement, wherein the RGB color is based on the set of images; and a processor configured for processing the application.
  • the set of images comprises a 4D tensor of size N x w x h x c, where N is a number of views, w is width of an image, h is height of the image, and c is a channel of the image.
  • the neural network chooses a frontal view as a reference view from the set of images and extracts a feature volume.
  • An apparatus comprising: a non-transitory memory configured for storing an application, the application comprising: a multiview stereo 3D convolutional neural network (MVS-3DCNN) configured for encoding an input image set to features; a human mesh recovery multilayer perceptron (HMR MLP) configured for regressing the features to human parameters; and a neural radiance field multilayer perceptron (NeRF MLP) configured for fine-tuning the MVS-3DCNN and decodes a query 3D ray (3D location and direction) to an RGB color and a clothes-to-body displacement; and a processor configured for processing the application.
  • MVS-3DCNN multiview stereo 3D convolutional neural network
  • HMR MLP human mesh recovery multilayer perceptron
  • NeRF MLP neural radiance field multilayer perceptron

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Multimedia (AREA)
  • Software Systems (AREA)
  • Evolutionary Computation (AREA)
  • General Health & Medical Sciences (AREA)
  • Health & Medical Sciences (AREA)
  • Computer Graphics (AREA)
  • Artificial Intelligence (AREA)
  • Databases & Information Systems (AREA)
  • Computing Systems (AREA)
  • Medical Informatics (AREA)
  • Human Computer Interaction (AREA)
  • Geometry (AREA)
  • Oral & Maxillofacial Surgery (AREA)
  • Image Analysis (AREA)
  • Processing Or Creating Images (AREA)
  • Image Processing (AREA)

Abstract

Une structure de capture de performance humaine neuronale (MVS-PERF) capture le squelette, la forme du corps et un déplacement de vêtements, et l'apparence d'une personne à partir d'un ensemble d'images multivues calibrées. Elle aborde l'ambiguïté de prédiction de la position absolue dans la récupération d'un maillage humain monoculaire, et relie la représentation volumétrique de NeRF à une capture de performance adaptée à l'animation. La structure MVS-PERF comprend trois modules pour extraire des cartes de caractéristiques d'images multivues et les fusionner en un volume de caractéristiques, régresser le volume de caractéristiques à un vecteur de paramètres humains nus, générer un maillage de corps moulant SMPL-X avec une pose du squelette, une forme du corps et une expression, et utiliser un champ de radiance neuronale et un champ de déformation pour inférer les vêtements en tant que le déplacement sur le corps nu à l'aide d'un rendu différentiable. Un maillage de corps vêtu est obtenu par ajout des vecteurs de déplacement interpolés aux sommets de maillage de corps moulant SMPL-X. Le champ de radiance obtenu est utilisé pour un rendu volumétrique exempt de vue du sujet d'entrée.
EP22715732.8A 2021-03-31 2022-03-31 Prédiction humaine neuronale multivue à l?aide d?un moteur de rendu différentiable implicite pour l?expression faciale, la forme et la pose du corps, et la capture de performance de vêtements Pending EP4292059A1 (fr)

Applications Claiming Priority (4)

Application Number Priority Date Filing Date Title
US202163168467P 2021-03-31 2021-03-31
US202163279916P 2021-11-16 2021-11-16
US17/701,991 US11961266B2 (en) 2021-03-31 2022-03-23 Multiview neural human prediction using implicit differentiable renderer for facial expression, body pose shape and clothes performance capture
PCT/IB2022/053034 WO2022208440A1 (fr) 2021-03-31 2022-03-31 Prédiction humaine neuronale multivue à l'aide d'un moteur de rendu différentiable implicite pour l'expression faciale, la forme et la pose du corps, et la capture de performance de vêtements

Publications (1)

Publication Number Publication Date
EP4292059A1 true EP4292059A1 (fr) 2023-12-20

Family

ID=81328451

Family Applications (1)

Application Number Title Priority Date Filing Date
EP22715732.8A Pending EP4292059A1 (fr) 2021-03-31 2022-03-31 Prédiction humaine neuronale multivue à l?aide d?un moteur de rendu différentiable implicite pour l?expression faciale, la forme et la pose du corps, et la capture de performance de vêtements

Country Status (5)

Country Link
EP (1) EP4292059A1 (fr)
JP (1) JP2024510230A (fr)
KR (1) KR20230150867A (fr)
CN (1) CN116134491A (fr)
WO (1) WO2022208440A1 (fr)

Families Citing this family (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN116758202A (zh) * 2023-03-14 2023-09-15 中国科学院深圳先进技术研究院 人手图像合成方法、装置、电子设备及存储介质
CN116824092B (zh) * 2023-08-28 2023-12-19 深圳星坊科技有限公司 三维模型生成方法、装置、计算机设备和存储介质
CN117238420A (zh) * 2023-11-14 2023-12-15 太原理工大学 一种极薄带力学性能预测方法及装置

Also Published As

Publication number Publication date
CN116134491A (zh) 2023-05-16
KR20230150867A (ko) 2023-10-31
JP2024510230A (ja) 2024-03-06
WO2022208440A1 (fr) 2022-10-06

Similar Documents

Publication Publication Date Title
US11961266B2 (en) Multiview neural human prediction using implicit differentiable renderer for facial expression, body pose shape and clothes performance capture
Li et al. Monocular real-time volumetric performance capture
US11941831B2 (en) Depth estimation
US20240005590A1 (en) Deformable neural radiance fields
KR20210042942A (ko) 비디오 데이터를 이용한 객체 인스턴스 매핑
WO2022208440A1 (fr) Prédiction humaine neuronale multivue à l'aide d'un moteur de rendu différentiable implicite pour l'expression faciale, la forme et la pose du corps, et la capture de performance de vêtements
KR102141319B1 (ko) 다시점 360도 영상의 초해상화 방법 및 영상처리장치
CN113689578B (zh) 一种人体数据集生成方法及装置
US20230130281A1 (en) Figure-Ground Neural Radiance Fields For Three-Dimensional Object Category Modelling
US20210374986A1 (en) Image processing to determine object thickness
CN116958492B (zh) 一种基于NeRf重建三维底座场景渲染的VR编辑方法
GB2567245A (en) Methods and apparatuses for depth rectification processing
CN117542122B (zh) 人体位姿估计与三维重建方法、网络训练方法及装置
CN117557714A (zh) 三维重建方法、电子设备及可读存储介质
CN113850900A (zh) 三维重建中基于图像和几何线索恢复深度图的方法及系统
GB2571307A (en) 3D skeleton reconstruction from images using volumic probability data
CN116310228A (zh) 一种针对遥感场景的表面重建与新视图合成方法
CN111783497A (zh) 视频中目标的特征确定方法、装置和计算机可读存储介质
CN116797713A (zh) 一种三维重建方法和终端设备
CN113570673B (zh) 三维人体和物体的渲染方法及其应用方法
CN116433852B (zh) 数据处理方法、装置、设备及存储介质
WO2023132261A1 (fr) Système de traitement d'informations, procédé de traitement d'informations et programme de traitement d'informations
CN116681818B (zh) 新视角重建方法、新视角重建网络的训练方法及装置
US20230177722A1 (en) Apparatus and method with object posture estimating
Johnston Single View 3D Reconstruction using Deep Learning

Legal Events

Date Code Title Description
STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: UNKNOWN

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: THE INTERNATIONAL PUBLICATION HAS BEEN MADE

PUAI Public reference made under article 153(3) epc to a published international application that has entered the european phase

Free format text: ORIGINAL CODE: 0009012

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: REQUEST FOR EXAMINATION WAS MADE

17P Request for examination filed

Effective date: 20230912

AK Designated contracting states

Kind code of ref document: A1

Designated state(s): AL AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HR HU IE IS IT LI LT LU LV MC MK MT NL NO PL PT RO RS SE SI SK SM TR

DAV Request for validation of the european patent (deleted)
DAX Request for extension of the european patent (deleted)