Classifications

• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
  • G06T17/00—Three-dimensional [3D] modelling for computer graphics
  • G06T17/20—Finite element generation, e.g. wire-frame surface description, tesselation
  • G06T17/205—Re-meshing
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
  • G06T17/00—Three-dimensional [3D] modelling for computer graphics
  • G06T17/20—Finite element generation, e.g. wire-frame surface description, tesselation
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
  • G06T19/00—Manipulating three-dimensional [3D] models or images for computer graphics
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
  • G06T2207/00—Indexing scheme for image analysis or image enhancement
  • G06T2207/20—Special algorithmic details
  • G06T2207/20112—Image segmentation details
  • G06T2207/20116—Active contour; Active surface; Snakes
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
  • G06T2210/00—Indexing scheme for image generation or computer graphics
  • G06T2210/44—Morphing
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
  • G06T2219/00—Indexing scheme for manipulating 3D models or images for computer graphics
  • G06T2219/021—Flattening

Definitions

• the present invention relates to a method and system for producing a three-dimensional deformable model of an element.
• deformable models are also used in three-dimensional or 3D animation (Bao-Cai Yin's "Mpeg-4 compatible 3D facial animation based on morphable model", Cheng-Zhang Wang, Qin Shi, and Yan- Feng Sun, In Machine Learning and Cybernetics, 2005, Proceedings of 2005 International Conference on, volume 8, pages 4936 ⁇ 1941, Vol 8, Aug 2005, and "Statistical Generation of 3D Facial Animation Models” by Rudomin, A. Bojorquez, and H.
• step 3 Creation of a vector space specific to the study subject by the use of a statistical analysis method such as PCA (Principal Component Analysis), ACI (Independent Component Analysis) or their derivatives .
• PCA Principal Component Analysis
• ACI Independent Component Analysis
• the last step 3) of this process gives rise in particular to a so-called mean vector and modes of deformation, the linear combinations of which make it possible not only to reform the learning examples but also to generate new elements (new ones). faces in the case of a deformable model of faces for example).
• Blanz and Vetter have proposed the use of an algorithm of flow or optical flow or "Optical Flow Algorithm" in English ("Hierarchical motion-based frame rate conversion” of James R Bergen and R Hingorani, Technical report David Sarno Research Center, 1990).
• the laser used for scanning acquisitions, or "scanning" in English provided a cylindrical representation (also called 2.5 D).
• a two-dimensional or 2D image of the texture was immediately available and used for the implementation of the aforementioned algorithm.
• this algorithm requires that the deformations from one example to the other are weak (like successive images of a video), which has no problem. reason to be in the general case.
• the cylindrical representations have the major disadvantage of being generating occlusions. While these are relatively rare in the case of faces, making the Blanz and Vetter method usable, it is different in the case of more complex forms, such as ears, for which the loss of information can be prohibitive.
• Chen Li and associates take advantage of the particular form of their subject of study, in this case the ear, and data available, in this case a photo and a depth map of the ear profile view, to develop the hierarchical growth algorithm of mesh triangles or "Triangle Mesh Hierarchical Growth” in English language ("A novel 3D ear reconstruction method using a single image” by Chen Li, Zhichun Mu, Feng Zhang, and Shuai Wang, in Intelligent Control and Automation (WCICA), 2012 10th World Congress, pages 4891 -4896, IEEE, 2012).
• a depth map also called a 2.5 D image or "z map” is a pixel-based z coordinate image, which is usually created using a 3-D camera. The gray levels in the depth map represents the height values.
• the matching method is based on global and non-local geometric considerations, such as the intersection of a right from one end of the image with a curve at the other end, it dilutes or totally loses the semantic information conveyed by the image.
• characteristic points of the ear such as tragus or anti-tragus can not be associated with one or more of the constructed descriptive points.
• Kaneko et al. (Shoken Kaneko's Ear shape modeling for 3D audio and acoustic virtual reality: The shapebased average hrtf ", Tsukasa Suenaga, Fujiwara Mai, Kazuya Kumehara, Futoshi Shirakihara, and SaSatoshi Sekine, Audio Engineering Society Conference: 61 st International Conference Audio for Games, Audio Engineering Society, 201 6.) uses X-ray scans or scans of volunteer ear molds and favors the use of non-rigid 3D mapping methods ("A new Haili Chui and Anand Rangarajan, Computer Vision and Image Understanding, 89 (2): 1 14-141, 2003, "Bing Jian and Baba C, point-to-point matching algorithm for non-rigid registration" Vemuri Pattern Analysis and Machine Intelligence, IEEE Transactions on, 33 (8): 1 633-1 645, 201 1.).
• the meshes are composed of about 3000 vertices and the vectors of deformation transforming a mesh of reference in the others of the base are sought with the aid of mixtures of Gaussians.
• a method for producing a three-dimensional deformable model of an element from an initial basis of examples of such elements provided with data for determining, for each of the elements of the initial base, a three-dimensional mesh surface based on points and a triangular network connecting said points, wherein: for each element example of the initial base, at each point of its mesh surface, the value of at least one parameter representative of the shape of the surface of the element at this point is determined (by measurement or calculation), to obtain an improved base of examples of elements;
• the mesh surface is flattened in three dimensions to obtain a two-dimensional representation of said mesh surface; on the set of two-dimensional representations of the meshed surfaces of said elements, a plurality of respective points is matched by using said determined values of the parameter or parameters representative of the shape of the surface of the element at said points and a method of analyzing said two-dimensional representations of the meshed surfaces;
• the invention is not dependent on the presence of texture information and can handle data sets that are devoid of such data, such as MRI results.
• said one or more parameters representative of the shape of the surface of the element at a point of the mesh surface of an example of the initial base comprise a local curvature at said point and / or a descriptor of form at that point.
• said local curvature comprises a minimum curvature and / or a maximum curvature and / or a Gaussian curvature and / or a mean curvature.
• the shape descriptor comprises a shape correction surface patch histogram or SPHIS for the acronym "Surface Patch Histogram of Index Shape" in the English language.
• the method may be configured to detect the more or less pronounced presence of one or more types of shapes rather than merely measuring the curvature.
• said flattening uses an ABF, LSCM, ABF ++, or H LSCM method.
• said mapping uses a division of the two-dimensional representations into N c curvature levels distributed uniformly over the range of values taken by the values of the parameter or parameters representative of the shape of the surface. of the element.
• two-dimensional representations can be segmented according to objective and reproducible criteria.
• said mapping uses a division of the two-dimensional representations into N c levels of curvature account of the statistical distribution of the values taken by the values of the parameter or parameters representative of the shape of the surface of the element.
• mapping utilizes a number N TM anu points mapped manually.
• the operator precisely controls their positioning. This is particularly useful when building small models (for testing purposes or for lack of learning examples).
• the method is semi-automatic, and progressively from said mapping, the number N TM anu points mapped manually the current element decreases with the number of items processed.
• the method is automatic based on active contours, and the number N TM anu of manually mapped points is zero.
• said element is a right ear and / or a left ear, and / or the head, and / or the torso of an individual.
• a system for developing a three-dimensional deformable model of an element from an initial basis of examples of such elements provided with data for determining, for each of the elements of the initial base, a three-dimensional mesh surface based on points and a triangular network connecting said points, comprising a computer configured to implement the method as previously described.
• FIGS. 1 to 7 schematically illustrate a method according to one aspect of the applied invention; to human ears;
• FIG. 8 to 10 schematically illustrate a method according to one aspect of the invention applied to human faces.
• the present invention is an alternative to the aforementioned methods and allows the creation of a deformable model of any type of subject or element from the study of its morphology.
• the present invention does not require any texture information and thus effectively avoids the pose and illumination problems experienced by the flow or optical flow algorithms, such as the algorithms called structure from motion or SFM for acronym for "structure from motion” in English or algorithms called structure from shadows or SFS for acronym for “structure from shading” in English.
• the invention makes it possible to adapt naturally to three-dimensional or 3D data as to those in 2.5D.
• the present invention makes it possible to preserve the semantic information, or in other words to preserve the physical meaning conveyed by a zone, a group of vertices or even a single vertex.
• the vertices composing the nose of the middle form will also compose the nose of any face of the model after deformation. This observation is also valid for substructures as in the present case: the tip of the nose, the right and left nostrils or the ridge.
• Figure 1 shows the major steps of the method according to one aspect of the invention.
• FIG. 1 illustrates a method 1 for producing a three-dimensional deformable model of an element from an initial base of examples, loaded 2 in the computer means implementing the method.
• elements provided with data for determining, for each of the elements of the initial base, a three-dimensional mesh surface based on points and a triangular network connecting said points, wherein:
• the value of at least one parameter representative of the shape of the surface of the element is determined 3 (by measurement or calculation) point, to obtain an improved base of examples of elements;
• the flat surface is flattened or unfolded in three dimensions to obtain a two-dimensional representation of said mesh surface
• Each sample element of the initial database can be subsampled.
• Figures 2a and 2b show an example of a model obtained from an ear base.
• Figure 2a is shown the non-meshing middle ear
• Figure 2b is shown the same deformed ear according to the third mode of deformation. In this case it is about straight ears.
• each mode of deformation represents a set of displacements type undergone by the elements of the point cloud. It is possible to see these types of displacement as the data of a direction and a speed of movement for each point.
• the data of a multiplicative coefficient which could be assimilated to a duration in the previous analogy, makes it possible to calculate the exact displacement.
• the present model thus makes it possible to highlight physical substructures of the ear which tend to evolve together (or on the contrary separately if one works by complementarity).
• the gray level of each point is associated with its deviation from its position in the average shape (the higher the gray level, the larger the difference).
• a base of training examples is supposed to be available, each of the examples allowing, directly or after treatments, the reconstruction of a meshed surface in R 3 .
• the result of each measurement is associated with the point used for its realization.
• This measurement can be summarized as a local curvature, as shown in Figure 3, such as the minimum, maximum, Gaussian or average curvature at the point considered, to a more complex shape descriptor such as an index surface patch histogram. form or SPHIS for acronym for "Surface Patch Histogram of Index Shape" in English, or an equivalent or a combination of the above.
• the 3D ear of Figure 3 at gray levels that depend on the local mean curvature, the higher the local average curvature, the darker the gray is.
• FIGS. 4a and 4b An unfolding of the surface which thus makes it possible to obtain a representation of each mesh in the form of a 2D image, as illustrated in FIGS. 4a and 4b, respectively corresponding to the left and right ears of subject 9 after calculation of curvature and flattening, noted im2D and a connected graph, classically noted Gc but not shown.
• the gray levels are such that the greater the local average curvature, the darker is the gray.
• This unfolding also called “unwrapping” in English, can be done in many ways, as with flattening algorithms based on angles, with ABF acronym for "Angle-based Flattening” in English, algorithms of least significant LSCM acronym for Least Square Conformai Maps in English, or their derivatives (ABF ++, HSLCM for "Hierarchical Least Square Conformai Maps” in English, ).
• a mapping of a maximum of points is performed from 2D images using the characteristics measured in point 2) and the analysis methods related to 2D image processing, as shown in Figure 5.
• Figure 5 On Figure 5 are shown on the left in 2D and on the right in 3D the same ear after manual mapping of 88 points according to the isocurbures 9, which are materialized on the left 2D image.
• the gray levels are such that the higher the average local curvature, the darker the gray.
• the points retained during the matching are then used to downsample the initial 3D meshes.
• the resulting point clouds are then used to build the actual model using conventional construction tools such as principal component analysis, acronym ACP or ACI acronym. .).
• the base used consists of the ten freely accessible examples of the SYMARE database for "Sydney-York Morphological And Recording of Ears" in English.
• all the left ears of these ten pairs of ears have been symmetrized with respect to the sagittal plane so as to have twenty straight ears (the ten initial lines and the ten lines coming from the symmetrization of the ten left ones).
• the meshes of the ears thus obtained are subsampled to about 6900 vertices.
• This step purely optional, is present in order to optimize the digital processing times and to facilitate the subsequent integration of possible other learning examples.
• Point 2 of the description of the invention is then carried out.
• the local mean curvature was retained as a geometric feature and applied as texture to the 3D meshes, as shown in Figure 3.
• mapping of the vertices of the connected graphs required the following steps:
• n c isocurrent lines, as illustrated in the example of Figure 6, with n c G [1, N C ], themselves chosen to ensure a uniform distribution of vertices.
• Tj ⁇ tj, iel ⁇ all of my triangles.
• Each element tj j potentially contains vertices of the graph whose barycentric coordinates are calculated in the reference proper to the triangle tj.
• N c and N s mami can of course be set to other values.
• the curvature levels may not be uniformly distributed over the available range but take into account the statistical distribution of curvature values.
• the N s mami points selected in step 2) of the mapping can be automatically or semi-automatically, for example:
• the method is applied to human faces.
• the face database used in this example consists of Examples 2, 5, 6 and 14 of the UWA face database 3D database of the University of Western Australia (UWA), available at next address:
• Step 2 of the description of the invention is then carried out following the same methodology as for the example described based on ears.
• the steps of the method described above may be performed by one or more programmable processors executing a computer program for performing the functions of the invention by operating on input data and generating output data.
• a computer program can be written in any form of programming language, including compiled or interpreted languages, and the computer program can be deployed in any form, including as a stand-alone program or as a subroutine, element or other unit suitable for use in a computing environment.
• a computer program can be deployed to run on one computer or multiple computers at a single site or spread across multiple sites and interconnected by a communications network.

Landscapes

• Engineering & Computer Science (AREA)
• Physics & Mathematics (AREA)
• Computer Graphics (AREA)
• Software Systems (AREA)
• General Physics & Mathematics (AREA)
• Theoretical Computer Science (AREA)
• Geometry (AREA)
• Computer Hardware Design (AREA)
• General Engineering & Computer Science (AREA)
• Processing Or Creating Images (AREA)
• Image Processing (AREA)

Applications Claiming Priority (2)

Publications (1)

Family

ID=56411768

Family Applications (1)

Country Status (5)

Families Citing this family (11)

Family Cites Families (7)

• 2016
  • 2016-05-27 FR FR1654765A patent/FR3051951B1/fr active Active
• 2017
  • 2017-05-15 US US16/300,044 patent/US10489977B2/en active Active
  • 2017-05-15 CN CN201780032523.6A patent/CN109844818B/zh active Active
  • 2017-05-15 WO PCT/EP2017/061607 patent/WO2017202634A1/fr not_active Ceased
  • 2017-05-15 EP EP17723989.4A patent/EP3465629A1/de not_active Ceased
• 2019
  • 2019-10-18 US US16/656,993 patent/US10762704B2/en active Active

Also Published As

Similar Documents

Legal Events