CN117237529A - 4D face data acquisition method and device - Google Patents
4D face data acquisition method and device Download PDFInfo
- Publication number
- CN117237529A CN117237529A CN202311095187.3A CN202311095187A CN117237529A CN 117237529 A CN117237529 A CN 117237529A CN 202311095187 A CN202311095187 A CN 202311095187A CN 117237529 A CN117237529 A CN 117237529A
- Authority
- CN
- China
- Prior art keywords
- face
- dimensional
- data
- groups
- point cloud
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 43
- 230000008859 change Effects 0.000 claims abstract description 13
- 238000013507 mapping Methods 0.000 claims description 39
- 238000012545 processing Methods 0.000 claims description 33
- 239000011159 matrix material Substances 0.000 claims description 17
- 238000013519 translation Methods 0.000 claims description 17
- 238000009499 grossing Methods 0.000 claims description 15
- 241000746998 Tragus Species 0.000 claims description 12
- 238000013480 data collection Methods 0.000 claims description 5
- 230000008569 process Effects 0.000 claims description 5
- 239000013598 vector Substances 0.000 claims description 5
- 238000006073 displacement reaction Methods 0.000 claims description 3
- 239000000203 mixture Substances 0.000 claims description 3
- 230000007704 transition Effects 0.000 claims description 3
- 238000003384 imaging method Methods 0.000 abstract description 4
- 230000014509 gene expression Effects 0.000 description 8
- 238000005516 engineering process Methods 0.000 description 7
- 230000003068 static effect Effects 0.000 description 7
- 230000000694 effects Effects 0.000 description 3
- 239000011664 nicotinic acid Substances 0.000 description 3
- 238000013473 artificial intelligence Methods 0.000 description 2
- 238000013499 data model Methods 0.000 description 2
- 238000011161 development Methods 0.000 description 2
- 238000001914 filtration Methods 0.000 description 2
- 238000004458 analytical method Methods 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 238000004364 calculation method Methods 0.000 description 1
- 239000004053 dental implant Substances 0.000 description 1
- 238000013461 design Methods 0.000 description 1
- 238000003745 diagnosis Methods 0.000 description 1
- 238000002059 diagnostic imaging Methods 0.000 description 1
- 238000010586 diagram Methods 0.000 description 1
- 230000006872 improvement Effects 0.000 description 1
- 230000001788 irregular Effects 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 210000003205 muscle Anatomy 0.000 description 1
- 238000000547 structure data Methods 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
- 230000001360 synchronised effect Effects 0.000 description 1
- 238000012549 training Methods 0.000 description 1
- 230000000007 visual effect Effects 0.000 description 1
Landscapes
- Image Processing (AREA)
Abstract
The invention provides a 4D face data acquisition method and a device, the 4D face data acquisition method is adapted to a corresponding 4D face data acquisition device, the 4D face data acquisition device comprises 3 groups of three-dimensional acquisition modules, the 3 groups of three-dimensional acquisition modules are distributed in a delta shape, each group of three-dimensional acquisition modules comprises a structure light unit and a camera unit, the light wave frequency ranges of the structure light units of the 3 groups of three-dimensional acquisition modules are different, the 4D face data acquisition method is based on the 4D face data acquisition device which is provided with 3 groups of three-dimensional acquisition modules, the 3 groups of three-dimensional acquisition modules work in the unique light wave frequency ranges, the 3 groups of three-dimensional acquisition modules do not interfere with each other when in work, so that time-sharing multiple scanning is not needed, the 3 groups of three-dimensional acquisition modules work simultaneously, the scanning time is greatly reduced, the scanning efficiency is improved, the continuous scanning imaging of the three-dimensional face is realized, and the change range of the surface condition does not need to be strictly required.
Description
Technical Field
The invention relates to the technical field of face acquisition, in particular to a 4D face data acquisition method and device.
Background
In order to acquire a full face surface three-dimensional data model of a face, a three-dimensional scanning device is required to perform three-dimensional face scanning, and the existing dynamic 4D face acquisition technology mainly has two implementation ideas. One is static three-dimensional scanning, then the static three-dimensional model is subjected to bionic traction and deformation by means of some artificial intelligence methods, and a dynamic effect is simulated. The other idea is to perform static three-dimensional scanning for multiple times, obtain a three-dimensional model each time, match and align the obtained multiple sets of three-dimensional digital models in software of a third party after multiple times of scanning, and then sequentially display the three-dimensional models according to time.
The bionic traction static three-dimensional model simulates a dynamic effect, and only the original model which is collected statically is accurate three-dimensional data of a patient, and the data generated by deformation are inaccurate although the dynamic effect exists, so that bionic reality only depends on the prediction of an artificial intelligence algorithm, accurate data cannot be provided for doctors, and the significance of clinical reference is not great.
According to the multi-time static collection method of different expressions, each three-dimensional model in a final sequence is derived from collected original data, accuracy is high, but a collection process is complex and difficult to operate, specifically, each static scanning requires that a scanned target keeps one stiff expression, and another stiff expression is replaced after scanning and three-dimensional reconstruction are completed, and for continuity of a subsequent three-dimensional model sequence, the change amplitude of each expression of the target is required to be small enough, and each expression needs to be kept for a short period of time. This is hardly possible for a normal patient seeking a diagnosis without specialized training.
Accordingly, the prior art is still in need of improvement and development.
Disclosure of Invention
The invention provides a 4D face data acquisition method and device, and mainly aims to solve the technical problems in the prior art.
The first aspect of the invention provides a 4D face data acquisition method, which comprises the following steps:
the method comprises the steps that a 4D face data acquisition device is provided, the 4D face data acquisition device comprises 3 groups of three-dimensional acquisition modules, the 3 groups of three-dimensional acquisition modules are distributed in a delta shape in a overlooking view, each group of three-dimensional acquisition modules comprises a structure light unit and a camera unit, and the light wave frequency ranges of the structure light units of the 3 groups of three-dimensional acquisition modules are different;
simultaneously carrying out three-dimensional dynamic scanning and photographing on the front face of the face by using 3 groups of three-dimensional acquisition modules of the 4D face data acquisition device to obtain a plurality of groups of face three-dimensional point cloud data and a plurality of groups of face map data which dynamically change under a time scale;
carrying out gridding treatment on each group of face three-dimensional point cloud data to obtain face gridding data;
mapping processing corresponding to the face mapping data is carried out on each group of face gridding data, and a three-dimensional face model is obtained;
and forming a dynamic sequence by the obtained multiple groups of three-dimensional face models based on a time scale to obtain a 4D face model.
In an optional implementation manner of the first aspect of the present invention, after the mapping processing corresponding to the face mapping data is performed on each set of the face gridding data to obtain a three-dimensional face model, the forming a dynamic sequence by using the obtained plurality of sets of three-dimensional face models based on a time scale, and before obtaining a 4D face model, the method further includes:
and in the three-dimensional space, adjusting the space pose of the first group of three-dimensional face models on a time scale by utilizing the space key point information of face recognition, obtaining a rotation translation matrix, and applying the rotation translation matrix to the subsequent groups of three-dimensional face models.
In an optional implementation manner of the first aspect of the present invention, in the three-dimensional space, adjusting a spatial pose of the first group of three-dimensional face models on a time scale by using spatial key point information of face recognition, obtaining a rotation translation matrix, and applying the rotation translation matrix to each subsequent group of three-dimensional face models includes:
taking the mass center of the first group of three-dimensional face models on the time scale as the origin of coordinates of a world coordinate system;
two non-collinear space vectors formed by the left and right tragus points to the corresponding eye lower frame points form an XY plane;
taking the normal direction of the XY plane pointing to the top of the head as a Z axis, and taking the left tragus point to the right tragus point as an X axis;
taking a Cartesian right-hand coordinate system as a reference, wherein the directions of the Cartesian right-hand coordinate system, which are perpendicular to an X axis and a Z axis, and pointing to the nose tip point from the origin of coordinates are Y axes;
and respectively calculating the relation between three axes and the origin of coordinates by using the model coordinates of the key points on the three-dimensional face model to obtain a rotation translation matrix of the three-dimensional face model changing to the world coordinate system.
In an optional implementation manner of the first aspect of the present invention, the performing meshing processing on each set of three-dimensional point cloud data of the face, before obtaining the face meshing data, includes:
performing outlier removal processing on each group of face three-dimensional point cloud data;
and smoothing the three-dimensional point cloud data of the face remained after the outlier is removed.
In an optional implementation manner of the first aspect of the present invention, the processing for removing outliers on each set of three-dimensional point cloud data of the face includes:
in the three-dimensional space, taking each point in the three-dimensional point cloud data of the human face as a center, and counting each point in the three-dimensional space as r 1 The number of other points present in the range;
for each point, if the radius r 1 If the number of other points in the space range is smaller than a preset experience threshold, judging that the point is an outlier point and needing to be removed from the three-dimensional point cloud data of the face.
In an optional implementation manner of the first aspect of the present invention, the smoothing the three-dimensional point cloud data of the face remaining after the outlier removal includes:
for each point in the three-dimensional point cloud data of the face, defining a radius r by taking each point in the three-dimensional point cloud data of the face as a center in a three-dimensional space 2 Is a smoothing region of (a);
judging the composition linear queue of each direction for all points in each smoothing processing area, and finding out the point deviating from the queue;
and performing displacement compensation on the points deviated from the queue so as to smoothly transition all the points in the three-dimensional point cloud data of the human face.
In an optional implementation manner of the first aspect of the present invention, the performing meshing processing on each set of three-dimensional point cloud data of the face to obtain face meshing data includes:
forming a triangle by every three points in the three-dimensional point cloud data of the human face;
and (3) orderly arranging and connecting the obtained triangles in a common point or common edge mode to form a netlike continuous surface so as to obtain face meshing data.
In an optional implementation manner of the first aspect of the present invention, the mapping processing corresponding to the face mapping data is performed on each set of face gridding data, and obtaining the three-dimensional face model includes:
according to the arrangement mode of camera coordinates, preliminarily aligning the face mapping data with the face gridding data, and then carrying out reprojection;
acquiring corresponding coordinates of each point in the face meshing data on the face mapping data, and grouping the acquired corresponding coordinate sets according to triangles in the face meshing data;
traversing all triangles in the face gridding data, and remapping each group of corresponding coordinates obtained by grouping to three vertexes of the triangle;
and adjusting the area texture of the inner area of the triangle in a linear stretching or compressing mode to finish the texture mapping of the face gridding data, thereby obtaining the three-dimensional face model.
In an optional implementation manner of the first aspect of the present invention, the 4D face data collecting device further includes a base and a remote control lifting module installed on the base, the remote control lifting module is provided with a door-shaped auxiliary support, 3 groups of three-dimensional collecting modules are installed on the remote control lifting module and two side arms of the door-shaped auxiliary support respectively, and a display control module is further installed on the remote control lifting module, and is used for operably displaying the 4D face model.
The invention provides a 4D face data acquisition device, which comprises 3 groups of three-dimensional acquisition modules, wherein the 3 groups of three-dimensional acquisition modules are distributed in a delta shape in a overlook view, each group of three-dimensional acquisition modules comprises a structure light unit and a camera unit, and the light wave frequency ranges of the structure light units of the 3 groups of three-dimensional acquisition modules are different;
the 4D face data acquisition device is internally provided with a data processing module, and the data processing module is used for simultaneously carrying out three-dimensional dynamic scanning and photographing on the front face of the face by utilizing 3 groups of three-dimensional acquisition modules of the 4D face data acquisition device to obtain a plurality of groups of face three-dimensional point cloud data and a plurality of groups of face mapping data which dynamically change under a time scale; carrying out gridding treatment on each group of face three-dimensional point cloud data to obtain face gridding data; mapping processing corresponding to the face mapping data is carried out on each group of face gridding data, and a three-dimensional face model is obtained; and forming a dynamic sequence by the obtained multiple groups of three-dimensional face models based on a time scale to obtain a 4D face model.
The beneficial effects are that: the invention provides a 4D face data acquisition method and a device, the 4D face data acquisition method is adapted to a corresponding 4D face data acquisition device, the 4D face data acquisition device comprises 3 groups of three-dimensional acquisition modules, the 3 groups of three-dimensional acquisition modules are distributed in a delta shape, each group of three-dimensional acquisition modules comprises a structure light unit and a camera unit, the light wave frequency ranges of the structure light units of the 3 groups of three-dimensional acquisition modules are different, the 4D face data acquisition method is based on the 4D face data acquisition device which is provided with 3 groups of three-dimensional acquisition modules, the 3 groups of three-dimensional acquisition modules work in the unique light wave frequency ranges, the 3 groups of three-dimensional acquisition modules do not interfere with each other when in work, so that time-sharing multiple scanning is not needed, the 3 groups of three-dimensional acquisition modules work simultaneously, the scanning time is greatly reduced, the scanning efficiency is improved, the continuous scanning imaging of the three-dimensional face is realized, and the change range of the surface condition does not need to be strictly required.
Drawings
FIG. 1 is a flow chart of one embodiment of a 4D face data acquisition method of the present invention;
fig. 2 is a schematic diagram of an embodiment of a 4D face data acquisition device according to the present invention.
Detailed Description
It should be noted in advance that the terms "first," "second," "third," "fourth," and the like in the description and in the claims of the invention and in the above-described figures, if any, are used for distinguishing between similar objects and not necessarily for describing a particular sequential or chronological order. It is to be understood that the data so used may be interchanged where appropriate such that the embodiments described herein may be implemented in other sequences than those illustrated or otherwise described herein. Furthermore, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, system, article, or apparatus that comprises a list of steps or elements is not necessarily limited to those steps or elements expressly listed or inherent to such process, method, article, or apparatus.
For convenience of understanding, the following describes a specific flow of an embodiment of the present invention, referring to fig. 1, a first aspect of the present invention provides a 4D face data collection method, which includes the following steps:
s100, a 4D face data acquisition device is provided, see fig. 2, the 4D face data acquisition device comprises 3 groups of three-dimensional acquisition modules 10,3 groups of three-dimensional acquisition modules 10 are distributed in a delta shape, each group of three-dimensional acquisition modules 10 comprises a structure light unit and a camera unit, and the light wave frequency ranges of the structure light units of the 3 groups of three-dimensional acquisition modules 10 are different. In the invention, each three-dimensional acquisition module is responsible for the respective optimal three-dimensional scanning range, and 3 groups of three-dimensional acquisition modules 10 are combined to just cover the complete range of 180 degrees of the front face of the human face.
In the invention, a structured light unit is responsible for projecting coded structured light stripes to the surface of a scanned object (human face) according to a specific sequence, a camera unit synchronously captures a plane image of the scanned object covered with the stripes, then the image captured by the camera unit is subjected to binarization processing, stripe positions are extracted, a series of stripe decoding and three-dimensional point cloud reconstruction algorithms based on a triangulation principle are obtained for each stripe edge, and three-dimensional data of the surface of the scanned object are calculated to complete three-dimensional scanning.
In the invention, three groups of three-dimensional acquisition modules work in respective unique light wave frequency bands. For example, the projection of module A projects a light wave band lambda 1 Left and right light stripes, corresponding to the industrial camera lens of the module A can only pass through the light wave band lambda 1 The left and right signals, i.e. the camera can only capture images of the object in the corresponding light wave band. Similarly, the projection and camera of module B operates at lambda 2 The projection of the module C and the camera work at lambda 3 Left and right light wave bands. By the design, the three modules do not interfere with each other during operation, so that time-division scanning is not needed, the three modules can work simultaneously, and the time of single scanning is greatly reduced.
In the invention, the highest frame rate of the camera unit can reach 400Hz, and the high-frequency structure light stripe adopted by the three-dimensional structure light scanning module has 18 stripe pictures, so that the device can reach the full face scanning speed of 20Hz at the highest, and the scanned person can make natural expressions in the equipment working area according to the requirements of doctors by benefiting from the synchronous scanning method of the frequency division and non-time division of the invention, can synchronously scan and capture, and is subsequently displayed in the visualized three-dimensional space.
In an alternative embodiment of the present invention, the 4D face data collecting device further includes a base 20 and a remote control lifting module 30 mounted on the base 20, the remote control lifting module 30 is provided with a door-shaped auxiliary support 40,3 groups of three-dimensional collecting modules 10 are respectively mounted on the remote control lifting module 30 and two side arms of the door-shaped auxiliary support 40, a display control module 50 is further mounted on the remote control lifting module 30, and the display control module 50 is used for operably displaying the collected 4D face model.
In the invention, the inverted Y-shaped scanning module is integrally arranged on a remote control lifting module 30, a base 20 is arranged below the remote control lifting module 30 to ensure the integral stability of the device, an electric lifting rod is arranged above the base 20, and a scanned person stands in a working area right in front of the device, and the lifting of the remote control lifting module 30 is controlled by a remote controller so as to conveniently scan the scanned persons with different heights.
The display control module 50 is also installed on the remote control lifting module 30, and mainly has functions of software interface display and operation equipment, and performs point cloud meshing and 3D texture mapping of each group of three-dimensional face data, and matching alignment and connection between two adjacent groups of three-dimensional face data under the module to finally form a dynamic sequence of a three-dimensional face model, form a 4D face model, and summarize and extract the data change direction, trend and motion track of the 4D sequence based on face analysis data performed by each 3D face model.
S200, simultaneously carrying out three-dimensional dynamic scanning and photographing on the front face of the face by using 3 groups of three-dimensional acquisition modules 10 of the 4D face data acquisition device to obtain a plurality of groups of face three-dimensional point cloud data and a plurality of groups of face map data which dynamically change under a time scale; specifically, each dynamic acquisition of the three-dimensional acquisition module 10 includes several complete face scanning processes, so that it includes multiple sets of complete face three-dimensional point cloud data.
In the present invention, for three-dimensional point cloud data obtained by each acquisition, filtering is first performed, that is, in step S300, gridding processing is performed on each group of three-dimensional point cloud data of a face, and before face gridding data is obtained, the steps include: performing outlier removal processing on each group of face three-dimensional point cloud data; and smoothing the three-dimensional point cloud data of the face remained after the outlier is removed.
The processing for removing outliers from the three-dimensional point cloud data of each group of faces comprises the following steps: in the three-dimensional space, taking each point in the three-dimensional point cloud data of the human face as a center, and counting each point in the three-dimensional space as r 1 The number of other points present in the range; for each point, if the radius r 1 If the number of other points in the space range is smaller than a preset experience threshold, judging that the point is an outlier point and needing to be removed from the three-dimensional point cloud data of the face. Specifically, to remove outliers, a window is first set to a radius r 1 In the three-dimensional space, the point is taken as the center, and the distance between the front, back, left, right, up and down is counted as r 1 Within the range of (2), the number of other points present, n. When the number n of points is smaller than a certain empirical threshold, this point is defined as an outlier and needs to be removed in the point cloud. And traversing all points in the point cloud, and performing the outlier removing method on the points, so that outlier noise interference suffered by the point cloud can be reduced.
The smoothing the three-dimensional point cloud data of the face remaining after the outlier removal comprises: for each point in the three-dimensional point cloud data of the face, defining a radius r by taking each point in the three-dimensional point cloud data of the face as a center in a three-dimensional space 2 Is a smoothing region of (a); judging the composition linear queue of each direction for all points in each smoothing processing area, and finding out the point deviating from the queue; and performing displacement compensation on the points deviated from the queue so as to smoothly transition all the points in the three-dimensional point cloud data of the human face. Specifically, the smoothing of the point cloud aims to finely adjust the spatial position of partial points in a positive manner, so that the point cloud is smooth on the whole, and data of irregular positions are reduced. In short, a row of 5 points is formed, wherein 4 points are integrally arranged on a straight line, only a certain point in the middle is slightly deviated from the queue, and the deviated points return to the queue through the smoothing of the point cloud, so that the whole point cloud becomes smooth. Also byA radius r 2 Traversing the whole point cloud, adjusting all points in the window to be smoother every time, and finishing the smoothing of the whole point cloud.
In addition, in the invention, normal calculation is required to be performed on the processed point cloud. The normal of a point is the direction in which the pointing is in three dimensions. Represented generally by a normalized three-dimensional vector, N (a, b, c), where a 2 +b 2 +c 2 =1. And (3) including a certain point and points with distances smaller than a certain set value at the front and back positions in the up, down, left, right and front directions together to form a point set in a window. The normal to each point is not a fixed allocation but rather needs to be calculated in combination with the positions of other points within the nearby window. Fitting the points in the window to a curved surface, such as a sphere or other curved surface, the tangent plane of each point in the curved surface can be calculated, and the normal of the tangent plane is the normal of the point.
S300, carrying out gridding processing on each group of face three-dimensional point cloud data to obtain face gridding data; in the present invention, the step of performing gridding processing on each group of face three-dimensional point cloud data to obtain face gridding data includes: forming a triangle by every three points in the three-dimensional point cloud data of the human face; and (3) orderly arranging and connecting the obtained triangles in a common point or common edge mode to form a netlike continuous surface so as to obtain face meshing data. Specifically, the three-dimensional face point cloud data after filtering and normal processing are calculated, and surface three-dimensional grid reconstruction, namely triangularization of the point cloud is needed.
S400, mapping processing corresponding to the face mapping data is carried out on each group of face gridding data, and a three-dimensional face model is obtained; in the invention, after the face meshing data are obtained, 3D texture mapping is carried out on the face meshing data.
In an optional implementation manner of the first aspect of the present invention, the mapping processing corresponding to the face mapping data is performed on each set of face gridding data, and obtaining the three-dimensional face model includes: according to the arrangement mode of camera coordinates, preliminarily aligning the face mapping data with the face gridding data, and then carrying out reprojection; acquiring corresponding coordinates of each point in the face meshing data on the face mapping data, and grouping the acquired corresponding coordinate sets according to triangles in the face meshing data; traversing all triangles in the face gridding data, and remapping each group of corresponding coordinates obtained by grouping to three vertexes of the triangle; and adjusting the area texture of the inner area of the triangle in a linear stretching or compressing mode to finish the texture mapping of the face gridding data, thereby obtaining the three-dimensional face model.
Specifically, 3D texture mapping is performed. Texture coordinates of each three-dimensional point cloud are first acquired. The three-dimensional point cloud book is formed by a plane photo and depth information. Therefore, according to the arrangement mode of camera coordinates, aligning the two-dimensional color texture map shot by the camera and the reconstructed point cloud, and then carrying out re-projection to enable the color map to correspond to the three-dimensional point cloud row by row and column by column. That is, coordinates of each three-dimensional point corresponding to the texture map, i.e., texture coordinates of the point, are obtained. For three points in each triangle, a group of texture coordinates of the corresponding color texture map is provided, all the triangles are traversed, the texture coordinates of the points are remapped to three vertexes of the triangle, and color texture area blocks are distributed in the inner area of the triangle in a linear stretching or compressing mode to complete the three-dimensional texture map. This completes all work from one three-dimensional point cloud to the final corresponding mesh model.
According to the method, three-dimensional reconstruction is carried out on the data acquired by each dynamic acquisition, and a group of multiple three-dimensional data models are obtained. If the acquisition speed is 20Hz, the time interval for each model acquisition is only 0.05s. The position and expression changes of the model are thus subtle. The three-dimensional models of the plurality of groups can be considered to be perfectly aligned between every two.
In order to make the obtained three-dimensional face model rotatable, in an optional implementation manner of the first aspect of the present invention, after the performing mapping processing corresponding to the face mapping data on each set of the face gridding data to obtain the three-dimensional face model, the forming a dynamic sequence by the obtained sets of three-dimensional face models based on a time scale, and before obtaining the 4D face model, further includes: and in the three-dimensional space, adjusting the space pose of the first group of three-dimensional face models on a time scale by utilizing the space key point information of face recognition, obtaining a rotation translation matrix, and applying the rotation translation matrix to the subsequent groups of three-dimensional face models.
The step of obtaining a rotation translation matrix by adjusting the space pose of the first group of three-dimensional face models on a time scale in a three-dimensional space by utilizing the space key point information of face recognition, and the step of applying the rotation translation matrix to the following groups of three-dimensional face models comprises the following steps: taking the mass center of the first group of three-dimensional face models on the time scale as the origin of coordinates of a world coordinate system; two non-collinear space vectors formed by the left and right tragus points to the corresponding eye lower frame points form an XY plane; taking the normal direction of the XY plane pointing to the top of the head as a Z axis, and taking the left tragus point to the right tragus point as an X axis; taking a Cartesian right-hand coordinate system as a reference, wherein the directions of the Cartesian right-hand coordinate system, which are perpendicular to an X axis and a Z axis, and pointing to the nose tip point from the origin of coordinates are Y axes; and respectively calculating the relation between three axes and the origin of coordinates by using the model coordinates of the key points on the three-dimensional face model to obtain a rotation translation matrix of the three-dimensional face model changing to the world coordinate system.
In the present invention, the centroid (x d ,y d ,z d ) For the new world coordinate system O-XYZ, the coordinate origin is represented by the left and right tragus points (x a1 ,y a1 ,z a1 ),(x a2 ,y a2 ,z a2 ) To the respective corresponding under-eye frame points (x b1 ,y b1 ,z b1 ),(x b2 ,y b2 ,x b2 ) Two spatial vectors (x a1 -x b1 ,t a1 -y b1 ,z a1 -z b1 ),(x a2 -x b2 ,y a2 -y b2 ,z a2 -z b2 ) Forms an XY plane, and the normal direction of the XY plane pointing to the top of the head is taken as a Z axis (x 3 ,y 3 ,z 3 )＝(x a1 -x b1 ,y a1 -y b1 ,z a1 -z b1 )×(x b2 -x a2 ,y b2 -y a2 ,z b2 -z a2 ) Taking the left tragus point to the right tragus point as the X axis (X 1 ,y 1 ,z 1 )＝(x a2 -x a1 ,y a2 -y a1 ,z a1 -z a1 ) With the Cartesian right-hand coordinate system as a reference, the directions perpendicular to the X-axis and the Z-axis, respectively, and pointing from the origin of coordinates to the tip of the nose are the Y-axis (X 2 ,y 2 ,z 2 )＝(x 1 ,y 1 ,z 1 )×(x 3 ,y 3 ,z 3 ). By using the model coordinates of the key points of the three-dimensional model, respectively calculating the relational expression of three axes of O-XYZ and the origin of coordinates, finally obtaining a rotation translation matrix M of the three-dimensional model changing to the world coordinate system rt ，
For a certain acquired dynamic three-dimensional data, the nature of the dynamic three-dimensional data is a group of a plurality of static three-dimensional data, and the dynamic three-dimensional data is displayed in a visual three-dimensional space according to the concept of video frames. The video frame playing speed adjusting setting can freely adjust the speed of dynamic playing and can stay at a certain frame still. By constructing the rotation translation matrix, the whole dynamic three-dimensional data can be scaled, translated and rotated by utilizing the mouse. The 4D display was completed.
S500, forming a dynamic sequence by the obtained multiple groups of three-dimensional face models based on a time scale, and obtaining a 4D face model. In an alternative embodiment of the present invention, all three-dimensional face model data are uploaded to the cloud (for example, the ali cloud) in an encrypted manner under the support of the internet, and any authorized port, no matter where, only needs to be connected to the network, can synchronously download the data for 4D display.
With the development of digital three-dimensional technology in the direction of medical imaging, doctors related to medical projects such as orthodontics, dental implants and the like in the field of oral medical treatment have realized that the support and help of three-dimensional digital technology to clinical projects will be increasingly large. This complements the creation of new application requirements and scenarios. One of the important points is the application of dynamic digital three-dimensional technology in industry.
Imaging technology for outpatient applications has clearly seen a change from 2D to 3D and further to 4D. Specifically, the camera can capture a 2D planar picture of the face, and the three-dimensional scanner can further obtain three-dimensional data, and compared with 2D, 3D not only increases depth information, but also captures scale information of the scanned object. Namely, the data reserved by the 3D digital model can restore multi-dimensional information such as the length, the area, the volume and the like of the target in a 1:1 mode. By means of the 3D technology, a doctor can acquire face structure data of all angles of a patient only through one three-dimensional scanning data, and the method is rapid, convenient, accurate and effective.
The 4D face data acquisition method provided by the invention is used as an innovation based on a 3D technology, not only supports the conventional 3D scene application, but also can assist a time dimension to a 3D digital model of a patient to capture the face information of the patient in a dynamic 3D mode in a full-angle, full-scale and full-time period mode. The invention provides key information such as digital occlusion track, muscle change and the like for doctors, and aims to provide the 4D face data acquisition method and device which are high in precision, high in speed and strong in feasibility.
In addition, the second aspect of the present invention provides a 4D face data collecting device, referring to fig. 2, where the 4D face data collecting device includes 3 groups of three-dimensional collecting modules 10, and when viewed from a top view, the 3 groups of three-dimensional collecting modules are distributed in a delta shape, each group of three-dimensional collecting modules includes a structural light unit and a camera unit, and light wave frequency ranges of the structural light units of the 3 groups of three-dimensional collecting modules are different; the 4D face data acquisition device is internally provided with a data processing module, and the data processing module is used for simultaneously carrying out three-dimensional dynamic scanning and photographing on the front face of the face by utilizing 3 groups of three-dimensional acquisition modules of the 4D face data acquisition device to obtain a plurality of groups of face three-dimensional point cloud data and a plurality of groups of face mapping data which dynamically change under a time scale; carrying out gridding treatment on each group of face three-dimensional point cloud data to obtain face gridding data; mapping processing corresponding to the face mapping data is carried out on each group of face gridding data, and a three-dimensional face model is obtained; and forming a dynamic sequence by the obtained multiple groups of three-dimensional face models based on a time scale to obtain a 4D face model.
In summary, the invention provides a 4D face data acquisition method and a device, the 4D face data acquisition method is adapted to a corresponding 4D face data acquisition device, the 4D face data acquisition device comprises 3 groups of three-dimensional acquisition modules, the 3 groups of three-dimensional acquisition modules are distributed in a delta shape, each group of three-dimensional acquisition modules comprises a structural light unit and a camera unit, the light wave frequency ranges of the structural light units of the 3 groups of three-dimensional acquisition modules are different, the 4D face data acquisition method is based on the 4D face data acquisition device which is provided with 3 groups of three-dimensional acquisition modules, the 3 groups of three-dimensional acquisition modules work in the unique light wave frequency ranges, the 3 groups of three-dimensional acquisition modules do not interfere with each other when in work, so that the 3 groups of three-dimensional acquisition modules do not need to scan for many times, the scanning time is greatly reduced, the scanning efficiency is improved, the continuous scanning imaging of the three-dimensional face is realized, and the change range of the surface condition does not need to be strict.
The above embodiments are only for illustrating the technical solution of the present invention, and not for limiting the same; although the invention has been described in detail with reference to the foregoing embodiments, it will be understood by those of ordinary skill in the art that: the technical scheme described in the foregoing embodiments can be modified or some technical features thereof can be replaced by equivalents; such modifications and substitutions do not depart from the spirit and scope of the technical solutions of the embodiments of the present invention.
Claims (10)
1. The 4D face data acquisition method is characterized by comprising the following steps of:
the method comprises the steps that a 4D face data acquisition device is provided, the 4D face data acquisition device comprises 3 groups of three-dimensional acquisition modules, the 3 groups of three-dimensional acquisition modules are distributed in a delta shape, each group of three-dimensional acquisition modules comprises a structure light unit and a camera unit, and the light wave frequency ranges of the structure light units of the 3 groups of three-dimensional acquisition modules are different;
simultaneously carrying out three-dimensional dynamic scanning and photographing on the front face of the face by using 3 groups of three-dimensional acquisition modules of the 4D face data acquisition device to obtain a plurality of groups of face three-dimensional point cloud data and a plurality of groups of face map data which dynamically change under a time scale;
carrying out gridding treatment on each group of face three-dimensional point cloud data to obtain face gridding data;
mapping processing corresponding to the face mapping data is carried out on each group of face gridding data, and a three-dimensional face model is obtained;
and forming a dynamic sequence by the obtained multiple groups of three-dimensional face models based on a time scale to obtain a 4D face model.
2. The method for collecting 4D face data according to claim 1, wherein after the mapping process corresponding to the face mapping data is performed on each set of the face gridding data to obtain a three-dimensional face model, the forming a dynamic sequence by the obtained plurality of sets of the three-dimensional face models based on a time scale, and before obtaining the 4D face model, further includes:
and in the three-dimensional space, adjusting the space pose of the first group of three-dimensional face models on a time scale by utilizing the space key point information of face recognition, obtaining a rotation translation matrix, and applying the rotation translation matrix to the subsequent groups of three-dimensional face models.
3. The method of claim 2, wherein the adjusting spatial pose of the first group of three-dimensional face models on a time scale by using spatial key point information of face recognition in a three-dimensional space to obtain a rotation translation matrix, and applying the rotation translation matrix to each subsequent group of three-dimensional face models comprises:
taking the mass center of the first group of three-dimensional face models on the time scale as the origin of coordinates of a world coordinate system;
two non-collinear space vectors formed by the left and right tragus points to the corresponding eye lower frame points form an XY plane;
taking the normal direction of the XY plane pointing to the top of the head as a Z axis, and taking the left tragus point to the right tragus point as an X axis;
taking a Cartesian right-hand coordinate system as a reference, wherein the directions of the Cartesian right-hand coordinate system, which are perpendicular to an X axis and a Z axis, and pointing to the nose tip point from the origin of coordinates are Y axes;
and respectively calculating the relation between three axes and the origin of coordinates by using the model coordinates of the key points on the three-dimensional face model to obtain a rotation translation matrix of the three-dimensional face model changing to the world coordinate system.
4. The 4D face data collection method according to claim 1, wherein the step of performing gridding processing on each set of the face three-dimensional point cloud data, before obtaining face gridding data, includes:
performing outlier removal processing on each group of face three-dimensional point cloud data;
and smoothing the three-dimensional point cloud data of the face remained after the outlier is removed.
5. The method of 4D face data collection according to claim 4, wherein said removing outliers from each set of three-dimensional point cloud data of the face comprises:
in three dimensions, inEach point in the three-dimensional point cloud data of the human face is taken as a center, and the distance between each point in the three-dimensional space is counted as r 1 The number of other points present in the range;
for each point, if the radius r 1 If the number of other points in the space range is smaller than a preset experience threshold, judging that the point is an outlier point and needing to be removed from the three-dimensional point cloud data of the face.
6. The method of 4D face data collection according to claim 4, wherein smoothing the three-dimensional point cloud data of the face remaining after outlier removal comprises:
for each point in the three-dimensional point cloud data of the face, defining a radius r by taking each point in the three-dimensional point cloud data of the face as a center in a three-dimensional space 2 Is a smoothing region of (a);
judging the composition linear queue of each direction for all points in each smoothing processing area, and finding out the point deviating from the queue;
and performing displacement compensation on the points deviated from the queue so as to smoothly transition all the points in the three-dimensional point cloud data of the human face.
7. The method of 4D face data collection according to claim 4, wherein the performing gridding processing on each set of the face three-dimensional point cloud data to obtain face gridding data includes:
forming a triangle by every three points in the three-dimensional point cloud data of the human face;
and (3) orderly arranging and connecting the obtained triangles in a common point or common edge mode to form a netlike continuous surface so as to obtain face meshing data.
8. The method of claim 7, wherein the mapping the face gridding data of each group to the face mapping data to obtain a three-dimensional face model comprises:
according to the arrangement mode of camera coordinates, preliminarily aligning the face mapping data with the face gridding data, and then carrying out reprojection;
acquiring corresponding coordinates of each point in the face meshing data on the face mapping data, and grouping the acquired corresponding coordinate sets according to triangles in the face meshing data;
traversing all triangles in the face gridding data, and remapping each group of corresponding coordinates obtained by grouping to three vertexes of the triangle;
and adjusting the area texture of the inner area of the triangle in a linear stretching or compressing mode to finish the texture mapping of the face gridding data, thereby obtaining the three-dimensional face model.
9. The 4D face data collecting method according to claim 1, wherein the 4D face data collecting device further comprises a base and a remote control lifting module mounted on the base, the remote control lifting module is provided with a door-shaped auxiliary support, 3 groups of three-dimensional collecting modules are mounted on the remote control lifting module and two side arms of the door-shaped auxiliary support respectively, and a display control module is further mounted on the remote control lifting module and is used for displaying the 4D face model in an operable manner.
10. The 4D face data acquisition device is characterized by comprising 3 groups of three-dimensional acquisition modules, wherein the 3 groups of three-dimensional acquisition modules are distributed in a delta shape, each group of three-dimensional acquisition modules comprises a structure light unit and a camera unit, and the light wave frequency ranges of the structure light units of the 3 groups of three-dimensional acquisition modules are different;
the 4D face data acquisition device is internally provided with a data processing module, and the data processing module is used for simultaneously carrying out three-dimensional dynamic scanning and photographing on the front face of the face by utilizing 3 groups of three-dimensional acquisition modules of the 4D face data acquisition device to obtain a plurality of groups of face three-dimensional point cloud data and a plurality of groups of face mapping data which dynamically change under a time scale; carrying out gridding treatment on each group of face three-dimensional point cloud data to obtain face gridding data; mapping processing corresponding to the face mapping data is carried out on each group of face gridding data, and a three-dimensional face model is obtained; and forming a dynamic sequence by the obtained multiple groups of three-dimensional face models based on a time scale to obtain a 4D face model.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202311095187.3A CN117237529A (en) | 2023-08-28 | 2023-08-28 | 4D face data acquisition method and device |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202311095187.3A CN117237529A (en) | 2023-08-28 | 2023-08-28 | 4D face data acquisition method and device |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN117237529A true CN117237529A (en) | 2023-12-15 |
Family
ID=89095786
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202311095187.3A Pending CN117237529A (en) | 2023-08-28 | 2023-08-28 | 4D face data acquisition method and device |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN117237529A (en) |
-
2023
- 2023-08-28 CN CN202311095187.3A patent/CN117237529A/en active Pending
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN108629801B (en) | Three-dimensional human body model posture and shape reconstruction method of video sequence | |
| CN111784754B (en) | Tooth orthodontic method, device, equipment and storage medium based on computer vision | |
| Plankers et al. | Articulated soft objects for video-based body modeling | |
| CN1266655C (en) | Human limb three-dimensional model building method based on image cutline | |
| CN101916454B (en) | Method for reconstructing high-resolution human face based on grid deformation and continuous optimization | |
| CN113421328B (en) | Three-dimensional human body virtual reconstruction method and device | |
| WO2016003258A1 (en) | Face model generation method for dental procedure simulation | |
| CN112308932B (en) | Gaze detection method, device, equipment and storage medium | |
| CN108154550A (en) | Face real-time three-dimensional method for reconstructing based on RGBD cameras | |
| US11928778B2 (en) | Method for human body model reconstruction and reconstruction system | |
| CN103021017A (en) | Three-dimensional scene rebuilding method based on GPU acceleration | |
| CN106504321A (en) | Method using the method for photo or video reconstruction three-dimensional tooth mould and using RGBD image reconstructions three-dimensional tooth mould | |
| CN106485207B (en) | A kind of Fingertip Detection and system based on binocular vision image | |
| EP1522051A2 (en) | Discrete linear space sampling method and apparatus for generating digital 3d models | |
| CN112401369B (en) | Body parameter measurement method, system, device, chip and medium based on human body reconstruction | |
| CN112734890A (en) | Human face replacement method and device based on three-dimensional reconstruction | |
| CN103679797A (en) | Human limb modeling method adopting deformable models and virtual human model control platform | |
| Achenbach et al. | Accurate Face Reconstruction through Anisotropic Fitting and Eye Correction. | |
| CN114049442A (en) | Three-dimensional face sight line calculation method | |
| CN117237529A (en) | 4D face data acquisition method and device | |
| CN114155288A (en) | AR space visual three-dimensional reconstruction method controlled through gestures | |
| CN113284249A (en) | Multi-view three-dimensional human body reconstruction method and system based on graph neural network | |
| CN115205737B (en) | Motion real-time counting method and system based on transducer model | |
| Rosenhahn et al. | Automatic human model generation | |
| Varghese et al. | Computational simulation of human fovea |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |