KR101556992B1 - 3d scanning system using facial plastic surgery simulation - Google Patents

Abstract

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a three-dimensional scanning system using facial shaping simulation, and is a portable apparatus that guides a three-dimensional scanner shooting point by recognizing a face direction and a camera direction by performing face tracking, A three-dimensional scanning device for simultaneously recording photographing time point information while acquiring an image automatically; A three-dimensional scanning post-processing device for reconstructing and reconstructing a mesh-like three-dimensional model based on the information recognized by the three-dimensional scanning device, and performing texture mapping at a shooting point of each of the RGB images and the reconstructed mesh model; ; And a 3D model processing unit that visualizes the 3D mesh model processed through the 3D scan after-processing apparatus, changes the texture of the mesh model in real time based on the texture mapping information closest to the camera position, Output device; .

Description

BACKGROUND OF THE INVENTION 1. Field of the Invention [0001] The present invention relates to a 3D scanning system,

The present invention relates to a three-dimensional scanning system, and more particularly, to a 3D scanning system which quantitatively measures a structure of a face, simulates a molding result to increase a patient's satisfaction and provides quantitative facial information to a doctor, And a system capable of designing a plan.

Conventional image rendering techniques and techniques for restoring three-dimensional faces are disclosed in Korean Patent Laid-Open No. 10-2002-0012598 (hereinafter referred to as "Prior Art 1") and Korean Patent Laid-Open No. 10-2005-0045774 , 'Preceding document 2') are registered and disclosed.

The above prior art document 1 discloses a method of rendering an image comprising the steps of: receiving a plurality of texture images (TGa, TGb, ...) of a single object (OB) observed from different viewpoints, ; Performing normal rendering processing on the objects (OB) to be plotted in detail according to a predetermined determination criterion when a plurality of identical objects (OB) are to be rendered; And selecting a texture image according to the direction in which the generated texture images TGa, TGb, ... are to be floated, and using the selected texture image as a texture for the objects OB to be simply plotted .

Also, in the above-mentioned prior art document 2, an apparatus for generating a three-dimensional face based on feature point information of a face image by receiving a two-dimensional face image of a user having various viewpoints, A central processing unit for processing and generating image information according to image and character information input from the plurality of user interface units and transmitting the generated image information to the corresponding user interface unit; And a system unit.

In the conventional medical 3D scanning system including the prior art documents 1 and 2, a mesh system is created by three-dimensionally reconstructing an image photographed at various points of view by providing an illumination system that produces uniform light conditions, and RGB The image is merged to map a color texture image to the surface of the model.

It is manufactured in a fixed installation form for shooting at various view points with the lighting system, and there is a restriction that the space is large, and it is difficult to use while moving.

In the case of the conventional mobile 3D scanning system, there is an inconvenience that the user must position and photograph the 3D scanner at predetermined shooting time points, and there is no lighting system that creates a uniform light condition, A seam artifact is generated due to a difference in luminosity between RGB images captured at various points in time (see FIG. 1A).

In order to overcome this problem, the blending algorithm reduces the color difference of overlapping parts. However, even in this case, additional calculation time is required for executing the algorithm, and distortion occurs in the original image, which causes a problem affecting medical observation.

There is a problem in that it is difficult to spread a 3D scanning system that increases the patient's satisfaction and assists the surgeon in planning the surgery more safely and conveniently.

Korean Patent Publication No. 10-2002-0012598. Korean Patent Publication No. 10-2005-0045774.

SUMMARY OF THE INVENTION The present invention has been made in view of the above problems, and it is a first object of the present invention to provide a hand-held three-dimensional scanner capable of performing face tracking simultaneously during a scan, By recognizing the direction of the camera, it is possible to guide the shooting timing of the 3D scanner, automatically record the RGB image while simultaneously recording the shooting time information when reaching the predefined shooting times, To facilitate the process.

A second object of the present invention is to provide a method and apparatus for performing texture mapping on an RGB image photographed at a position closest to a camera view in 3D rendering (texture switching) , It is possible to provide an optimal color texture image at the time when the user views the three-dimensional model, omit the blending algorithm, shorten the calculation time of the system, and cause problems in medical diagnosis .

According to an aspect of the present invention, there is provided a three-dimensional scanning system using facial shaping simulation, which is a portable apparatus that performs face tracking to recognize a face direction and a camera direction, A three-dimensional scanning device that simultaneously records shooting time point information while acquiring an image automatically when a predetermined shooting time point is reached; A three-dimensional scanning post-processing device for reconstructing and reconstructing a mesh-like three-dimensional model based on the information recognized by the three-dimensional scanning device, and performing texture mapping at a shooting point of each of the RGB images and the reconstructed mesh model; ; And a 3D model processing unit that visualizes the 3D mesh model processed through the 3D scan after-processing apparatus, changes the texture of the mesh model in real time based on the texture mapping information closest to the camera position, Output device; .

The three-dimensional scanning device may further include a scanner position-tracking unit for recognizing the position and direction of the three-dimensional space in real time for each frame; Based on the recognized depth map, the shape of the face is analyzed every frame to measure the head pose and the distance from the sensor to the face, A tracking unit; The control unit controls the RGB camera to automatically perform the RGB photographing when the camera camera approaches the preset camera view using the direction and the distance information of the face extracted by the face tracking unit in advance A camera viewpoint determination unit; And a photographing unit photographing the RGB image under the control of the camera view determining unit when the camera view determined by the camera view determining unit is satisfied, storing the camera internal parameters at the photographing time, the three-dimensional position and orientation information of the device together; And a control unit.

The face tracking unit may use a Depth Map or a Depth Map reconstructed from a Volume of TSDF (Timed Synchronous Data Flow) as an input value.

In addition, the 3D scanning apparatus may include a scan determination unit for determining a portion that is not scanned by mapping voxel information for each part of the face by mapping a face shape obtained through the face tracking unit on TSDF Volume; And a control unit.

The scan determination unit may determine whether the TSDF value has not been updated or not and determine a portion that is not scanned by estimating an update frequency with a weight value to determine whether the update frequency is less than a specific threshold value do.

The 3D scanning post-processing apparatus may further include: a mesh converting unit for extracting surface information from the TSDF volume recognized through the 3D scanning apparatus and generating a three-dimensional mesh-shaped model; A method of removing unnecessary double vertices from a three-dimensional model generated through the mesh transforming unit and performing mesh reduction or mesh simplification to reduce the number of polygons and using a Hole-filing algorithm A mesh reconstruction unit that fills non-scanned portions; An image reconstruction unit for improving color information of a plurality of RGB images; And a texture mapping unit for performing texture mapping of the three-dimensional model of the mesh and the RGB images. And a control unit.

In addition, the texture mapping unit performs projection based on the camera internal parameters in the position and direction of the 3D scanner at the time of photographing at which the RGB image is acquired, and detects the positions of the vertices of each face of the mesh And maps the RGB color coordinate system for each piece of information.

The texture mapping unit generates texture mapping information corresponding to the number of RGB images through texture mapping, and stores texture mapping information together with the 3D scanner position and direction information.

The 3D model output device may include a rendering unit for rendering the 3D mesh model processed through the 3D scanning post-processing apparatus at a time point designated by the user to output the 3D mesh model as a two-dimensional image; A rendering camera viewpoint calculation unit for calculating a viewpoint position closest to the camera position using the viewpoint information as a rendering reference through the rendering unit and extracting texture mapping information that is closest to the camera position according to the calculated viewpoint position; A texture changing unit for changing a texture of a mesh model displayed in the rendering unit in real time based on the texture mapping information closest to the camera position extracted through the rendering camera viewpoint calculating unit; A portion-to-be-measured portion for visualizing the distance for each predetermined region using the face shape obtained through the 3D scanning device, and overlaying the overlapped portion on the two-dimensional image rendered by the rendering portion; A virtual forming unit for dividing a main part of a face and simulating each divided area; And a control unit.

According to the present invention, the problem of the conventional fixed three-dimensional scanning system which can not be moved and has a space limitation for operation is improved by employing a hand-held three-dimensional scanner, Dimensional scanners, the user can manually move to a predetermined point in time, and the problem of increase in the amount of calculation due to the execution of the blending algorithm of the overlapped portion of the RGB image and the problem of image deformation in the blending application region are improved, A hand-held three-dimensional scanner is provided to construct a safe and comfortable molding simulation system.

The present invention provides an automated texture mapping and switching system for a hand-held three-dimensional scanner, including a real-time three-dimensional scanner position tracking technique, a three-dimensional reconstruction technique, and a texture mapping technique , And contributes to the development of complex technologies such as 3D scanning equipment and medical image processing software, which are vulnerable in Korea.

FIG. 1A is an exemplary view showing a conventional fixed type three-dimensional scanner. FIG.
BACKGROUND OF THE INVENTION 1. Field of the Invention [0002] The present invention relates to a three-dimensional scanning system,
FIG. 3 is a view showing an example of a head posture, a face orientation, and a distance from a sensor to a face moved by pitch, roll, and yaw extracted through the face tracking unit according to the present invention.
FIG. 4 illustrates an example of an RGB image captured through a photographing unit according to the present invention. FIG.
FIG. 5 and FIG. 6 are views showing an example of a form of a three-dimensional scanning apparatus according to the present invention.
FIG. 7 illustrates an example of a scanning method of a three-dimensional scanning apparatus according to the present invention. FIG.
FIG. 8 illustrates an example of mesh simplification through a mesh reconstruction unit according to the present invention; FIG.
FIG. 9 is an exemplary view showing a state where a portion not scanned is filled through a mesh reconstruction unit according to the present invention; FIG.
FIG. 10 is a view showing an example in which a texture is changed in real time from a front texture to a right texture through a texture changing unit according to the present invention; FIG.
FIG. 11 is a view showing a predetermined portion such as a distance between two eyes according to the present invention, a straight line of a half-length caret, an angle of a nose and a straight line, and the like.
FIG. 12 is a view showing an example of a simulation before the simulation about the nose region through the virtual forming unit according to the present invention, and a simulation using the mesh deforming method. FIG.

Specific features and advantages of the present invention will become more apparent from the following detailed description based on the accompanying drawings. It is to be noted that the detailed description of known functions and constructions related to the present invention is omitted when it is determined that the gist of the present invention may be unnecessarily blurred.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS The present invention will now be described in detail with reference to the accompanying drawings.

The three-dimensional scanning system using the facial shaping simulation according to the present invention will be described with reference to FIGS. 2 to 12. FIG.

FIG. 2 is a block diagram schematically showing a three-dimensional scanning system S using a face forming simulation according to the present invention. As shown in FIG. 2, the three-dimensional scanning apparatus 100, the three- And a three-dimensional model output device 300.

The three-dimensional scanning device 100 is a hand-held three-dimensional scanner that performs face tracking to recognize the direction of the face and the direction of the camera, thereby guiding the time of shooting the three-dimensional scanner, The face tracking unit 120, the camera-view-point determining unit 130, the scan-position determining unit 130, and the scan-position determining unit 130. [ (140) and a photographing unit (150).

Specifically, the scanner position locating unit 110 recognizes the position and direction on the three-dimensional space in real time every frame using an ICP (Iterative Closest Point) algorithm.

ICP (Iterative Closest Point) is an algorithm mainly used in computer graphics. It is a method of calculating scale conversion, rotation and movement, etc., in order to match the measurement data to a model when there is measurement data for any model will be.

The face tracking unit 120 analyzes the shape of the face at every frame based on the recognized depth map and measures the head position and the distance from the sensor to the face, And distance information.

Here, the face tracking unit 120 uses a depth map (Depth Map) or a reconstructed depth map (TSDF) as an input value.

Fig. 3 shows a head pose (a) moving by pitch, roll, yaw, etc., and a distance b from the face direction to the face from the sensor.

The camera-view-point determining unit 130 previously sets a camera-viewpoint for optimally photographing the entire face, and approximates the camera-viewpoint by using the direction and distance information of the face extracted through the face-tracking unit 120 , It controls to perform RGB shooting automatically. Here, when reaching the camera view point once taken, control is performed so that the RGB photographing is not performed again.

At this time, the camera-view-point determining unit 130 sets camera viewpoints for various positions such as front, left, and right in advance in order to photograph the entire face optimally, Can be generated.

On the other hand, the scan may not be performed, or a portion that is insufficient may occur. Accordingly, the scan determination unit 140 according to the present invention can increase the usability for improving the scan quality by schematizing the face position with respect to a portion that is not scanned or is insufficient.

That is, the scan determination unit 140 determines the non-scanned portion by mapping the face shape obtained through the face tracking unit 120 on the TSDF Volume and checking the voxel information for each part of the face.

More specifically, the scan determination unit 140 determines whether the TSDF value has not been updated by checking the voxel information for each part of the face, and determines that the corresponding part is not scanned.

In addition, the scan determination unit 140 determines whether the update count is less than a specific threshold value by estimating the update count as a weight value. If the update count is less than a specific threshold value, the scan determination unit 140 determines that the error of the distance measurement is not sufficiently fused , It is judged that the corresponding portion is a portion where the surface is not smoothly scanned, that is, a portion where the scan is insufficient.

When the camera view determined by the camera view determining unit 130 is satisfied, the photographing unit 150 photographs the RGB image under the control of the camera view determining unit 130 and displays the camera internal parameters at the photographing time, Dimensional position and orientation information are stored together. At this time, the photographed RGB image is as shown in FIG. 4, and the photographing unit 150 can transmit the photographed information to the scanner position tracking unit 110.

5 to 7, the IR projection (a), the RGB CMOS sensor (b), the IR CMOD (b), and the IR CMOS A sensor c and an LED lighting d with a built-in battery, and a USB interface device f which can be interfaced with a handle e and other devices can be connected to the lower end thereof.

5, the structure light camera of the active system is applied to the 3D scanning apparatus 100. However, the present invention is not limited to this, and the time-of-flight (TOF) depth Cameras, and passive Stereo Vision cameras, all of which can output depth maps.

The 3D scan after-treatment apparatus 200 generates and reconstructs a mesh-like three-dimensional model based on the recognized information through the 3D scanning apparatus 100, and reconstructs the mesh model and the RGB images The mesh reconstructing unit 220, the image reconstructing unit 230, and the texture mapping unit 240, as shown in FIG.

Specifically, the mesh converting unit 210 extracts surface information from the TSDF volume recognized through the 3D scanning apparatus 100, and generates a three-dimensional model in the form of a mesh.

The mesh reconstructing unit 220 reconstructs an unnecessary double in the three-dimensional model generated through the mesh transform unit 210 to improve the data processing speed in the texture mapping unit 240 and the three-dimensional model output apparatus 300, 8, the number of polygons is reduced by performing mesh decimation or mesh simulation, and a portion not scanned using a Hole-filing algorithm (See FIG. 9).

Here, a Duplicated Vertex refers to a vertex having the same positional coordinates on a three-dimensional space.

The image reconstructing unit 230 performs image enhancement and noise elimination using an image enhancement algorithm in a plurality of RGB images to improve the color information so as to help a medical diagnosis of a doctor.

The texture mapping unit 240 performs a texture mapping of the three-dimensional model of the mesh and the RGB images.

More specifically, the texture mapping unit 240 performs projection based on the camera internal parameters in the position and orientation of the three-dimensional scanner at the time of photographing, at which the RGB image is acquired, to obtain vertices of each face of the mesh (UV (LUV)) for each position information of the vertices of the vertices.

Here, the texture mapping information is generated by the number of RGB images, and is stored together with the three-dimensional scanner position and orientation information.

The three-dimensional model output device 300 visualizes the three-dimensional mesh model processed through the three-dimensional post-scan processing device 200, changes the texture of the mesh model in real time based on the texture mapping information closest to the camera position, 2, the rendering unit 310, the rendering camera viewpoint calculation unit 320, the texture modification unit 330, the site-specific survey unit 340, And a virtual forming unit 350. [0035]

Specifically, the rendering unit 310 performs a ray-casting algorithm at a time point designated by the user to output a three-dimensional mesh model processed through the three-dimensional scan-after-treatment apparatus 200 as a two-dimensional image And render it.

At this time, the user can receive the control information through a keyboard, a mouse, a haptic device, or the like and move the viewpoint.

The rendering camera viewpoint calculation unit 320 calculates a viewpoint position closest to the camera position by using the viewpoint information that is a rendering reference specified by the rendering unit 310, And extracts the mapping information.

At this time, when looking at the face model, the direction of the viewpoint is directed toward the center of the face. Therefore, the rendering camera viewpoint calculator 320 can approximate only the distance between the rendering point and the camera point by ignoring the direction.

That is, regardless of the direction, the Euclidean distance of the position of the rendering point and the camera position is obtained with only the X, Y, and Z coordinates, or the Manhattan distance is obtained, and the texture mapping of the camera position having the smallest distance Information is extracted.

In addition, the rendering camera viewpoint calculation unit 320 performs a ray-casting algorithm at the viewpoint, and launches a plurality of rays into a model into a field-of-view (viewing angle) at a viewpoint, We collect the set of regions, and select the camera position where the set of rendering points and intersection are the largest.

The texture changing unit 330 changes the texture of the mesh model displayed in the rendering unit 310 on the basis of the texture mapping information closest to the extracted camera position through the rendering camera view computing unit 320 .

Fig. 10 is an example showing a face front texture (a) and a right texture (b), showing that the texture is changed in real time from the front texture (a) to the right texture (b).

The site-specific surveying unit 340 visualizes the distance of each predetermined site using the face shape obtained through the face-tracking unit 120 of the 3D scanning apparatus 100, And outputs the overlay.

Here, as shown in FIG. 11, the predetermined portion may be a distance (a) between two eyes, a half length (b), a straight line of a cunt and an angle (c) of a straight line of a nose and a jaw.

The virtual forming unit 350 divides main parts such as nose, eye, forehead, and ball using the face shape obtained through the face tracking unit 120 of the 3D scanning apparatus 100, Simulation is performed using the Mesh Deformation method.

FIG. 12 is a diagram showing an example before a simulation about a nose portion and a view (b) simulating using a mesh deforming method.

Here, the mesh deformation for each divided region is a deformation as the volume increases when a specific amount of implant or cosmetic liquid is injected.

On the other hand, the virtual forming unit 350 may arbitrarily modify the shape of the divided main region based on user's control information through a keyboard, a mouse, a haptic device, or the like.

That is, if you select a specific vertex or vertices in a circular shape and take a Drag / Sweep motion, the vertices selected in that direction move and the surface is continuous, Vertices are also connected to vertices that are connected with a certain percentage of the motion vector.

While the present invention has been particularly shown and described with reference to preferred embodiments thereof, it will be understood by those skilled in the art that various changes in form and details may be made therein without departing from the spirit and scope of the invention as defined by the appended claims. It will be appreciated by those skilled in the art that numerous changes and modifications may be made without departing from the invention. Accordingly, all such appropriate modifications and changes, and equivalents thereof, should be regarded as within the scope of the present invention.

100: three-dimensional scanning device 200: three-dimensional scanning post-processing device
300: Three-dimensional model output device 110: Scanner positioning part
120: face-tracking unit 130: camera-
140: scan determination unit 150:
210: mesh conversion unit 220: mesh reconstruction unit
230: image reconstruction unit 240: texture mapping unit
310: rendering unit 320: rendering camera viewpoint calculation unit
330: texture changing unit 340:
350: virtual forming part

Claims (9)

As a portable device, a face tracking is performed to recognize a direction of a face and a direction of a camera, thereby guiding a 3D scanner photographing point, and automatically acquiring an image when reaching predefined photographing points, and simultaneously recording photographing point information A three-dimensional scanning device;
A three-dimensional scanning post-processing device for reconstructing and reconstructing a mesh-like three-dimensional model based on the information recognized by the three-dimensional scanning device, and performing texture mapping at a shooting point of each of the RGB images and the reconstructed mesh model; ; And
Dimensional mesh model processed through the 3D scan after-treatment apparatus, the texture of the mesh model is changed in real time based on the texture mapping information closest to the camera position, and a three-dimensional model output Device; , ≪ / RTI &
The three-dimensional post-scan processing apparatus includes:
A mesh transformer for extracting surface information from a TSDF volume recognized through the 3D scanning device and generating a mesh-shaped three-dimensional model;
A method of removing unnecessary double vertices from a three-dimensional model generated through the mesh transforming unit and performing mesh reduction or mesh simplification to reduce the number of polygons and using a Hole-filing algorithm A mesh reconstruction unit that fills non-scanned portions;
An image reconstruction unit for improving color information of a plurality of RGB images; And
A texture mapping unit for performing texture mapping of the three-dimensional model of the mesh type and each RGB image; Dimensional scanning system using a facial shaping simulation.
The method according to claim 1,
The three-dimensional scanning device includes:
A scanner position tracking unit for recognizing a position and a direction on a three-dimensional space in real time every frame;
Based on the recognized depth map, the shape of the face is analyzed every frame to measure the head pose and the distance from the sensor to the face, A tracking unit;
The control unit controls the RGB camera to automatically perform the RGB photographing when the camera camera approaches the preset camera view using the direction and the distance information of the face extracted by the face tracking unit in advance A camera viewpoint determination unit; And
A photographing unit photographing an RGB image under the control of a camera view judging unit when the camera view determined by the camera view judging unit is satisfied and storing the camera internal parameters at the photographing time and the three-dimensional position and direction information of the device together; Dimensional scanning system using a facial shaping simulation.
3. The method of claim 2,
The face-
Wherein the depth map is reconstructed from a Depth Map or a Timed Synchronous Data Flow (TSDF) volume as an input value.
3. The method of claim 2,
The three-dimensional scanning device includes:
A scan determining unit for determining a non-scanned portion by mapping voxel information for each part of the face by mapping a face shape obtained through the face tracking unit on TSDF Volume; Dimensional scanning system using a facial shaping simulation.
5. The method of claim 4,
The scan determination unit may determine,
And judges whether the TSDF value has not been updated or whether the update frequency is less than or equal to a specific threshold by estimating the update frequency with a weight value, 3D scanning system.
delete The method according to claim 1,
Wherein the texture mapping unit comprises:
Projection is performed according to the camera internal parameters in the position and orientation of the three-dimensional scanner at the time of photographing at which the RGB image is acquired and the color coordinate system of RGB is set for each position information of the vertices of each face of the mesh. Dimensional scanning system using face-forming simulation.
8. The method of claim 1 or 7,
Wherein the texture mapping unit comprises:
Wherein the texture mapping information is generated by the texture mapping for the number of RGB images, and the texture mapping information is stored together with the three-dimensional scanner position and orientation information.
The method according to claim 1,
The three-dimensional model output apparatus includes:
A rendering unit for rendering a three-dimensional mesh model processed through the three-dimensional post-scan processing apparatus as a two-dimensional image at a time point designated by the user;
A rendering camera viewpoint calculation unit for calculating a viewpoint position closest to the camera position using the viewpoint information as a rendering reference through the rendering unit and extracting texture mapping information that is closest to the camera position according to the calculated viewpoint position;
A texture changing unit for changing a texture of a mesh model displayed in the rendering unit in real time based on the texture mapping information closest to the camera position extracted through the rendering camera viewpoint calculating unit;
A portion-to-be-measured portion for visualizing the distance for each predetermined region using the face shape obtained through the 3D scanning device, and overlaying the overlapped portion on the two-dimensional image rendered by the rendering portion; And
A virtual forming unit for dividing a main portion of a face and simulating each divided region; Dimensional scanning system using a facial shaping simulation.

Images