KR102457180B1 - 3D modeling system with integrated control - Google Patents

Abstract

A 3D modeling system capable of integrated control is provided. The 3D modeling system includes: a plurality of cameras disposed around a subject to shoot 2D images at various angles with respect to a subject; a monitoring device monitoring the subject prior to camera shooting and generating a preparation image; a display device providing shooting guide information for 3D modeling to the subject; and a control device 3D modeling facial expressions of the subject from a plurality of 2D images shot by the plurality of cameras.

Description

3D modeling system with integrated control

The present invention relates to a 3D modeling system, and more particularly, to a system for 3D modeling a facial expression of a subject through integrated control.

Photogrammetry is a method of constructing 3D shape information based on 2D photos. It uses information from multiple photos to find the location information of the subject and camera, and finds and synthesizes information about specific patterns and pixels in the image information. The synthesized data is converted into 3D data to generate modeling data. In particular, a technique of 3D meshing 2D images captured by multiple cameras is called 3D reconstruction.

Conventionally, many processes from the camera capturing step for generating a large number of 2D photos to the 3D reconstruction step are not controlled in an integrated way, and are manually controlled by an administrator. For example, in the case of camera shooting, it is necessary to collectively change their settings so that all cameras and lights have predetermined setting values depending on the purpose. Conventionally, the administrator manually changes the settings for each camera and light By doing so, errors easily occurred in their setting values. If all cameras or lights do not have the same setting values, regularity in multiple captured images may be destroyed and fatal errors may occur in the 3D reconstruction process.

Also, in the prior art, when photographing a camera for 3D modeling, a photographer verbally requests a subject or a subject for position alignment or facial expression. From the point of view of the person being photographed, it is not easy to accurately understand the instructions of the photographer only with such verbal requests, so there is a problem that it takes a lot of work time.

An object of the present invention is to provide a 3D modeling system capable of integrated control.

The problems to be solved by the present invention are not limited to the problems mentioned above, and other problems not mentioned can be clearly understood by those of ordinary skill in the art to which the present invention belongs from the description below. will be.

A 3D modeling system according to an embodiment of the present invention for achieving the above object is a system for 3D modeling a facial expression of a subject using photogrammetry, and is disposed around the subject and has various angles with respect to the subject. a plurality of cameras for capturing 2D images; a monitoring device for generating a preparation image by monitoring the subject before the camera is photographed; a display device providing photographing guide information for 3D modeling to the subject; and a control device for 3D modeling the facial expression of the subject from a plurality of 2D images captured by the plurality of cameras.

Here, the control device may display the preparation image and the shooting guide information together on the display device. The preparation image may be changed and displayed on the display device according to a change in the state of the subject, and the control device may determine whether camera shooting is possible according to whether the preparation image satisfies a predetermined criterion defined in the shooting guide information. can be evaluated.

The photographing guide information may include: position guide information on a position required for the subject for 3D modeling; Expression guide information for the expression required for the subject for 3D modeling; and motion guide information for a motion required for the subject for 3D modeling.

The user interface provided by the display device may include: a location guide area for displaying the preparation image in real time while a subject guide line is displayed as the location guide information; an expression guide text area displaying the expression guide information as text; an expression guide image area displaying the expression guide information as an image; and a motion guide area displaying feedback on the motion of the preparation image.

The control device provides a warning message when the preparation image deviates from the subject guide line, and the control unit detects the facial expression of the subject from the preparation image in real time to evaluate camera shooting suitability based on the facial expression guide information, , the control device may detect the movement of the subject from the preparation image in real time to evaluate suitability for camera shooting.

The control device performs image integrity verification on the plurality of 2D images before performing 3D modeling, and the image integrity verification determines whether the number of the plurality of 2D images matches the number of the plurality of cameras , determining whether the exposure values of the plurality of 2D images are within a predetermined range, and determining the degree of defocusing of the plurality of 2D images.

Specific details of other embodiments are included in the detailed description and drawings.

As described above, according to the 3D modeling system according to the present invention, there are excellent effects as follows.

First, since the control device can control all cameras collectively, it is possible to prevent an error in the setting value when the camera setting value needs to be changed according to the purpose.

Second, the subject or subject can directly check whether his or her condition is suitable for shooting before shooting by checking his/her own image (i.e., preparation image) as well as the photographer’s requirements (i.e., shooting guide information) through the display device. have.

The subject may arrange his/her position through the display device so that his/her preparation image does not deviate from the subject guide line. If the preparation image deviates from the subject guide line, the control device may induce position alignment of the subject or the subject by providing a warning message.

The person to be photographed may check the expression requested by the photographer in text and images through the display device, and may make an expression according to the text and image. The control device detects the expression of the subject or the subject in real time from the prepared image using a facial recognition algorithm, etc. can do.

When it is impossible to take a picture with the camera due to the movement of the person to be photographed, the control device may provide feedback to the person to be photographed through the display device.

Third, the control device can increase data reliability before 3D modeling by automatically performing image integrity verification on a plurality of 2D images captured by a plurality of cameras. Through image integrity verification, it is possible to determine whether the number of images matches, the appropriateness of the image exposure value, and the appropriateness of image focusing.

Fourth, the control device automatically classifies and groups the image data with secured integrity based on the subject (subject), file name, shooting date, facial expression (shooting pose), etc., and stores it in a database for easy data storage and reuse .

1 is a diagram schematically showing the configuration of a 3D modeling system according to an embodiment of the present invention.
FIG. 2 is a diagram schematically showing the configuration of the control device of FIG. 1 .
FIG. 3 is a diagram illustrating a user interface provided by the display device of FIG. 1 .
4 is a flowchart sequentially illustrating a 3D modeling method according to an embodiment of the present invention.

Advantages and features of the present invention and methods of achieving them will become apparent with reference to the embodiments described below in detail in conjunction with the accompanying drawings. However, the present invention is not limited to the embodiments disclosed below, but will be implemented in a variety of different forms, only these embodiments allow the disclosure of the present invention to be complete, and common knowledge in the technical field to which the present invention belongs It is provided to fully inform the possessor of the scope of the invention, and the present invention is only defined by the scope of the claims. Like reference numerals refer to like elements throughout.

Hereinafter, a 3D modeling system according to an embodiment of the present invention will be described in detail with reference to FIGS. 1 to 3 . 1 is a diagram schematically showing the configuration of a 3D modeling system according to an embodiment of the present invention. FIG. 2 is a diagram schematically showing the configuration of the control device of FIG. 1 . FIG. 3 is a diagram illustrating a user interface provided by the display device of FIG. 1 .

The 3D modeling system 10 according to an embodiment of the present invention is a system for 3D modeling the facial expression of a subject using photogrammetry, and includes a photographing booth 20, a camera 22, and a monitoring device ( 24), the display device 30 and the control device 100 are included.

The photographing booth 20 is composed of supports arranged in a substantially cylindrical shape, and has a photographing space in which a subject or a person to be photographed is disposed. A plurality of cameras 22 and lighting means (not shown) are installed in the photographing booth 20 . The plurality of cameras 22 are disposed around the subject to photograph the 2D image at various angles with respect to the subject.

A monitoring device 24 for generating a preparation image by monitoring the subject before the camera is photographed is installed around the subject disposed in the photographing booth 20 . Preferably, the monitoring device 24 may be provided in plurality to photograph the front, left, and right sides of the subject. The monitoring device 24 may be formed of a webcam, a camera, or the like. In this embodiment, the monitoring device 24 is described as an example of a configuration separated from the camera 22 installed in the photographing booth 20, but the present invention is not limited thereto, and at least some of the cameras 22 are monitoring devices. (24) may be performed simultaneously.

The display device 30 is disposed in front of the subject, and displays the preparation image (ie, real-time image of the subject) captured by the monitoring device 24 and shooting guide information for 3D modeling on the screen to provide the subject to the subject. . The subject or the person to be photographed can directly check whether his or her condition is suitable for shooting before shooting by checking his/her own image (ie, preparation image) as well as the requirements of the photographer (ie, shooting guide information) through the display device 30 . can

The control device 100 controls operations of the camera 22 , the monitoring device 24 , and the display device 30 . The control device 100 3D models the facial expression of the subject from the plurality of 2D images captured by the plurality of cameras 22 . The control device 100 includes a camera control unit 110 , a subject guide unit 120 , an image verification unit 130 , a 3D reconstruction unit 140 , and a database 150 .

The camera controller 110 may collectively set shooting options or setting values of the plurality of cameras 22 . These shooting options include, for example, ISO (sensitivity), F-Stop (aperture ratio), and Aperture (aperture). In addition, the camera control unit 110 may collectively set the setting values of the lighting means installed in the photographing booth 20 . In this way, the camera control unit 110 can minimize the preparation time by collectively setting all the cameras 22 and the lighting means before shooting. In addition, the camera controller 110 may control all cameras 22 to shoot simultaneously. Therefore, all captured images can maintain the same brightness and focus.

The subject guide unit 120 displays shooting guide information for 3D modeling and a preparation image captured by the monitoring device 24 together on the display device 30 . When there is a change in the state of the subject's position, expression, movement, etc., the preparation image displayed on the display device 30 is also changed. The subject guide unit 120 may evaluate whether the camera can be photographed according to whether the preparation image satisfies a predetermined criterion defined in the photographing guide information.

Here, the photographing guide information includes location guide information, facial expression guide information, and movement guide information. The location guide information is information for guiding a location required for a subject for 3D modeling. The expression guide information is information for guiding an expression required for a subject for 3D modeling. The movement guide information is information for guiding a movement or a floating posture required for a subject for 3D modeling.

As shown in FIG. 3 , a user interface provided by the display device 30 includes a location guide area 38 , an expression guide area 32 , 34 , a motion guide area 36 , and a background area surrounding them. is composed of

The location guide area 38 displays the preparation image of the subject in real time while the subject guide line 39 is displayed as location guide information. The subject guide unit 120 may provide a warning message when the preparation image deviates from the subject guide line 39 . Accordingly, the subject or the photographed person can check his/her own image (ie, the preparation image) through the display device 30 in real time and arrange the positions so that the preparation image does not deviate from the subject guide line 39 . The front and/or side of the subject may be displayed in the location guide area 38 .

The expression guide areas 32 and 34 are areas in which expression guide information is displayed, and include an expression guide text area 32 displaying expression guide information as text and an expression guide image area 34 displaying expression guide information as an image. include For example, when the photographer requests AU (action unit) 4 from the subject, the expression guide text area 32 displays the text "Inner eyebrow lowering" corresponding to AU 4, and the expression guide image area 34 An image corresponding to AU 4 is displayed. The subject guide unit 120 detects and analyzes the expression of the subject from the prepared image in real time using a facial recognition algorithm, etc., and evaluates the expression of the subject based on the expression guide information to determine suitability for camera shooting. For example, when the expression of the subject is evaluated below a predetermined threshold score based on a perfect score of 100, the subject guide unit 120 determines that the expression of the subject is unsuitable for camera shooting and provides feedback to the subject to change the expression of the subject. can induce

The motion guide area 36 is an area in which motion guide information for a motion required for the subject is displayed, and may display feedback on the motion of the preparation image. The subject guide unit 120 may detect the movement of the subject from the preparation image in real time to evaluate suitability for camera shooting. For example, when the photographer requests the subject to be immobile, the subject guide unit 120 measures the movement distance of the ready image due to the movement of the subject, and when the movement distance is greater than a predetermined threshold, the display device 30 Feedback may be provided via the motion guide area 36 . For example, the movement guide area 36 may display a first feedback (unable to recognize a face) when (i) the subject is absent or face recognition is not possible, and (ii) when the movement distance of the subject is greater than a threshold point, the first feedback can be displayed. 2 feedback (facial movement) may be displayed, and (iii) if the movement distance of the subject is smaller than a critical point, a third feedback (photographing ready completion) may be displayed. The first to third feedbacks may be displayed in text, image, color, or the like.

The image verification unit 130 performs image integrity verification before performing 3D modeling on a plurality of 2D images generated by camera photographing. Image integrity verification includes first, determining whether the number of the plurality of 2D images matches the number of the plurality of cameras, second, determining whether exposure values of the plurality of 2D images are within a predetermined range, third, the plurality and determining the degree of defocusing of the 2D image of The image verification unit 130 first checks whether the number of cameras 22 and the number of photographed images are the same through verification of the number of images to verify whether all cameras 22 have properly photographed. The image verification unit 130 verifies whether the subject or the photographed person receives light sufficiently and evenly through the second exposure value verification. Here, the exposure value is a value that defines the combination of the aperture and shutter speed of the camera, and represents the amount of light that is exposed to the image pickup device in photography. The image verification unit 130 verifies whether the image is blurred due to out-of-focus through the third focus verification. The image verification unit 130 may generate thumbnails for a plurality of 2D images, and the administrator may determine whether to use the corresponding images through the thumbnails.

The 3D reconstruction unit 140 performs 3D reconstruction by 3D meshing the 2D image with integrity. For example, a commercial program Reality Capture, etc. may be used for 3D reconstruction. The 3D reconstruction unit 140 outputs 3D modeling and texture after performing 3D reconstruction.

The database 150 may automatically classify and store the 2D image data for which integrity is secured based on a subject (subject), a file name, a shooting date, an expression (photographing pose), and the like. The database 150 may store various other data required for 3D modeling.

Hereinafter, a 3D modeling method according to an embodiment of the present invention will be described with reference to FIG. 4 . 4 is a flowchart sequentially illustrating a 3D modeling method according to an embodiment of the present invention.

First, the camera controller 110 collectively sets shooting options or setting values of the plurality of cameras 22 ( S10 ).

The subject guide unit 120 displays a preparation image in which the monitoring device 24 captures the subject in real time in a state in which shooting guide information for position alignment, facial expression, and movement of the subject for 3D modeling is displayed on the display device 30 . It is displayed together on the device 30 (S20).

When the subject guide unit 120 determines that the preparation image satisfies a predetermined criterion defined in the shooting guide information, the camera control unit 110 simultaneously operates all the cameras 22 to photograph the subject to generate a plurality of 2D images. do (S30).

The image verification unit 130 performs image integrity verification before performing 3D modeling on a plurality of 2D images (S40).

The image verification unit 130 automatically classifies the 2D image data for which integrity is secured based on the subject (subject), file name, shooting date, facial expression (photographing pose), etc. and stores it in the database 150 (S50). .

The 3D reconstruction unit 140 performs 3D reconstruction on the 2D image for which integrity is secured, and then outputs 3D modeling and texture (S60).

The aforementioned control device, camera control unit, subject guide unit, image verification unit, 3D reconstruction unit, etc. are computer programs, for example, stored or loaded in a computer readable storage medium. It may be composed of a plurality of program modules implemented by one or more processors. Also, each step mentioned in the flowchart may be composed of a computer program, a program module, or a computer-executable instruction.

Although the embodiments of the present invention have been described with reference to the accompanying drawings, those of ordinary skill in the art to which the present invention pertains can realize that the present invention can be embodied in other specific forms without changing its technical spirit or essential features. you will be able to understand Therefore, it should be understood that the embodiments described above are illustrative in all respects and not restrictive.

10: 3D modeling system
20: Shooting booth
22: camera
24: monitoring device
30: display device
32, 34: expression guide area
36: movement guide area
38: position guide area
39: Subject guidelines
100: control device
110: camera control unit
120: subject guide unit
130: image verification unit
140: 3D reconstruction unit
150: database

Claims (5)

A system for 3D modeling the facial expression of a subject using photogrammetry, comprising:
a plurality of cameras disposed in the vicinity of the person to be photographed and simultaneously photographing a plurality of 2D images at various angles with respect to the person to be photographed;
a monitoring device for generating a preparation image, which is a real-time image of the subject, before the camera is photographed;
a display device providing photographing guide information for 3D modeling to the subject; and
Comprising a control device for 3D modeling the facial expression of the subject from a plurality of 2D images taken by the plurality of cameras,
The control device displays the preparation image and the shooting guide information together on the display device,
The preparation image is changed and displayed on the display device according to a change in the state of the subject, and the control device determines whether camera shooting is possible according to whether the preparation image satisfies a predetermined criterion defined in the shooting guide information. evaluate,
The display device is disposed in front of the person to be photographed and displays the preparation image and the photographing guide information on the screen so that the person to be photographed can directly check the real-time image of the person and the requirements of the photographer through the display device before photographing to provide it to the subject,
The photographing guide information includes movement guide information on the movement required for the subject for 3D modeling, and the control device detects the movement of the subject from the preparation image in real time and evaluates camera shooting suitability. displaying feedback on the display device;
The control device performs image integrity verification on the plurality of 2D images before performing 3D modeling, and the image integrity verification determines whether the number of the plurality of 2D images matches the number of the plurality of cameras A 3D modeling system comprising: According to claim 1,
The shooting guide information,
location guide information for a location required for the subject for 3D modeling; and facial expression guide information for facial expressions required for the subject for 3D modeling. 3. The method of claim 2,
The user interface provided by the display device,
a location guide area for displaying the preparation image in real time while a subject guide line is displayed as the location guide information; an expression guide text area displaying the expression guide information as text; an expression guide image area displaying the expression guide information as an image; and a motion guide region displaying feedback on the motion of the preparation image. 4. The method of claim 3,
The control device provides a warning message when the preparation image deviates from the subject guide line,
3D modeling system, characterized in that the control device detects the expression of the subject in real time from the preparation image and evaluates suitability for camera shooting based on the expression guide information. According to claim 1,
The verification of the image integrity further comprises determining whether the exposure values of the plurality of 2D images are within a predetermined range and determining the degree of defocusing of the plurality of 2D images.

Images