KR100912872B1 - Apparatus and method for producing three-dimensional caricature - Google Patents

Abstract

The present invention relates to an apparatus and method for generating a 3D caricature, which accurately extracts feature point information of facial components using an ASM technique specialized for a face, and modifies a 3D polygon face basic model based on the feature point information. The main objective is to create a three-dimensional caricature with high similarity to the face, and perform preprocessing to detect the position of the eye in the input face image and normalize the size of the face image. After detecting each initial position and loading feature points by preloading ASM to the preprocessed face image to correspond to the detected initial position, the 3D polygon face caricature is modified by modifying the 3D polygon face basic model according to the coordinate values of the extracted feature points. It characterized in that to generate.

3d caricature ASM

Description

3D caricature generation device and method {APPARATUS AND METHOD FOR PRODUCING THREE-DIMENSIONAL CARICATURE}

1 is a schematic block diagram of a three-dimensional caricature generating apparatus according to the present invention.

FIG. 2 is a detailed configuration diagram of the preprocessor shown in FIG. 1.

3 is a flowchart of a three-dimensional caricature generation method according to the present invention.

4 is a detailed flowchart of the preprocessing step in FIG. 3.

FIG. 5 is a flowchart of a method of restoring a three-dimensional caricature transmitted in FIG. 3.

6 is a diagram illustrating an example of a face image.

7 and 8 illustrate results of binarizing a face image according to a set threshold value.

9 is a view showing that the size of the eye occupies in the face image according to the change in the threshold value of binarization.

10 is a diagram illustrating an example of extracting feature points from an input face image according to the present invention.

FIG. 11 is a diagram illustrating an example in which ASM adaptation performance is improved by initial position detection of a facial component.                 

12 is a diagram for describing generating a 3D polygon face caricature.

FIG. 13 is a diagram for explaining deforming a 3D polygon face basic model.

14 is a diagram illustrating an example of a caricature with emphasis on exaggeration.

Explanation of symbols on the main parts of the drawings

10 ... Memory 11 ... Learning data DB

13 ... 2D Face Texture DB15 ... 3D Polygon Face Base Model DB

17 ... 3D Body Model DB 30 ... Image Processing Unit

31 Pre-processing unit 33 ... Face component initial position detection unit

35 ... ASM loading part 37 ... ASM adaptation part

39.Caricature generator 39a ... Face caricature generator

39b ... body caricature generator 40 ... projection unit

50 ... Caricature storage 70 ... Caricature transmission and reception

The present invention relates to a method and apparatus for generating a three-dimensional caricature, in particular, using feature-specific Active Shape Model (ASM) techniques to accurately extract feature point information on the face components and based on this feature point information three-dimensional polygon face The present invention relates to an apparatus and method for generating a three-dimensional caricature with high similarity to a user's face by modifying a basic model.

In general, a caricature is a picture of a person or anthropomorphic animals or animals, aimed at humor, humor, satire, etc., and includes meanings such as user's curiosity, surrogate satisfaction, and the ideal person required by the individual. Sports athletes have been used as a promotional means, but nowadays it is widely used by the general public.

In the traditional way, caricatures have been completed by professional caricature designers, but with increasing public interest, technologies are being developed that can create caricatures without the need for expert hands.

For example, a user may select and combine clip arts required to form a face contour such as eyes, nose, and mouth from a clip art database, and complete various final accessories by applying various accessories or clothes. However, this method has a lot of complaints due to the lack of similarity due to the problem that the user is very difficult to select the clip art closest to their facial features.

Korean Patent Laid-Open No. 2000-63391 (caricature manufacturing method) first completes the overall face shape from the face in the picture, calculates the position, ratio, and type of each element to be applied to the face shape, and finally the various sizes and A method of manufacturing a caricature is disclosed by extracting a model for each face element from a database of face-each face elements having a size and then synthesizing the model. However, since only the position information of each face element is used in comparison with the face size, accurate feature point extraction for each face element is almost impossible, and thus, the creation of a caricature close to the face feature is difficult.

In addition, Korean Patent Publication No. 2000-49421 (Caricature manufacturing system and method of manufacturing a caricature using the system) compares the caricature for each part that stores the caricature for each part of the photo edited by the software and each part of the face. A method of converting each part of a face needed into data in a database and converting it into a caricature is disclosed. However, since this method requires a photo editing process in which the original data composed of natural colors are separated into black and white by tonal conversion, it is impossible to generate a caricature having a substantially colored color.

In addition, Korean Patent Laid-Open Publication No. 2000-64110 (apparatus and method for automatically generating a character based on a face image) extracts face shape information by using corresponding point information of an input face image and a reference image, and based on the extracted face image. A method of resynthesizing to obtain a more natural and sophisticated caricature image is disclosed. However, this method takes a simple method of synthesizing a pre-prepared reference image with a face image as in Korean Patent Laid-Open No. 2000-63391, and does not specifically present a face feature point extraction from the face image.

On the other hand, the above-mentioned patents are all related to the generation of two-dimensional caricature, but in recent years, research on the generation of three-dimensional caricature using the face modeling technique is actively in progress.                         

As one of such three-dimensional caricature generation techniques, as described in US Patent No. 5,960,099 entitled 'System and Method for Creating a Digitized Likeness of Persons', There is a technique of estimating the characteristics of the face, that is, the position of each part of the face through statistical data using two face images on the side, and generating a portrait with a changed expression. However, this technique has a problem in that it is not possible to accurately find the facial features of the original image because the facial features are estimated through statistical data when the portraits of the facial expressions are changed using the front and side face images.

As another 3D caricature generation technology, Korean Patent Publication No. 2001-69820 (Method and System for Acquiring 3D Face Image from 2D Image) includes eye, nose, mouth, ear, eyebrow and face outlines from front and side pictures. After extracting the outline model passing through the nose, hair features, etc., a user selects a plurality of three-dimensional face shapes similar to the extracted face features from a three-dimensional face shape database (a lot of real human faces are accumulated). Disclosed is a technique for selecting a three-dimensional face shape most similar to one's own face photograph. However, the most similar three-dimensional face shape is obtained through simple comparison with the face shape database which is already built in advance, and has a disadvantage of requiring a separate manual process by a correction expert to correct more detailed parts.

The technical problem to be achieved by the present invention is to accurately extract feature point information for facial components using ASM technique that is specialized for the face, and to modify the 3D polygon face basic model based on the feature point information to the user's face and It is to generate three-dimensional caricature with high similarity.

In order to achieve the above object, a three-dimensional caricature generating apparatus according to the present invention comprises a memory unit storing an ASM and a three-dimensional polygon face basic model required for generating a three-dimensional caricature, and detecting an eye position from an input face image. Pre-processing unit for normalizing the size, Facial component initial position detection unit for detecting each initial position of the facial component in the pre-processed face image, Loading the ASM stored in the memory unit to correspond to the detected initial position in the pre-processed face image An ASM loading unit, an ASM adaptor adapted to extract the feature points for the facial component by adapting the loaded ASM; And a caricature generation unit for generating a 3D polygon face caricature by loading the 3D polygon face basic model from the memory unit and modifying the loaded 3D polygon face base model according to the coordinate values of the feature points extracted through the ASM adaptor. It is characterized by.

In another preferred embodiment of the present invention, it characterized in that it further comprises a projection for obtaining a two-dimensional face caricature by projecting the three-dimensional polygon face caricature in two dimensions.

In another preferred embodiment of the present invention, the caricature transceiver further comprises extracting and transmitting the coordinate values and the memory reference values of the feature points in the three-dimensional polygon face caricature.                     

Hereinafter, the configuration and operation of a three-dimensional caricature generating device according to the present invention will be described in detail with reference to the accompanying drawings.

1 is a schematic block diagram of a three-dimensional caricature generating apparatus 100 according to the present invention, a memory unit 10, an image processing unit 30, a projection unit 40, a caricature storage unit 50, a caricature transmission and reception unit It consists of 70.

In the memory unit 10, training data necessary for generating a 3D caricature, a 2D face texture, a 3D polygon face basic model, a 3D body model, and the like are stored in a database form. These learning data DB 11, the two-dimensional face texture DB 13, the three-dimensional polygon face base model DB 15, and the three-dimensional body model DB 17 are briefly described.

The training data DB 11 stores front / side ASMs obtained by using various types of eyebrows, eyes, noses, mouths, and facial contours as learning patterns, average position coordinates of these ASMs, and morphological change characteristic data of the ASMs. In the two-dimensional face texture DB 13, two-dimensional face textures for texturing the three-dimensional polygon face base model are stored. Here, the two-dimensional face textures can be changed to reflect various color information in the three-dimensional face caricature, for example, the lip color of the make-up and the change in the brightness of the skin color due to lighting are applied to the three-dimensional face caricature. It is structured to generate more natural and sophisticated caricatures by reflecting.

In addition, the three-dimensional polygon face base model DB 15 stores various types of three-dimensional polygon face base models. These three-dimensional polygon face base models are constructed based on anthropological standard data reflecting racial and national characteristics. The three-dimensional body model DB 27 stores three-dimensional body models for processing caricatures of body components such as hair, body, glasses, accessories, clothes, and shoes.

The memory unit 10 may include other data in addition to the training data, the two-dimensional face texture, the three-dimensional polygon face basic model, and the three-dimensional body model.

The image processor 30 detects feature point information of each part of the face using the ASM technique and generates a 3D caricature reflecting the user's face features by modifying the 3D face basic model based on the detected feature point information. It consists of a preprocessor 31, a facial component initial position detector 33, an ASM loading unit 35, an ASM adaptor 37, and a caricature generator 39.

The preprocessor 31 detects the position of the eye from the input face image and performs preprocessing to normalize the size of the face image. Hereinafter, the preprocessor 31 will be described in more detail with reference to FIG. 2.

FIG. 2 is a diagram showing a detailed configuration of the preprocessor shown in FIG. 1. As shown in FIG. 2, the preprocessor 31 includes a black and whiteization unit 21, a face region detector 22, an eye detector 23, and And a face normalization unit 28.

If the input face image is a color image, the black-and-white unit 21 converts the image into a black-and-white image. In the color image of RGB (Red, Green, Blue) method, the color component and the brightness component are mixed together, and thus the brightness of the feature point is extracted. This is because errors can be caused by change.                     

The face region detector 22 detects a face region from an input image using a Gabor Filter Response. The face region detection method using a Gabor filter includes a Gabor filter having various directions and frequencies in the input face image. By applying a set of these and detecting the facial region according to the response value, it is a kind of pre-processing method generally used in the image processing field, so a detailed description thereof will be omitted.

The eye detection unit 23 has a circular shape in the face area detected by the face area detection unit 22 and is positioned on the upper part of the face area and shows a black area in the middle of the white, that is, the center point (the pupil) of both eyes. Detect. Here, the first reason for detecting the center point of the two eyes is that the center points of the two eyes which are symmetrical from side to side are most easily detected. This can be easily detected by

An example of a face image input for caricature generation is illustrated in FIG. 6. When a face image having a high resolution is input as shown in FIG. 6 (a), the eyes are extracted and then the eyes are extracted based on the extracted edge information of the eyes. Although the center point can be detected, when the face image having a low resolution is input as shown in FIG. 6 (b), since the eye is recognized as almost one point, the edge of the eye cannot be detected properly and the edge of the eye is detected. Even if the eyebrows and eyes are not well distinguished, there is a risk of misrecognizing the eyebrows as eyes. In addition to the edge detection method, if a black and white face image having a brightness level of 256 levels based on a threshold is divided into black and white, the pupil of the eye is very low in brightness and thus becomes black. The binarization technique of detecting black circular regions as the center points of two eyes is also disclosed. However, the conventional binarization technique using only one threshold value is used to detect eyebrows and eyes in a face image having low resolution and irregular distribution. There is a limitation in that it is impossible to accurately detect the center point of the two eyes because it cannot be distinguished well.

Therefore, in order to accurately detect the position of both eyes even in a low resolution face image, the threshold value should be appropriately changed according to the state of the face image. In the present invention, the eyebrow and the eye through the eye detector 23 will be described as follows. The binarization is performed while increasing the threshold value until all of them have a constant black area. For this purpose, the eye detection unit 23 is a binarization unit 24, a grouping unit 25, a labeling unit 26, and a determination unit. And (27).

When the black and white face image is input, the binarization unit 24 sets a change range of the threshold for binarization, sets an initial value of the threshold within the change range of the threshold, and sets the brightness above the threshold based on the set threshold. Pixels with white are assigned to black and pixels with brightness below the threshold are assigned to black.

 An example of a face image binarized according to a set threshold is illustrated in FIGS. 7 and 8. As shown in FIGS. 7 and 8, when the face image is binarized according to the set threshold value, the eyebrows are first detected, and as the threshold value is increased, the eyes, the nose, the mouth, and the like appear in black.

When the binarized face image is input, the grouping unit 25 performs grouping to group pixels having black color into various groups in the areas corresponding to the eyebrows and the eyes. In this case, when grouping a pixel having a black color, the grouping means searching black neighboring pixels located around the pixel to find the black pixels and grouping the found black pixels into a group.

The labeling unit 26 labels the various groups detected by the grouping unit 25 to order the various groups, and then extracts the region having the same label as an isolated region.

The determination unit 27 checks whether there is a pair of isolated areas where the size of the eyebrows and the eyes are always paired and the eyebrows are located above the eyes, and the size change in the horizontal direction with respect to the upper isolated area is not significantly different. If it is present, it is recognized as an eyebrow candidate region, and if there is a pair of isolated regions where the size change is not significantly different in the horizontal direction near the eyebrow candidate region, it is recognized as an eye candidate region.

If there is no pair of eyebrow candidate regions and one pair of eye candidate regions among the isolated regions extracted through the labeling unit 26, that is, if neither one of the two eyebrows and the two eyes is detected, the determination unit 27. ) Performs binarization by increasing the threshold until both eyebrows and both eyes are detected so that the eyebrows are not mistaken as eyes.

If both eyebrows and both eyes are detected, the determination unit 27 calculates an error rate of the eye candidate area based on the information on the size, shape, and position of the eye to determine whether the eye candidate area is correct. It is determined whether the error rate is within an acceptable range, and if the error rate of the eye candidate region is determined to be an acceptable range, that is, if it is determined that the eye candidate region is an eye, the center points of the two eyes are detected in the eye candidate region. The description will be made in detail.

The face normalizer 28 performs 1: 1 mapping of the size of the face area with the 3D face basic model stored in the 3D polygon face base model DB 15 based on the position of the eye detected by the eye detector 23. Normalize to a size that can be, and if the face image is slightly inclined, correct it at a normal angle.

An example of preprocessing a face image through the preprocessor 31 is illustrated in FIGS. 10A to 10D. First, when the face image is input, the preprocessing unit 31 converts the face image to black and white as shown in FIG. 10 (a) to remove the influence of illumination and noise, and then as shown in FIG. 10 (b). The Gabor filter is applied to detect a rectangular face area that includes face-like features. Then, the preprocessing unit 31 detects the center points of the two eyes in the face area as shown in FIG. 10 (c), and then detects the center of the face area and the two eyes as shown in FIG. 10 (d). On the basis of this, the size of the face area is normalized and the slope is corrected.

On the other hand, if the feature points for each part of the face are extracted by using the general ASM technique, the eyebrows may be misrecognized as the eyes when the feature points are extracted, or face components may move in strange directions.

In order to prevent this phenomenon, the present invention performs the ASM adaptation by setting the approximate positions of the eyebrows, eyes, nose, mouth, or facial contours in advance through the facial component initial position detector 33, which is described below. The initial position setting will be described in more detail.

When extracting the feature points from the input face image, it is very important that ASM adaptation is performed where the ASM of each face component is placed in the face image. That is, the initial position where ASM is loaded has a great influence on the performance of ASM adaptation performed afterwards. For example, if the initial position where ASM is loaded is set well, the feature points can be extracted accurately with only a few adaptations. If the initial position is set incorrectly, the feature point extraction may not work properly.

The face component initial position detector 33 applies a canny filter to a black and white face image to detect an initial position of each face component, where the canny filter uses edges using Gaussian masks. Detects the initial position where the ASM will be loaded by detecting.

In addition, in addition to the use of the Canny filter as described above when detecting the initial position of the face component, the eyebrows according to the geometric position characteristics between the face components based on the center point of the two eyes detected by the eye detector 23 You can also detect the initial positions of the eyes, nose, and mouth.

10 (e) to 10 (g) show an example of extracting feature points by detecting an initial position of each facial component from an input face image and loading an ASM at the detected initial position. As shown in FIG. 10 (e), when the initial position of the eyebrow, eye, nose, mouth, or face contour is detected through the Canny filter, the ASM is applied to the detected initial position as shown in FIG. 10 (f). By adapting the loaded and loaded ASM, it is possible to accurately extract the feature points for the facial component as shown in FIG. 10 (g).

On the other hand, maintaining a constant relationship between ASMs for each facial component also helps to extract feature points accurately. For example, the right eye ASM is on the same horizontal axis as the left eye ASM, no matter how adapted. If you establish a relationship that the right eye ASM should always be below the right eyebrow ASM, ASM adaptation performance is improved.

11 shows an example in which the ASM adaptation performance is improved by applying an initial position detection of a facial component and a constraint based on the relationship between each ASM. As shown in FIG. 11 (a), when ASMs that are not given relevance are loaded on the input face images based only on eye positions, feature points may be incorrectly extracted from the input face images as the ASM adaptation proceeds. have. However, as shown in FIG. 11 (b), after setting the initial position of the facial component through the facial component initial position detecting unit 33, the ASM faces related to each other based on the set initial position. When loaded into the image, it can be seen that the feature points are accurately extracted even with a small number of adaptation processes.

That is, when the initial position of each facial component is detected through the facial component initial position detector 33, the ASM loading unit 31 is mutually different from the training data DB 11 on the face image so as to correspond to the detected initial position. Loading relevant ASMs can improve ASM adaptation performance, allowing fast and accurate extraction of feature points for each facial component.                     

In particular, in the present invention, since the learning data DB 11 stores ASMs learned at various resolutions to cope with various resolution changes, it is difficult to provide a user interface capable of capturing a clear face image, for example. Even if a low resolution face image is input from a mobile phone with a small camera, feature points for eyebrows, eyes, nose, mouth, or facial contours can be accurately extracted.

The ASM adaptor 37 adapts the loaded ASM by using the morphological change feature data of the previously obtained ASM, extracts feature points for the input face components, and then extracts the X and Y coordinate values of the extracted feature points. Output to (39). Here, the ASM adaptation means that the feature points of the ASM loaded in the face image are connected in a straight line and searched for pixels in the normal direction with respect to the straight line at the midpoint thereof, where the maximum correlation with the basic learning value occurs in the search area. It is to find the feature points on the eyebrows, eyes, nose, mouth, or face contour by repeating the process of changing the pixel to a new midpoint and changing the position of the feature points according to the changed midpoint.

However, until now, there have been no criteria on how to set the search area in ASM adaptation. As a result, if the search area is too wide, it is possible to extract the feature points in the wrong place by greatly escaping the area to be searched, or when the search area is too small. Too much time was spent on adaptation.

In addition, even if most of the feature points were extracted in a desired area, if one or two feature points were extracted from the wrong place, there was a fear that the shape of the feature points would be completely disturbed. For example, the nostrils and the concave portion of the upper lip are similar in shape and position, but if the search areas of the feature points seeking the lip line are too large, the nostrils may be mistaken for the upper lip. . In other words, if one or two feature points are incorrectly extracted in the adaptation phase of the ASM, this increases the likelihood that other feature points will be incorrectly extracted.

Therefore, it is necessary to limit the search area so that the feature point is not extracted incorrectly in the ASM adaptation step. To this end, in the present invention, the ASM adaptation is performed in a state in which the search area of the feature point extraction for each face component is limited as follows. To perform.

For example, in the case of the eye, the search area of the upper feature points of the eye is limited to 1/2 or less of the distance to the eyebrows, and the search area of the lower feature points of the eye is limited to 1/2 or less of the upper and lower widths of the eye. Also, in the case of the nose, the search area of the feature points corresponding to the nostrils and the nostrils is limited to be 1/2 or less of the distance from the upper lip, and in the case of the lips, the search area of the upper feature points of the lip is less than 1/2 the distance from the nostril The search area of the lower feature points of the lips is limited to 1/3 or less of the upper and lower widths of the lips.

The limit value of the search region is a value capable of minimizing a feature point extraction error obtained through a number of experiments, and the reliability of the feature point extraction can be increased by the limitation of the search region.

Referring again to FIG. 1, the caricature generator 39 is composed of a face caricature generator 39a for generating a three-dimensional face caricature and a body caricature generator 39b for generating a three-dimensional body caricature. It will be described in detail in the description with respect to Figure 3 and will be outlined below.

The face caricature generation unit 39a generates a three-dimensional face caricature reflecting the user's face features by modifying the three-dimensional polygon face basic model using the facial feature point information extracted through ASM adaptation, and the body caricature generation unit 39b. Generates a 3D body caricature based on a 3D body model such as body components such as hair, body, glasses, accessories, clothes, shoes, etc. stored in the 3D body model DB 15.

If the two-dimensional caricature is required, the projection unit 40 generates a two-dimensional caricature by projecting the three-dimensional caricature generated through the caricature generation unit 39 in two dimensions, and the caricature storage unit 50 is the caricature generation unit 39 Save the 3D caricature created by

On the other hand, the three-dimensional caricature generated by the caricature generation unit 39 may be transmitted to the other three-dimensional caricature generation device 100 by wireless or wired through the caricature transmission and reception unit 70, at this time the caricature transmission and reception unit 70 In order to reduce the volume of data to be transmitted, the coordinate values and the memory reference values of the feature points are extracted from the 3D caricature and transmitted to another 3D caricature generating device 100. The coordinate values and the memory reference values of the transmitted feature points may be restored to the 3D caricature through the caricature transceiver 70 in the other 3D caricature generating apparatus 100. A method of restoring the 3D caricature is described with reference to FIG. 5. This will be described in detail in the description.

For reference, the above-described components of the present invention may be variously modified and equivalent embodiments by those skilled in the art when implemented in hardware. For example, in the present invention, the memory unit 10 in which other data is stored besides the learning data, the 2D face texture, the 3D polygon face base model, and the 3D body model may include an ASM loading unit 35 and an ASM adapting unit. If the 37 or the caricature generator 39 includes a separate memory, the memory may be divided into a plurality of memories, and the ASM loading unit 35, the ASM adaptor 37, or the caricature generator 39 may be configured. Such components may be implemented in software running in a microprocessor.

On the other hand, the three-dimensional caricature generation method according to the present invention, a pre-processing step of detecting the position of the eye in the input face image and normalizes the size of the face image, detecting each initial position of the face components in the pre-processed face image Extracting feature points by loading the ASM onto the preprocessed face image to correspond to the detected initial position, and then extracting the feature points by loading the ASM, and loading the feature point extracted through the ASM adaptation after loading the 3D polygon face basic model. And modifying the loaded 3D polygon face basic model according to the coordinate values of the 3D polygon face caricatures.

In another preferred embodiment of the present invention, the method further comprises obtaining a two-dimensional face caricature by projecting the three-dimensional polygon face caricature in two dimensions.

In another preferred embodiment of the present invention, the method further comprises extracting and transmitting the coordinate values and the memory reference values of the feature points in the three-dimensional polygon face caricature.

Hereinafter, a three-dimensional caricature generation method according to the present invention will be described in detail with reference to the accompanying drawings.

3 is a flowchart of a three-dimensional caricature generation method according to the present invention.

When the face image is input, the preprocessing of detecting the position of the eye from the input face image and normalizing the size of the face image is performed (S10). Hereinafter, the preprocessing step (S10) will be described in detail with reference to FIG.

4 is a detailed flowchart of a face image preprocessing step of FIG. 3.

First, if the input face image is a color image, it is converted into a black and white image (S11), and then a face region is detected from the black and white face image using a Gabor filter response (S12).

Next, binarization is performed by increasing the threshold value until both eyebrows and both eyes appear in a black area in the face image. First, the threshold range for binarization is set, and then the threshold value is changed. An initial value of the threshold value is set within the range, and pixels having a brightness higher than or equal to the threshold are white and pixels having a brightness lower than or equal to the threshold are black based on the set threshold (S13 to S14).

Here, it is preferable that the change range of the threshold value is determined according to the brightness of the input face image, because if the face is generally dark or light, setting the change range of the threshold value based on this can reduce unnecessary computation amount.

After the binarization of the face image is completed, grouping is performed to group the black pixels into various groups in the areas corresponding to the eyebrows and the eyes (S15). At this time, if the calculated distance between the black pixels horizontally separated is less than a predetermined distance, it is possible to reduce the error caused by noise by connecting the black pixels to each other, the distance between the connectable black pixels is usually 640 * It is preferable not to exceed 5 in a face image having a resolution of 480, and this value may vary depending on the resolution of the face image.

Next, the groups are grouped by labeling the grouped groups and the regions having the same label are extracted as the isolated regions (S16). At this time, the labeling algorithm may be a four-way connection tracking algorithm or an eight-way connection tracking algorithm. If the number of groups in the face image is large, it takes much time for labeling. It is desirable to consider it as noise and to eliminate it.

Then, it is determined whether there is a pair of eyebrow candidate regions and a pair of eye candidate regions among the isolated regions extracted through labeling, and when either one of the eyebrows and the eyes is not detected, both the pair of eyebrows and the pair of eyes Binarization is performed while increasing the threshold until detection (S17 to S19). At this time, the threshold value is preferably increased within the set change range.

If there is a pair of eyebrow candidate regions and a pair of eye candidate regions among the isolated regions, the error rate of the eye candidate region is determined based on information on the size, shape, and position of the eye to verify that the eye candidate region is correct. The calculation determines whether the error rate is within an acceptable range. Hereinafter, the method of calculating the error rate of the eye candidate region will be described in more detail.

In general, the eyes occupy 0.2% to 2.5% of the total area of the face and have an oval shape. The eyes are located vertically upward in the face area, and the left and right eyes are located on the same horizontal line. The information may be used to determine whether the eye candidate region fits the eye.

That is, the error rate of the eye candidate region may be calculated based on information about the size, shape, and position of the eye, and it may be determined whether the eye candidate region fits the eye according to the calculated error rate of the eye candidate region.

First, in order to calculate the error rate of the eye candidate region, the size of the eye candidate region, the ratio of the horizontal length to the vertical length, the deviation value with respect to the vertical position and the horizontal position are calculated with respect to the reference value.

Here, the size of the eye candidate region, the ratio of the width to the length, the vertical position and the horizontal position can be obtained simply according to the information of the isolated region obtained when labeling. This is because an XY coordinate value for an isolated region is obtained, and the size, ratio of width to height, vertical position, and horizontal position of the isolated region can be calculated according to the XY coordinate value.

On the other hand, since the eye size, the ratio of the horizontal length to the vertical length, the vertical position, and the horizontal position may change according to the threshold value of the binarization, the reference value according to the threshold value of the binarization should be obtained when calculating the deviation value for the eye candidate region. do.

An example in which the reference value for the eye candidate region is changed according to the binarization threshold is shown in FIG. 9. As shown in FIG. 9, when the threshold is increased, the size of the eye occupies in the face image is increased. Deviation values for the candidate areas should be obtained based on the reference value for the eye. Similarly, for the ratio of the horizontal and vertical lengths of the eyes, the vertical and horizontal positions of the eyes, The deviation value for the candidate area should be obtained.

In the present embodiment, the reference value for the eye size is set to 0.2% to 2.5% of the entire face area, so that if the size of the candidate candidate area is 0.2% to 2.5% of the entire face area, it is determined as the eye. By setting the reference value for the ratio of the lengths to 1.0 to 3.2, it is determined that the ratio of the horizontal length and the vertical length of the eye candidate region is not less than 1.0 or 3.2 or more, that is, the eye candidate region is not elliptical.

Also, in the present exemplary embodiment, the reference value for the vertical position of the eyes in the entire face region is set to 1/3 so that when the eye candidate region is located upward in the vertical direction from the face region, it is judged as the eyes, and the eye is horizontal in the entire face region. By setting the reference value for the position to 5 °, if the left / right eye candidate areas exist on almost the same horizontal line, it is judged as an eye.

Deviation values for the size of the eye candidate region, the ratio of the horizontal length to the vertical length, the vertical position, and the horizontal position are D ₁ , D ₂ , D ₃ , If D ₄ and weights for each deviation value are k ₁ , k ₂ , k ₃ , and k ₄ , the equation for the error rate Er for the eye candidate region is given by Equation 1 below.

In Equation 1, weights k ₁ , k ₂ , k ₃ , and k ₄ for each deviation value may be determined according to a learning result of a face image, for example, eyes 0.2% to 2.5 in the entire face region. If the learning result shows that the area is likely to have a size of%, the weight (k ₁ ) for the size of the eye candidate area may be set to a larger value than the other weights (k ₂ , k ₃ , k ₄ ). The weights k ₁ , k ₂ , k ₃ , and k ₄ for the deviation value may be changed.

If the error rate Er for the eye candidate region is determined to be an acceptable range (for example, 30%), that is, if the size, shape, or position of the eye candidate region is determined to be appropriate, The center point is detected (S24 to S25).

When calculating the error rate (Er) for the eye candidate area as described above, all the deviation values for the size, shape, and position are taken into consideration, thereby minimizing the possibility of misrecognizing the eyebrows as the eyes, thereby reducing the face with low resolution In the image, the center point of both eyes can be detected accurately.

On the other hand, if the user's eyebrows are white, the eyebrows may not be detected at all. In this case, the determination unit 27 recognizes this as an eyebrow candidate area when eyes on both sides are detected through binarization, and the eye candidate area in the lower area. Binarization is performed while increasing the threshold until the detection (S11 to S19). However, if a pair of eye candidate regions are not detected even after performing the binarization process according to the maximum threshold value, an area recognized as an eyebrow candidate region is set as an eye candidate region (S20 to S21), and an error rate for the eye candidate region (Er If the calculated error rate Er is determined to be an acceptable range, the center point of the two eyes is detected in the eye candidate region (S24 to S25).

In addition, when one eyebrow of the user is covered by the hair, only one eyebrow may be detected. In this case, the determination unit 27 ignores the detection of only one isolated region (eyebrow) through the binarization and continues to perform the binarization. When a pair of left and right eyes are detected through binarization, the pair is recognized as an eyebrow candidate region and binarization is performed while increasing a threshold value until an eye candidate region is detected in the lower region (S11 to S19). However, even when the binarization process according to the maximum threshold is not detected, if the symmetrical eyebrow candidate region is not detected, the region recognized as the eyebrow candidate region is set as the eye candidate region (S20 to S21), and the error rate for the eye candidate region ( If the calculated error rate Er is determined to be an acceptable range, the center point of the two eyes is detected in the eye candidate region (S24 to S25).

On the other hand, when one eyebrow and one eye are covered by the hair, even if binarization is performed according to the maximum threshold value, a pair of isolated regions are not detected. In this case, if only one eye candidate region exists on either side of the eye, the eye After generating a pair of eyes based on the candidate region (S22 to S23), if the calculated error rate Er is determined to be an acceptable range, the error rate Er is calculated for the pair of generated eye candidate regions. Detect the center point of both eyes in (S24 ~ S25).

In the present invention, except that both eyes are covered by the hair, the reason is that both eyebrows may be covered by the hair in the general facial image, but the eyes are not very rarely covered by both eyes.                     

In addition, in the present embodiment, the verification conditions of the eye candidate region are limited to the size, the ratio of the horizontal length to the vertical length, the vertical position, and the horizontal position. Accordingly, the error rate is calculated for the eye candidate region. It is also possible to use conditions.

Next, the size of the face region is normalized to a size that can be mapped 1: 1 with the 3D face base model stored in the 3D polygon face base model DB 15 based on the detected positions of the two eyes (S26). In this case, when the face image is slightly inclined, it may be corrected at a normal angle.

Referring to FIG. 3 again, after the preprocessing of the face image as described above is completed, if the face component initial position detector 33 detects an initial position of each face component by applying a Canny filter to the face region (S30), The ASM loading unit 31 loads the ASM to the preprocessed face image to correspond to the detected initial position (S50).

On the other hand, in addition to using the Canny filter as described above when detecting the initial position of the facial components, eyebrows, eyes, nose according to the geometric position characteristics between the facial components based on the center point of the two eyes detected through the binarization It is also possible to detect the initial position of the mouth.

Next, the ASM adaptor 37 adapts the loaded ASM using the morphological change characteristic data of the ASM stored in the training data DB 11 to contour the eyebrow, eye, nose, lips, or face from the preprocessed face image. Extract feature points for (S70).

When the feature point is extracted, ASM adaptation is performed in a state in which the search region of the feature point extraction for each facial component is limited in advance. The limitation of the search region has been described in detail with reference to FIG. 1, and thus a detailed description thereof will be omitted. .

Next, the face caricature generation unit 39a loads the 3D face base model and the 2D face texture from the 3D polygon face base model DB 15 and the 2D face texture DB 13 (S90), and the loaded After mapping the 2D face texture to the 3D face base model (110), the 3D face base model is modified according to the coordinate values of the extracted feature points to generate a 3D polygon face caricature (S130).

Here, the two-dimensional face texture is to apply skin color, eyebrow color, lip color, etc. to the three-dimensional face basic model, as shown in FIG. 12 (a), and two-dimensional as shown in FIG. 12 (b). Mapping the face texture to the three-dimensional face base model reveals a more natural and sophisticated three-dimensional face caricature.

The deformation of the 3D face basic model according to the coordinate values of the extracted feature points will be described as follows. The feature points extracted through the ASM adaptor 37 have XY coordinate values. When the feature points are applied to the 3D face base model based on the XY coordinate values, the polygonal components on the 3D face base model are displaced and consequently Similar to the user's face, the three-dimensional face basic model is deformed.

FIG. 13 is a diagram for explaining deformation of a 3D polygon face basic model. As shown in FIG. 13, basic feature points on a 3D face basic model are included in a triangle formed by three points A, B, C, feature points A, and BC. Suppose a given polygon is D, DELTA ABC is Area, DAC is A1, DAC is A2, and DDA is A3.

When the feature points A ', B', and C 'extracted through ASM adaptation are applied on the three-dimensional face base model, the base feature points on the face base model move to A → A', B → B ', and C → C', respectively. Accordingly, the polygons on the three-dimensional face basic model are also displaced, wherein the displacement of the polygon D is given by Equation 2 below.

이다.In Equation 2, w1, w2, and w3 are values of A1 divided by Area, A2 divided by Area, and A3 divided by Area:

to be.

When the polygon D is displaced according to the application of the feature point, the polygons around the polygon D are also displaced in the same manner, thereby deforming the basic three-dimensional face model similarly to the user's face.

When applying the extracted feature points on the three-dimensional face basic model, information other than the XY coordinate value, for example, the coordinate value in the Z-axis direction, uses the anthropological standard data of the three-dimensional face basic model. For example, if the X, Y coordinates of the feature points extracted through ASM adaptation are (170, 210), and if they are applied on the three-dimensional face basic model, the X, Y, Z coordinates of the feature points are (170, 210, 35). Where the coordinate value 35 in the Z-axis direction is an anthropometric standard data value of the three-dimensional face basic model.

On the other hand, as described above, in addition to using the anthropometric standard data of the three-dimensional face basic model, it is also possible to generate a three-dimensional caricature more similar to the user using the side face image as described below.

First, when the side face image is input, the preprocessing unit 31 detects the end point of the nose and the eyebrows from the input side face image and normalizes the side face image based on this, and the face component initial position detection unit 33 performs the preprocessing. Each side initial position of the face component is detected in the extracted face image. Then, the ASM loading unit 35 loads the side ASM stored in the memory unit 10 into the preprocessed side face image so as to correspond to the detected side initial position, and the ASM adaptor 37 adapts the loaded side ASM. YZ coordinate values of feature points for eyebrows, eyes, noses, lips, or face contours are extracted from the preprocessed side face images. Next, the face caricature generation unit 39a generates a three-dimensional polygon face caricature by modifying the three-dimensional polygon face basic model according to the XY and YZ coordinate values of the feature points extracted by the ASM adaptor 37.

In addition, in order to generate a three-dimensional caricature that is similar to the user but emphasizes the flatness or exaggeration, the face basic model may be modified by using a snake algorithm. Hereinafter, the three-dimensional face basic model is modified by using the snake algorithm. A brief description will be given of how to do this.

Snake algorithm is a method of extracting lines, edges, contours, etc. in the image by repeatedly performing energy minimization such as image forces (imgae forces), in the present invention by applying the Snake algorithm shape of the mutual position and feature points between the feature points The XY position transformation is performed while keeping the s possible, and the remaining peripheral points are XYZ transformed based on the coordinate values on the basic model, thereby creating a caricature that is similar to the user's face but emphasizes the flatness or exaggeration.

An example of a caricature with exaggerated emphasis is shown in FIG. 14. As illustrated in FIG. 14, various three-dimensional characters similar to the user's face may be generated by using an exaggerated three-dimensional face base model and applying only important features in the user's face components to the three-dimensional face base model.

Referring back to FIG. 3, after the 3D polygon face caricature is generated, the body caricature generator 39b generates a 3D body caricature based on the 3D body model stored in the 3D body model DB 17 ( S150), and finally synthesizes the three-dimensional polygon face caricature and the three-dimensional body caricature to complete the three-dimensional caricature (S170).

On the other hand, if a two-dimensional caricature is required to generate a two-dimensional caricature by projecting the three-dimensional caricature in two dimensions through the projection unit 40 (S190 ~ S210), if the transmission of the three-dimensional caricature is required, The coordinate value and the memory reference value are extracted and transmitted (S230 to 250). Hereinafter, the method of restoring the 3D caricature based on the coordinate values and the memory reference values of the transmitted feature points will be described.

FIG. 5 is a flowchart of a method of restoring a three-dimensional caricature transmitted in FIG. 3.

First, when the coordinate values and the memory reference values of the feature points extracted from the 3D caricature are received, the face caricature generation unit 39a receives the memory received from the 3D polygon face base model DB 15 and the 2D face texture DB 13. A 3D face base model and a 2D face texture corresponding to the reference value are loaded (S310).

Next, the face caricature generation unit 39a maps the two-dimensional face texture to the three-dimensional face base model, and deforms the three-dimensional face base model according to the coordinate values of the received feature points to restore the three-dimensional polygon face caricature ( S330 ~ S350).

Then, the body caricature generation unit 39b loads the three-dimensional body model corresponding to the received memory reference value from the three-dimensional body model DB 17 to restore the three-dimensional body caricature, and the three-dimensional polygon face caricature thus restored. And 3D body caricature is synthesized to restore the 3D caricature (S370 ~ S390).

As described above, the three-dimensional caricature transmission and restoration method may be provided with a caricature sticker vending machine, a computer, a mobile phone, a PDA as well as a database capable of restoring the three-dimensional caricature, as well as between the three-dimensional caricature generating apparatuses 100 and 100. It can be applied to various fields such as a communication terminal, a game using an avatar, and a chat internet service. For example, a 3D caricature transmitted from a caricature sticker vending machine can be restored on an internet avatar site, or a 3D caricature transmitted from a computer is a mobile phone. Can be restored from

Although the present invention has been described with reference to one embodiment shown in the drawings, this is merely exemplary, and it will be understood by those skilled in the art that various modifications and equivalent other embodiments are possible. Therefore, the true technical protection scope of the present invention will be defined by the technical spirit of the appended claims.

 According to the 3D caricature generation apparatus and method of the present invention, it is possible to accurately extract the feature point information of the elements that characterize the face by using the ASM technique specialized for the face to generate a three-dimensional caricature with high similarity to the user's face It has an effect.

In addition, according to the present invention, when a two-dimensional caricature is required, the three-dimensional caricature can be easily converted into a two-dimensional caricature by projection, and when transmitting the three-dimensional caricature, only the face feature information and the memory reference values used for the three-dimensional caricature are transmitted. Therefore, the system burden can be minimized.

In addition, according to the present invention, there is an effect that the caricature sticker vending machine, a computer, a mobile phone, a communication terminal such as a PDA, and the like can be easily restored and used in the three-dimensional caricature.

Claims (34)

An apparatus for generating a caricature based on a face image of a user, A memory unit for storing an ASM required for generating a 3D caricature and a 3D polygon face basic model; A pre-processing unit for detecting the position of the eye in the input face image and normalizing the size of the face image; A facial component initial position detector for detecting respective initial positions of facial components in the preprocessed facial image; An ASM loading unit loading the ASM stored in the memory unit into the preprocessed face image so as to correspond to the detected initial position; An ASM adaptor adapted to extract the feature points for the facial component by adapting the loaded ASM; And Caricature generation unit for generating a three-dimensional polygon face caricature by modifying the loaded three-dimensional polygon face base model according to the coordinate values of the feature points extracted through the ASM adaptor after loading the three-dimensional polygon face base model from the memory unit 3D caricature generating device comprising a. The method of claim 1, wherein the preprocessing unit, A black and whiteization unit converting the input face image into a black and white image when the input face image is a color image; A face region detector for detecting a face region from the input face image using a Gabor filter response; An eye detector for detecting a position of an eye in the face region detected by the face region detector; And And a face normalization unit for normalizing the face region size to a size that can be mapped 1: 1 with the 3D polygonal face base model stored in the memory unit based on the position of the eye detected by the eye detector. 3D caricature generating device. The method of claim 2, wherein the eye detection unit, A binarization unit for binarizing the input face image based on the threshold value; A grouping unit for grouping pixels having a black color in a region corresponding to an eyebrow and an eye when a binarized face image is input from the binarization unit; A labeling unit for labeling the various groups detected by the grouping unit, ordering the various groups, and extracting a region having the same label as an isolated region; And And a determination unit configured to set an eyebrow candidate region and an eye candidate region among the isolated regions extracted through the labeling unit, and determine whether the set eye candidate region fits the eye. The apparatus of claim 3, wherein the determination unit performs binarization while varying a threshold value until both the eyebrow candidate region and the eye candidate region are detected. The apparatus of claim 1, wherein the face component initial position detector detects each initial position of a face component by applying a Canny filter to the preprocessed face image. The apparatus of claim 1, wherein the memory unit further includes shape change feature data of the ASM, and the ASM adaptor further comprises eyebrows, eyes, and eyebrows from the preprocessed face image using the shape change feature data of the ASM stored in the memory unit. 3D caricature generating device, characterized in that for extracting feature points for the nose, lips or facial contour. The apparatus of claim 1, wherein the ASM adaptor restricts a search region for feature point extraction for a face component when the feature point is extracted. The 3D polygon face base model of claim 1, wherein the memory unit further comprises a 2D face texture for texturing the 3D polygon face base model, and the caricature generation unit generates the 2D face texture stored in the memory unit. 3D caricature generating device, characterized in that for mapping to. The method of claim 1, wherein the memory unit further comprises a three-dimensional body model for generating a three-dimensional body caricature, the caricature generation unit to generate a three-dimensional body caricature based on the three-dimensional body model stored in the memory unit 3D caricature generating device characterized in that. The method of claim 1, wherein when the input face image is a side face image, The preprocessing unit detects the end point of the nose and the eyebrows in the input side face image and normalizes the side face image based on this, The face component initial position detection unit detects each side initial position of the face component in the preprocessed face image, The ASM loading unit loads the side ASM stored in the memory unit in the preprocessed side face image so as to correspond to the detected side initial position, The ASM adaptor adapts the loaded side ASM to extract coordinate values of feature points for a face component from the preprocessed side face image, and And the caricature generator generates a 3D polygon face caricature by modifying the 3D polygon face basic model according to the coordinate values of the feature points extracted by the ASM adaptor. The apparatus of claim 1, further comprising a projection unit configured to obtain a two-dimensional face caricature by projecting the three-dimensional polygon face caricature generated through the caricature generation unit in two dimensions. The apparatus of claim 1, further comprising a caricature transceiver configured to extract and transmit coordinate values and memory reference values of feature points in the 3D polygon face caricature generated by the caricature generator. The method of claim 12, wherein the coordinate values and the memory reference values of the feature points of the 3D polygon face caricature are received from the caricature transceiver. The caricature generation unit, After loading the 3D polygon face base model and the 2D face texture corresponding to the received memory reference value from the memory unit, the 2D face texture is mapped to the 3D polygon face base model and coordinates of the received feature points are obtained. And a three-dimensional polygon face caricature to restore the three-dimensional polygon face caricature according to the value. In the method for generating a caricature based on a face image of a user, A preprocessing step of detecting a position of an eye in the input face image and normalizing the size of the face image; Detecting respective initial positions of facial components in the preprocessed facial image; Extracting feature points by loading the ASM onto the preprocessed face image to correspond to the detected initial position and then adapting the loaded ASM; And And after loading the 3D polygon face base model, generating the 3D polygon face caricature by modifying the loaded 3D polygon face base model according to the coordinate values of the extracted feature points through the ASM adaptation. 3D caricature generation method. The method of claim 14, wherein the pretreatment step, Converting the input face image into a black and white image if it is a color image; Detecting a face region in the input face image using a Gabor filter response; Detecting an eye position in the detected face region; And And normalizing a face region size to a size that can be mapped 1: 1 with the 3D polygon face basic model based on the detected eye position. The method of claim 15, wherein the eye position detection step, Binarizing the input face image based on the threshold value; Grouping black pixels in the binarized face image; Labeling several groups detected by the grouping, ordering the various groups, and then extracting a region having the same label as an isolated region; And And setting an eyebrow candidate region and an eye candidate region among the extracted isolated regions and detecting a center point of two eyes from the set eye candidate region. The method of claim 16, wherein binarizing the face image comprises: Setting a threshold value change range according to the brightness of the input face image, setting an initial value of the threshold value within a change range of the threshold value, and binarizing the input face image based on the threshold value set; 3D caricature generation method comprising a. The method of claim 16, wherein the grouping step, And connecting the black pixels to each other if the calculated distance between the black pixels horizontally separated is less than or equal to a predetermined distance. The method of claim 16, wherein the labeling step, And removing a group having a predetermined size or less from among the plurality of groups detected by the grouping as noise. 17. The method of claim 16, wherein when there is no pair of eyebrow candidate regions and one pair of eye candidate regions among the extracted isolated regions, the threshold value is determined until both the pair of eyebrow candidate regions and the pair of eye candidate regions are detected. 3. The method of claim 3, further comprising performing binarization while changing. The method of claim 16, wherein detecting the center points of the two eyes from the eye candidate region comprises: Calculating deviation values for the eye candidate area by comparing the information of the eye candidate area with previously stored reference values; Calculating an error rate of the eye candidate region by adding weights to each of the deviation values and summing all weighted deviation values; And And determining whether an error rate of the eye candidate region is within an acceptable range. 22. The method of claim 21, wherein the prestored reference values comprise values for eye size, ratio of horizontal to vertical length, vertical position or horizontal position. 22. The method of claim 21, wherein the information on the eye candidate region is compared with reference values for the corresponding threshold value when calculating the deviation value for the eye candidate region. 22. The method of claim 21, wherein when only a pair of eyebrow candidate regions are detected among the extracted isolated regions, an error rate for the eye candidate region is calculated after setting the eyebrow candidate region as an eye candidate region. How to create caricatures. 22. The method of claim 21, wherein if only one of the left and right eyebrow candidate regions is detected among the extracted isolated regions, the eyebrow candidate region is set as an eye candidate region and then an eye pair is generated based on the eye candidate region. 3. The method of claim 3, wherein the error rate of the pair of eye candidate regions is calculated. The method of claim 14, wherein the initial position detection of the face component comprises: And determining an initial position of each facial component by applying a Canny filter to the preprocessed facial image. 15. The method of claim 14, wherein extracting the feature points comprises extracting feature points for eyebrows, eyes, noses, lips or facial contours from the preprocessed facial image using the morphological change feature data of the ASM. 3D caricature generation method characterized in that. 15. The method of claim 14, wherein the search region for feature point extraction is limited in advance when the feature point is extracted for the face component. 15. The method of claim 14, further comprising mapping a two-dimensional face texture to the three-dimensional polygonal face base model. 15. The method of claim 14, further comprising generating a three-dimensional body caricature based on the three-dimensional body model. The method of claim 14, wherein when a side face image is input, Detecting an end point of the nose and an eyebrow from the input side face image and normalizing the side face image based on the detected end face; Detecting each initial position of a side of a face component in the preprocessed face image; Loading a side ASM into the preprocessed side face image so as to correspond to the detected side initial position; Adapting the loaded side ASM to extract coordinate values of feature points for a face component from the preprocessed side face image; And And generating a 3D polygon face caricature by modifying the loaded 3D polygon face basic model according to the coordinate values of the feature points extracted through the ASM adaptation. 15. The method of claim 14, further comprising the step of projecting the three-dimensional polygon face caricature in two dimensions to obtain a two-dimensional face caricature. 15. The method of claim 14, further comprising extracting and transmitting coordinate values and memory reference values of feature points in the three-dimensional polygon face caricature. The method of claim 33, wherein the coordinate values and the memory reference values of the feature points of the three-dimensional polygon face caricature are received. Loading a 3D polygon face base model and a 2D face texture corresponding to the received memory reference value; Mapping the two-dimensional face texture to a three-dimensional polygonal face base model; And And reconstructing a three-dimensional polygon face caricature by modifying a three-dimensional polygon face basic model according to the coordinate values of the received feature points.

Images