KR100935482B1 - System and method for providing synthetic images - Google Patents

Abstract

The present invention relates to a composite image providing system for providing a composite image obtained by synthesizing a face image and a background image to a user by being connected to a user terminal via wired or wireless network. The face image receiving unit for receiving a face image from a user terminal, 3D face generator for generating 3D faces from face images, face pose converter for generating 3D faces with poses converted from 3D faces using pose data received from pose data extractors, and sizes received from size and position data extractors And a face image conversion server including a face size and position conversion unit for generating a 3D face whose face size and position are converted from the 3D face using position data, and a 2D face generation unit for generating a 2D face from the 3D face. Background to send the phone number for the selected background image to the user Unknown selection unit, a pose data extractor for extracting pose data of a face from a background image, a size and position data extractor for extracting size and position data of a face region from a background image, and a background image received from a 2D face generator. An image synthesizing server including an image synthesizing unit for synthesizing an image is provided.

Description

System and method for providing synthetic images

The present invention relates to a method for synthesizing an image, and more particularly, to a system and a method for providing a service through the Internet or a mobile by synthesizing a face image of a person on a background image.

Recently, as the digital camera has been widely used, a technology of synthesizing a human face image with a background image or a face image of several individual images taken together is widely used. In particular, since almost all mobile phones are equipped with a digital camera, a basic function of synthesizing a face photograph and a special icon taken with a mobile phone camera or a special background image is already provided by the mobile phone.

However, such image compositing technology and service has a problem that the reality is lowered because the composition of each of the background image and the face image using only the position adjustment. In particular, when a two-dimensional face image is to be synthesized into a background image, it does not match the direction of the background image or the face direction of other people already included in the background, resulting in an awkward composite image.

In addition, when simply adjusting the size or position of the image when composing the face image to the background image, the lighting method of the background image and the illumination method of the face image are different, so the reality of the synthesized image is deteriorated.

And since such image compositing has to be done by the user by using special software, it is difficult for the general public who has not acquired image compositing technology to make realistic composite image.

The present invention has been invented in view of the above points, and when the face image is synthesized in the background image, a realistic composite image is generated in consideration of the pose, size, position, and lighting of the face image, and automatically provided to the mobile phone user. The goal is to provide a system and method that can do this.

In the composite image providing system according to an aspect of the present invention, in the composite image providing system for providing a composite image synthesized by combining a face image and a background image to a user connected to the user terminal via wired or wireless network, the face image from the user terminal A face image receiving unit for receiving a 3D face generating unit for generating a 3D face from the received face image, a face pose converting unit for generating a 3D face having a pose converted from the 3D face using the pose data received from the pose data extracting unit, A face size and position converter for generating a 3D face whose face size and position are converted from the 3D face using the size and position data received from the size and position data extracting unit, and a 2D face generator for generating a 2D face from the 3D face. A face image conversion server for performing Background image selection unit to send a picture number to the user, Pose data extraction unit to extract the pose data of the face from the background image, Size and position data extraction unit to extract the size and position data of the face area from the background image, 2D face generation It is provided with an image synthesizing server including an image synthesizing unit which receives a face image from the unit and synthesizes the background image.

At this time, the background image is preferably made of a template (Template) using the 3D coordinate system and stored in the composite image providing system.

The pose data extractor extracts the feature points of the basic model face from the region where the user's face image is to be synthesized in the background image template, extracts the rotation values of the feature points based on the x, y, and z axes, and the face pose converter rotates them. It is preferable to generate a 3D face whose pose is converted using the value.

In the composite image providing system, the image synthesizing server further includes a lighting information extracting unit extracting lighting information from the background image, and the face image converting server applies the lighting information received from the lighting information extracting unit to the 3D face and the lighting of the background image. It is preferable to further include an illumination information applying unit for generating a matching 3D face. In this case, the background image stores data about the location, type, and color of the light, and the lighting information extractor extracts data about the location, type, and color of the light. The type of light can be 'Spotlight', 'Directional Light' or 'Point Light'.

According to another aspect of the present invention, there is provided a composite image providing method comprising: a composite image providing method performed in a composite image providing system connected to a user terminal and a wired or wireless network to provide a composite image obtained by synthesizing a face image and a background image to a user. (A) the user terminal accessing the composite image providing system to select a background image, (b) the user terminal receiving a telephone number for the selected background image, (c) the user terminal received the telephone number Transmitting a face image to the composite image providing system using the following steps; (d) generating a 3D face image from the received facial image by the composite image providing system; and (e) pose data of the background image by the composite image providing system. Converting the pose of the 3D face using the (F) composite image providing system the background image Converting the size and position of the 3D face using the size and position data; (g) generating a 2D face image from the 3D face by the composite image providing system; and (h) creating a 2D face image from the 3D face. Synthesizing the images, and (i) synthesizing the image transmission system to the user terminal.

At this time, the background image in step (a) is preferably made of a template (Template) using the 3D coordinate system and stored in the composite image providing system.

And, step (d), (d1) detecting the face area in the received face image, (d2) extracting each feature point of the face in the detected face area, (d3) the user of the 3D face model The method may include generating a user 3D face model reflecting a facial feature point, and (d4) mapping a texture of the user face region detected in the step (d1) to the user 3D face model.

Also, in the step (e), the feature point of the basic model face is extracted from the region where the user's face image is to be synthesized in the background image template, and the rotation value of the feature point is extracted based on the x, y, and z axes to pose the 3D face. It is preferable to convert.

In the composite image providing method, after step (f), the synthetic image providing system further comprises the step of generating a 3D face to which the lighting information of the background image is applied, the location, type, Extract data about color and apply it to 3D faces. The kind of light may be 'Spotlight', 'Directional Light' or 'Point Light'.

As described above, according to the present invention, when combining a face image with a background image, not only the size or position of the face image is adjusted, but also a pose and lighting method can be combined with the background image to generate a realistic composite image. It works.

In addition, there is an effect that a mobile phone user can be provided with a composite image using his face image easily.

Hereinafter, exemplary embodiments of the present invention will be described in detail with reference to the accompanying drawings. However, the following embodiments are provided to those skilled in the art to fully understand the present invention, and may be modified in various forms, and the scope of the present invention is limited to the embodiments described below. It doesn't happen.

1 is a block diagram illustrating a composite image providing system according to a preferred embodiment of the present invention, Figure 2 is a flow chart illustrating a composite image providing method according to a preferred embodiment of the present invention.

First, referring to FIG. 1, the composite image providing system 110 according to a preferred embodiment of the present invention is connected to the user terminal 100 through a network in a wired or wireless manner, and as a user terminal 100, a wired or wireless communication such as a mobile phone or a computer. It is not limited to a specific thing as long as this terminal is possible.

As shown in FIG. 1, a composite image providing system 110 according to a preferred embodiment of the present invention receives a face image from a user and converts the image to be suitable for compositing with a background image. And an image synthesizing server 130 for synthesizing the converted face image with the selected background image. The face image conversion server 120 includes a face image receiving unit 121, a 3D face generating unit 122, a face pose converting unit 123, a face size and position converting unit 124, an illumination information applying unit 125, and 2D. And a face generator 126, and the image synthesis server 130 includes a background image selector 131, a pose data extractor 132, a size and position data extractor 133, and an illumination information extractor 134. And an image synthesizer 135.

2 shows a flow of a composite image providing method performed in the composite image providing system 110 shown in FIG. Hereinafter, the composite image providing method performed by the composite image providing system 110 will be described in detail.

First, the user selects a background image for synthesizing his face image (step 200). The background image is prepared in advance and stored in the composite image providing system 110, and the user can select a desired one from a plurality of background images through the wired Internet or the wireless Internet.

The background image stores the size information, the location information, the pose information, and the lighting information of the region where the face image is synthesized so as to be matched with the user's face image. Information about the background image selected by the user is recorded in the background image selection unit 131 of the image synthesis server 130.

The background image can be made into a template using, for example, a 3D coordinate system. The 3D coordinate system consists of x, y, and z axes, and the x and y axes represent the actual size of the background image. That is, when the size of the background image template to be synthesized is 320 * 240, the coordinates of the background image are x (L) = 320, y (H) = 240, and z = 0. Thus, the 3D face model of the user to be synthesized is located in 320 * 240 * 0 coordinates. Here, the z-coordinate of the 3D face model is the central axis of the 3D face model. The position of the 3D face model changes only the x and y values and is placed at a specific position of the background image template. For example, FIG. 3A shows an image original before being made into a background image, and FIG. 3B shows a background image in which an area for a user's face is represented by an empty space.

When the user selects a background image, the background image selection unit 131 of the image synthesis server 130 uses a phone number for the selected background image, for example, a short message service (SMS) number or a multimedia message service (MMS). In step 201, it is transmitted to the server. Since all the background images stored in the composite image providing system 110 are given individual phone numbers, the user receives a phone number corresponding to the selected background image.

Next, the user transmits the face image to the composite image providing system 110 using the received telephone number (step 202). The face image may be a picture taken by the user with a mobile phone camera, and may use an SMS service or an MMS service method. The face image receiving unit 121 of the composite image providing system 110 stores the face image received from the user. Therefore, when the face image is registered in the face image receiving unit 121, the user transmits only the unique number of the background image to be synthesized by SMS to the SMS without the hassle of attaching the face image to MMS every time to provide the desired composite image. I can receive it.

The method of receiving a composite image using the mobile messaging service and generating the composite image desired by the user is that when the user generates the composite image, a unique MMS number of the composite image is required without the user having to access a mobile or wired Internet service. If you know that you can get a composite image anytime, anywhere. In particular, the service can be provided at a low cost to the user who feels the burden of the connection cost of the mobile Internet.

Subsequently, the 3D face generator 122 of the face image conversion server 120 generates a 3D face image from the received 2D face image (step 203). A detailed method of generating a 3D face image will be described with reference to FIG. 4.

As shown in FIG. 4, the face region is detected from the received 2D face image. The texture of the detected face area is used when mapping to 3D face modeling. Then, each feature point (eyebrow, eyes, nose, lips, facial shape, etc.) of the face is extracted in the detected face region. Subsequently, a user 3D face model is generated in which the user's face feature points are reflected from the existing 3D face model. Then, the user's face texture is mapped to the user's 3D face model. Therefore, since the texture of the face photograph input by the user is used, a realistic 3D face can be generated.

After generating the 3D face of the user, the pose of the face is converted using the pose data of the background image (step 204). That is, the pose data extractor 132 of the image synthesis server 130 extracts the pose data from the background image and transfers the pose data to the face pose converter 123, and the face pose converter 123 uses the pose data. To generate a 3D face whose pose is converted from the 3D face.

In detail, the face pose converter 123 converts a pose of the 3D face so that the 3D face image of the user may naturally match the background image template, and uses the rotation value of the 3D face model. The rotation value of the 3D face model can be extracted from the original background image as follows.

As shown in FIG. 5, the feature point of the basic model face is extracted from a region in which the face image of the user is to be synthesized in the background image template. In order to know the rotation of the face, for example, 9 feature points (left eye, right eye, brow center point, nose end point, left ball center point, right ball center point, left chin end point, right chin end point, chin center point) The facial analysis technology can be automatically extracted using AAM (Active Appearance Model) technology.

The facial feature points of the extracted basic model may be compared with the front 3D face model of the corresponding model face to calculate how each feature point is moved. FIG. 6A shows the position of the feature point of the 3D face model when it is in front, and FIG. 6B shows the position of the feature point of the 3D face model required by the background image template. That is, the left eye (E1), the right eye (E2), the center of the cranial center (M0), the nose end point (N0), the left ball center point (C1), the right ball center point (C2), the left jaw end point (S1) of Figure 6a, Right jaw end point (S2), jaw center point (S0), left eye (E1), right eye (E2), cranial center point (M0), nose end point (N0), left ball center point (C1), right ball of FIG. 6B The rotation value is extracted by comparing the center point C2, the left jaw end point S1, the right jaw end point S2, and the jaw center point S0 with each other.

The rotation value of the 3D face model can be extracted based on the x, y, and z axes. 7 shows the measurement of the degree of inclination of the head on the basis of the x-axis. Using the left eye (E1) and the right eye (E2), based on the lumbar feature point (M0) of the basic 3D face model, the angle at which the 3D face model is rotated about the x-axis, that is, the angle ∠ Rx is tilted. .

8 illustrates measuring the degree of turning the 3D face model to the side based on the y-axis. As shown in FIG. 8, the value of the 3D face rotated may be extracted through the ratio of the distance Δ (C1-S1) of C1-S1 and the distance Δ (C2-S2) of C2-S2. . That is, if the length of C1-S1 is longer than the length of C2-S2, data for turning the head of the 3D face model to the left is calculated.

9 illustrates measuring the degree to which the 3D face model moves the head up and down based on the z-axis. As shown in FIG. 9, when the lengths of N0 and S0 are measured and this length is shorter than the lengths of N0 and S0 of the basic 3D face model facing the front, it is determined that the 3D face model bows or raises his head. If the length of S1 and S2 is longer than the default value of the basic 3D face model, it is determined that the head is raised because the jaw is forward. do. Through the lengths of N0, S0, S1, and S2, data obtained by moving the head up and down can be calculated.

The face pose converter 123 generates a 3D face whose pose is converted from the 3D face of the user using the extracted pose data as shown in FIGS. 7-9.

After converting the pose of the face, the size and position of the face image are converted using the size and position data of the background image (step 205). That is, the size and position data extraction unit 133 of the image synthesis server 130 extracts the size and position data from the background image and transfers it to the size and position conversion unit 124, the size and position conversion unit 124 The size and position data are used to convert the size and position of the 3D face so that the face image is positioned at an appropriate position of the background image.

First, the size of the 3D face model is converted as follows. As shown in FIG. 10, the size and position data extractor 133 extracts face size data Δx (x2-x1) and Δy (y2-y1) required by the background image template.

However, as shown in FIG. 11, the 3D face model generated from the 2D face image transmitted by the user has a basic size of Fx, Fy, and Fz. Then, the face pose is converted by the face pose converting unit 123, and Fx, Fy, and Fz are also converted together. Accordingly, by adjusting the size of the face image to the size (Δx, Δy) of the background image template by using the converted Fx and Fy data, the face size and position conversion unit 124 converts the size of the user's face. do.

And, the position of the 3D face model can be converted as follows.

In the background image template, a position value in which the 3D face model of the user is to be placed is input in advance. That is, as shown in FIG. 12, the position where the face image of the user is synthesized is stored in the background image template, and the size and position data extractor 133 extracts the position data from the background image template. Referring to FIG. 12, a position at which a user's face is to be synthesized may be referred to as Tx and Ty by arbitrarily selecting a center point of the face, and the z-axis of the 3D face model is fixed to '0'. Then, the face size and position conversion unit 124 converts the user's 3D face model in accordance with the position data Tx, Ty of the background image template.

After converting the size and the position, a face image to which the lighting information of the background image is applied is generated (operation 206). That is, the lighting information extracting unit 134 of the image synthesis server 130 extracts the lighting information from the background image and transmits it to the lighting information applying unit 125, and the lighting information applying unit 125 transmits the lighting information to the 3D. Apply to the face to create a 3D face that matches the lighting of the background image.

The reason that the lighting effect is applied to the 3D face model is to give the 3D face model a lighting effect similar to that required by the background image template, so that the user can actually take a picture in the background. Since this lighting effect can be performed in three-dimensional space, the user's 3D face model is required. Since the user's 3D face model is rotated to the pose required by the background image, z is the z-axis width of the 3D face model. Obtain the axis data value Δz (to get the side profile of the user's face)

The z-axis data value Δz may be calculated as follows from Δx and Δy, which are the size of the 3D face model required by the pose transformed Fx, Fy, Fz and the background image.

Fx: Fz = Δx: Δz

Fy: Fz = Δy: Δz

△ Δz = (Fz * Δx) / Fx or Δz = (Fz * Δy) / Fy

In addition, the background image template stores the data of the position, type, and color of the light, and when the illumination information extraction unit 134 extracts the position, type, and color data from the background image, the illumination information application unit 125 A 3D face model with lighting effects is created according to the location, type, and color of the user 3D face model.

Specifically, FIG. 13 is a diagram for explaining the position and direction of illumination in the 3D face coordinate system. Referring to FIG. 13, coordinates where the lighting is to be positioned in the 3D face coordinate system may be designated as Lpx, Lpy, and Lpz, and the direction of the illumination shining at such a position may be set as Ldx, Ldy, and Ldz. When the lighting effect is applied to the user 3D face model according to the location and direction information of the light extracted from the background image template, the user's face image can be matched with the atmosphere of the background image.

Next, a description will be given of the type of illumination, three kinds of light in three dimensions. For example, the types of lights that can be set in the 3D face coordinate system are 'Spotlight', 'Directional Light', and 'Point Light'.

'Spotlight' is a way of setting the light to emit a cone. In the light source, that is, the position of the light stored in the background image template, the user may express a shadow on the 3D face model by emitting light in a conical shape to the 3D face model of the user. This 'spotlight' effect may be used when the background image is a stage or an enclosed space to make the background image more atmospheric. In the case of 'Spotlight', since the direction of light emission from the light source is required, Ldx, Ldy, and Ldz data, which are direction information as well as position information of illumination, are required.

'Directional Light' is a light that has the same effect as sunlight, so it can be used when the background space is outdoors. This 'Directional Light' makes it possible to create a smooth effect on the 3D face model in the background image. Since 'Directional Light' is like sunlight, the direction of the light does not need to be set. Therefore, only the position information of the illumination is required, and all values of Ldx, Ldy, and Ldz are '0'.

'Point Light' is a light that emits light in all directions centering on the light source, and it can give an effect as if the bulb is turned on. In 'Point Light', only the location information of the lighting is needed and the direction of the lighting does not need to be set, and the user's 3D face model can be synthesized in the background image by giving the effect that it is taken indoors.

Subsequently, when the lighting color is described, various lighting colors used in the background image provided to the user must be set to match the 3D face model. In this case, the illumination color can be set using an RGB color code. 14 is an RGB color table of illumination that can be given to a 3D face model. FIG. 15 illustrates a 3D face model in which light types and colors are reflected at arbitrary lighting positions. In this way, the color of the light may be adjusted to convert the 3D face image to match the background image selected by the user. Referring to FIG. 15, it is possible to know a difference in lighting effects applied to the 3D face model according to the type and color of lighting.

As described above, after the pose, size, position, and illumination information are reflected in the face image, the 2D face generator 126 of the face image conversion server 120 generates a 2D face image from the 3D face (step 207). The reason for generating the 2D face image is that the background image to be synthesized is 2D, and the synthesized image to be transmitted to the user is also 2D.

Next, the image synthesizing unit 135 of the image synthesizing server 130 receives the 2D face image from the 2D face generating unit 126 and synthesizes it with the background image (step 208). This will be described with reference to FIG. 16. FIG. 16A illustrates a state in which a lighting effect is applied to a face after the pose, size, and position of the 3D face model are converted based on data required by the background image template in the 3D coordinate system. 16 (a) is synthesized with (b) of FIG. 16, which is a background image template, and finally (c) of FIG.

Accordingly, the composite image providing system 110 transmits (c) of FIG. 16, which is a composite image obtained by synthesizing the user's face image and the background image, to the user (step 209). Image transmission may use the MMS method, and the user may use the received composite image as a background on his terminal.

1 is a block diagram illustrating a composite image providing system according to a preferred embodiment of the present invention;

2 is a flowchart illustrating a method for providing a composite image according to a preferred embodiment of the present invention;

FIG. 3A shows the original image before being made into the background image, and FIG. 3B shows the background image in which the area where the user's face is to be placed is shown as empty space.

4 is an exemplary view illustrating a method of generating a 3D face image from a 2D face image;

5 is an exemplary diagram illustrating extracting feature points of a basic model face from a region in which a face image of a user is to be synthesized in a background image template;

Figure 6a is a view showing the position of the feature point of the 3D face model when the front, Figure 6b is a view showing the position of the feature point of the 3D face model required by the background image template,

7 is an exemplary diagram illustrating extracting a rotation value of a 3D face model based on the x-axis.

8 is an exemplary diagram illustrating extracting a rotation value of a 3D face model based on the y-axis.

9 is an exemplary diagram illustrating extracting a rotation value of a 3D face model based on the z axis;

10 is an exemplary view illustrating that the size and location data extraction unit extracts the face size data required by the background image template;

11 is an exemplary diagram for explaining size setting of a 3D face model;

12 is an exemplary view illustrating that the size and location data extraction unit extracts location data from a background image template;

13 is a view for explaining the position and direction of the illumination in the 3D face coordinate system,

14 is an exemplary diagram illustrating an RGB color table of illumination that can be given to a 3D face model.

15 is an exemplary view illustrating a 3D face model in which light types and colors are reflected at arbitrary lighting positions;

FIG. 16 is an exemplary view illustrating the synthesis of a 2D face image and a background image.

Claims (13)

In the composite image providing system that is connected to the user terminal and wired or wirelessly through a network to provide a composite image of a face image and a background image to the user, A face image receiver for receiving a face image from a user terminal, a 3D face generator for generating a 3D face from the received face image, and a pose converted from a 3D face using the pose data received from the pose data extractor. Face size and position conversion unit using the size and position data received from the face pose converting unit, size and position data extracting unit, and the face size and position converting unit to generate a 3D face whose face size and position are converted from the illumination information extracting unit. A face image conversion server including an illumination information applying unit generating a 3D face that matches the illumination of the background image by applying the 3D face to the 3D face, and a 2D face generating unit generating a 2D face from the 3D face; Background image selection unit to send the user's phone number for the background image selected by the user, Pose data extraction unit for extracting the pose data of the face from the background image, Size and position of extracting the size and position data of the face area from the background image An image synthesis server including a data extractor, an illumination information extractor for extracting information about a location, a type, and a color of a light from a background image, and an image synthesizer that receives a face image from a 2D face generator and synthesizes the background image with a background image; But The background image is a composite image providing system, characterized in that the template is made using a 3D coordinate system and stored in the composite image providing system. delete The method of claim 1, The pose data extracting unit extracts a feature point of the basic model face from a region of the background image template to which the user's face image is synthesized, extracts a rotation value of the feature point based on the x, y, and z axes, And the face pose converter generates a 3D face whose pose is converted using a rotation value. delete delete The system of claim 1, wherein the type of illumination is 'spotlight', 'direction light', or 'point light'. In the composite image providing method performed in the composite image providing system that is connected to the user terminal and a wired or wireless network through the network to provide a composite image of the face image and the background image to the user, (a) the user terminal accessing the composite image providing system and selecting a background image; (b) the user terminal receiving a phone number for the selected background image; (c) transmitting, by the user terminal, a face image to the composite image providing system using the received telephone number; (d) generating a 3D face image from the received face image by the composite image providing system; (e) converting a pose of the 3D face by using the pose data of the background image by the composite image providing system; (f) the composite image providing system converting the size and position of the 3D face using the size and position data of the background image; (g) generating a 3D face to which the lighting information of the background image is applied by extracting data on the position, type, and color of the light stored in the background image and applying the same to the 3D face; (h) generating a 2D face image from the 3D face by the composite image providing system; (i) synthesizing the 2D face image and the background image by the composite image providing system; (j) comprising the step of transmitting the composite image to the user terminal by the composite image providing system, In step (a), the background image is a composite image providing method, characterized in that the template is made using a 3D coordinate system and stored in the composite image providing system. delete The method of claim 7, wherein step (d), (d1) detecting a face area in the received face image; (d2) extracting each feature point of the face in the detected face area; (d3) generating a user 3D face model in which the facial feature points of the user are reflected from the 3D face model; and (d4) mapping a texture of the user face region detected in the step (d1) to a user 3D face model. The method of claim 7, wherein the step (e), The composition of the 3D face is transformed by extracting feature points of the basic model face from the region where the user's face image is to be synthesized in the background image template and extracting rotation values of the feature points based on the x, y, and z axes. How to provide an image. delete delete The method of claim 7, wherein the type of illumination is 'Spotlight', 'Directional Light' or 'Point Light'.

Images