KR20090125905A - Apparatus and method for complexing image in visual communication terminal - Google Patents

Abstract

PURPOSE: An apparatus and a method for combining images in an image communication terminal are provided to combine various character images with a facial image obtained from a camera, thereby showing the face of a user. CONSTITUTION: A face area decision unit(130) decides a face area. An outline detection unit(135) detects an outline area in the face area. A characteristic point decision unit(140) decides each characteristic point in an eye area, a nose area, and a lip area. An image combining unit(145) combines an image of a camera with a prepared character image based on the characteristic point.

Description

Apparatus and method for complexing image in visual communication terminal

The present invention relates to an image synthesizing apparatus and method of a video communication terminal, and in particular, by synthesizing various characters, such as a mustache, glasses or tear images, to a face image obtained from a camera, the emotional state of the user to the video communication terminal on the other side of the call The present invention relates to an image synthesizing apparatus and method for a video communication terminal capable of expressing a user's face decorated with various characters.

In recent years, thanks to the rapid development of information and communication fields, mobile communication terminals have become so popular that they are essential portable devices for the whole nation. Furthermore, voice calls and text messages have been the main functions. With the advent of the new models with various functions are released one after another to replace the old models quickly.

In particular, a third generation communication network supporting video telephony or a video conferencing system, such as a W-CDMA network or a next generation (fourth generation) mobile communication network, which is a development model thereof, such as a wireless broadband (WiBro) network is established. In the state of being stopped or moving, a video telephone service is introduced to exchange voice and video with a call counterpart through a mobile communication terminal.

As the video phone service became commercially available, on the other hand, there was a problem that one's self-hidden appearance could be shown to the other party on the phone without filtering. For example, sleeping eyes, messed up hair, or bare face without a toilet can be passed on to people who are not acquaintances, which can interfere with the privacy of users.

Accordingly, the present inventors have led to the present invention by studying an image synthesis method that can protect the user's face and further enhance the user's face in a fun or beautiful way.

The present invention has been made to solve the above-described problem, and finds a feature part of the face (eg, eyes, nose or mouth) in an image obtained from the camera, and based on the found feature part, It is an object of the present invention to provide an image synthesizing apparatus and method for a video communication terminal that can decorate a user's face by synthesizing a character image prepared in advance.

In order to achieve the above object, an image synthesizing apparatus of an image communication terminal according to the present invention comprises: a binarization unit for binarizing a camera image acquired by a camera into a skin region and a non-skin region; A face region determiner configured to determine a face region among the flesh colors divided by the binarization part; An outline detection unit detecting an outline area in the face area determined by the face area determination unit; The face region determined by the face region determiner is divided into four parts and divided into an eyebrow region, an eye region, a nose region, and a lip region, and a feature point of the eye is determined from an outline region existing in the eye region, and is present in the nose region. A feature point determination unit for determining a feature point of the nose in the contour region, and a feature point determination unit for determining a feature point of the lips in the contour region existing in the lip region and the feature point determined by the feature point determination unit, which are previously prepared with the camera image. It comprises an image synthesis unit for synthesizing the character image.

In the above configuration, the binarization unit preferably converts the RGB component of the camera image into a YUV format and then binarizes it. That is, the binarization unit binarizes (colors and uncolors) the camera image with color components excluding brightness components.

Preferably, the feature point determination unit equalizes the spacing between the regions in four equal parts of the eyebrow, eye, nose and lip regions.

The image synthesizing apparatus may further include a noise removing unit configured to perform erosion and expansion operations on the binary image by the binarization unit and output the same to the face region determination unit.

Preferably, the face region determiner determines a face region using a labeling operation. That is, the face region determination unit determines the largest one of the flesh color areas labeled by the labeling operation as the face region.

The contour detector may detect the contour region by using a horizontal Sobel mask operation.

The feature point determiner may perform a labeling operation on the contour areas detected by the contour detector, and remove those having a predetermined size or less from the labeled contour areas. That is, according to the function of the feature point determiner, since the thing less than a predetermined size (blemish or noise) is excluded, the feature point can be more accurately determined.

The image synthesis unit generates a first synthesis image by finding a change region in the camera image based on the feature point determined by the feature point determiner and replacing the change region with a character image prepared in advance, and the first synthesis image. It is preferable to generate a second composite image by finding a return region and replacing the return region with the camera image.

The character image is composed of a character part and a black or white part that is distinguished from the character part, and the black or white part is preferably the return area.

Preferably, the image synthesizing unit corrects a defect area in the secondary composite image.

The image synthesizing unit replaces the defective area with the camera image.

In addition, the video synthesizing method of the video communication terminal according to the present invention is performed in the video communication terminal is provided with a video synthesizing apparatus for synthesizing the camera image obtained from the camera and the character image prepared in advance, the color gamut of the camera image (A) binarizing the non-pigmented region; (B) determining a face region from the skin color areas classified by the skin color in step (a); (C) detecting the contour area from the face area determined by step (b); Dividing the face area determined by the step (c) into four equal parts into eyebrow area, eye area, nose area and lip area, determine the feature point of the eye in the contour area existing in the eye area, present in the nose area Determining a feature point of the nose in the contour area, and determining the feature point of the lips in the contour area existing in the lip area based on the feature point determined by (d) and (d). And (e) synthesizing the prepared character image.

In the above configuration, step (a) is preferably performed by converting the RGB component of the camera image to YUV format and then binarizing.

The step (d) is preferably performed by equalizing the spacing between the areas in four equal parts.

The image synthesizing method further includes (ab) performing an erosion and expansion operation on the binary image according to the step (a), wherein the step (b) includes a face region in the binary image that has undergone the step (ab). Preferably by determining

Step (b) is preferably performed by determining a face region using a labeling operation.

Step (c) is preferably performed by detecting the contour area using a horizontal Sobel mask operation.

The step (d) is preferably performed by performing a labeling operation on the contour areas detected by the step (c), and removing those below the predetermined size among the labeled contour areas.

In step (e), a first synthesis image is generated by finding a change area in the camera image based on the feature point determined in step (d) and replacing the change area with a character image prepared in advance. The method may be performed by finding a return region in the second composite image and generating a second composite image by replacing the returned region with the camera image.

The character image is composed of a character part and a black or white part that is distinguished from the character part. In the step (e), the search for the return area is preferably a process of finding the black or white part.

The image synthesizing method may further comprise (f) correcting a defective area in the secondary composite image generated by the step (e).

The step (f) is preferably performed by replacing the defective area with the camera image.

According to an image synthesizing apparatus and method of a video communication terminal according to the present invention, a feature part of a face (for example, an eye, a nose, or a mouth) is found in an image obtained from a camera, and the image is obtained from the camera based on the found feature part. By synthesizing the character image prepared in advance in the image, there is an effect that can hide the part of his or her face that you do not want to show to the other person during the video call or decorate the user's face in a fun or beautiful way.

Hereinafter, a video synthesizing apparatus and method for a video communication terminal according to a preferred embodiment of the present invention will be described in detail with reference to the accompanying drawings. A network configuration in which a video telephony service can be implemented with reference to FIG. Let's look at.

1 is a diagram illustrating a network in which a video phone service may be implemented.

As shown in FIG. 1, a network configuration of a system for implementing a video telephony service includes a computer, a wired telephone, and a mobile communication terminal (hereinafter, collectively referred to as a video communication terminal) each having a video telephony function (100). ) Is connected by the video communication network (200). In addition, the video communication network 200 may be connected to the Internet 400 to the gateway 300, whereby the video communication terminal 100 may download various characters necessary for image synthesis from the character providing server 500. .

Here, the video communication network 200 may be implemented as a W-CDMA network, the specific configuration of which is a radio signal transmission and reception, wireless channel encoding and decoding between the mobile station, that is, the mobile communication terminal, signal strength and image quality measurement, A base station that performs baseband signal processing, diversity, radio resource management, and self-maintenance functions, and a wireless network controller that controls a plurality of base stations, performs matching with base stations, inter-cell handover processing, and call control. It provides a video call and various additional functions between a radio network controller, a video telephone line switching network to which a wired telephone or a webcam-equipped computer is connected, and a video telephone line switching network and a wireless network controller. It may comprise a Mobile Switching Center (MSC) that provides connectivity to various other equipment and external networks.

Next, the Internet 400, as is well known, is a network of networks that connect a computer network based on a communication protocol called TCP / IP (Transmission Control Protocol / Internet Protocol) scattered around the world through wired / wireless networks. Should be interpreted as Here, the method of wirelessly accessing the Internet 400 is based on a platform such as WAP (Wireless Application Protocol) or WIPI (Wireless Internet Platform for Interoperability), or accesses through a mobile communication network or uses a public wireless LAN and an access point. Wi-Fi, a wireless Internet service that can provide high-speed Internet access even when stationary or slow, at ADSL quality and cost, and also known as a portable Internet.

2 is a block diagram of an image communication terminal equipped with an image synthesizing apparatus according to an embodiment of the present invention, and FIGS. 7 to 15 are screen views of images processed by the image synthesizing apparatus shown in FIG. 2.

As shown in FIG. 2, the video communication terminal 100 of the present invention fleshes out an image (hereinafter, referred to as a 'camera image') obtained by the camera 115 and the camera 115 as shown in FIG. 7. Binarization unit 120 for binarizing regions and other regions, noise removal unit 125 for removing noise components from binarized images by binarization unit 120, and outputting them, as shown in FIG. The face region determination unit 130 determines the face region among the skin regions classified as flesh colors by the binarization of 120, and as shown in FIG. 9, the outline region in the face region found by the face region determination unit 130. , Contour detection unit 135 for detecting boundary portions of eyes, nose and mouth, feature point determination unit 140 for determining a feature point of a face, that is, a reference point for image synthesis, in the contour area detected by the contour detection unit 135, FIG. As shown in 10 to 12 The image synthesizer 145 for synthesizing and outputting the camera image and the character image based on the feature points, the character image storage unit 150 and the antenna storing various character images downloaded from the character providing server 500. RF transmission / reception to transmit an image synthesized by an RF transmitter / receiver 155, a key input unit 165, a display unit 160, and an image synthesizer 145, which transmit / receive a radio frequency (RF) signal with a base station. The controller 110 may be configured to collectively control the video communication terminal 100, such as to control the receiver 155.

In the above-described configuration, the binarization unit 120 converts the RGB component of the camera image into the YUV format (see Equation 1 below), and then, at a V value, for example, a value between 10 and 60 is a flesh component. The value is to binarize the camera image by dividing it into non-pigmented components. As a result, the image information binarized into the black region (colored region) and the white region (non-colored region) as shown in FIG. 7 is output to the noise removing unit 125. Here, although the range of the V value recognized as the flesh component is set to 10 to 60, it may be set to another range depending on race.

Y = 0.3R + 0.59G + 0.11B

U = (B-Y) * 0.493

V = (R-Y) * 0.877

As is well known, YUV is a format in which color is represented by three pieces of information: Y (luminance component), U (difference between luminance component and red component), and V (difference between luminance component and blue component). As shown in the above equation, the binarization unit 120 binarizes an image using a V (or U) value of a color component in a state in which a luminance (brightness) component is excluded.

Next, the noise removing unit 125 performs an erosion operation and a dilation operation during a binary morphology operation that is well known as an image processing technique. Other areas are clearly distinguished.

More specifically, the erosion operation serves to expand the background (other regions) and reduce the size of the object with respect to the object (shaded area). As a result, referring to FIG. 7, blemish components around the face area are removed or reduced. Instead, the size of the original image (face area) is also reduced, which undergoes an expansion operation to restore the smaller image to its original size. In other words, the expansion operation extends the outermost angle of the object. Thus, the size of the object is expanded and the background is reduced.

Next, the face region determiner 130 performs a function of determining the face region in the binary image.

Specifically, the face region determiner 130 first performs a labeling operation of assigning different numbers to the skin regions.

5 illustrates an example of such a labeling operation.

That is, the face region determiner 130 reconstructs the binary image by assigning different numbers to the skin regions using an 8-near-pixel (or 4-near-pixel) labeling technique. Here, the near-pixel refers to a pixel adjacent to the pixel of interest to be numbered, as is well known in the field of image processing.

For example, as illustrated in FIG. 5, the face region determiner 130 assigns the number of '31' to 255 (color component) to the pixel of interest. Next, the center is moved to one of eight adjacent pixels, and then the same number '31' is labeled on the corresponding pixel. This repetitive labeling results in no labeling component that can move the center, thus completing the labeling process for a single skinning region. In the same manner as above, a different number from '31' is assigned to the other skin region.

After finishing the labeling process for all skin areas as described above, other parts are removed while leaving only the largest area 20 among those labeled as shown in FIG. 8.

After removing the remainder except the large area 20, the face area 10 including the large area 20 is determined.

Next, the contour detector 135 performs a sobel mask operation that is well known in the image processing art.

6 is a diagram illustrating an example of an application result of a Sobel mask operation.

As shown in FIG. 6, when the leftmost circle image is calculated by covering the Sobel mask in the vertical (G _x ) direction and the horizontal (G _y ) direction, the vertical outline and the horizontal outline are detected.

That is, the contour detector 135 detects the contour region 11 corresponding to the eyes, nose, mouth, and eyebrows by applying a horizontal Sobel mask to the face region 10 as shown in FIG. 9.

Next, the feature point determiner 140 determines a feature point of the face in the outline region detected by the outline detector 135.

Specifically, the feature point determiner 140 first labels the contour regions detected by the contour detector 135 with different numbers.

Next, the face contours and blemishes are removed. In other words, one of the labeled ones is removed below a certain size, leaving only the contour area 11 corresponding to the eyes, nose, mouth and eyebrows.

Next, the facial region 10 is divided into four regions in the vertical direction, that is, eyebrow region, eye region, nose region and lip region. Here, the feature point determiner 140 may equalize the spacing between the regions in four equal parts of the facial region 10. In other words, the feature point determiner 140 divides the four regions at equal intervals. Of course, such an interval may cause an error in distinguishing a feature part of the face by including the neck part in the face area 10. Nevertheless, simplification of image processing, that is, processing at equal intervals is most preferable in order to smoothly determine feature points even in low-end computers. In addition, since the neck is behind the face rather than the face, that is, the present inventors binarized under various conditions, and according to the results, it was confirmed that most of the neck was not included in the face region. As a result, it was confirmed that the features of the face were correctly classified even at the intervals of the backs.

Next, as shown in FIG. 9, the eye feature point 14 is determined in the outline region existing in the eye region, the nose feature point 15 is determined in the outline region existing in the nose region, and is present in the lip region. The lip feature point 16 is determined in the outline region, and further, the eyebrow feature point 13 is determined in the outline region existing in the brow region. Here, as illustrated in FIG. 9, the feature point determiner 140 may determine coordinates located at the bottom of each of the eyebrow, eye, nose, and lip regions as the feature points. In this case, which coordinate is determined as the feature point in the contour area 11 may vary depending on the character image to be mapped. For example, in the case of image synthesis to give the effect of eyelash makeup, the coordinate located at the top of the eye region may be a feature point.

As a result, the image synthesizing unit 145 covers the face of the Spider-Man mask as shown in FIG. 10 or sunglasses as shown in FIG. 11 based on the feature points determined by the feature point determining unit 140. Is covered on the face, or the candy is mapped to the lips as shown in FIG.

Specifically, the image synthesizing unit 145 first finds the change area 31 to be replaced with the character image 40 in the camera image 30 based on the feature point, for example, the eye feature point 14, and places the character image therein. Substituting for 40, as shown in FIG. 13, a primary composite image is generated.

Next, a return area to be returned to the original camera image 30 is found in the changed area 31 replaced with the character image 40. That is, as shown in FIG. 13, the character image 40 includes a character part that is substantially necessary, such as glasses and a mustache, and a black part distinctly distinguished from the character part. The image synthesis unit 145 includes the black part, that is, the return area. Will find. Here, the image synthesis unit 145 may find the return area using, for example, conditional sentence 1 below. In addition, when the character portion in the character image 40 is a dark color system, the unnecessary portion may be white to contrast with this.

Conditional statement 1

if (image-> imageData [(640 * y + x) * 3] == 0)

Subsequently, as shown in FIG. 14, the returned region may be replaced with the original camera image 30 to generate a second composite image and output the second composite image to the controller 110. For example, the image synthesis unit 145 may replace the return region with the original camera image 30 by using the following execution statement for the return region that satisfies the conditional statement 1 described above.

Execution statement

image-> imageData [(640 * y + x)] = image_copy-> imageData [(640 * y + x)];

On the other hand, when viewing the secondary composite image generated in this way, there may be a problem in that a pixel of the camera image 30 proximate to the character image 40 does not have an original value and a defect occurs in the return process.

Therefore, once again, the image synthesis unit 145 finds the defect area 32 and corrects it to the camera image 30 to generate the final composite image as shown in FIG. 15, and outputs it to the control unit 100. It is desirable to. Here, the image synthesis unit 145 may find the defect area 32 using, for example, conditional sentence 2 below.

Conditional statement 2

if ((unsigned char) image-> imageData [(640 * y + x) * 3] <10 && (unsigned char) image-> imageData [(640 * y + x) * 3 + 1] <10 && (unsigned char) image-> imageData [(640 * y + x) * 3 + 2] <10)

3 and 4 are flowcharts illustrating an image synthesizing method of a video communication terminal according to an embodiment of the present invention.

First, as shown in FIG. 3, an image is acquired by the camera 115 in step S11. Next, the binarization unit 120 converts the RGB component of the camera image into the YUV format in step S13, and proceeds to step S15 to binarize the camera image by dividing it into a skin color component and other non-color skin components.

Next, in operation S17, the noise removing unit 125 performs erosion and expansion operations on the binary image to remove blemishes.

Next, the face area determiner 130 performs a labeling operation for assigning different numbers to the skin color areas in step S19, and then proceeds to step S21 to display the largest area 20 among the labeled ones; Other parts are removed, leaving only the FIG. 8). Next, as shown in FIG. 8, the facial region 10 including the large region 20 is determined.

Next, in step S23, the contour detector 135 detects the contour region 11 corresponding to the eyes, nose, mouth, and eyebrows by applying a horizontal Sobel mask to the face region 10.

Next, the feature point determiner 140 labels the contour areas detected by the contour detector 135 in a different number at step S25, and then proceeds to step S27 to remove contours and blemishes of the face. Next, in step S29, the facial region 10 is divided into four regions in the vertical direction, that is, the eyebrow region, the eye region, the nose region, and the lip region, and the flow proceeds to step S31 to determine the eye feature point 14 in the eye region. The nose feature point 15 is determined in the nose area, the lip feature point 16 is determined in the lip area, and the eyebrow feature point 13 is determined in the eyebrow area.

Next, in step S33, the image synthesizing unit 145 synthesizes the camera image and the character image based on any one of the determined feature points. A detailed routine thereof will be described with reference to FIG. 4.

That is, as shown in FIG. 4, the image synthesizing unit 145 finds a change region in the camera image based on the feature point in step S331, and then proceeds to step S332 to replace the found change region with a character image for primary synthesis. Create an image. Then, in step S333, after finding the return region in the primary composite image, the flow proceeds to step S334 to replace the return region with a camera image to generate a second composite image.

Next, in step S335, after finding the defective area in the secondary synthesized image by the image synthesizer 145, the secondary synthesized image is corrected by replacing the defective area with the camera image in step S336. Then, the process proceeds to step S337 to output the data of the corrected final composite image to the controller 110.

The video synthesizing apparatus and method of the video communication terminal of the present invention are not limited to the above-described embodiments, and various modifications can be made within the scope of the technical idea of the present invention.

1 is a diagram illustrating a network in which a video phone service may be implemented.

2 is a block diagram of a video communication terminal having an image synthesizing apparatus according to an embodiment of the present invention.

3 and 4 are flowcharts illustrating an image synthesizing method of a video communication terminal according to an embodiment of the present invention.

5 illustrates an example of such a labeling operation.

6 is a diagram illustrating an example of an application result of a Sobel mask operation.

 7 to 15 are diagrams illustrating screens of images processed by the image synthesizing apparatus shown in FIG. 2.

*** Explanation of symbols for the main parts of the drawing ***

100: video communication terminal

110: control unit 115: camera

120: binarization unit 125: noise removing unit

130: face area determination unit 135: contour detection unit

140: feature point determiner 145: image synthesizer

150: character image storage unit 155: RF transmission and reception unit

160: display unit 165: key input unit

200: video communication network 300: gateway

400: Internet 500: Character server

Claims (23)

A binarization unit for binarizing the camera image acquired by the camera into a skin region and a non-skin region; A face region determiner configured to determine a face region among the flesh colors divided by the binarization part; An outline detection unit detecting an outline area in the face area determined by the face area determination unit; The face region determined by the face region determiner is divided into four parts and divided into an eyebrow region, an eye region, a nose region, and a lip region, and a feature point of the eye is determined from an outline region existing in the eye region, and is present in the nose region. A feature point determiner for determining a feature point of the nose in the outline region to determine a feature point of the lips in the outline region existing in the lip region; And an image synthesizing unit for synthesizing the camera image with a character image prepared in advance on the basis of the feature point determined by the feature point determining unit. The method of claim 1, The binarization unit converts the RGB component of the camera image into a YUV format, and then binarizes the video synthesizing apparatus of the video communication terminal. The method of claim 2, And the feature point determiner is configured to equalize the spacing between the areas in four equal parts of the face area. The method of claim 1, And a noise removing unit for performing an erosion and expansion operation on the binary image by the binarization unit and outputting the noise to the face region determination unit. The method of claim 1, And the face region determiner determines a face region using a labeling operation. The method of claim 1, And the contour detector detects the contour region using a horizontal Sobel mask operation. The method of claim 1, And the feature point determiner performs a labeling operation on the contour areas detected by the contour detector, and removes those having a predetermined size or less from the labeled contour areas. The method according to any one of claims 1 to 7, The image synthesis unit generates a first synthesis image by finding a change region in the camera image based on the feature point determined by the feature point determiner and replacing the change region with a character image prepared in advance, and the first synthesis image. And a second composite image by replacing the returned region with the camera image by finding a return region in. The method of claim 8, The character image is composed of a character portion and a black or white portion that is distinct from the character portion, And the black or white portion is the return region. The method of claim 9, And the image synthesizing unit corrects a defect area in the secondary synthesized image. The method of claim 10, And the image synthesizing unit replaces the defective area with the camera image. It is performed in a video communication terminal equipped with a video synthesizing apparatus for synthesizing a camera image obtained from the camera and a character image prepared in advance, (A) binarizing the camera image into a skin region and a non-color region; (B) determining a face region from the skin color areas classified by the skin color in step (a); (C) detecting the contour area from the face area determined by step (b); Dividing the face area determined by the step (c) into four equal parts into eyebrow area, eye area, nose area and lip area, determine the feature point of the eye in the contour area existing in the eye area, present in the nose area (D) determining a feature point of the nose in the contour region, and determining a feature point of the lips in the contour region existing in the lip region; and And (e) synthesizing the camera image with a previously prepared character image based on the feature point determined by step (d). The method of claim 12, The step (a) is performed by converting the RGB component of the camera image into a YUV format and then binarizing the video. The method of claim 13, The step (d) of the video communication terminal, characterized in that the step of equalizing the interval between the four areas in equal division. The method of claim 12, (A) further comprising the step of performing the erosion and expansion operation on the binary image by the step (a), The step (b) is performed by determining a face region in the binary image which has passed the step (ab). The method of claim 12, The step (b) is performed by determining a face region using a labeling operation. The method of claim 12, The step (c) is performed by detecting the contour region using a horizontal Sobel mask operation. The method of claim 12, The step (d) is performed by performing a labeling operation on the contour areas detected by the step (c), and removing those having a predetermined size or less among the labeled contour areas. Image synthesis method. The method according to any one of claims 12 to 18, In step (e), a first synthesis image is generated by finding a change area in the camera image based on the feature point determined in step (d) and replacing the change area with a character image prepared in advance. And a second composite image is generated by finding a return region in the second composite image and replacing the returned region with the camera image. The method of claim 19, The character image is composed of a character portion and a black or white portion that is distinct from the character portion, The search of the return region in the step (e) is a process of synthesizing the video or communication terminal, characterized in that finding the black or white part. The method of claim 20, And (f) correcting the defective area in the secondary synthesized image generated by the step (e). The method of claim 21, The step (f) is performed by replacing the defective area with the camera image. A computer-readable medium having recorded thereon a program for executing the video synthesizing method of the video communication terminal according to any one of claims 12 to 22.

Images