KR20030062586A - Human area detection for mobile video telecommunication system - Google Patents

Abstract

PURPOSE: A method for extracting a man's area in mobile communications is provided to use areas properly separated according to the kind of application by precise extraction of the face, head and body areas of a man. CONSTITUTION: A method for extracting a man's area in mobile communications includes the steps of extracting a face area from an image(S14), extracting a head area on the basis of the extracted face area(S15), extracting a body area on the basis of the extracted face area(S16), and assigning a man's area by uniting the extracted face area, head area, and body area together. First, an original image is reduced before carrying out the previous steps(S12), and the reduce image is restored to the original image size after carrying out the previous steps(S18).

Description

HUMAN AREA DETECTION FOR MOBILE VIDEO TELECOMMUNICATION SYSTEM}

The present invention relates to a method for detecting a face region in an image and a method for detecting a human region in an image by detecting a head and body region based on the detected face region.

In a mobile terminal environment capable of video communication and multimedia services, it is difficult to transmit a large amount of data in real time through a limited network environment. Since the quality of the image data to be transmitted and the amount of data are proportional to each other, high data rates must be guaranteed at the same time to ensure high image quality. The difference in image quality felt by the user in the video communication is more severe in the user area than in the entire frame area of the communication image. This is because, due to the characteristics of the video communication, the attention is focused on the caller (user) and the appearance of the caller's face and the like is the main area of interest. Therefore, if only the user region is transmitted with higher image quality, higher immersion quality can be realized with a lower data amount.

This requires a technology that automatically separates the user from the background. The technology that automatically separates the user from the background is not only video communication, but when offline editing such as video mail is possible, the background other than the face is switched to another interesting template, and in some cases, the dress or hairstyle is edited and transmitted. To make it possible.

This service requires a technology that separates the user and the background or the user's face and the background from the video communication video, and in particular, considering the limited network environment, whether the image quality as well as the real-time processing is possible is an important issue. do.

Most of the techniques for separating specific objects from image frames are difficult to process in real time or do not provide satisfactory results for various data. In particular, it is more difficult to separate objects such as people from the mobile camera environment, ie moving background.

Conventionally, there is a technique of separating an object using a difference image and an edge image. This method is simple to use, and fast processing is possible, but it is difficult to separate on a complicated screen, and the background is stationary and the moving object is separated on the premise that an object is difficult to apply to a mobile camera environment. Another technique for separating an object in the related art is a technique for separating a specific object based on color information in an image frame. This technique is also difficult to apply to mobile communication environments that require real-time processing because it takes considerable processing time to separate a certain area of color.

Among techniques for separating objects based on color information, there is a method of extracting a face region. In this method, a range corresponding to flesh color is defined in advance in the color space to designate a face area using only pixels within the range, that is, pixels having flesh color. As a method of detecting a face region based on flesh color, a method of extracting a flesh color region by converting an RGB color coordinate into a YIQ coordinate and then specifying the flesh color range in a YIQ color coordinate, or using an L * a * b color coordinate from a set of training data A method of extracting a facial region using a method of extracting a flesh color color range using a Principle Component Analysis (PCA) to express an effective flesh color color range with a smaller amount of information, and then using a fuzzy set. There is a method using the concept of calculating the probability that a particular color is a flesh color using

All of the above-described color-based face region extractions use a range of skin colors in advance, which has the advantage of requiring fast processing time, but is not suitable for practical application for the following reasons. .

When analyzing the color of the flesh area in the images acquired by using the same place with various places and various photographing devices, the colors appear very different for each image. That is, since the skin color regions appear in various colors on the color coordinates according to the colors, if all possible skin color ranges are specified in advance, the range becomes too broad and often includes areas other than the skin color in one image. This is mainly caused by color distortion caused by lighting that is distorted by color, color distortion caused by an image acquisition device dependent on an image acquisition device, and color distortion caused by a playback device dependent on an image reproducing device. have. The second and third cases described above are caused by the reason why the image acquisition and reproduction technique does not reach the level that can reflect the color of the natural image as it is, and to apply its own color filter to correct it.

Due to the above-described problems, a method using template matching has also been proposed as a face region extraction method using information other than color. In this method, a template of a human face is constructed and the configured template is scanned and matched from the minimum size to the maximum size for the entire image area.

However, this method has a large number of template matching, which results in a very long processing time even with a small template size.

Another conventional technique is a method of detecting a face by combining the method using the skin region and the method using a template.

The conventional object separation technique described so far uses only a still image, and thus uses less information than that of a moving image, thereby reducing its accuracy. In addition, the problem of processing time or accuracy is a problem that is difficult to apply in a mobile camera environment.

SUMMARY OF THE INVENTION An object of the present invention is to provide a method for detecting a human region in which a human region is separated from an image frame for video communication in real time and applied to object-based bit rate adjustment or video editing.

In addition, an object of the present invention is to provide a face detection method that can be used in the application by accurately separating the face region in real time in the image frame for video communication.

In order to achieve the above object, the present invention includes extracting a face region from an image, extracting a head region based on the extracted face region, extracting a body region based on the extracted face region, and extracting Designating a human area by adding the combined facial area, head area, and body area; It provides a method for extracting a human region comprising a.

In order to achieve the above object, the present invention includes extracting a face region from an image, setting a head reference region based on the extracted face region, obtaining a head reference color from the set head reference region, Obtaining an area having a color similar to the head reference color, excluding an area where an edge is generated from the area having a color similar to the obtained head reference color, and selecting an area adjacent to the face among the remaining areas from which the edge generation area is excluded Designating as a head region; It provides a head region extraction method comprising a.

In order to achieve the above object, the present invention includes extracting a face region from an image, specifying a body frame based on the extracted face region, setting a body reference region based on the extracted face region, Obtaining a body reference color from the body reference region, obtaining an area having a color similar to the body reference color from the inside of the designated body frame, and obtaining a region having flesh color from the inside of the body frame; It provides a body region extraction method comprising a.

In addition, in order to achieve the above object, the present invention comprises the steps of constructing a color histogram (hist) for only pixels corresponding to the skin color region in the image, when the color histogram is configured only a partial region around the center of the image Comprising a center color histogram (center_hist) for the pixels corresponding to the flesh color region, forming a flesh color group using the color histogram (hist, center_hist), by separating the region of the image for each color group Designating a face area; It provides a face region extraction method comprising a.

In addition, the present invention to reduce the image to 44 * 36, to achieve the above object, by analyzing the distribution in the color space of the pixel corresponding to the flesh color range in the reduced image to perform the facial region extraction through flesh color color grouping Specifying a head reference region based on the extracted face region and extracting a head region by checking a hair reference color within the head reference region; specifying a body reference region based on the extracted face region Extracting a body region by checking a body reference color in the body reference region, designating a human region by adding the extracted face region, the head region, and the body region and restoring it to the original image size; It provides a method for extracting a human region comprising a.

1 is a diagram showing an example of an image for explaining object-based bit rate adjustment;

2 is a diagram illustrating an example of switching a background of an image;

3 is a flowchart showing the overall process of the human area detection method of the present invention.

4 is a flowchart illustrating a facial region detection process in the present invention.

5 is a diagram for explaining a face model and an MBR for face detection;

Figure 6 is a view for explaining the color grouping in the present invention

7 is a view for explaining a head reference region in the present invention.

8 is a view for explaining a head region detection method in the present invention;

Figure 9 is a view for explaining the frame for extracting the head region in the present invention

10 is a flowchart showing a head region extraction process in the present invention.

11 is a view for explaining a body reference region in the present invention

Figure 12 is a view for explaining the body frame in the present invention

Figure 13 is a flow chart showing a body region extraction process in the present invention

14 is a diagram showing an example of a human region extracted by the present invention;

Fig. 15 is a flowchart for explaining an embodiment of the face detection method of the present invention.

FIG. 16 is a view for explaining a center region applied to the face detection method of FIG.

17 is a flowchart showing a color grouping process in FIG.

18 illustrates an HMMD color space

19 illustrates quantization in the HMMD color space.

20 is a view for explaining that a face region composed of several colors is defined as one color group by a color grouping method.

21 is a view for explaining an example of effectively separating only a face region even when a region other than the face is flesh-colored.

As mentioned above, application of the separation technology of the human domain for video communication may include bit rate adjustment and video editing to ensure a constant image quality even in a low network environment.

[Object-based Bit-rate control]

This technique is to reduce the number of allocated bits according to the important areas of the screen to keep the important areas of high quality and to reduce the quality of the other areas. This requires automatic separation of Region of Interest (ROI). In the case of video communication, the human area will be the main area and the background will be the non-important area. Also, the face area becomes a more important area among the human area, and the eyes and mouth can be regarded as more important areas in the face area.

By defining a multi-level ROI in this way, it is possible to maintain an optimum picture quality according to the network environment. In addition, video mail can be encoded into a smaller size video by allocating a non-significant area from the beginning, which is very important because it is directly related to the communication service charge as well as the problem of occupying the communication line. In other words, as the size of the image increases, the user's fee also increases, so encoding to a small size of the image is very economical for the user.

Fig. 1 (a) shows the original frame image, and Fig. 2 (b) shows the original color information after separating the ROI (in this case, the human area) from the image of (a). The background area is an example in which the size of data is reduced by representing only brightness information without color information.

FIG. 1C illustrates a case where the human area is expressed by the number of bits higher than the background area, and (d) is an example in which the entire screen is expressed by the low number of bits. In the case of (d), the deterioration of the image quality can be easily felt. However, in the case of (b) and (c), although the large amount of data has been reduced, the user does not have enough inconvenience to feel the deterioration of the image quality. This will be easily understood considering that the human area, which is an important area for the user, has a great influence on the image quality. In a real communication environment, such deterioration may be caused by frame dropping or blocking.

[Video Mail Editor]

The advent of mobile terminals equipped with cameras also enables video mail services. In the case of video mail, it is possible to compress and transmit a smaller size image by using the above-described separation of important areas. Instead of the conventional choice of giving up one of the communication fee and the image quality, the user can select a low communication rate while maintaining some image quality. In addition, various video mail editing can be performed by using a separation technique of a face region and an eye region. FIG. 2 shows an example in which a human area and a background area are separated from the original image, and a background is switched by synthesizing another background with the separated human area. 2 (a) is an original image, and (b) is a background switched image.

The human region extraction method according to the present invention includes extracting a face region, extracting a head region, and extracting a body region.

3 shows the overall process of the human area detection method of the present invention for extracting the human area. First, when an image is input to CIF or QCIF, resizing is performed to reduce the input image to a size of 44 * 36 pixels (S11 and S12). This resizing is because processing images of this size allows for very fast processing without affecting the overall performance.

Next, a face region is first extracted from the reduced image (S13 and S14). The extraction of the face region is performed through the process of extracting the face candidate region based on the flesh color (S13) and the process of identifying the face region (S14). After extraction of the face region is performed, the head region is extracted based on the extracted face region (S15), and the body region is extracted based on the extracted face region (S16). Here, the extraction order of the head region and the body region may be reversed.

Next, the extracted face, head, and body areas are combined to be a human area, and after the post-processing process S17 (this process will be described in detail later), the image is resized to the original image size. Through the process (S18), the extraction of the human domain is finished.

The human region extraction process described so far is described in more detail by dividing into a face region extraction method, a head region extraction method, and a body region extraction method.

1. Facial region extraction method

Extraction of facial areas is based on skin area extraction. The skin color area is a method of investigating whether a color value of each pixel of an image corresponds to a skin color range of a human. This method has a high processing speed, but there is a problem that the range of flesh color is too wide due to the variety of colors that are variously modulated by the lighting or the image capturing device. In other words, even when pixels belonging to the skin color range are taken, a problem is generally obtained in which regions other than the human skin are mistaken for skin color and extracted. In order to solve this problem, in the present invention, when similar color groups are gathered among the pixels belonging to the flesh color area, they are regarded as one color group, and the final face area is extracted by extracting the face candidate area for each color group. The method was taken.

In this way, even if there is a background mistaken for skin color due to the wide range of skin color, it can be distinguished from the real face area. However, even with this method, pixels inside all face areas are not extracted as flesh pixels due to various color distortions such as shadows. For this reason, even if a facial region is extracted based on the skin color region, a process of confirming whether the extracted skin color region is a real face region is necessary.

A representative example of a method of identifying a face area is a method of creating a face template in advance and comparing the same, and then identifying a similar area as a face area. In general, checking the face area is used to accurately extract the initial area because the skin area requires a relatively long time when the flesh area does not accurately point to the face area, and then checks the face area in successive frames. The process may be omitted or a restrictive method may be used, such as checking only around the previous area.

This process of face extraction is shown in FIG.

First, color pixels are extracted from an input image, and color histograms are obtained and color grouped (S21, S22, S23). Color grouping refers to the grouping of major colors grouped together on a histogram. 6 shows an example of color grouping. In FIG. 6, the similar colors gathered were designated as one group using the distribution indicated by the color histogram. In a next step S24, regions composed of pixels belonging to each color group are designated as face candidate regions, and face regions are extracted through the above-described face region checking processes S25 and S26.

The extraction of the face area is very important because the head and body areas are extracted using the result of the face area extraction. In particular, since the head region collects the face region and neighboring head candidate regions as the head region, the shape of the face region is important to properly neighbor the face region. When face shapes appearing differently depending on a person merely rely on the flesh area, there is a problem of being distorted by noise. Therefore, a predefined face shape model is used. In the present invention, a face model as shown in FIG. 5 is applied to be suitable for extracting a head region. That is, if the face area is caught in a rectangle as shown in Fig. 5 (b), the face area used for head and body area extraction is extracted as shown in Fig. 5 (b) by the model shown in Fig. 5 (a). do. As described above, the extracted face region is first expressed as a minimum bounded rectangle (MBR) including a face and then converted into a predefined face model (S25 and S26).

15 shows another embodiment of the facial region detection method of the present invention. The face detection method of FIG. 15 analyzes a color distribution only on pixels having a color of a predefined skin color range in a main image frame, obtains a main skin color color group, and then selects a facial region in order of color group having the highest importance. By designating as a candidate, it is possible to apply fast and effective face region extraction by adaptively applying skin-dependent color gamut.

First, when an image is input, all pixels are compared and determined whether the color of each pixel belongs to the skin color region (S51). Judgment conditions for the skin region were defined as follows using three axes of Hue (h), Sum (s), and Diff (d) in the HMMD color space.

Skin Area Conditions in the HMMD Color Space:

if (((h> 300 && h <360) ∥ (h <60 && h> 0)) && s> 100 &amp; & s <480 && d> 5 && d <100)

skin color;

else

Not flesh;

A color that satisfies the conditions defined above is defined as flesh color. The HMMD color space is one of the color spaces such as RGB, YCrCb, etc., and has a property that the distance in the color space well reflects the difference in color visible to the eye. The shape of the color space is a double cone shape as shown in Fig. 18A, where the vertical axis is sum representing brightness, the axis from the center to the outer edge is Diff representing color purity, and The angle represented by the circle, which is a cross section, means a hue representing a color. Fig. 18B shows a cross section of this color space. The conversion from the RGB color space to the HMMD color space is performed as follows.

max = MAX (R, G, B)

min = MIN (R, G, B)

if (max-min)> 0)

｛

// requires HUE

if (R == max) H = ((G-B) / (max-min));

else

｛

if (G == max) H = (2.0+ (B-R) / (max-min));

else

｛

if (B == max) H = (4.0+ (R-G) / (max-min));

｝

｝

H * = 60;

if (H <0.0) H + = 360;

｝

else

H = -1;

hue = H;

sum = max + min;

diff = max-min;

The RGB color space has the characteristic that the distance in the color space does not reflect the visible color difference. Therefore, in the HMMD color space where the distance in the color space reflects the actual color difference, the skin color condition is defined as described above. After that, the judgment conditions were again defined in the RGB color space to become the same condition.

The reason for re-creating the same condition in the RGB color space is because the input image is an image input by a computer and is basically expressed in the RGB color space, so that it can be directly determined without converting to another color space.

However, in order to define the flesh color range directly in the RGB color space, it is difficult to set the exact range because of the characteristics of the RGB color space described above, so first set the range in the HMMD color space, and then use the same conditions in the RGB color space. To convert.

Since the HMMD color space can be 1: 1 bidirectionally mapped to the RGB color space, the same condition can be converted to the same condition. When the flesh color condition definition in the HMMD color space is considered in the converted RGB color space, it can be expressed as follows. Can be.

Skin Area Conditions Defined in the RGB Color Space

if (r> b && r> g)

｛

if (b> g)

if (r + g> 100 && r + g <480 && g-g> 5 && r-g <100)

skin color;

else

Not flesh;

else

if (r + b> 100 && r + b <480 && r-b> 5 & r-b <100)

skin color;

else

Not flesh;

｝

else

Not flesh;

By applying the above conditions, a color histogram (hist) is formed on the pixels determined as the skin color region (S52). To construct the color histogram, the color space must be quantized. HMMD color space is quantized to 72 levels as follows.

First, after dividing into six parts by the hue axis, the six subdivided regions are further divided into four by the Diff axis, and each of the divided regions by three parts by the sum axis. (A) of FIG. 19 shows six divisions in the Hue axis, and (b) shows four divisions and three divisions in the Diff and Sum axes, respectively. The basis of quantization as described above was to calculate the distribution of values constituting one face for each element of hue, sum, and diff by experiment, and then determine the degree of equalization based on this. For example, the sum is divided into three parts because of its relatively low distinction ability, even though one face is widely distributed by illumination, while the hue is divided into six parts because it has a narrow distribution within one face.

As described above, after extracting the histogram for all regions of the input image determined to be flesh color, the histogram (cneter_hist) is extracted only for the center region of the image. The central area means a constant area from the center as shown in Fig. 16, for example, a partial area having a length corresponding to 66% of the total image height on the vertical axis and 42% of the total image width on the horizontal axis. Based on the center. The reason for constructing the histogram separately in the central area is that in the color grouping step to be described later, even if the color distribution is the largest color, the priority is higher than that of the centrally gathered color even if the color is the largest color. This not only helps to group all similar colors in the face area together by first creating a group of colors centered at the time of the color group, but also allows the designation of a color group that is most likely to be the face area among the extracted color groups. .

For this reason, if there is face region information extracted from a previous frame, a central histogram may be extracted from a face region previously extracted instead of a central partial region.

As described above, once both the histogram (hist) and the center histogram (center_hist) are obtained, color grouping is performed using the two histograms (S54). Specific examples of grouping are described in detail below.

The 72 flesh colors are usually grouped into 2-3 color groups by color grouping. Among the two to three color groups, one color group having a high degree of importance is first designated as a face region candidate (S55), and finally a face region can be extracted through face verification operations such as template matching. If there is no face matching in the area belonging to the color group having the first priority, the template matching is attempted in the area belonging to the color group having the next priority. In most cases, however, the facial region is detected by template matching in the region belonging to the color group having the first importance.

The color grouping process (S54) performed with the two histograms (hist, center_hist) is shown in detail in FIG.

First, it is determined whether color grouping is attempted first (S61). In the first attempt, bin _max having the largest bin value in the center histogram center_hist is obtained (S62).

Ten thousand and one bin _max value (V (bin _max)) a certain threshold value (Th1), obtain the bins having a similar color and color (C (bin _max)) of greater than bin _max set to the set A, and (S63, S64, S65 ), Otherwise exit.

In a next step (S66), all the bin values belonging to the set A are added in the histogram (hist), and the values are compared with the threshold Th2 (S67). In the histogram, when the sum of the bin values in set A is greater than the threshold Th2, the bins in set A are set to correspond to one color group and the color group of importance 1 is defined. S68). Then, the histogram (hist) excludes all the beans belonging to the set A (S69). If the sum of all the bin values belonging to the set A in the histogram is smaller than the threshold Th2, the process moves to the first step S61 except for bin _max in the histogram.

Once the above process is executed once, the process starts from step S63 and the largest bin value bin _max is found in the histogram (hist), and the subsequent steps (S64-S69) are repeated.

20 shows that the face region composed of several colors is defined as one color group by the proposed color grouping method. (A) of FIG. 20 is an original image, and (b) is an example of extracting a face region using the entire skin color range as before. In this example, no skin region exists in the region other than the face, so that the facial region was extracted relatively accurately. However, since the skin color range is quantized to 72 colors, it can be seen that the same face area is composed of several skin colors as shown in (c). However, in (d), one face is expressed as one flesh color group through the color grouping process, and the face region is easily extracted.

21 shows an example of effectively separating only the face area even when the area other than the face is flesh colored by the proposed method. (a) is the original image, and (b) is an example of extracting a face region using the entire skin color range as before. As can be seen from the figure, it is difficult to extract a face region because a wide area different from the face is classified as skin color. Figures (c) and (d) show that the present invention accurately extracts the face and background areas even though they belong to the same skin color range.

As can be seen from Fig. 20 and Fig. 21, it can be seen that a relatively accurate face region is extracted using the skin color group region. However, small noises captured around the face area in the drawing are not a problem for human eyes, but when processed by a computer, the noise can be removed to capture the correct area. In the present invention, after converting the image representing the skin color group region into a grid image composed of N * M grids, the color region is naturally connected by connecting the skin color group region with a connected component to leave only the largest element. Removal method was used.

As such, when the original image is composed of N * M grids, if the original image is CIF (352 * 288), the image is blocked with an 8 * 8 grid, and the original image is QCIF (176 *). In the case of 144), the image is blocked by a 4 * 4 grid. When the density of flesh color pixels in each grid exceeds a certain threshold, the grid is defined as a flesh color grid, so that the actual processing shows the same processing speed as the 44 * 36 image processing regardless of the image size.

The method described so far has introduced a method of obtaining a connection element by blocking the original image to extract the face region after the skin color grouping on the original image. However, as another embodiment, the original image may be first reduced to 44 * 36 size, and then the flesh color grouping may be performed using the reduced image, and then the connection element may be directly obtained based on the result. The results of the two methods may be slightly different, but both methods perform well in a video communication environment where faces appear large in size throughout the image. However, the latter method is advantageous in terms of processing speed.

2. Head region extraction method

The human head region is extracted using the extracted face region. Extraction of the head region uses three types of information such as color, edge, and shape. The method of using each information and the reason thereof are as follows.

2.1. Color information

Human hair color varies not only with race but also with individual characteristics and preferences. Therefore, it is not possible to extract a hair region by designating a specific color as a hair color. In addition, since the shape of the hair varies, it is not possible to forcibly separate a specific area based on the face area and call it a head area. In the present invention, the average color is taken from some areas (called a hair reference region) having a high probability of being a hair area based on the face area, designated as a hair reference color, and then the areas having the same color are obtained to obtain hair. Use the method of designating a region candidate. This can solve problems with various hair colors.

Given the face area, the head reference area is defined as shown in FIG. In Fig. 7 (a), an average collar was taken with both parts of the upper end of the face area as the head reference area. By taking both of the head reference regions in this manner, it is possible to reduce the problem of finding the wrong head reference collar due to noise. If the top of the face is attached to the end of the screen, most of the head reference area is located outside the screen, so that the head reference collar cannot be obtained as shown in FIG. In this case, as shown in Fig. 7B, the head reference area is obtained at both ends of the face area. Whether the top of the face is attached to the end of the screen can be easily determined through the coordinates of the face area.

When using the average color information of the taken head reference region, the performance depends on the given color space. In the present invention, as described above, the HMMD color space capable of expressing Hue, Sum, and Diff is used.

When representing pixel values of a given image in the HMMD color space, a wide range of predefined hair colors is defined as hair color = (sum <150.AND.diff <30) .OR. (Sum <230 && diff <15) It was. In the HMMD color space, sum represents brightness and diff represents color purity. When purity is low, ie gray, it's more likely that you're a head, even though the brightness is quite bright. This means that the head can shine brightly by lighting. Therefore, when the diff representing purity is 15 or less, the hair color is designated as the sum representing brightness is 230 or less. On the other hand, if the diff is higher than 15 but lower than 30, the hair color is designated only when the sum is 150 or less. This is because, if the purity is somewhat high, the color is more strongly felt than the gray color, so it is not likely to be the hair. As described in the definition of hair color, the broad hair color is not limited by the hue representing the color. This is because if it is not a fresh color, it is regarded as a candidate for hair color regardless of the color value. In addition, since the hue that takes the longest time to convert the input image represented by RGB to HMMD is not required, processing time is also increased.

Most of the methods described above can produce good results. However, if the face region is not accurate or an exceptional case such as baldness occurs, the head reference region may include an area other than the actual head. There may be. To solve this problem, hair color range was used to have higher accuracy than usual. In other words, a wide range of hair colors is specified in advance so that the pixels included in the head reference area participate in the average color only when they belong to the hair color range. This method has a problem that cannot be solved when it has an exceptional hair color, but is widely applicable to most problems due to its wide range, and may be selectively applied because the hair color range may not be used in exceptional cases. In other words, it has a higher accuracy than the general case, and the selective use is possible to solve even if the exception is less accurate.

2.2. Edge information

In a mobile communication environment, images are often acquired outdoors. In the case of an outdoor background, if a background such as a forest or a tree is located behind the head, it is often not distinguished by the same hair color. In this case, edge information is used to distinguish the head region from the background region. Pixels whose edges are detected in the obtained head candidate region are excluded. In general, no edge is generated in the head region, and an edge is detected in the boundary or background between the head region and the background region. Therefore, when the edge detected pixels are excluded, part of the background mistaken for the hair color is excluded, and the boundary between the head area and the background area is mostly excluded.

Next, a connected component that connects the pixels connected to the remaining areas into one area is obtained. In this case, since the boundary between the head region and the background region is erased by the edge, the head region and the background region are labeled with different regions. Therefore, if only the connection element attached to the extracted face is determined as the head region, most of the background region is erased. An example of this process is shown in FIG.

In FIG. 8, connection elements of gray 81 and darker color 82 are extracted around the face. In FIG. 8, the gray area 81 is the area mistaken for the head area due to the background or similar hair color, and the darker color area 82 is the actual head area. Due to the removal of the edge pixels, the border area of most of the head area and the background area is separated as shown in FIG. There may be a small number of exceptionally stuck pixels, such as the area 83 shown in the upper left of the head, but this noise may be erased by morphology techniques such as opening.

Morphology is a common way that image processing is heavily used and will be briefly described later.

In FIG. 8, when only the connection element adjacent to the face region is deleted, only the actual head region is extracted, and the wrongly extracted background is excluded.

There are many filters used to find the edges, but the 2 * 2 Roberts filter is used in the present invention. The Roberts filter is described in Equation 1 below, and it is determined that there is an edge only when the filter is applied to the image changed to gray and the value is larger than the threshold value of 25.

2.3. Shape information

The above-described method for extracting the head region is all made inside the previously defined frame around the face region. An example of a frame is shown in FIG. When the face region is extracted, the shape frame is reconstructed at a corresponding size and aspect ratio, and the head region extraction process is performed in the reconstructed shape frame. This has the effect of preventing the worst case by forcibly limiting the area away from the head even if it is mistakenly mistaken as the background area head area due to a processing error. In addition, since processing is performed only in a limited area, unnecessary processing time can be reduced. The frame is defined to be much larger than the general head area, preventing the actual head area from being excluded by the frame.

So far, each method of extracting the head region using color, edge, and shape information has been described.

10 shows the overall process of extracting the head region by combining the processes described so far. The image used as the input is a 44 * 36 (CFI / 8 = 44 * 36) image because the image has already been resized in the entire process described above, and the face region has already been extracted.

First, the head candidate region is defined using the frame using the face region extracted previously (S31). From the next step (S32) all the process is made of the candidate area limited by the frame. Among the following target pixels, a pixel corresponding to a predefined hair color range is obtained and called a set H (S32). Next, the edge-generated pixel is obtained using the Roberts filter, and this is called a set E (S33). Using these two sets (set H, set E), a set A (= H ∩ E ^c ) of pixels having a hair color without edges is obtained (S34). Next, a head reference area is obtained using the face area obtained above (S35), and a head reference color is obtained by averaging pixel colors corresponding to the hair color in the head reference area (S36). In addition, only a color similar to the head reference color is left in the set A, which is called a set hair (S37). Next, after obtaining the connection element from the set hair, the process is completed by determining only the connection element adjacent to the face area as the head area (S38, S39). In the above-described process, the order of each step may be changed for convenience.

3. Body Area Extraction Method

After the facial and head regions are extracted by the method described so far, the body region is extracted based on the extracted face region. Unlike other areas, the body area has various types and shapes of clothes worn by a person, making it difficult to extract the body area accurately. In the present invention, since a practical technical proposal for an actual application is an object, if the body region can be automatically extracted from a given problem, the body region is extracted as much as possible, and if not, the extraction is forcibly performed using a predefined body frame. do. Since the extraction of the body region is used for the bit rate adjustment described above, there is no big problem even if the exact region cannot be extracted, and since the body is rarely caught on the screen in a mobile communication environment, it is not a big problem in actual application.

There are two methods of extracting body regions, using color information and shape information.

3.1. Body Area Extraction Method Using Color Information

A reference body region is designated based on the extracted face region. The body reference area designates a place where the user's clothes are likely to be the area of the user's clothes. As shown in FIG. 11, the body reference area takes two left and right areas which are in contact with each other when the ends of the face area are connected to the bottom of the screen. Next, as in the head region extraction, an average color value is taken from the body reference region and designated as the body reference color, and then, an area composed of pixels having a color similar to the body reference color is designated as the body region.

3.2. Body Region Extraction Method Using Shape Information

Once the face area is determined, an approximate area corresponding to the human body can be predicted. It is the frame that designates the candidate area wider than this approximate area. The body frame is designated as shown in FIG. In other words, given a face area, the size of the frame is adjusted according to the size of the face and the frame is positioned at the bottom of the face. In this frame, the above-described color-based processing is performed using the limited body area as the candidate area of the body. Because color-based processing is performed on the area limited to the body frame, even if the background includes a color similar to that of the user's clothes, the area may be excluded by this frame.

In addition to the user's clothes, the body region includes the user's body such as the neck and arms but where the actual skin is exposed. Therefore, when the body region is extracted only by the body reference collar, the neck or the arm may be excluded. Therefore, not only areas with a color similar to the body reference collar within the body frame but also fleshed areas should be included in the body area. That is, the body region is extracted by designating a region having a color or flesh color similar to the body reference collar within the body frame as the body region.

Since the body region can take various clothes, the user may not be able to extract the actual body region if the body region is taken only with a collar similar to the body reference collar. For example, if a user wears a solid color of clothes, the body area may be appropriately taken. However, if the hair is stretched to the body area or the clothes of various colors are worn, an appropriate area may not be taken. Therefore, if the body reference area includes various colors other than a constant color, it is considered that the body area cannot be taken by the above-described method, and the area defined by the body frame is forcibly separated and designated as the body area.

Figure 13 shows the body region extraction process described so far. First, the body reference region described in FIG. 11 is designated using the extracted face region (S41), and a body reference color is obtained by taking an average color value in the designated region (S42). Next, the body frame described in FIG. 12 is designated using the extracted face region (S43), and a pixel similar to the body reference color is taken inside the designated body frame region, and this is called a set body (S44). Next, a pixel value having a flesh color is taken inside the body frame region, and this is called a set skin (S45). Next, the body region extraction process is completed by designating a region of the sum of the two sets of bodies and skins as the body region (S46).

4. Post processing

After the face region extraction, the head region extraction, and the body region extraction described above, the entire body region is extracted through post-processing. Post-processing consists of trimming the noisy edge area and filling the hole to remove the hole in the extracted body area.

Since the extracted human region is composed of pixels, the edge of the extracted region may not be smooth, and there may be noise such as a small branch protruding or entering. Simple morphology is performed to remove this noise. First, dilation using an element of size A is performed. Dilation using an element of size A fills and smooths the grooves smaller than A size. Next, an erosion using an element of size B is performed. This will remove any overhanging twigs smaller than B. In the present invention, size A is set slightly larger than size B. Since dilation is noise cancellation using region expansion, and erosion is noise removal using region reduction, two morphology processes are used to define a region larger than the originally extracted region. This is to set the area larger than the original extracted area because the case where the important area is mistaken and recognized as the background in the application of bit rate adjustment is more important than the case where the background is mistaken as the important area. After the two morphology processes, the holes in the extracted area are removed.

[Improvement of Processing Speed]

The human region extraction method described so far is reduced to 44 * 36 size when an image is input, and then processed, thereby enabling fast time processing. In general, video communication data has a screen size of CIF (352 * 288) or QCIF (176 * 144). In the present invention, since all are reduced to 44 * 36 size, the CIF is reduced to 1/8 of the width and width of the QCIF. In this case, it is reduced to 1/4. Performance in terms of processing time and algorithm accuracy generally has a tradeoff. Therefore, it is difficult to satisfy both aspects, so it is important to understand the characteristics of a given problem and to consider which one is important. In the present invention, while satisfying both aspects to some extent, the degree of performance degradation and speed improvement by each process caused by the reduction of the area is measured to achieve an overall harmony.

In the present invention, the basic unit algorithms used in the face region extraction process, the head region extraction process, and the body region extraction process, which are the main processes of the present invention, are mainly processed by a color-based algorithm such as a morphology application part, a skin-color extraction area, and an edge-based algorithm Asking for time. In particular, if you compare the information that requires a lot of processing time, it is morphology algorithm> edge based algorithm> color based algorithm.

In order to reduce their processing time, the following three steps can be considered.

The first is to use the original image as it is. This method is not preferable because it takes too much processing time to process the pixel in the original image size.

The second way is to use a grid. In other words, after dividing a given image into n * m grids to obtain edges or flesh colors for each grid, each grid is an average view of color and edges. In this case, basic color pixels or edge pixels are extracted from the original image. However, the process of applying and determining the obtained color pixels and edge pixels is performed in grid units, and thus processing time is reduced and processed. That is, the most time-consuming population algorithms use the grid, which can save a lot of time. However, this method requires the original processing time in color or edge-based algorithms.

The third is to zoom out and zoom in again. That is, a case in which a given image is reduced to n * m and then processed using the reduced image. In this case, all the algorithms use the reduced image, which enables fast processing time. However, the smaller the size, the greater the effect of an error of one pixel on the entire image. As a result, noise is sensitive. Therefore, it is a matter of how much the image can be reduced, and the degree of influence due to image reduction may also be different for each unit algorithm.

In terms of the role of each of these three algorithms, the degree of performance degradation and the speed improvement when the image is reduced and processed instead of the grid method described above is considered. The improvement is 150%, and the color-based algorithm shows 1% performance degradation and 50% speed improvement, and the edge-based algorithm shows 2% performance degradation and 100% speed improvement.

As you can see, the speed improvement of each of the three algorithms is very large, but the performance degradation is very slight. Relatively small edge-based algorithms reduce the performance due to noise, but the absolute degradation is not so great. In particular, in the aspect of application of bit rate adjustment, the error of 1 to 2% is not an important point compared to the processing time, so the present invention used a method of reducing and processing an image.

The size of the reduction was determined by experimenting that the size of 44 * 36 is four times the minimum size without any problem in processing video communication data. Making sure that the size is a multiple of 4 is for optimization in actual processing using registers and the like.

Examples of the results extracted using the human region extraction method described so far are shown in Figs. 14A to 14E. In each example of Fig. 14, the left image is the original image and the squares marked on the original image represent the MBR of the face region extracted automatically. In the example of FIG. 14, the right image is the extracted human body image, and the extracted region is expressed in units of blocks because the processing is reduced to 44 * 36 and magnified again.

Figure 17 (a) is an example with a background similar to the color of the face, (b) is an example with a background similar to the color of the hair, (c) is an example in which the arm is included in addition to the clothes, (d) This is an example of processing a low quality image, and (e) shows an example in which the head is included in the body reference region and is incorrectly separated.

The present invention provides a method for automatically separating a user from a background in video communications in a mobile camera environment. The present invention provides an object-based video communication data application by providing a method for separating a user area at a processing time fast enough to enable real-time processing.

In particular, the present invention can be applied as a core technology that can be usefully used for mobile communication, such as bit rate adjustment or automatic video mail editing, which is adaptively applied to the network load, and solves both problems in terms of time and performance. The practical application was made possible.

According to the present invention, an accurate face area extraction, a relatively accurate head area extraction, and an approximate body area extraction method can be used to properly separate areas according to the application type. In particular, this approach can be suitably used for video communication data with a large face area.

In the present invention, since the unit of object separation is made of 8 * 8 block units in the case of CIF image and 4 * 4 block units in the case of QCIF, the bit rate control or object based on macroblock unit of 16 * 16 size is used. It can be used appropriately for coding.

In addition, the present invention solves the problem of skin color diversity that is variously extracted for each image by extracting a face region based on color, but separating the skin candidate region from a given image exactly with the real human region and the background having a color similar to the flesh color. At the same time, the skin color area in an image is separated according to the object. In addition, in order to accurately extract the face area in video communication, if a plurality of skin areas appear, the areas are first separated based on the skin color that exists at the position where the face is likely to appear, so that the face smaller than the background part is accurately extracted based on the color. can do.

Claims (29)

Extracting a face region from an image, extracting a head region based on the extracted face region, extracting a body region based on the extracted face region, the extracted face region, a head region, a body region Designating a sum as a human area; Human region extraction method comprising a. The method of claim 1, further comprising the step of reducing the original image before all the steps, and further comprising the step of reconstructing the original image size after all the steps of using the reduced image. Way. The method of claim 1, wherein the facial region extraction comprises: obtaining a pixel corresponding to a skin color range in the image, analyzing the distribution in the color space of the skin color pixels, grouping the skin color colors, and configured for each skin color color group Determining whether the image is a face area, expressing the identified face area with a predefined face model; Human area extraction method characterized in that made by. 2. The method of claim 1, further comprising a post-processing step comprising morphology processing and hall clearing processing. Extracting a face region from the image, setting a head reference region based on the extracted face region, obtaining a head reference color from the set head reference region, and a region having a color similar to the obtained head reference color. Obtaining, excluding an area where an edge is generated from an area having a color similar to the obtained head reference color, and designating an area adjacent to the face as a head area among the remaining areas from which the edge generation area is excluded; Hair region extraction method comprising a. The method of claim 5, wherein the head reference color is obtained by using an average value of a color for only pixels corresponding to a predefined hair color range among the pixels included in the head reference area. . 6. The method of claim 5, wherein the head reference region is designated as an upper left and right partial regions of the face region or an upper partial region from the left and right ends of the facial region based on the extracted face region. The method of claim 5, wherein the designating of the head region comprises determining a connected component of the remaining pixels and designating only the elements adjacent to the face as the head region. 9. The method for extracting a head region according to claim 8, wherein prior to designating only the element adjacent to the face as the head region, morphology processing is first performed to solve a problem in which a region other than the head is connected to the head due to noise. 6. The method of claim 5, further comprising specifying a head frame based on the extracted face region, wherein all the processing is limited to pixels included in the head frame. The method of claim 5, wherein the extraction of the face region comprises obtaining an MBR including a face and using a face model having a size and a ratio suitable for the MBR. Extracting a face region from an image, specifying a body frame based on the extracted face region, setting a body reference region based on the extracted face region, and obtaining a body reference color from the body reference region Obtaining an area having a color similar to the body reference color from the inside of the designated body frame, and obtaining an area having flesh color from the inside of the body frame; Body region extraction method comprising a. The apparatus of claim 12, wherein the body frame is a shape model of a body defining a body area larger than a general case when a face area is given, and when the face area is extracted, the body frame is extended or proportional to its size and position. Body area extraction method characterized in that the reduced position after positioning. The apparatus of claim 12, wherein the setting of the body reference area comprises: a predetermined area that is proportional to the size of the face area from the left and right end coordinates of the face area based on the horizontal axis coordinates, and the face area based on the vertical axis coordinates; Body area extraction method characterized in that it is set to a predetermined area from the coordinates below a certain portion from the bottom coordinates to the bottom of the screen. The method of claim 12, wherein the body region is forcibly separated using the body frame when the color distribution is not constant in the body reference region. Comprising a color histogram (hist) for only the pixels corresponding to the skin color region in the image, the center color histogram (center_hist) targeting the pixels corresponding to the skin color region only for a partial region when the color histogram is configured Configuring a color scheme using the color histogram (hist, center_hist), and designating a face region by separating an image area for each flesh group; Facial region extraction method comprising a. 17. The method of claim 16, wherein the partial region is defined as a rectangle smaller than the original image size centering on the center of the image, and the width and height of the original image are defined for the partial region, respectively. And a horizontal / vertical length of each rectangle satisfies width * a and height * b (a <b <1), respectively. The method of claim 16, wherein the partial region is defined as a face region in a previous frame. 17. The method of claim 16, wherein the flesh color grouping comprises: determining a bin bin _max having a maximum value among bin values of a color histogram, and similarity between the representative color value of the bin _{max and} the representative color value of each other bin; Measuring bins, grouping bins with the measured similarity below a predetermined threshold and bin _max into a group, and finally determining the bins as a group when the sum of all bin values included in the group is greater than or equal to a predetermined threshold. And after configuring the one color group, repeating the grouping with respect to the remaining bins except for the bins included in the color group; Facial region extraction method characterized in that consisting of. 17. The method of claim 16, wherein the flesh color grouping defines an empty bin _max having a maximum value among bins of a center color histogram (center_hist) when constituting a first color group, and a color when constituting a color group after a second. Determining the bin bin _max having the maximum value among the bin values in the histogram, and then measuring the similarity between the representative color value of bin _{max and} the representative color value of each bin in the color histogram. Grouping bins having a similarity below a predetermined threshold and bin _max into a group, and finally determining the bins as a group if the sum of all bin values included in the group is above a predetermined threshold; After the group is formed, the second and subsequent color groups are selected for the remaining bins except those included in the color group. Repeating the step of sex; Facial region extraction method characterized in that consisting of. 17. The method of claim 16, wherein if the facial region is included in the region of the grouped flesh color group, the region is composed of pixels corresponding to the first flesh color group. A face region extraction method comprising performing a face region extraction process for a region. 17. The method of claim 16, wherein the step of separating the regions of the image for each skin color group comprises: blocking an image in which each skin color group is specified into N * M-sized blocks (grids); Setting a block in which the frequency of the corresponding pixel is greater than or equal to a predetermined threshold value to a flesh color block, obtaining a connected component connecting neighboring flesh color blocks, and designating the largest connected element as a face region among them; Facial region extraction method characterized in that consisting of. 17. The method of claim 16, further comprising: reducing the original image before the face region extraction process, and restoring the original image size after performing the face region extraction using the reduced image; Facial region extraction method further comprises. Reducing the image to 44 * 36, analyzing the distribution in the color space of the pixel corresponding to the flesh color range in the reduced image, and performing face region extraction through flesh color color grouping, based on the extracted face region Specifying a head reference region and extracting a head region by checking a head reference color within the head reference region, specifying a body reference region based on the extracted face region, and checking a body reference color within the body reference region Extracting a body region, designating a human region by adding the extracted face region, the head region, and the body region and restoring it to an original image size; Human region extraction method comprising a. 25. The method according to claim 3 or 24, wherein the flesh color range is selected from the following conditions in an RGB color space; if (r> b && r> g) ｛ if (b> g) if (r + g> 100 && r + g <480 && g-g> 5 && r-g <100) skin color; else Not flesh; else if (r + b> 100 && r + b <480 && r-b> 5 & r-b <100) skin color; else Not flesh; ｝ else Not flesh; ; According to claim 1, wherein the human area extraction method for defining the flesh color range. 25. The method of claim 3 or 24, wherein the fleshing range is determined by the following conditions in the HMMD color space; if (((h> 300 && h <360) ∥ (h <60 && h> 0)) && s> 100 &amp; & s <480 && d> 5 && d <100) skin color; else Not flesh; ; According to claim 1, wherein the human area extraction method for defining the flesh color range. 25. The method according to claim 3 or 24, wherein the color space for extracting the skin color region is an HMMD color space, and the skin color range sections in the Hue, Diff, and Sum axes are respectively M / N / K (M> N> K). A method for extracting human domains, comprising quantizing by dividing into equal parts. 25. The method of claim 24, wherein the range of the hair color defined as the head reference color in the head reference region is a case where the Diff is less than 30 and the Sum is less than 150, or the Diff is less than 15 and the Sum is less than 230 in the HMMD color space. Human region extraction method characterized in that. 25. The method of claim 24, wherein a region having a color similar to the body reference color in the body reference region is obtained by calculating similarity in the HMMD color space.