KR20160130911A - Method and apparatus for masking face by using multi-level face features - Google Patents

Abstract

According to the present invention, a method for masking a face by using multi-level face features may include: a step of extracting an interest region in a video frame; a step of extracting and analyzing a spatial feature of the face from the extracted interest region so as to detect a first face region; a step of detecting a second face region by using temporal information between frames from the extracted interest region; a step of generating an integrated facial region by integrating the detected first face region and the second face region; and a step of applying the masking to the generated integrated face region. Accordingly, the present invention can maximize the face features for masking the face in the video image photographed in an unrestricted environment.

Description

TECHNICAL FIELD [0001] The present invention relates to a facial masking method and a facial masking method using multi-level facial features,

The present invention relates to a face masking technique, and more particularly, to a face masking technique for detecting a face region from input video in order to protect the privacy of characters appearing in a video, and a multilevel face feature suitable for masking the detected face region The present invention relates to a facial masking method and apparatus using the same.

In recent years, surveillance cameras (eg CCTV) have been installed in various places throughout the country for the purpose of surveillance and crime tracking. Such trends have the advantage of providing efficiency in crime prevention and investigation, The privacy of an individual who is unintentionally exposed by the public is exposed to the public.

In particular, since face exposure can be a clue to know the identity, a technique for masking the face of the individual so as to protect privacy is required.

Here, in order to perform facial masking, a detection process of accurately locating a face region is required. Detection of a face region in a video acquired in an unconstrained environment such as CCTV is performed using the following conducive elements, There may be a variety of challenging factors such as varying angle changes, face lighting changes, blurred faces, occlusion of faces, and the like.

As is well known, face masking systems for privacy protection in large-scale video are very inefficient in terms of time and cost when manually specifying face regions of people.

Therefore, a technique of automatically detecting the face region of people is required. Typical examples include a Viola-Jones face detection method (Prior Art 1) and a Saliency map-based face detection method 2).

Prior art 1

Conventional art 1 is the first real-time face area detector having stable detection performance, and it is widely used as a base of many face area detectors integrated into penCV (Open Computer Vision Library) which is a representative image processing recognition library.

This prior art 1 discloses a technique for generating a hair-like characteristic feature based on structural information of a face in a strong classifier of a cascade structure composed of a combination of weak classifiers through a boosting technique Lt; / RTI >

However, the conventional art 1 shows a very stable detection performance when the characteristics of the test image and the image used in the training are similar to each other, but otherwise the detection rate sharply drops.

Prior Art 2

The conventional art 2 is a method proposed in order to solve the problem of the lowering of the detection rate of the conventional art 2 described above in the unconstrained environment. In addition to the conventional art 1, a map corresponding to the skin color and an object area, Celanese map was used.

However, the Celianity map has a problem that the efficiency is drastically reduced for an input image including a complex texture background which is not monotonous.

Korean Patent Publication No. 2011-0070735 (Publication date: June 24, 2011)

[1] Viola, Paul, and Michael J. Jones. &Quot; Robust real-time face detection. "International journal of computer vision 57.2 (2004): 137-154. [2] El-Barkouky, Ahmed, et al. "Face detection at a distance using saliency maps." Computer Vision and Pattern Recognition Workshops (CVPRW), 2012 IEEE Computer Society Conference on. IEEE, 2012.

The present invention detects a face area using a spatial level feature and applies a masking to a face area in a manner of detecting a face area using a temporal level feature, We propose a face masking method using multi - level facial features that can perform facial masking in video images.

The problems to be solved by the present invention are not limited to those mentioned above, and another problem to be solved by the present invention can be clearly understood by those skilled in the art from the following description will be.

According to one aspect of the present invention, there is provided a method for extracting a region of interest, the method including extracting a region of interest in a video frame, extracting and analyzing spatial features of the extracted region of interest, Detecting a second face region using temporal information between frames; generating an integrated face region by integrating the detected first face region and the detected second face region; And a face masking method using a multilevel face feature.

The area of interest of the present invention can be extracted in a rectangular shape.

The region of interest of the present invention can be extracted through a skin color pixel detection technique.

The step of detecting the first face region of the present invention includes the steps of extracting each analysis score value for the low level face feature, the intermediate level face feature, and the high level face feature from the ROI, Determining whether the face region is fused to determine whether the face region is fused; and detecting a coordinate value of the region as the first face region when the face region is identified.

The low-level face feature of the present invention can use skin color information.

The mid-level facial features of the present invention may utilize local facial features of both eyes and mouth regions.

The high level face feature of the present invention may utilize the global face feature.

The step of detecting the second face region may include the steps of obtaining a previous coordinate value in a previous frame corresponding to the coordinate value based on the forward position tracking, Obtaining a subsequent coordinate value within the frame, and detecting the obtained previous coordinate value and the subsequent coordinate value as the second face region.

According to another aspect of the present invention, there is provided an image processing apparatus for extracting and analyzing a spatial feature of a face from an extracted interest region, A second face region detection block for detecting a second face region using temporal information between frames extracted from the extracted region of interest and a second face region detection block for detecting the integrated face and the second face region by integrating the detected first face region and the second face region, The present invention provides a face masking apparatus using a multilevel face feature including a face region integration unit for generating a face region masking unit for applying a masking to the generated integrated face region.

The first face region detection block of the present invention includes a first face feature analysis unit for extracting a first analysis score value for a low level face feature from the region of interest, A third face feature analyzing unit for extracting a third analysis score value for a high-level face feature from the region of interest, a second face feature analyzing unit for extracting an analysis score from the extracted first through third analysis results, And a spatial face detection unit for detecting a coordinate value of the area as the first face area when the face area is discriminated as the face area.

The first facial feature analysis unit of the present invention can use the skin color information as a low-level facial feature.

The second facial feature analysis unit of the present invention can use the local facial features of both eyes and mouth regions as the intermediate level facial feature.

The third face characteristic analyzing unit of the present invention can use the global face feature as the high level face feature.

The second face area detection block may include a forward coordinate obtaining unit for obtaining a previous coordinate value in a previous frame corresponding to the coordinate value based on the forward position tracking, And a temporal face detecting unit for detecting the obtained previous coordinate value and subsequent coordinate values as the second face area.

The face area integrating unit of the present invention can generate the integrated face area by integrating the coordinate value, the previous coordinate value, and the subsequent coordinate values.

In the present invention, face masking is performed on a video image photographed in an unconstrained environment by detecting the face region using the spatial level feature and applying the masking to the face region in a manner of detecting the face region using the temporal level feature It is possible to realize maximization of the face characteristic.

1 is a block diagram of a face masking apparatus using a multilevel face feature according to the present invention.
2 is a detailed configuration diagram of the first face area detection block shown in FIG.
3 is a detailed configuration diagram of the second face area detection block shown in FIG.
4 is a flowchart illustrating a main process of performing face masking using a multi-level face feature according to the present invention.
FIG. 5 is a pictorial example of a concept for extracting a region of interest from a video frame according to the present invention.
FIG. 6 is a photographic example showing an example of performing a window scan within each ROI to determine face area coordinates according to the present invention.
FIG. 7A is a photograph illustrating a face region detection based on a low level face feature, FIG. 7B is a face region detection based on an intermediate level face feature, and FIG. 7C is a photograph example for explaining a face region detection based on a high level face feature, respectively.
FIG. 8A shows an example of adding the coordinates of the face region detected based on the forward position tracking, FIG. 8C shows an example of detecting the face detected based on the reverse position tracking, Fig. 8 is a photograph showing an example in which coordinates of a region are added.
FIG. 9 is a photograph illustrating a result of a video frame in which masking through blurring is applied to face regions detected from a region of interest according to the present invention. FIG.

First, the advantages and features of the present invention, and how to accomplish them, will be clarified with reference to the embodiments to be described in detail with reference to the accompanying drawings. While the present invention has been described with reference to exemplary embodiments, it is to be understood that the invention is not limited to the disclosed exemplary embodiments, but, on the contrary, It will be understood by those of ordinary skill in the art that various changes in form and details may be made therein without departing from the spirit and scope of the invention as defined by the appended claims.

In the following description of the present invention, detailed description of known functions and configurations incorporated herein will be omitted when it may make the subject matter of the present invention rather unclear. It is to be understood that the following terms are defined in consideration of the functions of the present invention, and may be changed according to intentions or customs of a user, an operator, and the like. Therefore, the definition should be based on the technical idea described throughout this specification.

Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings.

FIG. 1 is a block diagram of a face masking apparatus using a multi-level face feature according to the present invention. The face masking apparatus includes a ROI extracting unit 102, a first face region detecting block 104, a second face region detecting block 106, A frame memory 108, a face area integration unit 110, a face area masking unit 112, a frame output unit 114, and the like.

Referring to FIG. 1, the ROI extracting unit 102 may provide a function of extracting (identifying) a ROI from a video frame (video image) input from the outside. That is, rectangular regions having a high probability of existence of a face in the input video frame are extracted as an interest region so as to reduce the calculation time of the face masking and reduce false positives in detecting the face region, To the first face area detection block 104, and the like.

Here, various methods including extraction of a skin color pixel (skin color pixel) and the like can be applied to extraction (determination) of a region of interest.

FIG. 5 is a photograph illustrating a concept of extracting a region of interest from a video frame according to the present invention. In FIG. 5, a left picture shows an input video frame, and a right picture shows a region of interest As a result of extraction, a region indicated by white corresponds to a skin color pixel, and a green box indicates regions of interest having a rectangular shape set based on the skin color.

Next, the first face region detection block 104 extracts and analyzes the spatial features of the face from the extracted region of interest transmitted from the region of interest extraction unit 102, and detects (discriminates) the first face region, (E.g., low level, middle level, high level, etc.) within the regions of interest of the input video frame, and analyzes and classifies the analysis results into a face Area coordinates may be outputted and transmitted to the second face area detection block 106.

That is, the first face region detection block 104 performs a window scan in each ROI for determination of the face region coordinates, that is, as shown in FIG. 6 as an example, The first face area detection block 104 may include a configuration as shown in FIG. 2, and may be configured to detect the first face area by determining whether the window is a face area or not. have.

FIG. 2 is a detailed configuration diagram of the first face region detection block shown in FIG. 1, where the first face region detection block 104 includes a first face feature analysis unit 202, a second face feature analysis unit 204, A third face feature analysis unit 206, a face area determination unit 208, and a spatial face detection unit 210. [

Referring to FIG. 2, the first facial characteristic analyzing unit 202 may provide a function of extracting a first analysis score value for a low-level facial feature from the region of interest and transmitting it to the facial region discriminating unit 208 For example, skin color information can be used as a low-level face feature, and the skin color information can be converted into an analysis score value by calculating a ratio of skin pixels in the window, for example, as shown in Fig. 7A.

The detection of the face region based on the low-level facial feature is relatively high recall in an image having an angle change due to the rotation of the face or a low resolution image or an image having a motion blur, Can be realized.

Next, the second facial characteristic analyzing unit 204 may provide a function of extracting a second analysis score value of the intermediate level facial feature from the region of interest and transmitting it to the facial region discriminating unit 208, As an intermediate level facial feature, for example, the local facial features of both eyes and mouth regions can be used. As shown in FIG. 7B, for example, eye and mouth area detection are performed, The number of elements can be converted into an analysis score value.

Detection of the face area based on the intermediate level facial feature can be applied to a moderate angle change within 45 degrees left and right. When the occlusion is present in a part of the face, the component detection of the uncovered area It is possible to realize a relatively high accuracy.

The third facial feature analysis unit 206 may provide a function of extracting a third analysis score value of the high-level facial feature from the region of interest and delivering the third analysis score value to the facial region discriminator 208, As a level facial feature, for example, a holistic facial feature can be used and, as shown in FIG. 7C as an example, a frontal facial region detection and a profile facial region detection in a window are respectively performed and considered It is possible to convert the number of windows classified into the face region among the windows close to the window into an analysis score value.

The detection of the face area based on the high level face feature has an advantage that the detection rate is relatively increased when the face region image used for the classifier learning and the face region of the test are similar in angle.

That is, the present invention can discriminate the face region by analyzing three levels (low level, middle level, high level) facial features in each window set in the region of interest.

Next, the face region determination unit 208 fuses the first through third analysis score values extracted through the first through third face region analysis units 202, 204, and 206 to determine whether or not the face region is a face region And transmits the determination result to the spatial face detection unit 210. [0154] FIG.

That is, the three score values based on three levels (low level, middle level, high level) facial feature analysis have different ranges, so normalization (e.g., min-max normalization, etc.) And the three normalized score values calculated in the window are input to the binary classifier as a three-dimensional vector to determine whether the corresponding window is a face region or not.

Here, various kinds of binary classifiers (e.g., a support vector machine (SVM), etc.) can be used, and positive windows corresponding to the face region for classifier training and a negative the three score values calculated in the negative windows are used for the binary classifier learning.

Next, the spatial face detection unit 210 detects the coordinate values of the area as the first face area (the face area detected using the spatial feature level) when the face area determination unit 208 notifies the determination to the face area To the second face area detection block 106 of FIG. 1, and so forth.

Referring again to FIG. 1, the second face area detection block 106 detects the second face area using the temporal information between video frames from the ROI extracted through the ROI extracting unit 102 (in the forward direction forward) and reverse face tracking) to detect (discriminate) the second face region. For this, a configuration as shown in FIG. 3 may be included.

FIG. 3 is a detailed configuration diagram of the second face region detection block shown in FIG. 1, and may include a forward coordinate acquisition unit 302, a reverse coordinate acquisition unit 304, a temporal face detection unit 306, and the like.

Referring to FIG. 3, the forward-direction coordinate acquiring unit 302 receives a previous frame (temporally adjacent to a t-frame) corresponding to a coordinate value of a face region detected (tracked) from an input frame (E.g., the previous t-1 frame) from the frame memory 108 and transfer it to the temporal-face detecting unit 306, for example.

In addition, the reverse-direction-coordinate obtaining unit 304 obtains a reverse frame of the frame corresponding to the coordinate value of the face region detected (tracked) from the input frame (t frame) based on the reverse- 1 frame) from the frame memory 108 and deliver it to the temporal-face detecting unit 306. The time-

The temporal face detecting unit 306 detects a previous coordinate value and a subsequent coordinate value obtained through the forward coordinate obtaining unit 302 and the backward coordinate obtaining unit 304 using the second face region Face region) of the face region to the face region integration unit 110 of FIG. 1, and so on.

Referring again to FIG. 1, the face area integration unit 110 integrates the detected face region (first face region) using the spatial feature level and the detected face region (second face region) using the temporal feature level To generate the integrated face region, and to transmit the generated integrated face region to the face region masking unit 112. [

That is, face area coordinates obtained as a result of face area detection (FD) using the spatial level feature in the current frame at time t, face tracking in the forward direction (FFT) in the current (t) ), And at the time t + 1 frame, the face area coordinates obtained as a result of performing face tracking (BFT) in the reverse direction to the current (t) frame are all incorporated.

At this time, the frame for performing reverse face tracking at the current frame reference (time t) is up to the frame of time T (t-T + 1).

FIG. 8A shows an example of adding the coordinates of the face region detected based on the forward position tracking, FIG. 8C shows an example of detecting the face detected based on the reverse position tracking, Fig. 8 is a photograph showing an example in which coordinates of a region are added.

Referring to FIG. 8A, it can be seen that the coordinates of the detected face region are displayed (green box) using the spatial level feature output from the first face region detection block 104. Referring to FIG. 8B, It can be seen that the coordinates of the face area acquired by the forward direction coordinate acquiring unit 302 in the area detecting block 106 are additionally displayed (red box).

Through this, it can be seen that face detection based on the forward tracking is useful for detecting face regions detected in the video frame in the early part but not in the video frame in the latter part.

Referring to FIG. 8C, it can be seen that the coordinates of the face area obtained by the reverse direction coordinate obtaining unit 304 in the second face area detecting block 106 are additionally displayed (blue box) It can be seen that the region detection is useful for detecting face regions that were detected in the video frame of the latter half but not yet detected in the video frame of the latter half.

That is, in the present invention, it is difficult to detect the face area even when the spatial level features are insufficient (for example, when the face rotation angle is too large and the eyes and mouth are not visible or the occlusion is very severe) The recall can be increased.

Next, the face area masking unit 112 applies masking (e.g., masking through blurring) to the integrated face area (coordinate values of the integrated face area) transmitted from the face area integrating unit 110, That is, it is possible to provide a function of applying masking so that visual identification is impossible.

Then, the frame output unit 114 can provide a function of masking the face regions detected from the input video frame, for example, outputting a video frame as shown in Fig. 9 to a display side (not shown) have.

Next, a series of processes for applying the face masking based on the multi-level face feature through the face masking apparatus according to the present embodiment having the above-described configuration will be described in detail.

4 is a flowchart illustrating a main process of performing face masking using a multi-level face feature according to the present invention.

4, when a video frame is input from the outside, the ROI extracting unit 102 extracts (identifies) a ROI (for example, ROI) from an input video frame (video image) ). Here, extraction (discrimination) of a region of interest, for example, skin color pixel detection and the like can be applied.

In response, the first facial feature analysis unit 202 in the first facial region detection block 104 extracts a first analysis score value for a low-level facial feature from the region of interest, and the second facial feature analysis unit 204 Extracts a second analysis score value for the intermediate level face feature from the ROI, and the third face feature analysis unit 206 extracts a third analysis score value for the high level face feature from the ROI (step 404 ).

Here, in the low-level face feature, skin color information can be used, for example, and converted into an analysis score value by a method of calculating the proportion of skin pixels in the window.

Further, in the intermediate level face feature, for example, local face features of both eyes and mouth regions can be used, eye and mouth region detection can be performed, and the number of face components detected by this process can be used as an analysis score value Can be converted.

In the high-level facial feature, for example, a holistic facial feature can be used, and a frontal facial region detection and a facial facial region detection are respectively performed in a window, and a facial region Can be converted into an analysis score value.

Next, in the face area determination unit 208, the first through third analysis score values extracted through the first through third face area analysis units 202, 204, and 206 are fused to determine whether or not the face region is a face region (Step 406).

If it is determined that the face region is a result of the classification in step 406, the spatial face detection unit 210 detects the coordinate values of the face region as a face region detected using the first face region, that is, the spatial feature level Step 408).

In the forward-direction coordinate acquisition unit 302 in the second face-area detection block 106, the previous frame (t frame) corresponding to the coordinate value of the face region detected (tracked) from the input frame (step 410), and the backward coordinate acquisition unit 304 obtains the previous frame in the input frame (t-1 frame) based on the backward position tracking (T + 1 frame temporally adjacent to the t frame) corresponding to the coordinate value of the face region detected (traced) from the frame memory 108 (step 412).

In response to this, the temporal face detection unit 306 detects each of the obtained previous coordinate values and subsequent coordinate values as a face region detected using the second face region, that is, the temporal feature level (Step 414).

The face region integration unit 110 integrates the detected face region (first face region) using the spatial feature level and the detected face region (second face region) using the temporal feature level, (Step 416).

In the face area masking unit 112, masking is applied to the integrated face area (coordinate values of the integrated face area) transmitted from the face area integration unit 110 so that the face in the video image can not be visually identified, Masking via blurring is applied (step 418).

Thereafter, the frame output unit 114 outputs a video frame having masking applied to the face regions detected from the input video frame, for example, to the display side (step 420).

It will be understood by those skilled in the art that various changes in form and details may be made therein without departing from the spirit and scope of the present invention as defined by the following claims. It is easy to see that this is possible. That is, the embodiments disclosed in the present invention are not intended to limit the scope of the present invention but to limit the scope of the present invention.

Therefore, the scope of protection of the present invention should be construed in accordance with the following claims, and all technical ideas within the scope of equivalents should be interpreted as being included in the scope of the present invention.

Claims (13)

Extracting a region of interest in a video frame;
Extracting and analyzing a spatial feature of the face from the extracted ROI to detect a first face region;
Detecting a second face region using temporal information between frames from the extracted region of interest;
Generating an integrated face region by integrating the detected first face region and the detected second face region;
A process of applying masking to the generated integrated face area
A method for facial masking using multi-level facial features,
The method according to claim 1,
Wherein the region of interest comprises:
Extracted by skin color pixel detection technique
Facial masking method using multi - level facial features.
The method according to claim 1,
The step of detecting the first face region comprises:
Extracting analytic score values for low-level facial features, intermediate-level facial features, and high-level facial features from the region of interest;
Determining whether each of the extracted analysis score values is a face region by fusing the extracted analysis score values;
Detecting a coordinate value of the area as the first face area when the face area is identified;
A method for facial masking using multi-level facial features,
The method of claim 3,
The low-
Using skin color information
Facial masking method using multi - level facial features.
The method of claim 3,
The intermediate level facial feature may comprise:
Using the local facial features of both eyes and mouth regions
Facial masking method using multi - level facial features.
The method of claim 3,
The high-
Using full face features
Facial masking method using multi - level facial features.
The method of claim 3,
Wherein the step of detecting the second face region comprises:
Obtaining a previous coordinate value in a previous frame corresponding to the coordinate value based on the forward position tracking;
Acquiring an intra-frame intra-frame coordinate value corresponding to the coordinate value based on the reverse position tracking;
Detecting the obtained previous coordinate value and subsequent coordinate values as the second face region
A method for facial masking using multi-level facial features,
An interest region extracting unit for extracting a region of interest from an input video frame,
A first face region detection block for extracting and analyzing spatial features of the face from the extracted region of interest and detecting a first face region,
A second face region detection block for detecting a second face region using temporal information between frames extracted from the extracted region of interest,
A face area integration unit for integrating the first face region and the second face region to generate an integrated face region,
A face region masking unit for applying masking to the generated integrated face region,
Wherein the face masking device is a face masking device.
9. The method of claim 8,
Wherein the first face area detection block comprises:
A first facial feature analysis unit for extracting a first analysis score value for a low-level facial feature from the region of interest;
A second face feature analyzing unit for extracting a second analysis score value for an intermediate level face feature from the ROI,
A third face feature analyzing unit for extracting a third analysis score value for the high-level face feature from the region of interest,
A face area determination unit for determining whether or not the extracted first to third analysis score values are fused to determine whether or not the face region is fused;
A face detection unit detecting a coordinate value of the area as the first face area when the face area is determined as the face area,
Wherein the face masking device is a face masking device.
10. The method of claim 9,
Wherein the first facial feature analyzing unit comprises:
Using skin color information as low-level facial features
Facial masking device using multi - level facial features.
10. The method of claim 9,
Wherein the second face feature analyzing unit comprises:
Using the local facial features of both eyes and mouth regions as the mid-level facial feature
Facial masking device using multi - level facial features.
10. The method of claim 9,
Wherein the third face characteristic analyzing unit comprises:
Using the global face feature as the high-level face feature
Facial masking device using multi - level facial features.
10. The method of claim 9,
Wherein the second face area detection block comprises:
A forward coordinate obtaining unit for obtaining a previous coordinate value in a previous frame corresponding to the coordinate value based on the forward position tracking,
A backward coordinate acquiring unit for acquiring in-frame intra-frame coordinate values corresponding to the coordinate values based on backward position tracking;
And a temporal face detecting unit for detecting the obtained previous coordinate value and subsequent coordinate values as the second face region,
Wherein the face masking device is a face masking device.

Images