JP5029545B2 - Image processing method and apparatus - Google Patents

Description

  The present invention relates to an image processing technique using a computer, and more particularly, to an image processing technique aiming at a so-called “skin-beautifying” effect that corrects wrinkles and spots appearing on a skin portion of a human face image so as to be inconspicuous.

  A photo retouch technique that takes a photographic image as digital data into a computer and performs various processing on the image on the computer is used in various fields. In particular, there is a great demand for improving the skin texture of human face images, and attempts to remove or reduce unnecessary parts such as wrinkles and spots that impair the aesthetics have been widely conducted from still images to movies. Yes. Recently, along with the improvement in computer performance, various retouch processes can be applied to photographic images using a personal computer even at an individual level.

As an image processing method for so-called smoothing processing that effectively reduces and eliminates such unnecessary portions, an ε-filter, an MTM filter, and the like are known.
These filters are a kind of average value filter and originally intended to remove high-frequency noise components with small amplitudes, but when applied to image signals such as face images, boundary information such as edges Since unnecessary portions (small amplitude signals) such as wrinkles can be removed while preserving the image, it is particularly suitable for applications such as ID photos where the finished quality is important.
For example, Patent Document 1 discloses a technique related to “skin vine” using an ε-filter using different thresholds, and Patent Document 2 discloses an MTM filter as a noise removal method for image data from the viewpoint of signal processing. The characteristics of various two-dimensional filters are discussed.
Japanese Patent Laid-Open No. 2001-118084 JP 2006-60841 A

In addition, in the smoothing process of the face image, it is desirable to perform the smoothing process only on the skin part so as not to impair the details of eyes, lips and hair, and the skin area mask indicating the skin part of the face is used. There are many examples where processing is performed only on the part. A technique for recognizing a skin color area from a face image and generating a skin area mask in this manner is disclosed in, for example, Patent Document 3 by the same applicant as the present invention.
JP 2005-276182 A

However, when such a “skin-beautifying” function is to be put into practical use, the following problems arise.
For example, when applied to an ID photo, the size of the ID photo varies greatly depending on the application, and the size of the image data also changes greatly. In addition, since the face size in the frame also changes depending on the shooting distance, the size of the face image on the image data has a large fluctuation range.
In this way, even if a two-dimensional filter having a uniform size is applied to face image data having a wide size range, a desired effect cannot be obtained, and a large variation occurs in the smoothing effect by the filter. Therefore, in order to obtain a stable smoothing effect regardless of the size of the face, it is necessary to change the size of the two-dimensional filter according to the face size of the image data.

  On the other hand, as the face image size increases, the size of the two-dimensional filter also increases. However, the calculation time of each pixel increases with the area ratio of the two-dimensional filter, so that the calculation time of filtering processing significantly increases as the face image increases. A new problem arises. For this, a method of reducing the amount of calculation by thinning out pixels in the filtering process can be considered, but on the other hand, other problems such as image quality degradation and generation of pseudo noise occur.

  Therefore, the present invention can obtain a uniform smoothing effect regardless of the size of the captured face image, and can be processed at high speed with little quality deterioration even for a large-sized face image. An object of the present invention is to provide such an image processing method and apparatus.

The image processing method of the present invention that solves the above-described problems is intended to beautify the skin by correcting unwanted images such as spots, wrinkles, and scratches that appear on the skin of the image signal representing the face image of a person. A person image input stage for inputting image data including a face to a computer, and a skin area extraction stage for identifying a face area from the person image, extracting a skin part from the face area, and generating skin area mask data indicating the skin area And, based on the skin area mask data, performing a filtering process by a two-dimensional matrix of a square array on each pixel in the skin area, and
An image processing method for correcting an unnecessary image portion of a face image, comprising:
The smoothing processing step includes a face actual dimension setting step for designating an actual size of a face, an unnecessary image actual dimension setting step for designating an actual size of the unnecessary image portion, and the unnecessary image actual dimension as the face. Matrix size determination step of integrating the value divided by the actual size and the number of pixels of the face image obtained from the face area, and determining an odd value closest to the integration result as the number of pixels on one side of the two-dimensional matrix It is characterized by including.

  Further, when the number of pixels on one side of the two-dimensional matrix is 5 or more, the calculation target pixels used in the filtering process are selected by thinning out in a stepping stone shape.

  Further, the calculation target pixel used in the filtering process is selected concentrically with respect to the target pixel at the center of the two-dimensional matrix with the same target image as the center.

The second invention of the present invention is
Human image input means for inputting image data including a person's face to a computer, and a skin area for specifying a face area from the person image, extracting a skin from the face area, and generating skin area mask data indicating the skin area Extraction means, smoothing processing means for performing filtering processing by a two-dimensional matrix of a square array on each pixel in the skin region based on the skin region mask data, and an unnecessary image portion to be subjected to the filtering processing on the real object An unnecessary image actual dimension input means for setting a dimension, a value obtained by dividing the unnecessary image actual dimension by a preset average face size dimension, and the number of pixels of the face image obtained from the face area; Matrix size determining means for determining an odd value closest to the integration result as the number of pixels on one side of the two-dimensional matrix;
An image processing apparatus comprising:

 Here, when the number of pixels on one side of the two-dimensional matrix is five or more, the calculation target pixels used in the filtering process are selected by thinning out in a stepping stone shape.

  Further, the calculation target pixel used in the filtering process is selected concentrically with respect to the target pixel at the center of the two-dimensional matrix, centering on the target image.

  According to the image processing method and apparatus of the present invention, by obtaining a two-dimensional matrix having a size corresponding to the size of the face image data to be processed, constant filtering regardless of the size of the face image. The effect can be obtained, and the processing target pixels in the filtering process are arranged concentrically, so that high-speed processing with little image quality deterioration can be realized.

Hereinafter, embodiments of the present invention will be described with reference to the drawings.
FIG. 1 is a functional block diagram of an image processing method of the present invention, and FIG. 2 shows an example of a hardware configuration when this is realized by computer processing.
FIG. 3 is a diagram illustrating a processing flow of image processing according to the present invention, FIG. 4 is a diagram illustrating a process of generating a skin region mask, FIG. 5 is a diagram illustrating a relationship between a face size and a two-dimensional matrix, and FIG. FIG. 7 is a diagram showing the principle, and FIG. 7 is a diagram showing the relationship between a two-dimensional matrix and a thinning pattern.

  The operation unit 21 in FIG. 1 is used to give an instruction regarding image processing of the present invention using the keyboard 6 and the mouse 7 while obtaining visual information from the monitor 4 in FIG. The processing unit 20 indicates that image processing is performed with reference to data and parameters stored in the data storage unit 27. The image input function 22, the skin region extraction function 23, the smoothing processing function 24, and the image output used in the image processing of the present invention. It consists of function 25 and the like. The I / O external connection unit is mainly for input / output of image data. For example, in the case of input, a data storage medium or a digital imaging device is connected, and in the case of output, data storage is performed. It is assumed that a printer or the like is connected in addition to the media.

FIG. 3 shows a detailed flow relating to the smoothing processing of the present invention.
First, the face portion of the person image input to the computer in the image input stage (S11) is specified by face detection processing. There are various methods for face detection, but the “Haar-like feature” is used in the flow shown in FIG.
The object detection method using the Haar-like feature is characterized in that identification is performed at high speed based on a light / dark pattern of an object to be detected, and is used for face detection in still images and moving images. . In particular, high detection accuracy is exhibited for a frontal, hatless person image such as an ID photo. A face detection result using the Haar-like feature is represented as an area including local information of the face such as eyes, nose, and mouth, and information such as the center and size of the face.

Note that the principle of object detection based on the Haar-like feature, an example of application to face detection, and the like are introduced in the following documents (Non-Patent Document 1 and Non-Patent Document 2).
Paul Viola and Michel J. Jones: "Rapid Object Detection Using a Boosted Cascade of Simple Features", Proceedings of the 2001 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, pp.511-518, (2001) Rainer Lienhart and Jochen Maydt: "An Extended Set of Haar-likeFeatures for Rapid Object Detection", Proceedings of the 2002 IEEE International Conference on Image Processing, Vol.1, pp.900-903, (2002)

 In addition, a method for identifying a face from an image and acquiring a face size and position using Haar-like features has already been provided as an image processing library (OpenCV library developed and published by Intel). ), It is easy to derive a face rectangular area representing the face area by using these. A face rectangular area 42 shown in FIG. 4B is a face area obtained by using the Haar-like feature and OpenCV, and the face rectangular area indicates the position (center) of the face in the processed image. It represents the size (width and height of the face).

  When the face portion is specified in this way, it is possible to detect a skin region in which the skin portion of the face is extracted with reference to the color of the part indicating the average skin color of the face. In the present invention, mask data generation is performed using the method of extracting a human skin region and the method of creating mask data disclosed in Patent Document 3 (S13). The mode of the mask data generated by the skin region extraction is shown in FIG.

On the other hand, the size of the face in the width direction (number of pixels) obtained from the face rectangular area based on the Haar-like feature, the actual face dimensions (S14) and unnecessary image actual dimensions (S15) set separately, The matrix size (the number of pixels on one side), which is the number of pixels of the filter, can be determined (S16).
Note that the actual face size need not be measured for each individual, and in the configuration of FIG. 1, for example, a value of an average Japanese face size of 170 mm is stored in the data storage unit 27 in advance as data ( Actual face size 273). The unnecessary image actual size is a size of an unnecessary portion to be corrected such as a stain or a wrinkle, and a numerical value measured in millimeters is set by combining the actual size and unit of the face. 2 may be input from the keyboard 6 in FIG. 2 or selected from several sizes with the mouse 7.

To calculate the filter matrix size,
a = L × M / F
(Where L is the face size on the image data, that is, the number of pixels in the width direction of the face rectangular area,
M is the unnecessary image actual size,
F is the actual size of the face)
Is obtained by setting the odd value closest to a obtained here. If this value is A, then A
A = INT (a / 2 + 1) * 2-1
(However, INT is an operator that represents a rounded-down integer type)
It can be expressed by the following formula.
Thereby, since the matrix size A changes according to the face size of the input image data, a constant filtering effect can be obtained regardless of the size of the face image data and the pixel density.
FIG. 5 (a) shows a state of an unnecessary portion on the face of a person, and FIGS. 5 (b) and 5 (c) show a matrix size on the face image. (B) shows a case where the face image is a small size, and (C) shows a case where the face image is a large size, A and A ′ in the figure show how the matrix size A changes depending on the size of the face image. Represent.

Using the filter of the matrix size A obtained above, the filtering process (S17) is performed with reference to the skin area mask data obtained in the skin area extraction (S13). Here, the filtering process is performed using the MTM filter as an example. Will be explained.
The MTM filter is basically an averaging filter, but the target pixel and the neighboring pixels are compared. If the difference is small, the correlation between the target pixel and the neighboring pixels is strong, and this neighboring pixel is calculated. In contrast, when the difference is large, the filter effect is focused on the small amplitude signal by excluding it from the calculation target. FIG. 6 shows the principle of the MTM filter.

On the other hand, since the amount of computation in such filtering processing depends on the size of the filter, the amount of computation increases with the area ratio as the number of pixels on one side of the two-dimensional matrix increases, resulting in a significant increase in processing time. .
In order to avoid an increase in processing time, it is desirable to reduce the amount of calculation by thinning out the processing target pixels during the filtering process. Therefore, in the present invention, a pixel pattern to be calculated is set in advance on the two-dimensional matrix of the filter, and the calculation amount is reduced, that is, the calculation speed is increased by excluding the set pixels from the calculation target. is doing.
FIG. 7 shows an example of a thinning pattern on a two-dimensional matrix, where (A) shows a vertical and horizontal grid pattern, and (B) shows a staggered pattern. Note that the thinning effect cannot be obtained when the number of pixels on one side of the two-dimensional matrix is 3 or less, and is effective when the number of pixels on one side is 5 or more.

  For example, in the example of FIG. 7A, the size of the two-dimensional matrix is 7 × 7 and the number of neighboring pixels centering on the pixel of interest 61 is 48 pixels, but the calculation target pixel 62 is selected in a stepping stone shape. As a result, the total number of calculation target pixels is reduced to 8 pixels. As a result, the processing time for the repetitive calculation for each pixel can be greatly shortened, and the processing time of the entire smoothing process can be shortened.

However, in a pattern with repetitive regularity such as the stepping stone pattern of (A) and (B) above, this pattern interferes with the pattern included in the original image to be processed, and the filtering result includes interference fringes and the like. Side effects such as the appearance of secondary patterns may occur. For example, with respect to a linear image such as hair on the skin portion, a shading pattern may remain in this portion.
In order to avoid such a side effect, it is effective to eliminate regularity from the thinning pattern. As a result of the experiment, a pattern in which the pixels to be calculated are arranged in a concentric circle as shown in FIG. 7C is more desirable. And got the result.

  The result of the smoothing process can be output to a data recording medium (CD, DVD) or printer as an image output (S18).

FIG. 2 is a hardware configuration example when the image processing method of the present invention is realized by computer processing. The image processing apparatus 1 includes a CPU 2, a memory 3, a monitor 4, an I / O 5, a keyboard 6, a mouse 7 and a hard disk 8 that are connected via a bus 9.
The memory 3 is a data storage location for storing a program to be executed and temporary work data when the CPU 2 executes the program 82.
The monitor 4 is a display device that displays visual information, and the keyboard 6 and the mouse 7 are input devices that receive information input and instructions from an operator and transmit the signals to the CPU.
I / O 5 represents a connection port for connection with the outside, and is connected to a printer or other system via an external device such as a digital camera, a CD, or a DVD, or a network, for example.
The hard disk 8 includes data 81 including information to be handled by the system, an image processing program for executing each step of the smoothing process flow of FIG. 3, a program 82 executed by the CPU, and the system functions as a computer. OS83 etc. which are the basic software for this are stored.
Further, the above-described image processing and input / output control are realized by the CPU 2 in FIG.

Functional block diagram of the image processing method of the present invention Hardware configuration diagram The figure which shows the processing flow of the image processing of this invention Diagram showing the process of generating a skin area mask Diagram showing the relationship between face size and two-dimensional matrix Diagram showing the principle of MTM filter Diagram showing the relationship between two-dimensional matrix and thinning pattern

Explanation of symbols

1 Image processing device 2 CPU
3 Memory 4 Monitor 5 I / O
6 keyboard 7 mouse 8 hard disk 20 processing unit 21 operation unit 27 data storage unit 42 face rectangular area 43 skin area mask 61 target pixel 62 calculation target pixel

Claims (6)

A human image input stage for inputting image data including a human face to a computer;
A skin area extraction step of identifying a face area from the person image, extracting a skin portion from the face area, and generating skin area mask data indicating the skin area;
Based on the skin region mask data, a smoothing processing step of performing filtering processing with a two-dimensional matrix of a square array on each pixel in the skin region;
An image processing method for correcting an unnecessary image portion of a face image, comprising:
The smoothing processing step includes an actual face dimension setting step for designating an actual face dimension,
An unnecessary image actual dimension setting step for designating the dimension of the unnecessary image part on the actual object;
The value obtained by dividing the unnecessary actual image size by the actual face size is integrated with the number of pixels of the face image obtained from the face area, and the odd value closest to the integration result is the number of pixels on one side of the two-dimensional matrix. A matrix sizing step defined as:
An image processing method comprising: 2. The image processing method according to claim 1, wherein when the number of pixels on one side of the two-dimensional matrix is five or more, the calculation target pixels used in the filtering process are selected by thinning out in a stepping stone shape. The image processing method according to claim 2, wherein calculation target pixels used in the filtering process are concentrically selected with respect to the target pixel at the center of the two-dimensional matrix with the same target image as the center. Person image input means for inputting image data including a person's face to a computer;
A skin area extracting unit that identifies a face area from the person image, extracts a skin portion from the face area, and generates skin area mask data indicating the skin area;
Based on the skin area mask data, smoothing processing means for applying a filtering process with a square array of two-dimensional matrix to each pixel in the skin area;
Unnecessary image actual dimension input means for setting the actual dimension of the unnecessary image portion to be subjected to the filtering process;
A value obtained by dividing the unnecessary image actual size by a predetermined average face size and the number of pixels of the face image obtained from the face area are integrated, and an odd value closest to the integration result is calculated as the two-dimensional value. Matrix size determining means defined as the number of pixels on one side of the matrix;
An image processing apparatus comprising: The image processing apparatus according to claim 4, wherein when the number of pixels on one side of the two-dimensional matrix is 5 or more, calculation target pixels used in the filtering process are selected by thinning out in a stepping stone shape. The image processing apparatus according to claim 5, wherein calculation target pixels used in the filtering process are concentrically selected with respect to the target pixel at the center of the two-dimensional matrix with the same target image as the center.

Images