JP2009294955A - Image processor, image processing method, image processing program and recording medium with the same program recorded thereon - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide image correction technology for removing a spectacles pattern in an image by generating a spectacles model adaptable to spectacles with various frame shapes. <P>SOLUTION: A pre-processing part 12 of an image processor 10 identifies and detects a face region from input image data. A spectacles model generation part 15 creates a spectacles model under the consideration of textures such as spectacles frame shapes and the shapes of eyeballs in a spectacles frame from the learning pattern of a sample image. A feature extraction means part 14 applies the spectacles model to the face region, and extracts a spectacles model parameter. A spectacles identification part 16 reads an identification parameter for identifying the presence of spectacles, and decides the presence of the spectacles in input image data with a spectacles model parameter. A region extraction part 18 divides the input image data into arbitrary regions. An interpolation part 19 creates an arbitrary texture from the data of the division regions, and integrates the respective regions. This integration result is output by an output part 20. <P>COPYRIGHT: (C)2010,JPO&INPIT

Description

  The present invention relates to an image processing technique for determining presence / absence of spectacles and a spectacle region included in an image and accurately removing spectacle patterns.

  In the field of computer vision today, face authentication technology is frequently used as one of biometrics in surveillance systems, pet robots, human interfaces, and the like. For face authentication, the face image registered in advance and the input image are compared and recognized, so basically face images without obstructions such as partial occlusion, glasses and ornaments that cause misrecognition. It is necessary to register.

  Therefore, when registering a face image of a person wearing glasses, there is a need to register two types of images: a face image when wearing glasses and a face image when wearing glasses. In the face authentication system, a technique for removing only the eyeglass region from the face image is required in order to save such troubles twice and to reduce misrecognition due to partial masking of the face.

The glasses removal processing procedure can be divided into five parts: “pre-processing unit”, “glasses model generation unit”, “glasses detection unit”, “region extraction unit”, and “interpolation unit”. As an example of a conventional spectacle removal method, there is a method using a parametric spectacle frame model (see Non-Patent Document 1). In this technique, the eyeglass region is removed by the following procedure. That is, a face image of a person wearing glasses is input, and a spectacle frame region is detected by comparing a spectacle frame model prepared in advance with line information obtained by detecting edges of the input image. Next, after the luminance value of the detected spectacle frame region is removed, the luminance value of the spectacle frame region is linearly interpolated with the luminance value of the skin color portion of the peripheral region. A spectacles-removed image is generated by adding a deviation to the linearly interpolated portion and smoothing it.
Yasuyuki Saito, Yukiko Kenmochi, and Kazuya Kotani: “Extraction and removal of eyeglass frame regions from facial images using parametric eyeglass frame models”, IEICE Transactions, D-II Vol. J82-D-II No. 5 pp. 880-890, May 1999 Yuichi Araki, Nobutaka Shimada, Yoshiaki Shirai: “Detection of face and estimation of face direction independent of background and face direction”, Science Technique Vol. 101 no. 569 pp. 87-94, 2002 T.A. F. Cootes, G .; J. et al. Edwards, and C.I. J. et al. Taylor. “Active appearance models”. IEEE TPAMI, VOL. 23, NO6, JUN 2001: 681-685 Nello Cristianini, John Shawe-Taylor, “An Induction to Support Vector Machines”: And Other Kernel-Based Learning Methods, Cambridge P94 to Cambridge P Jian Sun, Lin Liang, Fang Wen, Heung-Yung Shum: “Image Vectorizing using Optimized Gradient Meshes”, ACM Transactions on Graphics, Vol. 26, no. 3, Article 11, Publication date: July 2007.

  However, the conventional spectacle removal method has the following problems. That is, since the shape and color of a spectacle frame are generally various, even if a spectacle frame model is created and used, erroneous detection of the spectacle frame occurs. This can be attributed to the fact that the spectacle frame is detected only from the line information obtained by the edge extraction, and that the spectacle frame only model is used. In order to realize the removal of the spectacle region from the face image, not only the spectacle frame position is estimated from the line information obtained from the edge extraction, but also additional information such as the spectacle frame and the texture inside the spectacle lens is necessary. It can be said that there is.

  In addition, since a face image including glasses is used as an input, this technique cannot be actually applied without a prior information that a face exists in the input image and glasses are included.

  Furthermore, even if the detected luminance value of the spectacle frame region is linearly interpolated or smoothed with the luminance value of the flesh-colored portion of the surrounding region, the contrast at the boundary between the spectacle frame region and the adjacent frame inner region and the outer region of the spectacle frame Is discontinuous and the processing result image may appear unnatural, so that the effect is not sufficient.

  Accordingly, a first object of the present invention is to generate a spectacle model that can accommodate spectacles having various frame shapes and remove a spectacle pattern. Further, it is a second problem to identify the presence of a face and the presence or absence of glasses for enabling the removal of glasses regardless of the input of any image regardless of the presence or absence of glasses or the presence of a face. Furthermore, a third problem is to interpolate the removal region so that the processing result of the eyeglass removal looks natural.

  The present invention is a technical idea created to solve the above-mentioned problems, and the invention according to claims 1 to 8 detects a spectacle region in consideration of texture information in addition to the shape of a spectacle frame and an eyeball. This solves the first problem. The invention according to claim 2.5 solves the second problem by making it possible to identify the presence of a face in a captured image and the presence or absence of glasses. The invention described in claim 3.6 solves the third problem by dividing the face area into a plurality of small areas, generating a layer including a texture in consideration of the texture of the entire area, and synthesizing the layer. .

  Specifically, the invention described in claim 1 is an image processing apparatus that identifies the presence or absence of glasses from a captured image of a person, and generates a face image from which the glasses pattern is removed. Eyeglass model generating means for generating a spectacle model that takes into account the frame shape and the shape of the eyeball inside the spectacle frame, and the texture inside the spectacle frame including the spectacle frame and the eyeball, and the face area and the spectacles in the input captured image A feature extraction unit that extracts a feature amount of a spectacle region from a model, a spectacle identification unit that compares the feature amount of the spectacle region with a feature amount of a learning pattern, and determines the presence or absence of spectacles in the captured image; An area extraction unit that divides the captured image into arbitrary areas from the feature values of the face area and the spectacle area, and generates an arbitrary texture from the data of each of the divided areas. And includes an interpolation means for integrating the regions, it is characterized by outputting the integrated results.

  The invention according to claim 2 includes pre-processing means for identifying the presence or absence of a face in the captured image and extracting the face area, wherein the feature extraction means searches for the spectacle area, The spectacles model parameter obtained as a result is extracted as a feature value, and the spectacles identification unit reads the learning pattern identification parameter stored in the database as a feature value, and uses the spectacle model parameter to store in the captured image. It is characterized by determining the presence or absence of glasses.

  According to a third aspect of the present invention, the region extracting means calculates a coordinate value of the spectacle region, a coordinate value of the spectacle frame region, and a coordinate value of the eyeball region in the input image data, and a plurality of the face regions. Means for dividing and dividing the area into a plurality of areas, and the interpolation means fills the hollow pixel values of the objects in each area cut by the area extraction means, and creates a layer of an arbitrary area And means for automatically generating a texture from the object and correcting the texture, and means for converting and integrating the plurality of layers into a single image data.

  The invention according to claim 4 is an image processing method for identifying the presence or absence of eyeglasses from a captured image of a person and generating a face image from which the eyeglass pattern is removed, wherein the eyeglass model generating means A first step of generating a spectacle model that takes into account the spectacle frame shape, the shape of the eyeball inside the spectacle frame, and the texture inside the spectacle frame including the spectacle frame and the eyeball; The second step of extracting the feature amount of the spectacle region from the face region and the spectacle model, and the spectacle identification means compares the feature amount of the spectacle region with the feature amount of the learning pattern, and the spectacles in the captured image A third step of determining whether or not there is a fourth step, and a region extracting unit divides the captured image into arbitrary regions from the feature amounts of the face region and the spectacle region; The interleaving unit has a fifth step of generating an arbitrary texture from the data of each divided area and integrating the areas, and a sixth step of outputting the integration result of the fifth step. It is said.

  According to a fifth aspect of the present invention, the preprocessing means further includes the step of identifying the presence or absence of a face in the captured image and extracting the face region, and the second step searches for the spectacle region. Then, the eyeglass model parameter obtained as a result of the search is extracted as a feature amount, and the third step reads the learning pattern identification parameter stored in the database as the feature amount, and uses the eyeglass model parameter to perform the imaging. It is characterized by determining the presence or absence of glasses in the image.

  According to a sixth aspect of the present invention, the step of calculating the coordinate value of the spectacle region, the coordinate value of the spectacle frame region, and the coordinate value of the eyeball region in the input image data includes a plurality of face regions. The fifth step includes a step of filling the pixel value of the cavity of the object in each region cut by the region extraction means, and a layer of an arbitrary region. A step of automatically generating a texture from the object and correcting the texture; and converting and integrating the plurality of layers into a single image data.

  A seventh aspect of the present invention relates to an image processing program that causes a computer to function as each means constituting the image processing apparatus according to any one of the first to third aspects.

  The invention according to claim 8 relates to a recording medium on which the image processing program according to claim 7 is recorded.

  According to the first to eighth aspects of the present invention, the spectacle region is detected in consideration of the texture information in addition to the spectacle frame and the shape of the eyeball. Since a large amount of information is used, it is possible to reduce false detection as a result.

  In particular, according to the invention described in claim 2.5, it is possible to identify the presence of the face in the captured image and the presence or absence of the glasses, so that any image data can be input regardless of the presence or absence of the glasses or the face. It becomes like this.

  According to the invention described in claim 3.6, the face area is divided into a plurality of small areas, and a layer including a texture that takes into consideration the texture of each of the entire areas is generated and synthesized. An image processing result from which the eyeglass pattern is removed can be obtained without impairing the feeling.

  Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings. FIG. 1 is a functional block diagram of an image processing apparatus according to an embodiment of the present invention. The image processing apparatus 10 is configured by a computer, and includes an image data input unit 11, a preprocessing unit 12, a first learning pattern storage unit 13, a feature extraction unit 14, a spectacle model generation unit 15, a spectacle identification unit 16, and a second learning. A pattern storage unit 17, a region extraction unit 18, an interpolation unit 19, and an output unit 20 are included.

  Specifically, the image processing apparatus 10 includes hardware resources such as a normal computer component, for example, a CPU (Central Processor Unit), a memory (RAM), a hard disk drive device, a communication device, and the like. And the processes of the functional blocks 11 to 20 are executed in cooperation with the installed software. Among these, the learning pattern storage units 13 and 17 are constructed as a database in the hard disk drive.

  Then, the overall processing of the image processing apparatus 10 will be schematically described along the flow of FIG. 2. First, the image data input unit 11 is realized by the communication device or the like, and has an imaging means such as a digital camera. The captured digital image data is input via the network, and the input captured image data (hereinafter abbreviated as input image data) is transmitted to the preprocessing unit 12 (S1).

  Next, the pre-processing unit 12 compares the input image data with the feature amount of the pattern stored in the first learning pattern storage unit 13 to perform face detection and identify whether or not a face exists ( S2). At this time, if it is identified that there is a face in the input image, a portion that is considered to be a face is cut out and transmitted to the feature extraction unit 14 (S3). On the other hand, if it is identified that there is no face in the input image, the process ends.

  The feature extraction unit 14 extracts a spectacle model parameter serving as a feature amount and sends it to the spectacle identification unit 16 (S4). The spectacle model parameters are generated based on the input face area image data and the spectacle model generated by the spectacle model generation unit 15.

  The eyeglass identification unit 16 uses the feature amount (glasses model parameter) transmitted from the feature extraction unit 14 and the identification parameter stored in the second learning pattern storage unit 17 to obtain the identification result of the presence or absence of glasses (S5). ). At this time, when it is determined that there is glasses (S6), the following processes (S7) to (S9) are continued. On the other hand, when it is determined that there is no glasses, the process ends. Here, in the second learning pattern storage unit 17, learning samples are generated from a large number of learning images in advance, identification parameters are generated by learning, are held, and are transmitted to the eyeglass identification unit 16.

  The area extraction unit 18 divides the digital image data into a plurality of areas and extracts them from the image data of the face area obtained by the preprocessing unit 12 and the feature amount (glasses model parameter) obtained by the feature extraction unit 14. (S7).

  In the interpolation unit 19, each region obtained by the region extraction unit 18 is layered as shown in FIG. 3, the raster-represented image data of each layer is converted into a vector representation, color correction is performed, and the generated data is generated. The result obtained by integrating the layer data is converted into a raster representation and transmitted to the output unit 20 (S8). The output unit 20 outputs the transmitted image data (S9). The output image data is displayed on a display means such as a monitor. Hereinafter, the processing of each of the functional blocks 11 to 20 will be described individually and specifically.

(1) Pre-processing unit 12
The execution method of the process in the preprocessing unit 12 is shown in the flowchart of FIG. The pre-processing unit 12 performs face detection from the input image data to cut out a face area (S10) to (S12). This face detection may be performed by the method described in Non-Patent Document 2. Here, for example, a face detection method that can detect a face facing in a direction other than the front is used. In general, in face detection, learning is performed in advance using a large number of learning images, and the identification parameters obtained as a result of the learning are accumulated in a learning pattern storage unit. The presence / absence of a face in the image is identified using the identification parameter, and the position in the image is specified.

  Similarly, in the present invention, at the time of face detection (S10), an identification parameter is read from the first learning pattern storage unit 13 and used for identification. The first learning pattern storage unit 13 has a function of performing learning from a large number of input learning images to generate and hold an identification parameter.

  Next, when it is identified that there is a face in the input image data (S11), face area extraction (S12) is performed. If it is identified that there is no face, the process ends. In the face area extraction (S12), image data of a face area having pixel values of the face area and image data of a background area having background pixel values other than the face area are generated. First, two copies of the input digital image data are created in order to generate image data for the face area and image data for the background area. Subsequently, using the coordinate value information of the eyes, nose, and mouth, which are facial parts in the image obtained as a result of the face detection (S10), the pixel values in an elliptical area of any size including all parts Select. Then, in one copy image, all the pixels other than the selected elliptical area are converted into white (or black) to obtain face area image data. Similarly, in the other copy image, all the pixels in the selected elliptical area are converted to white (or black) as background area image data.

(2) Eyeglass model generation unit 15
FIG. 5 shows a flowchart of processing in the spectacle model generation unit 15. The spectacle model generation unit 15 generates a spectacle model (S13, S14) and outputs retained data (S15). The “Active Appearance Model” (hereinafter abbreviated as “AAM”) is used in the spectacle model generation processing.

  “AAM” is a method of matching an unknown image with a generated model composed of two processes: creation of a statistical model called an appearance model and parameter adjustment by model search. In the appearance model creation process, a statistical model having a correlation between the coordinate value of the feature point representing the shape of the object and the luminance value representing the appearance as a parameter is created. The parameter adjustment process by model search results in an optimization problem that changes the parameters of the generated appearance model to minimize the image residual between the generated appearance model and the input unknown image, and solves the optimization problem Thus, the input unknown image is matched with the generated model. The spectacle model generation unit 15 only creates an appearance model, which is the first step of “AAM”.

  The following outlines the appearance model creation. The appearance model is created using a learning image data set in which feature points are arranged. First, a vector in which the coordinates of feature points in one learning image are arranged is set as a shape vector “x”, and an average shape “x bar” of the shape vectors of all learning images is obtained. Next, in each learning image, the shape is normalized by the average shape “x bar”, and the luminance values in the normalized shape are arranged to obtain the texture vector “g”. Similar to the shape vector, the average texture “g bar” of the texture vectors of all learning images is calculated. The shape vector “x” and the texture vector “g” can be modeled as shown in Equation (1).

Here, “Q _s ” and “Q _g ” are eigenvectors, and “c” is an appearance parameter that controls both shape and texture. By using the appearance parameter “c”, not only the shape but also the texture can be expressed. Conversely, if the appearance parameter “c” is obtained, the shape and texture can be calculated. (See Non-Patent Document 3 for the basic principle of “AAM”).

Specifically, a face image with spectacles and coordinate values of feature points arranged on the face image as shown in FIG. 6 are prepared as learning image data, and the spectacle frame and the inside of the spectacle lens, the shape of the eyeball, From the texture, the above “x bar”, “g bar”, “Q _s ”, and “Q _g ” are calculated. However, the coordinates of the spectacle frame partial feature points in the image in FIG. 6 are (x _f , y _f ), the coordinates of the lens partial feature points in the image (x _l , y _l ), and the eyeball partial feature points in the image Is set to (x _e , y _e ). At this time, the shape vector “x” is expressed by Expression (2).

“X bar”, “g bar”, “Q _s ”, and “Q _g ” obtained from the learning image data set are held in a temporary storage means such as a memory as an appearance model and transmitted to the feature extraction unit 14. To do. The appearance model may be stored in a hard disk drive device.

(3) Feature extraction unit 14
A method for executing processing in the feature extraction unit 14 is shown in the flowchart of FIG. The feature extraction unit 14 receives the face region image data transmitted from the preprocessing unit 12 and uses the spectacle model created by the spectacle model generation unit 15 to solve the optimization problem, thereby searching for the spectacle region. Do. The spectacle model parameter obtained as a result of the search is used as a feature amount, and the feature amount and the face area data transmitted from the preprocessing unit 12 are transmitted to the spectacle identification unit 16.

Specifically, the spectacle region is searched by parameter adjustment by model search, which is the second step of “AAM” (S16). Here, when searching for the spectacle region from the image data, in addition to the appearance parameter “c” representing the shape and texture of the model, a new parameter “in consideration of the position, size, and rotation of the spectacles in the image data of the face region” p "is used. The image residual between the texture vector “g _s ” created from the pixel values of the part of the image data of the face region that is considered to be glasses and the texture vector “g _m ” created using the glasses model is expressed as “r (p) = g _s −g _m ”,“ p ”is changed so that“ r (p) ”is minimized, and an optimal solution is obtained. “C” is calculated backward from the obtained optimal solution, and this is used as a spectacle model parameter. The details of the optimization problem here are the same as in Non-Patent Document 3.

(4) Second learning pattern storage unit 17
In the second learning pattern storage unit 17, identification function data for identifying the presence or absence of glasses is generated and transmitted to the glasses identification unit 16. For generation of identification function data, “Support Vector Machine (hereinafter abbreviated as SVM)” is used. “SVM” is a technique for constructing a pattern discriminator for solving a two-class classification problem. The discriminator is configured as a hyperplane that maps an input vector to a virtual higher-order feature space by a non-linear mapping and serves as a boundary that maximizes the generalization ability between the two classes in the feature space. The basic principle of “SVM” here is the same as in Non-Patent Document 4.

  A flowchart of the processing procedure of the second learning pattern storage unit 17 is shown in FIG. The second learning pattern storage unit 17 includes a learning sample generation unit 17-1 and an identification function data generation unit 17-2. The learning sample generation unit 17-1 generates the same number of learning samples as the learning image from a large number of input learning images. The generated learning sample includes a feature amount and a label indicating a class to which the learning sample belongs.

  In FIG. 8, the number of learning samples “n = 0” is set before the label input (S19). Two types of images, that is, a face image of a person wearing glasses and a face image of a person not wearing glasses are input to the learning sample generation unit 17-1 (S20). Note that the input face image belongs to the class with glasses for the face image of a person wearing glasses, and belongs to the class without glasses for the face image of a person without glasses. The label of the face image belonging to “y = 1” and the label of the face image belonging to the class without glasses is given as “y = −1”.

  Using the spectacle model generated by the spectacle model generation unit 15 for the input set of learning images and labels, parameter adjustment by model search is performed in the same manner as the parameter optimization processing (S16) of the feature extraction unit 14 (S16). S21). The obtained spectacle model parameter is used as the feature amount of the learning image. The obtained feature amount and label set is set as one learning sample (n = 1) (S22). This process is repeated (S 23, S 24), and is performed on all prepared learning images, and feature quantities are respectively extracted to generate “n ← n + 1” learning samples. This learning sample may be stored in a temporary storage means such as a memory.

In the discrimination function data generation unit 17-2, the learning sample generated in the learning sample generation unit 17-1 is input, and learning is performed with “SVM” to obtain discrimination function data. In the learning of “SVM”, when l feature quantities “x _i (i = 1, 2,..., L)” are given together with the label “y”, the discriminant function “f (x)” It is given in the form of (3). The equation (3) is solved and optimized (S25).

Here, “α ^* , β ^* ” is an optimal solution of the maximization problem of the following equation (4).

In the discrimination function data generation unit S18, “α ^* , β ^* ” and “x _i ” generated by learning are accumulated in the second learning pattern storage unit 17 as discrimination function data (S26). The discriminant function of equation (3) may be a non-linearly expanded one.

(5) Glasses identification unit 16
The execution method of the process in the spectacles identification part 16 is shown in the flowchart of FIG. The eyeglass identification unit 16 performs a process for identifying the presence or absence of eyeglasses. When discriminating between a class with glasses and a class without glasses, the feature quantity generated by the feature extraction unit 14 and the identification function data are called from the second learning pattern storage unit 17, and which class is used by using them. Is identified (S27).

  For example, the feature amount generated by the feature extraction unit 14 is set as an input “x”, and is substituted into Expression (3). In the identification using “SVM”, when an unknown input “x” is given, the class to which the unknown image belongs is determined by the code of the identification function “f (x)”. Specifically, the class with glasses is set when “f (x) ≧ 0”, and the class without glasses is set when “f (x) <0”. Only when it is determined that the feature amount input here is a spectacle pattern, the data is transmitted to the region extraction unit 18. If it is determined that there is no spectacle pattern, the process ends here. In addition to the “SVM”, for example, nearest neighbor method, Fisher's linear identification method, neural network, subspace method, or the like may be used in the identification processing.

(6) Region extraction unit 18
A method for executing processing in the region extracting unit 18 is shown in the flowchart of FIG. The region extraction unit 18 includes a spectacle region position calculation processing unit 18-1 and a region cutout unit 18-2. The spectacle region position calculation processing unit 18-1 calculates the position of the spectacle region in the image using the spectacle model parameters obtained by the feature extraction unit 14 (S30).

Specifically, the shape vector “x” is calculated by performing reverse calculation from the spectacle model parameter “c”. Thereafter, the in-image coordinate values (x _f , y _f ) of the spectacle frame partial feature points in the obtained shape vector are calculated as the spectacle region position. Next, (x _f , y _f ) and (x _l , y _l ) are calculated as spectacle frame region positions, and (x _e , y _e ) are calculated as eyeball region positions (S31).

  Next, the processing (S32 to S34) of the area cutout unit 18-12 will be described with reference to FIG. FIG. 11 is an explanatory diagram of each area in the area extraction process. The clipping process generates image data in which all pixel values other than the specific area are replaced with white (or black), as in the face area extraction (S12) of the preprocessing unit 12. First, the spectacle region shown in FIG. 11B calculated from the face region shown in FIG. 11A is selected, and the image data storing the pixel values of the spectacle region is used as spectacle region data. The face base area shown in FIG. 11C is saved and the pixel value inside the eyeglass area is changed to white (or black) (S32). Further, image data in which the pixel values of the spectacle frame region portion shown in FIG. 11 (d) are stored from the spectacle region is created, and image data in which only the pixel values of the eyeball region portion in FIG. 11 (e) are stored from the spectacle region. Create (S33). As shown in FIG. 11F, the image data of the lens region is obtained by setting the pixel values of the frame region and the eyeball region to white (or black) from the eyeglass region (S34).

(7) Interpolation unit 19
The interpolation unit 19 vectorizes each region generated by the region extraction unit 18 and divides the layer. Vectorization is a series of image data described as a set of pixels that can only take discrete values, such as digital image data called raster representation, which is described as a line segment connecting feature points and vertices of an object called vector representation. Conversion to image data expressed in a format that can take values. Here, a gradient mesh is used as a specific vectorization technique. The gradation mesh is a graphic technique that can realistically express the texture of an object by dividing the shape of a target object into a mesh and applying a color gradation along the mesh. Further, vectorization by gradation mesh is automatically generated by an optimization method as described in Non-Patent Document 5.

  The structure of the gradation mesh in Non-Patent Document 5 will be outlined below with reference to FIG. FIG. 12A shows an example of a parametric curved surface in which each coordinate is expressed as a function of the parameter “u, v”, and this is hereinafter referred to as a patch. The Ferguson patch used in Non-Patent Document 5 is expressed by Equation (5).

  The gradation mesh is composed of patches and mesh lines, and the points on the Ferguson patch have four types of information: positions for expressing the mesh lines, position differentiation, color, and color information differentiation. FIG. 12B is an example of a gradation mesh, and two curves passing through the point “q” are mesh lines. For example, in the case of the point “q” in FIG. 12B, the position information of Expression (6), the differential information of the position of Expression (7), the RGB color information of Expression (8), and the RGB color of Expression (9) The gradation mesh is expressed by equation (10).

  In Non-Patent Document 5, optimization in vectorization is performed according to the following procedure. That is, first, the mesh is initialized by creating a plurality of patches, for example, by equally dividing a region to be vectorized in the input raster image. Now, when P patches are generated and the image to be vectorized is “I”, the mesh is optimized and vectorized by minimizing the energy function of equation (11).

  A specific processing procedure in the interpolation unit 19 is shown in the flowchart of FIG. The interpolation process includes a face base layer creation unit 19-1, a lens layer creation unit 19-2, and an eyeball layer creation unit 19-3 in FIG. All the information created by each of the layer creation units 19-1 to 19-3 is synthesized into one image by the layer integration processing S43.

  Hereinafter, processing of each of the layer creation units 19-1 to 19-3 will be described. FIG. 14 is a conceptual diagram for explaining the processing in the face base layer creation unit 19-1 in the flowchart of FIG. The face base layer creation unit 19-1 uses the face base region information created by the region extraction unit 18, and uses the face base hole filling process (S38) to open the eyeglass region hole that was opened when the face base region was cut out. Next, face-based vectorization processing (S39) is performed.

  Specifically, the face base hole filling process (S38) interpolates the color of the eyeglass region by the average of the pixel values of the pixels at the boundary with the hole region in the face base region of FIG. An image having the same color as the skin is also generated in the hole region as shown in (b). The face base vectorization process (S39) creates a gradation mesh in which the face base area is optimized by the optimization method. FIG. 14C shows mesh shape information automatically created by optimization. A texture is also generated on the mesh by optimization. The generated vector image as shown in FIG. 14D having both mesh information and texture information is used as a face base layer, and the generated data is transmitted to the layer integration processing S43 and the lens layer creation unit 19-2.

  The eyeball layer creation unit 19-3 uses the information on the eyeball region created by the region extraction unit 18 and performs the eyeball region vectorization process (S42) using the optimization method. The vector image generated here is used as an eyeball layer, and the generated data is transmitted to the layer integration processing S43 and the lens layer creation unit 19-2.

  The lens layer creation unit 19-2 uses the lens region information created by the region extraction unit 18 and performs lens region vectorization processing (S40) using the optimization method. The vector image generated here is used as a lens layer. Next, lens layer color correction (S41) is performed using the information of the generated three layers of the face base layer, the eyeball layer, and the lens layer. The details of the lens layer color correction (S41) will be described with reference to FIG. The generated layer has a mesh as shown in FIG. 15A and a mesh point that is an intersection of the mesh. The mesh point retains color information and can control the color condition in the vicinity of the mesh point. First, the color of the face base side mesh point in FIG. 15B, which is the mesh point of the outer contour portion of the lens layer, is the same coordinate value location in the corresponding face base layer as shown in FIG. 15C. Replace with color. Next, the color of the eyeball side mesh point in FIG. 15B which is the mesh point of the inner contour portion is replaced with the color of the location of the same coordinate value in the corresponding eyeball layer as shown in FIG. To control the color.

  In layer integration processing S43, the background area, face base layer, lens layer, and eyeball layer cut out in the preprocessing are sequentially overlapped in order from the bottom and converted into a single raster image data. The process of converting to a raster image is to convert an image that has been vector-expressed into image data that takes discrete values. For automatic generation of a vectorized image, “subdivisionmesh” may be used in addition to the method of non-patent literature.

  In addition, the present invention is not limited to the above-described embodiment, and can be constructed as a program that causes a computer to function as each of the functional blocks 11 to 20 of the image processing apparatus 10, for example. This program may cause the computer to execute all the processes of the functional blocks 11 to 20, or may cause the computer to execute a part of the functions.

  This program can be provided to a computer by downloading from a website or the like. Also, it is stored in a recording medium such as a CD-ROM, DVD-ROM, CD-R, CD-RW, DVD-R, DVD-RW, MO, HDD, Blu-ray Disk (registered trademark) and provided to a computer. May be. Since the program code itself read from the recording medium realizes the processing of the above embodiment, the recording medium also constitutes the present invention.

1 is a functional block diagram of an image processing apparatus according to an embodiment of the present invention. The flowchart which shows the whole process. The schematic of layering by the interpolation means. The flowchart which shows the process of the same preprocessing part. The flowchart which shows the process of the spectacles model production | generation part. The conceptual diagram of the spectacles model which the same feature extraction part uses. The flowchart which shows the process of the feature extraction part. The flowchart which shows the process of the 2nd learning pattern memory | storage part. The flowchart which shows the process of the spectacles identification part. The flowchart which shows the process of the same area extraction part. The conceptual diagram of the extraction area | region of the same area extraction part. The conceptual diagram which shows the structure of the gradation mesh of the interpolation part. The flowchart which shows the process of the interpolation part. The conceptual diagram which shows the process sequence in the same face base layer preparation part. The conceptual diagram which shows the process sequence of the same lens layer color correction.

Explanation of symbols

DESCRIPTION OF SYMBOLS 10 ... Image processing apparatus 11 ... Image data input part 12 ... Pre-processing part (pre-processing means)
13 ... 1st learning pattern memory | storage part 14 ... Feature extraction part (feature extraction means)
15 ... Eyeglass model generation unit (glasses model generation means)
16 ... Glasses identifying unit (glasses identifying means)
DESCRIPTION OF SYMBOLS 17 ... 2nd learning pattern memory | storage part 17-1 ... Learning sample production | generation part 17-2 ... Discrimination function data generation part 18 ... Area extraction part (area extraction means)
18-1 ... Eyeglass region position calculation unit 18-2 ... Region cutout unit 19 ... Interpolation unit (interpolation means)
19-1 ... Face base layer creation unit 19-2 ... Lens layer creation unit 19-3 ... Eyeball layer creation unit 20 ... Output unit

Claims (8)

An image processing apparatus that identifies the presence or absence of glasses from a captured image of a person and generates a face image from which a glasses pattern is removed,
A spectacle model generation means for generating a spectacle model taking into account the spectacle frame shape and the shape of the eyeball inside the spectacle frame, and the texture inside the spectacle frame including the spectacle frame and the eyeball, from the learning pattern of the sample image;
Feature extraction means for extracting a feature amount of a spectacle region from the face region in the input captured image and the spectacle model;
A pair of glasses identifying means for comparing the feature amount of the glasses region and the feature amount of the learning pattern to determine the presence or absence of glasses in the captured image;
An area extracting unit that divides the captured image into arbitrary areas from the feature quantities of the face area and the eyeglass area;
Interpolating means for generating an arbitrary texture from the data of each divided area and integrating the areas;
An image processing apparatus that outputs the integration result. Recognizing the presence or absence of a face in the captured image, and comprising preprocessing means for extracting the face region;
The feature extraction means performs a search of the spectacle region, extracts spectacle model parameters obtained as a result of the search as a feature amount,
The eyeglass identification unit reads an identification parameter of a learning pattern stored in a database as a feature amount, and uses the eyeglass model parameter to determine the presence / absence of eyeglasses in the captured image. Image processing apparatus. The area extracting means calculates a coordinate value of the eyeglass area, a coordinate value of the eyeglass frame area, and a coordinate value of the eyeball area in the captured image; and a means for dividing the face area into a plurality of areas. With
The interpolation means includes a means for filling pixel values of a cavity of an object in each area separated by the area extraction means, a means for creating a layer of an arbitrary area, and a texture automatically generated from the object to generate the texture. The image processing apparatus according to claim 1, further comprising: a unit that corrects the plurality of layers, and a unit that converts and integrates the plurality of layers into a single image. An image processing method for identifying the presence or absence of glasses from a captured image of a person and generating a face image from which a glasses pattern is removed,
A first step in which the spectacle model generation means generates a spectacle model taking into account the spectacle frame shape and the shape of the eyeball inside the spectacle frame, and the texture inside the spectacle frame including the spectacle frame and the eyeball, from the learning pattern of the sample image;
A second step in which a feature extraction unit extracts a feature amount of the spectacle region from the face region in the input captured image and the spectacle model;
A third step in which the eyeglass identification means compares the feature amount of the eyeglass region and the feature amount of the learning pattern to determine the presence or absence of glasses in the captured image;
A fourth step in which an area extracting unit divides the captured image into arbitrary areas from the feature quantities of the face area and the eyeglass area;
A fifth step in which an interpolating unit generates an arbitrary texture from the data of the divided areas and integrates the areas;
A sixth step of outputting the integration result of the fifth step;
An image processing method comprising: Further comprising the step of identifying the presence or absence of a face in the captured image in the pre-processing means and extracting the face region;
In the second step, the eyeglass region is searched, and eyeglass model parameters obtained as a result of the search are extracted as feature amounts.
5. The third step is to read an identification parameter of a learning pattern stored in a database as a feature amount, and to determine the presence or absence of glasses in the captured image using the spectacle model parameter. Image processing method. The fourth step includes calculating a coordinate value of the spectacle region, a coordinate value of the spectacle frame region, and a coordinate value of the eyeball region in the input image data, and dividing and dividing the face region into a plurality of regions. While having
The fifth step includes a step of filling a pixel value of a cavity of an object in each region separated by the region extraction means, a step of creating a layer of an arbitrary region, a texture is automatically generated from the object, and the step The image processing apparatus according to claim 4, further comprising a step of correcting a texture, and a step of converting and integrating the plurality of layers into a single image data.   An image processing program for causing a computer to function as each means constituting the image processing apparatus according to claim 1.   A recording medium on which the image processing program according to claim 7 is recorded.

Images