JP2000311248A - Image processor - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an image processor which can fast extract an optional feature value from an image with high strength and accuracy and also can easily synthesize a portrait of high picture quality against a problem that much time is required for performing the image processing to extract the feature value of a specific object included in an image, the wrong feature value is extracted by mistake or many positions must be designated when a processing range is directly designated by means of a position designation means, etc. SOLUTION: This image processor is provided with an input means 11 which inputs the images, a storage means 12 which stores the inputted images, an arithmetic means 13 which performs an optional operation, a position designation means 14 which can designate an optional position in an image, a recognition means which recognizes the position and size of an object included in an image and a feature extraction means 10 which extracts an optional feature from a relevant image by inputting one or more positions of a relevant object included in the image as long as the relation set between the position and size of the object satisfies a fixed constraint condition.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention is used for information devices and electronic devices such as personal computers, word processors, workstations, portable information tools, copiers, scanners, facsimiles, televisions, videos, and video cameras. Specific features related to the image, such as the position, size, and shape of the eyes and mouth in a human image, are extracted, and based on the extracted information, the input image can be converted to a state desired by the operator, such as a cartoon style. The present invention relates to an image processing apparatus capable of generating an image having a taste.

[0002]

2. Description of the Related Art Conventional techniques relating to image processing of the present invention are described below. In addition, when citing the following documents, the relevant document name will be expressed in the format of "Document []". Reference [1]: R. Brunelli and T.M.
Poggio, “Face Recognition
n: Features versus Templa
tes ”, IEEE Transactions o
n Pattern Analysis and Ma
chine Intelligence, Vol. 1
5, No. 10, pp. 1042-1052, 1
993. Reference [2]: M.A. Turk and A. Pen
land, “Face Recognition”
Using Eigenfaces ", Processe
dings of IEEE Conference
on Computer Vision and Pa
ttern Recognition, p. 586-
591, 1991.・ Document [3]: Ryushi Funayama, Naokazu Yokoya, Hidehiko Iwasa, Haruo Takemura, “Cooperation of multiple dynamic network models and its application to facial parts extraction”, IEICE technical report, PRU9
5-179, p. 15-22, 1995.・ Reference [4]: Tatsushi Funayama, Hajime Takezawa, Minehiro Konya, Mitsuhiro Tootani, “Adaptive Segmentation of Face Region with Simple User Specification”, IPSJ 55th Annual Convention,
6AB-3, 1997.・ Reference [5]: Mikio Takagi, supervised by Hirohisa Shimoda, “Image Analysis Handbook”, University of Tokyo Press, 1991. Reference [6]: M.A. Kass. : Snakes: Ac
five Contour Models, Int.
J. Comput. Vision, p. 321, 1
988.・ Reference [7]: Satoshi Hosoi et al., Recognition and Synthesis of Hairstyles, IEICE Technical Report, PRMU97-15
5, pp. 25-32 (1997). -Document [8]: JP-A-10-320543-Document [9]: JP-A-10-255017-Document [10]: JP-A-08-305880-Document [11]: JP-A-10-240921 [Patent Document 1]-Document [12]: Japanese Patent Laying-Open No. 11-015947 (1) [Prior Art Related to Claims 1, 2, and 6] Generally, the following method is used to extract an arbitrary feature amount in an image. is there. Here, the arbitrary feature amount refers to an arbitrary coordinate of a specific object included in an image, a shape or size that can be expressed by a combination of coordinates, or a specific type or attribute. It is a code that can be used. For example, if a human face is included in the image, and the specific object referred to here is the right eye of the person, as one of the arbitrary feature amounts, for example, the coordinates of the inner corner of the right eye in the image and For example, a rectangle circumscribing the right eye in the image can be represented by a combination of upper left coordinates and lower right coordinates, and for example, the right eye is a sagging eye. It can be a code representing the type.

[0003] As a primitive method, it is conceivable that an operator visually inputs an arbitrary feature amount to an image displayed on a display device by using a position specifying means or the like. This is, of course, the complexity of work, the non-objectivity in which different features can be input by the operator even for the same image and the same object and the same feature, the environment for working with the same image and the same object and the same feature, There is a problem of non-constancy and the like in which different feature amounts can be input depending on states such as time. Therefore, this is often applied only in some places of high quality and small amount of video production, or in special situations such as art work production where non-objectivity etc. is respected as the sensitivity of workers.

In recent years, with the spread of personal computers, word processors, digital cameras, and the like, it has become possible to easily acquire images and perform various processes on the acquired images to enhance practicality and entertainment. Is coming. As one of the image processes, there is a case where an arbitrary feature amount as described above is extracted and various processes are performed according to the extracted feature amount. For example, if an attempt is made to convert the body color of a car in an acquired image into an arbitrary color, it is necessary to extract a feature amount indicating which region in the image is the body of the car. For example, when a person's face is included in an acquired image, in order to determine who the person is (hereinafter, referred to as individual identification), various feature amounts (eyes, nose,
It is necessary to extract the position and shape of the mouth).

[0005] In addition to the above, various applications are conceivable, and it is considered that it is very useful to extract an arbitrary feature amount in an image robustly, with high accuracy, and at high speed.

Various methods have been proposed as a basic method for extracting an arbitrary feature quantity existing in an image, such as a template matching method, a projection method, an eigenspace method, and a method using color information.

[0007] Template matching is described, for example, in the above-mentioned document [1], and is used to determine who owns a face in an image including a human face.
A plurality of patterns to be determined, that is, a plurality of grayscale images of a person's face are stored here, and individual identification can be performed by using the similarity between the input image and each storage pattern as a feature amount.

The projection method is also described in the above-mentioned document [1]. The luminance value and the differential value of the input image are summed in the horizontal or vertical direction to form a histogram, thereby forming a rectangle circumscribing the eye. Coordinates and the like can be extracted as feature amounts.

The eigenspace method is described in the above reference [2], and stores a pattern as in template matching, but stores the pattern as a probability distribution of a plurality of images instead of a single image. Is what you do. Therefore, compared to the template matching, this method has an advantage in that the influence of variations in the target pattern, for example, even a non-expression face of the same person changes with time or environment is reduced.

Further, as described in the above document [3], the color distribution of the object to be extracted is statistically obtained in advance, and the coordinates of pixels having a color applicable to the distribution in the input image are calculated as feature quantities. There is a method of extracting as This can be applied to, for example, an application in which only skin color pixels are extracted and made closer to the color in the white direction, and the other colors are converted to make the skin look fair. (2) [Prior Art Related to Claim 3] To binarize an image in order to obtain a region to be recognized, the following method is usually employed. First, convert the image as needed. For this conversion, a method of conversion into luminance, conversion by a differential operator, or the like is used. Next, a threshold is determined. There are a method of determining the threshold value using a fixed value, a method of performing a discriminant analysis, arithmetic averaging, a median or the like on pixel values in an image, and the like. Next, the threshold is compared with all pixel values, and an image is generated with a new pixel value of 1 if the pixel value is larger than the threshold and 0 if the pixel value is smaller than the threshold. (3) [Prior Art Related to Claims 4, 5, and 8] To detect the position and size of face parts such as eyes and mouth,
Usually, the following method is used. First, there is no user specification, and there is a method of detecting a face or a face part in an image by various methods such as a template matching method, a projection method, an eigenspace method, and a method using color information. In addition, the user specifies only the approximate position without specifying the face size, or the user specifies the face position, approximate position and approximate size, and uses template matching, projection method, eigenspace method, color information There is a method of estimating a part of an image in which a face or a face part is considered to be present based on information specified by a user, and applying the above method only to that part.

In addition, the conditions of the input image are kept constant by taking a picture while keeping the positional relationship between the camera and the face constant at the time of taking the picture, and the position of the face or face parts in the image is kept at a fixed position / size. There is a method of detecting a face or a face part in an image by using the above-mentioned various methods after setting the value within the range. (4) [Prior Art Related to Claim 7] To determine the shape of an eye, the following method is usually used. Assuming that the position and size of the eyes are already known, the shapes of the eyes are classified in advance into a plurality of categories using a method such as the template matching or the eigenspace method described above, and for each category, the category is expressed. A pattern having image features is created, the pattern is stored, the similarity between the pattern and the input image is calculated, and the closest one is set as the eye shape in the input image. (5) [Prior Art Related to Claim 9] In order to detect the position and size of the eyebrows, the following method is usually employed. Using a method such as the template matching or the eigenspace method described above, a pattern having an image characteristic representing an average eyebrow shape is created in advance, the pattern is stored, and the pattern is moved while moving on the input image. The similarity between the corresponding partial image of the input image and the partial image is calculated, and the position of the pattern on the input image when the similarity is the highest is defined as the position of the eyebrow. The size of the eyebrows is determined from the size of the pattern at that time. Alternatively, utilizing the feature that the eyebrows are composed of pixels darker than the surrounding pixels, the input image is binarized, and a pattern having image characteristics representing an average eyebrow shape in the binarized image in the same manner Is created, and the position and size of the eyebrows are detected by using a method such as template matching or the eigenspace method. (6) [Prior Art Related to Claim 10] In order to determine the shape of eyebrows, the following method is usually used. Assuming that the position and size of the eyebrows are already known, the shapes of the eyebrows are classified in advance into a plurality of categories using a method such as the template matching or the eigenspace method, and for each category, the category is expressed. A pattern having image features is created, the pattern is stored, the similarity between the pattern and the input image is calculated, and the closest one is used as the shape of the eyebrows in the input image. Alternatively, the input image is binarized by utilizing the feature that the eyebrows are composed of pixels darker than the surrounding pixels,
Similarly, a pattern having image features representing each category in the binarized image is created, and the shape of the eyebrows is similarly determined by using a method such as template matching or the eigenspace method. (7) [Prior Art Related to Claim 11] To determine the shape of the jaw, the following method is usually used. First, after extracting the face outline, the closest reference outline shape is selected by performing template matching between the shape and a reference shape prepared in advance. Here, as a method of extracting the face outline, for example, a method using an active outline model can be considered. This assumes a temporary outline called an initial outline in the vicinity of the extraction target, and moves the points on this outline based on the energy function set to be the minimum on the true outline, thereby shifting the outline. This is the method to be found. At this time, in addition to the energy representing the smoothness of the contour and the energy for contracting the contour, an energy characterizing the contour of the object (hereinafter referred to as image energy) is used as the energy function. As the image energy, for example, an edge image obtained by extracting an edge of the original image is used. As a method of extracting a contour using the active contour model as described above, there is the Snake method described in the above-mentioned reference [6]. (8) [Prior Art Related to Claims 12 and 13] To classify a hair shape, for example, as described in the above-mentioned reference [7], a hair color is extracted and a pixel close to the hair color is extracted. By doing so, a hair region is cut out and its shape is classified. To extract the hair color, for example, as described in the above reference [7], first, the skin color and the background color are extracted, and then the skin color of the face is extracted by extracting the skin color or a pixel similar thereto. Extract the parts,
In addition, a background portion is extracted by extracting a background color or a pixel close to the background color, and further, an area where hair pixels are present is estimated from the position and width of the face, and the hair color is determined from the hair pixels in this area. Was. (9) [Prior Art Related to Claims 14 and 15] To classify a hair shape, a hair region is cut out by extracting a hair color or a pixel close to the hair color using the hair color. 7],
After reducing the resolution, a template matching method has been used to classify the template pattern prepared in advance into a category to which the template pattern having the highest similarity belongs. (10) [Prior Art Related to Claims 16 and 17] In a device for generating a portrait by selecting parts, a hair part is determined by selecting one hair part based on the hair shape. Also,
As described in the above reference [8], there is also a method of combining a binarized hair region with a small part that expresses the hair of the bangs. (11) [Prior Art Regarding Claims 18 and 19] The above-mentioned document [9] discloses a technique of synthesizing a portrait based on a feature amount extracted from an image. This technology adjusts the correction amount of the placement feature based on gender or adult, based on the extracted location information based on gender or adult and comparison with the average arrangement in each, The arrangement information of the face parts is determined. However, in this technique, the arrangement information of the face part is determined separately from the face contour, and the face part is merely synthesized with the face contour. (12) [Prior Art Regarding Claims 20 and 21] In the conventional portrait synthesizing techniques, many techniques for simplifying an editing operation based on symmetry of a face have been reported. For example, there is a technique described in the above document [10] in which the right eye and the left eye receive the same editing operation. However, there is not much known a technique for simplifying an editing operation that focuses on the fact that a part of the same color or a part having the same characteristics as a face part receives the same editing operation.

[0012]

However, the technique described above still has the following problems. (1) [Inventions of Claims 1 and 2] A method of extracting a feature amount by template matching is as follows.
Although the feature is that the processing is simple and easy to realize, there is the following problem due to the characteristic that the pattern to be compared with the input image is stored as an image.

In the case of a stored pattern image (hereinafter, referred to as a template image) having a large number of pixels, that is, a template image including finer features, the feature amount of an object which is not an object in the input image is set as an object. While the risk of becoming the same as or close to the feature value of the existing object is reduced, the appearance of the target object in the input image is slightly different from the template image, There is a risk that the feature value will be far away from the feature value. The converse is also true: if the number of pixels of the template image is small, the feature amount of the non-target object becomes the same as or close to the feature amount of the target object, and instead, the feature amount of the target object is changed. There is an advantage that the feature amount does not change much even if the appearance changes slightly.

Therefore, for example, in consideration of the problem of detecting the position of the right eye using template matching from an input image including a human face image, a template image that can correctly detect the position of the eye in a certain image is considered. May be prepared, an erroneous position may be detected in another image, for example, a position in the background where the grayscale pattern is similar to that of the eyes.

The conventional method of extracting a feature using a projection method also has a feature that processing is simple and easy to realize. However, this method is used to extract the luminance value and differential value of an input image horizontally or vertically. Sum up the directions to create a histogram,
The feature amount is extracted by examining the histogram, and when a histogram is created, if there is something other than the target object in the direction of summing, the histogram changes accordingly, and the correct feature amount is determined. It is difficult to extract.

A method of extracting a feature using the eigenspace method as in the related art has almost the same restrictions as a method using template matching. However, the eigenspace method uses a pattern (hereinafter, referred to as a dictionary image) expressed as a probability distribution of a plurality of images instead of a single image as a stored pattern. It has the feature that it is more robust. However, creating a dictionary image requires an enormous amount of work.

A method of extracting a feature amount using color information as in the prior art is based on the following method. An incorrect feature value is extracted.

That is, in any of the above methods, when processing is performed on the entire input image, an erroneous feature amount may be extracted. In order to avoid this, the acquisition of the image is usually performed in a controlled environment, that is, for example, when photographing a person, a uniform color curtain or the like is installed on the back of the person and no extraneous matter is reflected in the background Or always keep the distance between the imaging system and the subject constant, or irradiate a certain amount of light from a certain direction, or make sure that the face faces forward and the face does not rotate. . However, providing the above-mentioned restrictions greatly impairs the convenience of an apparatus employing these methods.

Therefore, usually, the following method is used in which the range for processing is limited by some method to relax the above-mentioned limitation.

In order to limit the range for performing the above processing by using the position specifying means, the following method is usually employed. When extracting the positions of the right eye, left eye, and mouth of a person as features from an image including a person face, it is necessary to first set a search range for processing the right eye. The search range is represented by a rectangle, and at least two points must be specified to represent the rectangle.
Similarly, it is necessary to specify at least two points for the left eye and the mouth, respectively, and it is necessary to specify at least six points in total. As a result, there arises a problem that many points must be specified. (2) [Inventions of Claims 3, 5, and 8] Conventionally, as a method of performing binarization to obtain a region to be recognized, binarization of luminance, binarization after differentiation, and color information Binarization, etc. based on
Various methods, such as discriminant analysis and average values, have been used in determining the threshold value used in valuation. However, each system has advantages and disadvantages.
Can obtain the desired area, but the desired area cannot be obtained by the method B, and the desired area can be obtained by the method B in another image, but the method A
May not be able to obtain the desired area.
It is impossible to obtain a desired result with all images, even if the method is adopted or the method B is adopted. (3) [Inventions of Claims 4, 5 and 8] Conventionally, in detecting the position and size of face parts such as eyes and mouth, a method of detecting without specifying anything by the user, A method has been performed in which only the approximate position is specified without specifying the size and the information is detected from the information. However, with these methods, it is impossible to predict the size of the face part, and it is necessary to perform detection in an unnecessarily large range, so that useless calculation occurs and the detection result may be erroneously wrong. is there. In addition, the user can predict the size of the face part by a method in which the user specifies the position, approximate position, and approximate size of the face and detects from the information. However, specifying the approximate size of the face may cause the user to incorrectly specify the approximate size of the face due to problems such as concealment of the face by the hair. Sufficient accuracy cannot be obtained for the size of the face parts. Therefore, since it is necessary to perform detection in an unnecessarily large range, useless calculation occurs and the detection result may be erroneously large. In addition, in a method in which the position and size of the face and face parts in the face image are captured in advance within a certain range, it is not necessary to perform detection in an unnecessarily large range, and the detection result is unlikely to be significantly erroneous. However, there is a problem that it is impossible to detect a face or a face part from an arbitrary input image. (4) [Inventions of Claims 6 and 8] An image processing method such as template matching or projection is applied to an input image that has been conventionally used to detect the position and size of a face part, and the position of the face part is determined. In the method of detecting the size, since the applied method is greatly affected by the inclination of the face part, the face in the input image is required to be horizontal, and the face in the image is inclined. If
There is a problem that the accuracy of detecting the position and size of the face part is significantly reduced. (5) [Invention of Claim 7] When a method of determining the shape of an eye by template matching or the eigenspace method is used, first, the position and size of the eye need to be correctly detected. The above two methods are very vulnerable to misregistration, and the calculated feature amount is large if the template image or dictionary image is slightly displaced from the corresponding partial image in the input image. May be different. In addition, a template image or a dictionary image corresponding to each category whose shape is to be determined must be prepared in advance, and if the target object has a shape that is not included in the prepared category, a correct The feature amount cannot be calculated, and a large amount of work is required to prepare a template image and a dictionary image. (6) [Invention of Claim 9] When a technique of detecting the position and size of eyebrows by template matching or the eigenspace method is used, a template image or a dictionary image must be prepared in advance, but generally, it is generally used. Since eyebrows can take various shapes depending on individual differences, it is difficult to detect the positions and sizes of all eyebrows with one template image or dictionary image. Therefore, a template image or a dictionary image is created for each category set in advance, and the template matching and the eigenspace method are applied to each of the categories to obtain the position of the eyebrows with the most appropriate similarity in the most appropriate template image or dictionary image. And size must be detected.

However, as described above, preparing a template image or a dictionary image requires a large amount of work, and an erroneous position and size are detected even when one template image or a dictionary image is used. If there is a possibility that a plurality of template images or dictionary images are used, the risk of detecting an erroneous position and size increases.

Further, when a technique such as template matching or the eigenspace method is used, the size of the template image or dictionary image must be the same as the size of the target object existing in the input image. If is not known, prepare template images or dictionary images of a plurality of sizes, perform processing between all of them and the input image, or enlarge or reduce the input image, and enlarge or reduce all of them. The processing must be performed between the template image and the dictionary image. This results in a huge amount of processing.

When the input image is binarized, the threshold value must be appropriately set. However, a binarization method generally used is used due to factors such as different lighting conditions depending on the image. It is difficult to obtain a threshold value for clearly separating the eyebrows from those other than the eyebrows simply by using the eyebrows. (7) [Invention of Claim 10] When a technique of determining the shape of eyebrows by template matching or the eigenspace method is used, first, the position and size of eyebrows need to be correctly detected. The above two methods are very vulnerable to misregistration, and the calculated feature amount is large if the template image or dictionary image is slightly displaced from the corresponding partial image in the input image. May be different. In addition, a template image or a dictionary image corresponding to each category for which a shape is to be determined must be prepared in advance, and when the target object has a shape that is not included in the prepared category,
Correct feature amounts cannot be calculated, and a large amount of work is required to prepare template images and dictionary images. (8) [Invention of Claim 11] Normally, it is necessary to extract the contour of a face including a chin before performing shape determination. However, in general, sharpness is determined by lighting conditions, face orientation, changes in facial expressions, and the like. Contours are not always obtained. In particular, when an edge image is used, problems such as a break in the contour and a false contour appear easily. For this reason, the conditions of the target image often become severe, such as shooting the subject under a substantially uniform background under a constant lighting condition.

Next, when a method of determining the shape of the obtained contour line by template matching is applied, even if the correct contour line can be extracted by the face contour extraction, there is a large individual difference in the jaw contour line. The number of reference templates is increased, that is, a large number of dictionary images are required. Although it is possible to reduce the number of dictionary images and reduce the number of classifications, it is difficult to adjust a common reference dictionary image taking individual differences into account. (9) [Inventions of Claims 12 and 13] The background color is uniform or close to it so as to extract the background color and extract the background region from the entire image and prevent the hair region from entering the background region. It was necessary, and it was difficult to create a portrait from ordinary snapshots. (10) [Invention of claim 14] In the method of classifying the hair shape by template matching, it is necessary to accurately detect and classify the "separation" referred to as a so-called "seven-three division" or "center division". And
It has been difficult to perform detailed shape classification, such as determining the roundness of the hairline shape and classifying it into "square" or "round". (11) [Invention of claim 15] Since the hair shape is determined based only on the extracted hair region and hair characteristics, for example, when it is difficult to distinguish the hair region from the skin region such as gray hair, the hair shape is erroneously determined. Misjudgments such as "thin hair" were sometimes caused. (12) [Inventions of Claims 16 and 17] In an apparatus for generating a portrait by selecting a part, one type of hair part corresponds to one of the classified categories of hair shape, so that the front hair and the back hair are separated. It was difficult to generate harmonious and realistic hair shapes. A hair image formed by a method of combining a binarized hair region with a small part representing the hair of the bangs is often inferior in beauty as compared with a case where the entire hair part is created in advance, It was also difficult to harmonize with the rest of the face. Furthermore, since the binarized image of the hair region is used, there is a problem that if there is any part where the binarization processing is incomplete, the part is directly output to the user and gives a sense of incongruity. (13) [Inventions of Claims 18 and 19] In the conventional technology for automatically creating images such as portraits,
When arranging face parts such as nose / mouth / eyebrows / ears, place face parts in fixed positions, or determine the relative positional relationship of eyes / nose / mouth / eyebrows / ears on a real image. The arrangement was based on the reference. For this reason, in a face with a wide face, the eyes are shifted toward the center, and in a face with a narrow face, the eyes and eyebrows protrude from the face. Also, even with a wide face of Odeco, the eyes came to the top and the image was unnatural. This is because the fact that the arrangement of the face parts is determined by the shape of the face outline was ignored. You can also change the placement of parts according to age and gender, although this reflects the general tendency of face contours,
It did not directly reflect fine characteristics such as differences in face width and differences in eye position. (14) [Invention of Claim 20] Although a human face is composed of a plurality of face parts, it is normal for one person to have a very strong correlation between the face parts. As facial parts of portraits, there are colors such as black / brown / white for one hairstyle, and the variation is large depending on the person, but the color of the bangs and back hair of one person is different Is extremely rare.
However, since the forehead and the back hair are separate parts as the face parts, when the color of the forehead is changed to a certain color by the recognition or editing operation, if the back hair remains in a different color, an uncomfortable feeling occurs. Conversely, when the color of the back hair is changed, if the bangs have a different color, the same discomfort occurs. Similarly, an uncomfortable combination, such as a case where the forelock is perm and the back hair is straight, may cause discomfort. (15) [Invention of claim 21] One part data of a face part used for a portrait is composed of the part itself and a shadow part generated by the part. It is stored in data storage means. In the prior art, the components extracted by the face component data extraction means are synthesized as they are. For this reason,
For example, if a part of the bangs is drawn after drawing the parts of the eyes, the shadow of the bangs may be drawn on the eyes and the eyes may be crushed. This is because the shape of the shadow of the bangs is determined by the shape of the bangs, but ignores that the shadow itself is part of the face contour to which the shadow is projected. There is also a drawback that only the shadow is written even if there is no face on which the shadow is projected, such as a face outline. Further, as in the above document [11], there is a method in which parts are divided into layers and stored, and are synthesized for each layer. However, since one piece of data is divided into a plurality of parts, data management becomes complicated. And there is a disadvantage that the data capacity is required more than necessary.

SUMMARY OF THE INVENTION The present invention has been made to solve the above-described problems. It is an object of the present invention to take a long time to perform image processing for extracting a feature amount of a specific object in an image. Any feature amount in an image can be robust and high without extracting erroneous feature amounts, and without specifying many positions when directly specifying the range of processing using position designation means. It is an object of the present invention to provide an image processing apparatus that can accurately and quickly extract and easily synthesize a high-quality portrait.

[0026]

According to a first aspect of the present invention, there is provided an image processing apparatus comprising: an input unit for inputting an image; a storage unit for storing the input image; Calculating means for performing, position specifying means for specifying an arbitrary position in the image, and recognizing the position and size of the object arranged in the image, and determining the relationship between the position and size of the object. When a certain constraint condition is satisfied, a feature extraction amount means for extracting an arbitrary feature in the image by inputting a position of one or more of the objects in the image is provided. .

According to the above arrangement, when the relationship between the position and the size of the object arranged in the image satisfies a certain constraint, one or more positions in the image are input, and the input position As a result, it is possible to set an appropriate search range for performing image processing for extracting a feature amount of a target object.

That is, in the conventional method, in order to set the search range, for example, if there are three objects, at least six points have to be specified to set the search range for each object. By using the image processing device, it is possible to set an appropriate search range with fewer designated points. [Claim 2] An image processing apparatus according to claim 2 of the present invention is
2. The image processing apparatus according to claim 1, wherein, when the input image includes a face, an eye, a nose, a mouth, an eyebrow, an ear, an outline, and a hair constituting the face are face parts. A position, a size, and a position of at least one corresponding face part among the face parts;
It is characterized in that a shape is extracted as a feature amount.

According to the above construction, in addition to the operation of the first aspect, the positions of face parts such as eyes, nose, mouth, eyebrows, ears, contours, hair, etc.
Features such as size and shape can be extracted robustly and with high accuracy.

In the present image processing apparatus, for example, when the positions of the right eye, left eye, and mouth of a person included in the input image are extracted as feature amounts, two points of the right eye and the left eye are extracted by using the above-mentioned position specifying means. Is specified. Here, the position and size of the right eye, the left eye, and the mouth follow a certain constraint, that is, the size of the left and right eyes is almost the same, which is obtained by multiplying the distance between the eyes by a certain coefficient. The mouth is located vertically below the center of the line connecting the eyes, and the distance and the size of the mouth are larger than the value obtained by multiplying the distance between the eyes by a certain coefficient. Values that are not far apart.

That is, in the prior art, six points have to be designated by the position designation means in order to limit the search range so that an erroneous value is not extracted in the feature extraction processing. The same effect can be obtained only by designating two points. [Claim 3] An image processing apparatus according to claim 3 of the present invention is
3. The image processing apparatus according to claim 2, wherein the feature amount extracting unit binarizes the input image using a plurality of methods and a plurality of thresholds, and determines a position, a size, and a shape of a region in the images, It is characterized by comprising an area detecting means for detecting an area to be recognized by selecting an image having the highest reliability.

In the above configuration, when determining a threshold for performing binarization to obtain a region to be recognized, binarization is performed using a plurality of thresholds by a plurality of methods, and a plurality of thresholds obtained as a result are obtained. By comparing the area with the position, shape, size, etc. of the recognition target whose outline is known in advance, and making the area closest to the position, shape, size of the recognition target the recognition target, the recognition target area is robust and It can be extracted with high accuracy. According to a fourth aspect of the present invention, there is provided an image processing apparatus according to the second aspect, wherein the feature amount extracting means is provided with two or more specified by the position specifying means. A face part recognizing means for detecting the position and the size of the face part by estimating the size of the face part from the distance relationship between the positions of the face parts.

An image processing apparatus according to a fifth aspect of the present invention comprises:
5. The image processing apparatus according to claim 4, wherein the face part recognition unit detects the target of the position and the size to be detected is an eye of the face part.

An image processing apparatus according to claim 8 of the present invention provides:
5. The image processing apparatus according to claim 4, wherein the face part recognizing means detects the position or size of the face part by the mouth of the face part.

With the above configuration, the size of the face part is predicted with high accuracy, the detection process is performed only within a necessary and sufficient range in the image, and the position and size of the face part are detected with high accuracy with a small amount of calculation. Can do it. [Claim 6] An image processing apparatus according to claim 6 of the present invention comprises:
6. The image processing apparatus according to claim 4, wherein said face part recognizing means includes means for rotating a face image such that a line connecting left and right eyes is horizontal with respect to the detected positions of both eyes. It is characterized by.

According to the above configuration, the position and size of a face part can be detected with high accuracy even when an image with a tilted face is given as an input. [Claim 7] An image processing apparatus according to claim 7 of the present invention is
The image processing apparatus according to claim 4, wherein the face part recognizing unit sets a search range based on the detected position and size of the eyes, and displays an inclination and a thickness of the eyes in the range. It is characterized by comprising means for detecting a feature and determining an eye shape.

According to the above arrangement, an image feature representing the inclination and the thickness of the eyes is detected within the set search range,
Since the shape of the eyes is determined, it is possible to avoid a risk that an erroneous feature amount is extracted due to a shift between the template image or the dictionary image and the corresponding partial image in the input image. Furthermore, when the target eye shape is a shape that is not included in the category prepared in advance, it is possible to avoid a danger that a correct feature amount cannot be calculated. Furthermore, it is necessary to prepare a template image and a dictionary image, and the amount of work can be significantly reduced. [Claim 9] An image processing apparatus according to claim 9 of the present invention is characterized in that:
5. The image processing apparatus according to claim 4, wherein the position and size of the face part recognition means are eyebrows of the face part.

According to the above arrangement, the positions of two or more face parts are specified by the position specifying means, and the size of the eyebrows is predicted from the distance relationship between the specified positions, thereby extracting a feature amount. The range in which processing is to be performed can be limited to an appropriate size.

Then, binarization is performed on the processing range. In order to solve the above-described problem, the image processing apparatus performs a plurality of binarizations to obtain a region to be recognized. Method, binarizing with a plurality of thresholds, determining the position, size, shape, and the like of the region in those images, and selecting the most reliable image to detect the recognition target with high accuracy. It is characterized by:

According to this configuration, the position of two or more face parts is designated by the position designation means, the size of the eyebrows is predicted from the distance relationship between the designated positions, and the processing for extracting the characteristic amount is performed. In addition to limiting the range to be performed to an appropriate size, the size of the eyebrows can be estimated. That is, for example, when the face parts whose positions are specified by the position specifying unit are the right eye and the left eye, the size of the eyebrows is not a value greatly separated from a value obtained by multiplying the distance between the eyes by a constant coefficient. can do.

Therefore, the size of the region separated when binarization is performed is compared with the estimated size of the eyebrows.
By obtaining a binarization threshold value such that their sizes are not so far apart, an area representing an eyebrow can be detected with high accuracy. [Claim 10] An image processing apparatus according to claim 10 of the present invention is the image processing apparatus according to claim 4 or 9, wherein
The face part recognizing means sets a search range based on the detected position and size of the eyebrows, detects image characteristics representing the thickness and how to bend the eyebrows in the range, and determines the shape of the eyebrows. It is characterized by comprising.

According to the above arrangement, the image feature representing the thickness and the way of bending of the eyebrows is detected within the set search range, and the shape of the eyebrows is determined. It is possible to avoid the danger that an erroneous feature amount is extracted due to a deviation from the partial image in the input image to be performed. further,
In the case where the target eyebrow shape is a shape that is not included in the category prepared in advance, it is possible to avoid a risk that a correct feature amount cannot be calculated. Furthermore, it is necessary to prepare a template image and a dictionary image, and the amount of work can be significantly reduced. [11] The image processing apparatus according to claim 11 of the present invention, in the image processing apparatus according to claim 2, wherein the feature extracting means is based on one or more position information specified by the position specifying means. And a contour recognizing means for detecting a contour feature of a jaw and determining a shape thereof. In other words, a feature image representing the features of the chin contour is created more prominently from information characterizing one or more faces obtained from the position specifying means, and the contour is detected by a dynamic contour model using the image as image energy. And expressing the detected contour as a distance function consisting of a distance and a direction from the value portion of the face, calculating the feature of the distance function, and comparing the reference feature to obtain a jaw contour. The shape is determined.

Therefore, the operator of the image processing apparatus first specifies the center of the face of the person included in the input image by the position specifying means. The face center may be specified directly, or may be estimated from the specification of other facial features, for example, the coordinates of both eyes and mouth. Next, an initial outline coordinate sequence including the face of the person is obtained. Next, a color difference between adjacent pixels on a straight line connecting the face center coordinates and each coordinate on the initial contour is calculated, a coordinate middle point between the target pixels is set as a coordinate value, and an image having the calculated color difference as a pixel value (hereinafter, referred to as an image) , A color difference map image). Next, a jaw contour line is detected using an active contour model using this color difference map image as image energy. Next, the obtained contour is expressed as a function (hereinafter, referred to as a distance function) composed of the coordinates of the value inside the face, for example, the distance from the face center and the direction (angle). Next, the feature of the distance function is compared with the feature of the distance function of the reference contour shape, and the contour shape having the closest distance function is determined as the jaw shape of the input image.

The accuracy of the color difference map image creation may be enhanced by utilizing the fact that the image is a human face of the target image. For example, when obtaining the color difference, the skin color and the other colors may be determined separately. In other words, the color difference between pixels classified as skin color is reduced by lowering the color difference detection accuracy.
The effects of noise and wrinkles can be made less likely to be reflected in the color difference map image. Conversely, the border between the neck and the chin often has the same flesh color, and it is difficult to produce a color difference. Therefore, when detecting a color difference on a straight line from the center to the neck, the detection accuracy may be increased. Note that the direction of the neck position can be estimated if the coordinates of the mouth are the values.

After the color difference map image is created as described above, for example, by assuming an ellipse as the face outline, the pixel value (= color difference) on the ellipse coordinate centered on the face center coordinate and the pixel values on both sides thereof Are averaged to obtain the pixel value. Alternatively, when other features other than the face outline are known separately, for example, if the center coordinates of the mouth are the values, a line having the straight line connecting the center coordinates of the mouth and the center of the face on the symmetry axis is used.
The pixel values of the pixels may be averaged and used as the pixel value.
This makes it possible to create an energy image in which the features of the jaw shape are taken into account, and it is possible to perform more stable jaw detection even on an input image in which a clear outline does not appear or an image with much noise.

Also, when a distance function is created from a contour line, the influence of noise and illumination is eliminated as much as possible by using the unique characteristics of the contour of the human face, so that the jaw features are more remarkably represented. To modify the distance function. For example, a distance function such as averaging can be corrected based on the shape of a face such as an ellipse or symmetry as in the case of creating a color difference map.

Next, the distance function is compared with features of the distance function such as the position of the inflection point of the distance function, the number of inflection points, and the slope between the inflection points. This is done by comparing each with the feature. And
The reference shape having the most similar reference distance function is determined as the corresponding contour shape.

Further, if a distance function is represented by a method of describing a curve on a plane in the frequency domain, for example, by using a Fourier descriptor, the Fourier coefficient calculated thereby can be positioned as a feature of the distance function. By comparing with the coefficient of the distance function of the reference contour shape,
Shape determination can be performed in the same manner as described above.

The feature of the reference distance function to be compared is that the distance function may be normalized beforehand and stored in a memory as a table, or only the information such as the required positions of the inflection points which are required in advance may be used. May be stored. When a Fourier descriptor is used, a coefficient of a required order may be stored. These methods do not need to have a reference shape to be compared as a dictionary image as compared with template matching, which is advantageous in terms of memory cost and processing speed.

When the Fourier descriptor is used, the fact that the lower-order terms of the Fourier coefficients reflect a rough curve shape and the higher-order terms reflect a more detailed curve shape is used. By comparing the terms (1) and (2), it is possible to obtain a determination result in which effects such as noise and individual differences are eliminated as much as possible. According to a twelfth aspect of the present invention, in the image processing apparatus according to the second aspect, the feature extracting means includes at least one position information designated by the position designation means. And a hair recognizing means for estimating the height of the crown and the height of the hairline on the basis of the above, and recognizing the hair region.

According to a thirteenth aspect of the present invention, in the image processing apparatus according to the twelfth aspect, the hair recognizing means includes a hair color extracting means for extracting a hair color. .

According to the above configuration, it is not necessary to extract the background region from the entire image, so that the background color does not need to be uniform or close to it, and the hair color can be extracted even from a normal snapshot, etc. Can be created. [14] The image processing apparatus according to claim 14 of the present invention, in the image processing apparatus according to claim 12, wherein the hair recognizing means is based on one or more position information specified by the position specifying means. A hair feature extracting means for extracting a feature of the hair part, a hair contour extracting means for extracting a hair contour using the feature of the hair part, and a hair classifying means for classifying the hair using the hair contour. It is characterized by comprising.

According to the above-described configuration, instead of using template matching, the hair contours are extracted, so that the "separation" referred to by the so-called "seven-three division" or "center division" is accurately detected. Fine shape classification can be performed, for example, by determining the roundness of the shape of the hairline, and by classifying it into “square” or “round”. [15] The image processing apparatus according to claim 15 of the present invention, in the image processing apparatus according to claim 12, wherein the hair recognizing means extracts a face contour feature, and a hair feature. And a hair classification means for classifying the hair using the facial contour feature.

According to the above arrangement, for example, when the height of the top of the face contour line is lower than the head height by a certain threshold or more, it is determined that there is a considerable amount of hair, and the hair area such as white hair is determined. Even when it is difficult to distinguish between a skin region and a skin region, it is possible to eliminate or reduce erroneous judgments such as "thin hair". [16] The image processing apparatus according to claim 16 of the present invention, in the image processing apparatus according to claim 12, wherein the hair recognizing means is a bangs feature extracting means for extracting features of a bangs part, and a back hair part. And a back hair feature extracted by the bangs feature extraction unit and a back hair feature extracted by the back hair feature extraction unit when an image including a hair portion is input. Are used to determine bangs parts.

According to the above configuration, the bangs part is determined by using the characteristics of both the bangs and the back hair. For example, if there is a division on the left side in the upper part of the hair, even the bangs part flows from the left side. By selecting an item that matches the "left division" as described above, a more realistic portrait with less discomfort can be created. Further, since a hair part prepared in advance is used, the hair area is converted to a binarized image, for example, by a method of combining a binarized hair area with a small part representing the hair of the bangs. In many cases, a more beautiful portrait can be created as compared with a method in which part or all of the hair image is output as it is as part or all of the hair image. Is not directly visible to the user as output, so it is difficult to give a sense of incongruity. [17] The image processing apparatus according to claim 17 of the present invention, in the image processing apparatus according to claim 12, wherein the hair recognizing means is a bangs feature extracting means for extracting features of a bangs part, and a back hair part. And a back hair feature extracted by the bangs feature extraction unit and a back hair feature extracted by the back hair feature extraction unit when an image including a hair portion is input. Is used to determine the back hair part.

According to the above configuration, the back hair part is determined by using both characteristics of the bangs and the back hair. For example, if the front hair part of the forehead has a division on the left side, the back hair part is also determined on the left side. By selecting an item that matches the "left division" in which there is a division, a more realistic or less unnatural portrait can be created. Further, since a hair part prepared in advance is used, the hair area is converted to a binarized image, for example, by a method of combining a binarized hair area with a small part representing the hair of the bangs. In many cases, a more beautiful portrait can be created as compared with a method in which part or all of the hair image is output as it is as part or all of the hair image. Is not directly visible to the user as output, so it is difficult to give a sense of incongruity. [Claim 18] An image processing apparatus according to claim 18 of the present invention has means for obtaining face part feature information of the size and shape of a face part in an image and a plurality of face part types corresponding to the feature information. A face part data storage unit storing a plurality of part data for each face part type; and a face part data extraction unit for extracting appropriate part data from the face part data storage unit based on the face part characteristic information. Means for arranging another face part at a position suitable for the contour by determining the arrangement position of the part for each face outline part type stored in the face part data storage unit with each of the extracted face part data. It is characterized by comprising means.

With the above configuration, after determining which face part to use, such as the size and shape of the face part in the image,
A part arrangement method such as a part arrangement position and a part size is determined for each face contour part in the face parts, and each piece of face part data is arranged. By arranging the parts based on the face outline, not only the parts do not protrude from the face but also the face parts can be arranged at the position and size most suitable for the shape of the face outline. Determining facial part placement information for each facial contour to generate a facial caricature without disintegration when facial parts are placed, even if the facial balance is completely different due to the concept of portraits such as dramatic and comic styles Is possible. [19] The image processing apparatus according to claim 19 of the present invention, in the image processing apparatus according to claim 18, means for obtaining information of the position of the face part in the image, and the obtained position information of the face part. A means for correcting the arrangement position of another face part determined in accordance with the face outline based on the face contour to determine the arrangement position of the face part.

According to the above configuration, all variations of the face having the same face contour but having slightly different positions of the face parts are supported. Therefore, another face determined based on the position information of the face parts in the image is used. The arrangement position of the face part is corrected within the allowable range of the face outline, and a more appropriate part arrangement position is determined. Further, the position of the face part extracted from the input image is compared with a statistical standard position in the face contour of the face part, and the difference and the correction allowable range determined for each face contour determine the arrangement of the face part. Is corrected. This makes it possible to reflect the delicate feature of the position of the face part of the input image on the portrait while avoiding a failure such that the face protrudes from the face contour. [Claim 20] The image processing apparatus according to claim 20 of the present invention is the image processing apparatus according to claim 18, wherein the face part data arranged at the time of caricature composition relates to a specific color of a set of related parts. It is characterized in that, when a change of a part is designated, there is provided means for automatically generating a data which does not cause inconsistency by automatically changing the part with respect to other related parts.

According to the above configuration, when the color of one part is determined to be any of black / brown / white, for example, between parts having a strong correlation between face parts such as bangs and back hair or bangs and mustache. By automatically changing the color of the other part to the same color, without explicitly specifying the color change of the other part,
Automatically generate a portrait with no discomfort. Similarly, for a shape that has a strong correlation between the front hair and the back hair, such as whether the hair is perm, an editing operation without a sense of incongruity can be performed by reflecting an instruction for one on the other. [21] The image processing apparatus according to claim 21 of the present invention, in the image processing apparatus according to claim 18, wherein one of the face part data stored in the face part data storage unit is one of the face part data. Has more than one layer structure,
When combined with other facial part data, means for changing the order of layers based on the color information in the part and determining the arrangement order of the part and the layers constituting the part in an appropriate order is provided. Features.

According to the above configuration, in each pixel or region of a face part or a part forming a portrait such as a hat, a pixel or a pixel determined to constitute a shadow projected on a face contour from its color information. By extracting a region, drawing pixels or regions in advance, and then drawing individual face parts, it is possible to prevent a shadow of another part from being placed on a certain part. Further, when there is no component to which a shadow is projected, by stopping the drawing of the pixel or the region itself, drawing at a place where no face outline exists can be prohibited.

[0061]

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS An embodiment of an image processing apparatus according to the present invention will be described below with reference to the drawings. First Embodiment [First Embodiment] (Inventions of Claims 1 and 2) FIG. 1 is a diagram showing functional blocks of an image processing apparatus according to the present embodiment.

The image processing apparatus according to the present embodiment determines the feature amount of an object included in an image input from the outside, that is, an image acquired from the outside of the apparatus (hereinafter referred to as an input image), for example, the position of the right eye. , A device that performs extraction processing from the input image. That is, as shown in FIG. 1, an input device 11 as an image acquisition unit for acquiring an electronic image, and an input device 1
Storage device 1 for storing the input image and the like acquired by
2, an arithmetic unit 13 for performing a process for extracting a feature from the input image, a position specifying device 14 as a unit for specifying an arbitrary position of the input image, and an output device for outputting the extracted feature to the outside 15 are provided. Note that the feature amount extraction unit 10 for extracting the feature amount of the object includes a storage device 12
And an arithmetic unit 13.

In this embodiment, image processing executed by the image processing apparatus having the above configuration will be described. Here,
The input image contains a face, and the extracted features are eyes, nose,
The case where the position, size, shape, and the like of face parts such as the mouth, eyebrows, ears, contours, and hair are described.

First, an image is input from the input device 11.
The operator uses the position specifying device 14 while viewing the input image displayed on the output device 15 such as a display device,
Specify an arbitrary position on the input image.

FIG. 2 is a diagram showing an example of position designation and an example of a search range determined by the position designation. This is an image showing that the right eye position 22, the left eye position 23, and the mouth position 24 of the input human image 21 are specified. Right eye position Pre, left eye position Ple, mouth position P
Assuming that m, a range 25 in which image processing is performed to extract a feature amount related to the right eye (hereinafter, referred to as a right eye search range)
Is a width Wre and a height Hr around the right eye position Pre.
This is represented by a rectangle e.

Here, the width Wre and the height Hre are determined as follows. In other words, the positional relationship and the size of each face part in the human face follow a certain constraint.
That is, it is considered that a certain ratio exists between the distance between the eyes and the size of the eyes. Assume that the size of the eyes is expressed by a width Weye and a height Heye of a rectangle circumscribing the eyes.
Each is represented as follows.

Weye = Cew × L Heye = Ceh × L where Cew is the eye width Wey with respect to the distance L between the eyes.
In the average value of the ratio e, Ceh is the average value of the ratio of the eye height Heye to the distance L between the eyes. These coefficients Cew and Ceh are measured in advance for a plurality of human faces with respect to the distance between the eyes and the width and height of the eyes, and the average value is obtained. Due to individual differences, the width and height of the eyes will vary, so by multiplying this by an appropriate coefficient,
For almost all persons, a rectangular width Wr representing a sufficiently large search range such that the right eye is included therein
e and the height Hre can be determined as follows.

Wre = Mew × Weye Hre = Meh × Heye Here, Mew and Meh are obtained in advance for a plurality of persons from the variance values measured for the distance between the eyes and the width and height of the eyes.

The above process can be similarly applied to the left eye, mouth, etc. That is, the search range of the left eye and the search range of the mouth can be similarly obtained. Here,
Although the search range is determined based on the distance L between the eyes, another reference may be provided. For example, the search range may be determined based on the distance between the center of the line connecting both eyes and the line connecting the position of the mouth, that is, the height of the eye and the mouth.

As described above, by using the position specifying device to input the position of an object whose position and size comply with a fixed constraint, the feature amount of the object can be extracted with a small number of input positions. The search range can be set appropriately.

Therefore, the features of the present embodiment are summarized as follows. (1) An image processing apparatus according to claim 1, wherein input means for inputting an image, storage means for storing the input image, operation means for performing an arbitrary operation, and an arbitrary position in the image are designated. Position specifying means capable of recognizing the position and size of an object placed in the image, and when the relationship between the position and size of the object satisfies a certain constraint, one or more of And a feature extraction amount means for extracting an arbitrary feature in the image by inputting the position of the object.

According to the above configuration (1), when the relationship between the position and the size of the object arranged in the image satisfies a certain constraint, one or more positions in the image can be input. It is possible to set an appropriate search range for performing image processing for extracting a feature amount of a target object from the input position.

That is, in the conventional method, in order to set a search range, for example, in the case of three objects, at least six points had to be specified in order to set a search range for each of the three objects. By using the image processing device, it is possible to set an appropriate search range with fewer designated points. (2) In the image processing apparatus according to the second aspect, when the feature extraction unit includes a face in the input image, eyes, nose,
The mouth, eyebrows, ears, contour, and hair are face parts, and the position, size, and shape of at least one of the face parts are extracted as feature amounts.

According to the above configuration (2), in addition to the operation of the above (1), features such as the position, size and shape of facial parts such as eyes, nose, mouth, eyebrows, ears, contours, hair, etc. The quantity can be extracted robustly and with high precision. For example, when the positions of the right eye, left eye, and mouth of a person included in the input image are extracted as the feature amounts, two positions of the right eye and the left eye are extracted by using the position designating means.
Specify a point. Here, the position and size of the right eye, the left eye, and the mouth follow a certain constraint, that is, the size of the left and right eyes is almost the same, which is obtained by multiplying the distance between the eyes by a certain coefficient. The mouth is located vertically below the center of the line connecting the eyes, and the distance and the size of the mouth are larger than the value obtained by multiplying the distance between the eyes by a certain coefficient. Values that are not far apart.

That is, in the prior art, six points must be designated by the position designation means in order to limit the search range so that an erroneous value is not extracted in the feature extraction processing. The same effect can be obtained only by designating two points. [Second Embodiment] (Inventions of Claims 3, 4, and 5) This embodiment will be described with reference to FIGS. FIG. 17 is a flowchart showing the eye detection operation.
FIG. 18 is an example of various images used for eye detection processing (results of detection, conversion, extraction, and the like).

First, the user specifies the approximate positions of both eyes and the mouth in the image with a pointing device (position specifying device in FIG. 1) such as a pen or a mouse. Next, an image near a location specified by the user is cut out.

The calculation method of the range to be cut out is as follows.

First, the distance el between the approximate positions of both eyes specified by the user is calculated. The obtained distance el is multiplied by predetermined constants EW and EH to determine the width ew and height eh of the region to be cut out (100). Next, an image is cut out at the determined width and height with the approximate positions of both eyes specified by the user as the centers (101). still,
This cut-out image is hereinafter referred to as an eye periphery image.

Next, a method of obtaining an area of an eye in an image around the eye as a recognition target, that is, a method of obtaining a recognition target area by binarizing an input image will be described.

First, the image around the eyes is converted into a luminance image (111) as shown in FIG. 18 (102). Subsequently, the threshold value th1 is determined by applying the discriminant analysis method to the luminance image. Next, a predetermined value is added to or subtracted from the threshold value th1,
The thresholds th2 to th5 are determined based on the following equation (1
03).

Th2 = th1-20 th3 = th1-10 th4 = th1 + 10 th5 = th1 + 20 In this embodiment, the threshold value is determined by the above method. However, it is not possible to use or use another threshold value determination method. Easy. Next, the threshold values th1 to
Using th5, binarized images Img1 to Img5 are obtained (104). Reference numeral 142 in FIG. 18 is an example of a binarized image obtained by the above method. In this example, the eye portion to be recognized is black, but there is also a black portion at the upper left, which indicates that only the region to be recognized cannot be separated. Of course, there are cases where the recognition target area can be correctly separated by the above method.

Next, the input image is converted into a differential image (1
05). For how to generate differential images, see Sobel
Various methods are known, such as a method using an operator, and any of them may be used. (These methods are methods that can be easily realized by a person skilled in the art.) Therefore, a detailed description will not be given here.) Subsequently, thresholds th6 to th10 are determined in the same manner as that performed for the luminance image for the differential image (10
6) Binarization is performed to obtain binarized images Img6 to Img10 (107).

FIG. 18 shows an example of a binarized image 113 obtained by the above method. In this example, the eye portion to be recognized is white, and there is another small white portion. However, the region to be recognized is almost separated. Of course, there are cases where the recognition target area cannot be correctly separated by the above method. In addition to the above method, there are various methods for binarizing an input image, such as a method using color information, and it is easy to apply the method to the present invention. In this embodiment, the variation of the threshold value in each system and the number of binarized images generated thereby are set to "5". However, this may be an arbitrary number. It is also easy to carry out the change.

Subsequently, a region which is considered to be a recognition target in each binarized image is obtained. In the present embodiment, since the recognition target is the detection of the eye region in the face image, it is a dark region or a region with a high differential value in the luminance image. Therefore, in the binarized images Img1 to Img5, the pixel value is 0, that is, a black pixel, and in the binarized images Img6 to Img10, the pixel value is 1, that is, a white pixel is a pixel considered to be a recognition target. Each of the binarized images I
A process of extracting only a continuous region having the largest area from mg1 to Img10 is performed, and the remaining regions are set as recognition target candidates Area1 to Area10, respectively (108).

In FIG. 18, reference numeral 114 denotes a result obtained by extracting only the continuous area having the maximum area from 113 in FIG. 18. Reference numeral 115 in 113 denotes a continuous area having the maximum area. It is an example. In 114, a small white area that was not the eye area that was in 113 has disappeared,
The recognition target area has been correctly separated. Here, information known in advance regarding the recognition target is used, and an area that seems to represent the recognition target best is selected from the recognition target candidates Area1 to Area10.

Using the statistical properties of the distance between the eyes and the size of the eyes in the face, the expected eye size is calculated from the approximate position of the eyes specified by the user.
Specifically, the distance between the approximate positions of the eyes specified by the user is calculated, and the distance between the eyes is multiplied by the average value of the ratio of the eye size, which is obtained in advance. The size is determined (109).

Next, recognition target candidates Area 1 to Area 1
Find the magnitude of zero. In this embodiment, the size of the circumscribed rectangle is set as the size of the recognition target candidates Area1 to Area10. Reference numerals 144 and 145 in FIG. 18 indicate the sizes of examples 115 of recognition target candidates. Next, each recognition target candidate Area1
The size of the eye is compared with the expected eye size to find the closest one, and the recognition target candidate having the closest size is used as the recognition target detection result (110). In the example 115 of the recognition target candidate, the recognition target is correctly separated, and has a size equal to the size of the recognition target. Therefore, the sizes 144 and 145 of the example 115 of the recognition target item are values close to the predicted eye size. On the other hand, examples of recognition target candidates 143
Does not correctly separate the recognition target, so its size is not close to the predicted eye size. Therefore, in this example, the example 115 of the recognition target candidate that correctly separates the recognition target is selected, and the eyes are recognized stably and with high accuracy.

Therefore, the characteristics of this embodiment are summarized as follows. (3) In the image processing apparatus of the third aspect, the feature amount extraction unit binarizes the input image with a plurality of methods and a plurality of thresholds,
Determine the position, size and shape of the area in those images,
It is characterized by comprising an area detecting means for detecting an area to be recognized by selecting an image having the highest reliability. Although the area detecting means is not shown, the arithmetic unit 1
3 or the feature extraction unit 10 realizes the function.

According to the above configuration (3), when determining a threshold value for performing binarization in order to obtain an area to be recognized, binarization is performed using a plurality of threshold values by a plurality of methods. By comparing the obtained multiple regions with the position, shape, size, etc. of the recognition target whose outline is known in advance, the region closest to the position, shape, and size of the recognition target is set as the recognition target. The target region can be extracted robustly and with high accuracy. (4) The image processing apparatus according to (4), wherein the feature amount extracting means predicts the size of the face part from the distance relationship between the positions of the two or more face parts specified by the position specifying means, thereby obtaining the face part. And a face part recognizing means for detecting the position and size of the face part. Although the face part recognizing means is not shown, it is included in the arithmetic unit 13 or the feature amount extracting unit 1.
0 implements that function. (5) An image processing apparatus according to a fifth aspect is characterized in that the target of the position and the size detected by the face part recognizing means are eyes of the face part.

According to the above configurations (4) and (5), the size of the face part is predicted with high accuracy, the detection process is performed only within a necessary and sufficient range in the image, and high accuracy is achieved with a small amount of calculation. Then, the position and size of the face part can be detected. This can be effectively applied to the eyes. [Third Embodiment] (Invention of Claim 6) This embodiment will be described with reference to FIGS. FIG. 19 is a flowchart of correcting the face inclination using the eye detection result, and FIG. 20 is an example of a screen in which the face inclination is corrected based on the eye detection result.

First, both eyes are detected (116). Next, the center of each of the detected eyes is calculated using the following equation. For the center of both eyes, the average of the position coordinates epu, epd of the upper and lower ends of the detected circumscribed rectangle of both eyes is the vertical center position epy, and the average of the left and right end position coordinates epl, epr is the horizontal center position epx. (117).

Epy = (epu + epd) / 2 epx = (epl + epr) / 2 Reference numerals 121 and 122 in FIG. 20 denote examples of a face image with a tilted face, and reference numeral 124 denotes an eye image of the face image with a tilt in the above method. Is detected, the center position of both eyes is calculated, and the center position of both eyes is connected. Subsequently, the image is rotated so that the line connecting the centers of both eyes is horizontal. Specifically, in order to calculate the angle of a line connecting the centers of both eyes, a vector of a line connecting the center positions of both eyes is obtained (118). Assuming that the magnitude of this vector is (x, y), the angle of the vector is obtained by the following equation (11)
9).

K = atan (y / x); k is the inclination of a line connecting the center positions of both eyes,
That is, it is also the inclination of the entire face in the image. Next,
Rotate k degrees (120). The method of rotating the image is not specifically described since it is considered that a person skilled in the art can easily realize this. By this rotation, the line connecting the centers of both eyes is corrected to be horizontal, and the face and facial parts in the image are also corrected for tilt. 12 in FIG.
Reference numeral 3 denotes an example of the image after the rotation, and reference numeral 148 denotes a line connecting the center positions of both eyes in the rotated image.
According to this, it is understood that the line connecting the center positions of both eyes is horizontal, and that the face in the image is also corrected in inclination and stands upright. Since the inclination is corrected by the above method,
Even when an inclined face is input, in the subsequent processing, processing such as recognition may be performed on the image whose face inclination has been corrected, so that stable and highly accurate recognition can be performed as compared with the conventional method.

Therefore, the features of this embodiment are summarized as follows. (6) The image processing apparatus according to claim 6, further comprising means for rotating the face image such that a line connecting the left and right eyes is horizontal with respect to the positions of both eyes detected by the face part recognizing means. And Since this means is provided in a face part recognizing means (not shown), its function is realized in the arithmetic unit 13 or in the feature quantity extracting unit 10 like the face part recognizing means.

According to the above configuration (6), the position and size of a face part can be detected with high accuracy even when an image with a tilted face is given as an input. [Fourth Embodiment] (Invention of Claim 7) This embodiment will be described with reference to FIGS. FIG. 3 is a diagram illustrating an example of an eye search range for recognizing an eye shape, a histogram, and detected inner and outer corners of the eye.
FIG. 4 is a diagram showing an example of a template for detecting the inner corner of the eye. FIG.
FIG. 6 is a flowchart illustrating an example of a non-skin region, and FIG. 6 is a flowchart illustrating an operation for obtaining an eye opening search range and an eye corner search range.

A method for determining the shape of the left eye will be described with reference to FIGS.

The search range 31 is set so as to include the left eye therein, and the positions of the inner and outer corners of the eye are detected. This can be achieved, for example, by the following method. In the case of the human eye, the eye is configured such that the upper and lower eyelids meet. Therefore, the point where the boundary between these two eyelids meets can be the inner or outer corner of the eye.

The image within the search range is differentiated in the vertical direction (S
61), and the histogram is created by projecting the differential value in the vertical direction (S62). Reference numeral 32 schematically represents a histogram created by differentiating the image 31 in the vertical direction and projecting the differential value in the vertical direction. Then, the histogram is scanned from left to right (S63). Since the differentiation in the vertical direction extracts the horizontal component of the boundary line between the two eyelids, when it is projected in the vertical direction, the following features will be exhibited. That is, when scanning from left to right, the histogram is flat because there is no eye at the left end in the search range and it is a skin portion. Looking to the right, when reaching the inner corner of the eye, the upper and lower eyelids start from there, so the histogram rises sharply. And
When reaching near the outer corner of the eye, the histogram starts to fall again.

Therefore, when the histogram is scanned from left to right within the eye search range, the point where the histogram rises sharply is likely to have the inner corner of the eye. Here, an appropriate threshold value is provided, and when the amount of change in the histogram rise exceeds the threshold value (S64), an appropriate range before and after that point is set as a search range 33 for detecting the next inner corner (S65). ). Similarly, if an appropriate threshold value is set and the amount of change in histogram descending exceeds the threshold value (S66),
An appropriate range before and after that is set as a search range 34 for next detecting the outer corner of the eye (S67). However, in the case of the outer corner of the eye, there are many cases where there is no clear boundary as compared with the inner corner of the eye, and it is desirable to set the search range wider than that of the inner corner of the eye.

After the search range of the inner corner is set, the position of the inner corner is detected within the search range. This includes
For example, it can be realized by the following method. A small template for detecting the inner corner (hereinafter, referred to as the inner corner template) is set. In the example of FIG. 4, the size of the inner corner template is 4 pixels × 4 pixels, and three types are prepared according to the shape of the inner corner. In the case of an eye like 44 in which the inner corner is lowered, an inner eye template such as 41 is used, and in the case of an eye such as 46 whose inner corner is raised, an inner eye template such as 43 is used. In the case of a simple eye, an inner corner template such as 42 is used.

The above-mentioned inner-eye template is moved within the inner-eye search range (S71), and the similarity between the image and the corresponding image is calculated (S72). The similarity S is defined, for example, by the following equation.

[0102] Here, W is a white area of the inner template, B is a black area of the inner template, I (p) is a luminance value at the pixel p, N (W),
N (B) is the number of pixels in the white and black regions of the inner corner template, respectively. The similarity S is calculated in the entire inner-corner search range (S73), and the point having the highest similarity S is set as the inner-corner position 35 (S74).

After the search range of the outer corner of the eye is set, the position of the outer corner of the eye is detected within the search range. Since the border of the outer corner of the eye is often more ambiguous than the inner eye, a template-based method such as the inner eye may not work well. For this, for example, a method of finding the center of gravity within the search range of the outer corner of the eye (S81) is used. Center of gravity (g
x, gy) is defined, for example, by the following equation.

[0104] Here, R is a region representing the search range of the outer corner of the eye, X (p)
Is the X coordinate of point p, and Y (p) is the Y coordinate of point p. The center of gravity (gx, gy) is set as the outer corner of the eye 36 (S82).

When the inner and outer eye positions 35 and 36 are detected, the inclination 37 of the eyes can be obtained.

Next, the thickness of the eyes is determined. In order to determine the eye thickness, for example, the following method is used. Separate the skin and non-skin regions within the eye search range. First, the skin color within the eye search range is analyzed. In the search range image 50 of the eye in FIG. 5, the color distribution of an area apparently considered to be skin, for example, the outer edge 51 of the search range image is examined. Specifically, it is assumed that the colors of the pixels constituting the human skin follow a normal distribution with a certain color as an average,
The average and variance of the colors of the pixels in the outer edge 51 are obtained. A probability density function representing the color of the skin is obtained from the average and the variance, and the function is applied to the eye search range image, whereby the skin and non-skin regions can be separated. This technique uses the technique described in the above-mentioned reference [4] or reference [12].

Here, the area 52 shown in black in the schematic diagram 55 schematically shows an area separated as a non-skin area. This includes most of the pixels that make up the eye. When this area is projected in the horizontal direction and in the direction rotated forward and backward, the range in which the area exists is determined (53, 54, 55).
The minimum length of the range is defined as the thickness of the left eye.

Thus, the inclination and the thickness of the eyes are measured,
Based on these two parameters, an eye shape code corresponding to a preset category can be obtained.

Although the processing for the left eye has been described above, for the right eye, exactly the same processing can be applied by inverting the search range image horizontally.

Therefore, the characteristics of this embodiment are summarized as follows. (7) The image processing apparatus according to claim 7 sets a search range based on the position and size of the eyes detected by the face part recognition means,
It is characterized by comprising means for detecting an image feature representing the inclination and thickness of the eye in that range and determining the shape of the eye. Since this means is provided in a face part recognizing means (not shown), its function is realized in the arithmetic unit 13 or in the feature quantity extracting unit 10 like the face part recognizing means.

According to the above configuration (7), an image feature representing the inclination and thickness of the eyes is detected within the set search range, and the shape of the eyes is determined.
Alternatively, it is possible to avoid a danger that an erroneous feature amount is extracted due to a difference between the dictionary image and the corresponding partial image in the input image. Furthermore, when the target eye shape is a shape that is not included in the category prepared in advance, it is possible to avoid a danger that a correct feature amount cannot be calculated. Furthermore, it is necessary to prepare a template image and a dictionary image, and the amount of work can be significantly reduced. Fifth Embodiment (Invention of Claim 8) In this embodiment, detection of a mouth will be described with reference to FIGS. FIG. 15 is a flowchart showing a mouth detection operation, and FIG. 16 is a diagram showing an example of a mouth detection image and an image of a projection result.

The mouth is detected for an image whose inclination has been corrected by rotation. First, a distance el2 between the centers of the detected eyes is determined. Next, from the detected distance el2 between the centers of both eyes, constants MW and MH, which are predetermined so as to have a size enough to include the mouth, are multiplied by the distance between the centers of both eyes. Range width mw and height mh
Is calculated. Subsequently, the calculated width mw and height m of the range
According to h, an image is cut out about the approximate position of the mouth designated by the user (125). Reference numeral 132 in FIG. 16 is an example of an image around the mouth that has been cut out.

Next, a region near the center in the horizontal direction of the image of the cut out mouth is determined (126), and the image is projected horizontally (127). Reference numeral 133 in FIG. 16 is an example of an image around the mouth that has been cut out. Reference numeral 136 denotes a region in which projection in the horizontal direction in 133 is performed, and reference numeral 139 schematically shows a projection result. The area where this projection is performed is a height m of a range where a value obtained by multiplying a predetermined constant MHW by the width mh of the image near the clipped mouth is projected, and a height of a range where the height of the image near the clipped mouth is projected. It is calculated as

Next, a portion having the lowest value is searched from the projection result 139. The lowest point of this value is the vertical position of the lip tear (128). At this time, it is assumed that the image near the mouth is tilted and not corrected, and the lip cleft is not horizontal. In FIG. 16, reference numeral 134 denotes an example of such an image around the mouth. Reference numeral 137 denotes a region in the horizontal projection in 134, and reference numeral 139 schematically shows a projection result. In 134, the cleft is not horizontal, so the position in the height direction alone cannot accurately represent the position of the cleft. In the projection result 139, it can be seen that the sharpness of the projection result is lost, and the position of the lip cleft cannot be detected. In other words, in the present invention, the mouth is detected after rotating the image in advance so that the entire face is correctly erect and the lip cleft is horizontal according to the result of the eye detection, so that the cleft of the lip is more stably and more accurately than the conventional method. It is possible to perform detection.

Subsequently, a vertically and horizontally long region is determined in the vicinity of the detected cleft lip (129) and projected vertically (13).
0). Reference numeral 135 in FIG. 16 is an example of the image around the mouth in that case, and reference numeral 138 is an area for performing vertical projection. This vertical projection area is obtained by projecting the width of the image near the cut mouth onto the height mvh of a range in which a value obtained by multiplying the height mh of the image near the cut mouth by a predetermined constant MVH is projected. And the calculated position of the detected lip cleft in the height direction is calculated as the center of the projected range in the height direction.

A result 141 of FIG. 16 is obtained by vertically projecting 138 in FIG. The width of the part with a low value of 138 is the width of the cleft lip. The determination of the portion having a low value of 138 is performed by the following method. First, discriminant analysis is performed on the value of the projection result to determine a threshold thm. Next, the threshold thm obtained,
The values of the projection results are compared, the portion having a value lower than thm is a portion having a cleft, the portion having a value lower than thm at the leftmost portion is at the left end portion of the cleft, and the portion having a value lower than thm at the rightmost portion is a portion having a cleft. This is the right end of the cleft lip. The interval between the right end and the left end of the cleft is the width of the cleft, and the position and width of the cleft are recognized as described above (131).

Although not particularly shown, if the lip in the image is tilted, the lip may protrude from the region where the vertical projection is performed, or the height of the region where the vertical projection is performed must be increased. In both cases, it is clear that the detection accuracy of the lip cleft width decreases. In other words,
According to the present invention, the position and width of the lip cleft can be detected stably and with high accuracy by correcting the inclination of the face.

Therefore, the characteristics of this embodiment are summarized as follows. (8) The image processing apparatus according to claim 8 is characterized in that the position and the size detected by the face part recognition means are the mouth of the face part. Although the face part recognition means is not shown, its function is realized in the arithmetic unit 13 or the feature quantity extraction unit 10.

According to the configuration (8), the size of the mouth of the face part is predicted with high accuracy, the detection process is performed only within a necessary and sufficient range in the image, and the face is accurately calculated with a small amount of calculation. The position and size of a part can be detected. [Sixth Embodiment] (Invention of Claim 9) In this embodiment, a method for extracting the position and size of the left eyebrow will be described with reference to FIGS. FIG. 9 is a diagram illustrating an example of an eyebrow search range and a binarized image for detecting the position and size of eyebrows, and FIG. 10 is a flowchart illustrating an operation for detecting the position and size of eyebrows. It is.

First, a search range large enough to include the left eyebrow therein is set (S101). The search range is
It may be obtained from the positions of both eyes specified by the position specifying device in the same manner as when obtaining the eye search range, or may be obtained from the second embodiment (using the image processing device according to claim 5). Alternatively, it may be obtained from the position of the left eye. In any case, in advance, for a plurality of human faces, the position of the eyes and the distance between the eyes,
Alternatively, it is necessary to measure and obtain the relationship between the position of the left eye and the search range in which the left eyebrow is included. FIG.
15 shows a search range image 91 of the left eyebrow and images 92 to 96 obtained by binarizing the image 91 with a plurality of thresholds. Here, it is assumed that the image 94 is an image binarized with an optimum threshold value for separating eyebrows.

In general, there are various factors such as contrast of an input image, hair, shadow, and the like, and it is very difficult to binarize an image to obtain an optimal threshold for separating eyebrows. Therefore, the image processing apparatus solves the above-described problem by comparing binarized images with a plurality of thresholds and determining a threshold value of an image that is considered to have the best eyebrow separation.

That is, the images 92 to 96 are binarized images (S103) in which the lower and upper limits of the threshold are determined (S102), and the thresholds are set at regular intervals therebetween. For example, the lower limit and the upper limit of the threshold value can be determined by using the P-TILE method or the like. In the case of the lower limit, the dark pixels are 5% of the whole, and in the case of the upper limit, the dark pixels are 50% of the whole. Other suitable techniques may be used. In addition, P-
For the TILE method, a technique described in reference [5] may be used.

With respect to the image binarized by each threshold value,
The following processing is performed. First, the binarization here uses a method of expressing dark pixels as white pixels and bright pixels as black pixels, and an area of adjacent white pixels is called a label. This shows that the binarized image 92 is composed of three labels 92a to 92c.

The search range of the eyebrow is set with a sufficient size so as to include the eyebrow in the search range. Therefore, when binarized, the label in contact with each side of the search range image has an area other than the eyebrow. It can be considered as separated. That is, in the binary image 92, it can be seen that the label 92c is not the eyebrow but the hair separated, and is not a label constituting the eyebrow. Therefore, the labels remaining after removing the labels in contact with the respective sides of the search range image remain as label candidates constituting eyebrows (S104).

Of the labels remaining as candidates, labels having a small area, for example, labels having an area of 1% or less of the eyebrow search range image, may be caused by noise in the image or other factors. It can be considered that dark pixels have been separated. Therefore, the label remaining after removing the small area label is used as a pixel constituting the eyebrows (S105).

A rectangle circumscribing the remaining label is obtained as described above, and this width and height are used as the width and height of the eyebrow candidate. The width and height are compared with the average width and height of eyebrows estimated from the distance between the eyes, and the binarized image including the eyebrow candidate with the smallest difference is separated from the eyebrows best. (S10)
6). In this binarized image, the position and size of the label that is considered to separate the eyebrows are set as the position and size of the left eyebrow in the input image (S107).

The processing for the left eyebrow has been described above. However, for the right eyebrow, the same processing can be applied by inverting the search range image horizontally.

Therefore, the characteristics of this embodiment are summarized as follows. (9) The image processing apparatus according to claim 9 is characterized in that the position and size detected by the face part recognizing means are eyebrows of the face part. Although the face part recognition means is not shown, its function is realized in the arithmetic unit 13 or the feature quantity extraction unit 10.

According to the configuration of the above (9), the position of two or more face parts is specified by the position specifying means, and the size of the eyebrows is predicted from the distance relationship between the specified positions. The range in which the processing at the time of extraction should be performed can be limited to an appropriate size. Then, binarization is performed on the processing range. In order to obtain a region to be recognized, binarization is performed using a plurality of methods and a plurality of thresholds. The method is characterized in that a method is provided for detecting the recognition target with high accuracy by determining the position, size, shape, etc. of the image and selecting the most reliable image. Therefore, the positions of two or more face parts are specified by the position specifying means, the size of the eyebrows is predicted from the distance relationship between the specified positions, and the range in which the processing for extracting the feature amount is to be performed is appropriately large. In addition to the restriction, the size of the eyebrows can be estimated. That is, for example, when the face parts whose positions are specified by the position specifying unit are the right eye and the left eye, the size of the eyebrows is not a value greatly separated from a value obtained by multiplying the distance between the eyes by a constant coefficient. can do.

Therefore, the size of the region separated when binarization is performed is compared with the estimated size of the eyebrows.
By obtaining a binarization threshold value such that their sizes are not so far apart, an area representing an eyebrow can be detected with high accuracy. Seventh Embodiment In the present embodiment, a method of determining the shape of the left eyebrow will be described with reference to FIGS.

FIG. 11 is a diagram showing an example of a rectangle circumscribing the eyebrows. FIG. 12 is a diagram showing an example of quantization for recognizing the shape of the eyebrows. FIG. 13 is a diagram showing how the eyebrows bend. FIG. 14 is a flowchart showing the operation for detecting, and FIG. 14 is a flowchart showing the operation for detecting the thickness of the eyebrows.

First, the search range 111 is set so as to include the left eyebrow therein (S131). FIG.
In the image 112 binarized with the optimum threshold value to extract the position and size of the eyebrows using the embodiment (the image processing apparatus of claim 9), the position and size of the eyebrows are separated from the eyebrows. This is shown as the bounding rectangle 113 of the label. The label image within the rectangle is quantized (S132). FIG.
Shows an example in which quantization is performed to a size of 3 × 2.
Here, the color of the ground is represented by white, and the color of the eyebrows is represented by black. Here, example 121 of rising eyebrow and example 12 of falling eyebrow
Although 2 is shown, when this is quantized to a size of 3 × 2, it becomes as shown in schematic diagrams 124 and 125. here,
For example, among the 3 × 2 blocks, the portions A and B shown in the schematic diagram 123 are observed, and when A white and B black, the rising eyebrow, and conversely, when A black and B white, the falling eyebrow can do. Similarly, other bent shapes can also be detected by examining the same quantization pattern (S133), and thus, it is possible to detect how the eyebrows are bent.

Next, the thickness of the eyebrows is detected. To detect the thickness of the eyebrows, for example, the following processing is performed. First,
A label (hereinafter referred to as an eyebrow label) in which an eyebrow is separated in an image binarized with an optimum threshold value for extracting the position and size of an eyebrow using the sixth embodiment (the image processing apparatus of claim 9) (Hereinafter referred to as an image). For this contraction processing, the technique of Reference [5] may be used.
A contraction process is performed, the number of times until the eyebrow label disappears is measured, and the thickness of the eyebrow is determined based on the number of times. That is,
In the case of a thin eyebrow, the eyebrow label disappears with a smaller number of contraction processes than in the case of a thick eyebrow, so by measuring the number of times until the eyebrow label disappears, the thickness of the eyebrow can be detected. .

As described above, the bending and thickness of the eyebrows are measured, and based on these two parameters, the eyebrow shape code corresponding to the preset category can be obtained.

The processing for the left eyebrow has been described above. However, for the right eyebrow, the same processing can be applied by inverting the search range image horizontally.

Therefore, the characteristics of this embodiment are summarized as follows. (10) The image processing device according to claim 10 sets a search range based on the position and size of the eyebrows detected by the face part recognizing means, and detects an image feature representing the thickness of the eyebrows and how to bend in the range. And a means for determining the shape of the eyebrows. Since this means is provided in a face part recognizing means (not shown), its function is realized in the arithmetic unit 13 or in the feature quantity extracting unit 10 like the face part recognizing means.

According to the configuration of the above (10), the image characteristics representing the thickness and the manner of bending of the eyebrows are detected within the set search range, and the shape of the eyebrows is determined, so that the template image or the dictionary image And the corresponding partial image in the input image is shifted, thereby avoiding the risk that an erroneous feature amount is extracted.
Further, when the shape of the target eyebrow is not included in the category prepared in advance, it is possible to avoid a risk that a correct feature amount cannot be calculated. Furthermore, it is necessary to prepare a template image and a dictionary image, and the amount of work can be significantly reduced. [Eighth Embodiment] (Embodiment 11) In this embodiment, an image processing operation for judging a human jaw contour shape in an original image shown in FIG. 2 will be described with reference to FIGS.

FIG. 23 is a flowchart for explaining an image processing operation performed by the image processing apparatus shown in FIG. 1, and FIG. 24 is a diagram for explaining the arrangement of the center coordinates and the initial contour in the input image. FIG. 25 is a diagram for explaining a method of calculating a color difference on a straight line connecting a point on the initial contour and the center coordinates, and FIG. 26 is a diagram schematically illustrating an example of calculating a color difference. FIG. 27 is a diagram for explaining a case where a face is elliptical as a method for performing color difference calculation specialized for a face contour shape, and FIG. 28 is a method for performing color difference computation specialized for a face contour shape. FIG. 29 is a diagram for describing a case in which the fact that the face is symmetrical about the center axis is used, and FIG. 29 is a diagram for explaining a method of calculating a distance function from the extracted face contour. FIG. Is a diagram for explaining a method of comparing the distance function and the reference distance function obtained from the input image.

Here, an image processing operation for judging the shape of the person jaw contour in the original image 21 shown in FIG. 2 will be described with reference to the flowchart of FIG.

First, it is assumed that the input device 11 stores the target original image 21 in the storage device 11. First, the operator specifies position information for specifying the center of the face using the position specifying device 14 and determines the center position of the face on the original image (step S201). The center position of the face may be directly designated by the operator, or may be designated by 22 to 2 in FIG.
Specify the center coordinates of the right eye, left eye, and mouth as shown in 4.
The center may be calculated as the center position of the face.

Next, in step S202, an initial contour is arranged near the face contour. The arrangement of the initial contour is directly designated by the operator using the position designation device 14 or, as described above, when the arrangement of other parts of the face such as eyes and mouth is known, the information is used based on such information. It may be automatically arranged at an appropriate position. For example, a region surrounding the eye and mouth regions is set as the initial contour. The relative distance between the eyes and the mouth may be statistically checked in advance, and may be arranged so as to surround the eyes and the mouth with an appropriate margin. FIG. 24 shows an example for explaining an image in which the center position 210 and the initial contour 211 are determined.

Next, in step S203, a color difference between adjacent pixels on a straight line connecting the face center coordinates and each coordinate on the initial contour is calculated from the original image, the center position, and the initial contour, and the coordinates between the target pixels are calculated. An image (color difference map image) is created with the calculated color difference as the pixel value, with the middle point as the coordinate value.

Here, as a method of calculating the color difference, for example, a difference value is calculated by subtracting a luminance value for each single color light of pixel data between pixels.
The sum of the difference values for each single color light is calculated as a color difference.
The color difference calculation method may use another method. For example, the pixel data is calculated based on the hue (H) from the luminance value of each monochromatic light.
It may be converted into an HSV value expressed by / saturation (S) / luminance (V), the position of the two pixels for which the color difference is to be determined in the HSV space, and the distance value between them may be used as the color difference. Alternatively, an average color may be calculated not for adjacent pixels but for every five consecutive pixels, for example, and a color difference between the average colors may be calculated. When calculating the color difference, the detection accuracy of the color difference may be changed by utilizing the fact that the target is a human face. For example, if the pixel values of the two pixels to be compared when calculating the color difference have a value close to the pixel value representing the skin color, it is considered that the two points are likely to be pixels within the face outline, and the detection accuracy of the color difference is determined. Lowering the influence of noise and the like. On the other hand, it is highly possible that both the chin and the neck have pixel values representing the skin color, and it is better to increase the detection accuracy when detecting the chin boundary which is the boundary. Therefore, at the time of detecting a color difference on a straight line from the center to the neck, the detection accuracy of the color difference is increased, and the jaw boundary is easily detected. Note that the position of the neck can be estimated in the coordinate direction if the coordinates of the mouth are the values.

For example, the face center 212 as shown in FIG.
FIG. 26 is a schematic diagram showing a case where a color difference on a straight line 214 connecting the coordinate point 213 on the initial contour with the color difference 213 is obtained. Reference numeral 215 denotes an arrangement of pixel values on a straight line, and reference numeral 216 denotes a difference in pixel value between two adjacent points. That is, in this example, 216 indicates the arrangement of the color differences.

Further, after detecting the color difference, a color difference map image more specialized for the face outline shape may be created by further utilizing the unique characteristic of the human face outline. For example, assuming that the face has a similar shape to an ellipse, as shown in FIG. 27, the color difference of a total of three points of two points adjacent to one point on an elliptic curve of an arbitrary size centered on the center of the face is averaged. Then, the color difference of the coordinates is stored again to suppress the influence of noise. Alternatively, assuming that the face outline has left-right symmetry, as shown in FIG. 28, the color differences of two coordinates with the straight line connecting the face center and the mouth coordinates as the axes of symmetry may be averaged to obtain the respective color differences.

As described above, by using a person's face as a constraint, an energy image specializing in expressing jaw-shaped features can be created, and a clear contour line does not appear. More stable jaw detection can be performed on an input image or an image with much noise.

Next, in step S204, the initial contour is moved according to the active contour model to extract the contour. As the energy function E, for example, the sum E = E1 + E2 + E3 of the internal energy E1 representing the smoothness of the contour, the energy E2 for shrinking the contour, and the image energy E3 characterizing the object contour is obtained, and this E is minimized. Move the contour.

Here, the color difference map image created in step S203 is used for the image energy E3. Image energy E3 at an arbitrary point P (x, y) on the image
(P) represents the color difference value on the color difference map image corresponding to P by D
When (P) is set, it is obtained from Expression 201.

E3 (P) = α × (MAX (D) −D (P)) (Equation 201) where MAX (D) is the maximum value of the color difference in the color difference map image, and the coefficient α is the energy function E Means the degree of contribution of image energy. According to Equation 201, the smaller the color difference is, the larger the image energy is, and the easier the contour is to move. Conversely, the larger the color difference, the smaller the image energy and the more difficult it is for the outline to move.

Next, in step S205, step S205
A distance function is calculated based on the contour obtained in step 204. That is, a contour r is a function r = L consisting of the coordinates of the value inside the face, for example, the distance r from the face center and the direction (angle) θ.
(Θ). FIG. 29 is a schematic diagram showing this state.
Shown in

For L (θ), r may be obtained when the value of θ is changed by a unit angle. For example, the range (the direction in which the neck is viewed from the center of the face) that more clearly expresses the jaw shape is expressed in units of The angle may be narrowed to increase the amount of information as compared to other directions. Further, the distance function may be represented as a Fourier descriptor represented by, for example, Expression 202.

[0152] Here, the coefficient exp () in which A (n) represents the curve shape represents the power of the base of natural logarithm, s represents the distance on the curve, and L represents the total length of the closed curve. Details regarding the Fourier descriptor are disclosed, for example, in reference [5].

Next, in step S206, step S
The shape is determined by comparing the features of the distance function obtained in 205 with the reference distance function. Here, the reference distance function is a distance function created in advance from a jaw-shaped contour serving as a reference. The contour of the jaw shape serving as a reference is an image in which the contour is manually detected in advance, a similar jaw shape,
For example, what is necessary is to classify into a base type, a round type, and the like, and use an average of contours detected manually for each classification. The comparison of the distance functions includes, for example, positioning the position of the inflection point on the distance function, the number of inflection points, the slope between the inflection points, and the like as the characteristics of the distance function, and the characteristics of the distance function of the reference contour shape. By comparing with When making a comparison,
It is necessary to normalize in advance so that the position matches the reference distance function. The position and number of inflection points and the inclination between inflection points are obtained in advance for the reference shape, the information is stored in a memory, and the information of the inflection point of the distance function obtained in step S205 is obtained. May be compared as appropriate. Then, the shape having the reference distance function closest to the comparison result is determined as the determination result. Note that the comparison of the distance functions can be performed simply by comparing the sum of the differences with the reference distance function.

FIG. 30 is a diagram schematically showing this state. Z in FIG. 30 indicates a difference from the reference function. When the reference distance function is B (θ), the difference sum Z1 is given by Expression 203.

[0155] That is, the shape having B (θ) that minimizes Z1 may be determined as a similar shape. In this method, θ
It is necessary to prepare B (θ) for the reference shape in the memory for the reference shape, but it is possible to easily perform more detailed shape classification and judgment.

If a distance function is expressed using a method of describing a curve on a plane in the frequency domain, for example, using a Fourier descriptor, the Fourier coefficient calculated by this method can be positioned as a feature of the distance function. By comparing with the distance function coefficient of the reference contour shape,
Shape determination can be performed in the same manner as described above.

Let Ab be the coefficient of the Fourier descriptor of the reference shape.
When (n) is set, the difference Z2 from the target distance function is expressed by the following equation (20).
4, the shape having Ab (n) in which Z2 is the minimum is determined as a similar shape.

[0158] Generally, a low-order term of the Fourier coefficient reflects a rough curve shape, and a high-order term reflects a more detailed curve shape. Therefore, by comparing low-order terms, that is, by finding Z2 by reducing the range of n in Equation 204,
It is possible to obtain a determination result in which influences such as noise and individual differences are eliminated as much as possible.

By the above operation, more stable jaw detection can be performed even on an image in which the photographing conditions are poor, and an image in which a clear contour line does not appear or an image with much noise can be obtained, and the influence of noise and individual differences can be obtained. It is possible to judge the jaw shape which is excluded.

Therefore, the features of the present embodiment are summarized as follows. (11) An image processing apparatus according to claim 11, further comprising a contour recognizing means for detecting a jaw contour feature based on one or more pieces of position information designated by the position designating means and judging its shape. It is characterized by becoming. In other words, a feature image that represents the features of the jaw contour more prominently is created from the information that characterizes one or more faces obtained from the position specifying unit,
The method is characterized in that a contour is detected by a dynamic contour model using the image as image energy, and the detected contour is expressed as a distance function consisting of a distance and a direction from the value portion of the face. It is characterized in that the jaw contour shape is determined by calculating the feature of the function and comparing the reference feature. This contour recognition means is provided in a face part recognition means (not shown).
0 implements that function.

According to the configuration (11), the operator of the image processing apparatus first specifies the center of the face of the person included in the input image by the position specifying means. The face center may be specified directly, or may be estimated from the specification of other facial features, for example, the coordinates of both eyes and mouth. Next, an initial outline coordinate sequence including the face of the person is obtained. Next, a color difference between adjacent pixels on a straight line connecting the face center coordinates and each coordinate on the initial contour is calculated, a coordinate middle point between the target pixels is set as a coordinate value, and an image having the calculated color difference as a pixel value (hereinafter, referred to as an image) , A color difference map image). Next, a jaw contour line is detected using an active contour model using this color difference map image as image energy. Next, the obtained contour is converted into the coordinates of the value inside the face, for example, the distance and direction (angle) from the face center.
(Hereinafter referred to as a distance function).
Next, the feature of the distance function is compared with the feature of the distance function of the reference contour shape, and the contour shape having the closest distance function is determined as the jaw shape of the input image.

Here, the accuracy of the color difference map image creation may be enhanced by utilizing the fact that the image is a human face of the target image. For example, when obtaining the color difference, the skin color and the other colors may be determined separately. In other words, the color difference between pixels classified as skin color is reduced by lowering the color difference detection accuracy.
The effects of noise and wrinkles can be made less likely to be reflected in the color difference map image. Conversely, the border between the neck and the chin often has the same flesh color, and it is difficult to produce a color difference. Therefore, when detecting a color difference on a straight line from the center to the neck, the detection accuracy may be increased. Note that the direction of the neck position can be estimated if the coordinates of the mouth are the values.

After the color difference map image is created as described above, for example, by assuming an ellipse as the face outline, the pixel value (= color difference) on the ellipse coordinate centered on the face center coordinate and the pixel values on both sides thereof Are averaged to obtain the pixel value. Alternatively, when other features other than the face outline are known separately, for example, if the center coordinates of the mouth are the values, a line having the straight line connecting the center coordinates of the mouth and the center of the face on the symmetry axis is used.
The pixel values of the pixels may be averaged and used as the pixel value.
This makes it possible to create an energy image in which the features of the jaw shape are taken into account, and it is possible to perform more stable jaw detection even on an input image in which a clear outline does not appear or an image with much noise.

Also, when creating a distance function from a contour line, the characteristic of the jaw can be represented more prominently by using the unique characteristics of the human face contour to minimize the influence of noise and lighting. To modify the distance function. For example, a distance function such as averaging can be corrected based on the shape of a face such as an ellipse or symmetry as in the case of creating a color difference map.

Next, the distance function is compared with those of the distance function, such as the position of the inflection point of the distance function, the number of inflection points, and the slope between the inflection points. This is done by comparing each with the feature. And
The reference shape having the most similar reference distance function is determined as the corresponding contour shape.

If a distance function is expressed using a method of describing a curve on a plane in the frequency domain, for example, using a Fourier descriptor, the Fourier coefficient calculated by this method can be positioned as a feature of the distance function. By comparing with the distance function coefficient of the reference contour shape,
Shape determination can be performed in the same manner as described above.

The feature of the reference distance function to be compared may be that the distance function may be normalized in advance and stored in a memory as a table, or only the information such as the position of the inflection point that is required in advance is normalized. May be stored. When a Fourier descriptor is used, a coefficient of a required order may be stored. These methods do not need to have a reference shape to be compared as a dictionary image as compared with template matching, which is advantageous in terms of memory cost and processing speed.

When the Fourier descriptor is used, the fact that the low-order term of the Fourier coefficient reflects a rough curve shape and the high-order term reflects a more detailed curve shape is used. By comparing the terms (1) and (2), it is possible to obtain a determination result in which influences such as noise and individual differences are eliminated as much as possible. Ninth Embodiment (Inventions of Claims 12 to 17) This embodiment will be described with reference to FIGS. FIG. 31 is a block diagram showing the configuration of the image processing apparatus of the present embodiment, FIG. 32 is a flowchart showing the processing of the image synthesizing apparatus of FIG. 31, FIG. 33 is an explanatory diagram relating to hair color extraction, and FIG. FIG. 3 is an explanatory diagram related to bangs classification, and FIG.
FIG. 5 is an explanatory diagram relating to back hair classification. Hereinafter, each step will be described in detail according to the processing flow of FIG. 32 and with reference to the explanatory diagrams of FIGS. 33, 34, and 35.

First, a face image is input by the input means 321 and stored in the storage means 322 (step S34).
1). Next, the right eye, the left eye,
The approximate position of the mouth and the face outline are input and stored in the storage unit 322 (step S342). Each unit operates by using the arithmetic unit 323 with reference to the image stored in the storage unit 322 and information on the right eye, left eye, mouth, and face outline,
Classify bangs and back hair, or determine hair parts. Although the approximate positions of the right eye, the left eye, and the mouth are input here, these three points are not necessarily required. For example, instead of these three points, the nose and the mouth are replaced.
It is also conceivable to enter a rough position of the point.

As the information on the right eye, left eye, mouth, and face outline, the designated information may be used as it is, but the second embodiment (claim 5) or the fifth embodiment (claim 8) The accuracy can be further improved by using the position detected by the method described above. Although the input image may be used as it is, the image may be used as it is, based on the detected right eye, left eye, and mouth position in advance by the method described in the third embodiment (claim 6). The accuracy can be improved by performing a rotation process so that the right and left eyes are horizontal or close to it, or performing image processing such as a low-pass filter. When performing image rotation processing, rotation processing for the same angle is performed on the right eye, left eye, mouth, and face contour.

The hair color extracting means 326 extracts the hair color as follows (step S343). This extraction method will be described with reference to FIG. Note that, as shown in FIG. 33, in the following description, the y-coordinate is set in such a direction that the value increases from the upper side to the lower side.

First, the right eye,
Based on the coordinates of the left eye and the mouth, a skin color is extracted using pixel values in a region near the nose. In this case, the average value may be simply calculated. For example, the average value and the variance may be calculated once, and the average and the variance may be calculated again except for the pixels that are largely deviated from the average. Here, performing skin color extraction is useful for hair color extraction and hair feature extraction as described later, but is not always necessary, and the skin color extraction means can be omitted.

Next, using the coordinates of the right eye, left eye, and mouth, the initial estimated value f of the crown height ft and the hairline height fh is calculated.
t0 and fh0 are determined. For example, the average value of the y coordinate of the right eye and the left eye is y_eye, and the y coordinate of the mouth is y_m.
If out, coefficients k_ft, k_f appropriately determined
h, ft0 = y_eye-k_ft × (y_mouth
−y_eye) fh0 = y_eye−k_fh × (y_mouth
−y_eye).

Next, sampling rectangles ABFE and EFDC are set based on the above ft0 and fh0. Here, the y coordinate of E and F is ft0, and the y coordinate of C and D is fh0.
And the y coordinate of A and B is ft0− (fh0−f
t0) (AE = EC). Also, A,
The x-coordinates of E and C are near the right eye (coming to the left on the image) or slightly to the left of the right eye (on the image).
The x coordinate of D may be near the left eye or slightly to the right (on the image) of the left eye.

Next, the height ft of the EF is set to an appropriate threshold value ft_
For up, ft_down, ft0−ft_up << = ft << = ft0 + ft
The search is performed by moving up and down within the range of _down, and a point at which the degree of separation between the pixel value in the rectangle ABFE and the pixel value in the rectangle EFDC is maximized is set as the estimated value of the crown height ft. This degree of separation is
The average value of the pixel values in the rectangle ABFE is A1, the variance is V1,
The average value of the pixel values in the rectangular EFDC is A2, the variance is V2,
If the average value of the pixel values in the ABDC is A3, the variance is V3, and the area ratio between the rectangle ABFE and the rectangle EFDC is S1: S2, then: S1 × (A1-A3) × (A1-A3) + S2 × (A
2-A3) × (A2-A3)｝ / V3. If the image is a color image, the pixel value may be calculated assuming that the pixel value is a three-dimensional vector.

Next, a background color is extracted in the rectangle ABFE. At this time, the lower side EF is at the height ft after being moved by the above search. In this case, the average value and the variance may be obtained once, and the average value and the variance may be obtained again except for the pixels that are largely deviated from the average. Note that extracting the background color in this way is useful for hair color extraction as described below, but is not always necessary.

Further, hair color is extracted in the rectangular EFDC. At this time, the upper side EF is at the height ft after being moved by the above search. Although it is conceivable that the average value is simply calculated, pixel values other than the hair pixels are included in the average calculation, and it is considered that the accuracy is reduced. Therefore, for example, the average value may be performed as follows.

Using the average and variance of the flesh color and the average and variance of the background color, the average and variance are calculated excluding the pixels close to the flesh color and the pixels close to the background color. Further, by using the average and the variance of the hair color, the average and the variance are calculated again except for the pixels which have already been removed in the above calculation and the pixels which are largely deviated from the average. At this time,
The number of removed pixels is large, and the number of pixels used for calculation as hair color (hereinafter referred to as “hair color pixel”) is equal to a certain threshold n_
If the value is less than sh, it is considered that the hair color has not been stably extracted because the hair is thin, so that step S34 is performed.
The hair characteristic extraction of No. 5 is skipped, and the process jumps to the hair classification of step S346 (step S344). In this case, the hair classification means 335 sets the hair classification to “light hair”. When one or both of the skin color and the background color are not extracted, the hair color can be extracted by omitting the process of removing the pixels that are not extracted, but the accuracy is reduced. It is thought that.

The hair characteristic extracting means 329 comprises one or both of the front hair characteristic extracting means 327 and the back hair characteristic extracting means 328, and extracts hair characteristics (step S34).
5).

An operation example of the bangs feature extracting means 327 will be described below.

Using the average value and the variance of the hair color and the average value and the variance of the skin color, for each pixel in the image,
If the skin color is closer to the skin color than the hair color and it does not greatly deviate from the average value of the skin color, it is a non-hair pixel, otherwise,
Label it as a hair pixel. Thereby, a hair region can be extracted. If the skin color is not extracted in step S343, the respective pixels in the image may be labeled as hair pixels if they do not greatly deviate from the average value of the hair color, and otherwise may be labeled as non-hair pixels. . The hair region itself is considered to be one bangs feature itself, and further, by using this, a mesh of about 11 mesh width × 7 mesh length is set at an appropriate position that is considered to include the bangs, The number of hair pixels in each mesh is defined as a bangs feature (hereinafter, this feature is referred to as a "bangs mesh feature").
For example, bangs characteristics are extracted.

An example of the operation of the back hair characteristic extracting means 328 will be described below.

Using the average value and the variance of the hair color and the average value and the variance of the skin color, for each pixel in the image,
If the color is closer to the hair color than the skin color and does not greatly deviate from the average value of the hair color, the hair pixel is labeled, and if not, the label is a non-hair pixel. Thereby, a hair region can be extracted. When the skin color feature is not extracted in step S343, for each pixel in the image,
If it does not deviate significantly from the average value of the hair color, it may be labeled as a hair pixel, otherwise, it may be labeled as a non-hair pixel. The hair area is itself considered to be a back hair feature, but it is further used to provide a considerable amount of hair in the case of long hair, including, for example, so-called "semi-long" hair. In the case of short-haired hair, instead, a rectangular area that seems to have little hair is set on both the left and right sides of the face, and the number of hair pixels in those rectangles is taken (hereinafter, this feature is referred to as A back hair feature is extracted.

In the above description, the extraction method has been described assuming that the forehead feature and the back hair feature are separate, but the hair feature may be extracted by considering both as one. For example, the hair pixel area is common to the bangs and the back hair, and for example, for each pixel in the image, if it does not greatly deviate from the average value of the hair color, it is labeled as a hair pixel, otherwise, it is labeled as a non-hair pixel. It can also be created by attaching.

In the fourteenth aspect, one or both of the front hair contour and the rear hair contour are extracted by using the hair contour extracting means 332. The hair contour extracting means 332
It comprises one or both of the forehead contour extraction means 330 and the back hair contour extraction means 331.

The forelock contour extracting means 330 operates as follows using the hair region extracted by the forehead characteristic extracting means 327.

From the middle point of the right and left eyes, the image is scanned right above the image to the end of the image, and the longest hair pixel run is detected. With the lowest point of this run as the starting point, the contour is traced in the left direction (on the image), below a threshold value of a certain y coordinate (the y coordinate value is large) determined based on the coordinates of the right and left eyes, and The tracking is terminated when it comes to the left of a threshold value of a certain x coordinate similarly determined. Next, with the lowest point of the run as a starting point, the contour is traced in the right direction (on the image), and is lower than a certain y-coordinate threshold value determined based on the coordinates of the right and left eyes (the y-coordinate value is large). And, when it comes to the right of a threshold value of a certain x coordinate similarly determined, the tracking is completed. Further, the left contour and the right contour are joined to form a bangs contour.

The back hair contour extracting means 331 operates as follows using the hair region extracted by the back hair feature extracting means 328.

From the middle point of the right and left eyes, the image is scanned right above the image to the end of the image, and the longest hair pixel run is detected. With the top point of this run as the starting point, the contour is traced to the left (on the image), below a certain y-coordinate threshold value determined based on the coordinates of the right and left eyes (the y-coordinate value is large), and similarly When it comes to the left of a threshold value of a certain x coordinate, the tracking ends. Next, with the top point of the run as a starting point, the contour is traced in the right direction (on the image), and a threshold value (y) below a certain y coordinate determined based on the right eye and left eye coordinates (y
The tracking is finished when the coordinate value is large), and when it comes to the right of a threshold value of a certain x coordinate similarly determined. Further, the left contour and the right contour are joined to form a back hair contour.

In the above, each extraction method has been described assuming that the forehead outline and the back hair outline are separate. However, the hair outline is tracked by assuming the two as one and tracking the outline using the hair region. May be extracted.

According to the fourteenth aspect, the hair characteristic extracting means 3
29 may use the extracted hair contour to extract another hair feature. For example, it is conceivable to detect the top point of the forehead contour and use it as the forehead feature, or to scan the back hair contour and detect the point where the dent inside the hair becomes the maximum, and use it as the back hair feature. .

[0192] The hair classification means 335 is a bangs classification means 333.
And one or both of the back hair classification means 334, and the hair features obtained by the hair feature extraction means 329,
According to the fourteenth aspect, the hair style is classified using the hair contour obtained by the hair contour extracting means 332 (step S346). It is also conceivable to classify the hair by considering the bangs and back hair as one, without distinguishing or separating them.

An example of the operation of the bangs classification means 333 will be described below.

Using the extracted bangs mesh feature, if the number of meshes whose number of hair pixels is equal to or greater than a certain threshold value c2 is equal to or greater than a certain threshold value m_fc, the hair is classified as “Oppappa”. Here, “okap” is a hairstyle in which the forehead is mostly covered with hair. If the number of meshes whose number of hair pixels is less than c2 and is equal to or greater than another threshold value c1 satisfying c1 <c2 is equal to or greater than a certain threshold value m_fm, the mesh is classified as "blind." The term “blind” as used herein refers to a hairstyle in which a considerable amount of hair is covered on the forehead, but a considerable amount of skin is visible from the gap in the hair.

According to the fourteenth aspect of the present invention, the hair style is further classified using the features of the hair contour as follows (see FIG. 34).

In the case where neither “okappa” nor “blind” is described above, first, at the upper part of the bangs contour, it is checked how much the contour is depressed to the side (upward) of the hair area. If there is not much, it is roughly classified into "without division", and if not, it is classified into "with division".

Next, as for “no division”, the linearity at the top of the bangs contour is examined. If the linearity is large (close to a straight line), it is “square” (see FIG. 34 (a)). The case is classified into "round" (see FIG. 34B).

Further, with respect to “with a split”, an appropriately determined df1 <df2 <df3 is determined by using the x-coordinate (hereinafter referred to as x_df) of the uppermost point of the detected bangs contour.
In contrast to thresholds df1, df2, df3, and df4 of <df4, when x_df <df1, "one ninety division" (FIG. 34)
(See FIG. 34 (c)), df1 <= x_df <df2, "37 divisions", df2 <= x_df <= df3, "center division" (see FIG. 34 (d)), df3 <x_df
In the case of <= df4, it is classified as "divided into seven," and in the case of df4 <x_df, it is classified as "divided into nineteen."

An example of the operation of the back hair classification means 334 will be described below.

Using the extracted back hair rectangle features, if the number of hair pixels is equal to or greater than a certain threshold value n_b, the hair is classified as “long hair”, otherwise, it is classified as “short hair”.

According to the fourteenth aspect of the present invention, the hair style is further classified as follows using the features of the hair contour (see FIG. 35).

The above “long hair type” and “short hair type” are broadly classified.
Furthermore, using the x-coordinate (hereinafter, referred to as x_db) of the point where the inward indentation of the detected back hair contour inside the hair region becomes maximum, db1 <db2 <db3 <db4 appropriately determined.
For the thresholds db1, db2, db3, db4, x_
In the case of db <db1, "one ninety", db1 <= x_d
In the case of b <db2, “divide by three”, db2 <= x_db
In the case of <= db3, it is subdivided into "center division" (see FIG. 35 (a)), in the case of db3 <x_db <= db4, it is subdivided into "73 divisions", and in the case of db4 <x_db, it is subdivided into "91 divisions". However, if the detected indentation of the back hair contour into the inside of the hair is not so large even at the point where it becomes the maximum, it is subdivided into "No split detected" (see FIG. 35 (b)).

[0203] According to the fifteenth aspect, the face contour feature extracting means 336 is provided, and the hair classifying means 335 classifies the hair as follows using the face contour features in addition to the hair features.

First, the face outline feature extracting means 336 obtains the minimum value of the y-coordinate of the face outline point, and sets this as the height of the top of the face outline.

Next, the hair classifying means 335 determines, for example, the height of the uppermost part of the face contour (hereinafter referred to as y_ft),
Using the average value y_eye of the y coordinate of the right eye and the left eye,
If y_eye-y_ft <hf with respect to a certain threshold value hf, in step S344, even if the number of hair color pixels is small, the classification is not performed as “light hair”, and the process proceeds to step S345, where the hair characteristics are determined. After the extraction, in step S346, these are used to perform classification into an appropriate category other than "thin hair".

The fore-hair part determining means 337 and the back-hair part determining means 338 operate as follows to determine a hair part (step S347). It is not always necessary to have both of the bangs part deciding means 337 and the back hair parts deciding means 338. One of the bangs part and the back hair part can be determined by the user's specification, and the bangs and the back hair can be determined. It is also conceivable to use a hair part that is regarded as one without distinguishing or separating the hair parts.

The bangs part deciding means 337 outputs the classification result of the bangs categorizing means 333, for example, "Okappa", "Blind", "Square type", "Round type", "Nineteenth part", "Seventh part". One of the parts created in advance corresponding to these is determined and output in accordance with ", middle division", "73 divisions", and "91 divisions". At this time, the color of the component may be determined according to the hair color extracted in step S343.

In the sixteenth aspect, the bangs classification means 33
In addition to the classification result of No. 3, the classification result of the back hair classification means 334 is also referred to. For example, if the front hair classification is “square type” and the rear hair classification is “short hair / 91 class”, “square type” not,
The bangs pattern corresponding to "Nine parts" is output. Or, although there is not so clear division in the bangs,
"Square type" with hair flowing from the left side (right side on the image) to the right side (left side on the image)
It is also conceivable to prepare a different pattern and output this.

[0209] The back hair part determination means 338 outputs the classification result of the back hair classification means 334, for example, "short hair type / one-part",
In accordance with the “long hair system / middle classification”, one of the parts created in advance corresponding to these is determined and output.

According to the seventeenth aspect, the back hair classification means 33
In addition to the classification result of No. 4, the classification result of the bangs classification means 333 is also referred to. For example, if the back hair classification is “long hair type / partition is not detected” and the bangs classification is “Nine classification”, the back hair pattern Outputs the one corresponding to "Long hair style / 91".
Alternatively, if the back hair classification is "long hair / 91 divisions" and the bangs classification is "37 divisions", the division positions are different between the bangs classification and the back hair classification. Is considered to have a high degree of reliability, so that the back hair pattern corresponding to the "long hair type / 37 division" is output.

In the above, hair color extraction, hair feature extraction, hair classification, and hair part determination are performed, but it is not always necessary to perform all of them. For example, in the invention of claim 13, the processing up to the hair color extraction in step S343 is performed, steps S344 to S347 are omitted, the type of the shape of the hair part is determined by user designation, and only the color of the hair part is determined.
It is also conceivable to determine according to the extracted hair color.

Although the hair classification means and the hair part determination means are separate means, considering that the classification result can be regarded as a hair part type, the division between them is not always clear. Not something.

Further, hair parts are created in advance, and hair parts are determined and output. However, one or a plurality of patterns prepared in advance are appropriately applied based on the result of hair classification. It is also conceivable that the output is deformed.

[0214] The skin color extracting means 325, the hair color extracting means 326 for extracting the hair color by the hair recognizing means, the bangs characteristic extracting means 327 for extracting the characteristics of the bangs, and the rear means. A hair feature extracting means 329 for extracting the features of the hair part, comprising a back hair feature extracting means 328 for extracting the features of the hair part, a forehead contour extracting means 330 for extracting the contour of the bangs, and after extracting the contour of the back hair Hair contour extraction means 331
A hair contour extraction means 332 for extracting a hair contour using the features of the hair part, which is composed of: a hair contour extraction means 332 for classifying bangs, and a hair contour classification means 334 for classifying back hair Hair classification means 335 for classifying hair using
A face contour feature extraction unit 336 for extracting features of the face contour;
Bangs part determining means 3 for determining bangs parts from the contour of the face
37 and back hair part determination means 338 for determining back hair parts
It is assumed that a hair recognizing means is configured from the above.

Therefore, the features of the present embodiment are summarized as follows. (12) In the image processing apparatus according to the twelfth aspect, the feature recognition unit estimates the crown height and the hairline height based on one or more position information designated by the position designation unit, and recognizes the hair region. It is characterized by comprising means. (13) An image processing apparatus according to a thirteenth aspect is characterized in that the hair recognizing means includes a hair color extracting means for extracting a hair color.

According to the above configurations (12) and (13),
Since it is not necessary to extract the background region from the entire image, the background color does not need to be uniform or close to it, and the hair color can be extracted from a normal snapshot or the like, or a portrait can be created. (14) The image processing apparatus according to claim 14, wherein the hair recognizing unit is configured to perform the operation based on at least one position information specified by the position specifying unit.
Hair feature extracting means for extracting features of the hair part, hair contour extracting means for extracting a hair contour using the features of the hair part, and hair classification means for classifying the hair using the hair contour It is characterized by becoming.

According to the structure of (14) above, in order to extract the outline of the hair instead of using the template matching, the "separation" referred to as a so-called "seven-three division" or "center division" is used. Detecting and classifying with high accuracy, judging the roundness of the hairline,
It is possible to perform fine shape classification such as classification into “round type”. (15) An image processing apparatus according to claim 15, wherein the hair recognizing means includes a face contour feature extracting means for extracting face contour features, and a hair classifying means for classifying hair using the hair features and the face contour features. It is characterized by becoming.

According to the above configuration (15), for example, when the height of the top of the face contour line is lower than the head height by a certain threshold or more, it is determined that there is a considerable amount of hair, so When it is difficult to distinguish between the hair area and the skin area,
It is possible to eliminate or reduce erroneous judgments such as "thin hair". (16) An image processing apparatus according to (16), wherein the hair recognizing means includes a bangs feature extracting means for extracting the features of the forehead part, and a back hair feature extracting means for extracting the features of the back hair part, and the image including the hair part. Is input, and a bangs part is determined by using the bangs feature extracted by the bangs feature extraction unit and the back hair feature extracted by the back hair feature extraction unit.

According to the configuration (16), the bangs part is determined by using the characteristics of both the bangs and the back hair. For example, if there is a division on the left side in the upper part of the hair, even the bangs part is left. By selecting an item that matches the "left division" that flows from the person, a more realistic portrait with less discomfort can be created. Further, since a hair part prepared in advance is used, the hair area is converted to a binarized image, for example, by a method of combining a binarized hair area with a small part representing the hair of the bangs. In many cases, a more beautiful portrait can be created as compared with a method in which part or all of the hair image is output as it is as part or all of the hair image. Is not directly visible to the user as output, so it is difficult to give a sense of incongruity. (17) An image processing apparatus according to claim 17, wherein the hair recognizing means includes a bangs feature extracting means for extracting the features of the bangs portion, and a back hair feature extracting means for extracting the features of the back hair portion. Is input, the back hair part is determined using the bangs feature extracted by the bangs feature extraction unit and the back hair features extracted by the back hair feature extraction unit.

According to the configuration (17), since the back hair part is determined by using the characteristics of both the front hair and the back hair, for example, if the left side of the forehead part of the forehead has a split, the back hair By selecting a part that matches the "left division" that has a division on the left side, a more realistic or less unnatural portrait can be created. Also, since hair parts prepared in advance are used, for example,
As in the technique of combining a binarized hair region with a small part representing the hair of the bangs, a part or all of the binarized image of the hair region is directly used as a part or all of the hair image. Compared to the output method, more beautiful portraits can be created in many cases, and even if there is an incompletely processed part, it will not be visible to the user as output as it is, so it is difficult to give a sense of incongruity . <Embodiment 2> This embodiment will be described with reference to FIGS. FIG. 36 is a schematic block diagram of an image processing apparatus for synthesizing a portrait image, FIG. 37 is a diagram of position coordinates at which the center position of each face part is to be arranged, and FIG. 38 is an example of the shape of a face outline. FIG. 39 shows an example of drawing a shadow by a conventional method, FIG. 40 shows an example of drawing a shadow according to the present invention, and FIG. 41 is a flowchart of a method of drawing a shadow included in the component arrangement means 408 of FIG.

First, referring to FIG. 36, the configuration of a portion relating to the composition of a portrait image of the present invention will be described.

The image input means 400 is a means for inputting an original image for making a portrait, such as a scanner, and the feature amount extracting means 401 is the first embodiment (first to ninth aspects of the present invention). The facial contour determining means 402 selects the corresponding facial part based on the characteristic quantity, and the facial part data storing means 403 stores the facial part image data. The determining unit 404 determines a face outline of the person from the extracted feature amount, and extracts information on the deformation and arrangement of the part corresponding to the outline from the part deformation / arrangement information storage unit 405. 406 is a means for correcting the arrangement information drawn from 405 to generate a more similar portrait based on the feature amount of 401, and the component deformation means 407 is a means for correcting the deformation information received from 406. Then, the face part data is transformed into a shape suitable for the face contour to the size of the face part or the like. The image output means 409 is a means for outputting a synthesized portrait image such as a CRT or a printer.
Numeral 10 denotes a unit for changing the arrangement position and the deformation ratio of the parts with respect to the synthesized portrait image. [Tenth Embodiment] (Embodiment 18) In this embodiment, the face features extracted by the feature amount extracting means 401 are used to determine the closest face contour part from the prepared face contour parts. Determined by 404. For example, face contour parts such as a standard face / small face / wide face / round face as shown in FIG. In this face outline part, the coordinates at which the center position of each face part is to be arranged and the deformation ratio of the part are stored in the part deformation / arrangement information storage means 405 in the form of a table or a function as shown in FIG. Have been. A detailed example table is shown below as a specific example.

X coordinate y coordinate x direction enlargement ratio y direction enlargement ratio Back hair ｛500, 500, 0.92, 1.00｝ left eyebrows ｛410, 670, 0.85, 0.85｝ right eyebrows 590, 670, 0.85, 0.85｝ Left eye ｛410, 615, 0.80, 0.80｝ Right eye ｛590, 615, 0.80, 0.80｝ Nose ｛500, 510, 0.85, 0. 85 口 mouth ｛500, 450, 0.95, 0.95｝ left ear ｛310, 600, 1.00, 1.00｝ right ear 690, 600, 1.00, 1.00｝ bangs ｛500, 500, 0.92, 1.00 ｘ x-coordinate and y-coordinate are places where individual parts should be placed.
This is an example in which the size of the entire portrait is 1000 × 1000 and the center of the face is (500, 500). The enlargement ratio is indicated by a ratio where the size of each part of the standard face is 1.00. The part position is closer to the center than the standard face, and the parts are specified slightly smaller, so that the part configuration is optimal for the shape of the fine face. The part deforming means 407 and the part arranging means 408 arrange the face parts on the face outline or on the lower surface (back hair or ears) based on the values in this table.

Therefore, the features of this embodiment are summarized as follows. (18) An image processing apparatus according to claim 18 has means for obtaining face part feature information of the size and shape of a face part in an image (feature amount extraction means) and a plurality of face part types corresponding to the feature information. A face part data storage unit for storing a plurality of part data for each face part type; and a face part data extraction unit for extracting appropriate part data from the face part data storage unit based on the face part characteristic information. Means for arranging another face part at a position suitable for the contour by determining the arrangement position of the part for each type of face outline part stored in the face part data storage unit with the extracted face part data. Means (component arranging means).

According to the above configuration (18), after deciding which face part to use, such as the size and shape of the face part in the image, the component arrangement position is determined for each face contour part in the face part. And a part placement method such as a part size are determined, and each face part data is placed. By arranging the parts based on the face outline, not only the parts do not protrude from the face but also the face parts can be arranged at the position and size most suitable for the shape of the face outline. Determining facial part placement information for each facial contour to generate a facial caricature without disintegration when facial parts are placed, even if the facial balance is completely different due to the concept of portraits such as dramatic and comic styles Is possible. [Eleventh Embodiment] (Embodiment 19) In this embodiment, FIG. 22 showing a feature amount used for face part arrangement correction and FIG. 21 showing a processing flow of the correction amount calculation
This will be described with reference to FIG.

With reference to the width 151 of the face outline and the position (height) 156 of the eyes in the face image, the distance 153 between the centers of the eyes with respect to the width of the face outline (width) and the distance (width) between the centers of the eyebrows
152, distance (height) between the center of the eyebrow and the center of the eyebrow
154, distance (height) 15 between eyes, center of eyes and center of mouth
5 are obtained, the values of several hundred persons are measured in advance, the average is stored, and the ratio is calculated by comparing and dividing the result with the recognition result of the face part position in the face image (149). Specifically, the ratio Rew of the average Mew of the distance (width) 153 between the centers of the eyes to the width of the face contour and the detection result Dew, the average Mebw of the distance (width) 152 between the centers of the eyebrows, and the detection result Debw
Ratio Rebw, the average Mebh of the distance (height) 154 between the center of the eye and the center of the eyebrow and the center of the eyebrow and the ratio Rebh of the detection result Debh, the distance (height) between the center of the eye, the center of the eye and the center of the mouth
The ratio Rmh between the average Mmh of 155 and the detection result Dmh is calculated.

Rew = Dew / Mew Rebw = Debw / Mebw Rebw = Debh / Mebh Rmh = Dmh / Mmh Thus, the feature of the face part position of the face in the face image is extracted. Next, a correction amount of the arrangement position of the face part is calculated based on the extracted feature of the face part position. In the present embodiment, the respective ratios obtained above are multiplied by respective constants determined in advance in consideration of the degree of emphasis and the like,
The obtained value is used as the correction amount of the face part arrangement position in the composite image (150).

That is, a constant for determining the vertical position correction amount of the eyebrows Kebh A constant for determining the horizontal position correction amount of the eyebrows Kebw A constant for determining the horizontal position correction amount of the eyebrows The vertical position of the Kew mouth The constant Kmh for determining the correction amount is multiplied by each of the ratios obtained above, and the vertical position correction amount of the eyebrows Hebh The horizontal position correction amount of the eyebrows Hebw The horizontal position correction amount of the eye Hew The vertical position of the mouth The correction amount Hmh is obtained.

Hebh = Rebh × Rebh Hew = Rebw × Rebw Hew = Rew × Rew Hmh = Rmh × Rmh The layout of the facial parts is corrected based on the correction amount of the layout of the facial parts obtained as described above, and a portrait image is synthesized. By performing the above, it is possible to synthesize a portrait image in which the arrangement is suitable for the face contour part to be used and at the same time, the feature of the face part position of the face in the face image is reproduced or emphasized.

Therefore, the characteristics of this embodiment are summarized as follows. (19) An image processing apparatus according to claim 19, further comprising: means () for obtaining information on the position of the face part in the image, and another face determined corresponding to the face outline based on the obtained position information of the face part. It is characterized by comprising means (part arrangement means) for correcting the arrangement position of the parts (arrangement position correction means) and determining the arrangement position of the face parts.

According to the configuration of (19), all face variations having the same face contour but subtly different face part positions are supported. The arrangement position of the other face parts determined in step 18 is corrected within the allowable range of the face contour, and a more appropriate arrangement position of the parts is determined. Further, the position of the face part extracted from the input image is compared with a statistical standard position in the face contour of the face part, and the difference and the correction allowable range determined for each face contour determine the arrangement of the face part. Is corrected. This makes it possible to reflect the delicate feature of the position of the face part of the input image on the portrait while avoiding a failure such that the face protrudes from the face contour. [Twelfth embodiment] (Invention of claim 20) In the face part data storage means 403 of FIG. 36, the front hair part and the back hair part are stored as completely different parts, and each of them is human hair. There are provided hair color parts such as black / brown / white / gold so that the color can be easily expressed. The normal human hair color is the same for both the forehead and the back hair, and the feature amount extraction unit 401 uses the face part data extraction unit 4.
Since a feature amount indicating that the two colors are the same is passed to 02, no problem occurs in the generation result. However, if an instruction to change the color of the bangs is given by the edit designation input means 410, it will be inconsistent with the back hair. Therefore, when the bangs are instructed to change to a part having a different color, the face part data extracting means 40
2 automatically changes the color of the back hair to a part of the same color at the same time. This can prevent the inconsistency that the color of the forelock and the back hair is different. Conversely, when a color change instruction is given to the rear hair, the color of the front hair is automatically changed. Such simultaneous color change of multiple parts is not only for bangs and back hair,
This may be performed for all parts that need to be unified to the same hair color, such as eyebrows and mustaches. Also, assuming that the face parts also have the same constraint conditions, if a change of the skin color of the face outline from fair skin to tan is instructed, the ear and nose parts are also changed to the corresponding skin color parts. Embodiments are also conceivable. In addition, it is conceivable that not only the color but also an item having a strong correlation between the bangs and the back hair, such as whether the hair is perm, may be similarly changed in conjunction with each other.

Therefore, the features of this embodiment are summarized as follows. (20) In the image processing device according to the twentieth aspect, when a change of a part related to a specific color of a set of related parts is designated in the face part data arranged at the time of portrait composition, the other related parts are Also, a means (part deforming means) for automatically generating data which does not cause inconsistency by automatically changing parts is provided.

According to the configuration of the above (20), the color of one of the parts having a strong correlation between the face parts such as the bangs and the back hair or the bangs and the mustache is changed to, for example, black / brown / white. If it is determined that the color of the other part is automatically changed to the same color, a portrait without any discomfort is automatically generated without explicitly specifying the change of the color of the other part. Similarly, for a shape that has a strong correlation between the front hair and the back hair, such as whether the hair is perm, an editing operation without a sense of incongruity can be performed by reflecting an instruction for one on the other. [Thirteenth Embodiment] (Invention of Claim 21) Each face part stored in the face part data storage means 403 of FIG. 36 is internally stored as an image having a layered structure called a shadow. . If a drawing is performed by simply combining components having this layer structure, as shown in FIG. 39, a shadow of another component overlaps a certain component and is drawn, resulting in an unnatural picture. Therefore, when arranging each part, as shown in the flowchart of FIG. 41, the color of the pixel or the layer structure constituting each part is checked, and if the color is a shade color of a flesh color (for example, the RGB value is r = 238, g = 160, b = 87), first, only the pixel or layer structure corresponding to the shadow color of the component is drawn in advance, and then the pixel or layer of a non-shadow color is drawn. This makes it possible to generate a natural image as shown in FIG. This is possible because the number of colors used in the face part is extremely limited.

Furthermore, if only the skin color shadow layer has no face outline under the pixel area to be drawn, the drawing itself is suppressed, so that only the skin color shadow is drawn on the back hair or the background. It is also possible to prevent unnaturalness.

Therefore, the features of this embodiment are summarized as follows. (21) In the image processing apparatus according to claim 21, when one face part data has two or more layer structures among the face part data stored in the face part data storage unit, the other When combined with the face part data, there is provided means (part arranging means) for changing the order of layers based on the color information in the part and determining the arrangement order of the part and the layers constituting the part in an appropriate order. It is characterized by becoming.

According to the above arrangement, in each pixel or region of a face part or a part constituting a portrait such as a hat, it is determined from the color information that a shadow projected on the face outline is formed. Pixels or regions are extracted, and the pixels or regions are drawn in advance, and then individual face parts are drawn, thereby preventing a part from shadowing another part. Further, when there is no component to which a shadow is projected, by stopping the drawing of the pixel or the region itself, drawing at a place where no face outline exists can be prohibited.

Based on the contents described above,
When performing image processing to extract the characteristic amount of a specific object in an image, it does not take a long time to process or extract an incorrect characteristic amount, and directly specifies the processing range using a position specifying unit or the like. In this case, it is possible to provide an image processing apparatus that can robustly, accurately and quickly extract an arbitrary feature amount in an image without specifying many positions, and can easily synthesize a high-quality portrait.

In the above description, the target of image processing is described as a face. However, the present invention is not limited to the face, and various arbitrary images including, for example, a picture of a picture, a picture taken by a personal computer, a digital camera, and a video, and the like. The present invention can also be applied to an object by changing the role and function of the constituent means and parts.

The contents in each of the embodiments described so far are not limited to the contents described above unless the gist of the present invention is changed.

[0240]

The image processing apparatus according to the present invention has the following effects in each claim.

According to the first aspect of the present invention, when the relationship between the position and the size of the object arranged in the image satisfies a certain constraint condition in the image input from the outside, an arbitrary position in the image is By inputting the position of one or more of the objects in the image using the position specifying means that can specify the object, an appropriate search range for performing image processing for extracting the feature amount of the object is set. can do. This makes it possible to extract feature values robustly, with high accuracy, and at high speed. In addition, since the same effect can be obtained with a smaller number of designated points as compared with the case where the search range is designated by using the position designation means, there is an effect of reducing the amount of work.

According to a second aspect of the present invention, an image input from the outside includes a face, and the extracted feature quantity is an eye,
It is characterized by the position, size, shape, and the like of at least one corresponding facial part in the facial parts such as the nose, mouth, eyebrows, ears, contours, and hair. Therefore, the relationship between the position and the size of each face part can be immediately applied by examining a plurality of faces in advance, and when constructing an application using the feature amount, the processing is robust and highly efficient. Accurate and high-speed operation is possible. Further, the same effect can be obtained with fewer designated points as compared with the case where the search range is designated by using the position designation means. This has the effect of reducing the amount of work, and has the effect of increasing the convenience of the application.

In claim 3 of the present invention, when performing binarization to obtain a region to be recognized, binarization is performed using a plurality of methods and a plurality of thresholds, and the position and size of the region in the image are determined. , Shape, and the like, and selecting the most reliable image, the recognition target can be detected with high accuracy.

According to a fourth aspect of the present invention, the position of two or more face parts is specified by the position specifying means, and the size of the face part is predicted from the distance relationship between the specified positions, thereby achieving high precision. The position and size of the face part can be detected.

According to a fifth aspect of the present invention, utilizing the fact that the size of the eyes is within a certain range, the positions of two or more face parts are specified by the position specifying means, and the distance between the specified positions is specified. In order to predict the size of the face part from the relationship or to perform binarization in order to obtain a region to be recognized, binarization is performed using a plurality of methods and a plurality of thresholds, and the position and size of the region in the image are determined. Judging the shape, etc., and selecting the most reliable image, or detecting the position of both eyes, rotating the face image so that the line connecting the left and right eyes is horizontal, or any of the above By arbitrarily combining the above methods, the recognition target can be detected with high accuracy.

According to the sixth aspect of the present invention, it is possible to improve the face part detection accuracy by detecting the positions of both eyes and rotating the face image so that the line connecting the left and right eyes is horizontal. become.

According to the seventh aspect of the present invention, an image feature representing the inclination and thickness of the eyes is detected within the set search range, and the shape of the eyes is determined. Due to the shift from the corresponding partial image in the input image, it is possible to avoid the danger that an erroneous feature amount is extracted. further,
When the shape of the target eye is a shape that is not included in the category prepared in advance, it is possible to avoid a risk that a correct feature amount cannot be calculated. Furthermore, it is necessary to prepare a template image and a dictionary image, and the amount of work can be significantly reduced.

According to an eighth aspect of the present invention, utilizing the fact that the size of the mouth is within a certain range, the positions of two or more face parts are specified by the position specifying means, and the distance between the specified positions is specified. In order to predict the size of the face part from the relationship or to perform binarization in order to obtain a region to be recognized, binarization is performed using a plurality of methods and a plurality of thresholds, and the position and size of the region in the image are determined. Judging the shape, etc., and selecting the most reliable image, or detecting the position of both eyes, rotating the face image so that the line connecting the left and right eyes is horizontal, or any of the above By arbitrarily combining the above methods, the recognition target can be detected with high accuracy.

According to a ninth aspect of the present invention, the positions of two or more face parts are specified by the position specifying means, the size of the eyebrows is predicted from the distance relationship between the specified positions, and the characteristic amount relating to the eyebrows is extracted. In addition to restricting the range in which the processing is performed to an appropriate size, the size of the eyebrows can be estimated. That is, the size of the region separated when binarization is performed is compared with the estimated size of the eyebrows,
By obtaining a binarization threshold value such that their sizes are not so far apart, an area representing an eyebrow can be detected with high accuracy.

According to the tenth aspect of the present invention, an image feature representing the manner in which the eyebrows are bent and the thickness of the eyebrows is detected within the set search range, and the shape of the eyebrows is determined. It is possible to avoid the danger that an erroneous feature amount is extracted due to the difference between the image and the corresponding partial image in the input image. Further, when the shape of the target eyebrow is not included in the category prepared in advance, it is possible to avoid a risk that a correct feature amount cannot be calculated. Furthermore, it is necessary to prepare a template image and a dictionary image, and the amount of work can be significantly reduced.

According to the eleventh aspect of the present invention, more stable jaw detection is performed even for an image in which the photographing conditions are poor and a clear contour line does not appear or for an image with much noise.
It is possible to perform a jaw shape determination in which the effects of noise, individual differences, and the like are eliminated as much as possible.

According to the twelfth and thirteenth aspects of the present invention, it is not necessary to extract the background area from the entire image, so the background color does not need to be uniform or close to it, and the hair color can be extracted even from a normal snapshot. Thus, a portrait can be created.

According to the fourteenth aspect of the present invention, in order to extract the outline of the hair instead of using template matching, the "separation" referred to as a so-called "seven-three division" or "central division" is used. Detecting and classifying well, and determining the roundness of the hairline shape,
It is possible to perform fine shape classification such as classification into “round type”.

In claim 15 of the present invention, for example,
If the height of the top of the face contour line is lower than a certain threshold value by comparison with the head height, it is determined that there is a considerable amount of hair. Erroneous judgments such as "thin hair" can be eliminated or reduced.

In the sixteenth aspect of the present invention, since the bangs part is determined using both the bangs and the back hair characteristics, for example, if there is a division on the left side in the upper part of the hair, the bangs part is also left. By selecting one that matches the "left division" that flows from one side, a more realistic portrait with less discomfort can be created. Further, since a hair part prepared in advance is used, the hair area is converted to a binarized image, for example, by a method of combining a binarized hair area with a small part representing the hair of the bangs. In many cases, a more beautiful portrait can be created as compared with a method in which part or all of the hair image is output as it is as part or all of the hair image. Is not directly visible to the user as output, so it is difficult to give a sense of incongruity.

According to the seventeenth aspect of the present invention, since the back hair part is determined by using the characteristics of both the bangs and the back hair, for example, if there is a split on the left side in the forehead part of the forehead, the back hair part However, by selecting an item that matches the "left division" in which there is a division on the left side, a more realistic caricature with less discomfort can be created. Further, since a hair part prepared in advance is used, the hair area is converted to a binarized image, for example, by a method of combining a binarized hair area with a small part representing the hair of the bangs. In many cases, a more beautiful portrait can be created as compared with a method in which part or all of the hair image is output as it is as part or all of the hair image. Is not directly visible to the user as output, so it is difficult to give a sense of incongruity.

According to the eighteenth aspect of the present invention, while the features of each part of the face obtained by the feature amount extracting means are reflected to the maximum extent, a constraint condition based on the shape of the face contour is added to remove the excessive features. It is possible to prevent the reflection and automatically synthesize a portrait that is optimal and has no breakdown.

According to a nineteenth aspect of the present invention, a means for obtaining information such as the position of a face part in an image, and a decision corresponding to the face contour according to the eighteenth aspect, based on the obtained position information of the face part. Means for correcting the arrangement position of the other face parts, and determining the more appropriate arrangement position of the parts, in addition to the effect of claim 18, in addition to the effect of the arrangement of the face parts of the face in the image, It is possible to combine the emphasized portrait image.

[0259] According to a twentieth aspect of the present invention, by using a constraint condition relating to colors in a portrait, generation of an image that is inappropriate as a portrait is suppressed, and a color change designation of an extra part is given to the user. Can be reduced.

According to a twenty-first aspect of the present invention, a natural image is obtained by automatically determining or suppressing the drawing order of layers constituting a part without unnecessarily complicating the data of the stored face part. Can be automatically generated.

[Brief description of the drawings]

FIG. 1 is a schematic block diagram of an image processing apparatus according to an embodiment of the present invention.

FIG. 2 is a diagram illustrating an example of position designation and an example of a search range determined by a designated position.

FIG. 3 is a diagram showing an example of an eye search range for recognizing an eye shape, a histogram, and detected positions of the inner and outer corners of the eye.

FIG. 4 is a diagram illustrating an example of a template for detecting an inner corner of the eye;

FIG. 5 is a diagram illustrating an example of an eye search range for detecting eye thickness, a skin color sampling region, and a skin / non-skin region.

FIG. 6 is a flowchart showing an operation for obtaining an inner and outer corner search range.

FIG. 7 is a flowchart showing an operation for detecting an inner corner position.

FIG. 8 is a flowchart showing an operation for detecting the position of the outer corner of the eye.

FIG. 9 is a diagram illustrating an example of an eyebrow search range and a binarized image for detecting the position and size of eyebrows.

FIG. 10 is a flowchart showing an operation for detecting a position and a size of an eyebrow.

FIG. 11 is a diagram illustrating an example of a rectangle circumscribing eyebrows.

FIG. 12 is a diagram illustrating an example of quantization for recognizing the shape of eyebrows.

FIG. 13 is a flowchart showing an operation for detecting how an eyebrow is bent.

FIG. 14 is a flowchart showing an operation for detecting the thickness of eyebrows.

FIG. 15 is a flowchart showing an operation of mouth detection.

FIG. 16 is a diagram illustrating an example of a mouth detection image and a projection result.

FIG. 17 is a flowchart illustrating an eye detection operation.

FIG. 18 is a diagram illustrating an example of an eye detection image and conversion and extraction results.

FIG. 19 is a flowchart showing an operation of tilt correction using an eye detection result.

FIG. 20 is a diagram illustrating an example of tilt correction using an eye detection result.

FIG. 21 is a flowchart illustrating an operation of calculating a face part arrangement correction amount.

FIG. 22 is a diagram illustrating feature amounts used for face part arrangement correction.

FIG. 23 is a flowchart schematically showing an operation of the image processing apparatus according to the eleventh embodiment of the present invention.

FIG. 24 is a diagram for explaining the arrangement of center coordinates and initial contours in an input image in the embodiment of claim 11 of the present invention.

FIG. 25 is a diagram for explaining a method of calculating a color difference on a straight line connecting a point on the initial contour and the center coordinates in the embodiment of claim 11 of the present invention.

FIG. 26 is a diagram schematically showing an example of calculating a color difference in the embodiment according to claim 11 of the present invention.

FIG. 27 is a diagram for explaining a case where the ellipse shape of a face is used as a method for performing a color difference calculation specialized for a face contour shape according to the eleventh embodiment of the present invention.

FIG. 28 is a diagram for explaining a case of utilizing the fact that a face is bilaterally symmetric with respect to a center axis as a method for performing color difference calculation specialized for a face contour shape according to an embodiment of the present invention. is there.

FIG. 29 is a diagram for explaining a method of calculating a distance function from an extracted face contour in the embodiment of claim 11 of the present invention.

FIG. 30 is a diagram for explaining a method for comparing a distance function obtained from an input image with a reference distance function according to the eleventh embodiment of the present invention.

FIG. 31 is a block diagram showing a configuration of an image processing apparatus according to claims 12 to 16 of the present invention.

FIG. 32 is a flowchart showing processing of the image synthesizing apparatus according to claims 12 to 16 of the present invention.

FIG. 33 is an explanatory diagram related to hair color extraction.

FIG. 34 is an explanatory diagram related to bangs classification.

FIG. 35 is an explanatory diagram relating to back hair classification.

FIG. 36 is a schematic block diagram of an image processing apparatus relating to the synthesis of a portrait image according to another embodiment of the present invention.

FIG. 37 is a diagram of position coordinates where the center position of each face part is to be arranged.

FIG. 38 is an example of the shape of a face contour.

FIG. 39 is an example of shadow drawing by a conventional method.

FIG. 40 is a drawing example of a shadow according to the present invention.

FIG. 41 is a flowchart of a shadow drawing method included in the component placement unit 408 of FIG. 36.

[Explanation of symbols]

 Reference Signs List 10 feature amount extraction unit 11 input device 12 storage device 13 arithmetic device 14 position designation device 15 output device 400 image input means 401 feature quantity extraction means 402 face part data extraction means 403 face part data storage means 404 face contour determination means 405 part deformation / Placement information storage means 406 placement position correction means 407 part deformation means 408 part placement means 409 image output means 410 edit designation input means

Continuation of the front page (72) Inventor Sou Takezawa 22-22, Nagaike-cho, Abeno-ku, Osaka-shi, Osaka Inside (72) Inventor Yoshinori Nagai 22-22, Nagaike-cho, Abeno-ku, Osaka-shi, Osaka (72) Inventor Ai Ito 22-22 Nagaike-cho, Abeno-ku, Osaka-shi, Osaka Inside (72) Inventor Minehiro Konya 22-22 Nagaike-cho, Abeno-ku, Osaka-shi, Osaka F-term (Reference) 5B050 BA06 BA12 CA07 EA09 EA12 EA13 EA19 FA09 FA19 5L096 BA18 FA06 FA15 FA69 GA38 GA51 JA22 LA05

Claims (21)

[Claims] 1. An input unit for inputting an image, a storage unit for storing the input image, an arithmetic unit for performing an arbitrary operation, and a position specifying unit for specifying an arbitrary position in the image. Recognizing the position and size of an object placed in the image, and inputting the position of one or more of the objects in the image when the relationship between the position and size of the object satisfies a certain constraint. And a feature extraction unit for extracting an arbitrary feature in the image. 2. The image processing apparatus according to claim 1, wherein the feature amount extracting means includes a face, an eye, a nose, a mouth, an eyebrow, an ear,
The contour and hair are face parts, and at least one of the face parts
2. The image processing apparatus according to claim 1, wherein the position, the size, and the shape of the corresponding face part are extracted as feature amounts. 3. The feature amount extracting unit binarizes an input image by a plurality of methods and a plurality of thresholds, determines the position, size, and shape of a region in the image, and determines the highest reliability. 3. The image processing apparatus according to claim 2, further comprising an area detecting unit that detects an area to be recognized by selecting an image. 4. The feature quantity extracting means predicts the size of the face part from the distance relationship between the positions of two or more face parts specified by the position specifying means, thereby obtaining the position and size of the face part. 3. The image processing apparatus according to claim 2, further comprising a face part recognizing means for detecting the height. 5. The image processing apparatus according to claim 4, wherein the target of the position and size to be detected by the face part recognition means is an eye of the face part. 6. The face part recognizing means, wherein a line connecting left and right eyes is horizontal with respect to a detected position of both eyes.
6. The image processing apparatus according to claim 4, further comprising: means for rotating a face image. 7. The face part recognizing means sets a search range based on the detected position and size of the eyes, detects image features representing the inclination and thickness of the eyes in the range, and determines the shape of the eyes. 5. The apparatus according to claim 4, further comprising:
7. The image processing apparatus according to any one of claims 1 to 6. 8. The image processing apparatus according to claim 4, wherein said face part recognition means detects a position and a size of a mouth of the face part. 9. An image processing apparatus according to claim 4, wherein said face part recognizing means detects a position or a size of an eyebrow of the face part. 10. The face part recognizing means sets a search range based on the detected position and size of the eyebrow, detects image features representing the thickness and how to bend the eyebrow within the range, and detects the shape of the eyebrow. The image processing apparatus according to claim 4, further comprising: means for determining 11. The method according to claim 1, wherein the feature extracting means includes a contour recognizing means for detecting a contour feature of a jaw and determining a shape thereof based on one or more position information designated by the position designating means. 3. The image processing apparatus according to claim 2, wherein: 12. A hair recognizing means for estimating a head height and a hairline height based on one or more position information specified by the position specifying means, and recognizing a hair region. 3. The image processing apparatus according to claim 2, wherein: 13. The image processing apparatus according to claim 12, wherein said hair recognizing means includes a hair color extracting means for extracting a hair color. 14. A hair feature extracting means for extracting a feature of a hair portion based on one or more pieces of position information specified by the position specifying means, and a hair using the feature of the hair portion. 13. The image processing apparatus according to claim 12, further comprising a hair contour extracting means for extracting a contour, and a hair classifying means for classifying the hair using the hair contour. 15. The hair recognizing means includes a face contour feature extracting means for extracting facial contour features, and a hair classifying means for classifying hair using the hair features and the facial contour features. The image processing apparatus according to claim 12, wherein: 16. The hair recognizing means includes bangs feature extracting means for extracting features of a bangs part, and back hair feature extracting means for extracting features of a back hair part, and when an image including a hair part is inputted, 13. The image processing apparatus according to claim 12, wherein a bangs part is determined using the bangs feature extracted by the bangs feature extraction unit and the back hair feature extracted by the back hair feature extraction unit. 17. The hair recognizing means includes bangs feature extracting means for extracting features of a bangs portion, and back hair feature extracting means for extracting features of a back hair portion, and when an image including a hair portion is input, 13. The image processing apparatus according to claim 12, wherein a back hair part is determined by using a bangs feature extracted by the bangs feature extraction unit and a back hair feature extracted by the back hair feature extraction unit. . 18. A means for obtaining face part feature information of the size and shape of a face part in an image, a plurality of face part types corresponding to the feature information, and storing a plurality of part data for each face part type. Face part data storage means, a face part data extraction means for extracting appropriate part data from the face part data storage unit based on the face part characteristic information, and a face part data An image processing apparatus comprising means for arranging another face part at a position suitable for the contour by determining the arrangement position of the part for each face outline part type stored in the part data storage unit. . 19. A means for obtaining information on the position of a face part in an image, and based on the obtained position information on the face part, correct the arrangement position of another face part determined corresponding to the face outline, and 19. The image processing apparatus according to claim 18, further comprising: means for determining an arrangement position of the component. 20. In the facial part data to be arranged at the time of portrait composition, when a part change designation is made for a specific color of a set of related parts, the parts are automatically changed also for the other related parts. 19. The image processing apparatus according to claim 18, further comprising means for generating data that does not cause inconsistency. 21. When one face part data has two or more layers in the face part data stored in the face part data storage unit, the face part data is combined with another face part data. 19. The apparatus according to claim 18, further comprising means for changing the arrangement order of the layers based on the color information in the parts and determining the arrangement order of the parts and the layers constituting the parts in an appropriate order.
The image processing apparatus according to any one of the preceding claims.