JP4530173B2 - Method and system for detecting the position of a facial part - Google Patents

Description

  The present invention relates to a method for automatically detecting positions of facial parts such as eyes, mouths, and noses from facial images.

  Detection of the position of facial parts such as eyes, mouth, and nose in the facial image is performed by overlaying color images on the eyes, mouth, nose, other facial parts of the subject's facial image, and their surrounding areas Is needed when forming.

  In addition, the detection of the eye position is performed to adjust the size and position of the image when a hairstyle simulation image is formed by overlaying the model image of the hairstyle on the face image of the subject. It is also used in image blindfolding, face image frame processing, creation of personal authentication images, and the like.

Conventionally, as a method for detecting an eye position from a face image, a method of detecting a face by extracting a skin color region and further detecting an eye by pattern matching or the like has been used (Non-Patent Document 1, Patent Document 1). ).

2005 5th LSI IP Design Award Winning Paper “Face candidate point detection method that enables fast and reliable face detection” (Nikkei Business Publications) JP 2004-94917 A

  However, the extraction of the skin color area is affected by the lighting environment. Therefore, reliability becomes a problem when an unspecified number of face images taken in various lighting environments are targeted. Further, the pattern matching method has a problem that the amount of calculation becomes enormous.

  On the other hand, an object of the present invention is to enable automatic and reliable detection of positions of eyes, mouth and nose from a face image.

  When the present inventors create a plurality of grayscale images with successively changing brightness from the face image and observe the images that are faded in sequentially from the faded-out images on the high brightness side, If the appearing pixels belong to the pupil region, in this case, the pixels in the pupil region appear in pairs, and if the frequency of appearance over all grayscale images of the pixel regions that appear in pairs is accumulated, The position of the eyes can be specified, and from among the cluster of pixels obtained when the images faded in sequentially from the faded-out image are observed, the one having a specific relationship with the eye position is selected and It was found that the position of the mouth and the position of the nose can be specified by adding conditions for the positional relationship.

That is, according to the present invention, a plurality of grayscale images whose brightness has been sequentially changed are created from the face image, and the faded-out face region is faded in from the state where the grayscale image has faded out from the high brightness to the low brightness. A method of detecting gradually appearing pixel mass regions (hereinafter referred to as a fade-in method) is performed, and among the detected pixel mass regions, those appearing in pairs are selected as eye position candidates. Identify eye positions based on frequency of occurrence across all grayscale images of position candidates,
Groups of pixels detected by the fade-in method that have gradually stopped growing, grouped by overlapping vertical positions,
A circumscribed rectangle is formed for each grouped pixel cluster area,
Out of the circumscribed rectangles, those within a predetermined region predicted from the positional relationship with both eyes are considered as mouth position candidates,
Provided is a method for detecting the position of a face part that identifies a rectangle having a maximum width among circumscribed rectangles as mouth position candidates as a mouth position.

Further, the present invention forms a circumscribed rectangle for each of the clustered pixels detected by the fade-in method after the position of the mouth is determined by the above-described method, and the growth of pixels that gradually appear is stopped,
Among the circumscribed rectangles, those within a predetermined region predicted from the positional relationship with the positions of both eyes are the nose position candidates,
Among the circumscribed rectangles that are nose position candidates, the method of detecting the position of the face part that identifies the nose position with the lower side above the upper side of the circumscribed rectangle identified as the position of the mouth and the largest width I will provide a.

Furthermore, the present invention is a face part position detection system comprising a face image acquisition means and a calculation means, wherein the calculation means comprises:
The ability to create multiple grayscale images with varying brightness from face images,
A function that detects a cluster area of pixels that gradually appear in the faded-out face area as the grayscale image fades out from a state that fades out at high brightness to low brightness.
A function of selecting a clustered region of detected pixels that appears as a pair as an eye position candidate,
The ability to identify eye positions based on the frequency of appearance across all grayscale images of each eye position candidate,
A function of grouping the detected pixel cluster areas where the growth of the pixels that appear gradually stops and the vertical positions overlap,
A function to form a circumscribed rectangle for each grouped pixel cluster area,
A function that makes a candidate for the position of the mouth among the circumscribed rectangles of the grouped pixel cluster area within a predetermined area predicted from the positional relationship with both eyes,
A function that selects the circumscribed rectangle as the mouth position candidate having the maximum width as the mouth position,
A function to form a circumscribed rectangle for each detected pixel cluster area where the gradually appearing pixel growth has stopped,
A function that makes a nose position candidate a circumscribed rectangle for each cluster area of detected pixels within a predetermined area predicted from the positional relationship with both eyes,
Among the circumscribed rectangles that are candidates for the position of the nose, the detection system has a function of identifying the one with the maximum width as the position of the nose with the lower side above the upper side of the circumscribed rectangle identified as the position of the mouth I will provide a.

  According to the face part position detection method or detection system of the present invention, first, a cluster area of pixels obtained by the fade-in method is detected as an eye position candidate, and the appearance frequency over the entire gray scale image of the eye position candidate is detected. The position of the eye is specified based on. Next, a group of pixels obtained by the fade-in method, in which the growth of the gradually appearing pixels is stopped, is grouped, and a circumscribed rectangle of the grouped pixel cluster area is assumed. The position of the mouth is specified from the positional relationship and the width of the circumscribed rectangle. Furthermore, assuming a circumscribed rectangle that is a clustered area of pixels obtained by the fade-in method, and the growth of pixels that gradually appear is stopped, the position of the circumscribed rectangle and the eye, the position of the mouth, and the width of the circumscribed rectangle To identify the position of the nose. Therefore, regardless of the lighting environment, the skin color of the subject, the color of the pupil of the subject, the orientation of the face in the face image, etc., the positions of the eyes, mouth, and nose can be detected with high reliability, and pattern matching is unnecessary. Therefore, the positions of eyes, mouth, and nose can be detected by high-speed calculation processing.

  Therefore, the subject forms a hairstyle simulation image that fits various hairstyle images to the face image of the subject using the eye position as a reference, and the facial parts such as the eyes, nose, and mouth after makeup, and images of their surrounding areas. In the formation of a makeup simulation image to be fitted to the face image, a natural simulation image can be automatically and simply formed using an arithmetic device.

  Hereinafter, the present invention will be described in detail with reference to the drawings.

  FIG. 1 is a flowchart of a method according to an embodiment of the present invention for sequentially detecting the positions of eyes, mouth, and nose, and FIG. 2 is a block diagram of a system for detecting the position of a facial part that implements this method.

  The system 10 includes an imaging apparatus 1 and a personal computer main body 2, and a display 3, an image scanner 4, a printer 5, and the like are connected to the personal computer main body 2.

  The imaging device 1 is provided as means for acquiring a face image of a subject, and a commercially available digital camera can be used.

  The personal computer main body 2 has an image processing function for creating a plurality of gray scale images whose brightness has been changed successively (for example, one in which pixel values are changed by 2 to 3 gradations in a 256 gradation gray scale image layer). ing.

Gray scale image creation methods include (1) a method using the average of the maximum and minimum values of R, G, and B for each pixel, and (2) the average of each value of R, G, and B. (3) There is a method in which each value of R, G, B is multiplied by a predetermined weighting coefficient (NTSC, etc.) and then averaged, etc. In the present invention, In the method, for example, the following formula Y (lightness) = 0.298912 × R + 0.568811 × G + 0.114478 × B
It is preferable to change the gradation by 2 to 3 gradations using Such a gray scale image processing function can be obtained, for example, by installing in the personal computer main body 2 commercially available image processing software such as Adobe System Photoshop.

  Further, the personal computer main body 2 has a function of changing the contrast in accordance with the change in brightness (for example, a function of decreasing the contrast as the brightness is increased and increasing the contrast as the brightness is decreased), and the grayscale image hierarchy. A function that assigns list numbers in order of low or high brightness, and in the grayscale image hierarchy, pixels that gradually appear in the faded-out face area as the brightness fades from low fade to high brightness A function for detecting the cluster area of the detected pixel, a function for selecting the detected pixel cluster area as a pair of eye positions, and the frequency of appearance of each eye position candidate over the grayscale image of all layers A function that counts and identifies the eye position based on the value, and all grayscales for each eye position candidate A group that has the function of specifying the position of the eyes based on the frequency of appearance over the image, and that overlaps the vertical positions of the detected pixel cluster areas where the growth of the gradually appearing pixels has stopped A function to form a circumscribed rectangle for each grouped pixel cluster area, and a function to make a mouth position candidate a circumscribed rectangle of the grouped pixel cluster area in a specific positional relationship with both eyes , Among the circumscribed rectangles that are candidates for the position of the mouth, the function of specifying the largest width as the position of the mouth, and the function of forming a circumscribed rectangle for each cluster area of the detected pixels, Among the circumscribed rectangles for each mass area, a function that makes a specific positional relationship with both eyes a candidate for the position of the nose, and among the circumscribed rectangles that are candidates for the nose position, the lower side of the rectangle is the mouth position. And there above the upper side of the circumscribed rectangle of the horizontal width has a like function to identify the largest of the position of the nose.

  In the face part position detection method using the system 10, first, a face image, preferably an erect face image, for which the face part position is to be detected is acquired in the personal computer main body 2. The face image may be obtained, for example, by taking a face image of the subject with the imaging device 1 and importing the face image into the personal computer main body 2 or reading the face photograph of the subject with the image scanner 4, such as the Internet. You may acquire using a communication line.

  Next, if necessary, the number of pixels of the face image is adjusted to a size suitable for the face part position detection process. More specifically, if the number of pixels of the face image is too large, an excessive burden is imposed on the detection process, and if it is too small, the detection accuracy of the position of the face part decreases, so the image size is about 480 × 360 to 320 × 240. To change.

  In addition, if the face image is sharp, many fine edges will appear in the subsequent image processing, and this will cause noise in the detection of position candidates for each face part of the eyes, mouth, and nose. I do. In this case, it is preferable that the degree of blurring is weak to medium. For example, for each pixel, the weighting blur filter of 1 to 5 is used for the pixel of interest and the surrounding 25 pixels. Performs blurring to output the average value. This blurring process may be performed a plurality of times as necessary.

  Furthermore, in the face part position detection method of the present invention, color information is not necessary for the face image. Therefore, when the acquired image is a color image, it is converted to a grayscale image in order to reduce the amount of subsequent processing. Convert.

  Next, the eye position is detected in the acquired face image, but it is preferable to set a detection target area of the eye position in advance. For example, as shown in FIG. 3, a rectangular frame 21 is set as the eye position detection target region for the grayscale face image 20. As a method for setting the rectangular frame 21, assuming that 1/8 of the vertical size of the face image 20 is A, the area of 3 A length and 4 A width is the center O of the area, and the center coordinates of the face image 20 Here, the center coordinates of the face image 20 may be set at a position of 3.5 A from the upper side of the face image 20 on the horizontal center line of the face image 20. The method of setting the rectangular frame 21 as the eye position detection target region in this way can be applied to any upright image in which the upper body is shown.

  Next, as shown in FIG. 4, a plurality of grayscale images whose brightness has been sequentially changed are formed. More specifically, for example, in a grayscale image with 256 gradations, about 30 to 100 layers are created from an image with a pixel value of 0 that is completely faded out. The eye part always appears in 50 to 130 in the pixel values in this hierarchy. In this case, it is preferable to lower the contrast as the brightness is increased and to increase the contrast as the brightness is decreased, because the eye part appears more clearly in the fade-in image.

  As can be seen from FIG. 4, when the brightness of a grayscale image is increased and the faded-in image layer is formed by successively reducing the brightness from an image that has been completely faded out, normally, the cluster of pixels is usually first in the pupil. Is detected, and then a block of pixels is detected in the nose and mouth. Further, since the cluster of pixels in the pupil region appears as a pair of left and right, the group of pixels of the pair of left and right is selected. Depending on the hierarchy, pixel cluster areas appear in pairs in areas such as the mouth, eyebrows, and forehead in addition to the pupils. At this stage, these pixel cluster areas are also selected as eye position candidates.

  In the detection of the cluster area of pixels, it is preferable to reverse the gradation of the grayscale image whose brightness has been sequentially changed as shown in FIG. 5 in order to facilitate image processing.

  In addition, as a condition of the cluster of pixels to be detected as eye position candidates, when 1/8 of the vertical size of the grayscale face image 20 is A as described above, the size of the pixel cluster area is set to vertical 0. It is preferable to set within a range of 0.5 A to 0.7 A and lateral 0.7 A to 1.0 A. Usually, the size of the cluster area of the pixel at the eye position has a size within this range. By setting such a size condition, the size of the subsequent area required for detection of the eye position is set. The amount of processing can be reduced.

  FIG. 6 shows a circumscribed rectangle of a clustered area of pixels appearing in a face area in a fade-in image of a certain hierarchy, and further, a pair of these is connected by a straight line. In FIG. 6, the pixel cluster region is surrounded by a circumscribed rectangle for easy understanding of the pixel cluster region pair. However, when detecting the position of the eye, the pixel cluster region is not necessarily surrounded by the circumscribed rectangle. Not necessary.

It is preferable that the following (1) to (4) are set in advance as conditions for selecting a candidate eye position from among the clustered pixel regions.
(1) The lateral distance of the cluster area of a pair of left and right pixels is within a certain range.
(2) The vertical distance between the left and right pixel cluster areas is within a certain range.
(3) Of the paired left and right pixel cluster regions, the left pixel cluster region is in the left half region of the rectangular frame 21 of the face image.
(4) Of the paired left and right pixel cluster regions, the right pixel cluster region is in the right half region of the rectangular frame 21 of the face image.

  Next, as shown in FIG. 7, a pair of eye position candidates appearing for each fade-in image is accumulated over all layers of the grayscale image, the appearance frequency is counted, and the ranking is performed in descending order of the count number. Create a list. In this count, the eye position candidate that appears first when the brightness is gradually reduced from the faded-out face image and continues to appear until the final stage is normally the maximum count (ranking first). Therefore, in principle, the position of the eye position candidate with the maximum count is specified as the eye position.

However, if the following conditions (a) to (c) are all satisfied, or if the condition (d) is satisfied, the position of the eye position candidate with the second largest count is set as the eye position. As specified.
(a) When the second position candidate for the ranking is higher than the first position candidate for the ranking
(b) When the distance between the centers of the second-ranked position candidates is longer than the distance between the centers of the first-ranked position candidates
(c) When the left and right pupil regions of the second ranking eye position candidate are both outside the positions corresponding to the left and right pupils of the first ranking eye candidate
(d) The vertical distance between the first position candidate of the ranking and the second position candidate of the ranking is about the distance between the eyes and the eyebrows, and the first position candidate of the ranking is ranked first. When the position is above the second position candidate

  Of these, (a) to (c) are intended to prevent erroneous determination of the position of the mouth as the position of the eye, while the first ranking in the ranking may rarely be the mouth area. d) is for preventing erroneous determination that the eyebrows around the eyes are the positions of the eyes.

  As described above, as shown in FIG. 8, in the face image (original image before image processing), the eye position (more precisely, the position of the pupil) can be accurately detected.

  In the present invention, after detecting the position of the eyes, the position of the mouth and the position of the nose are sequentially detected. When the face image is tilted as shown in FIG. 9A, the position of the mouth and the position of the nose are detected. Prior to this, as shown in FIG. 9B, it is preferable to correct the inclination of the face image by rotating the coordinates of the face image so that the straight line connecting the eyes of both eyes is horizontal.

  Moreover, it is preferable to set in advance a rectangular frame as a detection target area of the mouth and nose in the face image. For example, as shown in FIG. 10, assuming that 1/8 of the vertical size is A for the face image 20, A is set in the vertical direction from the inverted regular triangle having both vertices as two vertices, and 0. A rectangular frame 21 ′ that is 5A wide is set.

  When detecting the position of the mouth, in the grayscale hierarchical image used for detecting the position of the eyes, the growth of pixels that gradually appear in the clustered area of the detected pixels, preferably within the set rectangular frame 21 ′. The stopped items are grouped according to their overlapping vertical positions. For example, as shown in FIG. 11, the pixel cluster regions A and B are grouped into one group C.

  When grouping pixel cluster areas, the maximum area after grouping is 1.1 times the distance between the eyes of the eyes in the horizontal direction and 0.6 times the distance of the eyes of the eyes in the vertical direction. Do not. Thereby, unnecessary noise can be excluded from the group of pixel cluster regions for detecting the position of the mouth.

  A circumscribed rectangle is formed for each grouped pixel cluster region, and the circumscribed rectangle has a specific relationship with the position of both eyes, more specifically, as shown in FIG. Draw an inverted equilateral triangle T as left and right vertices, draw a circle 40 centered on the lower apex of the inverted equilateral triangle T and whose diameter is 0.65 to 1.0 times the side length of the inverted equilateral triangle, On the other hand, when a circle 41 inscribed in the circumscribed rectangle 30 having the center of the circumscribed rectangle 30 of the group C as the center of the circle is drawn and this circle 41 overlaps the circle 40 described above, it corresponds to the circle 41 having the overlap. The circumscribed rectangle 30 to be used is set as a mouth position candidate.

  When there are a plurality of circumscribed rectangles as mouth position candidates, the circumscribed rectangle having the maximum width is specified as the position of the mouth.

  After specifying the circumscribed rectangle 30 as the position of the mouth in this way, the horizontal width of the specified circumscribed rectangle is set as the horizontal width of the pixels constituting the mouth.

  Here, when the width of the circumscribed rectangle selected as the mouth position does not cross the bisector of the straight line connecting the eyes of both eyes (the lower vertex of the inverted triangle), the bisector is the center. The circumscribed rectangle is expanded in the horizontal direction so that the circumscribed rectangle selected as the position of the mouth is symmetric, and the position of the mouth is specified by the expanded rectangle.

  If there is no circumscribed rectangle as the mouth position in the above processing, the eye position by the fade-in method is set as the next eye position candidate (the eye position of the next rank in the count ranking). ) And repeat mouth detection.

  In the present invention, after the position of the mouth is specified, the position of the nose is detected. When detecting the position of the nose, in the grayscale hierarchical image used in the detection of the eye position, preferably, as shown in FIG. 11, the cluster area of the detected pixels in the set rectangular frame 21 ′. The circumscribed rectangles 31, 32, 33, and 34 of the cluster region of each pixel are formed without specially grouping. Then, among the formed circumscribed rectangles 31, 32, 33, and 34, those having a specific relationship with the positions of both eyes, more specifically, as in the case of detecting the position of the mouth, as shown in FIG. Draw an inverted equilateral triangle T with the positions of both eyes at the left and right vertices, and center the lower vertex of the inverted equilateral triangle T, and the diameter is 0.5 to 0.7 times the length of one side of the inverted equilateral triangle. A circle 42 is drawn. On the other hand, a circle inscribed in each circumscribed rectangle 31, 32, 33, 34 having the center of each circumscribed rectangle 31, 32, 33, 34 as the center of the circle is drawn. When there is an overlap, circumscribed rectangles 32 and 33 corresponding to the circle having the overlap are used as nose position candidates.

  Of the circumscribing rectangles 32 and 33 as nose position candidates, the lower side is above the upper side of the circumscribing rectangle 30 specified as the mouth position, and the one with the largest width is the nose position. The position and width of the lower end of the nose are specified by the lower side of the circumscribed rectangle specified as the position of the nose.

  Here, when the width of the circumscribed rectangle specified as the position of the nose does not cross the bisector of the straight line connecting the eyes of both eyes (the lower vertex of the inverted triangle), the bisector is the center. The circumscribed rectangle is expanded in the horizontal direction so that the circumscribed rectangle selected as the nose position is symmetrical, and the position and width of the lower end of the nose are specified by the lower side of the elongated rectangle.

  In specifying the positions of the nose and mouth in this way, as shown in FIG. 9A, when the tilted face image is corrected as shown in FIG. 9B, the positions of the eyes, nose and mouth are changed. After the identification, the coordinates are rotated in the reverse direction, and their positions are converted into positions in the initial tilted face image.

  The present invention is useful in forming a makeup simulation image that requires detection of the position of the eyes, the position of the mouth, and the position of the nose.

It is a flowchart of the detection method of the position of a face part. It is a block diagram of the detection system of the position of a face part. It is explanatory drawing of the detection target area | region of the position of eyes. It is the gray scale image which changed lightness sequentially. It is a grayscale inverted image of a gray scale image in which the brightness is sequentially changed. It is explanatory drawing of the cluster area | region of a pixel in a certain fade-in image. It is explanatory drawing of the state which accumulated the position candidate of eyes over all the hierarchy of a gray scale image. It is the original image which displayed the position of the detected eye. It is a tilted face image. It is a face image with corrected inclination. It is explanatory drawing of the detection object area | region of the position of a mouth and the position of a nose. It is explanatory drawing of the circumscribed rectangle of the cluster area | region of a pixel, and its grouping. It is explanatory drawing of the selection method of the circumscribed rectangle made into the position candidate of a mouth or a nose.

Explanation of symbols

DESCRIPTION OF SYMBOLS 1 Imaging device 2 Personal computer body 3 Display 4 Image scanner 5 Printer 10 Face part position detection system 20 Gray scale face image 21 Rectangular frame 21 ′ as a detection target area of eye position Detection target of mouth position and nose position A rectangular frame 30 as a region A circumscribed rectangle 31, 32, 33, 34 that groups the circumscribed rectangles of the clustered region of pixels 40, 41, 42 yen

Claims (5)

Create multiple grayscale images with successively changing brightness from the face image, and as the grayscale image fades out from the high brightness to the low brightness, the cluster area of pixels that gradually appear in the faded out facial area Is detected (hereinafter referred to as a fade-in method), and a pair of detected pixel cluster regions is selected as an eye position candidate, and all gray scales of each eye position candidate are selected. Locate the eye based on the frequency of appearance across the image,
Groups of pixels that have been detected by the fade-in method and that have gradually stopped growing are grouped by overlapping vertical positions.
A circumscribed rectangle is formed for each grouped pixel cluster area,
Out of the circumscribed rectangles, those within a predetermined region predicted from the positional relationship with both eyes are considered as mouth position candidates,
A method for detecting the position of a face part that specifies a mouth rectangle having a maximum width among circumscribed rectangles as mouth position candidates.   Draw an inverted equilateral triangle with the pupils of both eyes as left and right vertices, and draw a circle with a diameter 0.65 to 1.0 times the length of one side of the inverted equilateral triangle centered on the lower vertex of the inverted equilateral triangle. The face part position detection method according to claim 1, wherein the circumscribed rectangle in which the circle and the inscribed circle located at the center of the circumscribed rectangle overlap is used as a mouth position candidate. After specifying the position of the mouth by the method according to claim 1, forming a circumscribed rectangle for each of the pixel cluster areas detected by the fade-in method, where the growth of pixels that gradually appears is stopped,
Among the circumscribed rectangles, those within a predetermined region predicted from the positional relationship with both eyes are set as nose position candidates,
Among the circumscribed rectangles that are nose position candidates, the method of detecting the position of the face part that identifies the nose position with the lower side above the upper side of the circumscribed rectangle identified as the position of the mouth and the largest width .   Draw an inverted equilateral triangle with the eyes of both eyes as the left and right vertices, and draw a circle with a diameter 0.5 to 0.7 times the length of one side of the inverted equilateral triangle centered on the lower vertex of the inverted equilateral triangle. The face part position detection method according to claim 3, wherein the circumscribed rectangle in which the circle and the inscribed circle located at the center of the circumscribed rectangle overlap each other is used as a nose position candidate. A face part position detection system comprising a face image acquisition means and a calculation means, wherein the calculation means comprises:
The ability to create multiple grayscale images with varying brightness from face images,
A function that detects a cluster area of pixels that gradually appear in the faded-out face area as the grayscale image fades out from a state that fades out at high brightness to low brightness.
A function of selecting a clustered region of detected pixels that appears as a pair as an eye position candidate,
The ability to identify eye positions based on the frequency of appearance across all grayscale images of each eye position candidate,
A function of grouping the detected pixel cluster areas where the growth of the pixels that appear gradually stops and the vertical positions overlap,
A function to form a circumscribed rectangle for each grouped pixel cluster area,
A function that makes a candidate for the position of the mouth among the circumscribed rectangles of the grouped pixel cluster area within a predetermined area predicted from the positional relationship with both eyes,
A function for specifying the mouth rectangle with the largest width among the circumscribed rectangles as mouth position candidates,
A function to form a circumscribed rectangle for each detected pixel cluster area where the gradually appearing pixel growth has stopped,
A function that makes a nose position candidate a circumscribed rectangle for each cluster area of detected pixels within a predetermined area predicted from the positional relationship with both eyes,
Among the circumscribed rectangles that are candidates for the position of the nose, the lower part is above the upper side of the circumscribed rectangle that is identified as the position of the mouth, and the face part has the function of identifying the largest width as the position of the nose Position detection system.

Images