JP2010015463A - Method for detecting eye position - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To detect automatically an eye position from a moving image with ease and high reliability. <P>SOLUTION: The method for detecting an eye position from a moving image includes the steps A: (A1) preparing from a frame of a moving image (original image) a blurred image (horizontal- and vertical-blurred image 24) that is blurred in two intersecting directions; obtaining a position of an intersection of edges in the blurring directions; (A2) preparing a face region detection image that has no gradation of the original image; (A3) and detecting a face region candidate in the face region detection image from a rectangular region that has an edge intersection in the gradation direction as an apex; and B: (B1) preparing a plurality of eye position detection images whose brightness changes for every face region candidate; (B2) detecting a pixel group that gradually appears in the image as the eye position detection image fades in low brightness from a state of fading out with high brightness; (B3) selecting pixel groups that appear in pairs as eye position candidates from the detected pixel group regions; and (B4) identifying the eye position candidates as eye positions based on their appearance frequencies. <P>COPYRIGHT: (C)2010,JPO&INPIT

Description

  The present invention relates to a method and a detection system for automatically detecting an eye position from a moving image.

  The position of the eye in the face image can be obtained by overlaying a hairstyle model image on the subject's face image to form a hairstyle simulation image, or by overlaying a makeup image such as lips or eyebrows on the subject's face image. In this case, the image is detected for adjusting the size of the image or aligning the image. The detection of the position of the eyes and the detection of the position of the face area are also performed in face image blindfolding processing, face image frame processing, creation of an image for personal authentication, and the like.

  Conventionally, as a method for detecting an eye position from a face image, a method of detecting a face by extracting a skin color region and further detecting an eye by pattern matching or the like has been used (Non-Patent Document 1, Patent Document 1). ).

2005 5th LSI IP Design Award Winning Paper “Face candidate point detection method that enables fast and reliable face detection” (Nikkei Business Publications) JP 2004-94917 A

  However, the extraction of the skin color area is affected by the lighting environment. Therefore, reliability becomes a problem when an unspecified number of face images taken in various lighting environments are targeted. Further, the pattern matching method has a problem that the amount of calculation becomes enormous. Therefore, the brightness of the image may fluctuate greatly, the number and position of the face image will fluctuate every moment, and the position of the eye will be changed in a movie that requires processing of several frames or more per second. Automatic detection has been a major issue.

  On the other hand, an object of the present invention is to enable automatic and highly reliable detection of the position of eyes and the position of a face region from a moving image by a simple method.

  The inventors of the present invention (1) create a blurred image in two directions such as horizontal and vertical intersecting from the original image, calculate the edge of the blurred image in the blur direction, and further determine the intersection between them, 2) By crushing the gradation of the original image including the face image, it is preferable to obtain an image that can discriminate between the hair area and the face area but not the face structure, and (3) Considering the intersection of the edges in the blur direction, the face image often exists in a rectangular area having this intersection as a vertex and having a higher brightness than the areas adjacent to the left and right and above, and therefore This rectangular area can be used as a face area candidate. (4) For each face area candidate, when multiple images with varying brightness are created, fade-out images on the high brightness side are sequentially faded to images on the low brightness side. First, (5) In this case, the pixels in the pupil area appear in pairs, and thus the eye position candidates can be determined, and (6) appear in pairs. The eye position candidate can be identified from the accumulation result of appearance frequencies over all eye position detection images of the eye position candidate, and this eye position identifying method is suitable for a moving image requiring processing speed. I found it.

That is, the present invention is a method for detecting the position of an eye in a moving image,
A. (A1) Create a blurred image blurred in two directions intersecting from a moving image frame (hereinafter also referred to as an original image), determine the position of the intersection of the edges in the blurred direction in the blurred image,
(A2) Create a face area detection image with the gradation of the original image collapsed,
(A3) In a face area detection image, a rectangular area having an intersection of edges in the blur direction as one vertex and having a higher brightness than the left and right and upper adjacent areas is detected as a face area candidate. Process,
B. (B1) For each face area candidate, create a plurality of eye position detection images with varying brightness,
(B2) As the eye position detection image fades in from a state of fading out at a high brightness to a low brightness, a cluster area of pixels that gradually appears in the eye position detection image is detected,
(B3) Select the clustered areas of the detected pixels that appeared in pairs as eye position candidates,
(B4) specifying the eye position candidate as the eye position based on the appearance frequency over all eye position detection images of each eye position candidate;
A method for detecting the position of an eye having

  Further, the present invention provides a face area position detection method for specifying a face area based on a face area candidate corresponding to the eye position after the eye position is specified by the above-described eye position detection method.

Furthermore, the present invention is an eye position detection system provided with moving image acquisition means and calculation means, wherein the calculation means comprises:
A function that creates a blurred image that is blurred in two directions intersecting with the original image, a function that detects edges in the blur direction of the blurred image and finds the position of those intersections, and a face area detection that collapses the gradation of the original image Image creation function, a face area detection image is a rectangular area whose intersection is the intersection of edges in the blur direction and has a higher brightness than the adjacent areas on the left and right and above As a function to create a plurality of images whose brightness has changed as eye position detection images for each face area candidate, as the image fades out from a state of fading out with high brightness to low brightness, A function for detecting a cluster area of pixels that gradually appear in the eye position detection image, a function for selecting a pair of detected pixel cluster areas that appear as a pair, and each eye Providing a detection system of the position of the eye has a function of specifying the position of the eye based on the frequency of occurrence over all eye position detecting image position candidate.

  Further, the present invention is a face region position detection system provided with a moving image acquisition means and a calculation means, the calculation means, in addition to the function of the calculation device of the eye position detection system described above, Provided is a face region position detection system having a function of identifying a face region candidate whose eye position is identified as a face region.

  According to the method or system of the present invention, a face area candidate that is likely to contain a face image is automatically detected in a moving image, and a plurality of images with different brightness values are formed for each face area candidate. Since the position is specified, regardless of the lighting environment, the subject's skin color, the subject's pupil color, the face orientation in the face image, etc., the position of the eye included in the frame of the moving image and the face corresponding to the position of the eye The area can be automatically detected in a short time with high reliability.

  Furthermore, according to the eye position detection method or system of the method or system of the present invention, the formation of a hairstyle simulation image for fitting various hairstyle images to an arbitrary face image based on the eye position, Simulation image by fitting multiple pieces of image information based on eye position, such as forming a makeup simulation image that fits a partial facial image after makeup to an arbitrary face image based on eye position When forming the image, it becomes possible to form a simulation image of a natural moving image in a short time.

  The present invention also applies to image processing such as painting the eye and its neighboring area, applying a mosaic, and applying a mosaic to the entire face area, such as a blindfold process for protecting personal information. The method or system can be suitably used.

  Hereinafter, the present invention will be described in detail with reference to the drawings.

  FIG. 1 is a flowchart of an embodiment of the eye position detection method of the present invention, and FIG. 2 is a block diagram of an eye position detection system for implementing this method.

  The system 10 includes an imaging apparatus 1 and a personal computer main body 2, and a personal computer main body 2 is connected to a display 3, a printer 4, and the like.

  The imaging device 1 is provided as means for acquiring a face image of a subject, and a commercially available camera that can capture a moving image such as a digital video camera or a web camera can be used. A video capture is connected between the imaging device 1 and the personal computer body 2 as necessary.

  The personal computer main body 2 is provided as a calculation means, and roughly, gray scale of a color image, gradation change, brightness correction, image size change, overall blur, blur in a predetermined direction such as horizontal and vertical, edge Image processing functions such as extraction, image superposition, figure enlargement / reduction functions, etc., and a function to create a fade-in image with a predetermined fade-in ratio by combining these functions as described later. It has a function of changing (for example, a function of decreasing the contrast as the brightness is increased and increasing the contrast as the brightness is decreased). These individual image processing functions can be obtained by, for example, installing commercially available image processing software such as Photoshop manufactured by Adobe System Co. in the personal computer main body 2.

  The personal computer body 2 has a function of managing a plurality of images with different brightness by assigning a list number in order of increasing or decreasing brightness. The brightness of a predetermined area on the image is compared with the brightness of other areas to perform predetermined extraction. More specifically, a horizontal / vertical blurred image is created as a blurred image that is blurred in two intersecting directions from the original image, and the horizontal / vertical blurred image In the function of detecting the horizontal edge and the vertical edge and obtaining the position of the intersection, the function of creating the eye position detection target area setting image in which the gradation of the original image is crushed, the eye position detection target area setting image, A function for setting a rectangular area having an intersection of the horizontal edge and the vertical edge as one vertex; a function for detecting an area having a higher brightness as compared with the adjacent area on the left and right and above the rectangular area as a face area candidate; Area A function to create multiple images with varying brightness as eye position detection images for each eye, as the image fades in from a state that fades out at high brightness to a low brightness, the eye position detection image gradually A function for detecting a cluster area of appearing pixels, a function for selecting a pair of detected pixel cluster areas appearing as eye position candidates, and an image for detecting the position of all eyes of each eye position candidate The function of specifying the position of the eyes based on the appearance frequency over the entire range.

  In addition, the personal computer main body 2 stores the position when a position of the eye is specified for a certain frame N, a function for selecting a frame transferred at a predetermined frame rate from the imaging apparatus 1 as necessary. A function of setting the periphery of the position as a face image area candidate for detection of the position of the eye in a frame for a predetermined time after that, when the position of the eye is specified for a certain frame, The maximum size of the face area candidate whose eye position is specified in the frame N is set as a face image size frame, the face image size frame is scanned on the face area detection image, and the peripheral area is compared with the center portion in the frame. A function for detecting a region with low brightness as a face region candidate, and when the position of an eye is specified for a certain frame, the position of the eye in frame N in the next frame N + 1 The specified face area candidate is used as a face area candidate, and the face area candidate is detected from an image obtained by removing the face area candidate whose eye position is specified in frame N from the original image of frame N + 1. Yes.

As shown in FIG. 1, the method for detecting the position of the eye in the moving image using the system 10 is roughly as follows. A step of setting a frame as an original image to be processed and detecting a face region candidate from the original image;
B. It consists of two steps: a step of detecting an eye position candidate for each face region candidate and specifying it.

  In the process A, first, a frame of a moving image to be detected for eye position is acquired in the personal computer main body 2 as an original image. As the original image, for example, a moving image including a face image may be taken by the imaging apparatus 1 and may be taken into the personal computer main body 2. The video may be acquired in the personal computer main body 2, and video capture is used as necessary.

  The number of face images included in one original image may be one or more, but the present invention is particularly effective when there are a plurality of face images and when the sizes of the plurality of face images are different.

  Next, pre-processing such as gray scaling, size adjustment, blurring, and brightness correction is performed on the original image as necessary.

  Of these, gray scale conversion is performed in order to reduce the amount of subsequent processing when the acquired original image is a color image because the eye position detection method of the present invention does not require color information.

As a gray scale method, for each pixel, (1) a method that uses the average of the maximum and minimum values of R, G, and B, and (2) the average of each value of R, G, and B A method to be used, (3) a method of multiplying each value of R, G, and B by a predetermined weighting coefficient (NTSC, etc.) and then averaging them can be used. For example, in the method (3), the following equation Y (output luminance) = 0.298912 × R + 0.568811 × G + 0.114478 × B
Can be used.

  The size adjustment is to adjust the original image to an image size suitable for the eye position detection process. Specifically, the long side is preferably 320 pixels or more from the viewpoint of detection accuracy. In view of speed, it is preferable to change the screen size so that the long side is about 480 pixels or less.

  If the face image in the original image is sharp, a lot of fine edges appear in the subsequent image processing, and blurring processing is performed as necessary because it becomes noise for detecting eye position candidates. In this case, it is preferable that the degree of blurring is weak to medium. For example, for each pixel, the weighting blur filter of 1 to 5 is used for the pixel of interest and the surrounding 25 pixels. Performs blurring to output the average value. This blurring process may be performed a plurality of times as necessary.

  The brightness correction is preferably performed when the face image detection image is created when the original image is extremely bright or dark. As a brightness correction method, for example, a brightness histogram of an original image is created for an original image that is too bright as shown in FIG. 3A (a) or an original image that is too dark as shown in FIG. After setting the count value of the pixel value which is 1.5% to 15% of the number, preferably 3% to 8% or less to zero, the maximum pixel value Xmax among the pixel values whose count value is not zero and The minimum pixel value Xmin is obtained. Next, as shown in FIGS. 3A (b) and 3B (b), the original image brightness correction is performed by substituting the pixel value Xi of the original image into the following equation to obtain the pixel value after the brightness correction.

However, if Xi <Xmin, Xi = Xmin,
If Xi> Xmax, Xi = Xmax.

  The brightness correction method is not limited to the above-described method, and various methods can be applied. For example, an arbitrary image is set as a reference image in advance, the average brightness is measured, and an appropriate brightness change amount for creating a horizontal / vertical blurred image and a face area detection image is determined. The ratio of the average brightness of the original image to the average brightness is calculated, and the brightness of the original image is corrected by changing the brightness of the original image by an amount obtained by multiplying the appropriate brightness change amount in the reference image by the average brightness ratio.

  Here, the appropriate brightness change amount in the reference image is determined as follows. First, an image in which the average brightness of the reference image is changed by one gradation in both the light and dark directions is created, a detection target area of an eye position described later is detected for each image, and the detection result is actually a face. It is judged visually whether it matches with the area where the image exists, and the success rate of the detection is calculated. Then, the average brightness of the image with the highest success rate is obtained, and the difference between this and the average brightness of the reference image is set as the appropriate brightness change amount.

  As the reference image, face images of 5 to 20 persons are shown, and the ratio of the maximum size and the minimum size of each face image is preferably within 1 to 3 times the face length. The eye position detection target area may be set by the following method.

  FIG. 4 shows the original image 20 before the pre-processing and the original image 21 after the pre-processing when the gray scale, size adjustment, blurring, and brightness correction are performed on the original image 20 as the pre-processing.

  After pre-processing, in step A (A1) in FIG. 1, a horizontal / vertical blurred image 24 as shown in FIG. 5C is created as a blurred image blurred in two intersecting directions. The horizontal / vertical blurred image 24 is a combination of the horizontal blurred image 22 created as shown in FIG. 10A and the vertical blurred image 23 created as shown in FIG. In the present invention, the two blur directions in the blurred image are not limited to the horizontal direction and the vertical direction, and may be two oblique directions as shown in FIG. 7, for example. In this case, it is preferable that the angle θ on the acute angle side be 60 ° or more as the crossing angle in the two directions. However, in order to simplify the subsequent calculation, it is preferable that the intersecting two directions are the horizontal direction and the vertical direction, and the horizontal / vertical blurred image 24 as shown in FIG. 5C is created.

  For example, in the case of the horizontal blurred image 22, an average pixel value of 80 to 120 pixels in the horizontal direction centered on the pixel is obtained for each pixel in the horizontal blurred image 22, and the pixel value is calculated. It can be obtained by performing the operation of setting the pixel value of the pixel for all the pixels. Similarly, in the vertical blurred image 23, the vertical direction centering on the pixel for each pixel with respect to the original image 20 It can be obtained by calculating an average pixel value of 80 to 120 pixels and setting the pixel value as the pixel value of the pixel for all pixels.

  In addition, as a method of creating the horizontal blurred image 22 and the vertical blurred image 23, first, a horizontal blurred image and a vertical blurred image are generated, and then, without combining the two steps of combining them, for each pixel, The average pixel value of the horizontal direction 80 to 120 pixels and the vertical direction 80 to 120 pixels centered on the pixel may be calculated simultaneously.

Next, as in the horizontal / vertical edge image 25 shown in FIG. 6, the horizontal edge and the vertical edge in the horizontal / vertical blurred image 24 are obtained, and the position of the intersection of these edges is obtained. For detection of a horizontal edge or a vertical edge, it is preferable to use an edge filter having a blurring function. For example, a horizontal edge filter or a vertical edge filter of the following formula can be used.

  On the other hand, in step (A2) of FIG. 1, a face area detection image in which the gradation of the original image is crushed is created.

  As the face area detection image, as shown in FIG. 8A, by changing the brightness of the original image 20, the hair area and the face area can be discriminated, but the face structure cannot be discriminated, that is, the hair area and the face area. However, it is preferable to create an image in which the high gradation side is crushed so that the inside of the face does not appear at all, or the face, eyes, nose, and mouth can be recognized but it is difficult to determine who the face is.

  Although the method for creating the face area detection image 26 depends on the shooting conditions of the original image 20, it is basically preferable to create an image with a fade-in ratio of 25 to 35% from the original image 20.

  Here, for example, when the original image has 256 gradations, an image having a predetermined fade-in ratio can be created as follows.

  First, in an image with 0% fade-in, the pixel values of all the pixels are uniformly set to 255.

  Further, in an image with fade-in x%, pixels having a pixel value of less than 2.55x in the original image have the pixel value of the original image, and pixels having a pixel value of 2.55x or more (provided that x> 0) are uniform. Suppose that it has a pixel value of 255.

  For example, in a 10% fade-in image, pixels with a pixel value of less than 25.5 (= 255 × 0.1) remain the same in the original image, and pixels with a pixel value of 25.5 or more have a uniform pixel value. 255.

  An image with 20% fade-in has a pixel value of less than 51.0 (= 255 × 0.2) in the original image, and a pixel value of 51.0 or higher in the original image remains the same. Set to 255.

  In an image with 100% fade-in, the pixel values of all the pixels remain the pixel values of the original image.

  As shown in FIG. 8B, the face area detection image 26 may be formed as a negative image with the gradation reversed, and the subsequent image processing may be performed with the gradation reversed.

  Further, the method of creating the face area detection image 26 is not limited to a method of creating an image having a predetermined fade-in ratio, and various binarization processing methods can be applied. For example, various binarization processes (generally referred to as variable threshold processes) for determining a threshold value based on the characteristics of the original image, such as a binarization process using the central brightness value of the brightness histogram of the original image 20 as a threshold value. Thus, the face area detection image 26 can be created.

  Next, as shown in step A (A3) in FIG. 1, a face area candidate expected to be a face area is detected in the face area detection image 26. The face area candidate is a face area detection image 26 and a rectangular area having one vertex at the intersection of the horizontal edge and the vertical edge of the horizontal / vertical blurred image, compared to the left and right and upper adjacent areas. To detect areas with high brightness. This is because, in a grayscale image, the face area is compared to the periphery of the head, hat, clothing, and other face areas, regardless of race or hair color, and with or without hair. This is based on the inventor's knowledge that it is bright.

As a specific method of detecting a face area candidate, for example, as shown in FIG. 9, first, a horizontal / vertical blurred image 25 of a horizontal / vertical blurred image is superimposed on a face area detection image 26, and the horizontal edge and the vertical edge At each intersection P of the edge, a small square 30 having the intersection P as one vertex is assumed, and the square is defined as the following (a) or The image is expanded in the lower right direction, the lower left direction, the upper right direction, and the upper left direction until the condition (b) is satisfied.
(a) When the proportion of pixels having a pixel value of 255 in the square becomes more than 60% and then becomes less than 60%
(b) When the ratio of the pixels having a pixel value of 254 or less becomes 1/3 or less after the ratio of the pixel value of 254 or less in the pixels forming the vertical side or the lower side of the line width of 1 pixel in the square expansion direction

Next, as shown in FIG. 10, in a region adjacent to the left and right and above the square 31 at the time when the expansion is stopped, each side of the square is a long side, and the length is 1/3 to 1/2 of the side of the square. Assuming rectangles 32a, 32b, and 32c having short sides as short sides, if the following condition (p) or (q) is satisfied for these rectangles 32a, 32b, and 32c, the square 31 is selected as a face area candidate. And Even when all the above three rectangles cannot be drawn because the square 31 is located at the corner of the face area detection image 26, the rectangle is drawn within the drawable range, and the following conditions (p) or (q) are satisfied. to decide.
(p) 10% or more, preferably 20% or more of all the pixels of the rectangles 32a, 32b, and 32c have a pixel value of 254 or less.
(q) Among the long sides of the rectangles 32a, 32b, and 32c, a pixel having a pixel value of 254 or less is 1/3 or more, preferably 2/5 or more of the length of the long side on the square 31 side.

  In this face area candidate detection method, the intersection point P is an intersection point of a horizontal edge generated at the boundary between the forehead and the hair and a vertical edge generated at the boundary between the face and the right and left hair areas. Since it is often a point that falls within the outer upper hair region or a point around the bottom of the earlobe, the small square 30 from the intersection point P is moved to the lower right direction, the lower left direction, the upper right direction, or the upper left direction. When the enlarged square 31 is a face area, the square 31 becomes brighter than the area adjacent to it and satisfies the above-described condition.

  The figure to be enlarged with the intersection point P as the base point is not limited to a square, but may be a rectangle or the like. The figure when the enlargement is stopped and the neighboring area for comparing the brightness are also the three rectangles as described above. Not limited to 32a, 32b, and 32c, for example, a U-shaped region surrounding the square 31 from the left and right sides and from above may be assumed. As long as the face area candidate search process can be completely automated by the calculation means, it can be set as appropriate.

  Thus, as shown in FIG. 11, a plurality of face area candidates 33 can be detected in the face area detection image 26. As shown in the figure, a plurality of face area candidates 33 may overlap in one area.

  After the face area candidate 33 is detected, a process B for detecting the position of the eyes for each individual face area candidate 33 is performed.

  In step B, first, in step (B1), in order to detect eye position candidates, a plurality of eye position detection images with varying brightness are formed as eye position detection images for each face region candidate. . More specifically, for example, as shown in FIG. 12, a fade-in image in which the brightness is gradually decreased from 0% fade-in according to the above-described image forming method with a predetermined fade-in ratio is reduced from 10% to 80% by 1%. Create about 70 levels in increments.

  In creating the fade-in image hierarchy, in order to increase the processing speed while maintaining the detection accuracy, a face area candidate 33 having a side larger than 80-120 pixels in advance is It is preferable to adjust the image size of each face region candidate 33 so that the size is 80 to 120 pixels, and it is preferable to perform blurring processing as necessary to reduce noise.

  Further, when forming a fade-in image, it is preferable to lower the contrast as the brightness is increased and to increase the contrast as the brightness is decreased, because an eye portion appears more clearly in the fade-in image.

  In addition, as the eye position detection image, for example, by gradually changing the γ value, a plurality of images whose brightness is changed as shown in FIG. 12 may be formed. It is possible to use.

  As can be seen from FIG. 12, when the brightness of a detection target region is increased and the brightness is decreased from an image that is completely faded out to form a hierarchy of fade-in images, normally, first, a cluster of pixels is first formed in the pupil. Then, a block of pixels is detected in the nose and mouth. The cluster of pixels in the pupil region appears as a pair of left and right.

  Therefore, in step B (B2), the pixels that gradually appear in the eye position detection image as the eye position detection image formed from the face area candidate 33 fades in from the state of fading out at high brightness to the low brightness. Detect the mass area. In Step B (B3), a group of left and right pixels is selected as an eye position candidate. Depending on the hierarchy, pixel cluster areas appear in pairs in areas such as the mouth, eyebrows, and forehead in addition to the pupils. At this stage, these pixel cluster areas are also selected as eye position candidates.

In this case, it is preferable that the following (1) and (2) are set in advance as a condition for selecting a candidate eye position from among the clustered regions of the paired pixels.
(1) The lateral distance of the cluster area of a pair of left and right pixels is within a certain range.
(2) The vertical distance between the left and right pixel cluster areas is within a certain range.

  FIG. 13 shows a block area of pixels appearing in a face area in a fade-in image in a hierarchy with a face area candidate 33 as a rectangle, and a pair of these areas connected by a straight line. .

  Next, in step (B4), eye position candidates are specified as eye positions based on the appearance frequencies of all eye position candidates over all eye position detection images. More specifically, first, as shown in FIG. 14, a pair of eye position candidates appearing for each fade-in image is accumulated over all layers of the eye position detection image, and the appearance frequency is counted. . Then, a ranking list is created in descending order of the number of counts, and the position of the eye position candidate with the maximum count is narrowed down to the eye position in principle. In this count, the eye position candidate that appears first when the brightness is gradually reduced from the faded-out face image and continues to appear until the final stage is normally the maximum count (ranking first).

  When narrowing eye position candidates, it is preferable to determine in advance whether or not the eye positions are appropriate according to the number of layers of the fade-in image. For example, when a 100-level fade-in image is created between pixel values 0 and 255, it is determined that the eye is not eye when the maximum count is 5 or less. When determining the suitability of the eye position by this method, the number of layers of the fade-in image and the maximum count number for judging the suitability of the eye may be determined so as to be proportional. Therefore, when the fade-in image created between pixel values 0 and 255 is 200 layers, it is determined that the maximum count number is 10 or less, which is not noticeable. The determination of the suitability of the eye position candidate according to the number of layers of the fade-in image may be made after determining the conditions (a) to (c) described below.

  The eye position candidates narrowed down by the appearance frequency are those that have the second largest count when the following conditions (a) to (c) are all satisfied or when the condition (d) is satisfied. The position candidate position is specified as the eye position.

(a) When the second position candidate for the ranking is higher than the first position candidate for the ranking
(b) When the distance between the center of the eye position candidate of the 2nd ranked eye is longer than the distance between the centers of the position candidate of the 1st ranked eye
(c) When the left and right pupil regions of the second ranking eye position candidate are both outside the positions corresponding to the left and right pupils of the first ranking eye candidate
(d) The vertical distance between the first position candidate of the ranking and the second position candidate of the ranking is about the distance between the eyes and the eyebrows, and the first position candidate of the ranking is ranked first. When the position is above the second position candidate

  Of these, (a) to (c) are intended to prevent erroneous determination of the position of the mouth as the position of the eye, while the first ranking in the ranking may rarely be the mouth area. d) is for preventing erroneous determination that the eyebrows around the eyes are the positions of the eyes.

  In this way, as shown in FIG. 15, it is possible to accurately detect the eye position (more precisely, the position of the pupil) in the detection target area of each eye position.

  The eye position specifying method described above is performed for all face area candidates set in step A. Further, when the position of the eye cannot be specified in any face area candidate, it is handled that there is no eye in that area.

  Further, the eye positions specified from the eye position candidates can be finally overlapped with the corresponding positions of the original image as shown in FIG. It becomes.

  Note that the reason why the eye position is not properly detected by the above method is that (i) the eye is not present in the image, (ii) the eye is present but the eye is closed, (iii) ) The image of the eye is blurred due to the eye moving at high speed.

  The position specified as the eye position from the eye position candidates is registered as the eye position in the frame.

  In the present invention, in order to increase the detection accuracy of the eye position, the above-described A.1. B. Detection of face area candidates; It is preferable to determine the detection of the eye position and the success or failure of the detection of the eye position.

  On the other hand, in order to increase the eye position detection processing speed according to the frame rate or according to the eye position detection purpose, the following method can be used.

  First, as shown in FIG. 1, when the position of the eye is specified in one frame N, the frame immediately after to 3 seconds later is the frame in which the position of the eye is specified immediately after the frame N. Sometimes, the area around the eye position in the frame N is set as a face area candidate to be used in the process B, and the time required to detect the face area candidate in the process A is shortened.

  Here, when the eye position is properly detected in one frame N, the setting around the eye position as the detection target region of the eye position in the next frame N + 1 is, for example, an intermediate point between eyes The horizontal direction is 1.5 to 3 times, preferably 1.8 to 2.2 times the distance between the eyes and the center of the eyes, and the vertical direction is 0 to the center distance between the eyes and the eyes. .5 to 2 times, preferably 0.8 to 1.2 times.

  When the position of the eye is specified in one frame N, a frame that uses the peripheral area of the eye position of the frame N as a detection symmetry area of the eye position is displayed immediately after the frame N to 3 seconds later. One of the reasons why the frame is determined to be a failure in the detection of the position of the eyes is that the face is moving violently in the moving image. This is because it is assumed that there is no second.

  After the eye position is specified in the frame N, in the frame immediately after to 3 seconds later, if the eye position is not specified in the peripheral area of the eye position in the frame N, the entire area of the frame is checked. This is performed from the detection of the face area in step A described above.

  On the other hand, after the eye position is specified in the frame N, if the eye position is specified in the peripheral region around the eye position of the frame N in the frame immediately after to 3 seconds after that, The specified eye position is registered, and in the subsequent frames, the eye position is detected in the same manner as described above using only the vicinity of the newly registered eye position as an eye position detection target region. If the position of the eye is specified even at one position in the frame immediately after the frame N to 3 seconds later, it is registered as the position of the eye in that frame.

  In the initial frame, if the eye position could not be specified at all, the process proceeds to the next frame without further detecting the eye position in this frame, and the eye position is detected in the face area of step A. From the detection of.

  As a second method for increasing the detection processing speed of the eye position, when the eye position is specified in one frame N, the eye position is specified in the frame N in the frame immediately after to 3 seconds later. The face image size frame is scanned on the face area detection image obtained in step A (A2), and the peripheral portion is lower than the center portion in the frame. An area having brightness is detected as a face area candidate, and is used for detecting the eye position in the subsequent process B.

  More specifically, as shown in FIG. 17, the face image size frame 35 is scanned on the face area detection image 26 obtained in step A (A2), and the peripheral portion is compared with the central portion in the frame. Frame areas 35a to 35e having low brightness are detected, and the detected frame areas 35a to 35e are set as face area candidates to be detected as eye positions. In the scanning of the face image size frame 35, for example, when the length of one side of the face image size frame 35 is X pixels, 1 to 0.35X pixels, preferably 0.2X to 0.3X pixels, more preferably 0.23X to When the face image size frame 35 is divided into 9 × 3 × 3 regions by moving the image vertically and horizontally by 0.27 × pixels, it is preferable to detect face region candidates based on the following criteria (1) to (3).

(1) The existence ratio of regions having pixel values of 254 or less (regions that are not white) has a higher average value in the surrounding eight regions than the central region.
(2) The existence ratio of the region having a pixel value of 254 or less is higher in the upper central region than in the central region.
(3) The existence ratio of the region having a pixel value of 254 or less is higher in the central left region or the central right region than in the central region.

  Of these criteria, (1) is for people with different skin and hair colors, with or without head hair, and in grayscale images, the face area is the head, hat, clothing, and other This is based on the fact that it is brighter than the surroundings of the face area, and is a reference necessary for detecting the face area.

  (2) is a front face image, when there is a dark region of head hair above the bright area of the face on the image, (3) is on the image, right side of the face, like a sideways face image Alternatively, this corresponds to the case where a dark region of hair exists on the left side, and these are used when the detection target region of the eye is set to be excessively wide only under the condition (1). Therefore, it is preferable to detect a frame area that satisfies only the condition (1) when priority is given to preventing detection omissions, and a frame that satisfies (1) and (2) at the same time if the purpose is to reduce the computation time. It is preferable to detect a region or a frame region that satisfies (1) and (3) simultaneously.

  Note that the scan of the face image size frame 35 and the detection of the face area candidate may be performed at the same time, or the face area candidate may be detected after the scan is completed.

  The face area candidate detection method by scanning the face image size frame 35 is that the frame size depends on the size of the face area candidate whose eye position is specified in the frame N. Although the detection accuracy is low as compared with detection of a face area candidate starting from creation of a blurred image, the face area candidate can be detected in a short time.

  On the other hand, as a method for improving the detection accuracy of the eye position, when the eye position is specified in one frame N, in the next frame N + 1, the face area candidate whose eye position is specified in the frame N is selected. It is preferable to detect the face area candidate from the image obtained by removing the face area candidate whose eye position is specified in the frame N from the original image of the frame N + 1, as the face area candidate used in the process B. In general, when the position of the eye is specified in the frame N, the position of the eye is often specified in the same region in the frame N + 1. According to this method, the eye in the frame N is specified in the frame N + 1. Since the eye position is detected while checking the position of the eye, the detection accuracy of the eye position can be improved.

  In the present invention, when the detection of the eye position cannot catch up with the transfer of the frame due to the size of the image or the frame rate, the eye position may be detected every predetermined number of frames.

  For the purpose of detecting the contents of the image and the eye position in the frame where the eye position is not properly detected or in the frame where the eye position is not detected to detect the eye position every predetermined number of frames. Accordingly, the eye position may be specified by complementing the eye position in the frame in which the eye position is detected, and the frame in which the eye position is not detected is not displayed on the display. Also good.

  In the present invention, after specifying the position of the eye in step B (B4), the face region is specified based on the face region candidate corresponding to the position of the eye in step B (B5). The position of the inside face area can also be detected. In this case, the rectangular area of the face area candidate corresponding to the specified eye position does not need to be used as the face area as it is, but a polygon area including the detected eye position according to the purpose of specifying the face area in the original image. Or a circular region or a flesh-colored region. Here, the size of the face region can be appropriately set according to the detected distance between both eyes, the size of the face region candidate, and the like. In addition to detecting the position of eyes in each face region candidate, the position of the mouth and nose is also detected using the fade-in method (Image Recognition and Understanding Symposium (MIRU2006), Proceedings CD-ROM, IS1-18 “ Face parts (hair, eyebrows, eyes, nose, mouth) are highly accurate and put into practical use ", published 2006.7.19), and at least an area that includes the nose or mouth in addition to the eyes may be used as the face area.

  In addition, when a plurality of face areas corresponding to the position of the eye and having different sizes or positions are overlapped, the face area may be determined based on these sum areas. For example, when the purpose is to perform mosaic processing of the face area in the original image, it is preferable to use the sum area in order to reliably perform mosaic processing on the original face area.

  The eye position detection method and face region position detection method according to the present invention include a skin color, a pupil color, a face orientation, and an illumination of each subject to be detected as an eye position in a plurality of face images. Regardless of environmental changes, the eye position can be detected at high processing speed with a simple and accurate calculation method. Therefore, it is possible to detect the position of the eyes from a plurality of face images regardless of whites or blacks.

  The positions of the eyes detected by the present invention are the following: when forming a hairstyle simulation image, when forming a makeup simulation image, when performing face image blindfold processing, when performing face image frame processing, It can be used in various scenes where the detection of the eye position is required, such as when creating.

  In addition, the position of the face area detected by the present invention is also applied to the case where the face image is blindfolded, the face color correction at the time of digital camera shooting or the face color correction of the shot image, the face color correction at the time of printer printing, Can be used.

It is a flowchart of the detection method of the position of the eye in the moving image, and the position of a face area. It is a block diagram of the detection system of the position of eyes and the position of a face area. It is explanatory drawing of brightness correction. It is explanatory drawing of brightness correction. An original image before preprocessing and an original image after preprocessing. They are a horizontal blur image, a vertical blur image, and a horizontal / vertical blur image. It is an image of a horizontal edge and a vertical edge detected from a horizontal / vertical blurred image. It is a blurred image in two diagonal directions. It is a face area detection image (positive image). It is a face area detection image (negative image). It is explanatory drawing of the square which makes the base point the intersection of a horizontal edge and a vertical edge for the detection of a face area candidate. It is explanatory drawing of the detection conditions of a face area | region candidate. It is the figure which showed the face area candidate in the image for face area detection. It is an eye position detection image in which the brightness of a face area candidate is changed. It is the figure which showed the position candidate of the eye to the fade-in image of a certain hierarchy. It is the figure which accumulated and displayed the eye position candidate over all the hierarchies in the eye position detection image of a certain hierarchy. It is the figure which displayed the position of the specified eye on the face area candidate. It is the figure which superimposed the position of the detected eye and the position of the face area on the original image. It is explanatory drawing of the method of detecting a face area candidate by scanning a face image size frame with the image for face area detection.

Explanation of symbols

DESCRIPTION OF SYMBOLS 1 Imaging device 2 Personal computer body 3 Display 4 Printer 10 Eye and face position detection system 20 Original image 21 Pre-processed original image 22 Horizontal blurred image 23 Vertical blurred image 24 Horizontal / vertical blurred image 25 Horizontal / vertical edge image 26 Face area detection image 30 Minute square 31 Squares 32a, 32b, and 32c that have stopped being expanded Rectangular 33 Face area candidate 35 Face image size frame P Intersection of horizontal edge and vertical edge in horizontal / vertical blurred image

Claims (16)

A method for detecting an eye position in a moving image,
A. (A1) Create a blurred image blurred in two directions intersecting from a moving image frame (hereinafter also referred to as an original image), determine the position of the intersection of the edges in the blurred direction in the blurred image,
(A2) Create a face area detection image with the gradation of the original image collapsed,
(A3) In the face area detection image, a rectangular area having an intersection of edges in the blur direction as one vertex and having a higher brightness than the left and right and upper adjacent areas is detected as a face area candidate. The process of
B. (B1) For each face area candidate, create a plurality of eye position detection images with varying brightness,
(B2) As the eye position detection image fades in from a state of fading out at a high brightness to a low brightness, a cluster area of pixels that gradually appears in the eye position detection image is detected,
(B3) Select the clustered areas of the detected pixels that appeared in pairs as eye position candidates,
(B4) specifying the eye position candidate as the eye position based on the appearance frequency over all eye position detection images of each eye position candidate;
A method for detecting the position of an eye.   When the eye position is specified in one frame N, the frame immediately after to 3 seconds later, when the frame N is the most recently specified frame, the position of the eye in the frame N is determined. The eye position detection method according to claim 1, wherein the peripheral area is a face area candidate used in step B.   When the eye position is specified in one frame N, the maximum size of the face region candidate whose eye position is specified in frame N in the frame immediately after to 3 seconds later is set as the face image size frame, and the process A ( The face image size frame is scanned on the face area detection image created in A2), and the area where the peripheral part is lighter than the central part in the frame is detected as the face area candidate used in step B. The method for detecting an eye position according to claim 1.   When the eye position is specified in one frame N, the face area candidate whose eye position is specified in frame N in the next frame N + 1 is set as the face area candidate used in step B. The method of detecting an eye position according to claim 1, wherein the face area candidate is detected in step A from an image obtained by removing the face area candidate whose eye position is specified from the original image of the frame N + 1.   The method of detecting an eye position according to any one of claims 1 to 4, wherein, in step A (A2), a face area detection image is created from an original image that has undergone lightness correction.   In step A (A2), when the original image has 256 gradations, an image with 0% fade-in has all pixels having a pixel value of 255, and an image with fade-in x% (where x> 0) A pixel having a pixel value less than 2.55x in the original image has a pixel value of the original image, and a pixel having a pixel value of 2.55x or more has a uniform pixel value 255. The eye position detection method according to claim 1, wherein an image having a fade-in of 25 to 35% is formed with respect to the original image.   The method for detecting an eye position according to claim 1, wherein two intersecting directions are a horizontal direction and a vertical direction. In step A (A3), assume a small square with the intersection of the horizontal edge and the vertical edge as one vertex, and use the intersection as a base point until the next condition (a) or (b) is satisfied. Expand downward, lower left, upper right and upper left,
(a) When the proportion of pixels having a pixel value of 255 in the square becomes more than 60% and then becomes less than 60%
(b) When the expansion stops when the ratio of pixels with a pixel value of 254 or less becomes 1/3 or less after the ratio of the pixel value of 254 or less in the vertical or lower side of the line width of 1 pixel in the square enlargement direction Assuming rectangles having one side of the square as a long side and a length of 1/3 to 1/2 of one side of the square as a short side, The eye position detection method according to claim 7, wherein a face region candidate satisfying the following condition (p) or (q) is used.
(p) 10% or more of all pixels have a pixel value of 254 or less
(q) Pixels having a length equal to or more than 1/3 of the long side of the rectangular square have a pixel value of 254 or less   A method for detecting the position of a face area, which specifies a face area based on a face area candidate corresponding to the eye position after the eye position is specified by the method according to claim 1. An eye position detection system including a moving image acquisition unit and a calculation unit, wherein the calculation unit includes:
A function that creates a blurred image that is blurred in two directions intersecting with the original image, a function that detects edges in the blur direction of the blurred image and finds the position of those intersections, and a face area detection that collapses the gradation of the original image Image creation function, a face area detection image is a rectangular area whose intersection is the intersection of edges in the blur direction and has a higher brightness than the adjacent areas on the left and right and above As a function to create a plurality of images whose brightness has changed as eye position detection images for each face area candidate, as the image fades out from a state of fading out with high brightness to low brightness, A function for detecting a cluster area of pixels that gradually appear in the eye position detection image, a function for selecting a pair of detected pixel cluster areas that appear as a pair, and each eye The position detection system of the position of the eye has a function of specifying the position of the eye based on the occurrence frequency over position detecting images of all eye candidates.   When the eye position is specified in one frame N, the frame immediately after to 3 seconds later, and when the frame N is the most recently specified frame, the position of the eye in the frame N is determined. The eye position detection system according to claim 10, further comprising a function of setting a peripheral area as a face area candidate.   When the eye position is specified in one frame N, the face area detection is performed using the maximum size of the face area candidate whose eye position is specified in frame N as the face image size frame in the frame immediately after to 3 seconds later. The position of the eye position according to claim 10, comprising a function of scanning a face image size frame on the image for detection and detecting, as a face region candidate, a region whose peripheral portion has a lower brightness than the central portion in the frame. Detection system.   When the eye position is specified in one frame N, in the next frame N + 1, the face area candidate whose eye position is specified in frame N is set as a face area candidate, and the eye position in frame N is specified. The eye position detection system according to claim 10, further comprising a function of detecting a face area candidate from an image obtained by removing the face area candidate from the original image of frame N + 1.   The system for detecting an eye position according to claim 10, wherein two intersecting directions are a horizontal direction and a vertical direction.   15. The eye position detection system according to claim 10, wherein the moving image acquisition means is a camera capable of shooting a moving image.   16. A system for detecting a position of a face area comprising a moving image acquisition means and a calculation means, wherein the calculation means includes a function of the calculation means according to claim 10 and an eye among face area candidates. A face area position detection system having a function of identifying a face area based on the position of the face.