JP2017084302A - Iris position detection device, electronic apparatus, program, and iris position detection method - Google Patents

Abstract

PROBLEM TO BE SOLVED: To accurately detect a horizontal iris position of an object person.SOLUTION: An iris position detection device (1) comprises a gradation processing unit (13) for generating a reference image, a brightness value data generation unit (brightness value graph generation unit 16) for generating brightness value data, an inner/outer conner-of-the-eye detection unit (reference point detection unit 17) for extracting a position of the reference image in which a vertical luminance value shows a steep change and an iris center detection unit (18) for extracting the centers of horizontal two points in the reference image.SELECTED DRAWING: Figure 1

Description

  The present invention relates to a black eye position detection device that detects a black eye position of a subject, an electronic device including the black eye position detection device, a program, and a black eye position detection method.

  2. Description of the Related Art Conventionally, a technique for detecting the center of a pupil by analyzing an image obtained by photographing a subject's eye with a camera and detecting the direction of the subject's line of sight based on the position of the center of the pupil has been developed. For example, Patent Document 1 illuminates a user's eyeball with a light projecting means, reads a signal of each pixel of a sensor that receives a reflected light beam from the eyeball, estimates a pupil center position, and determines a pupil edge from the position. A gaze detection device that detects a gaze of a user by extracting the gaze is disclosed. Further, Patent Document 2 discloses a three-dimensional viewpoint measurement apparatus including a calculation unit that extracts two pupils and the like of a subject by detecting a pupil part based on an image difference from a bright pupil image and a dark pupil image. Has been.

JP 2003-144388 A (published on May 20, 2003) Patent No. 4,500,992 (issued July 14, 2010) JP 2014-52758 A (published March 20, 2014)

  However, the line-of-sight detection apparatus disclosed in Patent Document 1 requires a dedicated device such as an IRLED (infrared light emitting diode) as a light projecting unit in addition to the camera. Therefore, the line-of-sight detection device has a problem in that it cannot detect the user's line-of-sight direction with a simple configuration. The 3D viewpoint detection device disclosed in Patent Document 2 also requires the first and second cameras and the first and second light sources as dedicated devices, and is as simple as the above-described line-of-sight detection device. There is a problem that the user's line-of-sight direction cannot be detected with a simple configuration.

  The present invention has been made to solve the above-mentioned problems, and its purpose is to detect the position of the black eye in the horizontal direction of the subject with high accuracy with a simple configuration using a camera built in a tablet or the like. There is to do.

In order to solve the above problem, a black-eye position detection device according to one aspect of the present invention includes:
Emphasizes the difference between the brightness value at the center of the input image and the brightness value at the periphery of the input image for the input image that represents the subject's eyes by the brightness value of each pixel in the plurality of pixels A gradation processing unit that generates a reference image by performing basic gradation processing,
Luminance value data representing the relationship between the vertical luminance value indicating the integration of the luminance values of a plurality of pixels arranged in the vertical direction in the reference image generated by the gradation processing unit and the horizontal position of the reference image. A luminance value data generation unit to generate;
From the luminance value data, a position near the end of the reference image showing a steep change in the vertical luminance value is extracted, and the extracted position is detected as the position of the eye or the eye corner of the subject. An eye corner detection unit;
For the region where the difference between each vertical luminance value continuously changing in the luminance value data and the vertical luminance value corresponding to the position of the eye or the corner of the eye continuously decreases from the maximum value, A black eye center detection unit that extracts two horizontal points and detects the center of the two extracted points as the center of the subject's black eye.

  According to the above configuration, the gradation processing unit generates the reference image in which the difference between the luminance value in the central portion and the luminance value in the peripheral portion of the input image is emphasized. Therefore, the gradation processing unit can emphasize the black eyes in the reference image more than the black eyes in the input image. Further, the black eye position detection device according to the present invention detects the black eye position in the horizontal direction of the subject using the luminance value data generated by the luminance value data generation unit, so that the processing when detecting the black eye position is performed. The amount of reference image data to be reduced can be reduced.

  Further, according to the above configuration, the black eye position detection device according to the present invention includes an eye / eye corner detection unit that detects the position of the eyes or corners of the subject from the luminance value data, and the center of the subject's black eyes from the luminance value data And a black eye center detection unit for detecting. Therefore, the black eye position detection device according to the present invention uses the position of the eye or eye corner detected by the eye / eye corner detection unit and the center of the black eye detected by the black eye center detection unit, so that the lateral direction of the subject As the black eye position, for example, the distance from the eye or the corner of the subject to the center of the black eye can be calculated.

  As described above, the black-eye position detection device according to the present invention can accurately detect the black-eye position in the lateral direction of the subject even when incorporated in a relatively low-performance electronic device such as a tablet with a camera function. .

In order to solve the above problem, in the black eye position detection device according to one aspect of the present invention,
(I) enhancement processing for emphasizing a difference in brightness value of each pixel in the plurality of pixels with respect to the input image, and (ii) applying a minimum value filter or a maximum value filter to each pixel in the plurality of pixels. A pre-processed image by performing at least one of a filtering process and (iii) a preparatory gradation process that emphasizes a difference between a luminance value at a central portion of the input image and a luminance value at a peripheral portion of the input image Is further provided with a preprocessing unit for generating
The gradation processing unit preferably uses the pre-processed image as the target image of the basic gradation process as an input image representing the eye of the subject based on the brightness value of each pixel in a plurality of pixels.

  According to the above configuration, the gradation processing unit applies (i) enhancement processing that emphasizes the difference between the luminance values of each pixel constituting the input image, and (ii) applies a minimum value filter or a maximum value filter. For the preprocessed image that has been subjected to any of the filtering process and (iii) the preparation gradation process that emphasizes the difference between the luminance value in the central portion of the input image and the luminance value in the peripheral portion of the input image, Apply basic gradation processing.

  Here, since the black eye in the image is more emphasized in the preprocessed image than the input image, the black eye in the image is more emphasized in the reference image generated by the gradation processing unit. It becomes. Therefore, the black eye position detection device according to the present invention can detect the black eye position in the lateral direction of the subject with higher accuracy.

In order to solve the above problem, in the black eye position detection device according to one aspect of the present invention,
Perform a preparatory gradation process that emphasizes the difference between the luminance value at the center of the input image and the luminance value at the periphery of the input image, and calculate the approximate center position of the luminance value distribution in the preparatory image after the preparatory gradation process An approximate center position calculation unit,
Preferably, the gradation processing unit sets the substantially center position as a starting point of luminance value enhancement related to the basic gradation processing.

  According to the above configuration, the black eye position detection device according to the present invention further includes a substantially center position calculation unit that calculates a substantially center position of the luminance value distribution in the prepared image after the preparation gradation process. In addition, the gradation processing unit sets the approximate center position as the starting point of the luminance value enhancement related to the basic gradation processing.

  Here, the approximate center position of the luminance value distribution in the preparation image substantially coincides with the center position of the subject's black eye, that is, the center position of the black eye portion in the preparation image. Therefore, the center position of the black eye portion in the prepared image or a position in the vicinity of the center position can be used as a starting point for luminance value enhancement related to the basic gradation process, and a reference image with more emphasized black eyes can be generated. it can. Therefore, the black eye position detection device according to the present invention can detect the black eye position in the lateral direction of the subject with higher accuracy.

  In order to solve the above problems, an electronic apparatus according to one aspect of the present invention includes the black-eye position detection device according to any one of the above aspects.

  According to the above configuration, it is possible to realize an electronic apparatus that can detect the black eye position in the lateral direction of the subject with high accuracy.

In order to solve the above problem, a black eye position detection method according to an aspect of the present invention includes:
Emphasizes the difference between the brightness value at the center of the input image and the brightness value at the periphery of the input image for the input image that represents the subject's eyes by the brightness value of each pixel in the plurality of pixels A gradation processing step for generating a reference image by performing basic gradation processing,
In the reference image generated in the gradation processing step, luminance value data representing a relationship between a vertical luminance value indicating integration of luminance values of a plurality of pixels arranged in the vertical direction and a horizontal position of the reference image. Luminance value data generation step for generating
From the luminance value data generated in the luminance value data generation step, a position near the end of the reference image where the vertical luminance value shows a sharp change is extracted, and the extracted position is extracted from the target person. An eye / eye corner detection process for detecting the position of the eye's eye or the eye corner;
The difference between each vertical luminance value that continuously changes in the luminance value data generated in the luminance value data generation step and the vertical luminance value corresponding to the position of the top or bottom of the eye is continuously from the maximum value. And a black eye center detecting step of extracting two points in the horizontal direction of the reference image and detecting the center of the two extracted points as the center of the subject's black eye.

  According to the above configuration, it is possible to realize a black eye position detection method capable of accurately detecting the black eye position in the lateral direction of the subject even in a relatively low-performance electronic device such as a tablet with a camera function. it can.

  The black eye position detection device according to each aspect of the present invention may be realized by a computer. In this case, the black eye position detection device is operated by operating the computer as each unit included in the black eye position detection device. A program for realizing the above and a computer-readable recording medium on which the program is recorded also fall within the scope of the present invention.

  According to one aspect of the present invention, it is possible to accurately detect a black eye position in a lateral direction of a subject with a simple configuration using an input image that represents the subject's eyes based on the brightness value of each pixel. it can.

It is a block diagram which shows the principal part of the black-eye position detection apparatus which concerns on Embodiment 1 and 2 of this invention. It is a flowchart which shows the whole flow of a subject's eyes | visual_axis detection by the tablet terminal provided with the black-eye position detection apparatus which concerns on this invention. (A) is a figure which shows an input image. (B) is a histogram of the entire image before enhancement processing. (C) is a histogram on the horizontal center line of the image before enhancement processing. (D) is a figure which shows the image after an emphasis process. (E) is a histogram of the entire image after enhancement processing. (F) is a histogram on the horizontal center line of the image after the enhancement processing. (G) is a figure which shows a pre-processing image. (A) is a figure which shows the outline of the determination method of the gradation magnification in a gradation magnification determination process. (B) is a figure which shows a reference | standard image at the time of performing a basic gradation process using the gradation magnification determined by the gradation magnification determination process. (A)-(c) is a figure which shows the outline of the processing method of a basic gradation process. (A) is a figure showing an example of an input picture. (B) is a figure which shows the image produced | generated by performing the horizontal basic gradation process with respect to the said input image. (C) is a figure which shows the image produced | generated by performing the basic gradation process of the vertical direction with respect to said input image. (A) is a figure showing an example of an input picture. (B) is a diagram showing a reference image generated when the input image is multiplied by a gradation magnification of 1.5 times in the vertical and horizontal directions. (C) is a figure which shows the reference | standard image produced | generated when said input image is multiplied by the gradation magnification of 2.0 times in the vertical direction and the horizontal direction. (D) is a figure which shows the reference | standard image produced | generated when said input image is multiplied by the gradation magnification of 2.5 times in the vertical direction and the horizontal direction. (A) is a figure showing an example of an input picture. (B) is a figure which shows each reference | standard image produced | generated when the gradation magnification which each differs in the vertical direction and the horizontal direction with respect to the said input image. (A) is drawing substitute photograph which shows the state of the image produced | generated when the input image in a figure is binarized with a specific threshold value. (B) is a drawing-substituting photograph showing the state of an image generated when a reference image having a vertical and horizontal gradation magnification of 1.5 times is binarized with a specific threshold value. (C) is a drawing-substituting photograph showing the state of an image generated when a reference image having a vertical and horizontal gradation magnification of 3.0 times is binarized with a specific threshold value. (A) is a figure which shows the outline of the production | generation method which produces | generates a vertical direction luminance value data table from the reference | standard image in a figure. (B) is a figure which shows an example of the vertical direction luminance value data table. (A) is a figure which shows the luminance value graph at the time of setting the production | generation range of a vertical direction luminance value to the range of the arrow A. FIG. (B) is a figure which shows the luminance value graph at the time of setting the production | generation range of a vertical direction luminance value to the range of the arrow B. FIG. (C) is a figure which shows a luminance value graph at the time of setting the production | generation range of a vertical direction luminance value to the range of the arrow C. FIG. (A) is a figure showing an example of a standard image. (B) is a figure which shows the distribution graph of the vertical direction luminance value produced | generated from the said reference | standard image. It is a figure which shows the outline of the calculation method of a subject's black eye position. It is a figure which shows an example of a brightness | luminance value graph, and the correspondence of the setting of the threshold value for black eye position detection, and the capture area | region in the input image by the said threshold value setting. It is a figure which shows the other example of the calculation method of the said black eye position. It is a flowchart which shows the black-eye position detection method by the black-eye position detection apparatus which concerns on Embodiment 1 of this invention. (A) is a figure showing an example of an input picture. (B) is a figure which shows the preparation image produced | generated by performing a preparation gradation process with respect to the said input image. (C) is a drawing-substituting photograph showing an example of an image generated when the above-mentioned prepared image is binarized with a specific threshold. (D) is a figure which shows an example of the image produced | generated by using the luminance value graph of a vertical direction, and the luminance value graph of a horizontal direction with respect to said preparation image. (A) And (b) is a figure which shows the outline of the processing method of a multistage gradation process. (C) is a drawing-substituting photograph showing an outline of a processing method of multistage gradation processing. It is a figure which shows the outline of the determination method of the gradation magnification in the case of setting a gradation starting point sequentially. It is a flowchart which shows the black-eye position detection method by the black-eye position detection apparatus which concerns on Embodiment 2 of this invention.

Embodiment 1
Hereinafter, embodiments of the present invention will be described in detail with reference to FIGS. Here, in each of the following embodiments, for convenience of explanation, the upper side is the upper side, the lower side is the lower side, the right side is the right side, and the left side is the left side for the convenience of explanation. In each of the following embodiments, the case where the black eye position detection devices 1 and 2 are built in the tablet terminals 100 and 200 as shown in FIG. 1 will be described as an example.

  The tablet terminals 100 and 200 are computer products in which an LCD (Liquid crystal display) provided with a touch panel is provided as a display unit 54 (see FIG. 2) on almost the entire surface of one side of a plate-shaped housing. . The black-eye position detection device according to the present invention is applied to various electronic devices such as portable game devices, various medical welfare devices, gaze measurement while driving a car, security devices such as security cameras in addition to tablet terminals. can do.

<Overall flow of target gaze detection using tablet devices>
First, with reference to FIG. 1, the overall flow of subject eye gaze detection by the tablet terminals 100 and 200 will be described. As shown in FIG. 1, first, the user captures the face of the subject with the camera unit 51 built in the tablet terminals 100 and 200 (step 100; hereinafter abbreviated as S100; subject photography step).

  Next, an input image generation unit 11 (see FIG. 1) such as an OpenCV (Open Source Computer Vision Library) included in the black eye position detection devices 1 and 2 incorporated in the tablet terminals 100 and 200 is photographed by the camera unit 51. An input image representing the eye of the subject is generated based on the brightness value of each pixel in the plurality of pixels from the face image of the subject (S101; input image generation step). The input image is an image representing either the right eye or the left eye of the subject. Which of the above-mentioned images is used as the input image may be set in advance in the tablet terminals 100 and 200, or may be arbitrarily selected by the user.

  Next, the black eye position detection devices 1 and 2 detect the position of the subject's lateral black eyes (hereinafter abbreviated as “subject's black eye position”) based on the input image (S102; black eye position detection step). ). Here, the black eye position of the subject indicates a linear distance in the horizontal direction from the eye of the subject to the center of the black eye (see FIG. 12). The horizontal distance from the subject's eye corner to the center of the black eye may be the subject's black eye position. The definition of “black eye center” will be described later.

  Next, a gaze direction estimation unit (not shown) provided in the tablet terminals 100 and 200 rotates the eyeball rotation center position from the camera coordinate system to the eyeball coordinate system based on the horizontal eye position of the subject. The matrix, target plane, and line-of-sight vector correction parameters are determined, and the line-of-sight direction of the subject is estimated (S103; line-of-sight direction estimation step, see Patent Document 3).

  Next, the user executes various gaze operation application software installed in the tablet terminals 100 and 200 using the estimated gaze direction data of the subject (S104; application software execution step).

<Specific configuration of tablet terminal>
Next, a specific configuration of the tablet terminal 100 will be described with reference to FIG. As illustrated in FIG. 1, the tablet terminal 100 includes a control unit 50, a camera unit 51, a storage unit 52, an operation input unit 53, a display unit 54, a battery unit 55, a transmission / reception unit 56, and an antenna unit 57.

  The control unit 50 comprehensively controls the tablet terminal 100 and includes the black eye position detection device 1. The black eye position detection device 1 controls various processes for detecting the black eye position of the subject, and details thereof will be described later.

  The camera unit 51 is an imaging unit that images a subject, and transmits the captured image data to the input image generation unit 11 described later. The storage unit 52 stores various control programs executed by the control unit 50, and includes a nonvolatile storage device such as a hard disk or a flash memory. The storage unit 52 may be provided outside the tablet terminal 100, for example. The operation input unit 53 acquires an input user operation. In the tablet terminal 100, a touch panel is used. The display unit 54 displays various images such as images resulting from the execution of various functions (application software) provided in the tablet terminal 100. The battery unit 55 supplies power to each unit constituting the tablet terminal 100. The transmission / reception unit 56 transmits / receives various data such as audio data via the antenna unit 57.

<Specific Configuration of Black Eye Position Detection Device According to this Embodiment>
Next, a specific configuration of the black eye position detection device 1 will be described with reference to FIG. As shown in FIG. 1, the black eye position detection device 1 includes an input image generation unit 11, a preprocessing unit 12, a gradation processing unit 14, a luminance value graph generation unit (luminance value data generation unit) 16, and a reference point detection unit (head). / Eye corner detection unit) 17, a black eye center detection unit 18, and a black eye position calculation unit 19.

  Here, in each of the following embodiments, the input image is a black and white image, and the brightness value of each pixel constituting each image including the input image is set to, for example, 256 gradations. Represent. In addition, the luminance value of the whitest (brighter) pixel in each pixel is “255”, and the luminance value of the blackest (darkest) pixel is “0”. However, the luminance value of the whitest (bright) pixel in each pixel may be “0”, and the luminance value of the blackest (darkest) pixel may be “255”.

  The input image may be a color image of two colors other than white and black. Further, the brightness value of each pixel may be expressed by, for example, 32 gradations.

  The input image generation unit 11 is not necessarily built in the black eye position detection device 1, and may be provided outside the black eye position detection device 1 or outside the tablet terminal 100, for example.

  The preprocessing unit 12 enhances the input image acquired from the input image generation unit 11 and emphasizes the difference in brightness value of each pixel in a plurality of pixels constituting the input image; A filter process for applying a minimum value filter to the pixel is performed. Then, the preprocessing unit 12 generates a preprocessed image by performing the above two processes.

  The gradation processing unit 14 sequentially applies the gradation magnification to the preprocessed image acquired from the preprocessing unit 12 (the input image representing the eye of the subject by the level of the luminance value of each pixel in the plurality of pixels, the target image). A reference image is generated by performing determination processing and basic gradation processing. Details of the gradation magnification determination process and the basic gradation process will be described later.

  The luminance value graph generation unit 16 determines, from the reference image generated by the gradation processing unit 14, the vertical luminance value indicating the average value of each luminance value in the vertical pixel row, and the horizontal position of the reference image. A luminance value graph (luminance value data) representing the relationship is generated. The Y axis of the luminance value graph represents a simple luminance value in which the minimum value of the vertical luminance value is replaced with “maximum value 1” and the maximum value of the vertical luminance value is replaced with “minimum value 0”. This represents the distance in the horizontal direction from the left end of the reference image (see FIGS. 11 and 12). Note that the vertical luminance value is not necessarily an average value of the luminance values, and may be a total value of the luminance values, for example. In other words, the vertical luminance value only needs to indicate an integration of luminance values of a plurality of pixels arranged in the vertical direction in the reference image (a ratio in which the maximum value is 1).

  The reference point detection unit 17 uses a point on the luminance value graph (an end portion of the reference image) indicating a sharp change in the simple luminance value (vertical luminance value) from the luminance value graph generated by the luminance value graph generation unit 16. (Near position) is extracted, and the X coordinate of the extracted point (position) is detected as the reference point, that is, the position of the eye of the subject. The reference point detection unit 17 may detect the position of the subject's eye corner as a reference point.

  The black-eye center detection unit 18 has two points at which the simple luminance value gradually decreases starting from the maximum value of the simple luminance value (vertical luminance value) in the luminance value graph generated by the luminance value graph generation unit 16. (Two points in the horizontal direction of the reference image) are extracted, and the X coordinate of the center of the two extracted points is detected as the center of the subject's black eye.

  The black eye position calculation unit 19 calculates the black eye position of the subject from the reference point detected by the reference point detection unit 17 and the center of the black eye detected by the black eye center detection unit 18. Note that the black eye position calculation unit 19 is not an essential component of the black eye position detection device 1. For example, the black eye center detection unit 18 may instead calculate the black eye position of the subject performed by the black eye position calculation unit 19. Good.

<Generation of Preprocessed Image by Preprocessing Unit>
Next, generation of a preprocessed image by the preprocessing unit 12 will be described with reference to FIG. Specifically, the preprocessing unit 12 performs enhancement processing by taking the following processing steps (1) and (2).
(1) By analyzing an input image as shown in FIG. 3A, the distribution of luminance values of each pixel constituting the input image is measured, and a reference histogram (FIG. 3B) is generated. To do. When attention is paid to the horizontal line (not shown) at the center of the input image, if the relationship between the horizontal position with the left end point at the origin as the origin and the luminance value distribution on the horizontal line is graphed, FIG. It becomes a graph as shown in (c).
(2) By expanding the reference histogram in the horizontal axis direction, in this example, the luminance value of the lowest (darkest) pixel among the pixels constituting the input image is converted to “0”, and A process of converting the luminance value of the pixel having the highest luminance value (brightest) to “255” is performed. Then, a preprocessing reference image as shown in FIG. 3D is generated. When the reference histogram is expanded in the horizontal axis direction, a histogram as shown in FIG. When the relationship between the horizontal position on the preprocess reference image center line (not shown) and the luminance value is graphed, a graph as shown in FIG.

  As described above, by performing the enhancement process on the input image, it is possible to further enhance the difference in the brightness value of each pixel constituting the input image. Further, when there are a plurality of input images, it is possible to eliminate uneven brightness and make the input images uniform.

Next, the preprocessing unit 12 generates a preprocessed image by performing a filtering process with the processing content shown in (3) below.
(3) For each pixel constituting the preprocessing reference image, the pixel having the lowest luminance value is selected from the total of nine pixels of the target pixel and the eight pixels surrounding the target pixel after sequentially setting the target pixel. Is replaced with the luminance value of the target pixel. Then, a preprocessed image as shown in (g) of FIG. 3 is generated.

  In this way, by applying the filtering process to the preprocess reference image, it is possible to reduce the reflection of disturbance light in the input image and to further enhance the black eye portion in the preprocess reference image. it can.

  Note that the above-described enhancement process and filter process are processes for detecting the subject's black eye position with higher accuracy, and thus it is not always necessary to perform two processes on the input image. That is, only one of the two processes may be performed, or the two processes may not be performed. Here, when the preprocessing unit 12 performs only enhancement processing, the preprocessing reference image becomes the preprocessed image as it is, and when only filter processing is performed, the input image is directly filtered. Further, when neither the enhancement process nor the filter process is performed, the black eye position detection device 1 may not include the preprocessing unit 12.

  Further, when the luminance value of the whitest pixel in each pixel constituting each image is represented by “0” and the luminance value of the blackest pixel is represented by “255”, the preprocessing unit 12 determines the maximum value for the preprocessing reference image. A filtering process for applying a filter is performed.

<Generation of reference image by gradation processing unit>
Next, generation of a reference image by the gradation processing unit 14 will be described with reference to FIGS.

  First, as gradation magnification determination processing, the gradation processing unit 14 includes an upper end region Su and a lower end region Sd in the horizontal direction H, a right end region Sr and a left end region in the vertical direction V in the preprocessed image as illustrated in FIG. For a total of four locations of Sl, a process of calculating an average luminance value of a plurality of pixels included in each location is performed. Next, in this example, the gradation processing unit 14 is a row of pixels in the horizontal direction H arranged on the uppermost side of the preprocessed image based on values obtained by dividing 255 (whitest color) by four average luminance values. (Hereinafter referred to as “upper end pixel column”) and a horizontal magnification H column (hereinafter referred to as “lower end pixel column”) arranged in the lowermost side of the preprocessed image are determined. In addition, a column of vertical V pixels arranged on the rightmost side of the preprocessed image (hereinafter referred to as “right end pixel column”) and a column of vertical V pixels arranged on the leftmost side of the preprocessed image ( Hereinafter, the gradation magnification of “leftmost pixel column” is determined (see FIG. 4B).

  By applying basic gradation processing to the preprocessed image using the gradation magnification determined in this gradation magnification determination processing, the reference image is a peripheral region (target person) of the black eye portion as shown in FIG. The area corresponding to the skin) is a white-out image. Therefore, the detection accuracy of the subject's black eye position is further increased by performing the gradation magnification determination process.

  As an example of the gradation magnification determination process, a value obtained by further averaging the average luminance value of the upper end region Su in the horizontal direction H and the average luminance value of the lower end region Sd is 130, and the average luminance value of the right end region Sr in the vertical direction V is The case where the value obtained by further averaging the average luminance value of the left end region Sl is 100 will be described as an example. In this case, for example, the gradation magnification of the upper end pixel column and the lower end pixel column can be determined to be double based on the calculation of 255/130, and the gradation magnification of the right end pixel column and the left end pixel column can be calculated to 255/100. 2.5 times can be determined based on this.

  Even if the average luminance value is a value close to 255, pixels that are originally dark (the luminance value is close to “0”) are not necessarily dark and white out even if gradation magnification determination processing is performed. . Therefore, a value obtained by adding 1 to the determined gradation magnification may be used as the final gradation magnification in order to make the whiteout more sure. In addition, the gradation magnification determination process is a process for further improving the detection accuracy of the subject's black eye position, and therefore the gradation magnification determination process may not be performed in the gradation processing unit 14. When this processing is not performed, for the gradation magnification of the upper end pixel column and the lower end pixel column and the gradation magnification of the right end pixel column and the left end pixel column, for example, a predetermined magnification value may be stored in the storage unit 52 in advance. Or you may enable it to set arbitrarily by a user.

  Further, in the above example of the gradation magnification determination process, the gradation magnification of the upper end pixel column and the lower end pixel column is determined to be double (symmetrical in the vertical direction), and the gradation magnification of the right end pixel column and the left end pixel column is determined to be 2.5 times. (Symmetry), but is not limited to this determination process. For example, the four end pixels are determined such that the gradation magnification of the upper end pixel row is determined directly from the average luminance value of the upper end region Su and the gradation magnification of the right end pixel row is directly determined from the average luminance value of the right end region Sr. It may be determined so that all the columns have different gradation magnifications.

  Next, the gradation processing unit 14 performs basic gradation processing for emphasizing the difference between the luminance value at the central portion and the luminance value at the peripheral portion of the preprocessed image, using the gradation magnification determined in the gradation magnification determination processing. Thus, a reference image is generated (see FIG. 5C). The central portion of the preprocessed image is a substantially circular, substantially elliptical, rectangular, polygonal, or indeterminate image area, and the width of the preprocessed image is the black eye of the subject in a state where the subject is looking directly in front. Assuming an input image such that the vertical center line (not shown) of the preprocessed image intersects (see FIG. 3A), the subject's black eye has moved to the most eye side Is set to be substantially the same as the total distance of the moving distance from the center line to the eye-eye end point of the black eye and the moving distance from the center line to the eye-eye end point of the black eye when moving most toward the eye corner side.

  Specifically, as the basic gradation process, the gradation processing unit 14 first generates a horizontal gradation array having the same number of elements as the numerical value of the horizontal width w (the number of pixels in the horizontal direction) of the preprocessed image. At this time, the element of the horizontal gradation array corresponding to the leftmost pixel in the preprocessed image is [0], and the element of the horizontal gradation array corresponding to the rightmost pixel is [w−1]. It becomes. Then, the gradation processing unit 14 stores the gradation magnification determined in the gradation magnification processing in the element [0] and the element [w−1].

  Next, when the number of pixels w in the horizontal direction is an even number, the gradation processing unit 14 counts from the left side in the preprocessed image, the element [w / 2-1] corresponding to the w / 2th pixel, w / 2 + 1th A gradation magnification of 1 is stored in each of the elements [w / 2] corresponding to the pixels. On the other hand, when the number of pixels w in the horizontal direction is an odd number, the number of pixels arranged in the center of the preprocessed image is one, so the gradation processing unit 14 uses the element corresponding to the w / 2th pixel [w / 2-1.] Stores a 1 × gradation magnification.

  Next, the gradation processing unit 14 increases the gradation magnification linearly toward both ends of the preprocessed image symmetrically with respect to the center line when the horizontal number of pixels w is an even number ( (See the graph in FIG. 5A), the gradation magnification is sequentially changed from the element [w / 2-2] to the element [1] and from the element [w / 2 + 1] to the element [w-2] in the horizontal gradation array. Store it. On the other hand, when the number of pixels w in the horizontal direction is an odd number, the gradation processing unit 14 converts the elements [w / 2-2] to elements [1] and elements [w / 2] to elements [w−] of the horizontal gradation array. 2] sequentially store gradation magnifications. In this case, the increase rate of the gradation magnification in the right direction and the increase rate of the gradation magnification in the left direction are slightly different.

  Then, the gradation processing unit 14 multiplies each column of pixels in the horizontal direction H in the preprocessed image by a horizontal gradation array in which gradation magnifications are stored in all elements.

  In this way, basic gradation processing is performed such that the height difference between the luminance value of the central portion Ch and the luminance value of the peripheral portion Ah is enhanced in the horizontal direction H of the preprocessed image ((a) in FIG. 5). reference).

  In the vertical direction V of the preprocessed image, the gradation processing unit 14 uses the same method as described above, and the height difference between the luminance value of the central portion Cv and the luminance value of the peripheral portion Av in the vertical direction V of the preprocessed image. A basic gradation process is performed such that the image is emphasized (see FIG. 5B).

  If the input image is multiplied by a gradation magnification that gradually enhances the luminance value from the center to the periphery, the initial value, the value of the element, and the magnification for each direction can be arbitrarily set. In addition, the shape of the central portion of the preprocessed image may be any of a circle, an ellipse, a rectangle (quadrangle), a polygon, or an indefinite shape. In other words, any pattern that roughly corresponds to the existence probability of black eyes may be used. .

  Here, as an example of the basic gradation processing by the gradation processing unit 14, 1 is added to the upper and lower pixel columns for a black and white image (not shown) having a horizontal width of 50 (the number of pixels in the horizontal direction H is 50 pixels). A case will be described in which a gradation magnification of .5 times is multiplied by a gradation magnification of 1.5 times each of the right end pixel row and the left end pixel row.

  As shown in the graph of FIG. 5A, the gradation processing unit 14 corresponds to the position of the 25th pixel and the 26th pixel (the distance in the horizontal direction from the left end of the preprocessed image is 25) counted from the left side. ) Is multiplied by 1 × gradation magnification. The gradation processing unit 14 linearly increases the gradation magnifications to be multiplied from the 24th pixel to the 2nd pixel counted from the left side and from the 27th pixel to the 49th pixel counted from the left side, respectively. That is, the gradation processing unit 14 increases the gradation magnification per pixel by 1.5 / 25 = 0.06 to each pixel from the 24th pixel to the second pixel, and the 27th pixel. To the 49th pixel are multiplied by the gradation magnification.

  Thus, by multiplying each column of pixels in the horizontal direction H by the gradation magnification, an image in which the central portion Ch is emphasized in black and the peripheral portion Ah is emphasized in white as shown in FIG. Can be generated.

  In addition, the gradation processing unit 14 multiplies each column of pixels in the vertical direction V in the black and white image by the gradation magnification in the same manner as described above, so that the center portion Cv as shown in FIG. It is possible to generate an image in which is emphasized in black and the peripheral portion Av is emphasized in white.

  In this way, by performing basic gradation processing in the horizontal direction H and the vertical direction V of the monochrome image, the gradation processing unit 14 is emphasized in black at the center as shown in FIG. An image in which the peripheral portion is emphasized in white is generated.

  Since the processing of the input image by the preprocessing unit 12 is not essential, the gradation magnification determination process and the basic gradation process (the luminance value in the central part of the input image and the peripheral part of the input image are directly applied to the input image. Basic gradation processing for emphasizing the difference from the luminance value may be performed. For example, when the basic gradation process in the horizontal direction H is performed on the input image as shown in FIG. 6A, the vertical direction including the black eye portion in the input image is shown in FIG. 6B. An image is generated in which a region that is elongated in black is emphasized in black, and a peripheral region of the region is emphasized in white. Furthermore, when the basic gradation process in the vertical direction V is performed, as shown in FIG. 6C, the black eye part and the area corresponding to the eye / eye corner part in the input image are emphasized in black, and other areas, In particular, it is possible to generate a reference image in which a region corresponding to a skin portion in an image is emphasized in white.

(Setting / adjustment of gradation magnification)
In the black-eye position detection device 1, in addition to the above-described configuration, for example, a configuration in which the user can arbitrarily set the gradation magnification, or a configuration in which the user can appropriately adjust the gradation magnification automatically determined by the gradation magnification determination processing, etc. are adopted. Can do. When adopting these configurations, how to set / adjust the gradation magnification is determined by comparing the advantages / disadvantages of increasing the magnification and the advantages / disadvantages of decreasing the magnification. preferable.

  As an example of the above-described comparative balance, an example is shown in which basic gradation processing is performed on the input image as shown in FIG. 7A using the gradation magnification set by the user (no processing by the preprocessing unit 12). Will be described. First, as shown in FIG. 7B, when the upper end pixel row, the lower end pixel row, the right end pixel row, and the left end pixel row of the input image are respectively multiplied by a gradation magnification of 1.5, the reference image (H × There is an advantage that the black eye portion and the eye / eye corner portion in the image V) are sufficiently emphasized in black. On the other hand, there is a demerit that the upper eyelid portion and the lower eyelid portion remain black in the reference image, and the skin portion in the image is not sufficiently whitened.

  Next, as shown in FIG. 7D, when each of the pixel columns is multiplied by a gradation magnification of 2.5, the skin portion in the reference image (H × V image) is sufficiently whitened. There is a merit to do. On the other hand, there is also a demerit that the eye / eye corner portion that is a reference for detecting the black eye position does not remain clearly in the reference image.

  Therefore, the merit / demerit in the above two cases is compared and weighed, for example, as shown in FIG. 5C, the gradation magnification is an average value of 1.5 times and 2.5 times. It is conceivable to set to 0 times. By setting the gradation magnification in this way, it is possible to whiten most of the skin portion while enhancing the black eye portion and the eye / eye corner portion in the reference image black.

  Further, for example, as shown in FIG. 8B, gradation magnifications multiplied by the upper end pixel row and the lower end pixel row in the input image (FIG. 8A), and gradation magnifications multiplied by the right end pixel row and the left end pixel row, It is also conceivable to select a reference image that can detect the black eye position with the highest accuracy by differentiating each of them.

  Further, the black eye position detection apparatus 1 can detect the black eye position of the subject by performing binarization processing as basic gradation processing on the reference image (see Embodiment 2). Whether or not the gradation magnification is appropriately set / adjusted is also important in detecting the black eye position by the binarization process.

  For example, when the binarization process is directly performed on the input image without generating the reference image, as shown in FIG. 9A, the threshold value of the luminance value serving as the reference for the binarization process is set. It becomes difficult. That is, even if the threshold value is set low (threshold value “44”), an image in which a part of the black eye portion disappears is obtained. Therefore, it is necessary to use a reference image that has been subjected to basic gradation processing by the gradation processing unit 14 even in detection of the black eye position by binarization processing.

  Next, as shown in FIG. 9B, binarization is performed with respect to a reference image in which the gradation magnification of the upper end pixel row and the lower end pixel row, and the gradation magnification of the right end pixel row and the left end pixel row are 1.5 times, respectively. When the processing is performed, even if the threshold value of the luminance value is set slightly higher (threshold value “65”), more black-eye portions remain than when the binarization processing is performed directly on the input image. It becomes an image.

  Further, as shown in FIG. 9C, the binarization processing is performed on the reference image in which the gradation magnification of the upper end pixel row and the lower end pixel row and the gradation magnification of the right end pixel row and the left end pixel row are 3.0 times, respectively. , Even if the threshold value of the luminance value is set higher (threshold value “111”), it remains more reliably than the black eye part and other parts other than the black eye part disappeared better than the above two cases. It becomes an image. However, since the eye / eye corner portion does not remain sufficiently in the reference image, the eye / eye corner portion does not remain well when binarization processing is performed.

  From the above, it can be seen that it is important to appropriately set / adjust the gradation magnification in order to improve the accuracy of black eye position detection.

  In each of the following embodiments, the gradation magnification is determined so as to increase linearly from the center to both ends in both the vertical direction V and the horizontal direction E with respect to the preprocessed image. The gradation magnification may be determined so as to change in a curved manner like a quadratic function. Further, the gradation magnification multiplied by the center of the preprocessed image does not need to be 1. However, for example, a gradation magnification of about 0.8 times at which the center of the preprocessed image becomes darker may be multiplied. In this case, since the starting point becomes darker in the basic gradation process using the substantially center position described later as the starting point of gradation, an effect of removing disturbance light reflected in the input image can be obtained.

<Generation of luminance value graph and detection of subject's black eye position>
Next, generation of a luminance value graph and detection of a subject's black eye position will be described with reference to FIGS.

(Generation of luminance value graph)
The luminance value graph generation unit 16 that has acquired the reference image from the gradation processing unit 14 first calculates the vertical luminance value of the column of pixels in the vertical direction V in the reference image, as shown in FIG. Every 1st to nth columns are obtained. Next, the luminance value graph generation unit 16 shows the correspondence between each column of pixels in the vertical direction V as shown in FIG. 10B and the vertical luminance value based on the calculation result. A luminance value data table (luminance value data) is generated.

  Regarding each pixel constituting the column of pixels in the vertical direction V necessary for calculating the vertical luminance value, which range of pixels is the target pixel for calculating the vertical luminance value, each of the following embodiments In the embodiment, the gradation processing unit 14 performs determination as follows. Note that the vertical luminance value calculation range is from hRange0, which is the Y coordinate of the uppermost pixel in the target pixel, to hRange1, which is the Y coordinate of the lowermost pixel, and in the vertical direction from each column, hRange0 to hRange1. It is assumed that the vertical luminance value is calculated from the luminance value of each pixel that is continuous.

  Specifically, the gradation processing unit 14 first generates a horizontal luminance value data table (luminance value data; not shown) indicating the correspondence between each column of pixels in the horizontal direction E and the horizontal luminance value. Next, the gradation processing unit 14 determines two pixels constituting a column of pixels in the vertical direction V corresponding to the two values immediately after the horizontal luminance value sharply decreases in the horizontal luminance value data table, respectively. The range of the target pixel is tentatively determined as the upper end pixel and the lower end pixel (the range indicated by the arrow C in FIG. 10A). Then, the gradation processing unit 14 excludes pixels having a high horizontal luminance value (corresponding to a portion that is over-exposed) from each pixel within the range of the target pixel that is provisionally determined. The pixel is determined as the final target pixel. When the target pixel for calculating the vertical luminance value is determined in this way, a luminance value graph to be described later is a graph as shown in FIG.

  By determining the target pixel for calculating the vertical luminance value as described above, it is possible to generate a vertical luminance value data table in which the difference between the vertical luminance values is more emphasized. Therefore, the black eye position in the horizontal direction of the subject can be detected with higher accuracy using the luminance value graph generated based on the vertical luminance value data table.

  In addition, as a method of determining the target pixel for calculating the vertical luminance value, for example, all the pixels constituting the column of the pixels in the vertical direction V (from the pixels in the first row to the pixels in the h row in FIG. 10A) ) May be the target pixel (range of arrow A in FIG. 10A). In this case, the luminance value graph described later is a graph as shown in FIG. Further, for example, the vertical width h of the reference image may be divided by 1/3, and a plurality of pixels included in a range corresponding to the central portion in the vertical direction V may be set as target pixels (indicated by an arrow B in FIG. 10A). range). In this case, a luminance value graph to be described later is a graph as shown in FIG.

  Here, using the vertical luminance value data table, a vertical luminance value distribution graph is generated in which the Y axis represents the vertical luminance value and the X axis represents the distance in the horizontal direction H from the left end of the reference image. In this case, the graph is as shown in FIG. In this graph, the value of the Y coordinate is smaller in the darker vertical V pixel row corresponding to the black eye portion in the reference image in FIG. Therefore, in order to make it easy to visually grasp the position in the horizontal direction H from the left end of the reference image, the luminance value graph generation unit 16 uses the vertical luminance value data table in FIG. A luminance value graph as shown in (b) of FIG.

(Detection of subject's black eye position)
Next, as shown in FIGS. 13A and 13B, the reference point detection unit 17, of the two points on the luminance value graph in which the simple luminance value shows a steep change, is closer to the Y axis. One point is extracted as a reference point P1. Then, the X coordinate of the extracted reference point P1 is detected as the position of the subject's eye. In the present embodiment, one of the two points on the luminance value graph showing a steep change in the simple luminance value is closer to the Y axis, which is the reference point P1, but the farther from the Y axis. One point may be the reference point P1. Which of the two points on the luminance value graph in which the simple luminance value shows a steep change is used as the reference point P1 is whether the eye of the subject displayed in the input image is the right eye or the left eye, or the Y axis is the reference It depends on whether it is set on the right or left side of the image.

  Next, as shown in FIGS. 13A and 13B, the black eye center detection unit 18 gradually decreases the simple luminance value starting from the maximum value of the simple luminance value in the luminance value graph 2. By extracting the points, the center P2 of the subject's black eye is detected.

  Specifically, the black eye center detection unit 18 first searches for a point (α) at which the simple luminance value on the luminance value graph is maximized. Then, a point where the simple luminance value increases rapidly after the decrease in the simple luminance value is once relaxed while comparing two adjacent simple luminance values on the luminance value graph in both the left and right directions starting from the point (α) (Hereinafter referred to as “pole”). Whether or not the extreme point has been searched is determined, for example, based on whether or not the increase rate of the value when comparing two adjacent simple luminance values is 5% or more (the increase rate value is not limited to 5%). . Then, the black eye center detection unit 18 that searches for the extreme points one by one on both the left and right sides uses the two locations that have been searched immediately before the decrease in the simple luminance value as gradual as the end points (β) of the subject's black eyes. The midpoint (γ) of a straight line connecting two end points (β) is extracted. Then, the black eye center detection unit 18 detects a point on the X axis representing the X coordinate of the extracted middle point (γ) as the center P2 of the subject's black eye.

  In this way, a portion where the decrease in the simple luminance value searched immediately before the extreme point search is regarded as the end point of the subject's black eye is over-exposed around the black eye portion in the reference image. It is possible to detect the center of the black eye while more reliably eliminating the above. Note that the black eye center detection unit 18 may perform black eye center detection by regarding the pole as an end point of the subject's black eye.

  Further, as shown in FIGS. 15A and 15B, the black eye center detection unit 18 extracts two points on the luminance value graph where the simple luminance value matches the threshold value for black eye center detection. Further, the midpoint of the straight line connecting the two points may be extracted, and a point on the X axis representing the X coordinate of the midpoint may be detected as the center P2 of the subject's black eye. When this method is adopted, it is important to appropriately set a black eye center detection threshold value used for detecting the black eye center P2. For example, when a threshold value for black eye center detection is set on the basis of the input image shown in FIG. 14, if the threshold value is set to 0.7, two points corresponding to the simple luminance value 0.7 on the luminance value graph The straight line distance between them is the closest to the subject's black eye width. Therefore, when detecting the black eye position of the subject based on the input image shown in FIG. 14, it is most appropriate to set the black eye center detection threshold to 0.7.

  Also, the black eye center detection unit 18 first attempts to detect the center P2 of the subject's black eye by a method using the maximum value of the simple luminance value on the luminance value graph, and when the detection is impossible. In addition, the center P2 of the black eye may be detected again by a method using a threshold value for black center detection. For example, when the simple luminance value to be compared becomes “0” before both or one of the two extreme points is searched, the black eye center detection unit 18 may switch to the method using the above threshold value. .

  Thus, the black eye center detection unit 18 detects the center P2 of the subject's black eye by employing at least one of the two methods described above. In other words, the black-eye center detection unit 18 determines that the difference between each vertical luminance value that continuously changes in the luminance value data and the vertical luminance value corresponding to the position of the subject's eyes or corners from the maximum value. For the continuously decreasing area, two lateral points of the reference image are extracted, and the center of the extracted two points is detected as the center P2 of the subject's black eye.

  Here, in the method of using the maximum value of the simple luminance value on the luminance value graph, as shown in FIG. 13B, the end point of the black eye that is closer to the extreme point from one of the extreme points in the luminance value graph. (Β), the point (α) and the end point (β) of the black eye through the other to the other of the extreme points are “the vertical luminance values that change continuously in the luminance value data and the subject's head. Alternatively, it corresponds to a “region where the difference from the vertical luminance value corresponding to the position of the corner of the eye continuously decreases from the maximum value”. Further, in the method using the black eye center detection threshold value, as shown in FIGS. 14 and 15B, the simple luminance value in the luminance value graph matches the black eye center detection threshold value (0.7). The area from one of the two points on the luminance value graph to the other of the two points via the vertex of the graph where the simple luminance value has the maximum value 1 is “each vertical luminance that changes continuously in the luminance value data. This corresponds to “a region where the difference between the value and the vertical luminance value corresponding to the position of the eye or the corner of the subject's eyes continuously decreases from the maximum value”.

  The threshold value for detecting the center of black eye can be arbitrarily set by the user through an input operation from the operation input unit 53 (see FIG. 1). In this threshold setting, a value obtained based on experimental data or the like may be stored in advance in the storage unit 52 or the like.

  Next, as shown in FIGS. 13A and 13B, the black eye position calculation unit 19 determines the difference between the X coordinate value of the center P2 of the black eye and the X coordinate value of the reference point P1 in the luminance value graph. Calculate Then, the black eye position calculation unit 19 detects a horizontal linear distance from the reference point P1 as a calculation result to the center P2 of the black eye as the black eye position of the subject.

<Method for Detecting Black Eye Position of Subject by Black Eye Position Detection Device According to this Embodiment>
Next, a method for detecting the black eye position of the subject by the black eye position detection device 1 will be described with reference to FIG.

  As shown in FIG. 16, first, the input image generation unit 11 acquires an image of the face of the subject photographed by the camera unit 51 and generates an input image (S200; input image generation step, similar to S101). Processing step). Next, the preprocessing unit 12 generates a preprocessed image by performing preprocessing (enhancement processing, filter processing) on the input image acquired from the input image generation unit 11 (S201; preprocessing step).

  Next, the gradation processing unit 14 performs gradation magnification determination processing on the preprocessed image acquired from the preprocessing unit 12, so that the gradation magnification of the upper end pixel row and the lower end pixel row of the preprocessed image, the right end pixel row, and The gradation magnification of the left end pixel column is determined (S202; gradation magnification determination step (gradation processing step)). Then, the gradation processing unit 14 generates a reference image by performing basic gradation processing on the preprocessed image using each determined gradation magnification (S203; basic gradation processing step (gradation processing step)).

  Next, the luminance value graph generation unit 16 generates a vertical luminance value data table by calculating the vertical luminance value of the column of pixels in the vertical direction V in the reference image acquired from the gradation processing unit 14 (S204). ; Vertical direction luminance value data table generation step (luminance value data generation step)). Then, the luminance value graph generation unit 16 generates a luminance value graph using the vertical luminance value data table (S205; luminance value graph generation step (luminance value data generation step)).

  Next, the reference point detection unit 17 extracts the reference point P1 using the luminance value graph generated by the luminance value graph generation unit 16, and detects the X coordinate of the reference point P1 as the position of the subject's eye. (S206; reference point detection step (eye / eye corner detection step)). Next, the black eye center detection unit 18 extracts two points in the luminance value graph generated by the luminance value graph generation unit 16 from which the simple luminance value gradually decreases starting from the maximum simple luminance value, A point on the X axis representing the X coordinate of the center of the two points is detected as the black eye center P2 of the subject (S207; black eye center detection step).

  Next, the black eye position calculation unit 19 calculates the difference between the X coordinate value of the black eye center P2 acquired from the black eye center detection unit 18 and the X coordinate value of the reference point P1 acquired from the reference point detection unit 17. Thus, the black eye position of the subject is detected (S208; black eye position detection step).

[Embodiment 2]
The following will describe another embodiment of the present invention with reference to FIGS. 1 and 17 to 20. For convenience of explanation, members having the same functions as those described in the embodiment are given the same reference numerals, and descriptions thereof are omitted.

  The black eye position detection device 2 according to the present embodiment is different from the black eye position detection device 1 according to the first embodiment in that the preprocessing unit 12 performs a preparation gradation process in addition to the enhancement process and the filter process. Further, in the black eye position detection device 2 according to the present embodiment, the gradation processing unit 14 sets an approximate center position, which will be described later, as a starting point of luminance value enhancement related to the basic gradation processing, and then the basic gradation until a predetermined condition is satisfied. Also in the point which repeats a process, it differs from the black eye position detection apparatus 1 which concerns on Embodiment 1. FIG. Further, as shown in FIG. 1, the black eye position detection device 2 according to the present embodiment also includes a substantially center position calculation unit 13 and a condition determination unit 15, so that the black eye position detection device 1 according to the first embodiment is also provided. And different. The black eye position detection device 2 is built in the tablet terminal 200.

  The preparatory gradation process performed by the pre-processing unit 12 according to the present embodiment is a process performed to facilitate the calculation of the above-described approximate center position, and the processing content includes the basic gradation process performed by the gradation processing unit 14 It is the same. That is, the preparatory gradation process refers to a process that emphasizes the difference between the luminance value in the central portion of the preprocessed image and the luminance value in the peripheral portion of the preprocessed image.

  Further, the preprocessing unit 12 generates a preparation image by performing a preparation gradation process on the preprocessed image. For example, when an enhancement process, a filter process, and a preparation gradation process are performed on an input image as shown in FIG. 17A, a preparation image as shown in FIG. 17B is generated.

  The approximate center position calculation unit 13 calculates the approximate center position of the luminance value distribution in the preparation image acquired from the preprocessing unit 12. Here, the “substantially center” is an approximate center deviated from the center of the black eye of the reference image, and is detected based on, for example, a reference image on which a binarization method has been applied. As this binarization method (threshold determination method), for example, (i) a method using a black eye center detection threshold value as a threshold value for binarization processing (see (a) and (b) of FIG. 15), ( ii) A method of generating a histogram of the entire reference image and using the peak value, (iii) A method of calculating an average value of luminance values of the entire reference image and using the average value, and (i) to (iii) There is a method using a value obtained by multiplying any of the values used in each method by a constant. Also, other binarization methods may be used. As a method of detecting the “substantially center”, for example, a method of calculating an intersection point in an image using a luminance value graph in the vertical direction V and the horizontal direction E described later is used without using a binarization method. It may be detected (see FIG. 17D).

  Specifically, the approximate center position calculation unit 13 binarizes the luminance value of each pixel constituting the preparation image with a predetermined threshold value. Then, the centroid of the region (two-dimensional figure) corresponding to the black eye portion in the preparation image generated by the binarization process is calculated as the approximate center position of the preparation image. As an example of the binarization process, when the binarization process is performed on the preparation image of FIG. 17B, the preparation image becomes an image as shown in FIG. The red area having the largest area in the image corresponds to the black eye portion in the preparation image.

  The method for calculating the approximate center position of the prepared image is not limited to the binarization process. For example, the approximate center position calculation unit 13 uses the preparation image to generate a luminance value graph in the vertical direction V of the preparation image such as the lower graph of the image in FIG. You may produce | generate the luminance value graph of the horizontal direction E of a preparation image like the graph of the right side. Then, as shown in FIG. 17D, the vertical luminance line in the vertical direction V passing through the point where the simple luminance value reaches the maximum value 1 in the vertical direction luminance value graph and the simple luminance in the horizontal direction E luminance value graph. It is also possible to calculate an intersection with a straight line in the horizontal direction E that passes through a point where the value is the maximum value 1, and to set the intersection as a substantially center position (“substantially center” in the figure). However, in this case, it is not necessary to express the maximum value of the simple luminance values in the vertical direction V and the horizontal direction by “1”, and the maximum value may be expressed using other values.

  Here, the luminance value graph in the horizontal direction E refers to the horizontal luminance value indicating the average value (or the added value) of the luminance values in the column of pixels in the horizontal direction E and the position in the vertical direction V of the preparation image. It is a graph showing a relationship. The Y axis of the luminance value graph in the horizontal direction E represents a simple luminance value in which the minimum value of the horizontal luminance value is replaced with “maximum value 1” and the maximum value of the horizontal luminance value is replaced with “minimum value 0”. The X axis represents the distance in the vertical direction V from the lower end of the reference image.

  In the present embodiment, the preprocessing unit 12 performs the preparation gradation process. However, when the preprocessing unit 12 does not perform the process, the approximate center position calculation unit 13 may perform the preparation gradation process.

  In the multi-stage gradation process described later, the condition determination unit 15 first has a threshold value for determining a first condition that shows a sharp change in luminance value near the right end or the left end in the result image after the basic gradation process. It detects whether the following areas exist. In other words, the condition determination unit 15 first determines whether or not the reference point P1 can be detected as the first condition. The condition determination unit 15 does not determine whether or not the first condition is satisfied in the subsequent multistage gradation process after determining that the first condition is satisfied once. Note that the first condition determination threshold value can be set, for example, by the same method as the setting of the black eye position detection threshold value used for black eye center detection by the black eye center detection unit 18.

  When the presence of the region is detected (when it is determined that the reference point P1 can be detected), the condition determination unit 15 determines the approximate center position of the result image and the result generated by the previous process. A linear distance connecting the approximate center position of the image is calculated as a movement amount of the approximate center position, and it is determined as the second condition whether or not the movement amount is equal to or less than a second condition determination threshold value. As the second condition determination threshold value, for example, a distance of about 5% with respect to the horizontal width of the image can be set. Further, as the second condition, the satisfaction condition may be that the number of repetitions of the multi-stage gradation process is a predetermined number (for example, 5 times).

<Multistage gradation processing performed by the gradation processing unit according to this embodiment>
The multistage gradation process performed by the gradation processing unit 14 according to the present embodiment will be described with reference to FIG.

  First, as a preparation for multi-stage gradation processing, as shown in FIG. 18A, the preprocessing unit 12 according to the present embodiment performs enhancement processing, filtering processing, and initial gradation (initial gradation) on an original image (input image). By performing the preparation gradation process, a first result image (corresponding to the result image 1 in the figure, a preparation image) is generated.

  Next, as shown in FIG. 18C, the approximate center position calculation unit 13 performs binarization processing on the first result image, thereby calculating the approximate center position of the first result image. . Then, the approximate center position calculated by the approximate center position calculation unit 13 by the gradation processing unit 14 according to the present embodiment is used as a starting point for luminance value enhancement related to basic gradation processing (hereinafter referred to as “gradation starting point”). A first gradation pattern (corresponding to N = 1 in gradation pattern N in the figure) is generated.

  Next, as shown in FIG. 18B, the gradation processing unit 14 applies the first gradation pattern to the first result image (corresponding to N = 1 in the result image N in the figure). The result image 2 is generated by performing the applied basic gradation process. Then, the approximate center position of the second result image is calculated by the approximate center position calculation unit 13.

  Next, when the condition determination unit 15 determines that the reference point P1 cannot be detected from the second result image, the gradation processing unit 14 according to the present embodiment uses the second gradation pattern (in the gradation pattern N in the drawing). Corresponding to the case of N = 2), and a third result image is generated. On the other hand, when the condition determination unit 15 determines that the reference point P1 can be detected from the second result image, the condition determination unit 15 changes the approximate center position of the first result image to the approximate center position of the second result image. A movement amount is calculated.

  When the movement amount is equal to or less than the second condition determination threshold value, the calculation of the approximate center position, the generation of the gradation pattern using the approximate center position as the gradation start point, and the basic gradation process using the gradation pattern are completed. To do. On the other hand, when it is larger than the threshold value, the gradation processing unit 14 generates a third result image.

  As described above, the approximate center position is calculated until it is determined by the condition determination unit 15 that the reference point P1 can be detected and the movement amount of the approximate center position is determined to be equal to or less than the second condition determination threshold. A series of processes of generating a gradation pattern and a basic gradation process are executed a plurality of times. The repetition of the series of processes is a multistage gradation process. Since the black eye portion in the reference image finally generated by the multistage gradation process is more emphasized, the black eye position can be detected with higher accuracy.

  Note that a series of processes of the calculation of the approximate center position, the generation of the gradation pattern, and the basic gradation process may be performed only once. That is, a result image obtained by applying a gradation pattern to a preparation image that has been subjected to preparation gradation processing may be used as a reference image. In this case, the condition determination part 15 becomes unnecessary.

  From the above, in the multistage gradation process executed by the gradation processing unit 14 according to this embodiment, one or more basic gradation processes are executed.

  Further, an input image may be used instead of the result image as an image to which the gradation pattern is applied. In this way, by fixing the image to which the gradation pattern is applied to the input image, even if the basic gradation process is repeated a plurality of times, the head portion that serves as a reference for detecting the black eye position is added to the finally generated reference image. Can be left.

<Relationship between gradation pattern generation and gradation magnification>
Next, the relationship between the generation of the gradation pattern and the gradation magnification will be described with reference to FIG.

  In the gradation pattern that is sequentially generated with the movement of the substantially central position, the value of the gradation magnification stored in each element of the horizontal gradation array generated by the gradation processing unit 14 varies for each generated gradation pattern. . Therefore, the gradation processing unit 14 according to the present embodiment uses a line in the vertical direction V that passes through the substantially center position as a reference, so that the gradation magnification increases linearly at the same increase rate toward both ends of the result image. Generate a directional gradient array.

  For example, as shown in FIG. 19, when a line in the vertical direction V passing through the approximate center position passes through the position m from the left end of the result image in the horizontal direction E, the gradation processing unit 14 first performs the horizontal gradation arrangement. The gradation magnification of g times is stored in the element [m]. Then, the gradation magnification is sequentially stored in each element at an increase rate of t per pixel toward both ends of the result image. In this case, since the gradation magnification increases at a constant increase rate from the gradation starting point at a position shifted from the center of the result image to both sides, the final arrival point element [0] and element [w−1] ], Gradation values of different values are stored.

<Method for Detecting Black Eye Position of Subject by Black Eye Position Detection Device According to this Embodiment>
Next, a method for detecting the black eye position of the subject by the black eye position detection device 2 will be described with reference to FIG. Note that the processes from S300 to S301 and S308 to S311 in FIG. 20 are the same as the processes from S200 to S201 and from S206 to S208 in FIG.

  As shown in FIG. 20, the pre-processing unit 12 generates a preparation image by performing an emphasis process and a filter process on the input image, and further performing a preparation gradation process (S302; preparation image generation step). . Next, the approximate center position calculation unit 13 that has acquired the image data of the preparation image calculates the approximate center position of the preparation image. And the gradation process part 14 which acquired the position data of a substantially center position sets a gradation starting point based on the said position data (S303; gradation starting point setting process).

  Next, the gradation processing unit 14 generates a gradation pattern based on the set gradation starting point (S304; gradation pattern generation step), and applies the gradation pattern to the preparation image (result image) to generate a result image ( S305: Result image generation step).

  Next, the condition determination unit 15 that has acquired the image data of the result image first determines whether or not the reference point P1 can be detected from the result image (S306; first condition determination step). If it is determined NO (hereinafter abbreviated as “N”) in S306, the condition determination unit 15 transmits a determination result to that effect to the approximate center position calculation unit 13. In this case, the black eye position detection device 2 performs the processing from S303 to S306 again.

  On the other hand, if it is determined as YES (hereinafter abbreviated as “Y”) in S306, the condition determination unit 15 further determines whether or not the movement amount of the approximate center position is equal to or less than the threshold value for the second condition (S307 No. 1). 2 condition determination process). If it is determined as N in S307, the condition determination unit 15 transmits a determination result to that effect to the approximate center position calculation unit 13. Then, the black eye position detection device 2 performs the processing from S303 to S307 again.

  On the other hand, if it is determined as Y in S307, the condition determination unit 15 transmits a determination result to that effect to the gradation processing unit 14. The gradation processing unit 14 that has received the determination result uses the finally generated result image as a reference image, and transmits the image data to the luminance value graph generation unit 16. Then, the black eye position detection device 2 generates a luminance value graph and then executes each process from S308 onward.

[Embodiment 3]
The control blocks (particularly the control unit 50) of the black-eye position detection devices 1 and 2 may be realized by a logic circuit (hardware) formed in an integrated circuit (IC chip) or the like, or a CPU (Central Processing Unit) may be realized. And may be realized by software.

  In the latter case, the black-eye position detection devices 1 and 2 include a CPU that executes instructions of a program that is software that implements each function, and a ROM (Read Only Memory) or a storage device (these are referred to as “recording media”), a RAM (Random Access Memory) for expanding the program, and the like. And the objective of this invention is achieved when a computer (or CPU) reads the said program from the said recording medium and runs it. As the recording medium, a “non-temporary tangible medium” such as a tape, a disk, a card, a semiconductor memory, a programmable logic circuit, or the like can be used. The program may be supplied to the computer via an arbitrary transmission medium (such as a communication network or a broadcast wave) that can transmit the program. The present invention can also be realized in the form of a data signal embedded in a carrier wave in which the program is embodied by electronic transmission.

  The present invention is not limited to the above-described embodiments, and various modifications are possible within the scope shown in the claims, and embodiments obtained by appropriately combining technical means disclosed in different embodiments. Is also included in the technical scope of the present invention.

DESCRIPTION OF SYMBOLS 1, 2 Black eye position detection apparatus 12 Pre-processing part 13 Substantially center position calculation part 14 Gradation processing part 16 Luminance value graph generation part (luminance value data generation part)
17 Reference point detector (eye / eye corner detector)
18 Black eye center detection unit 100, 200 Tablet terminal (electronic equipment)

Claims (6)

Emphasizes the difference between the brightness value at the center of the input image and the brightness value at the periphery of the input image for the input image that represents the subject's eyes by the brightness value of each pixel in the plurality of pixels A gradation processing unit that generates a reference image by performing basic gradation processing,
Luminance value data representing the relationship between the vertical luminance value indicating the integration of the luminance values of a plurality of pixels arranged in the vertical direction in the reference image generated by the gradation processing unit and the horizontal position of the reference image. A luminance value data generation unit to generate;
From the luminance value data, a position near the end of the reference image showing a steep change in the vertical luminance value is extracted, and the extracted position is detected as the position of the eye or the eye corner of the subject. An eye corner detection unit;
For the region where the difference between each vertical luminance value continuously changing in the luminance value data and the vertical luminance value corresponding to the position of the eye or the corner of the eye continuously decreases from the maximum value, A black-eye position detecting device, comprising: a black-eye center detecting unit that extracts two horizontal points and detects the center of the two extracted points as the center of the subject's black eye. (I) enhancement processing for emphasizing a difference in brightness value of each pixel in the plurality of pixels with respect to the input image, and (ii) applying a minimum value filter or a maximum value filter to each pixel in the plurality of pixels. A pre-processed image by performing at least one of a filtering process and (iii) a preparatory gradation process that emphasizes a difference between a luminance value at a central portion of the input image and a luminance value at a peripheral portion of the input image Is further provided with a preprocessing unit for generating
The gradation processing unit sets the preprocessed image as a target image of the basic gradation process as an input image representing the eye of the subject by the brightness value of each pixel in a plurality of pixels. Item 2. The black eye position detection device according to Item 1. Perform a preparatory gradation process that emphasizes the difference between the luminance value at the center of the input image and the luminance value at the periphery of the input image, and calculate the approximate center position of the luminance value distribution in the preparatory image after the preparatory gradation process An approximate center position calculation unit,
The black eye position detection apparatus according to claim 1, wherein the gradation processing unit sets the substantially center position as a starting point of luminance value enhancement related to the basic gradation process.   An electronic apparatus comprising the black eye position detection device according to claim 1.   The program for functioning a computer as each part in the black-eye position detection apparatus of any one of Claim 1 to 3. Emphasizes the difference between the brightness value at the center of the input image and the brightness value at the periphery of the input image for the input image that represents the subject's eyes by the brightness value of each pixel in the plurality of pixels A gradation processing step for generating a reference image by performing basic gradation processing,
In the reference image generated in the gradation processing step, luminance value data representing a relationship between a vertical luminance value indicating integration of luminance values of a plurality of pixels arranged in the vertical direction and a horizontal position of the reference image. Luminance value data generation step for generating
From the luminance value data generated in the luminance value data generation step, a position near the end of the reference image where the vertical luminance value shows a sharp change is extracted, and the extracted position is extracted from the target person. An eye / eye corner detection process for detecting the position of the eye's eye or the eye corner;
The difference between each vertical luminance value that continuously changes in the luminance value data generated in the luminance value data generation step and the vertical luminance value corresponding to the position of the top or bottom of the eye is continuously from the maximum value. And a black eye center detecting step for extracting two points in the lateral direction of the reference image and detecting the center of the two extracted points as the center of the eye of the subject. The black eye position detection method.