KR101831247B1 - Apparatus for focus measurement in eye tracking system using multi layer perception - Google Patents

Abstract

A focus measuring device for a long-distance line-of-sight camera is provided. The apparatus includes an image acquisition unit that acquires an image of an eye using a narrow angle camera, an input image generation unit that generates an input image by down sampling the acquired image of the eye, A brightness correction unit for correcting the corrected input image with an average gray level of data, a focus measurement method using a mask for further stopping the corrected input image, a focus measurement method using a steel mask, a Har wavel transform method, and a Daubish wavelet transformation method A normalizing unit for nonlinearly normalizing the measured four focus values to a value between 0 and 1, a normalizing unit for normalizing the four normalized focus values by using a sigmoid kernel function, A focus value summing unit for summing a single final focus value through a neural network, And a lens control unit for controlling the focus lens. The Harv wavelet transform method and the Daubish wavelet transform method each calculate a focus value in a four step scale, and calculate a focus value by dividing a high band value by a low band value Are summed with a predetermined weight to measure the focus value. According to the present invention, the focus lens can be moved to a position having a more accurate focus value.

Description

TECHNICAL FIELD [0001] The present invention relates to a focus measuring apparatus for a line-of-sight tracking system using a multi-layer neural network,

The present invention relates to a line-of-sight tracking system, and more particularly to a focus measurement of a camera.

Focus measurement is widely used in imaging applications such as medical imaging, photography, pattern recognition, and eye tracking. Various methods of focus measurement have been studied. The methods can be broadly divided into a spatial domain based method and a wavelet domain based method.

The spatial domain based method is a method of calculating the amount of high frequency using a mask and nonlinear scaling based on the calculated high frequency score to define a focus value. The wavelet domain based method is a method of filtering the input image into multiscale subbands using wavelet transform and defining the focus value at the ratio of low frequency and high frequency.

The spatial domain based method can accurately measure the high frequency components through simple calculation, while the wavelet domain based method can not only measure the focus value accurately but also reduce the influence of the brightness change. Both the spatial domain based method and the wavelet domain based method have good advantages, but there are limitations in accuracy due to several factors.

The factors that reduce the accuracy in the spatial domain based method are the change in brightness, the movement of the user, the error due to eyebrows and eyelashes in the image, and the limited filter range of the mask.

The factors that reduce the accuracy in the wavelet domain-based method are the user's motion, the error due to eyebrows and eyelashes in the image, the brightness abnormality of the illumination (too bright or too dark), and high-frequency noise synthesized in the high-frequency subband.

Such accuracy decreases the accuracy of the focus value. Therefore, it is required to overcome the reduction in the accuracy caused by the elements using a new method that combines the spatial and wavelet domain based methods properly.

In the present invention, a method of combining two methods using a multi-layer neural network is proposed.

SUMMARY OF THE INVENTION It is an object of the present invention to provide a method and apparatus for measuring an auto-focus of a camera.

It is another object of the present invention to provide a method and apparatus for moving a focus lens to a precisely focused position based on a focus value.

According to an aspect of the present invention, a focus measuring apparatus for a long-distance line-tracking camera includes an image acquiring unit for acquiring an image of an eye using a narrow-angle camera, a down-sampling unit for generating an input image by down- A brightness correction unit that corrects the average gray level of the input image to an average gray level of the learning data, a focus measurement method that uses the corrected input image with a daugman mask, a mask of Kang, A focus measuring unit for measuring four focus values using a Haar wavelet transform method and a Daubechies wavelet transform method respectively using the focus measurement method using the first focus value, the Haar wavelet transform method, and the Daubechies wavelet transform method, A normalization unit for non-linearly normalizing the values of the four normalized focus values, And a lens control unit for controlling the focus lens using the final focus value, wherein the Harv wavelet transform method and the Daubish wavelet transform method each calculate a focus value using a four step scale, And the ratio of the high band value divided by the low band value in each of the four step scales is summed by using a predetermined weight to measure the focus value.

According to the present invention, the focus lens can be moved to a position having a more accurate focus value.

According to the present invention, the ability to collect high frequency components in an image based on a spatial domain and the ability to reduce the influence of a change in brightness of an image based on a wavelet domain can be combined by using a multi-layer neural network.

1 shows the hardware of the focus measuring apparatus according to the present invention and the range of motion of the user on the Z distance.
2 shows an example of an image acquired by the coarse camera.
3 is a flowchart illustrating a focus measurement method according to the present invention.
Fig. 4 shows the performance according to the shape of the mask and the Z distance.
Fig. 5 shows the performance according to the steel mask and Z distance.
FIG. 6 shows focus values according to Z distance in two databases when using Dow Corning wavelet transform.
FIG. 7 shows focus values according to Z distance in two databases when using Har wavel wavelet transformation.
8 shows focus values according to Z distance in two databases when using a linear kernel in the MLP, according to the present invention.
FIG. 9 shows focus values according to Z distance in two databases when using the Sigmoid kernel in the MLP, according to the present invention.
FIG. 10 shows focus values according to Z distance in two databases when a hyperbolic tangent kernel is used in MLP according to the present invention.
11 is a block diagram showing an example of a focus measuring apparatus according to the present invention.

Hereinafter, exemplary embodiments of the present invention will be described in detail with reference to the accompanying drawings so that those skilled in the art can easily carry out the present invention. The present invention may, however, be embodied in many different forms and should not be construed as limited to the embodiments set forth herein. Also, in order to clearly illustrate the present invention in the drawings, portions not related to the present invention are omitted, and the same or similar reference numerals denote the same or similar components.

1 shows the hardware of the focus measuring apparatus according to the present invention and the range of motion of the user on the Z distance.

1, the hardware may comprise a wide angle camera, a coarse angle camera, a panning / tilting device, and a monitor (e.g., IPTV) with near infrared illumination at four corners, .

The wide angle camera is for face detection and eye detection on the detected face. The two cameras are fixed on the same axis.

The panning / tilting device is for adjusting the camera axis.

The four lights on the monitor represent the object at the line of sight and create reflected light on the cornea to calculate the line of sight.

The coarse camera acquires the image for focus measurement. The range of movement of the user on the Z-distance between the coarse camera and the user can be known.

2 shows an example of an image acquired by the coarse camera.

Referring to FIG. 2, the image captured by the narrow-angle camera is 1600 * 1200 pixels as shown in FIG. 2 (a). However, the wavelet transform requires a square size based on a multiple of two. Thus, the acquired image is downsampled to a size of 1024 * 1024 pixels as shown in FIG. 2 (b).

3 is a flowchart illustrating a focus measurement method according to the present invention.

Referring to FIG. 3, a snow image having a size of 1600 * 1200 pixels acquired by a coarse camera as shown in FIG. 2 is downsampled to a size of 1024 * 1024 pixels to become an input image (S300).

The input image is brightness compensated (S305). For example, the average gray level of the input image is corrected to the average gray level of the learning data.

There are four methods for the brightness-corrected input image: a spatial domain based method using a Daugman mask, a spatial domain based method using a mask of Kang, a Daubechies wavelet transform method, The focus values are measured by the Haar wavelet transform method (S310). For automatic focusing of the camera of the eye tracking device, it is obtained by using four methods (further mask, steel mask, Daubishi wavelet transformation, Hargh wavelet transformation). The reason for using these four methods in combination is to accurately estimate the difference between the acquired position and the focus position, thereby increasing the accuracy of the focus measurement. Therefore, the present invention can improve the performance of autofocus by controlling the focus lens more precisely, especially in the far-sight line tracking system.

Of the spatial domain based methods, more masks can acquire the highest frequency component in the image with a high and narrow bandwidth. The steel mask has middle and high band widths so that the focus value of the iris region, which is not the highest frequency region in the image, can be measured. Using the above two methods, the highest frequency region in the image and the focus value of the iris region can be measured.

In this way, the space-based method calculates the focus value using the value of the high-frequency portion, while the region-based method considers the low-frequency component as well as the high-frequency component.

In the wavelet-based method, the Dovasi wavelet transform method calculates the focus value by applying a high-pass-filter to the image step by step after the search of the iris region. If the DWI wavelet transform method is used without the line search of the iris region, only the focused image and the unfocused image can be classified.

The wavelet transform method of the wavelet-based method does not require the line search of the iris region. In contrast to DWI wavelet transform, the wavelet transform of Harsh wavelet transform is not applied to the whole image step by step but it shows the difference value between even element and continuous odd element and performs calculation once For elements, do not use them again in the same step.

In other words, the DWI wavelet transform method requires a region of interest (ROI) method and the filter is used in a stepwise manner. On the other hand, the wavelet transform method is a method using the entire image, But they have different characteristics.

The focus value measured by each of the four methods having different characteristics is normalized to a value between 0 and 1 (S315).

The four normalized focus values are input to a pre-trained Multi Layer Perception (MLP), and a final focus value is output (S320). The Z-distance is estimated by calculating the difference between the focal position and the image acquisition position through MLP using the normalized value as four inputs. After the linear Z-distance is estimated using the multilayer neural network, the focus lens of the camera of the gaze tracking device can be controlled based on the Z-distance.

For example, a multi-layer neural network can classify input images at four distances of 0, 5, 10, and 20 cm from the focused position, respectively. Depending on the output of the test procedure of the multilayer neural network, the focus lens is adjusted back and forth.

Various types of kernels can be used for multilayer neural networks. For example, a linear kernel, a sigmoid kernel, and a hyperbolic tangent kernel may be used.

Fig. 4 shows the performance according to the shape of the mask and the Z distance. 4 (a) shows a more limited mask, and FIG. 4 (b) shows a focus value according to the Z distance in two databases when using a further mask.

Referring to FIG. 4, the more masks can be represented by the "sinc" function in 2-D fourier transform as an 8 * 8 convolution kernel, as shown in the following equation.

Here, v and v are spatial frequencies, and D is the more mask.

In 2-D fourier space, the focus step is measured by calculating the sum of high frequencies in the spatial domain.

In order to obtain the focus value in the range from 0 to 1, the sum of spectrums is subjected to nonlinear normalization as shown in the following equation.

Here, x represents the total sum of spectra, c represents an offset value, and f (x) represents a focus value.

Fig. 5 shows the performance according to the steel mask and Z distance. 5 (a) shows a steel mask, and FIG. 5 (b) shows a focus value according to Z distance in two databases when using a steel mask.

Referring to FIG. 5, a steel mask can be expressed by a 5 * 5 convolution kernel and a 2-D Fourier transform as a band pass filter as shown in the following equation.

Where v and v are spatial frequencies and K is a steel mask. In the Fourier space, the focusing step is calculated by the sum of the spectra at the middle frequency and the high frequency.

The sum of the spectra is subjected to nonlinear normalization as in Equation (2) to obtain the focus value.

FIG. 6 shows focus values according to Z distance in two databases when using Dow Corning wavelet transform.

Referring to FIG. 6, a result of applying the multiscale DWI wavelet transform of four stages to the input image can be known. The average of the high frequency components and the average of the low frequency components at each scale are calculated by the total HH and LL subbands, respectively. The focusing step of each scale is calculated by the following equation as the ratio of the average of the high frequency components divided by the average of the low frequency components.

와

는 각각 고 주파수 성분의 평균과, 저 주파수 성분의 평균을 나타낸다. R은 하나의 스케일에서의 초점 단계를 정의하는 비율 이다. From here ,

Wow

Represent the average of the high frequency components and the average of the low frequency components, respectively. R is the ratio defining the focus step in one scale.

The focus step of the entire transformed image is the weighted sum of the four ratios of the four scales, as shown in the following equation.

와

는 각각 i번째 스케일의 고주파 성분의 평균과 저주파 성분의 평균을 의미한다. Here, f ₁ is the focus step in the entire transformed image, W _i is the weight of the i-th scale,

Wow

Means an average of the high-frequency components of the i-th scale and an average of the low-frequency components of the i-th scale, respectively.

The weight value can be a statistically derived value to make the four features common denominator.

The focus step is normalized as in the above equation to obtain a focus value on the Daubish wavelet transform.

FIG. 7 shows focus values according to Z distance in two databases when using Har wavel wavelet transformation.

Referring to FIG. 7, the wavelet transform process is the same as the DWT wavelet transform process. First, the input image is subjected to wavelet transform of four scales, and a ratio obtained by dividing the average of the high frequency components by the average of the low frequency components is obtained. The weighted sum of the four ratios is defined as the focus step of the entire image. The focus value is finally calculated by nonlinearly normalizing the focus step as shown in Equation (2).

The more mask, the strong mask, the dowby wavelet transform, and the wavelet transform of FIG. 4 to FIG. 7 are the four inputs of the learning of the multilayer neural network.

In a multi-layer neural network learning process, various kernel functions can be used. For example, a kernel function of either a linear kernel, a sigmoid kernel or a hyperbolic tangent kernel can be used.

Figure 8 shows focus values according to Z distance in two databases when using a linear kernel in an MLP, in accordance with the present invention.

9 shows focus values according to Z distance in two databases when using the sigmoid kernel in the MLP according to the present invention.

10 shows focus values according to Z distances in two databases when using a hyperbolic tangent kernel in the MLP, according to the present invention.

When comparing the slope and the standard deviation between the kernels of FIGS. 8 to 10, the sigmoid kernel has a good result, and other kernels have good results.

11 is a block diagram showing an example of a focus measuring apparatus according to the present invention.

11, the focus measuring apparatus of the far-sight line tracking camera includes an image acquisition unit 1105, an input image generation unit 1110, a brightness correction unit 1115, a focus measurement unit 1120, a normalization unit 1125, A focus value summing unit 1130, and a lens control unit 1135.

The image acquisition unit 1105 acquires an image of an eye using a narrow-angle camera.

The input image generating unit 1110 down-samples the acquired eye image to generate an input image.

The brightness correction unit 1115 corrects the average gray level of the input image to the average gray level of the learning data.

The focus measuring unit 1120 measures a focus measurement method using a mask, a focus measurement method using a steel mask, a Haar wavelet transform method, and a Daubishi wavelet transform method, as shown in FIG. 4 to FIG. Are used to measure four focus values. The Harv wavelet transform method and the Daubish wavelet transform method each calculate a focus value using four stages of scales and sum the ratios of the high band values divided by the low band values in each of the four step scales using predetermined weights, .

The normalizer 1125 non-linearly normalizes the measured four focus values to values between 0 and 1, respectively. Also, the normalization unit 1125 nonlinearly normalizes the final focus value summed by the focus value summing unit 1130 to a value between 0 and 1.

The focus value summing unit 1130 sums the normalized four focus values into one final focus value through a multi-layer perception using a sigmoid kernel. Multilayer neural networks can also use linear or hyperbolic tangent kernel functions.

The lens control unit 1135 controls the focus lens using the final focus value.

In the above-described exemplary system, the methods are described on the basis of a flowchart as a series of steps or blocks, but the present invention is not limited to the order of the steps, and some steps may occur in different orders . It will also be understood by those skilled in the art that the steps shown in the flowchart are not exclusive and that other steps may be included or that one or more steps in the flowchart may be deleted without affecting the scope of the invention.

The foregoing description is merely illustrative of the technical idea of the present invention, and various changes and modifications may be made by those skilled in the art without departing from the essential characteristics of the present invention. Therefore, the embodiments disclosed in the present invention are intended to illustrate rather than limit the scope of the present invention, and the scope of the technical idea of the present invention is not limited by these embodiments. The scope of protection of the present invention should be construed according to the following claims, and all technical ideas within the scope of equivalents should be construed as falling within the scope of the present invention.

Claims (10)

A focus measuring apparatus for a long distance line tracking camera,
An input image generating unit for generating an input image based on an image of a region in which an eye exists;
A brightness correction unit that corrects the brightness of the input image to a brightness corresponding to an average gray level;
A focus measuring unit for applying the plurality of different focus measurement methods to the corrected input image to generate the plurality of focus values corresponding to the corrected input image;
A normalization unit for non-linearly normalizing the plurality of focus values to a predetermined range of values;
A focus value summation unit for calculating a final focus value using the normalized plurality of focus values using a multi-layer neural network; And
And a lens control unit for controlling the focus lens using the final focus value. 2. The method of claim 1,
A focus measurement method using a mask of Kang, a Haar wavelet transformation method, and a Daubechies wavelet transformation method using a mask of a Kang, a focus measurement method using a daugman mask, a Kang mask, . The image processing apparatus according to claim 1,
Down-sampling an image of a region in which the eye exists, and generating the down-sampled image as an input image. The apparatus of claim 1,
And corrects the average gray level of the input image to the average gray level of the learning data. 2. The apparatus of claim 1, wherein the focus-
Wherein the final focus value is calculated by applying a multi-layer neural network using a sigmoid kernel function to the normalized plurality of focus values. 3. The method of claim 2,
The Harv wavelet transform method and the Daubish wavelet transform method each calculate a focus value using a scale of four steps and use the ratio of the high band value divided by the low band value in each of the four step scales to a predetermined weight to measure the focus value Wherein the focus detection unit detects the focus position of the focus lens. A method of measuring a focus of a camera,
Generating an input image based on an image of a region in which an eye exists,
Correcting the brightness of the input image to a brightness corresponding to an average gray level,
Generating a plurality of focus values corresponding to the corrected input image by applying a plurality of different focus measurement methods to the corrected input image,
Linearly normalizing the plurality of focus values to values of a predetermined range;
Calculating a final focus value using the plurality of normalized focus values using a multi-layer neural network;
And controlling the focus lens using the final focus value. 8. The method of claim 7,
A focus measurement method using a mask of Kang, a Haar wavelet transformation method, and a Daubechies wavelet transformation method using a mask of a Kang, a focus measurement method using a daugman mask, a Kang mask, . The method of claim 7, wherein the calculating of the final focus value comprises:
Wherein the final focus value is calculated by applying a multi-layer neural network using a sigmoid kernel function to the four normalized focus values. 9. The method of claim 8,
The Harv wavelet transform method and the Daubix wavelet transform method each calculate a focus value using a scale of four steps and use a ratio obtained by dividing a high band value by a low band value in each of the four step scales as a predetermined weight, And measuring the focal distance.

Images