WO2018207959A1

WO2018207959A1 - Image processing device and method

Info

Publication number: WO2018207959A1
Application number: PCT/KR2017/004881
Authority: WO
Inventors: 채용욱; 최태준
Original assignee: 주식회사 룩시드랩스
Priority date: 2017-05-11
Filing date: 2017-05-11
Publication date: 2018-11-15
Also published as: KR20190107738A

Abstract

An image processing method, according to one embodiment of the present invention, comprises the steps of: acquiring an eye image; detecting a contour of an image of a target object corresponding to a pupil or an iris from the eye image; calculating an elliptic equation for the target object on the basis of the contour and determining whether the elliptic equation is admitted as a final elliptic equation for the target object; and correcting the contour of the target object image by using a Fourier descriptor when the elliptic equation is not admitted as the final elliptic equation, wherein the Fourier descriptor can be acquired from shape information indicating shape characteristics of the target object image.

Description

Image processing apparatus and method

The present specification relates to an image processing apparatus and method for eye tracking, and more particularly, to an image processing apparatus and method for estimating the size of a pupil or an iris from an eye image.

The pupil is a hollow, circular space located in the center of the iris, through which external light is transmitted to the retina. The main function of these pupils is to reduce chromatic and spherical aberration by adjusting the amount of light reaching the retina and maximizing the depth of focus to maximize visual perception. In other words, the pupil is enlarged to detect a lot of light in the dark place, and the pupil size is reduced in order to reduce the amount of light reaching the retina in the bright place to make a clearer image. In addition, the pupil size changes due to the change of emotion, but the pupil size increases due to fear, surprise, and pain.

As such, the size of the pupil is closely correlated with various cognitive neuroscientific phenomena such as changes in light brightness, attention, cognitive load, and emotion, and is widely used in research in related fields. . In particular, in the field of eye recognition and tracking, it is important to estimate the exact size of the pupil or the iris containing the pupil from the face or eye image.

The present specification provides an image processing method for estimating the size of an iris or a pupil from an eye image.

In addition, the present specification provides an image processing method for performing an elliptic equation evaluation process to calculate an accurate elliptic equation for the iris or pupil covered by the eyelid, to accurately estimate the size of the pupil or iris.

In addition, the present disclosure provides a method for accurately estimating the size of the iris or the pupil by performing a process of correcting the outline of the iris image or the pupil image using a Fourier descriptor to remove the portion covered by the eyelid.

Image processing method according to an embodiment of the present invention comprises the steps of obtaining an eye image (eye image); Detecting an outline of an image of a target object corresponding to a pupil or an iris from the eye image; Calculating an elliptic equation for the target object based on the outline, and determining whether the elliptic equation is recognized as a final elliptic equation for the target object; If the elliptic equation is not recognized as the final elliptic equation, correcting an outline of the target object image using a Fourier descriptor, wherein the Fourier descriptor is shape information indicating a shape characteristic of the target object image. Obtained from; And when the elliptic equation is recognized as the final elliptic equation, acquiring information on at least one of an area, a shape, or a location of the target object using the final elliptic equation. It may include.

In an embodiment, when the outline of the target object image is corrected, determining whether the final elliptic equation is recognized may be calculated by recalculating an elliptic equation for the target object based on the corrected outline. It can be determined whether or not it is recognized as a final elliptic equation for the target object.

As an example, determining whether the final elliptic equation is recognized may include: calculating an error rate based on a distance between each outline point constituting the outline and an ellipse having the elliptic equation, and based on the error rate And determining whether the elliptic equation is a final elliptic equation for the target object.

In an embodiment, the shape information may be obtained based on a radial distance between the center of gravity of the outline points constituting the outline and each outline point.

In an embodiment, the step of correcting an outline of the target object image may include: removing noise on the shape information by using the Fourier descriptor; And removing a part associated with a neighboring object image covering the target object image from the outline by using a slope characteristic of the shape information.

In an embodiment, the removing of the noise on the shape information may include extracting a low frequency region of the Fourier descriptor; And restoring the shape information by inverse Fourier transforming the low frequency region.

In an embodiment, the low frequency region may be a frequency region including frequency components lower than a reference frequency.

The removing of the part associated with the surrounding object image may include: dividing the outline of the target object image and the graph of the shape information into a plurality of parts based on a peak value of the graph representing the shape information; Calculating a max-min gradient for each part; And removing the part having the largest value of the maximum-minimum gradient.

An image processing apparatus according to an embodiment of the present invention, comprising: a memory for storing data; And a processor coupled to a memory, the processor comprising: obtaining an eye image, detecting an outline of an image of a target object corresponding to the pupil or iris from the eye image, Calculate an elliptic equation for the target object based on the above, determine whether the elliptic equation is recognized as the final elliptic equation for the target object, and if the elliptic equation is not recognized as the final elliptic equation, a Fourier descriptor ( correcting the outline of the target object image using a fourier descriptor, wherein the Fourier descriptor is obtained from shape information indicating shape characteristics of the target object image, and when the elliptic equation is recognized as the final elliptic equation, Using an elliptic equation Group may obtain information about at least one of the size of the target object (area), shape (shape) or a position (location).

According to one embodiment of the present specification, an elliptic equation for the iris or the pupil may be calculated by performing an evaluation process of the elliptic equation for the iris or the pupil, and the size of the iris or the pupil obscured by the eyelid may be accurately estimated. have.

In addition, the present specification can remove the portion covered by the eyelid by performing the correction process of the outline of the iris image or the pupil image using the Fourier descriptor, through which the size of the iris or pupil covered by the eyelid can be accurately estimated. .

In addition, various effects according to embodiments of the present invention will be described later in detail with reference to the accompanying drawings.

1 is a block diagram of an image processing apparatus according to an embodiment of the present invention.

2 is an exemplary view of an eye image processed by an image processing apparatus according to an embodiment of the present invention.

3 is an enlarged view of an eye image processed by an image processing apparatus according to an embodiment of the present invention.

4 is a flowchart of an image processing method by an image processing apparatus according to an embodiment of the present invention.

5 is a flowchart illustrating a process of evaluating an elliptic equation by an image processing apparatus according to an embodiment of the present invention.

6 shows shape information and Fourier descriptor for an elliptic equation according to an embodiment of the present invention.

7 is a flowchart illustrating an operation of correcting an outline of a target object image by an image processing apparatus according to an exemplary embodiment.

FIG. 8 is an example of a graph showing detailed steps of outline correction of FIG. 7.

9 and 10 illustrate changes in an ellipse calculated based on an outline before or after the image processing apparatus according to an embodiment of the present invention performs outline correction.

11 is a comparison of results obtained by estimating the size of a target object based on an elliptic equation according to an elliptic equation evaluation and an outline correction according to an embodiment of the present invention.

12 illustrates a calculation time for an image processing method of an image processing apparatus according to an embodiment of the present invention.

FIG. 13 illustrates an iBrain interface system according to an exemplary embodiment of the present invention. FIG.

14 is a block diagram of a host device and a slave device according to an embodiment of the present invention.

FIG. 15 illustrates an iBrain interface (EBI) device according to an embodiment of the present invention.

16 is a block diagram of an EBI device in accordance with an embodiment of the present invention.

The terminology used herein is a general term that has been widely used as far as possible in consideration of the functions in the present specification, but may vary according to the intention of a person skilled in the art, convention or the emergence of a new technology. In addition, in certain cases, there is a term arbitrarily selected by the applicant, and in this case, the meaning will be described in the description of the corresponding embodiment. Therefore, it is to be understood that the terminology used herein is to be interpreted based not on the name of the term but on the actual meaning and contents throughout the present specification.

Moreover, although the embodiments will be described in detail with reference to the accompanying drawings and the contents described in the accompanying drawings, the present invention is not limited or restricted to the embodiments.

Hereinafter, exemplary embodiments of the present invention will be described in detail with reference to the accompanying drawings.

Referring to FIG. 1, an image processing apparatus 1 according to an embodiment of the present invention may include a processor 10 and a memory 20.

The processor 10 may be configured to perform an operation according to various embodiments of the present disclosure. For example, the processor 10 may be configured to perform image processing on an eye image. In particular, the processor 10 may be configured to perform an elliptic equation evaluation and outline correction process to be described later. In this way, the processor 10 may accurately estimate the size of a target object corresponding to the pupil or the iris in real time from the eye image. This will be described in detail below with reference to FIGS. 2 to 12.

In addition, at least one of a module, data, a program, or software for implementing an operation of the image processing apparatus 1 according to various embodiments of the present disclosure may be stored in the memory 20 and executed by the processor 10. In addition, the processor 10 may control other configurations.

The memory 20 is connected to the processor 10 and stores various information / data for driving the processor 10. The memory 20 may be included in the processor 10 or installed outside the processor 10 and connected to the processor 10 by a known means.

The specific configuration of the image processing apparatus 1 of FIG. 1 may be implemented such that various embodiments of the present invention described below are applied independently or two or more embodiments are applied together.

In an embodiment, the image processing apparatus 1 may be included as a module in a device or a system having a photographing function or a gaze tracking function for an eye image, or may be implemented as a separate device. If the image processing apparatus 1 is implemented as a separate device, the image processing apparatus 1 further includes a unit for obtaining an eye image from another device, for example, a communication unit for receiving an eye image from another device, and the like. can do. In addition, embodiments of another apparatus or system including the image processing apparatus 1 will be described below with reference to FIGS. 13 to 17.

As described above with reference to FIG. 1, the image processing apparatus may estimate the size of the pupil or the iris by processing the eye image. Here, the eye image may include images of both eyes or one eye.

In an embodiment, the eye image processed by the image processing apparatus may be a low resolution eye image. For example, as shown in FIG. 2, the eye image may be a low resolution camera (eg, a head mounted display (HMD) device, an eye brain interface (EBI) system / device, etc.) worn by a user. A low-resolution user's eye image captured by a 640 * 480 resolution camera). For another example, the eye image may be an eye image of a low resolution user photographed at a distance.

As such, when the eye image is a low resolution image, resolution of images of objects (eg, pupils, irises, etc.) included in the eye image may also be low. Thus, the contour of this object image will generally consist of a small number of contour points. In this case, all of the individual outline points constituting this outline have a significant meaning, so the precise processing thereof is very important to increase the accuracy of image processing. In other words, at low resolutions, small processing errors can have a big impact on the image processing results, so sophisticated processing of individual outline points is important.

For example, when an ellipse equation for a pupil is estimated based on the outline points of the pupil image to estimate the actual size of the pupil, the precise processing of the individual outline points constituting the pupil image by the image processing device is performed. The treatment is very important. In particular, if the pupil image is obscured by an image of another surrounding object (e.g., eyelid image, etc.), elaborate removal of the outline points of the obscured or obscured portion of the image processing device yields an accurate elliptic equation for the pupil. It is one of the very important factors for.

An image processing apparatus according to an embodiment of the present invention may be a device suitable for processing such a low resolution eye image. Hereinafter, various embodiments of the present disclosure will be described, with the eye image processed by the image processing apparatus as a low resolution image. However, it can be easily understood by those skilled in the art that, depending on the embodiment, the high resolution eye image may be processed by the image processing apparatus through the same or similar processing. In addition, the following mainly describes the image processing process for the pupil portion mainly, but the same or similar description can be easily applied to the image processing process for the iris portion including the pupil it will be readily understood by those skilled in the art. Can be.

3 is an enlarged view of an eye image processed by an image processing apparatus according to an embodiment of the present invention. In the embodiment of FIG. 3, the pupil and iris images in the eye image are hidden by the eyelid image. That is, the eye image of the embodiment of FIG. 3 may be an image showing the eye in a state where the pupil and the iris are covered by the eyelid. Here, the pupil image may be an image of the pupil or an image associated with the pupil part, and may indicate the shape of the pupil. In addition, the iris image is an image associated with the iris portion or the image of the iris, it may represent the shape of the iris. In addition, the eyelid image is an image of the eyelid or an image associated with the eyelid portion, which may represent the shape of the eyelid.

As in FIG. 3 (a), the pupil image 31 in the eye image may be covered by the eyelid image. The actual pupil (or iris) is circular in shape, but since the eye ball is spherical, the image of the pupil (or iris) on the photographed image is generally seen as elliptical. Thus, the contour of the pupil image (or iris image) is in the form of an ellipse. However, as in the embodiment of FIG. 3, when the pupil image is obscured by a straight eyelid image, the outline of the pupil image may not have the shape of an intact ellipse. In this case, the outline 32 of the pupil image detected from the eye image of FIG. 3A may be as shown in FIG. 3B.

The image processing apparatus may calculate an elliptic equation for the pupil based on the outline 32 of the pupil image in order to estimate the size of the pupil. This will be described in detail below with reference to FIG. 4. The outline 33 of the ellipse having the elliptic equation thus calculated may be as shown in FIGS. 3 (c) and 3 (d). Specifically, FIG. 3 (c) shows the outline 32 of the pupil image and the outline 33 of the ellipse having the calculated elliptic equation, and FIG. 3 (d) shows the outline of the ellipse having the calculated elliptic equation. (33) is shown.

3 (c) and 3 (d), the image processing apparatus has calculated an elliptic equation for the pupil based on the outline of the pupil image including the part obscured by the eyelid image. Thus, the shape of the ellipse represented by the calculated elliptic equation and the actual shape of the pupil differ. That is, the calculated elliptic equation cannot represent the exact size of the pupil.

Therefore, the image processing apparatus may perform an image processing process of removing a portion of the pupil image, which is covered by the eyelid image, in order to calculate an accurate elliptic equation. That is, an outline correction process may be performed to remove the hidden portion, which will be described below in detail with reference to FIG. 7. Through this, the image processing apparatus may again calculate an accurate elliptic equation for the pupil, and determine this as the final elliptic equation for the pupil. An elliptic equation evaluation process will be described in detail below with reference to FIG. 5.

The outline 34 of the ellipse having the elliptic equation thus recalculated may be as shown in FIGS. 3 (e) and 3 (f). Specifically, FIG. 3 (e) shows an outline 34 of an ellipse having an elliptic equation calculated again, and FIG. 3 (f) shows an ellipse (ellipse of FIG. 3 (d)) having an elliptic equation calculated first. The outline 33 is compared with the outline 34 of an ellipse (ellipse of FIG. 3 (e)) having an elliptic equation calculated again.

Unlike the embodiments of FIGS. 3 (c) and 3 (d), in the embodiments of FIGS. 3 (e) and 3 (f), the image processing apparatus is based on the outline of the pupil image from which the portion covered by the eyelid image has been removed. Since the elliptic equation for the pupil is calculated, the shape of the ellipse represented by the calculated elliptic equation and the actual shape of the pupil may be substantially the same. That is, the calculated elliptic equation may represent the exact size of the pupil. Thus, this elliptic equation is the final elliptic equation and can be used to obtain geometric information about the pupil.

4 is a flowchart of an image processing method by an image processing apparatus according to an embodiment of the present invention. In detail, FIG. 4 illustrates a method of estimating the size of a pupil or an iris from an eye image by an image processing apparatus according to an exemplary embodiment. Each step of FIG. 4 may be performed by the image processing apparatus or the processor of the image processing apparatus described above with reference to FIG. 2.

Referring to FIG. 4, the image processing apparatus may acquire an eye image (S410). As described above with reference to FIGS. 2 and 3, the eye image may include an image of the target object (target object image) and / or an image of a surrounding object covering the target object (a peripheral object image). In an embodiment, the target object may be a pupil or an iris. In this case, the surrounding object may be an eyelid that hides the pupil or iris (eg, the upper eyelid). Hereinafter, various embodiments of the present invention will be described with a focus on the case where the target object is the pupil or the iris and the surrounding object is the eyelid.

The image processing apparatus may detect an outline of an image of the target object from the eye image (S420). For example, the image processing apparatus may detect an outline of a pupil image or an iris image that is a target object image from the eye image. That is, the image processing apparatus may detect the outline of the image portion associated with the pupil or iris which is the target object from the eye image. In this case, as a method of detecting the outline of the image of the target object, there are a variety of methods depending on the type of image, the resolution, the type of the target object, and the like. As an embodiment, the image processing apparatus may detect the outline of the image of the target object from the eye image using one or more of these known methods. The outline of the detected target object image may be as shown in FIG. 3 (b).

The outline of the detected target object image may include a plurality of outline points. That is, the detected outline may be composed of a plurality of outline points. In this case, the number of outline points constituting the outline may vary according to the resolution of the image. For example, in the case of a low resolution image, the outline may consist of a relatively small number of outline points. Therefore, in the case of low resolution images, the processing of one outline point is very important as described above. Meanwhile, in the case of a high resolution image, the outline may be configured by a relatively large number of outline points.

Also, the outline of the detected target object image may include a portion associated with the image of the surrounding object covering the target object. For example, the outline of the detected pupil image or iris image may include a portion associated with the eyelid image that obscures the pupil image or iris image. For example, some of the plurality of outline points that make up the outline may be associated with an eyelid image that masks the pupil or iris. That is, some of the outline points may be outline points that indicate portions of the eyelid image. This has been described above with reference to FIG. 3.

The image processing apparatus may calculate an elliptic equation for the target object based on the outline of the target object image (S430). In an embodiment, the image processing apparatus may calculate an elliptic equation for the target object based on the plurality of outline points constituting the outline of the target object image. For example, the image processing apparatus may calculate parameters of the elliptic equation for the iris or pupil (eg, the center of the ellipse, the long axis, the length of the short axis, the slope, etc.) using the outline points of the iris image or the pupil image. . In this case, the elliptic equation and its parameters may be as shown in Equation 1 below.

Here, Xc and Yc represent the center point of an ellipse, a and b represent the length of a long axis and a short axis, respectively, and (theta) represents a slope.

As described above, since the actual shape of the target object corresponding to the pupil or iris is circular but the eye ball is spherical, the image of the target object on the photographed eye image is generally seen as an ellipse. Therefore, the elliptic equation calculated based on the outline of the target object image may be used as an equation for representing the shape or size of the target object.

The image processing apparatus may determine whether the calculated elliptic equation is recognized as the final elliptic equation for the target object (S440). This will be described below in detail with reference to FIG. 5. Here, the final elliptic equation may be an elliptic equation used to obtain or estimate geometric information about the target object. For example, the final elliptic equation may be an elliptic equation used to obtain information about at least one of an area, shape or location of the target object.

If the calculated elliptic equation is recognized as the final elliptic equation for the target object, the image processing apparatus uses the final elliptic equation to provide information on at least one of the area, shape or location of the target object. It can be obtained (S450). That is, the image processing apparatus may obtain at least one of size information indicating the size of the target object, shape information indicating the shape of the target object, or location information indicating the location of the target object. Through this, the actual size of the target object may be estimated.

Information about the estimated size and size of the pupil may be used to acquire information such as brightness change of the light, user's attention, cognitive load, and emotion change / analysis in real time. In addition, the estimated information may be used for gaze recognition and estimation.

If the calculated elliptic equation is not recognized as the final elliptic equation for the target object, the image processing apparatus may correct an outline of the target object image by using a Fourier descriptor (S460). As an embodiment, the Fourier descriptor may be obtained from shape information / data indicative of the shape characteristic of the target object image. This will be described below in detail with reference to FIGS. 6 to 7.

When the outline of the target object image is corrected, the image processing apparatus may repeat the above-described steps S430 to S460 on the corrected outline. For example, the image processing apparatus may recalculate an elliptic equation based on the corrected outline of the target object image, and determine whether the calculated elliptic equation is recognized as the final elliptic equation for the target object. When the calculated elliptic equation is recognized as the final elliptic equation for the target object, the image processing apparatus may acquire size information and the like of the target object using the elliptic equation. If the recalculated elliptic equation is not recognized as the final elliptic equation for the target object, the image processing apparatus may recorrect the outline of the target object image using the Fourier descriptor. In this case, the above-described process may be repeated until the calculated elliptic equation is recognized as the final elliptic equation for the target object.

5 is a flowchart illustrating a process of evaluating an elliptic equation by an image processing apparatus according to an embodiment of the present invention. Here, the elliptic equation evaluation process may correspond to step S440 of FIG. 4. That is, the elliptic equation evaluation process may be a step in which the image processing apparatus determines whether the elliptic equation is recognized as the final elliptic equation for the target object. Each step of FIG. 5 may be performed by a processor of the image processing apparatus or the image processing apparatus described above with reference to FIG. 2.

As described above in FIG. 3, the target object may be covered by the surrounding object. In this case, the outline of the target object image may include a portion that is covered by the surrounding object image having different shape characteristics. Therefore, the elliptic equation calculated based on this outline is difficult to accurately estimate the size of the target object and the like. Therefore, before the image processing apparatus obtains the size information and the like of the target object using the calculated elliptic equation, whether the calculated elliptic equation can be recognized as the final elliptic equation that can be used to estimate the size and the like of the target object. An elliptic equation evaluation process can be performed. In this way, the image processing apparatus may acquire accurate size information of the target object based on the exact elliptic equation, without unnecessary calculation process of estimating the size of the target object based on the incorrect elliptic equation.

In an embodiment, the image processing apparatus calculates an error ratio based on the distance between each outline point constituting the outline and an ellipse having an elliptic equation, and determines whether the elliptic equation is the final elliptic equation for the target object according to the error ratio. The elliptic equation can be evaluated in this way. Hereinafter, this will be described in detail with reference to FIG. 5.

Hereinafter, first, a process of calculating an error rate by the image processing apparatus will be described, and then a process of determining whether the elliptic equation is the final elliptic equation based on the error rate will be described. Here, the error rate may be a value representing a ratio of the outline points whose distance to the ellipse among the entire outline points exceeds the reference distance. That is, the error rate may be a value representing the ratio of the outline points corresponding to the ellipse equation, that is, the reference point spaced apart by a reference distance or more from the ellipse represented by the elliptic equation.

First, the image processing apparatus may calculate a distance from an ellipse having an elliptic equation with respect to each outline point constituting the outline (S510). That is, the image processing apparatus may calculate a distance (eg, a minimum distance) between each outline point and an ellipse. For example, the distance d between each outline point and an ellipse may be calculated using Equation 2 below.

For phosphorus (x, y),

Thereafter, the image processing apparatus may calculate an error rate based on the distance d between each outline point and the ellipse. In an embodiment, the image processing apparatus may calculate a ratio of outline points whose absolute value of the distance d is equal to or less than a reference value (for example, 0.1), and calculate an error ratio based on the ratio (S520). In this case, whether the absolute value of the distance d is less than or equal to the reference value may be determined using Equation 3 below.

If you write it down again,

When the error ratio is calculated, the image processing apparatus may determine whether the elliptic equation is the final elliptic equation for the target object based on the error ratio (S530). For example, the image processing apparatus may determine that the elliptic equation is the final elliptic equation for the target object when the error rate is less than the reference ratio (eg, 0.1). That is, when the ratio of the outline points whose distance to the ellipse among the entire outline points exceeds the reference distance is less than the reference ratio, the image processing apparatus may determine that the elliptic equation is the final elliptic equation for the target object. Alternatively, the image processing apparatus may determine that the elliptic equation is not the final elliptic equation for the target object when the error rate is less than the reference ratio (eg, 0.1).

An embodiment in which an elliptic equation whose error rate is less than the reference ratio is calculated is shown in FIG. 3 (d), and an embodiment in which an elliptic equation in which the error rate is less than the reference ratio is calculated is shown in FIG. In this way, the image processing apparatus checks to what extent the outline points exist on or within the reference distance of the ellipse calculated by the outline points, i.e., how well the outline points and the ellipse fit. Based on this, it can be determined whether the elliptic equation can be recognized as the final elliptic equation for the target object. By evaluating the elliptic equation through a process that is processed with a small amount of calculation using this simple algorithm, the image processing apparatus can evaluate the elliptic equation at a high processing speed.

In this specification, the Fourier descriptor is an outline-based such as a wavelet-based shape descriptor, a triangle-based shape descriptor, an ellipse-based shape descriptor. As one of the contour-based shape descriptors, it may be referred to as a Fourier shape descriptor or a Fourier-based shape descriptor.

Such a Fourier descriptor can be obtained from shape information / data that represents a characteristic (shape characteristic or shape signature) for the shape of a particular object. In one embodiment, the shape information may be obtained from the coordinates of the outline points of the particular object or image of the particular object. In another embodiment, shape information may be obtained from a radial distance between the outline points of a particular object or object image and the center of gravity (or centroid) of the outline points. Here, the radial distance may be information indicating the distance between the center point and the outer line point according to the angle change (eg, the counterclockwise angle change). In this case, the shape information may be in the form of a distance function indicating a radius distance according to the change in angle. That is, the shape information may be information expressed as a distance function indicating a radius distance according to the angle change. In the present specification, shape information may be referred to as a shape function, a distance function, or the like.

Hereinafter, shape information about an ellipse and a Fourier descriptor will be described by way of example. At this time, for convenience of explanation, assume an ellipse having a center of (0,0) and an inclination θc of zero, as shown in FIG. Further, it is assumed that the shape information is information represented by the distance function / information representing the above-described radial distance. At this time, the elliptic equation of the ellipse is as shown in Equation 5 below.

The radial distance of the specific coordinates (xi, yi) on the ellipse may be calculated through Equation 6 below.

Shape information can be calculated based on this radial distance information (or function). The graph of the shape information thus calculated may be as shown in FIG. 6 (b). Looking at the characteristics of the shape information obtained from the radial distance information or the radial distance information, in the graph for the radial distance information or shape information, the point with the maximum peak value and the point with the minimum peak value are It may be an edge or an inflection point (eg, points a, b, c, d) in the object.

The Fourier descriptor can be obtained by Fourier transforming this shape information. The graph of the Fourier descriptor thus obtained may be as shown in FIG. 6 (c). Such a Fourier descriptor may represent shape information / characteristics of a corresponding object or object image for each frequency component. However, in general, the high frequency component of the Fourier descriptor for a specific object may be treated as a noise component. For example, in the case of an ellipse object, as shown in FIG. 6C, the high frequency component of the Fourier descriptor may be treated as noise. Therefore, when the high frequency component is removed through the high-pass filter, the noise component of the object is removed. Accordingly, the object having the outline from which the noise is removed may be restored by inverse Fourier transform. Also, as more high frequency components are removed, the object can be simplified with more and more major shape features only. As such, the Fourier descriptor can be used to remove the noise portion of a particular object or object image while maintaining its original shape characteristics.

Hereinafter, a method of correcting the outline of the target object image by the image processing apparatus will be described with reference to FIGS. 7 to 8. The image processing apparatus may perform outline correction to calculate an accurate elliptic equation. To this end, the image processing apparatus may use the above-described Fourier descriptor.

7 is a flowchart illustrating an operation of correcting an outline of a target object image by an image processing apparatus according to an exemplary embodiment. FIG. 8 is an example of a graph showing detailed steps of outline correction of FIG. 7. In the embodiment of FIG. 8, it is assumed that the target object is a pupil. Each step of FIGS. 7 to 8 may be performed by the image processing apparatus or the processor of the image processing apparatus described above with reference to FIG. 2.

In the embodiment of FIG. 7, the correcting the outline of the image of the target object may be the step S460 of FIG. 4. That is, the step of correcting the outline of the image of the target object may be a step of correcting the outline of the target object image using a Fourier descriptor when the calculated elliptic equation is not recognized as the final elliptic equation for the target object. As an embodiment, the Fourier descriptor may be obtained from shape information / data indicative of the shape characteristic of the target object image.

In an embodiment, the process of adjusting the outline may include removing noise on the shape information using a Fourier descriptor and removing a part associated with the surrounding object image covering the target object image from the outline using a slope characteristic of the shape information. can do. Hereinafter, a process of removing noise will be described with reference to steps S710 to S730, and a process of removing a part associated with the surrounding object image will be described with reference to steps S740 to S770.

First, the image processing apparatus may acquire shape information about the target object image (S710). As described above, the shape information may provide information about shape characteristics of the target object image. This is the same as described above with reference to FIG. 6.

In an embodiment, the image processing apparatus may obtain shape information about the target object image based on coordinate information of outline points of the target object image. For example, the image processing apparatus may obtain shape information about the pupil image based on the coordinate information of the outline points of the pupil image.

In another embodiment, the image processing apparatus may obtain shape information about the target object image based on a radial distance between each outline point of the target object image and the center of gravity of the outline points. For example, the image processing apparatus may obtain shape information about the pupil image based on a radius distance between each outline point of the pupil image and the center of the outline points. For example, a graph representing shape information of the pupil image obtained from the eye image of FIG. 3A may be the same as that of FIG. 8A. As shown, the x-axis of the graph of shape information may represent an angle and the y-axis may represent a radial distance. That is, in this case, the shape information may be in the form of a distance function representing the distance according to the angle change.

The image processing apparatus may obtain a Fourier descriptor by Fourier transforming the shape information (S720). In this case, the Fourier descriptor may represent the shape characteristic / information of the target object image for each frequency component. For example, the Fourier descriptor may provide shape characteristics / information for each frequency component of the pupil image. The graph representing the Fourier descriptor thus obtained may be as shown in FIG. 8 (b). As shown, the x-axis of the graph of this Fourier descriptor may represent frequency and the y-axis may represent magnitude. As shown in FIG. 8 (b), the Fourier descriptor for the pupil image shows a form in which amplitudes are concentrated on frequency components below a specific frequency. That is, the Fourier descriptor for the elliptical pupil image may have a shape in which the size is concentrated in the low frequency component.

The image processing apparatus may recover the shape information by extracting the low frequency region of the Fourier descriptor and inversely transforming the low frequency region (S730). Through this, noise on shape information may be removed.

Here, the low frequency region means a frequency region including frequency components lower than a preset reference frequency. By way of example, the reference frequency may be a frequency that allows the recovered shape information to remain sufficiently free of noise while maintaining the original shape characteristics. The reference frequency may be set differently according to, for example, the type of the target object, the resolution of the image, and various other factors. That is, the reference frequency can be variable or flexible.

In an embodiment, the image processing apparatus may include a portion of the low frequency region from the lowest frequency component of the Fourier descriptor information to a frequency component lower than the reference frequency (eg, the 16th lowest frequency component). The shape information can be recovered by extracting the inverse Fourier transform. The reconstructed shape information may be in a state where noise is removed while maintaining original shape characteristics. The graph showing the shape information from which the noise is removed may be as shown in FIG. FIG. 8 (c) shows a graph showing the shape information of FIG. 8 (a) and a graph showing the shape information from which noise is removed.

The image processing apparatus may divide the outline of the target object image and the graph of the shape information into a plurality of parts based on the peak value of the graph representing the shape information (S740). In an embodiment, the image processing apparatus may subdivide the outline and the graph of the target object image into a plurality of parts based on the maximum peak value of the graph representing the shape information. For example, the image processing apparatus may subdivide the graph of the outline of the pupil image and the shape information into a plurality of parts based on the maximum peak value of the graph representing the restored shape information as shown in FIG.

In this case, as shown in FIG. 8D, each part of the outline of the target object image may be matched one-to-one with each part of the graph of shape information. That is, parts (1) to (6) of the outline of the left view of FIG. 8 (d) may correspond to parts (a) to (f) of the graph of the right view of FIG. 8 (d). At this time, for example, as shown in the right drawing of FIG. 8 (d), the part (part (a)) covered by the linear eyelid image in the outline of the oval pupil image is located far from the center of gravity. Therefore, since part (a) has a maximum peak value, as shown in the left figure of FIG. 8 (d), part (a) may correspond to part (1) having the maximum peak value of the shape information graph. In this manner, the image processing apparatus may correspond one-to-one to each part of the outline of the target object image and each part of the graph of the shape information.

The image processing apparatus may calculate a first order gradient of a graph representing shape information (S750). The graph of the primary gradient thus calculated may be as shown in FIG. 8 (e).

The image processing apparatus may calculate a max-min gradient for each part (S760). That is, the image processing apparatus may calculate a difference between the maximum gradient and the minimum gradient for each part. The maximum-minimum gradient thus calculated may be as indicated by the arrow in FIG. 8 (e). In this case, as shown in FIG. 8E, the part (part 1) covered by the eyelid image is almost straight, and thus may have the largest maximum-minimum gradient value.

The image processing apparatus may remove the part having the largest maximum-minimum gradient from the outline of the target object image (S770). In an embodiment, the image processing apparatus may remove all or some of the outline points associated with the part having the largest minimum-minimum gradient, thereby removing the part having the largest maximum-minimum gradient from the outline of the target object image. In this way, the image processing apparatus may correct the outline of the target object image. As described above, the image processing apparatus may use all of the portions that are covered by the peripheral object image having the shape characteristics different from the target object image from the outline of the target object image, using the slope characteristic (eg, the primary gradient value) of the shape information. Some can be removed.

The outline of the corrected target object image may be as shown in FIG. 8 (f). As shown in FIG. 8 (f), the corrected outline of the pupil image may be a state in which part or all of the part (part (a)) covered by the eyelid image is removed.

9 and 10 illustrate changes in an ellipse calculated based on an outline before or after the image processing apparatus according to an embodiment of the present invention performs outline correction. Meanwhile, FIG. 9 shows a change of an ellipse on the eye image, and FIG. 10 shows a change of an ellipse on the outline of the target object image.

9 (a) and 10 (a) show an ellipse a1 (or an outline of an ellipse) calculated based on an outline a3 of an ellipse object image before the image processing apparatus performs outline correction of the target object image. Indicates. 9 (a) and 10 (a), the calculated ellipse a1 does not substantially coincide with the outline a3 of the target object image. As described above, this is because the outline includes a part that is covered by the surrounding object rather than an ellipse. Therefore, the image processing apparatus may not recognize the calculated elliptic equation as the final elliptic equation for the target object, and may perform outline correction of the target object image. Also, the image processing apparatus may recalculate an elliptic equation based on the corrected outline of the target object image, and determine whether the recalculated elliptic equation may be recognized as the final elliptic equation for the target object. The ellipse a2 (or the outline of the ellipse) represented by the elliptic equation thus calculated may be the same as in FIGS. 9B and 10B.

9B and 10B illustrate an ellipse a2 calculated based on an outline a4 of an ellipse object image after the image processing apparatus performs outline correction of the target object image. As illustrated in FIGS. 9B and 10B, the recalculated ellipse a2 may substantially coincide with the outline a4 of the target object image. As described above, this is because the part whose outline is covered by the surrounding object is corrected to be removed. Accordingly, the image processing apparatus may recognize the calculated elliptic equation as the final elliptic equation for the target object, and acquire information on at least one of the size, shape, or position of the target object by using the final elliptic equation. Meanwhile, whether or not the calculated ellipse substantially coincides with the outline of the target object image may be confirmed by evaluating the elliptic equation described above with reference to FIG. 5.

FIG. 9 (c) is a diagram comparing the calculated ellipses before or after the image processing apparatus performs outline correction of the target object image. As shown in FIG. 9C, the ellipse a2 calculated after the outline correction is more closely matched to the outline of the target object image compared to the ellipse a1 calculated before the outline correction is performed. You can check it.

11 is a comparison of results obtained by estimating the size of a target object based on an elliptic equation according to an elliptic equation evaluation and an outline correction according to an embodiment of the present invention. In the embodiment of FIG. 11, it is assumed that the target object is a pupil. In this case, the size of the target object may be the area / area of the pupil. The area of the pupil can be calculated through the equation below.

Pupil area = a * b * π

Here, a and b represent the length of the long axis and short axis of the ellipse, respectively.

In the example of FIG. 11, series 1 shows a result graph when elliptic equation evaluation and outline correction are not performed, and series 2 shows a result graph when elliptic equation evaluation and outline correction are performed. The x-axis of the graph represents the frame number of the video image, and the y-axis represents the width of the pupil. In addition, the graph may be a graph using a median filter for each of 10 frames.

Referring to FIG. 11, in the case of series 2, a change in the area of an actual pupil may be represented more accurately than in the case of series 1. That is, when elliptic equation evaluation and outline correction are performed, it is possible to more accurately obtain information about the actual change in the pupil area.

Referring to FIG. 12, it can be seen that, in the total computation time for the entire process, the proportion of computation time for performing the proposed elliptic equation evaluation and outline correction is very small. Therefore, the additional computation time according to the proposed elliptic equation evaluation and outline correction does not significantly affect the increase of the total computation time. In other words, the proposed elliptic equation evaluation and outline correction can be performed very quickly. In other words, the proposed elliptic equation evaluation and outline correction can be performed with a small amount of computation. Through this, the image processing apparatus may estimate the shape of the pupil or the iris in real time.

## Hereinafter, an exemplary apparatus or system including the image processing apparatus described above or corresponding to the image processing apparatus will be described. However, the EBI device or EBI system of the embodiment of FIGS. 13 to 17 is merely an example of an apparatus or system that may include or correspond to an image processing apparatus, and the image processing apparatus may be included in various other apparatuses, Or other device.

FIG. 13 illustrates an iBrain interface system according to an exemplary embodiment of the present invention. FIG. 14 is a block diagram of a host device and a slave device according to an embodiment of the present invention.

Referring to FIG. 13, an EyeBrain Interface (EBI) system according to an embodiment of the present invention may include a host device 150 and a slave device 100.

The slave device 100 may represent various types of wearable devices that can be worn by a user. For example, the slave device 100 may represent a device that contacts / wears on a user's body part such as a head mounted display (HMD), a headset, a smart ring, a smart watch, an earset, an earphone, and the like. The slave device 100 may include at least one sensor to sense the biosignal of the user through a body part of the user. Here, the biosignal may represent various signals generated from the user's body according to the user's conscious and / or unconscious (eg, breathing, heartbeat, metabolism, etc.) behavior such as pulse, blood pressure, brain wave, and the like. In particular, in the present specification, the slave device 100 may sense the brain wave of the user as the biosignal of the user and transmit the sensing result to the host device 150.

The host device 150 may represent a device operating based on a sensing result of the biosignal received from the slave device 100. In more detail, the host device 150 may be various electronic devices that receive a biosignal sensing result of the user from the slave device 100 and perform various operations based on the received sensing result. The host device 150 may be, for example, various electronic devices such as a TV, a smartphone, a tablet PC, a smart car, a PC, a laptop, and the like.

The EBI system includes a slave device 100 and a host device 150 to provide a control scheme based on a biosignal of a user. Thus, even if the user does not perform a separate input intended for the system, the system directly senses the user's intention by sensing the user's biosignal and is controlled accordingly. As a result, the EBI system provides the user with a more convenient and intentional control method. Hereinafter, the configuration of the slave device 100 and the host device 150 will be described in more detail.

13 and 14, the slave device 100 includes a position marker unit 120, a gaze tracking unit 130, an EEG sensing unit 110, a sensor unit 260, a communication unit 250, and a processor 240. ) May be included.

The position marker unit 120 may include at least one light emitting device (eg, an infrared LED) that emits light. The host device 150 may track the position marker unit of the slave device 100 in real time, whereby the position of the user wearing the slave device 100, the position, the distance between the host device 150 and the user, and The relative position, etc. (hereinafter, 'user's position') can be detected.

When the position marker unit 120 includes a plurality of light emitting elements, the plurality of light emitting elements may be positioned in the position marker unit 120 spaced apart by a predetermined distance. In this case, the host device 150 may detect the relative distance between the host device 150 and the user by tracking the light emitting elements of each position marker unit 120 and measuring the separation distance between the light emitting elements in real time. Can be. For example, when the position marker unit 120 moves away from the host device 150, the separation distance between the light emitting elements measured in the host device 150 is reduced, and the position marker unit 120 is connected to the host device 150. When approaching, the separation distance between the light emitting devices measured by the host device 150 may increase. Based on this, the host device 150 calculates a ratio between the separation distance between the light emitting devices measured in real time and the predetermined separation distance between the actual light emitting devices, and thereby calculates the relative distance between the host device 150 and the user. The distance can be calculated.

In addition, the position marker unit 120 for tracking a user's position may be included in the slave device 100 in various forms, and the host device 150 includes the position, size, and location of these position marker units 120. The position of the user may be detected based on the number, location, and separation distance of the light emitting devices.

The gaze tracking unit 130 may track the gaze of the user. The gaze tracking unit 130 may be provided in the slave device 100 to be positioned around the eyes of the user to track the eyes of the user (eye movement) in real time.

The eye tracking unit 130 may include a light emitting device (eg, an infrared LED) that emits light and a camera sensor that receives (or senses) the light emitted from the light emitting device. The gaze tracking unit 130 may photograph light reflected from the user's eyes with a camera sensor and transmit the photographed image to the processor 240 (video analysis method). Hereinafter, for convenience of description, the eye tracking unit 130 to which the video analysis method is applied will be described as a reference, but is not limited thereto. The eye tracking unit 130 may include a contact lens method (mirror built-in contact) in addition to the above-described video analysis method. User's eyes using the reflected light of the lens, the eye tracking method using the magnetic field of the coiled contact lens) or the sensor attachment method (the eye tracking method using the electric field according to the eye movement by attaching the sensor around the eyes) Can be tracked.

The EEG sensing unit 110 may sense the EEG of the user. The EEG sensing unit 110 may include at least one electroencephalogram (EGE) sensor and / or a magnettoencephalography (MEG) and a near-infrared spectrometer (NIRS). The brain wave sensing unit 110 may be provided at a body (eg, head) contact position where the brain wave of the user may be measured when the user wears the slave device 100, and measure the brain wave of the user. The EEG sensing unit 110 measures an electrical / optical frequency that varies according to the EEG of the various frequencies generated from the body part of the contacted user or the activation state of the brain.

Since EEG is a biosignal, there is a difference between users. Therefore, even in the same cognitive state (for example, concentration / defocus / selection / search, etc.), different patterns of EEG can be extracted for each user. As a result, simply extracting the brain waves of the user and analyzing them on a uniform basis is less accurate in distinguishing the user's current cognitive state. Therefore, in order to accurately measure the cognitive state of the user based on the EEG, the present disclosure provides a calibration method of the EEG according to the current cognitive state for each user.

The sensor unit 260 may include at least one sensing means, and may sense the surrounding environment of the device 100 using the sensing means. In addition, the sensor unit 260 may transmit the sensing result to the processor. In particular, in the present specification, the sensor unit may sense a movement, a movement, and the like of the slave device 100 and transmit a sensing result to the processor 240.

The sensor unit 260 is an inertia measurement unit (IMU) sensor, a gravity sensor, a geomagnetic sensor, a motion sensor, a gyro sensor, an accelerometer, a magnetometer, an acceleration sensor, an infrared ray as a sensing means. The sensor may include an inclination sensor, an altitude sensor, an infrared sensor, an illuminance sensor, a global positioning system (GPS) sensor, and the like. The sensor unit 260 collectively refers to the various sensing means described above. The sensor unit 260 may sense various user inputs and environment of the device, and may transmit a sensing result to allow the processor to perform an operation accordingly. The above-described sensing means may be included in the slave device 100 as a separate element or integrated into at least one or more elements.

The communication unit 250 may communicate with an external device using various protocols, and may transmit / receive data through the communication unit 250. The communication unit 250 may connect to a network by wire or wirelessly to transmit / receive various signals and / or data. The slave device 100 may perform pairing with the host device 150 using the communication unit 250. In addition, the slave device 100 may transmit / receive various signals / data with the host device 150 using the communication unit 250.

The processor 240 may control the position marker unit 120, the eye tracking unit 130, the brain wave sensing unit 110, the sensor unit 260, and the communication unit 250. The processor 240 may control transmission / reception of signals (or data) between the above-described units.

In particular, in the present specification, the processor 240 may transmit a sensing result received from at least one sensor provided in the slave device 100 to the host device 150. In the present specification, the sensing result may refer to raw data obtained by using at least one sensor included in the slave device 100 or data processed through a predetermined algorithm.

In addition, the processor 240 may perform various operations for calibrating the user's gaze and brain waves, which will be described in detail later with reference to FIGS. 6 to 13.

In the above, the configuration units included in the slave device 100 according to an embodiment of the present invention have been described. The slave device 100 may optionally include some of the configuration units shown in FIGS. 1 and 2, and in addition, various units required for the purpose and operation of the device, such as a memory unit, a camera unit, and a power supply unit. May be further included.

The host device 150 may include a camera unit 140, a display unit 210, a communication unit 230, and a processor 220.

The camera unit 140 may photograph the position marker unit 120 of the slave device 100. In more detail, the camera unit 140 may capture the position marker unit 120 of the slave device 100 to obtain a captured image of the position marker unit 120. The camera unit 140 may transmit the acquired captured image to the processor 220, and the processor 220 may process the captured image to acquire a position of a user wearing the slave device 100. In this case, the processor 220 may acquire the position of the user by analyzing the position and size of the position marker units 120, the number of the light emitting elements included therein, the position, and the separation distance.

The camera unit 140 may be configured as a wide angle camera having an angle of view of about 60 degrees or more. When the camera unit 140 is configured as a general camera (camera having an angle of view of less than 60 degrees), the left and right angles of about 60 degrees in front of the host device 150, about 60 to about between the slave device 100 and the host device 150. You can track your position in the 90cm range. However, when the camera unit 140 is configured with a wide angle camera (camera having an angle of view of 60 degrees or more), the left and right angles of about 170 degrees in front of the host device 150, and between the slave device 100 and the host device 150 Position tracking is possible up to a distance of about 3m. Thus, the camera unit 140 of the present invention can be configured as a wide angle camera to obtain more accurate position data of the user.

The display unit 210 may display an image. Here, the image may represent a still image, a moving image, text, a virtual reality (VR) image, an AR (Augment Reality) image, or various other visual expressions including the same, which may be displayed on the screen. The display unit 210 includes a liquid crystal display, a thin film transistor-liquid crystal display, an organic light-emitting diode (OLED), a 3D display, a transparent It may include at least one of a transparent OLED (TOLED). In addition, the display unit 210 may be made of a metal foil, very thin grass, or a plastic substrate. In particular, in the case of a plastic substrate, a PC substrate, a PET substrate, a PES substrate, a PI substrate, a PEN substrate, an AryLite substrate, or the like may be used.

The communication unit 230 may communicate with an external device using various protocols, and may transmit / receive data through the communication device 230. The communication unit 230 may connect to a network by wire or wirelessly to transmit / receive various signals and / or data. The host device 150 may perform pairing with the slave device 100 using the communication unit 230. In addition, the host device 150 may transmit / receive various signals / data with the slave device 100 using the communication unit 230.

The processor 220 may control the camera unit 140, the display unit 210, and the communication unit 230. The processor 220 may control transmission / reception of signals (or data) between the above-described units.

In particular, in the present specification, the processor 220 may perform various commands (or operations) corresponding to the sensing result received from the slave device 100. For example, when the gaze coordinates of the user are received as a result of sensing, the processor 220 may execute a command for selecting a visual object (eg, an icon) at a specific position on the display unit 210 mapped to the gaze coordinates. Can be done. In addition, when user EEG data corresponding to the “focused” state is received as the sensing result, the processor 220 may execute a command for executing the selected visual object (eg, executing an application corresponding to the selected icon). have.

However, in this case, in order to know which point on the display unit 210 the gaze coordinates of the user received are mapped to the display unit 210, the gaze for mapping the specific gaze coordinates of the user and the specific coordinates on the display unit 210 to each other. Calibration needs to be preceded. In addition, as mentioned above, since the EEG pattern according to the cognitive state is different for each user, the EEG calibration for mapping the specific cognitive state of the user and the EEG of a specific frequency also needs to be preceded. Accordingly, the present invention can provide an EBC (Eye Brain Calibration) interface for simultaneously calibrating the user's gaze and brain waves.

In the above, the configuration units included in the host device 150 have been described. The host device 150 may optionally include some of the configuration units shown in FIGS. 13 and 14, and in addition, various units required for the purpose and operation of the device, such as a sensor unit, a memory unit, and a power supply unit. May be further included.

FIG. 15 illustrates an iBrain interface (EBI) device according to an embodiment of the present invention. 16 is a block diagram of an EBI device in accordance with an embodiment of the present invention.

In the present specification, the EBI device 400 may represent a device in which the slave device 100 and the host device 150 described above with reference to FIGS. 1 to 3 are integrated into one device. Therefore, the EBI device 400 may directly sense the biosignal and perform various operations based on the sensing result.

15 and 16, the EBI device 400 may be configured in the form of a wearable device that can be worn on a user's body. The EBI device 400 may include an EEG sensing unit 500, a gaze tracking unit 510, a communication unit 530, a display unit 540, and a processor 520. Since the description of the units included in the EBI device 400 overlaps with the description above with reference to FIG. 14, the following description will focus on the differences.

The EEG sensing unit 500 may sense the EEG of the user. The EEG sensing unit 500 may include at least one electroencephalogram (EGE) sensor and / or a magnettoencephalography (MEG). The brain wave sensing unit 500 may be provided at a body (eg, head) contact position where the user's brain wave may be measured when the user wears the EBI device, and measure the brain wave of the user.

The gaze tracking unit 510 may track the eyes of the user. The gaze tracking unit 510 may be provided in the EBI device 400 to be positioned around the eyes of the user to track the eyes of the user (eye movement) in real time. The gaze tracking unit 510 may include a light emitting device (eg, an infrared LED) that emits light and a camera sensor that receives (or senses) the light emitted from the light emitting device. The eye image photographed by the eye tracking unit 51 may be as shown in FIG. 15C. This is the same as described above with reference to FIG. 2.

The communication unit 530 may communicate with an external device using various protocols, and may transmit / receive data through the communication unit 530. The communication unit 530 may connect to a network by wire or wirelessly to transmit / receive various signals and / or data.

The display unit 540 may display an image. Here, the image may represent a still image, a moving image, text, a virtual reality (VR) image, an AR (Augment Reality) image, or various other visual expressions including the same, which may be displayed on the screen.

The processor 520 may control the EEG sensing unit 500, the eye tracking unit 510, the communication unit 530, the display unit 540, and the communication unit 530. The processor 520 may control transmission / reception of signals (or data) between the above-described units. The processor 520 may perform various operations corresponding to the sensing result received from the EEG sensing unit 500 and / or the gaze tracking unit 510.

In the above, the configuration units included in the EBI device 400 according to an embodiment of the present invention have been described. The EBI device 400 may optionally include some of the component units shown in FIG. 16. In addition, the EBI device 400 may further include various sensors required for the purpose and operation of the device 400 such as a sensor unit, a memory unit, and a power supply unit. Units may be further included.

For convenience of description, the drawings are divided and described, but the embodiments described in each drawing may be merged to implement a new embodiment. In addition, the display device is not limited to the configuration and method of the embodiments described as described above, the above embodiments are configured by selectively combining all or some of the embodiments so that various modifications can be made May be

In addition, while the preferred embodiments have been shown and described, the present specification is not limited to the above-described specific embodiments, and those skilled in the art without departing from the scope of the claims. Various modifications can be made by the user, and these modifications should not be understood individually from the technical spirit or prospect of the present specification.

Various embodiments have been described in the best mode for carrying out the invention.

It can be used in various control systems using brain waves and gaze of the user of the present invention.

Claims

Obtaining an eye image;

Detecting an outline of an image of a target object corresponding to a pupil or an iris from the eye image;

Calculating an elliptic equation for the target object based on the outline, and determining whether the elliptic equation is recognized as a final elliptic equation for the target object;

If the elliptic equation is not recognized as the final elliptic equation, correcting an outline of the target object image using a Fourier descriptor, wherein the Fourier descriptor is shape information indicating a shape characteristic of the target object image. Obtained from; And

If the elliptic equation is recognized as the final elliptic equation, obtaining information on at least one of an area, a shape, or a location of the target object using the final elliptic equation. Image processing method.
The method of claim 1,

When the outline of the target object image is corrected, determining whether the target elliptic equation is recognized is recalculated an elliptic equation for the target object based on the corrected outline, and the elliptic equation is applied to the target object. Image processing method for determining whether or not to be accepted as the final elliptic equation.
The method of claim 1,

Determining whether or not the final elliptic equation is recognized is:

Calculating an error ratio based on a distance between each outline point constituting the outline and an ellipse having the elliptic equation, and determining whether the elliptic equation is a final elliptic equation for the target object based on the error ratio. Image processing method comprising a.
The method of claim 1,

And the shape information is obtained based on a radial distance between the center of gravity of the outline points constituting the outline and each outline point.
The method of claim 1,

Correcting an outline of the target object image may include:

Removing noise on the shape information by using the Fourier descriptor; And

And removing a part associated with the surrounding object image that obscures the target object image from the outline using the slope characteristic of the shape information.
The method of claim 5, wherein

Removing noise for the shape information is:

Extracting a low frequency region of the Fourier descriptor; And

Restoring the shape information by inverse Fourier transforming the low frequency region.
The method of claim 6,

And the low frequency region is a frequency region comprising frequency components lower than a reference frequency.
The method of claim 5, wherein

Removing the part associated with the surrounding object image may include:

Dividing the outline of the target object image and the graph of the shape information into a plurality of parts based on a peak value of the graph representing the shape information;

Calculating a max-min gradient for each part; And

Removing the part having the largest value of the maximum-minimum gradient.
In the image processing apparatus,

A memory for storing data; And

A processor coupled to the memory, the processor comprising:

Acquire an eye image,

Detecting the contour of the image of the target object corresponding to the pupil or iris from the eye image,

Calculate an elliptic equation for the target object based on the outline, determine whether the elliptic equation is recognized as the final elliptic equation for the target object,

If the elliptic equation is not recognized as the final elliptic equation, the outline of the target object image is corrected using a Fourier descriptor, and the Fourier descriptor is obtained from shape information representing the shape characteristic of the target object image. Become,

Image processing, if the elliptic equation is recognized as the final elliptic equation, obtains information on at least one of the area, shape, or location of the target object using the final elliptic equation. Device.
The method of claim 9,

When the outline of the target object image is corrected, determining whether or not it is recognized as the final elliptic equation recalculates an elliptic equation for the target object based on the corrected outline, and the elliptic equation for the target object Determining whether the final elliptic equation is accepted.
The method of claim 9,

To determine whether the final elliptic equation is recognized is:

Calculating an error ratio based on a distance between each outline point constituting the outline and an ellipse having the elliptic equation, and determining whether the elliptic equation is a final elliptic equation for the target object based on the error ratio. And an image processing apparatus.
The method of claim 9,

And the shape information is obtained based on a radial distance between the center of gravity of the outline points constituting the outline and each outline point.
The method of claim 9,

Correcting the outline of the target object image is:

Removing noise on the shape information using the Fourier descriptor; And

And removing a part associated with the surrounding object image that obscures the target object image from the outline using the slope characteristic of the shape information.
The method of claim 13,

To remove noise for the shape information:

Extracting a low frequency region of the Fourier descriptor; And

And restoring the shape information by inverse Fourier transforming the low frequency region.
The method of claim 14,

And the low frequency region is a frequency region that includes frequency components lower than a reference frequency.
The method of claim 15,

Removing the part associated with the surrounding object image is:

Dividing the outline of the target object image and the graph of the shape information into a plurality of parts based on a peak value of the graph representing the shape information;

Calculating a max-min gradient for each part; And

And removing the part having the largest value of the maximum-minimum gradient.