KR20160057888A - Electronic device and method for determining a gaze of a user in electronic device - Google Patents

Abstract

Embodiments of the present invention may include: a camera unit capturing an image; and a processor detecting one or more reflections on a cornea generated by a plurality of light sources from the image, and determining a gaze by a mapping function corresponding to the number of the detected reflections from a plurality of mapping functions for mapping from an image plane to a screen plane. Also, a variety of other embodiments may be provided.

Description

TECHNICAL FIELD [0001] The present invention relates to an electronic device and a method for determining a line of sight in an electronic device,

Various embodiments of the present invention are directed to electronic devices and methods of determining a line of sight in an electronic device.

Examples of methods for judging and / or tracking a gaze of a user in an electronic device include a method of determining and determining a gaze using information such as an iris, a pupil, or a glint of the cornea, / Tracking methods are being studied.

In order to check what portion of the user's gaze is looking at on the display screen, the user is asked to look at a predetermined position and the direction of the user's gaze is modeled by analyzing, for example, iris, pupil, .

The technique of predicting and tracking the point of gaze (POG) of the user's viewpoint on the screen has been recently developed due to its intuitive and convenient usage method, the development of various human-device interface technologies And so on.

Recent line-of-sight technologies use IR cameras (IR cameras) and IR LED (light emitting diode) lighting devices for accurate performance as well as visible cameras. These techniques use the coordinates of a corneal reflection (hereinafter referred to as 'CR') in which a light source (for example, IR illumination) is reflected from the eye, together with the pupil center . This is called the pupil center corneal reflex (PCCR) technique, and there may be various eye tracking methods depending on the number of CRs used.

According to the present eye gaze determination or tracking method, eye gaze determination is performed by a gaze prediction method based on a predetermined number of CRs. For example, when a user gazes at a corner of a screen, the number of detected CRs is not sufficiently secured There is a disadvantage in that it is impossible to predict the gaze even if the calibration is normally completed.

The various embodiments of the present invention can provide an electronic device capable of eye gazing and tracking and a gaze determination method in an electronic device regardless of the number of CRs detected in a CR-based gaze prediction method, for example .

According to an embodiment of the present invention, there is provided an electronic device comprising: a camera unit for capturing an image; And a mapping corresponding to the number of detected detected points from a plurality of mapping functions for detecting at least one reflection point on the cornea generated by the plurality of light sources from the photographed image and mapping from an image plane to a screen plane And a processor for determining a line of sight by a function.

Also, an operating method of an electronic device according to any one of the various embodiments is characterized by the steps of: detecting at least one reflex point on the cornea generated by a plurality of light sources; Determining a pattern of the detected reflection points; And determining a line of sight by a mapping function corresponding to the number of reflection points among a plurality of mapping functions for mapping from an image plane to a screen plane.

The method of determining a line of sight in an electronic device and an electronic device according to various embodiments can determine and track a line of sight regardless of the number of CRs detected in a CR-based line of sight prediction method.

1 is a diagram showing an example of the configuration of an electronic device according to an embodiment of the present invention.
2 is a diagram illustrating modeling of eye tracking in accordance with various embodiments of the present invention.
3 is a diagram illustrating a mapping function of the gaze prediction method according to various embodiments of the present invention.
4 is a diagram illustrating a calibration method for eye tracking according to various embodiments of the present invention.
5 is a diagram illustrating a homography normalization method in accordance with various embodiments of the present invention.
6 is a diagram illustrating a method of measuring a line of sight according to various embodiments of the present invention.
Figure 7 is an illustration of examples of detectable reflections according to various embodiments of the present invention.
8 is a diagram showing a result of eye gaze measurement according to various embodiments of the present invention.
9 is a flow chart illustrating a calibration procedure of an electronic device according to various embodiments of the present invention.
10 is a flow chart illustrating a line-of-sight tracing procedure of an electronic device according to various embodiments of the present invention.
11 is a view showing a concept of a gaze measurement method according to various embodiments of the present invention.
12 is a diagram illustrating a method for modeling a detectable reflection point in accordance with various embodiments of the present invention.
13 is a diagram illustrating a method of encoding a pattern of detectable points of refraction in accordance with various embodiments of the present invention.
14 is a diagram showing a concept of a gaze measurement method according to various embodiments of the present invention.
FIG. 15 is a diagram illustrating a result of eye-tracking experiments according to various embodiments of the present invention.
16 is a diagram illustrating an electronic device in which a light source is disposed according to various embodiments of the present invention.
17 is a view showing a user's eyes according to various embodiments of the present invention.
18 is a diagram showing target points used for calibration and testing on the screen of an electronic device according to various embodiments of the present invention.
19 is a block diagram showing a detailed structure of an electronic device according to an embodiment of the present invention.
20 is a block diagram of a program module in accordance with various embodiments of the present invention.

Hereinafter, various embodiments of the present disclosure will be described with reference to the accompanying drawings. It should be understood, however, that this disclosure is not intended to limit the present disclosure to the particular embodiments, but includes various modifications, equivalents, and / or alternatives of the embodiments of the present disclosure. In connection with the description of the drawings, like reference numerals may be used for similar components.

In this document, the expressions "have," "may," "include," or "include" may be used to denote the presence of a feature (eg, a numerical value, a function, Quot ;, and does not exclude the presence of additional features.

In this document, the expressions "A or B," "at least one of A and / or B," or "one or more of A and / or B," etc. may include all possible combinations of the listed items . For example, "A or B," "at least one of A and B," or "at least one of A or B" includes (1) at least one A, (2) Or (3) at least one A and at least one B all together.

Expressions such as " first, "second," first, "or" second, " as used in various embodiments, Not limited. The representations may be used to distinguish one component from another. For example, the first user equipment and the second user equipment may represent different user equipment, regardless of order or importance. For example, without departing from the scope of the present disclosure, the first component may be referred to as a second component, and similarly, the second component may be named as the first component.

(Or functionally or communicatively) coupled with / to "another component (eg, a second component), or a component (eg, a second component) Quot; connected to ", it is to be understood that any such element may be directly connected to the other element or may be connected through another element (e.g., a third element). On the other hand, when it is mentioned that a component (e.g., a first component) is "directly connected" or "directly connected" to another component (e.g., a second component) It can be understood that there is no other component (e.g., a third component) between other components.

As used herein, the phrase " configured to " (or set) to be "adapted to, " To be designed to, "" adapted to, "" made to, "or" capable of ". The term " configured (or set) to "may not necessarily mean " specifically designed to" Instead, in some situations, the expression "configured to" may mean that the device can "do " with other devices or components. For example, a processor configured (or configured) to perform the phrases "A, B, and C" may be a processor dedicated to performing the operation (e.g., an embedded processor), or one or more software programs To a generic-purpose processor (e.g., a CPU or an application processor) that can perform the corresponding operations.

The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the scope of the other embodiments. The singular expressions may include plural expressions unless the context clearly dictates otherwise. All terms used herein, including technical or scientific terms, may have the same meaning as commonly understood by one of ordinary skill in the art of the present disclosure. Commonly used predefined terms may be interpreted to have the same or similar meaning as the contextual meanings of the related art and are not to be construed as ideal or overly formal in meaning unless expressly defined in this document . In some cases, the terms defined herein may not be construed to exclude embodiments of the present disclosure.

An electronic device in accordance with various embodiments of the present disclosure may be, for example, a smartphone, a tablet personal computer, a mobile phone, a videophone, an e-book reader reader, a desktop personal computer, a laptop personal computer, a netbook computer, a workstation, a server, a personal digital assistant (PDA), a portable multimedia player (PMP) , Mobile medical devices, cameras, or wearable devices such as smart glasses, head-mounted-devices (HMD), electronic apparel, electronic bracelets, electronic necklaces, (e. g., apps, e-tat, smart mirror, or smart watch).

In some embodiments, the electronic device may be a smart home appliance. Smart home appliances include, for example, televisions, digital video disk players, audio, refrigerators, air conditioners, vacuum cleaners, ovens, microwaves, washing machines, air cleaners, set- (Such as a home automation control panel, a security control panel, a TV box such as Samsung HomeSync ™, Apple TV ™ or Google TV ™), a game console (eg Xbox ™, PlayStation ™) A dictionary, an electronic key, a camcorder, or an electronic frame.

In an alternative embodiment, the electronic device may be any of a variety of medical devices (e.g., various portable medical measurement devices such as a blood glucose meter, a heart rate meter, a blood pressure meter, or a body temperature meter), magnetic resonance angiography (MRA) A global positioning system receiver, an event data recorder (EDR), a flight data recorder (FDR), an automotive infotainment device, a navigation system, a navigation system, Electronic devices (eg marine navigation devices, gyro compass, etc.), avionics, security devices, head units for vehicles, industrial or home robots, ATMs (automatic teller's machines) point of sale, or internet of things such as light bulbs, various sensors, electricity or gas meters, sprinkler devices, fire alarms, thermostats, street lights, toasters ), A fitness equipment, a hot water tank, a heater, a boiler, etc.).

According to some embodiments, the electronic device is a piece of furniture or a part of a building / structure, an electronic board, an electronic signature receiving device, a projector, Water, electricity, gas, or radio wave measuring instruments, etc.). In various embodiments, the electronic device may be a combination of one or more of the various devices described above. An electronic device according to some embodiments may be a flexible electronic device. Further, the electronic device according to the embodiment of the present disclosure is not limited to the above-described devices, and may include a new electronic device according to technological advancement.

1 is a diagram showing an example of the configuration of an electronic device according to an embodiment of the present invention. 1, an electronic device according to an exemplary embodiment of the present invention includes a controller 110, a light source 120, a camera 130, a display 140, an input unit 150, Or at least one of them. The control unit 110 may include at least one of a calibration processing unit 111, a mapping function determination unit 112, a reflection point detection unit 113, and a gaze determination unit 114.

The calibration processing unit 111 may perform a function of processing calibration of the user's gaze. According to various embodiments of the present invention, the calibration processing unit 111 may be implemented to perform calibration through a separate calibration setting menu. For example, when the user executes the calibration function through the input unit 150, a mark is displayed at at least one predetermined position through the display unit 140, and when the user looks at the mark, The calibration can be performed based on the user's gaze information.

The visual line determining unit 114 may perform a function of determining the user's line of sight. In addition, the gaze determining unit 114 may further perform a function of determining the gaze of the user and tracking the gaze of the user.

The gaze determination method of the gaze determination unit 114 may be implemented by various gaze determination algorithms, and various embodiments of the present invention are not limited to specific algorithms. For example, according to various embodiments of the present invention, the eye-gaze determining unit 114 models the shape of the eyeball using information such as iris, pupil, and corneal flicker of the user, and thereby determines or tracks the user's gaze .

In addition, according to various embodiments of the present invention, the gaze determination unit 114 may determine the user's gaze (or the gaze direction) in conjunction with the camera unit 130 or the light source unit 120. [ For example, the user's face or eyeball can be photographed through the camera unit 130, and the user's eyes can be determined by analyzing the photographed image.

In addition, according to various embodiments of the present invention, the light source unit 120 may emit light of at least one light source under the control of the controller 110. [ When the light source emits light through the light source unit 120, the gaze determining unit 114 photographs the user's face or eyeball through the camera unit 130, and obtains a light source (e.g., an eyeball The corneal reflex point (CR) reflected from the cornea) can be determined to determine the user's gaze.

The calibration information 161 processed through the calibration processing unit 111 and / or the line of sight information 163 determined through the line of sight determiner 114 may be stored in the storage unit 160. Also, according to various embodiments of the present invention, the calibration information 161 and / or the sight line information 163 may be stored corresponding to each user information.

According to various embodiments of the present invention, when the gaze determining unit 114 determines the gaze based on the corneal reflex point, the gaze determining unit 114 can determine and track gaze regardless of the number of corneal reflex points detected.

For example, according to various embodiments of the present invention, when the gaze determination or the gaze tracking function is operated, the reflection point detection unit 113 can detect the corneal reflection point from the image photographed through the camera unit 130. [

The mapping function determination unit 112 can determine a mapping function corresponding to the number of reflection points detected by the reflection point detection unit 113 and / or the pattern of reflection points. For example, the mapping function determining unit 112 refers to the mapping function information 162 stored in the storage unit 160 and determines a mapping function preset in correspondence with the number of the detected reflection points and / Can be determined as a mapping function. The mapping function is a function related to the number of reflection points and / or patterns applicable to the pattern, for example, a homography transform based normalization (HTN) scheme, an affine transform based normalization (ATN) And / or < / RTI > A specific example of a method for determining the mapping function corresponding to the number of reflection points and / or the pattern of reflection points will be described later.

When the mapping function to be applied to the gaze determination is determined by the mapping function determination unit 112, the gaze determination unit 114 determines information about the reflection point detected through the reflection point detection unit 113 (for example, ) To the mapping function determined through the mapping function determination unit 112, the user's gaze can be determined regardless of the number of reflection points detected.

In the various embodiments of the present invention, each functional unit or module may mean a functional and structural combination of hardware for performing the technical idea of various embodiments of the present invention and software for driving the hardware. For example, each functional unit or module may refer to a logical unit of a predetermined code and a hardware resource for executing the predetermined code, and may be a code that is physically connected, or a type of hardware Or may be easily deduced to the average expert in the field of the embodiments of the present invention.

An electronic device according to any one of the various embodiments of the present invention includes: a camera unit for photographing an image; And a mapping corresponding to the number of detected detected points from a plurality of mapping functions for detecting at least one reflection point on the cornea generated by the plurality of light sources from the photographed image and mapping from an image plane to a screen plane And a processor for determining a line of sight by a function.

According to various embodiments of the present invention, the processor can determine a line of sight by a mapping function corresponding to the detected pattern of the reflection points.

According to various embodiments of the present invention, the processor may be configured to determine a plurality of light source patterns by combining at least one selected light source of the plurality of light sources, and to determine, for each of the plurality of light source patterns, Function can be determined.

According to various embodiments of the present invention, the plurality of light source patterns may be mapped to coded data and stored.

According to various embodiments of the present invention, the plurality of light source patterns may be encoded into a binary code.

According to various embodiments of the invention, the light source may comprise an infrared (LED) light emitting diode.

According to various embodiments of the present invention, the camera unit may include an infrared camera.

According to various embodiments of the present invention, the processor may convert from the image plane to a virtual normalized plane and map the virtual normalized plane to the screen plane.

According to various embodiments of the present invention, the mapping function for mapping from the hypothetical normalized plane to the screen plane may be represented by a polynomial function.

According to various embodiments of the present invention, the processor may determine a polynomial mapping function for each of the patterns for a plurality of detected reflections during a calibration operation for gaze determination.

2 is a diagram illustrating modeling of eye tracking in accordance with various embodiments of the present invention. Referring to FIG. 2, eye tracking can be modeled by various methods.

For example, the gaze tracking method can be geometrically modeled using HN (homography normalization) as an example of gaze tracking modeling. The model based on the HN technique may be composed of three planes. First, a monitor screen plane (П _S ) 210 with four IR light sources (L ₁ -L ₄ ) 211, 212, 213, and 214 attached to four corners of the screen, four CR (G ₁ -G ₄ ) 230, 231, 232, 233, and 234, and a camera image plane (П _I ) 220. The corneal plane (П _C )

In the process of obtaining the input eye image, four IR light sources (L1 to L4) 211, 212, 213 and 214 are _first formed on the corneal (G ₁ -G ₄ ) 233, 234), followed by _{G 1 -G 4 (231, 232} , 233, 234) and a flat П _C (230) point PC (P), the image plane _{(image plane) (П I)} (220 through the camera on the ). Thus, finally, the PC (P) of the eye image and the four CRs (g ₁ -g ₄ ) 221, 222, 223, and 224 are generated.

3 is a diagram illustrating a mapping function of the gaze prediction method according to various embodiments of the present invention. For example, a geometric model of the PCCR scheme may be applied in accordance with various embodiments of the present invention. Referring to FIG. 3, the POG prediction in the HN scheme may be composed of two mapping functions (MF) as shown in FIG. The mapping from П _I (310) to П _N (320) and from П _N (320) to П _S (330). Where _N (320) is a normalized plane with a unit square size.

을 통하여 П_N(320) 평면으로 매핑될 수 있다. 이는 П_I(310)상의 4개의 CR들(g₁-g₄)과 П_N(320)의 네 모서리 지점(G₁-G₄)을 이용하여 가능하다.The P _I detected in the eye image was the homography function

Can be mapped to the П _N (320) plane. This is achieved by using the four corner points (G ₁ -G ₄₎ of the _I П four CR on the _{(310) (g 1 -g 4} ) and П _N (320).

을 이용하여 П_S(330)상의 점으로 매핑하여 최종 POG를 얻을 수 있다.

은 캘리브레이션 과정을 통해 얻을 수 있다.P _N , the PC on _N (320)

To obtain a final POG by mapping to points on the П _S (330).

Can be obtained through a calibration process.

로 나타낼 수 있으며, 상기 함수에 의해,П_I(310) 평면에서 П_S(330) 평면으로 매핑될 수 있다.Thus, the full mapping function

And can be mapped to the П _S (330) plane in the П _I (310) plane by the function.

을 계산할 수 있다. 상기 단계에서는 오류를 최소화하기 위한 방법으로 "ransac" 또는 최소 자승(least square) 알고리즘을 사용할 수 있다.For calibration, the user views four or nine (or more) points sequentially on the screen, stores the PC coordinates P _N on the _{N N} (1320) at each point in time and obtains corresponding coordinates Homography function with

Can be calculated. In this step, a "ransac" or a least square algorithm can be used as a method for minimizing the error.

The calibration method may be applied to at least one of various embodiments of the present invention, and various embodiments of the present invention may be applied to various calibration methods other than the calibration method, and the present invention is not limited thereto .

4 is a diagram illustrating a calibration method for eye tracking according to various embodiments of the present invention. 4, the user views the nine reference points 401, 402, 403, 404, 405, 406, 407, 408, 409 displayed on the screen in order, Eye tracking can be modeled through an eye image of a user photographed through a camera or the like.

가 결정될 수 있다. 상기

은 다항 함수(polynomial function)로 표현될 수 있다.5 is a diagram illustrating a homography normalization method in accordance with various embodiments of the present invention. Referring to FIG. 5, as a calibration method using the HN technique, a position of at least one reflection point CR in an eye image photographed as described above with reference to FIGS. 2 to 4 is confirmed, Can be mapped to positions on the image plane. For example, when a user gazes at a plurality of reference points set by the user, a virtual image having a size of a unit square from a plurality of photographed image planes (P _I ) ( _{N N)} ) 520 of the virtual normalized plane. The plane (П _N)) (520) normalization of the virtual back screen plane (П _S) is mapped to the 530, an of the virtual normalized plane (screen plane at П _N)) (520) (П _S) (530)

Can be determined. remind

Can be expressed as a polynomial function.

6 is a diagram illustrating a method of measuring a line of sight according to various embodiments of the present invention. Referring to FIG. 6, when the user gazes at a specific position on the screen, it is possible to estimate the position (the user's gaze) on the screen by the user by the mapping function modeled by the calibration method of FIG. 5 described above.

에 의해 다시 스크린 평면(П_S)(630)에 매핑될 수 있다. 상기 가상의 정규화된 평면(П_N)(620)에서의 동공의 중심(P)을 상기 스크린 평면(П_S)(630)으로 변환시킴으로써 사용자가 응시하는 화면상의 위치(POG)를 결정할 수 있다.For example, the CR in the image plane (П _I ) 610 generated from the photographed image may be checked for gaze tracking, and the image plane may be transformed into a virtual normalized plane (П _N ) (620) . The virtual normalized plane ( _{P N} ) (620) is determined by the polynomial function

To the screen plane ( _{P S} ) 630 again. It is possible to determine the position POG on the screen to be looked at by the user by converting the center P of the pupil in the virtual normalized plane P _{N 6} into the screen plane P _S 630.

Figure 7 is an illustration of examples of detectable reflections according to various embodiments of the present invention. Referring to FIG. 7, it can be seen that the number of CRs detected varies from the photographed image according to the direction of the line of sight of the user.

For example, in FIG. 7A, it can be seen that three CRs 701, 702 and 703 are detected, and one CR 704 is not detected. In FIG. 7 (b), it can be seen that two CRs 711 and 712 are detected and the remaining two CRs 713 are not detected. It can be seen that two CRs 721 and 722 are detected in FIG. 7 (c), and three CRs 731, 732 and 733 are detected in FIG. 7 (d).

7 (a) and 7 (b), CRs 704 and 713, which are not detected, can be generated. When a CR that is not detected as described above occurs, a mapping function according to one of the methods may not be applied. For example, when applying the HTN scheme that can be applied when four CRs are detected, if the number of detected CRs is 1, 2, or 3 as shown in FIG. 7, It may happen that you can not trace.

According to various embodiments of the present invention, even when the detected CRs occur or the number of detected CRs does not satisfy the condition of a predetermined mapping function (for example, a mapping function according to the HTN scheme) It is possible to track the gaze by selecting and applying any one of the mapping functions.

8 is a diagram showing a result of eye gaze measurement according to various embodiments of the present invention. Referring to FIG. 8, a mapping function set according to the number of CRs detected in the photographed image can not be applied. For example, if the number of CRs detected from the photographed image is four, eye gaze prediction is possible, but if the number of detected CRs is two, gaze prediction may not be possible. If the number of CRs detected from the photographed image is three, eye gaze prediction may be performed from four CRs by predicting the remaining CR that has not been detected.

9 is a flow chart illustrating a calibration procedure of an electronic device according to various embodiments of the present invention. Referring to FIG. 9, in operation 902, when the calibration mode is set, a plurality of light sources may be turned on in operation 904. It is possible to display a plurality of reference points on the screen for calibration and allow the user to gaze at each reference point, and then photograph the user's eye through the camera. In operation 906, a reflection point formed in the cornea can be detected from a plurality of photographed images gazing at the respective reference points.

In operation 908, a plurality of light source patterns (patterns of reflection points) may be determined by combining the detected reflection points corresponding to at least one light source. At operation 910, a mapping function may be determined for each light source according to the number of reflection point patterns or the number of reflection points.

10 is a flow chart illustrating a line-of-sight tracing procedure of an electronic device according to various embodiments of the present invention. In operation 1002, when the gaze tracking mode is executed, a plurality of light sources may be turned on in operation 1004. [ The user's eyes can be photographed through the camera in accordance with the execution of the gaze tracking mode and at operation 1006 at least one reflection point generated on each film from the photographed image can be detected.

At operation 1008, the pattern of the detected reflection points or the number of reflection points is checked, and at operation 1010, the user's gaze is determined by at least one mapping function preset in correspondence with the detected pattern of reflection points or the number of reflection points .

The operation of at least one of the operations shown in FIG. 9 or 10 may be omitted, and at least one other operation may be added between the operations. 9 or 10 may be processed in the order shown, and the order of execution of at least one operation may be modified and processed in accordance with the order of execution of other operations. In addition, the operations shown in FIG. 9 or 10 may be performed in the electronic device, or may be performed in the server. Further, at least one of the operations shown in FIG. 9 or 10 may be performed in the electronic device, and the remaining operations may be performed in the server.

The method of operating an electronic device according to any one of the various embodiments of the present invention is a method for determining a line of sight in an electronic device, the method comprising: detecting at least one reflection point on a cornea generated by a plurality of light sources; Determining a pattern of the detected reflection points; And determining a line of sight by a mapping function corresponding to the number of reflection points among a plurality of mapping functions for mapping from an image plane to a screen plane.

According to various embodiments of the present invention, an operation of determining a line of sight by a mapping function corresponding to the pattern of the detected reflection points may be further included.

Determining a plurality of light source patterns by combining at least one light source selected from the plurality of light sources, according to various embodiments of the present invention; And determining the mapping function for each light source pattern among the plurality of light source patterns.

According to various embodiments of the present invention, the plurality of light source patterns may be mapped to coded data and stored.

According to various embodiments of the present invention, the plurality of light source patterns may be encoded into a binary code.

According to various embodiments of the invention, the light source may comprise an infrared (LED) light emitting diode.

According to various embodiments of the present invention, an operation may be performed to photograph an eye image by an infrared camera.

According to various embodiments of the present invention, the act of determining the line of sight further comprises: converting from the image plane to a virtual normalized plane; And mapping the virtual normalized plane to the screen plane.

According to various embodiments of the present invention, the mapping function for mapping from the hypothetical normalized plane to the screen plane may be represented by a polynomial function.

In accordance with various embodiments of the present invention, it may further comprise determining a polynomial mapping function for each of the patterns for a plurality of detected reflections during a calibration operation for gaze determination.

11 is a view showing a concept of a gaze measurement method according to various embodiments of the present invention. Referring to FIG. 11, a mapping function according to a predetermined line-of-sight tracking method can be applied according to the number of CRs detected on the image plane 1110 or the CR pattern. According to various embodiments of the present invention, the image plane 1110 may be transformed into a virtual normalized plane 1120 by applying a predetermined mapping function according to the number of each detected CR or the pattern of the CR, as described above. And each virtual normalized plane 1120 can be transformed into a screen plane 1130. [

를 적용할 수 있으며, 이미지 평면(1110)상에서 검출된 CR의 개수가 3개일 경우 ATN 방식에 따른 매핑 함수

를 적용할 수 있으며, 이미지 평면(1110)상에서 검출된 CR의 개수가 2개일 경우 STN 방식에 따른 매핑 함수

를 적용할 수 있다.For example, when the number of CRs detected on the image plane 1110 is four, a mapping function according to the HTN scheme

If the number of CRs detected on the image plane 1110 is three, the mapping function according to the ATN scheme

If the number of CRs detected on the image plane 1110 is two, the mapping function according to the STN scheme

Can be applied.

상의 PC

의 좌표는

를 이용하여

상으로 매핑되어

를 얻을 수 있다. 본 발명의 다양한 실시 예에 따라, 9개 지점을 바라보며 두 평면상의 대응점들 간의 다항 계수들(polynomial coefficients)을 저장하는 다항 캘리브레이션(polynomial calibration) 방식을 사용할 수 있다.According to various embodiments of the present invention, when four CRs are detected in Figure 11,

Top PC

The coordinates of

Using

Gt;

Can be obtained. According to various embodiments of the present invention, a polynomial calibration scheme may be used to store polynomial coefficients between corresponding points on two planes looking at nine points.

상의 PC

좌표는

를 이용하여

상으로 매핑되어

을 얻을 수 있다. 상기

는 3개의 CR(g₁ 내지 g₃) 와 각각에 대응되는

상의 점 G₁ 내지 G₃ 간의 AT(affine transform)를 가능하게 하는 매핑 함수이다.

은 G₁ 내지 G₃으로 이루어지는 직각 이등변 삼각형(두 변의 길이는 단위 길이(unit length))이다.Also, according to various embodiments of the present invention, when three CRs are detected in FIG. 11, an affine transform (AT) normalization may be performed. for example,

Top PC

The coordinates are

Using

Gt;

Can be obtained. remind

Three CR (g ₁ to g ₃₎ and corresponding to an individual

Is an mapping function that enables an affine transform (AT) between the points G ₁ to G ₃ on the image plane.

Is a right-angled isosceles triangle consisting of G ₁ to G ₃ (the length of the two sides is unit length).

에서

로의 매핑을 위해 9개 지점을 바라보며 두 평면상의 대응점들 간의 다항 계수들을 저장하는 다항 캘리브레이션 방식을 사용할 수 있다.As in the case where four CRs are detected,

in

A polynomial calibration scheme that stores the polynomial coefficients between the corresponding points on the two planes can be used while looking at nine points for mapping to the two points.

상의 PC

좌표는

를 이용하여

상으로 매핑되어

을 얻을 수 있다. 상기

는 2개의 CR(g₁-g₂)과 각각에 대응되는

상의 점 G₁-G₂ 간의 ST(similarity transform)를 가능하게 하는 매핑 함수이다.

은 G₁ 및 G₂로 이루어지는 단위 길이를 갖는 직선일 수 있다. 마찬가지로,

에서

로의 매핑을 위해 9개 지점을 바라보며 두 평면상의 대응점들 간의 다항 계수들을 저장하는 다항 캘리브레이션 방식을 사용할 수 있다.In addition, according to the embodiment of the present invention, when two CRs are detected in FIG. 11, a similarity transform (ST) -based normalization can be performed. for example,

Top PC

The coordinates are

Using

Gt;

Can be obtained. remind

(G ₁ -g ₂ ) and two CR

Is a mapping function that enables a similarity transform (ST) between the points G _{1 and} G ₂ .

May be a straight line having a unit length consisting of G ₁ and G ₂ . Likewise,

in

A polynomial calibration scheme that stores the polynomial coefficients between the corresponding points on the two planes can be used while looking at nine points for mapping to the two points.

12 is a diagram illustrating a method for modeling a detectable reflection point in accordance with various embodiments of the present invention. Referring to FIG. 12, it is possible to analyze the number (for example, four, three, two, etc.), kind, position, and pattern of the CR that can be detected according to the movement of the user's eyeball. When a user looks extremely out of the screen, a single CR may be detected.

For example, according to the area of the screen to be examined by the user, four CRs can be detected when the user gazes at the front (1207). For example, four CRs may be detected in the case of gazing at the upper left corner (1204), gazing at the upper right corner (1205), gazing at the lower left corner (1209), gazing at the lower right corner (1210) As shown, the position and pattern of the CR detected according to each direction of gazing may be different.

Also, according to various embodiments of the present invention, one or two CRs may be detected when a user strikes off-screen (1201, 1202, 1203, 1206, 1208, 1211, 1212, 1213) The number, position, pattern, etc. of CRs detected in accordance with the direction may be different.

13 is a diagram illustrating a method of encoding a pattern of detectable points of refraction in accordance with various embodiments of the present invention. Figure 13 is a diagram illustrating the concept of analyzing and encoding a pattern of CRs that can be detected from four light sources (e.g., IR light sources) in accordance with various embodiments of the present invention.

Referring to FIG. 13, the number and patterns of detected CRs may vary depending on the direction in which the user looks. For example, the number of detected CRs can be detected as 1, 2, 3 or 4. In addition, the pattern of the detected CR can be detected in 15 patterns according to the position of the detected CR.

The pattern of the detectable CR can be encoded with a set code. For example, in FIG. 13, up to four CRs can be detected using four light sources, so that it can be encoded into a 4-bit binary code. According to various embodiments of the present invention, each digit of the encoded code may be mapped to a light source at a specific location. For example, as shown in the figure, the position of the binary code can be given in the counterclockwise direction from the light source at the upper right corner.

According to various embodiments of the present invention, when all four light sources are detected as illustrated in FIG. 13, they can be encoded into a binary code '1111', which can be '15' when converted to a decimal number. In addition, as illustrated in FIG. 13, if only the light source at the lower left is not detected and the remaining three light sources are detected, it can be encoded into a binary code '1011', and converted into a decimal number to be '11' .

14 is a diagram showing a concept of a gaze measurement method according to various embodiments of the present invention. Referring to FIG. 14, PC and CR can be detected from the captured eye image, and the detected CR pattern can be encoded as described above.

For example, it can be assumed that from the photographed eye image, the calibration of looking at nine points in the state of detecting four CRs is completed. According to various embodiments of the present invention, calibration may also be performed for various patterns in which three or two of the four CRs are detected in performing the calibration. For example, as a normalization method using three or two CRs, a normalization method (for example, an ATM technique or an STM technique) applicable to the number of CRs can be used.

When the calibration for the various numbers of CRs is completed, mapping functions P ₁₅ , P ₁₃ , P ₁₄ , P ₇ , P ₁₁ , P ₉ , P ₆ , P ₃ , P ₁₂ ) can be determined. For example, the mapping function P ₁₅ can be determined for four detected CRs, and the mapping functions P ₁₃ , P ₁₄ , P ₇ , and P ₁₁ can be determined for three detected CRs, The mapping functions P ₉ , P ₆ , P ₃ , and P ₁₂ can be determined.

If the mapping function corresponding to the number and pattern of the detected CRs is determined in the calibration process as described above, then when the gaze prediction or gaze tracking is performed, the corresponding mapping function is used according to the number and pattern of detected CRs The gaze can be judged. For example, as shown in FIG. 14, when the number of CRs detected in the captured eye image is 3 and the pattern is 13, normalization methods (for example, an ATM technique) applicable to the three CRs, The POG can be calculated by reading the polynomial mapping function P ₁₃ corresponding to the pattern 13, which is the confirmed pattern, from the storage unit and applying the polynomial mapping function P ₁₃ to the detected CR.

15 to 18 are diagrams showing experimental examples according to various embodiments of the present invention.

FIG. 15 is a diagram illustrating a result of eye-tracking experiments according to various embodiments of the present invention. Referring to FIG. 15, an IR light source was installed on the four corners of the monitor, and an eyeball image of 640x480 resolution was photographed using an IR camera. Experiments were performed on three subjects.

The distance between the screen and the user was 65cm. After performing the calibration by looking at the nine points on the screen, the distance between the screen and the screen was changed to 65cm, 55cm, and 75cm. Moving the head is to create an environment where CR detection is difficult. Horizontal and vertical rotation angle of 20 °, and left and right up and down parallel movement of 10 cm was defined as the permissible range of movement of the head.

FIG. 15 shows the performance comparison result between the HN technique operating with four CRs according to the prior art and the technique according to various embodiments of the present invention. The experimental results show the prediction error angle (degree) and the standard deviation at the test of 9 test points at each distance.

As shown in FIG. 15, when four CRs are detected, a visual prediction performance of about 1.19 degrees is obtained using the conventional HN technique, and the predictive performance of two or three CRs The gaze prediction performance of about 1.35 degrees was obtained. If the number of CRs detected in the entire experiment is examined, it can be seen that the ratio of detection of all four is 80.6%, and the ratio of detecting two or three is 19.4%. Thus, when using the technique according to various embodiments of the present invention, It can be understood that the gaze prediction can be performed by successfully coping with the situation where the result can not be obtained.

Hereinafter, the results of the eye gaze prediction experiment performed using the monitor of the 24-inch screen (1920x1080) will be described with reference to FIGS. 16 to 18. FIG. Referring to FIG. 16, IR light sources 1611, 1612, 1613, and 1614 may be provided at four corners of the screen 1600. Eye image of a resolution of 640 x 480 was shot using an IR camera 1620 provided on the monitor, and experiments were performed on six subjects as shown in Table 1 below.

17, the IR light sources 1711 are provided at four corners of the monitor 1700, and the distance between the screen and the user is maintained at 70 cm, and calibration is performed while looking at nine points while fixing the head . For the experiment, nine points on the screen 1721 were stared.

Table 1 shows the performance comparison results between the general HN technique (Ori_HN) and the proposed HTN, ATN, and STN techniques. Table 1 below shows the prediction error angles (degrees) and standard deviations during the test for the nine test points.

Ori_HN HTN ATN STN Subject 1 0.93 + 0.71 0.68 ± 0.36 0.64 ± 0.40 0.73 + - 0.33 Subject 2 0.81 ± 0.50 0.72 ± 0.27 0.74 + 0.24 0.77 ± 0.26 Subject 3 0.93 + - 0.46 0.73 + 0.44 0.63 + - 0.35 0.70 + - 0.37 Subject 4 0.71 + - 0.37 0.84 0.38 0.68 ± 0.45 0.97 + - 0.52 Subject 5 0.91 + - 0.52 0.75 + 0.39 0.88 0.44 0.89 ± 0.45 Subject 6 1.11 + - 0.51 0.70 + - 0.25 0.81 + 0.24 0.70 + - 0.27 medium 0.90 + - 0.51 0.74 + - 0.35 0.73 ± 0.35 0.79 + 0.37

<Table 1> shows that the result of HTN is improved than that of the conventional HN. Even when using only 3 or 2 CRs, the performance of the proposed method is superior to that of the HN method.

18 is a diagram showing target points used for calibration and testing on the screen of an electronic device according to various embodiments of the present invention. Referring to FIG. 18, a square display on the screen indicates target points set for calibration, and a cross mark indicates test points for eye tracking.

19 is a block diagram 1900 of an electronic device 1901 according to various embodiments. The electronic device 1901 may include all or part of the electronic device shown in Fig. 1, for example. The electronic device 1901 includes one or more application processors (APs) 1910, a communication module 1920, a SIM (subscriber identification module) card 1924, a memory 1930, a sensor module 1940, The device 1950 includes a display 1960, an interface 1970, an audio module 1980, a camera module 1991, a power management module 1995, a battery 1996, an indicator 1997, .

The AP 1910 may control a plurality of hardware or software components connected to the AP 1910 by, for example, operating an operating system or an application program, and may perform various data processing and operations. The AP 1910 may be implemented as a system on chip (SoC), for example. According to one embodiment, the AP 1910 may further include a graphics processing unit (GPU) and / or an image signal processor. The AP 1910 may include at least some of the components shown in FIG. The AP 1910 loads or processes commands or data received from at least one of the other components (e.g., non-volatile memory) into the volatile memory and stores the various data in the non-volatile memory .

The communication module 1920 includes a cellular module 1921, a WIFI module 1923, a BT module 1925, a GPS module 1927, an NFC module 1928 and a radio frequency (RF) module 1929 ).

The cellular module 1921 can provide voice communication, video call, text service, or Internet service, for example, through a communication network. According to one embodiment, the cellular module 1921 may utilize a subscriber identity module (e.g., SIM card 1924) to perform the identification and authentication of the electronic device 1901 within the communication network. According to one embodiment, the cellular module 1921 may perform at least some of the functions that the AP 1910 can provide. According to one embodiment, the cellular module 1921 may include a communication processor (CP).

Each of the WIFI module 1923, the BT module 1925, the GPS module 1927 or the NFC module 1928 includes a processor for processing data transmitted and received through a corresponding module . According to some embodiments, at least some (e.g., two or more) of the cellular module 1921, the WIFI module 1923, the BT module 1925, the GPS module 1927, or the NFC module 1928, (IC) or an IC package.

The RF module 1929 can transmit and receive a communication signal (e.g., an RF signal). The RF module 1929 may include, for example, a transceiver, a power amplifier module (PAM), a frequency filter, a low noise amplifier (LNA), or an antenna. According to another embodiment, at least one of the cellular module 1921, the WIFI module 1923, the BT module 1925, the GPS module 1927 or the NFC module 1928 transmits and receives RF signals through separate RF modules .

The SIM card 1924 may include, for example, a card containing a subscriber identity module and / or an embedded SIM and may include unique identification information (e.g., ICCID (integrated circuit card identifier) Subscriber information (e.g., international mobile subscriber identity (IMSI)).

The memory 1930 (e.g., memory 1930) may include, for example, an internal memory 1932 or an external memory 1934. The built-in memory 1932 may be a memory such as a volatile memory (e.g., a dynamic RAM, an SRAM, or a synchronous dynamic RAM (SDRAM)), a non-volatile memory : Programmable ROM (PROM), erasable and programmable ROM (EPROM), electrically erasable and programmable ROM (EEPROM), mask ROM, flash ROM, flash memory (e.g., NAND flash or NOR flash) , A hard drive, or a solid state drive (SSD).

The external memory 1934 may be a flash drive such as a compact flash (CF), a secure digital (SD), a micro secure digital (SD), a mini secure digital (SD) extreme digital), a memory stick, or the like. The external memory 1934 may be functionally and / or physically connected to the electronic device 1901 through various interfaces.

The sensor module 1940 may, for example, measure a physical quantity or sense an operating state of the electronic device 1901 to convert the measured or sensed information into an electrical signal. The sensor module 1940 includes a gesture sensor 1940A, a gyro sensor 1940B, an air pressure sensor 1940C, a magnetic sensor 1940D, an acceleration sensor 1940E, a grip sensor 1940F, (For example, RGB (red, green, blue) sensor 1940H), a biometric sensor 1940I, a temperature / humidity sensor 1940J, a light intensity sensor 1940K, violet sensor 1940M. Additionally or alternatively, the sensor module 1940 can be, for example, an E-nose sensor, an EMG sensor, an EEG sensor, an electrocardiogram sensor, an IR infrared sensor, an iris sensor, and / or a fingerprint sensor. The sensor module 1940 may further include a control circuit for controlling at least one sensor included in the sensor module 1940. In some embodiments, the electronic device 1901 further includes a processor configured to control the sensor module 1940, either as part of the AP 1910 or separately, so that the AP 1910 is in a sleep state , The sensor module 1940 can be controlled.

The input device 1950 may include, for example, a touch panel 1952, a (digital) pen sensor 1954, a key 1956, or an ultrasonic input device (1958). The touch panel 1952 may employ, for example, at least one of an electrostatic type, a pressure sensitive type, an infrared type, and an ultrasonic type. In addition, the touch panel 1952 may further include a control circuit. The touch panel 1952 may further include a tactile layer to provide a tactile response to a user.

The (digital) pen sensor 1954 may be, for example, part of a touch panel or may include a separate recognition sheet. The key 1956 may include, for example, a physical button, an optical key, or a keypad. The ultrasonic input device 1958 can sense data by sensing a sound wave from the electronic device 1901 to a microphone (e.g., a microphone 1988) through an input tool for generating an ultrasonic signal.

The display 1960 may include a panel 1962, a hologram device 1964, or a projector 1966. The panel 1962 may have the same or similar configuration as the display 140 of FIG. The panel 1962 can be embodied, for example, flexible, transparent, or wearable. The panel 1962 may be formed of one module with the touch panel 1952. The hologram device 1964 can display stereoscopic images in the air using interference of light. The projector 1966 can display an image by projecting light onto a screen. The screen may be located, for example, inside or outside the electronic device 1901. According to one embodiment, the display 1960 may further include control circuitry for controlling the panel 1962, the hologram device 1964, or the projector 1966.

The interface 1970 may be implemented in a computer system such as a high-definition multimedia interface (HDMI) 1972, a universal serial bus (USB) 1974, an optical interface 1976, or a D- subminiature (1978). The interface 1970 may include, for example, a mobile high-definition link (MHL) interface, a secure digital (SD) card / multi-media card (MMC) interface, or an infrared data association have.

The audio module 1980 can convert, for example, a sound and an electrical signal in both directions. The audio module 1980 may process sound information input or output through, for example, a speaker 1982, a receiver 1984, an earphone 1986, a microphone 1988, or the like.

According to one embodiment, the camera module 1991 may include at least one image sensor (e.g., a front sensor or a rear sensor), a lens, an image signal processor (ISP) ), Or a flash (e.g., LED or xenon lamp). The camera module 1991 may have the same or similar configuration as the camera unit 130 of FIG.

The power management module 1995 may manage the power of the electronic device 1901, for example. According to one embodiment, the power management module 1995 may include a power management integrated circuit (PMIC), a charger integrated circuit (PMIC), or a battery or fuel gauge. The PMIC may have a wired and / or wireless charging scheme. The wireless charging scheme may include, for example, a magnetic resonance scheme, a magnetic induction scheme, or an electromagnetic wave scheme, and may further include an additional circuit for wireless charging, for example, a coil loop, a resonant circuit, have. The battery gauge can measure, for example, the remaining amount of the battery 1996, the voltage during charging, the current, or the temperature. The battery 1996 may include, for example, a rechargeable battery and / or a solar battery.

The indicator 1997 may indicate a specific state of the electronic device 1901 or a portion thereof (e.g., AP 1910), e.g., a boot state, a message state, or a charged state. The motor 1998 can convert an electrical signal into a mechanical vibration and generate vibration or a haptic effect. Although not shown, the electronic device 1901 may include a processing unit (e.g., a GPU) for mobile TV support. The processing device for supporting the mobile TV can process media data conforming to standards such as digital multimedia broadcasting (DMB), digital video broadcasting (DVB), or media flow.

Each of the above-described components of the electronic device may be composed of one or more components, and the name of the component may be changed according to the type of the electronic device. In various embodiments, the electronic device may be configured to include at least one of the components described above, with some components omitted or further comprising additional other components. In addition, some of the components of the electronic device according to various embodiments may be combined into one entity, so that the functions of the components before being combined can be performed in the same manner.

20 is a block diagram 2000 of a program module 2010 according to various embodiments. According to one embodiment, the program module 2010 may include an operating system (OS) for controlling resources associated with the electronic device and / or various applications running on the operating system. The operating system may be, for example, android, iOS, windows, symbian, tizen, or bada.

The program module 2010 may include a kernel 2020, a middleware 2030, an application programming interface (API) 2060, and / or an application 2070. At least a portion of the program module 2010 may be preloaded on an electronic device or downloaded from a server.

The kernel 2020 may include, for example, a system resource manager 2021 or a device driver 2023. [ The system resource manager 2021 can perform control, assignment, or recovery of system resources. According to one embodiment, the system resource manager 2021 may include a process management unit, a memory management unit, or a file system management unit. The device driver 2023 may include, for example, a display driver, a camera driver, a Bluetooth driver, a shared memory driver, a USB driver, a keypad driver, a WIFI driver, an audio driver, or an inter- have.

The middleware 2030 may provide the functions commonly required by the application 2070 or may be used by the API 2060 to allow the application 2070 to efficiently use limited system resources within the electronic device. To provide various functions to the application (2070). According to one embodiment, the middleware 2030 includes a runtime library 2035, an application manager 2041, a window manager 2042, a multimedia manager 2043, a resource manager 2044, a power manager 2045, a database manager 2046, a package manager 2047, a connectivity manager 2048, a notification manager 2049, a location manager 2050, a graphic manager 2051, or a security manager 2052.

The runtime library 2035 may include, for example, a library module used by the compiler to add new functionality via a programming language while the application 2070 is running. The runtime library 2035 may perform input / output management, memory management, or functions for an arithmetic function.

The application manager 2041 can manage the life cycle of at least one of the applications 2070, for example. The window manager 2042 can manage GUI resources used in the screen. The multimedia manager 2043 can recognize a format required for reproducing various media files and can encode or decode a media file using a codec suitable for the format. The resource manager 2044 can manage resources such as a source code, a memory, or a storage space of at least one of the applications 2070.

The power manager 2045 operates together with a basic input / output system (BIOS), for example, to manage a battery or a power source and provide power information and the like necessary for the operation of the electronic device . The database manager 2046 may create, search, or modify a database for use by at least one of the applications 2070. The package manager 2047 can manage installation or update of an application distributed in the form of a package file.

The connection manager 2048 may manage a wireless connection, such as WIFI or Bluetooth, for example. The notification manager 2049 may display or notify events such as arrival messages, appointments, proximity notifications, etc. in a manner that does not disturb the user. The location manager 2050 can manage the location information of the electronic device. The graphic manager 2051 may manage a graphical effect to be provided to the user or a user interface related thereto. The security manager 2052 may provide security functions necessary for system security or user authentication. According to one embodiment, when the electronic device includes a telephone function, the middleware 2030 may further include a telephony manager for managing a voice or video call function of the electronic device.

The middleware 2030 may include a middleware module that forms a combination of various functions of the above-described components. The middleware 2030 may provide a module specialized for each type of operating system to provide a differentiated function. In addition, the middleware 2030 may dynamically delete some existing components or add new components.

The API 2060 may be provided in a different configuration depending on the operating system, for example, as a set of API programming functions. For example, for Android or iOS, you can provide one API set per platform, and for tizen, you can provide more than two API sets per platform.

The application 2070 may include a home 2071, a dialer 2072, an SMS / MMS 2073, an instant message 2074, a browser 2075, a camera 2076, an alarm 2077 ), Contact 2078, voice dial 2079, email 2080, calendar 2081, media player 2082, album 2083 or clock 2084, health care (e.g., Or providing information on the environment (e.g., providing atmospheric pressure, humidity, or temperature information, etc.), and the like.

According to one embodiment, the application 2070 may include an application (hereinafter referred to as an "information exchange application") for supporting information exchange between the electronic device and an external electronic device. The information exchange application may include, for example, a notification relay application for communicating specific information to the external electronic device, or a device management application for managing the external electronic device .

For example, the notification delivery application may include a function of transmitting notification information generated by another application (e.g., SMS / MMS application, email application, healthcare application, or environment information application) of the electronic device to an external electronic device . In addition, the notification delivery application may receive notification information from, for example, an external electronic device and provide the notification information to the user. The device management application may, for example, control the operation of at least one function of the external electronic device communicating with the electronic device (e.g., turn-on / turn-off of the external electronic device itself (E.g., install, delete, or update) an application running on the external electronic device or a service (e.g., call service or message service) provided on the external electronic device.

According to one embodiment, the application 2070 includes an application (e.g., a healthcare application) designated according to an attribute of the external electronic device (e.g., an attribute of the electronic device, the type of electronic device is a mobile medical device) . According to one embodiment, the application 2070 may include an application received from an external electronic device. According to one embodiment, the application 2070 may include a preloaded application or a third party application downloadable from a server. The names of the components of the program module 2010 according to the illustrated embodiment may vary depending on the type of the operating system.

According to various embodiments, at least some of the program modules 2010 may be implemented in software, firmware, hardware, or a combination of at least two of them. At least some of the program modules 2010 may be implemented (e.g., executed) by, for example, a processor (e.g., AP 1910). At least some of the program modules 2010 may include, for example, modules, programs, routines, sets of instructions or processes, etc. to perform one or more functions.

The term "module" or "function, " as used herein, may refer to a unit that includes one or a combination of two or more of, for example, hardware, software or firmware. The term "module" or "functional part" is used interchangeably with terms such as, for example, unit, logic, logical block, component, ). A "module" or "function" may be a minimum unit or a part of an integrally constructed component. A "module" may be a minimum unit or a portion thereof that performs one or more functions. "Modules" or "functional parts" may be implemented mechanically or electronically. For example, a "module" or "function" may be an application-specific integrated circuit (ASIC) chip, field-programmable gate arrays (FPGAs) or programmable logic arrays (FPGAs) -logic device).

At least a portion of a device (e.g., modules or functions thereof) or a method (e.g., operations) according to various embodiments may include, for example, computer-readable storage media in the form of program modules, As shown in FIG. The instructions, when executed by the processor, may cause the one or more processors to perform functions corresponding to the instructions. The computer-readable storage medium may be, for example, the storage unit 160. [

The computer readable recording medium may be a hard disk, a floppy disk, a magnetic media (e.g., a magnetic tape), an optical media (e.g., a compact disc read only memory (CD-ROM) but are not limited to, digital versatile discs, magneto-optical media such as floptical discs, hardware devices such as read only memory (ROM), random access memory (RAM) Etc.), etc. The program instructions may also include machine language code such as those produced by a compiler, as well as high-level language code that may be executed by a computer using an interpreter, etc. The above- May be configured to operate as one or more software modules to perform the operations of the various embodiments, and vice versa.

Modules or program modules according to various embodiments may include at least one or more of the elements described above, some of which may be omitted, or may further include additional other elements. Operations performed by modules, program modules, or other components in accordance with various embodiments may be performed in a sequential, parallel, iterative, or heuristic manner. Also, some operations may be performed in a different order, omitted, or other operations may be added.

According to various embodiments, there is provided a storage medium storing instructions which, when executed by at least one processor, cause the at least one processor to be configured to perform at least one operation, A method of determining a line of sight in an electronic device, the method comprising: detecting at least one reflection point on a cornea generated by a plurality of light sources; Determining a pattern of the detected reflection points; And determining a line of sight by a mapping function corresponding to the number of reflection points among a plurality of mapping functions for mapping from an image plane to a screen plane.

It is to be understood that both the foregoing general description and the following detailed description of the present invention are exemplary and explanatory and are intended to provide further explanation of the invention as claimed. And the like. Accordingly, the scope of various embodiments of the present invention should be construed as being included in the scope of various embodiments of the present invention without departing from the scope of the present invention, all changes or modifications derived from the technical idea of various embodiments of the present invention .

110: control unit 111: calibration processing unit
112: mapping function determination unit 113: reflection point detection unit
114: eye gaze determination unit 120:
130: camera unit 140: display unit
150: input unit 160:
161: Calibration information 162: Mapping function information
163: sight line information

Claims (20)

A camera unit for photographing an image; And
A mapping function for detecting at least one reflection point on the cornea generated by the plurality of light sources from the photographed image and for mapping the detected number of reflection points among a plurality of mapping functions for mapping from an image plane to a screen plane, And a processor for determining a line of sight by the electronic device.
2. The apparatus of claim 1,
And determines a line of sight by a mapping function corresponding to the detected pattern of reflection points.
3. The apparatus of claim 2,
Determining a plurality of light source patterns by combining at least one light source selected from the plurality of light sources and determining the mapping function for each light source pattern among the plurality of light source patterns.
3. The method of claim 2,
Wherein the plurality of light source patterns are mapped to encoded data and stored.
5. The method of claim 4,
Wherein the plurality of light source patterns are encoded in a binary code.
The method according to claim 1,
Wherein the light source comprises an infrared (LED) light emitting diode.
The camera system according to claim 1,
An electronic device, comprising an infrared camera.
2. The apparatus of claim 1,
Transforming the imaginary normalized plane from the image plane, and mapping the virtual normalized plane to the screen plane.
9. The method of claim 8,
Wherein the mapping function for mapping from the virtual normalized plane to the screen plane is represented by a polynomial function.
2. The apparatus of claim 1,
And determines a polynomial mapping function for each of the plurality of detected points of the detected calibration points during the calibration operation for gaze determination.
A method of determining a line of sight in an electronic device,
Detecting at least one reflex point on the cornea generated by the plurality of light sources;
Determining a pattern of the detected reflection points; And
Determining a line of sight by a mapping function corresponding to the number of reflection points from among a plurality of mapping functions for mapping from an image plane to a screen plane.
12. The method of claim 11,
And determining a line of sight by a mapping function corresponding to the detected pattern of reflection points.
13. The method of claim 12,
Determining a plurality of light source patterns by combining at least one light source selected from the plurality of light sources; And
And determining the mapping function for each light source pattern of the plurality of light source patterns.
13. The method of claim 12,
Wherein the plurality of light source patterns are mapped to coded data and stored.
15. The method of claim 14,
Wherein the plurality of light source patterns are encoded in a binary code.
12. The method of claim 11,
Wherein the light source comprises an infrared (LED) light emitting diode.
12. The method of claim 11,
Further comprising: capturing an eye image by an infrared camera.
The method as claimed in claim 11,
Converting the image plane to a virtual normalized plane; And
And mapping the virtual normalized plane to the screen plane.
19. The method of claim 18,
Wherein the mapping function for mapping from the hypothetical normalized plane to the screen plane is represented by a polynomial function.
12. The method of claim 11,
Determining a polynomial mapping function for each of the plurality of detected points of the plurality of detected points during a calibration operation for visual determination.

Images