KR102195068B1 - User terminal with Line-of-sight Matching function and method using the same - Google Patents

Abstract

The present invention relates to a user terminal with a line-of-sight matching function and a line-of-sight matching method using the same. The present invention includes: a database storing information on the size of the display screen of the user terminal and the position of the camera lens of the user terminal; a receiving unit receiving the input of an image taken with the camera; a face information detection unit acquiring user face and pupil position information by applying the input image to machine learning; a control unit acquiring an actual point-of-view value by estimating the user′s line of sight through the face and pupil position information and deriving a corrected final point-of-view value from the actual point-of-view value; an eye image generation unit generating a pupil image with a changed point of view by applying the final point-of-view value to the pupil image included in the input image; and an output unit matching the pupil image with the changed point of view to the image and outputting a corrected image on the display screen. With the user terminal of the present invention, it is possible to match an inconsistent line of sight attributable to the positional difference between the display screen and the camera by applying the point-of-view correction value to the actual line-of-sight value of the user. In addition, it is possible to correct the line of sight while maintaining the original image by generating a new eye image by applying deep learning to the user eye image.

Description

User terminal with Line-of-sight Matching function and method using the same}

The present invention relates to a user terminal equipped with a gaze matching function and a gaze matching method using the same, and more particularly, to a display screen and a camera lens mounted on a user terminal when self-photographing or video call is performed using a user terminal. The present invention relates to a user terminal equipped with a line-of-sight matching function that is generated due to different positions of each other and a line-of-sight matching method using the same.

In general, a user using a smartphone makes a self-portrait or video call while checking his or her image output through the display screen of the smartphone.

Currently, cameras mounted on smartphones are located outside the display screen. Therefore, in the case of self-photographing, the user takes a picture by looking at his or her image on the display screen, and when checking the photographed result, the user is photographed as if looking at a place other than the camera.

That is, when looking at an image captured while the user's gaze is fixed to the camera, the user's gaze is output as if it were located in front. Conversely, when looking at an image captured while the user's gaze is fixed on the display screen, the user's gaze is output as if it was pointing to another place. This is because the user's gaze is not toward the camera lens, but toward the display screen.

However, if the user's gaze is taken to face the front, it is difficult for the user to check the image or video displayed on the display screen due to the attachment position of the camera.

The technology behind the present invention is disclosed in Korean Patent Publication No. 10-0307854 (2001.11.02. Announcement).

The technical problem to be achieved by the present invention is to correct gaze matching that corrects mismatched gazes caused by different positions of a display screen and a camera lens mounted on a user terminal when a self-photographing or video call is made using a user terminal. An object of the present invention is to provide a user terminal equipped with a function and a gaze matching method using the same.

According to an embodiment of the present invention for achieving such a technical problem, in a user terminal equipped with a gaze matching function, a database storing the display screen size of the user terminal and location information of a camera lens mounted on the user terminal, the camera A receiving unit that receives an image captured through, a face information detector that obtains the user's face position information and the pupil position information by applying the input image to machine learning, the obtained user's face position information and the pupil position information A control unit for estimating the user's gaze by estimating the user's gaze and obtaining a corrected final viewpoint value from the obtained actual viewpoint value, and applying the final viewpoint value to the image of the pupil included in the input image And an eye image generation unit that generates the changed pupil image, and an output unit that matches the pupil image whose viewpoint is changed with the image and outputs a corrected image on the display screen.

The face information detection unit inputs the image captured through the camera into pre-built machine learning to learn the user's face position and the position of the pupil, and inputs the image acquired while the learning is completed into the machine learning. The position of the face and the position of the pupil may be detected, and information on the distance between the display screen and the user may be obtained by using the detected distance between the pupil and the pupil.

The control unit extracts an eye image from an image using the acquired position information of the pupil, and inputs the extracted eye image into pre-built machine learning to detect a coordinate value for the outer line of the eye and a coordinate value for the pupil. It is possible to learn how to do it, and estimate the actual viewpoint value of the user's eye gaze by inputting the image captured at the current viewpoint into machine learning on which the learning is completed.

The control unit determines whether the estimated actual viewpoint value is located inside the display screen, and when the actual viewpoint value is located inside the display screen according to the determined result, the standard viewpoint is applied to the actual viewpoint value. Value can be applied.

The final viewpoint value may be calculated through the following equation.

는 최종 시점값을 나타내고,

는 표준 시점값을 나타내고,

은 실제 시점값을 나타낸다. here,

Represents the final starting point value,

Represents the standard time point value,

Represents the actual starting point.

When the user's gaze is toward the camera, the standard viewpoint value may be obtained through pre-stored position information of a camera lens mounted on a user terminal, eye position information, and a distance value between the display screen and the user.

The eye image generator uses the input image to store eye images and corresponding coordinates for all the user's gaze directions in the form of a lookup table, and then an image matching the final viewpoint value from the lookup table. Can be extracted.

The eye image generation unit learns an image using a Generatvie Adversarial Network (GAN), and applies the final viewpoint value to the eye image of the user included in the input image when the learning is completed, and has a corrected viewpoint. Can be created.

In addition, according to another embodiment of the present invention, in a method for matching gaze using a learning-based user terminal, the camera is operated while storing the display screen size of the user terminal and the location information of the camera lens Receiving an image captured through machine learning, obtaining face location information and pupil location information of the user by applying the input image to machine learning, and obtaining the user's face location information and pupil location information Estimating the line of sight of to obtain an actual viewpoint value, and deriving a corrected final viewpoint value from the obtained actual viewpoint value, the pupil whose viewpoint has changed by applying the final viewpoint value to the image of the pupil included in the input image And generating an image, and matching the pupil image of which the viewpoint is changed with the image to output the corrected image on the display screen.

As described above, according to the present invention, the user terminal applies a viewpoint compensation value to the user's actual gaze value so that the discordant gaze can be matched due to the different positions of the display screen and the camera, and deep learning is performed on the user's eye image. By applying it to create a new eye image, it has the effect of correcting only the gaze while maintaining the original original image.

In addition, according to the present invention, the user terminal corrects inconsistent gaze caused by different positions of the display screen and camera during self-portrait or video call, so that it more naturally meets the gaze during self-portrait or video call. You will be able to shoot or make video calls.

1 is a block diagram illustrating a user terminal according to an embodiment of the present invention.
2 is a flow chart for explaining a method of line-of-sight matching according to an embodiment of the present invention.
3 is an exemplary diagram illustrating a method of obtaining a face position and an eye position by machine learning in step S220 shown in FIG. 2.
FIG. 4 is a flowchart illustrating S240 shown in FIG. 3.
FIG. 5 is an exemplary diagram for explaining the coordinate values of the pupil and the angle value of the eye line output through machine learning in step S242 shown in FIG. 4.
FIG. 6 is a diagram for explaining a method of obtaining a standard time point value in step S243 shown in FIG. 4.
FIG. 7 is a diagram illustrating a method of obtaining an actual viewpoint value in step S244 shown in FIG. 4.
8 is a diagram illustrating a method of extracting an image through deep learning in step S252 shown in FIG. 2.

Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings. In this process, the thickness of the lines or the size of components shown in the drawings may be exaggerated for clarity and convenience of description.

In addition, terms to be described later are terms defined in consideration of functions in the present invention, which may vary according to the intention or custom of users or operators. Therefore, definitions of these terms should be made based on the contents throughout the present specification.

Hereinafter, a user terminal equipped with a gaze matching function according to an embodiment of the present invention will be described in more detail with reference to FIG. 1.

As shown in FIG. 1, the user terminal 100 according to the embodiment of the present invention includes a database 110, a receiver 120, a face information detection unit 130, a control unit 140, and an eye image generation unit 150. And an output unit 160.

First, the database 110 stores the size of a display screen included in the user terminal and location information of a camera lens mounted in the user terminal 100.

In addition, the display screen is located in front of the user terminal and has different sizes depending on the specifications of the product. In addition, a camera lens for self-photographing or video calling is not located at the inner center of the display screen, but is located at the upper and outer sides of the display screen. In this way, the database 110 stores the size of the user terminal 100 input by the user and location information of the camera lens mounted on the user terminal 100. Here, the user terminal is a terminal capable of taking a picture or making a video call, and includes a smart phone, a PDA, and a notebook computer.

Then, the receiving unit 120 receives an image photographed by a camera mounted on the user terminal 100 and transmits the received image to the face information detection unit 130.

The face information detection unit 130 acquires the location of the user's face and the location of the pupils displayed through the display screen of the user terminal 100.

To explain this in detail, the face information detection unit 130 inputs the received image into pre-built machine learning. In this case, the face information detection unit 130 learns machine learning to output coordinates for the user's face position and eye position using a plurality of input images. In addition, when the machine learning learning is completed, the face information detection unit 130 detects coordinate values for the user's face position and eye position by inputting the image received at the current viewpoint into machine learning.

Then, the face information detector 130 acquires a distance value corresponding to the distance between the eye and the eye by using the coordinates of the detected eye position. Then, the face information detector 130 estimates the distance between the display screen of the user terminal and the user by using the obtained distance value between pupils.

The control unit 140 extracts an eye image from an image using a coordinate value for an eye position. In addition, the control unit 140 inputs the extracted eye image into machine learning to obtain positional information on the pupil. In addition, the control unit 140 derives a difference in gaze angle by substituting the acquired position information on the pupil into the acquired standard gaze value assuming that the user's gaze is located on the camera. In this case, the control unit 140 refers to the difference value of the derived gaze angle as an actual viewpoint value. Then, the controller 140 calculates a final viewpoint value according to the difference between the actual viewpoint value and the standard viewpoint value.

The eye image generator 150 applies the calculated final viewpoint value to the eye image to generate a pupil image with a changed viewpoint. In this case, the eye image generator 150 may generate an eye pupil image by two methods.

In the first method, the eye image generator 150 stores coordinates of a plurality of eye images and eye images having different gaze directions in the form of a lookup table, and then matches the final viewpoint value from the lookup table. Snow images can be extracted.

As a second method, the eye image generator 150 learns an eye image using a Generatvie Adversarial Network (GAN), and corrects by applying a final viewpoint value to the eye image of the user included in the received image when the learning is completed. It is possible to create an image of an eye with a defined viewpoint.

Finally, the output unit 160 attaches the pupil image whose viewpoint is corrected according to the final viewpoint value to the image, and outputs the corrected image through the display screen.

Hereinafter, a method of matching the gaze using a user terminal according to an embodiment of the present invention will be described in more detail with reference to FIGS. 2 to 7.

FIG. 2 is a flow chart for explaining a gaze matching method according to an embodiment of the present invention, and FIG. 3 is an example for explaining a method of acquiring a face position and a pupil position by machine learning in step S220 shown in FIG. 2 Is also.

First, the database 110 stores information on the size of a display screen input from a user and location information of a camera lens mounted on a position outside the display screen.

Then, as shown in FIG. 2, the user terminal 100 according to an embodiment of the present invention receives an image photographed by a camera mounted outside (S210). In this case, the image represents an image generated by taking a self-portrait of a user while performing a self-photographing or video call.

Then, the receiving unit 120 transmits the received photographed image to the face information detecting unit 130.

Then, the face information detector 130 obtains the user's face location information and the eye location information by applying the received image to machine learning (S220).

Then, the face information detector 130 acquires distance information between the display screen and the user's face by using the acquired position information of the pupil (S230).

In a more detailed description of the steps S220 and S230, first, the face information detection unit 130 receives a plurality of images captured by the user from the reception unit 120. In addition, the face information detection unit 130 inputs a plurality of received images to pre-built machine learning to learn.

Then, as shown in FIG. 3, machine learning tracks the user's face and pupils included in the input image according to the learned result, and sets a bounding box at positions corresponding to the face and eyes. Then, a bounding box set with four coordinate values is output by machine learning.

When the output of the coordinate values for the user's face position and the eye position is completed, the face information detector 130 acquires a distance value between the user's eyes and the eyes using the output coordinate values for the eye position. In addition, the face information detection unit 130 estimates the distance between the display screen of the user terminal and the user through the distance value between eyes and eyes.

Then, the face information detection unit 130 transmits the acquired coordinate values of the user's face position and eye position to the control unit 140.

Then, the controller 140 obtains the user's actual viewpoint value using the received coordinate value for the eye position, and derives the corrected final viewpoint value from the obtained actual viewpoint value (S240).

Hereinafter, step S240 will be described in more detail with reference to FIGS. 4 and 5.

FIG. 4 is a flow chart for explaining step S240 shown in FIG. 3, and FIG. 5 is an exemplary diagram for explaining the coordinate values of the pupil and the angle value of the eye line output through machine learning in step S242 shown in FIG. 4 to be.

First, the control unit 140 inputs and learns the eye image acquired from the captured image into pre-built machine learning (S241).

That is, machine learning is trained to receive an eye image and output a coordinate value for an outer line of the eyeball and a coordinate value for the pupil from the input eye image.

As shown in FIG. 5, the controller 140 extracts an eye image by using a coordinate value for a user's eye position, and inputs the extracted eye image as an input image into machine learning. Then, machine learning outputs the coordinate values for the outer line of the eyeball indicated in green and the coordinate values for the pupil indicated in red. In this case, the controller 140 outputs one output image corresponding to one input image input through machine learning. However, in FIG. 5, a plurality of output images corresponding to one input image are shown in order to explain that the position of the pupil and the outline of the eyeball are changed according to the user's gaze.

When the machine learning learning is completed in step S241, the controller 140 acquires a coordinate value for the eye line and a coordinate value for the pupil by using the eye image included in the image input at the current viewpoint (S242).

)을 획득한다(S243).In addition, the control unit 140 uses the position information of the camera lens mounted in the user terminal 100 stored in the database 110, the position information of the pupil obtained in step S230, and the distance between the display screen and the user.

) Is obtained (S243).

)으로 나타낸다. On the other hand, the value obtained when the user stares at the camera in front is the standard viewpoint value (

).

)을 획득하는 방법에 대해 더욱 상세하게 설명한다. Hereinafter, using FIG. 6, the standard viewpoint value (

How to obtain) will be described in more detail.

FIG. 6 is a diagram for explaining a method of obtaining a standard time point value in step S243 shown in FIG. 4.

)을 설정한다. As shown in (a) of FIG. 6, the controller 140 generates a vertical line v in the display screen centering on the position of the camera lens using the previously stored position information of the camera lens. Then, the control unit 140 generates a horizontal line h in the display screen using the position information of the pupil obtained in step S220. Then, the control unit 140 crosses the generated vertical line (v) and horizontal line (h) to mutually orthogonal

) Is set.

)으로 설정한다. 그리고 제어부(140)는 도 6의 (b)에 도시된 바와 같이, 사용자의 실제 얼굴에서 눈과 눈 사이의 중간 지점을 3D 공간상의 중심점(O)으로 설정한다. Then, the control unit 140 determines the midpoint between the user's eyes and the eyes output through the display screen.

). And, as shown in (b) of FIG. 6, the control unit 140 sets an intermediate point between the eyes and eyes on the user's real face as a center point O in the 3D space.

과 사용자의 눈과 눈 사이의 중간 지점(O) 사이의 거리는 r_o 로 표현할 수 있다. At this time, set on the display screen

The distance between the user's eyes and the middle point (O) between them can be expressed as r _o .

)과 시선에 대한 사이각(

)을 이용하여 표준시점값(

)을 획득하며, 표준시점값(

)은 (

,

)와 같이 표현된다. Then, the control unit 140 is the angle between the pupil position (

) And the angle between the gaze (

) Using the standard time point value (

), and the standard time point value (

) Is (

,

).

)은 교차점(

), 3D 공간상의 중심점(O) 및 디스플레이상의 중심점(

)에 의해 형성되고, 시선에 대한 사이각(

)은 디스플레이상의 중심점(

), 3D 공간상의 중심점(O) 및 카메라 렌즈의 위치(c)에 의해 형성된다. Here, the angle between the pupil's position (

) Is the intersection (

), center point in 3D space (O) and center point on display (

), and the angle between the line of sight (

) Is the center point on the display (

), the center point (O) in the 3D space and the position (c) of the camera lens.

)을 획득한다(S244). When the acquisition of the standard viewpoint value is completed in step S243, the controller 140 uses the coordinate value for the pupil obtained in step S242 to use the actual viewpoint value (

) Is obtained (S244).

At this time, the user may stare directly at the camera or may stare at his/her eyes output through the display screen.

)을 획득하는 방법에 대해 더욱 상세하게 설명한다. In the following, the actual viewpoint value (

How to obtain) will be described in more detail.

FIG. 7 is a diagram illustrating a method of obtaining an actual viewpoint value in step S244 shown in FIG. 4.

)을 획득하는 방법과 동일하게 실제 시점값(

)을 획득한다. In addition, the control unit 140 is a standard time point value (

In the same way as to obtain the actual point value (

).

)은 S242단계에서 획득한 눈동자의 좌표값을 이용하여 설정한다. 즉, 제어부(140)는 눈동자와 눈동자 사이에 위치하는 중간지점을 디스플레이상의 중심점(

)으로 설정한다. However, as shown in Figure 7 (a), the center point on the display (

) Is set using the coordinate value of the pupil obtained in step S242. That is, the control unit 140 selects an intermediate point located between the pupil and the pupil.

).

)을 디스플레이 화면 내의 수평선(h)보다 다소 낮은 위치에 설정할 수 있다. For example, assuming that the user is staring at his/her eyes output on the display screen, as shown in FIG. 7(b), the user's pupils are output on the display as if they are staring at the bottom of the display. . Then, the control unit 140 is the center point on the display according to the coordinate value of the pupil (

) Can be set at a position slightly lower than the horizontal line (h) in the display screen.

)을 디스플레이 화면 내의 수평선(h)과 동일한 위치에 설정할 수 있다. On the other hand, assuming that the user is staring at the camera, the control unit 140 controls the center point on the display according to the coordinate value of the pupil (

) Can be set at the same position as the horizontal line (h) in the display screen.

)과 시선에 대한 사이각(

)을 이용하여 실제 시점값(

)을 획득하며, 실제 시점값(

)은 (

,

)와 같이 표현된다. Then, the control unit 140 is the angle between the pupil position (

) And the angle between the gaze (

) Using the actual time value (

), and the actual time value (

) Is (

,

).

)은 교차점(

), 3D 공간상의 중심점(O) 및 디스플레이상의 중심점(

)에 의해 형성되고, 시선에 대한 사이각(

)은 디스플레이상의 중심점(

), 3D 공간상의 중심점(O) 및 카메라 렌즈의 위치(c)에 의해 형성된다. Here, the angle between the pupil's position (

) Is the intersection (

), center point in 3D space (O) and center point on display (

), and the angle between the line of sight (

) Is the center point on the display (

), the center point (O) in the 3D space and the position (c) of the camera lens.

)과 실제 시점값(

)을 하기의 수학식 1에 적용하여 최종 시점값(

)을 산출한다(S245). Then, the control unit 140 obtains the standard time value (

) And the actual point in time (

) Is applied to Equation 1 below, and the final time value (

) Is calculated (S245).

는 최종 시점값을 나타내고,

는 표준 시점값을 나타내고,

은 실제 시점값을 나타낸다here,

Represents the final starting point value,

Represents the standard time point value,

Represents the actual starting point

)은 표준 시점값(

)과 실제 시점값(

) 사이에 발생된 차이값에 의해 산출된다. 여기서 표준 시점값(

)은 사용자의 시선을 카메라에 위치하고 있다고 가정한 상태에서 획득한 값이므로, 사용자가 카메라를 정면으로 바라보는 경우, 실제 시점값(

)은 표준 시점값(

)이 동일하게 된다. 따라서, 최종 시점값(

)은 0이 된다. That is, the final starting point (

) Is the standard time point (

) And the actual point in time (

) It is calculated by the difference generated between. Where the standard time point (

) Is a value obtained under the assumption that the user's gaze is positioned at the camera, so when the user looks at the camera directly, the actual viewpoint value (

) Is the standard time point (

) Becomes the same. Therefore, the final starting point (

) Becomes 0.

When derivation of the final viewpoint value is completed in step S240, the control unit 140 transmits the derived final viewpoint value to the eye image generation unit 150.

Then, the eye image generator 150 generates a pupil image with a changed viewpoint by applying the final viewpoint value to the image of the pupil included in the input image (S250).

At this time, the eye image generator 150 generates an eye pupil image whose viewpoint is changed by two methods.

Looking at the first method, first, the eye image generator 150 collects eye images for all the user's gaze directions. In addition, the eye image generator 150 stores the collected eye image and coordinate values corresponding to the eye image in the form of a lookup table. Accordingly, the eye image generator 150 extracts an eye image by comparing the received final viewpoint value with a coordinate value stored in a lookup table.

As a second method, the eye image generator 150 extracts an eye image whose gaze is corrected through deep learning.

FIG. 8 is a diagram illustrating a method of extracting an eye image whose gaze is corrected through deep learning in step S252 shown in FIG. 2.

As shown in FIG. 8, the eye image generator 150 learns the pupil image using a previously constructed GAN (Generatvie Adversarial Network) network.

That is, the eye image generator 150 trains the GAN network to receive the user's eye image and the final viewpoint value using the GAN network and output an image of the pupil having a new line of sight value.

When learning of the GAN network is completed, the eye image generation unit 150 inputs an image of the user's eyes included in the input image into the GAN network. Then, the GAN network creates a pupil image with a corrected viewpoint by applying the final viewpoint value.

When the creation of the pupil image having a new viewpoint is completed in step S250, the output unit 160 matches the generated pupil image with the input image and outputs the corrected image (S260).

That is, the output unit 160 outputs a state in which only the gaze is corrected while maintaining the shape of the eyes included in the input image as it is.

As described above, the user terminal according to an embodiment of the present invention applies a viewpoint compensation value to the user's actual gaze value so that the discordant gaze can be matched due to the different positions of the display screen and the camera. By applying deep learning to create a new eye image, it has the effect of correcting only the gaze while maintaining the original original image.

In addition, the user terminal according to an embodiment of the present invention corrects the gaze caused by the different positions of the display screen and the camera in software when taking a self photo or performing a video call. You will be able to meet your eyes more naturally and make self-portraits or video calls.

The present invention has been described with reference to the embodiments shown in the drawings, but these are only exemplary, and those of ordinary skill in the art will understand that various modifications and equivalent other embodiments are possible therefrom. will be. Therefore, the true technical protection scope of the present invention should be determined by the technical idea of the following claims.

100: user terminal
110: database
120: receiver
130: face information detection unit
140: control unit
150: eye image generator
160: output

Claims (16)

In a user terminal equipped with a gaze matching function,
Database storing the display screen size of the user terminal and the location information of the camera lens mounted on the user terminal,
A receiving unit receiving an image captured through the camera,
A face information detector configured to obtain face location information and eye location information of a user by applying the input image to machine learning,
Extracting an eye image from the input image using the acquired eye position information, and inputting the extracted eye image into pre-built machine learning to detect a coordinate value for the outer line of the eye and a coordinate value for the pupil. A control unit that learns a method and estimates an actual viewpoint value for the user's eye gaze by inputting an image captured at the current viewpoint into machine learning on which the learning is completed, and derives a corrected final viewpoint value from the estimated actual viewpoint value,
An eye image generator configured to generate a pupil image with a changed viewpoint by applying the final viewpoint value to the image of the pupil included in the input image, and
And an output unit for matching the pupil image of which the viewpoint is changed to the image and outputting the corrected image on the display screen,
The face information detection unit,
The image captured by the camera is input into pre-built machine learning to learn the user's face position and eye position, and the user's face position and eye position by inputting the image acquired while learning is completed into machine learning Is detected,
A user terminal that acquires information on a distance between a display screen and a user by using the detected distance between the eyes and the eyes. delete delete The method of claim 1,
The control unit,
It is determined whether the estimated actual viewpoint value is located inside the display screen, and when the actual viewpoint value is located inside the display screen according to the determined result, a standard viewpoint value is applied to the actual viewpoint value. User terminal that calculates the final time value. The method of claim 4,
The final time value is,
A user terminal calculated through the following equation;

here,

Represents the final starting point value,

Represents the standard time point value,

Represents the actual starting point. The method of claim 5,
The standard time point is,
When the user's gaze is toward the camera, the user terminal is obtained through the position information of the camera lens, the position information of the eye, and a distance value between the display screen and the user. The method of claim 5,
The eye image generator,
A user terminal that stores eye images and corresponding coordinates for all the user's gaze directions in the form of a lookup table using the input image and then extracts an image matching the final viewpoint value from the lookup table. The method of claim 5,
The eye image generator,
A user terminal that learns an image using a Generatvie Adversarial Network (GAN), and generates a pupil image having a corrected viewpoint by applying the final viewpoint value to the user's eye image included in the input image when the learning is completed. In the gaze matching method using a user terminal,
Receiving an image captured through the camera while storing the display screen size of the user terminal and location information of the camera lens attached to the user terminal,
Applying the input image to machine learning to obtain face location information and pupil location information of the user,
Learning a method of extracting an eye image from an image using the acquired position information of the pupil, and inputting the extracted eye image into pre-built machine learning to detect the coordinate value of the outer line of the eye and the coordinate value of the pupil. Then, inputting the image received at the current time point into machine learning on which the learning was completed, estimating the actual viewpoint value of the user's eye gaze, and deriving the corrected final viewpoint value from the estimated actual viewpoint value,
Generating a pupil image with a changed viewpoint by applying the final viewpoint value to the image of the pupil included in the input image, and
Comprising the step of matching the pupil image of which the viewpoint is changed to the image and outputting the corrected image on the display screen,
The step of obtaining the user's face location information and the pupil location information,
Learning the location of the user's face and pupils by inputting the image captured through the camera into pre-built machine learning,
The step of detecting the position of the user's face and the pupil by inputting the image acquired while the learning is completed into machine learning, and
And acquiring information on a distance between a display screen and a user by using the detected distance between the pupil and the pupil. delete delete The method of claim 9,
The step of deriving the final viewpoint value,
It is determined whether the estimated actual viewpoint value is located inside the display screen, and when the actual viewpoint value is located inside the display screen according to the determined result, a standard viewpoint value is applied to the actual viewpoint value. A gaze matching method that calculates the final viewpoint value. The method of claim 9,
The final time value is,
A method of matching the gaze calculated through the following equation;

here,

Represents the final starting point value,

Represents the standard time point value,

Represents the actual starting point. The method of claim 13,
The standard time point is,
When the user's gaze is facing the camera, the user terminal is obtained through the location information of the camera lens mounted in the user terminal, the position information of the eye, and the distance value between the display screen and the user. The method of claim 13,
The step of generating the pupil image in which the viewpoint is changed,
Gaze matching that stores eye images for all the user's gaze directions and coordinates corresponding to them in the form of a lookup table using the input image, and then extracts an image matching the final viewpoint value from the lookup table. Angry way. The method of claim 13,
The step of generating the pupil image in which the viewpoint is changed,
Gaze matching that trains an image using GAN (Generatvie Adversarial Network) and generates a pupil image with a corrected viewpoint by applying the final viewpoint value to the user's eye image included in the input image when the learning is completed. Way.

Images