KR101492832B1 - Method for controlling display screen and display apparatus thereof - Google Patents

Abstract

An image display apparatus for displaying an image by tracking an eye of a user according to the present invention includes a display unit for displaying an image with a position, magnification, or rotation angle according to a control signal, a mode switching determination unit A camera module for photographing a user's eyeball, and an eye movement analyzer for analyzing a movement of the user's eyeball from an image captured by the camera module, and an eye movement analyzer for receiving an analysis result signal, And a control unit for generating a control signal for adjusting at least one of a position, a magnification, and a rotation angle.

Description

BACKGROUND OF THE INVENTION 1. Field of the Invention [0001] The present invention relates to an image control method,

The present invention relates to an image control method and an image display apparatus using the same, and more particularly, to an image control method for controlling an image displayed on an image display device by analyzing motion of a user's eyeball and a video display device using the same.

A video output screen of a portable terminal or a personal computer may be viewed continuously for a long time. In the case of using such a long time, unlike the intention of the user, there is a difference between the screen to be watched by the user and the screen to be watched by the actual user due to low concentration or drowsiness during the work.

There is a way to increase the concentration by working again after having a break after a certain working time, but in many cases, it does not have a break time at a specified time due to the nature of the work. The effect of fatigue or fatigue caused by long time work causes changes in user's posture and slowing of eye movement.

You can use your fitness more efficiently if you have improved convenience throughout the entire process. Particularly, when an incorrect posture is taken due to a change in attitude due to a loss of concentration, the consumption of the physical strength in performing the work becomes larger, which adversely affects the efficiency of work.

Therefore, it is necessary to develop an image display apparatus and an image display method in which a screen moves and changes according to a user's eye movement.

Korean Patent Publication No. 10-2012-0040320

SUMMARY OF THE INVENTION The present invention provides an image control method for controlling an image displayed on an image display device by analyzing movement of a user's eyeball and a video display device using the same.

According to an aspect of the present invention,

A video display device for displaying an image by tracking a user's eyeball,

A display unit for displaying an image with a position, magnification, or rotation angle according to a control signal;

A mode switching determination unit for determining whether or not to execute the eye tracking mode;

A camera module for photographing the user's eyeball;

An eye movement analyzer for analyzing a movement change of the user's eyeball from an image taken by the camera module; And

And a control unit for receiving the analysis result signal from the eye movement analyzing unit and generating the control signal for adjusting at least one of the position, magnification, and rotation angle of the image displayed on the display unit.

According to an embodiment, the mode switching determination unit may determine,

A timer for counting the elapse of a predetermined time; And

And an input sensing unit for sensing whether or not the user has input during the time set by the timer.

According to an embodiment, the mode switching determination unit may determine,

And a mode selection unit that is selected by a user to perform mode switching.

According to one embodiment,

The input of the user may be received through at least one of a keyboard input, a mouse input, a touch input, and an audio input.

According to one embodiment,

And a light emitting unit for illuminating the infrared rays toward the user ' s eyeball.

According to one embodiment, the control unit

And a controller for receiving an analysis result signal from the eye movement analyzing unit and adjusting the position of the image so that the image is displayed around the gazing point of the user's eye or adjusting the magnification of the image according to the distance between the user and the image display device, The control signal can be generated so as to adjust the rotation angle so that the image is displayed in parallel with the straight line formed by the two eyes of the eye.

According to one embodiment,

And a distance measuring unit for calculating a distance from the image captured by the camera module to the eyeball of the user and the camera module based on an area change of the user's eyeball or an interval between the two eyes.

According to one embodiment,

And a distance measuring unit for calculating a distance from the user's eye to the camera module using at least one of a proximity sensor, an infrared sensor, a radio frequency (RF) sensor, and an ultrasonic sensor.

An image control method for tracking an eye of a user and displaying an image on a display unit according to another aspect of the present invention includes:

Determining whether the eye tracking mode is executed;

Photographing a user's eyeball;

Analyzing a change in the user's eye movement from the photographed image; And

And generating the control signal for adjusting at least one of a position, a magnification, and a rotation angle of an image displayed on the display unit using the analysis result.

According to one embodiment, the display unit is electrically connected to user input means,

Wherein the step of determining whether to execute the eye tracking mode comprises:

After the user input through the user input means is detected, if the user's input is not detected again before the predetermined time elapses, the eye tracking mode can be executed.

According to one embodiment,

The user input means may use at least one of a keyboard input, a mouse input, a touch input, and an audio input.

According to one embodiment,

The step of analyzing the change of the user's eyeball movement may analyze the change of the gaze direction of the user's eyeball, the approach of the user's eyeball to the display unit, and the change of the gaze angle of the user's eyeball.

According to one embodiment,

The step of analyzing whether the user's eyeball is approaching the display unit may analyze a change in size of the user's eyeball or a distance between two eyes of the user from the photographed image.

According to one embodiment,

Wherein the step of analyzing whether the user's eyeball is approaching the display unit comprises:

The distance between the user's eyeball and the display unit can be calculated and analyzed from the photographed image.

According to one embodiment, the control signal < RTI ID = 0.0 >

And adjusting the magnification of the image according to the distance between the user and the image display device or adjusting the magnification of the image according to the distance between the user and the image display device, And to adjust the rotation angle so that the image is displayed side by side on the straight line.

The present invention can be embodied as a computer-readable recording medium having recorded thereon a program for causing a computer to execute a method according to the embodiments.

According to the image control method and the image display apparatus using the image control method of the present invention, even if a change in the gaze direction of the user's eye is caused by a user's use for a long time, the screen itself moves according to the current user gazing direction, You can keep gazing at the screen.

In addition, even when the gaze distance, such as the approach of the user's eye to the display unit, is changed, the user can continue to gaze at the screen in which the screen shape that the user originally gazed at is maintained.

Further, even when the gaze angle of the user's eye changes, the screen itself is rotated in accordance with the current gaze angle, so that the user can gaze at the screen continuously in a form that the user has originally gazed upon.

1 is a block diagram of an image display apparatus according to an embodiment of the present invention.
2 is an exemplary diagram for analyzing a gaze direction of a user's eye using a user's eye image acquired by a camera module unit of an image display apparatus according to an embodiment of the present invention.
3 and 4 are diagrams for explaining a method of controlling an image of a video display device according to an embodiment of the present invention.
5 is a flowchart illustrating an image control method according to an embodiment of the present invention.

Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings. First, it should be noted that the same components or parts among the drawings denote the same reference numerals whenever possible. In the following description of the present invention, a detailed description of known functions and configurations incorporated herein will be omitted so as not to obscure the subject matter of the present invention.

1, the image display apparatus according to an exemplary embodiment of the present invention includes a display unit 700 for displaying an image, a mode switching determination unit 100 for determining whether the eye tracking mode is executed, An eye movement analyzing unit 500 for analyzing a gazing direction of the user's eye from an image taken by the camera module, and a display unit 500 for displaying an analysis result signal from the eye movement analyzing unit. And a control unit 200 for generating a control signal for receiving an image displayed on the display unit, adjusting magnification, or adjusting a rotation angle.

A distance measuring unit 600 for calculating a distance from the image captured by the camera module to the eyeball of the user and the camera module, As shown in FIG.

The present invention detects a change in unintentional user's gaze direction due to a decrease in concentration and fatigue caused when a user uses the liquid crystal screen of a computer monitor or a mobile communication terminal for a long period of time, It is necessary to determine whether the use state of the user is a general use or an abnormal use due to use for a long time by moving a screen or a liquid crystal screen of the mobile communication terminal. Therefore, it is necessary to determine whether or not the image display apparatus is to execute the eye tracking mode.

The mode change determination unit 100 may include a timer 110 and an input detection unit 120. The mode change determination unit 100 may include a timer 110 and an input sensing unit 120. [ . The timer 110 counts the elapse of a predetermined time. The input detection unit 120 detects whether there is a re-input by the user again after the time set by the timer 110 has elapsed after the input by the user through the user input means. If there is a re-input within the set time, the current use state of the current user is determined to be normal use, and if there is no re-input within the set time, the current use state of the user is judged to be abnormal use. The set time may be a time settable by the user.

Here, the user's input through the user input means includes a keyboard input for inputting character numbers and the like through the keyboard, a mouse input for moving the cursor through the mouse or a mouse input for performing left / right click / A voice input for inputting a certain voice to a microphone formed in a video display device (monitor, mobile communication terminal, etc.), and the like. This is merely an example, but is not limited thereto.

In addition, the mode selection unit 130 may include an automatic mode switching determination by the timer 110 and the input sensing unit 120 as well as an eye tracking mode manually by user selection. The mode selection unit 130 may be implemented in hardware or software in an image display apparatus according to an embodiment of the present invention. That is, it may be implemented as a button at a predetermined position of the computer monitor, or may be implemented as a touch menu of the mobile communication terminal.

When the mode switching determination unit 100 switches from the normal mode to the eye tracking mode and is executed, the camera module unit 400 photographs the user's eyeball to acquire the user's eyeball image.

According to an embodiment, the apparatus may further include a light emitting unit 300 for illuminating the user's two eyes to effectively photograph the user's eyeball.

The light emitting unit 300 emits infrared rays and can illuminate the user's two eyes. The position and number of the light emitting unit 300 may vary depending on the embodiment. The light emitting unit 300 may be an infrared LED or an LED array in which an infrared LED is mounted on a substrate. Preferably symmetrically about the optical axis for uniform illumination.

When the infrared ray is illuminated by the light emitting unit 300, the eyeball image of the user is captured by the camera module unit 400. The camera module unit 400 may include a plurality of image sensors. The image sensor includes a lens having a predetermined curvature and serves to receive an optical signal of a subject, for example, an iris. The image sensor may be implemented by a CCD (Charge Coupled Device) or CMOS device having a plurality of pixels.

In addition to the image sensor, the camera module unit 400 can also use a close-up lens to take a close-up image of a subject, that is, an iris. The camera module unit 400 may be installed on one side of the image display device. A plurality of camera module units 400 may be installed in the image display device to photograph an eye image of the user.

The eye movement analyzing unit 500 may analyze the gazing direction of the user's eyeball from the image captured by the camera module unit 400. [ An analysis of the direction of the user's eye gaze by the eye movement analyzer 500 will be described with reference to FIG.

For example, the eye movement analyzing unit 500 receives an eye image of a user photographed by the camera module unit 400 as an image signal, and displays the image of the user's eyes 12 and / Identifies the outer shape (13) of the user's eye and continuously monitors changes in the pupil (12) and the outer shape (13) of the eye in the identified image.

It is detected that the center 12a of the user's pupil 12 is deviated from the center of the outer shape 13 of the eye to detect the center of the pupil 12 corresponding to the direction and the distance from the center of the outer shape 13 of the eye And generates an analysis result signal for moving a screen displayed on the display unit 700. Here, the eye movement analyzing unit 500 stores the initial coordinate information (coordinates of the eye taken when the eye tracking mode is started) related to the center 12a of the pupil 12 as the center of the eye outer shape 13 The change of the center 12a of the pupil 12 is recognized, and the analysis result signal is generated and sent to the control unit 200. [

In addition, the eye movement analyzing unit 500 analyzes whether or not the user's eyes are approaching the display unit 700. The analysis of whether or not the user's eyeball is accessed by the eyeball motion analyzer 500 is as follows.

First, a change in the size of the user's eyeball may be analyzed from the image captured by the camera module unit to determine whether the user is approaching the camera. When the user approaches the image display device, the area occupied by the eyeball in the image photographed by the camera module becomes large, and in the opposite case, the area becomes small. Therefore, the eye movement analyzing unit 500 can analyze the change in the eye size of the user in the photographed image to determine whether or not to approach the eye.

It is possible to determine whether or not to access by analyzing the distance change between the two eyes of the user from the image captured by the camera module unit. When the user approaches the image display device, the distance between the two eyes in the image captured by the camera module becomes small, and in the opposite case, the distance becomes large. Therefore, the eye movement analyzing unit 500 can determine the approach by analyzing the change in the distance between the two eyes of the user in the photographed image.

Third, it is possible to directly determine the distance from the image captured by the camera module to the eyeball of the user and the camera module to determine whether or not the camera module is accessible. The distance measuring unit 600 may be included to calculate the distance.

The distance measuring unit 600 may calculate the distance from the image captured by the camera module unit 400 to the camera module 400 of the user's eye. The distance measuring unit 600 may include at least one of a camera, a proximity sensor, an infrared sensor, a radio frequency (RF) sensor, and an ultrasonic sensor. The distance measuring unit 600 senses a user and outputs a sensing signal to the controller 200. [ The sensing signal includes distance information necessary for calculating the distance between the camera module unit 400 and the user's eyeball. The distance information may include at least one of a source used for user detection, for example, an irradiation speed of an infrared ray, an ultrasonic wave, a time of detecting the user's eye, and a shot image.

The distance from the user's eye to the camera module unit 400 can be calculated using at least one of a camera vision system, an infrared system, and an ultrasound system. For example, in the camera vision method, the eye image size of the user photographed by the camera module unit 400 in the manner of calculating the distance from the eye image size of the user photographed by the camera module unit 400 is measured by the camera module unit 400, Inversely proportional to the distance between them.

The infrared (or ultrasonic) method is a method of calculating the distance between the camera module unit 400 and the user's eye by receiving a reflected wave of infrared rays illuminated by the user's eye in the light emitting unit 300. After the infrared light is illuminated from the light emitting unit 300 to the user's eye, the reflected wave is received by the distance measuring unit 600 and transmitted to the control unit 200. At this time, the detection signal includes the reception time of the reflected wave as the distance information. The control unit 200 can calculate the distance between the camera module unit 400 and the user's eye based on the reception time of the reflected wave included in the detection signal and the irradiation speed of the infrared rays. The distance between the camera module unit 400 and the user's eye can be calculated by multiplying the reception time by the irradiation speed and then dividing by 2, for example, assuming that the irradiation speed of infrared rays and the speed of the reflected wave are the same.

The control unit 200 switches the mode of the image display device to the eyeball tracking mode when receiving the mode switching signal from the mode switching determination unit 100 and controls the light emitting unit 300, the camera module unit 400, 500, the distance measuring unit 600, and the display unit 700. In particular, the control unit 500 receives the analysis result signal from the ocular motion analysis unit 500, and generates a control signal for moving the image displayed on the display unit 700, adjusting the magnification or adjusting the rotation angle.

For example, as shown in FIG. 3, when the gaze direction of the user's eye after the start of the eye tracking mode is analyzed as being changed from the center of the display portion 700 to the lower right direction (refer to a reference symbol a) For example, in order to minimize movement, a control signal is generated to move a screen displayed so that the current gazing direction becomes the new center of the current screen, for example, in the upper left direction (see reference symbol b).

Also, for example, when the eyeball tracking mode is analyzed, when the eyeball of the user is analyzed as approaching to the display unit, the magnification of the image displayed on the display unit is reduced, and when the user eyeball is moved away from the display unit, And generates a control signal for enlarging the magnification of the image displayed on the display unit. More specifically, when the eyeball size value changed on the basis of the eyeball size value at the start of the eyeball tracking mode is increased by 50%, the magnification of the image is set to 70%, and when the eyeball size is decreased by 50% Can be specified. This magnification designation is merely an example, but the present invention is not limited thereto.

Therefore, even if the user changes his / her posture, the display unit can provide an experience of continuously viewing images of almost the same size.

Further, for example, as shown in FIG. 4, when the user's eyeball has been rotated by a predetermined angle (refer to reference c) with respect to the horizontal direction after the start of the eyeball tracking mode, And generates a control signal for turning the screen displayed so that the current gazing angle becomes the new horizontal of the current screen clockwise or counterclockwise (refer to reference numeral d). It is preferable to limit the rotation range to be 20 degrees in the clockwise direction with respect to the horizontal before the eye tracking mode is started, but the angle range is not limited thereto.

Therefore, even if the user changes his / her posture, the display unit can display the image to be inclined according to the inclination of the user's head, and the user can always view the image horizontally on the plane based on the plane formed by the two eyes.

Next, an image control method according to an embodiment of the present invention will be described with reference to FIG.

First, it is determined whether or not the eyeball tracking mode is to be executed in the state where the current image is displayed (S20). For example, whether or not the eyeball tracking mode is executed may be determined, It can be determined whether or not the input of the user is re-detected through the input means. For this, input detection by the timer 110 and the input sensing unit 120 is necessary, as described above. Also, it can be determined whether or not the eye tracking mode is selected by the user.

If it is determined in step S20 that the eye tracking mode has not been switched, the current normal mode is maintained (S30), and if the mode is switched to the eye tracking mode, the eye tracking mode is executed (S40)

When the eye tracking mode is executed, the user's eyeball is photographed (S50) from the camera module unit 400, and a change in the gaze direction of the user's eyeball, whether the user's eyeball approaches the display unit, (S60) The change of the gazing direction of the user, the approach of the user's eye, and the change of the gaze angle of the user's eye are the same as described above. The description is omitted.

Next, a control signal for adjusting at least one of a position, a magnification, and a rotation angle of an image displayed on the display unit 700 is generated using the analysis result (S70). Here, An image magnification signal for enlarging or reducing the magnification of the image according to the approach of the user's eye, an image rotation signal for rotating the screen according to the angle of gaze of the user's eye, and the like.

According to the image control method and the image display apparatus using the image control method according to an embodiment of the present invention, even if the direction of gazing at the user's eye changes due to the user's use for a long time, the screen is moved in accordance with the current user's gazing direction The user can continue to gaze at the screen that he was originally staring at.

In addition, even when the gaze distance changes, such as when the user's eyes approach the display unit, the user can keep gazing at a screen similar to the screen magnification originally used by the user.

Further, even when the gaze angle of the user's eye changes, the screen is rotated in accordance with the current gaze angle, so that the user can gaze at the original gaze continuously.

As described above, the image control method according to the present invention and the image display device using the image control method are described with reference to the drawings. However, the present invention is not limited to the embodiments and drawings disclosed in the present specification, It is to be understood that various modifications may be made by those skilled in the art.

100: Mode conversion determination unit 200:
300: light emitting part 400: camera module part
500: Eye movement analyzing unit 600: Distance measuring unit

Claims (16)

A video display device for displaying an image by tracking a user's eyeball,
A display unit for displaying an image with a magnification or a rotation angle according to a control signal;
A timer for counting whether or not a predetermined time has elapsed; and an input sensing unit for detecting whether or not a user has input during the preset time. If there is an input within a predetermined time, A mode switching judging unit for judging that the current use state of the current user is abnormal and executing the eye tracking mode if there is no input within a preset time;
A camera module unit for photographing a user's eyeball when the eyeball tracking mode is executed;
A distance measuring unit for calculating a distance from the image captured by the camera module to the eyeball of the user and the camera module based on an area change of the eyeball of the user or an interval between the two eyes,
An eye movement analyzer for analyzing a movement change of the user's eyeball from an image taken by the camera module; And
A control signal generating unit that receives an analysis result signal from the eye movement analyzing unit and adjusts the magnification of the image according to the calculated distance or adjusts the rotation angle so that an image is displayed in parallel with a straight line formed by two eyes of the user The control unit
And a display device. delete The apparatus according to claim 1,
And a mode selection unit for selecting the mode selected by the user and executing the mode switching. The method according to claim 1,
Wherein the input of the user is received through at least one of a keyboard input, a mouse input, a touch input, and an audio input. The method according to claim 1,
And a light emitting unit for illuminating an infrared ray toward a user's eyeball. delete delete The method according to claim 1,
Further comprising a distance measuring unit for calculating a distance from the user's eye to the camera module using at least one of a proximity sensor, an infrared sensor, an RF (Radio Frequency) sensor, and an ultrasonic sensor. . An image control method for tracking an eyeball of a user and displaying an image on a display unit,
If the user's input is detected within a predetermined period of time, it is determined that the current use state of the current user is normal, and if there is no input within the predetermined time, the current use state of the current user is determined to be abnormal Determining whether an eyeball tracking mode is to be executed;
Photographing a user's eyeball with the camera module unit when the eyeball tracking mode is executed;
Calculating a distance between the eyeball of the user and the camera module based on an area change of the user's eyeball or an interval between the two eyeballs;
Analyzing a change in the user's eye movement from the photographed image;
Generating a control signal for adjusting a magnification of an image according to the calculated distance using the analysis result or adjusting a rotation angle so that an image is displayed in parallel with a straight line formed by two eyes of a user; And
And displaying an image with a magnification or rotation angle of the adjusted image according to the control signal. delete The method of claim 9,
Wherein the user input is at least one of a keyboard input, a mouse input, a touch input, and an audio input. The method of claim 9,
Wherein analyzing the change in the user's eye movement comprises analyzing whether a change in a gaze direction of the user's eyeball, a proximity of the user's eyeball to the display unit, and a change in a gaze angle of the user's eyeball are analyzed. Way. delete delete delete A computer-readable recording medium storing a program for causing a computer to execute the method according to claim 9 or 11.

Images