KR101247780B1 - mobile phone with eyeball sensing function and display processing method there of - Google Patents

Abstract

The present invention provides a display unit for providing a screen for displaying arbitrary data, a memory for storing at least one or more data to be displayed on the screen, a camera unit for photographing a human face, and a face image photographed through the camera unit. And a control unit for extracting motion data and scrolling data displayed on the screen according to the extracted motion data.

According to the present invention, when the user wants to scroll the displayed data, the user can view the data reflecting the movement of the gaze by merely moving the gaze without directly manipulating a predetermined key.

Eyeball, movement, data, scrolling, shooting, screen

Description

Mobile communication terminal with eye detection function and its screen processing method {mobile phone with eyeball sensing function and display processing method there of}

1 is a block diagram of a mobile communication terminal according to an embodiment of the present invention.

2 and 3 are examples of application of a mobile communication terminal according to an embodiment of the present invention.

4 is an operation flowchart of a mobile communication terminal according to an embodiment of the present invention.

<Explanation of symbols for the main parts of the drawings>

10: mobile communication terminal 100: display unit

110, 120: window 111, 121: upper part

112, 122: Display portion 113, 123: OK button

200: camera unit 300: memory

400: control unit 410: eyeball image processing unit

420: motion data extractor 430: screen processor

500: key input unit 600: RF unit

700: voice processing unit 800: speaker

900: microphone

The present invention relates to a mobile communication terminal having an eye detection function and a screen processing method thereof, and to a mobile communication terminal having an eye detection function for scrolling data displayed on a screen by detecting a user's eye movement. It is about.

In a conventional mobile communication terminal, when a user wants to scroll data displayed on a screen, the user directly operates various function keys provided in the terminal.

However, in this case, the user had to manipulate a key every time for the displayed data to move, and if the key was located inside the slider or flip, it was inconvenient to open the slider or flip.

SUMMARY OF THE INVENTION The present invention has been made to solve such a conventional problem, and an object of the present invention is to provide a mobile communication terminal having an eye detection function for automatically scrolling data displayed on a screen without a user's direct manipulation, and a screen processing method thereof. have.

According to an aspect of the present invention for achieving the above object, a memory for storing at least one or more data to be displayed on the screen, a display unit for providing a screen for displaying any data, a camera unit for photographing a human face, And a control unit for extracting eye movement data from a face image photographed through a camera unit and performing scrolling of data displayed on the screen according to the extracted movement data. Provide a terminal.

Preferably, the data displayed on the screen includes at least one of menu items, text data, image data, and image data.

Preferably, the motion data includes at least one of direction data and velocity data of the user's eye movement.

Preferably, the control unit comprises an eyeball image processing unit for obtaining an eyeball image by identifying the eyeball from a face image photographed through a camera unit, a motion data extraction unit for extracting eyeball motion data by analyzing the acquired eyeball image, and extracted And a screen processor that scrolls the data displayed on the screen based on the motion data.

According to an aspect of the present invention, screen processing for performing a step of photographing a human face, extracting eye movement data from the photographed face image, and scrolling the data displayed on the screen according to the extracted motion data It provides a screen processing method of a mobile communication terminal comprising the step.

Preferably, the data displayed on the screen includes at least one of menu items, text data, image data, and image data.

Preferably, the motion data includes at least one of direction data and velocity data of eye movement.

Preferably, the screen processing step includes determining a direction and speed to scroll based on the motion data, and scrolling data displayed on the screen at the determined direction and speed.

Hereinafter, preferred embodiments of the present invention will be described with reference to the accompanying drawings.

1 is a block diagram illustrating a mobile communication terminal having an eye detection function according to an embodiment of the present invention.

Referring to FIG. 1, the mobile communication terminal 10 includes a display unit 100 for providing a screen on which arbitrary data is displayed, a camera unit 200 for photographing a human face, and eye movement based on the captured image data. A memory 300 for storing a programmed program (hereinafter, referred to as a "screen scrolling program") and a program stored in the memory 300 by reading a program stored in the memory 300 and programmed in the corresponding program. The control unit 400, a key input unit 500, an RF unit 600, a voice processing unit 700, a speaker 800, a microphone 900, and the like, which perform a procedure, are configured.

First, the display unit 100 is a display device such as a liquid crystal display (LCD). The display unit 100 displays a state of a mobile communication terminal or a progress of a program on a screen window (not shown) under the control of the controller 400.

In particular, the display unit 100 is provided to display the scrolled data under the control of the control unit by providing a screen window 110 (shown in FIG. 2) displaying at least one or more data according to an embodiment of the present invention. .

Referring to FIG. 2, the screen window 110 includes an upper portion 111 for informing information of data, a display portion 112 for displaying arbitrary data, and a confirmation button 113 according to an embodiment of the present invention. It is configured to include).

The camera unit 200 includes components of a digital camera such as a semiconductor image sensor such as a solid state imaging device (CCD) or a CMOS image sensor (CIS), a flexible printed circuit board (FPCB), a lens for a camera, at least one flash, and the like. According to a control signal of the controller 400, an operation of photographing an external shape of a face image of a user is performed.

The memory 300 stores a plurality of data to be displayed on the screen, a screen scrolling program for scrolling the screen data according to the movement of the eyeball of the user, and an operation program and a system program necessary for various mobile communication functions.

In this case, the plurality of data may include at least one of a menu item, text data, image data, and video data.

The controller 400 controls the overall operation of the mobile communication terminal and simultaneously extracts eye movement data from the face image photographed through the camera unit 200 according to the procedure of the screen scrolling program according to an embodiment of the present invention. Extracts and scrolls data displayed on the screen according to the extracted motion data.

To this end, the controller 300 may include an eyeball image processor 410 for identifying the eyeball from a face image photographed by the camera unit 200, a motion data extractor 420 for extracting eyeball motion data, It is configured to include a screen processor 430 for scrolling the data displayed on the screen according to the movement of the eye.

The eyeball image processor 410 identifies the eyeball from the face image photographed by the camera unit 100 and acquires the eyeball image.

The eyeball image processor 410 uses a face modeling technique to identify an eyeball from a face image and obtain an eyeball image.

In this case, the face modeling technique refers to a series of analysis processes for converting the face image photographed through the camera unit 100 into digital information for processing and transmission of the face image.

Known techniques for face modeling include Active Shape Modeling (ASM) and Active Appearance Modeling (AAM).

Among them, AAM searches for the part having the most similar texture based on the learned texture information while maintaining the shape of the face by using the shape information and texture information of the face. Technique.

A mobile communication terminal according to an embodiment of the present invention uses AAM to obtain an eyeball image from a face image.

In this case, for face modeling, a process of building a database of average face information is required. The average face information is information obtained by extracting face information of each person based on a predetermined criterion from face images of several people and calculating an average value of the extracted information.

To this end, the eyeball image processing unit 410 according to an embodiment of the present invention has been implemented to use a landmark technique for extracting shape information of a face from a face image, but the present invention is not limited thereto.

In this case, the landmark technique is a technique for extracting feature points of the face image, and designates a plurality of specific positions in the face image, and extracts feature points of the face image by reading information on the specified positions. Therefore, the eyeball image processor 410 acquires an image of the eyeball through the landmark technique.

In addition, the eyeball image processor 410 needs to clearly capture various types of poses in order to construct a database of average face information according to a face modeling technique. Therefore, in order to recognize this type of pose, a similar type of learning image should be constructed. Therefore, the shape of a face that can be generated in a photographic image is generally analyzed, and a learning image is constructed based on this.

In order to extract the average face information, the front face of the input image is referred to. Therefore, the eyes, the position of the eyes and the shape of the mouth will be the main ones, and the size of the nose, the size of the eyes, and the outline of the jaw will vary depending on individual differences. . Therefore, the database is composed based on the visual field, eyes, and mouth shape, which are the main causes of the main deviation, and additionally inputs the parts due to individual differences.

For example, a face image is photographed with respect to a pose in a state of looking at the front and a state of smiling at the front, and face shape information is extracted using the landmark technique.

As such, the eyeball image processor 410 extracts face shape information from the face image photographed by the camera unit 100, and face modeling based on preset average face information with respect to the extracted face shape information. The eyeball is determined through the eyeball image.

The motion data extractor 420 analyzes the eyeball image obtained by the eyeball image processor 410 to extract eyeball motion data.

At this time, the motion data includes at least one or more of the direction data, the velocity data for the movement of the eye according to an embodiment of the present invention.

In particular, the direction data among the motion data indicates which of the eyes moves up, down, left, or right.

Therefore, when the user changes the gaze downward, the motion data recognizes the direction of movement downward. In addition, when the user changes the gaze to the right, the motion data recognizes the direction of moving to the right.

Velocity data is calculated by recognizing the rate of change of eye movement when eye movement occurs. Therefore, if the user changes his gaze quickly, the user recognizes the speed data accordingly.

The screen processor 430 scrolls the data displayed on the screen based on the extracted motion data and displays the data on the screen window 120.

In detail, the screen processor 430 determines the speed and direction to scroll by using the speed data and the direction data included in the motion data according to an embodiment of the present invention, and scrolls the data according to the determined speed and direction. The operation of displaying on the window 120 is performed.

Referring to FIG. 3, the screen window 120 includes an upper portion 121 for notifying information of data and a display portion 122 for displaying data scrolled by the screen processor 430 according to an embodiment of the present invention. ) And the confirmation button 123.

On the other hand, the key input unit 500 is composed of a plurality of numeric keys, menu keys and function keys for performing various functions, and outputs the key data to the controller 300 by an external operation.

The RF unit 600 is controlled by the controller 300 and converts a signal output from the controller 300 into a wireless signal. In addition, the RF unit 600 converts the radio signal received through the antenna into a desired signal and outputs it.

The voice processing unit 700 modulates the voice signal input from the microphone 900 and converts the voice signal into voice data, and demodulates the voice data input from the RF unit 500 into the voice signal. Output through voice.

3 is an operation flowchart of a mobile communication terminal 10 according to an embodiment of the present invention.

First, the camera unit 200 photographs a human face (S1).

The eyeball image processor 410 extracts face shape information from the face image photographed by the camera unit 100 using a landmark technique, and extracts the eyeball image using face modeling on the extracted face shape information. Acquire (S2).

The motion data extractor 420 analyzes the obtained eyeball image and extracts eyeball motion data such as the eyeball direction and speed (S3).

The screen processor 430 determines the speed and direction to scroll based on the extracted motion data (S4).

In addition, the screen processor 430 scrolls the data displayed on the screen window 110 at the determined speed and direction and displays the data on the screen window 120 (S5).

Accordingly, it can be seen that the data displayed on the display portion 112 of the screen window 110 is displayed in the scrolled state in accordance with the movement of the eye in the display portion 122 of the screen window 120.

The present invention has been described with reference to preferred embodiments. However, those skilled in the art will understand that many other embodiments other than those specifically described also fall within the spirit and scope of the invention.

According to the present invention, by extracting the movement data of the eyeball from the face image photographed through the camera unit, and scrolling the data displayed on the screen according to the extracted movement data, the user can view the data displayed on the screen of the terminal When you want to scroll, you can see the data reflecting the movement of the eye by simply moving the eye without directly operating the preset key.

In addition, since the user can read various documents sequentially through the mobile communication terminal as the screen information is changed according to the speed of his eyes, the user can read the data of the document with the screen optimized for his reading speed.

In addition, even when the user is a disabled person who cannot operate the keypad of the mobile communication terminal, various data can be viewed only by the movement of the gaze.

In addition, even when all the data to be displayed on the screen of the mobile communication terminal are not displayed, the user may move his / her eyes to see the undisplayed portion of the data.

Claims (8)

A memory for storing at least one data to be displayed on the screen; A display unit providing a screen for displaying arbitrary data; A camera unit for photographing human faces, The eyeball motion data is extracted from the face image photographed through the camera, and the eyeball movement direction is recognized in the up, down, left, and right directions among the extracted motion data and scrolled in the recognized direction to be displayed on the screen. Mobile communication terminal with an eye detection function characterized in that it comprises a control unit for displaying the data that was not visible. The method of claim 1, wherein the data displayed on the screen A mobile communication terminal having an eye detection function comprising at least one of a menu item, text data, image data, and image data. The method of claim 1, wherein the motion data A mobile communication terminal equipped with an eye detection function, characterized in that it further comprises speed data on the movement of the user's eye. The method of claim 1, wherein the control unit An eyeball image processing unit for acquiring an eyeball image by identifying the eyeball from the face image photographed through the camera unit; A motion data extractor for analyzing the acquired eyeball image to extract motion data of the eyeball; And a screen processing unit for scrolling the data displayed on the screen based on the extracted motion data. Taking a picture of a person's face, Extracting eye movement data from the photographed face image; Recognizing a movement direction of the eyeball in the up, down, left, and right directions among the extracted motion data; And scrolling in the recognized direction to display data that is not visible on the screen. The method of claim 5, wherein the data displayed on the screen And at least one of a menu item, text data, image data, and image data. The method of claim 5, wherein the motion data And a velocity data of eye movement. The method of claim 5, wherein the displaying step Determining a direction and speed to scroll based on the motion data; And scrolling data displayed on the screen in the determined direction and speed.

Images