KR20160002589A - Method for Displaying Information and Displaying Device Thereof - Google Patents

Abstract

The present invention relates to an information display method. The information display method is used by a display device to employ user gaze tracking. The method includes the following steps: tracking the gaze of a user in an image when the user is sensed; displaying multiple objects, corresponding to multiple home appliances, connected to the display device through a home network, on a screen; extracting a specific object matched to the tracked user gaze among the objects being displayed on the screen; and displaying detailed information of the specific object extracted from the objects.

Description

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to an information display method,

The present invention relates to an information display method and a display apparatus therefor, and more particularly, to an information display method and a display apparatus therefor, which extract an object matched with a direction of a user's gaze through user's gaze tracing and extract detailed information or additional information And a display device therefor.

Recently, with the development of data communication technology, a home automation system (Home Automation System) has appeared that can remotely control home appliances connected through a home network as a home network function is developed based on the Internet .

The home network function means a function for connecting and remotely controlling various household appliances existing in a household through the Internet network, and is indispensably required for home automation.

In general, when the resident of the household is out of the home network and is located outside, the home network function accesses the Internet using the mobile terminal, inquires the status of various home appliances in the home, and performs a simple control operation .

However, even if a resident is located within a household, there is an inconvenience that the household appliances must be searched and controlled using a small mobile terminal.

SUMMARY OF THE INVENTION Accordingly, the present invention has been made keeping in mind the above problems occurring in the prior art, and it is an object of the present invention to provide an information display method of displaying relevant information of an object matched to a user's eyes, And a display device.

Another object of the present invention is to provide an information display method and a display device therefor, which display differentiated information about an object whose user's gaze is matched with an object whose user's gaze does not match through user's gaze tracking.

According to another aspect of the present invention, there is provided an information display method using a user's gaze tracking of a display device, the method comprising: tracking a user's gaze from a user's image when the user senses; Displaying a plurality of objects corresponding to a plurality of home appliances connected to the display device and a home network on a screen, and displaying a plurality of objects corresponding to the plurality of home appliances connected to the home network on a screen, Extracting an object, and displaying detailed information related to the extracted specific object among the plurality of objects.

As described above, according to the information display method and the display device therefor according to the embodiment of the present invention, related information of an object matched through the user's gaze tracking is displayed on the transparent display device in real time, It is possible to provide desired information.

In addition, according to an information display method and a display device therefor according to an embodiment of the present invention, by displaying detailed information or additional information on an object matched with a user's gaze in real time, detailed information and utilization information of the object can be displayed in real time As shown in FIG.

FIG. 1 is a block diagram schematically illustrating the configuration of an information display apparatus through user's gaze tracking according to an embodiment of the present invention. Referring to FIG.
2 is a flowchart schematically illustrating a method of displaying information through user's gaze tracking of a display device according to an embodiment of the present invention.
3 is a flowchart schematically illustrating a method of providing an additional function using user identification of a display device according to an embodiment of the present invention.
FIG. 4 is a view for explaining an information display method using eye tracking of a display device according to an embodiment of the present invention.
FIGS. 5A and 5B are views for explaining a method of displaying information according to user's gaze movement of a display device according to an embodiment of the present invention.
FIG. 6 is a diagram illustrating an operation method according to a gesture of a display apparatus according to an exemplary embodiment of the present invention. Referring to FIG.
7 is an exemplary view illustrating a service that can be provided when a display device according to an embodiment of the present invention is applied to a kitchen.

Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings. In the drawings, the same reference numerals are used to designate the same or similar components throughout the drawings. In the following description of the present invention, a detailed description of known functions and configurations incorporated herein will be omitted when it may make the subject matter of the present invention rather unclear.

Various aspects of the invention are described below. The inventions set forth herein may be implemented in a wide variety of forms, and it is to be understood that any particular structure, function, or all of the presented features are illustrative only. It will be apparent to those skilled in the art, based on the inventions set forth herein, that one aspect set forth herein may be implemented independently of the other, and that such aspects may be implemented in various ways Lt; / RTI > For example, an apparatus may be implemented or a method may be practiced using any number of aspects set forth herein. In addition, or in addition to one or more aspects described herein, or with the aid of structures, functions, or structures and functions other than these aspects, such devices may be implemented or such methods may be practiced.

Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings.

FIG. 1 is a block diagram schematically illustrating the configuration of an information display apparatus through user's gaze tracking according to an embodiment of the present invention. Referring to FIG.

1, a display device according to an embodiment of the present invention includes an image receiving unit 110, a gaze tracking unit 120, a user identifying unit 130, a control unit 140, a storage unit 150, And a display unit 160.

The image receiving unit 110 may acquire a peripheral image of a predetermined area, and the peripheral image may include a moving image. In this case, the peripheral image received through the image receiving unit 110 includes a user image, and may be an image obtained only when the user senses the user in real time. The image receiving unit 110 may be implemented as a camera and may be provided in the display device, and a plurality of image receiving units 110 may be provided to improve accuracy.

The eye tracking unit 120 extracts a user's eye movement by analyzing the user's image received through the image receiving unit 110 and detects the eye movement of the user based on the extracted eye movement of the user, And can be implemented through an Eye Tracking technique.

The user identification unit 130 may be implemented through a facial recognition program that analyzes a user image received through the image receiving unit 110 and recognizes and identifies a user's face.

The control unit 140 may control the display unit to display a plurality of objects connected to the home network on a screen. The control unit 140 may extract a specific object matching the user's gaze tracked through the gaze tracking unit 120 among a plurality of objects displayed on the screen. Then, the control unit 140 may control to display detailed information related to the extracted specific object among the plurality of objects. At this time, the extracted detailed information related to the specific object may include device information of the household appliance corresponding to the extracted specific object, device usage history, device usage description, and A / S information. In addition, the control unit 140 may determine, for at least one of the plurality of objects, at least one object other than the extracted specific object, current state information of the home appliances corresponding to the at least one objects Or an on / off state, etc.).

Meanwhile, when a specific user is identified through the user identification unit 130, the control unit 140 may control to accumulate / store the usage history of the specific user. In addition, the controller 140 extracts the history of the previously stored home appliance use history from the storage unit 150 at the request of the user, and stores the customized supplementary service personalized to the specific user on the basis of the extracted home appliance usage history . The controller 140 may control the operation of a plurality of objects connected through the home network.

The storage unit 150 may accumulate / store usage histories using specific objects for each user. For example, the specific object includes specific home appliances connected through a home network, and the storage unit 150 may store basic appliance information, device usage description, device A / S information of the specific home appliance, User history can be stored using home appliances. At this time, the storage unit 150 may accumulate / store user histories using the home appliances on a user-by-user basis.

The display unit 160 may provide a display screen under the control of the controller 140. For example, the display unit 160 may be implemented as a transparent flexible display.

As described above, the display apparatus according to an embodiment of the present invention can provide information related to specific objects that the user desires to obtain in real time without special input through user's eye tracking.

2 is a flowchart schematically illustrating a method of displaying information through user's gaze tracking of a display device according to an embodiment of the present invention.

As shown in FIG. 2, the display device according to an exemplary embodiment of the present invention can provide information desired by the user in real time without special selection or input through user's eye tracking.

First, the user can be detected from the user image and the user's eyes can be tracked within the display screen (S210). The user's gaze may be tracked based on the extracted content by extracting the user's eye or pupil movement from the user's image. In addition, the user's gaze may be achieved through a separate eye tracking technique.

Then, a plurality of objects connected to the home network can be extracted and displayed on the screen (S220). The display device may display a plurality of objects corresponding to a plurality of home appliances connected to the display device through a home network on a screen. That is, the display device is a transparent display device, which displays a plurality of home appliances located behind the transparent display device in such a manner as to be displayed in a transparent manner, wherein objects corresponding to the respective home appliances transmitted through the transparent display device . ≪ / RTI >

Then, a specific object matching the traced user's gaze may be extracted from among a plurality of objects displayed on the screen, and detailed information related to the extracted specific object may be displayed (S230, S240, S250). That is, the specific appliance matching the user's gaze can be recognized and extracted by the user, and detailed information related to the extracted specific appliance can be displayed on the screen. At this time, the detailed information may include device information of the home appliance, history of use of the appliance, description of the use of the device, and device A / S information.

On the other hand, for household appliances that are not matched to the user's eyes and are not extracted (i.e., not extracted because they are not recognized as selected by the user), only brief information such as status information indicating the current status such as on / [S260].

3 is a flowchart schematically illustrating a method of providing an additional function using user identification of a display device according to an embodiment of the present invention.

3, when a user identifies a specific user and wants to use a specific home appliance, the display device according to an exemplary embodiment of the present invention displays a list of frequently used It is possible to provide more functions.

First, a user can be detected and identified from the user image (S310). The user may be identified through a facial recognition program that recognizes and identifies a user's face through user image analysis. For example, if a user is detected from an image obtained through a predetermined camera, the face image of the user can be extracted from the image. Then, it is possible to identify a specific user depending on whether the extracted face image is stored in the database. At this time, if the face image of the specific user is not stored in the database, the user can be registered as a new user.

The device usage history may be accumulated and stored for the identified user (S320). The device usage history may include a predetermined number of recent usage histories of home appliances corresponding to a specific object extracted by the user's gaze tracking. If any one of the recent usage histories is selected, the device can be controlled to be in an operating state corresponding to the selected usage history. At this time, the display device is connected to a corresponding device through a home network, and can transmit a control signal for activating the device through the home network. In addition, the device use history may include a plurality of per-user usage histories of household appliances corresponding to specific objects extracted by the user's gaze tracking. Accordingly, when one user among the plurality of users is detected, the controller can transmit a control signal to the home appliance so as to be in an operating state corresponding to the detected use history of the user. At this time, the user detection method may include a user identification through face recognition, a touch or hand shape of a user (e.g., a number of fingers recognized when the user is designated several times as 1, 2, 3, 4, In other words, when a specific user who has been stored (or registered) in the database uses a predetermined home appliance, the usage history information can be accumulated and stored. For example, the accumulated / stored The use history information may include temperature and time information previously used for cooking using the oven when the household appliance is an oven and may include usage mode information of the washing machine when the household appliance is a washing machine have.

Thereafter, when a history of usage history of a specific user is extracted by the user identification, the home appliance can be activated by transmitting a control signal for setting a preference function frequently used by the specific user to the home appliance [S330 , S340, S350]. For example, in the case of a shared kitchen used by multiple users, the display device may recognize that the oven has been selected through the eye tracking of a particular user and may provide detailed information about the oven through the screen. Then, the display device recognizes the specific user's face, identifies a specific user, and provides the oven usage history stored on the screen to the identified specific user. When the oven is connected to the display device through the home network, the display device controls the home appliance to set a preference function frequently used by the specific user based on the home appliance use history of the specific user The home appliances can be activated by automatically transmitting signals.

As described above, according to the present invention, it is possible to provide home appliance related information that a user desires to obtain in real time without additional input through user's eye tracking, and can identify a specific user, The home networking / home automation system can be provided by automatically setting home appliances based on the history used in the home networking / home automation system.

4 is a view for explaining a method of displaying information using user's gaze tracking of a display device according to an embodiment of the present invention.

As shown in FIG. 4, the display device according to an embodiment of the present invention tracks the user's gaze and displays detailed information of a specific object (for example, a variety of household appliances or the like) Can be displayed.

The display device 420 may be installed between the plurality of objects 430 and 440 and the user 410. Accordingly, the display device 420 tracks the gaze of the user 410 through a plurality of gaze tracking modules 421, 422, and 423 mounted on a predetermined area, It is possible to recognize a specific object 430 to be selected. That is, the display device 420 selects a specific object 430 based on the gaze direction of the user 410, which is recognized by the gaze tracking module 421, 422, 423, And the user's gaze direction may be determined based on the face direction of the user and the pupil position.

Therefore, the information about the specific object 430 located on the rear surface of the display device 420 is displayed on the transparent material display device 420, but the user can easily recognize that the information is displayed on a specific object (located on the rear side of the display device) 430). ≪ / RTI >

In the case of a specific object 430 matching the tracked direction of the user 410, the display device 420 may display the latest detailed information related to the specific object 430. At this time, in the case of the other object 440 that does not match the direction of the user 410, only the current status information can be displayed.

FIGS. 5A and 5B are diagrams for explaining a method of displaying information according to a movement of a user's gaze of a display device according to an embodiment of the present invention.

5A, the display device 520 can display related information such as a refrigerator 530, an oven 540, and a microwave oven 550 located behind the display device 520 have. The display device 520 is located between the user 510 and various home appliances and may include a specific module 521 for tracking the eyes of the user 510 in a predetermined area. In other words, if the user 510 selects one of the refrigerator 530 (for example, refrigerator 530, oven 540, and microwave oven 550) through various display devices 520 ). Then, the display device 520 tracks the gaze of the user 510 using the specific module 521 for tracking the gaze and finds that the gaze of the user 510 is matched to the refrigerator 530 Detailed information 531 related to the refrigerator 530 can be displayed on the screen. At this time, the detailed information 531 may include device information of the device, device information use history, device usage description information, A / S information, and the like. On the other hand, the display device 520 may display only state information 541 and 551 indicating current state such as availability or on / off for other household appliances other than the refrigerator 530. [ The display device 520 can receive information related to various home appliances in real time through a wired or wireless home network.

5B, when the gaze of the user 510 moves to another home appliance (for example, the oven 540), the display device 520 is selected by the gaze of the user It is possible to display detailed information 541 mapped to the oven 540. At this time, the other home appliances 530 and 550 which do not match the gaze of the user 510, It is possible to display only the data 531 and 551 on the screen. Various kinds of information mapped to the respective home appliances can be searched and received in real time based on a home network.

At this time, the specific home appliances recognized by the user's eye-gaze tracking can correspond to a specific object (a screen representing information) displayed on the screen of the display device 520 in a one-to-one correspondence, If it is detected that the user's gaze is moved to another object in a state in which detailed information related to a specific object extracted (or recognized) by the user is displayed, the display device displays Detailed information can be displayed. That is, when detailed information related to another object is displayed, detailed information related to a specific object that has been displayed previously may be displayed on the screen only as status information indicating a brief current status.

As described above, the display apparatus according to an embodiment of the present invention can select and provide information of an object selected according to a user's gaze movement through real-time tracking without user input, And information that is not so different from each other.

FIG. 6 is a diagram illustrating an operation method according to a gesture of a display apparatus according to an exemplary embodiment of the present invention. Referring to FIG. The display device 620 and various home appliances 630, 640 and 650 shown in FIG. 6 may be connected in a home network manner.

As shown in FIG. 6, a display device according to an exemplary embodiment of the present invention may perform a specific operation according to a gesture input by a user. Particularly, if the microwave oven 650 is selected by eye tracking of the user 610, the display device 620 can display the detailed information 651 by extracting a specific object corresponding to the microwave oven 650 . At this time, if the predetermined user gesture input is detected while the detailed information 651 is being displayed through the display device 620, the display device 620 may terminate the display operation. The user gesture sensed herein may include a touch operation, or may include predetermined hand shapes and operations for terminating the display device 620, and the like.

Meanwhile, when the predetermined first user gesture is detected while the detailed information 651 related to the specific object extracted by the user 610 is tracked through the display device 620, the microwave oven 650 ) To transmit the activation signal and wait for the corresponding home appliance to use. Here, the first user gesture may be a predetermined user's hand shape for activating (for example, " ON ") a household appliance corresponding to a specific object.

When the second user gesture is detected in a state where the microwave oven 650 is activated by the activation signal, the deactivation signal may be transmitted to the microwave oven 650. Here, the second user gesture may be a predetermined user's hand shape for deactivating (for example, "OFF" or "standby") the household appliance corresponding to the specific object. However, the first user gesture and the second user gesture described above are not limited to a hand shape, and may be a motion using the body or a touch by a hand.

7 is an exemplary view illustrating a service that can be provided when a display device according to an exemplary embodiment of the present invention is applied to a kitchen.

As shown in FIG. 7, the display device according to an exemplary embodiment of the present invention can be applied to a kitchen and provide services such as kitchen appliance management and recipe provision.

First, as shown in FIG. 7, the user can activate the display device (S710). For example, the display device may be implemented as a transparent display. The user may activate the display device by operating a button 711 formed on the table through a gesture input (or a touch input).

Then, the display screen of the display device can be unfolded [S720]. At this time, the display device is a flexible display device 721 bent or bent freely and can be stored in a specific storage space in a state of being rolled up by a small pulley 722, for example .

Information related to various home appliances located in the kitchen can be displayed through the display device [S730]. At this time, information 732, 733, and 734 displayed through the display device 721 may include related information related to home appliances located on the back of the display device. Accordingly, the display device 721 can track the user's gaze and display related information related to the home appliance that the user is watching. At this time, the display device 721 tracks the eyes of the user to display detailed information 732 for a specific object matched with the user's gaze, and only brief status information 733 and 734 for other objects . For example, when the user watches the refrigerator, the display device 721 tracks the user's gaze and displays relevant information related to the refrigerator (e.g., an internal temperature, an antibacterial condition, and a refrigerator item list) It can be displayed as if it is being displayed.

In addition, the display device 721 may further display the home appliance usage information through the home network (S740, S750). For example, if the home appliance is a refrigerator, the display device 721 may provide a list of ingredients stored in the refrigerator to the user. The list of ingredients may be stored in advance in the refrigerator, Lt; / RTI > When a specific food ingredient 752 is selected from the list of ingredients through user input (e.g., gesture input, etc.), the display device accesses the Internet through the connected home network, Method 762, and the like.

Although the present invention has been fully described in connection with the preferred embodiments thereof with reference to the accompanying drawings, it is to be understood that the invention is not limited to the disclosed embodiments, but, on the contrary, And various modifications and changes may be made thereto without departing from the scope of the present invention.

Accordingly, the spirit of the present invention should not be construed as being limited to the embodiments described, and all of the equivalents or equivalents of the claims, as well as the following claims, belong to the scope of the present invention .

110: image receiving unit 120:
130: user identification unit 140:
150: storage unit 160: display unit

Claims (20)

A method of displaying information using user's gaze tracking of a display device,
Tracking the user's gaze from the user's image when the user is sensed;
Displaying on the screen a plurality of objects corresponding to a plurality of home appliances connected to the display device through a home network;
Extracting a specific object matching the traced user's gaze among a plurality of objects displayed on the screen; And
And displaying detailed information related to the extracted specific object among the plurality of objects.
The method according to claim 1,
The detailed information related to the extracted specific object is,
Wherein the information includes at least one of device information of a household appliance corresponding to the extracted specific object, device usage history, device usage description, and A / S information.

The method according to claim 1,
Wherein the step of displaying detailed information related to the extracted specific object comprises:
Wherein the current state information of the household appliances corresponding to the at least one object is displayed for at least one of the plurality of objects except for the specific object.
The method according to claim 1,
Displaying detailed information related to another object matched to the moved user's gaze among the plurality of objects when it is detected that the user's gaze is moved in a state where detailed information related to the extracted specific object is displayed; Further comprising the steps of:
The method according to claim 1,
And terminating a display operation of the display device when a predetermined user gesture is detected in a state in which detailed information related to the extracted specific object is displayed.
The method according to claim 1,
And transmitting an activation signal to a home appliance corresponding to the extracted specific object when the predetermined first user gesture is detected in a state in which detailed information related to the extracted specific object is displayed, .
The method according to claim 6,
And transmitting an inactivation signal to the home appliance when a predetermined second user gesture is detected while the home appliance is activated by the activation signal.
3. The method of claim 2,
The device usage history includes:
A predetermined number of recent usage histories of household appliances corresponding to the extracted specific object,
And transmitting a control signal to the home appliance so that the operating state corresponding to the selected usage history is selected when any one of the recent usage histories is selected.
3. The method of claim 2,
The device usage history includes:
And a plurality of usage histories of the plurality of users for the household appliances corresponding to the extracted specific objects,
And transmitting a control signal to the home appliance so that an operating state corresponding to the detected use history of the user is detected when any one of the plurality of users is detected.
1. A display device for displaying information using user's gaze tracking,
A gaze tracking unit for tracking a user's gaze from a user image when the user is sensed;
A control unit configured to control the display unit to display a plurality of objects corresponding to a plurality of home appliances connected to the home network on a screen and to display a specific object matching the traced user's gaze among a plurality of objects displayed on the screen A control unit for controlling the display unit to display detailed information related to the extracted specific object among the plurality of objects;
A display unit for providing a screen under the control of the control unit;
And a display device.
11. The method of claim 10,
The detailed information related to the extracted specific object is,
Wherein the information includes at least one of device information of the household appliance corresponding to the extracted specific object, device usage history, device usage description, and A / S information.
11. The method of claim 10,
The control unit
And displays the current state information of the household appliances corresponding to the at least one object with respect to at least one of the plurality of objects except for the specific object.
11. The method of claim 10,
The control unit
And displaying detailed information related to another object matched to the moved user's gaze among the plurality of objects when it is detected that the user's gaze is moved in a state in which detailed information related to the extracted specific object is displayed, / RTI >
11. The method of claim 10,
The control unit
Wherein when the predetermined user gesture is detected, detailed information related to the extracted specific object is displayed, the display operation of the display device is terminated.
11. The method of claim 10,
The control unit
Wherein when the first user gesture is detected, detailed information related to the extracted specific object is displayed, the display device transmits an activation signal to a home appliance corresponding to the extracted specific object.
16. The method of claim 15,
The control unit
And transmits a deactivation signal to the home appliance when a predetermined second user gesture is detected while the home appliance is activated by the activation signal.
12. The method of claim 11,
Wherein the device usage history includes a set number of recent usage histories for the household appliances corresponding to the extracted specific objects,
Wherein the controller transmits a control signal to the home appliances so that the operating state corresponding to the selected usage history is selected when any one of the recent usage histories is selected.
12. The method of claim 11,
Wherein the device usage history includes a plurality of usage histories of the plurality of users for the household appliances corresponding to the extracted specific objects,
Wherein the control unit transmits a control signal to the home appliance so that when any one of the plurality of users is detected, the operating state corresponding to the detected use history of the user is transmitted.
A computer readable medium comprising computer executable instructions for executing an application that displays information using user gaze tracking,
Instructions for tracking a user's gaze from a user image when the user is sensed;
Instructions for displaying on the screen a plurality of objects corresponding to a plurality of home appliances connected to the display device by a home network;
Instructions for extracting a specific object from among a plurality of objects displayed on the screen matching the traced user's gaze; And
And displaying detailed information related to the extracted specific object among the plurality of objects.
An application stored in a medium for displaying information using user gaze tracking,
Instructions for tracking a user's gaze from a user image when the user is sensed;
Instructions for displaying on the screen a plurality of objects corresponding to a plurality of home appliances connected to the display device by a home network;
Instructions for extracting a specific object from among a plurality of objects displayed on the screen matching the traced user's gaze; And
And instructions for displaying detailed information related to the extracted specific object among the plurality of objects.

Images