KR102358921B1 - Method for Displaying Information and Displaying Device Thereof - Google Patents

Abstract

The present invention provides a display apparatus and method for displaying in real time related information of a matched object through a user's gaze tracking, and displaying detailed information or additional information on an object to which the user's gaze matches in real time.

Description

Method for Displaying Information and Displaying Device Thereof

The present invention relates to a method for displaying information and a display device therefor, and more particularly, extracting an object matching a user's gaze direction through user gaze tracking, and extracting the extracted object and detailed information of the extracted object or The present invention relates to an information display method for displaying additional information and a display device therefor.

Recently, as the home network function is developed based on the Internet in conjunction with the development of data communication technology, a home automation system capable of remotely controlling home appliances connected through the home network has appeared. are doing

The home network function refers to a function capable of connecting and remote controlling various home appliances existing in a household through an Internet network, and is essential for home automation.

In general, when a resident of a household goes out and is located outside, such a home network function connects to the Internet network using a mobile terminal, inquires the status of various home appliances in their house, and performs a simple control operation. has been implemented in the form

However, even when a resident is located in a household, there is an inconvenience of having to search and control home appliances one by one using a mobile terminal having a small screen.

Accordingly, the present invention has been devised to solve the conventional problems as described above, and an object of the present invention is to track the user's gaze and display the related information of the object matching the user's gaze in real time, and an information display method thereof. To provide a display device for

Another object of the present invention is to provide an information display method for displaying differentiated information on an object with which a user's gaze matches and an object with which a user's gaze does not match through user gaze tracking, and a display apparatus therefor.

An information display method according to an embodiment of the present invention for achieving the above object, in the information display method using a user gaze tracking of a display device, when a user is detected, tracking the user gaze from the user image; And , displaying a plurality of objects corresponding to the plurality of home appliances connected to the display device through a home network on a screen; Extracting an object and displaying detailed information related to the extracted specific object from among the plurality of objects;

As described above, according to the information display method and the display device therefor according to an embodiment of the present invention, by real-time display of related information of an object matched through user gaze tracking on a transparent display device, the user can It has the effect of being able to provide the desired information.

In addition, according to the information display method and the display apparatus therefor according to an embodiment of the present invention, detailed information or additional information is displayed on an object with which a user's gaze matches in real time, so that detailed information and utilization information of the corresponding object are displayed in real time. It has the effect of being able to provide

1 is a block diagram schematically illustrating a configuration of an information display apparatus through user gaze tracking according to an embodiment of the present invention.
2 is a flowchart schematically illustrating a method of displaying information through a user's gaze tracking of a display apparatus according to an embodiment of the present invention.
3 is a flowchart schematically illustrating a method of providing an additional function using user identification of a display device according to an embodiment of the present invention.
4 is a diagram illustrating an information display method using eye tracking of a display apparatus according to an embodiment of the present invention.
5A to 5B are diagrams for explaining a method of displaying information according to a movement of a user's gaze of a display apparatus according to an embodiment of the present invention.
6 is an exemplary diagram illustrating an operation method according to a gesture of a display apparatus according to an embodiment of the present invention.
7 is an exemplary diagram illustrating a service that can be provided when the display device according to an embodiment of the present invention is applied to a kitchen.

Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings. First of all, in adding reference numerals to the components of each drawing, it should be noted that the same components are given the same reference numerals as much as possible even though they are indicated on different drawings. In addition, in describing the present invention, if it is determined that a detailed description of a related known configuration or function may obscure the gist of the present invention, the detailed description thereof will be omitted.

Various aspects of the invention are described below. It is to be understood that the inventions presented herein may be embodied in a wide variety of forms, and that any specific structure, function, or both presented herein is exemplary only. Based on the inventions presented herein, those of ordinary skill in the art to which the present invention pertains will realize that one aspect presented herein may be implemented independently of any other aspects, and two or more of these aspects may vary in various ways. It will be understood that they can be combined with For example, an apparatus may be implemented or a method may be practiced using any number of the aspects set forth herein. Further, such an apparatus may be implemented or such a method may be practiced using other structure, function, or structure and functionality in addition to or other than one or more aspects described herein.

Hereinafter, preferred embodiments according to the present invention will be described in detail with reference to the accompanying drawings.

1 is a block diagram schematically illustrating a configuration of a display apparatus for displaying information using user eye tracking according to an embodiment of the present invention.

As shown in FIG. 1 , the display device according to an embodiment of the present invention includes an image receiving unit 110 , an eye tracking unit 120 , a user identification unit 130 , a control unit 140 , a storage unit 150 , and The display unit 160 may be included.

The image receiver 110 may acquire an image of a surrounding area of a predetermined area, and the surrounding image may include a moving picture. In this case, the surrounding image received through the image receiving unit 110 includes a user image, and may be an image acquired only when the user is sensed by real-time sensing of the user. The image receiver 110 may be implemented in the form of a camera and may be provided in the display device, and may be provided in plurality to improve accuracy.

The eye tracking unit 120 extracts the user's eye movement by analyzing the user image received through the image receiving unit 110, and a specific gaze tracking the user's eye based on the extracted user's eye movement It may be implemented through an eye tracking technique.

The user identification unit 130 may be implemented through a predetermined facial recognition program for recognizing and identifying the user's face by analyzing the user image received through the image receiving unit 110 .

The controller 140 may control to display a plurality of objects connected to the display device through a home network on a screen. In addition, the controller 140 may extract a specific object matching the user's gaze tracked through the gaze tracking unit 120 from among a plurality of objects displayed on the screen. Then, the controller 140 may control to display detailed information related to the extracted specific object among the plurality of objects. In this case, the detailed information related to the extracted specific object may include device information, device usage history, device usage description and A/S information of the home appliance corresponding to the extracted specific object. In addition, the control unit 140, for at least one object other than the extracted specific object among the plurality of objects, current state information (eg, use of the home appliance) corresponding to the at least one object Indicated or on/off status, etc.) may be controlled to be displayed.

Meanwhile, when a specific user is identified through the user identification unit 130 , the control unit 140 may control to accumulate/store the home appliance usage history of the specific user. In addition, the control unit 140 extracts the home appliance use history stored in advance from the storage unit 150 in response to a user's request, and provides a personalized additional service to the specific user based on the extracted home appliance use history. can be controlled to provide. In addition, the controller 140 may control operations of a plurality of objects connected through the home network.

The storage unit 150 may accumulate/store a history of using a specific object for each user. For example, the specific object includes a specific home appliance connected through a home network. It is possible to store the user history of using the home appliance. In this case, the storage unit 150 may accumulate/store user histories of using the home appliance for each user.

The display unit 160 may provide a display screen under the control of the controller 140 . For example, the display unit 160 may be implemented in the form of a transparent flexible display.

As described above, the display device according to an embodiment of the present invention may provide related information on a specific object that the user wants to obtain in real time through user gaze tracking without a special input.

2 is a flowchart schematically illustrating a method of displaying information through a user's gaze tracking of a display apparatus according to an embodiment of the present invention.

As shown in FIG. 2 , the display device according to an embodiment of the present invention may provide information desired by a user in real time through user gaze tracking without a special selection or input.

First, by detecting the user from the user image, it is possible to track the user's gaze within the display screen [S210]. The user's gaze may be tracked based on the extracted content by extracting the user's eye or pupil movement from the user's image. In addition, the user's gaze may be made through a separate eye tracking technique.

In addition, a plurality of objects connected to the home network may be extracted and displayed on the screen [S220]. The display device may display a plurality of objects corresponding to a plurality of home appliances connected to the display device through a home network on a screen. That is, the display device is a transparent display device, and displays a plurality of home appliances positioned at the rear of the transparent display device through a display method, and an object corresponding to each home appliance transmitted through the transparent display device can recognize them

Then, it is possible to extract a specific object matching the tracked user's gaze from among the plurality of objects displayed on the screen, and display detailed information related to the extracted specific object [S230, S240, S250]. That is, a specific home appliance matching the user's gaze may be recognized as selected by the user and extracted, and detailed information related to the extracted specific home appliance may be displayed on the screen. In this case, the detailed information may include device information of the corresponding home appliance, device usage history, device usage description, device A/S information, and the like.

On the other hand, for home appliances that are not extracted because they do not match the user's gaze (that is, are not extracted because they are not recognized as selected by the user), brief information such as status information indicating whether to use or a current state such as on/off can only be displayed [S260].

3 is a flowchart schematically illustrating a method of providing an additional function using user identification of a display device according to an embodiment of the present invention.

As shown in FIG. 3 , the information display apparatus according to an embodiment of the present invention identifies a specific user and, when the identified user wants to use a specific home appliance, the specific user frequently You can provide more functions to use.

First, it is possible to detect and identify the user from the user image [S310]. The user may be identified through a predetermined facial recognition program that recognizes and identifies the user's face through user image analysis. For example, when a user is detected from an image obtained through a predetermined camera, a face image of the user may be extracted from the image. Then, it is possible to identify a specific user according to whether the extracted face image is pre-stored in the database. In this case, if the face image of the specific user is not previously stored in the database, it can be registered and used as a new user.

In addition, it is possible to accumulate and store a device use history for the identified user [S320]. The device use history may include a preset number of recent use histories for home appliances corresponding to a specific object extracted by user eye tracking. In addition, when any one of the recent use histories is selected, the device may be controlled to enter an operating state corresponding to the selected use histories. In this case, the display device is connected to the corresponding device through the home network, and may transmit a control signal for activating the corresponding device through the home network. In addition, the device use history may include a plurality of user-specific usage histories for the home appliance corresponding to a specific object extracted by the user's gaze tracking. Accordingly, when one user is detected from among the plurality of users, a control signal may be transmitted to the home appliance so as to be in an operating state corresponding to the sensed user's usage history. At this time, the user detection method is based on user identification through face recognition, the user's touch or hand shape (eg, the number of recognized fingers when the user is designated as 1, 2, 3, 4, etc. several times). In other words, when a specific user pre-stored (or registered) in the database uses a predetermined home appliance, the use history information can be accumulated and stored. The usage history information may include temperature and time information used before cooking using the oven when the home appliance is an oven, and may include information on a usage mode of the washing machine when the home appliance is a washing machine. have.

Thereafter, when a specific user's use of a corresponding home appliance is extracted by user identification, a control signal for setting a preferred function frequently used by the specific user may be transmitted to the corresponding home appliance to activate the corresponding home appliance [S330] , S340, S350]. For example, in the case of a communal kitchen used by multiple users, the display device may recognize that an oven is selected through eye tracking of a specific user and provide detailed information about the oven through a screen. Then, the display device may identify the specific user by recognizing the face of the specific user, and provide a pre-stored oven use history for the identified specific user on the screen. And when the oven is connected to the display device through a home network, the display device controls the display device to set a preferred function frequently used by the specific user based on the home appliance use history of the specific user. By automatically transmitting a signal, the corresponding home appliance can be activated.

As described above, according to the present invention, it is possible to provide home appliance-related information that a user wants to obtain in real time without a separate input through user eye tracking, and by identifying a specific user, the identified specific user frequently uses or transfers It has the advantage of being able to provide a state-of-the-art, automated home networking/home automation system by automatically setting home appliances based on the history of using it.

4 is a diagram illustrating a method for displaying information using user eye tracking of a display device according to an embodiment of the present invention, wherein a user 410, a display device 420, and a plurality of objects 430 and 440 (Home networking capable home appliance) shows the structure as viewed from above.

4 , the display device 420 according to an embodiment of the present invention tracks the gaze of the user 410 to match a specific object (the plurality of objects) to which the gaze of the user 410 is matched. (any one of 430 and 440) may be displayed.

The display device 420 may be installed between the plurality of objects 430 and 440 and the user 410 . Accordingly, the display device 420 tracks the gaze of the user 410 through a plurality of gaze tracking modules 421 , 422 , and 423 mounted in a predetermined area, and the user from among the plurality of objects 430 and 440 . 410 may recognize a specific object 430 to be selected. That is, the display device 420 selects a specific object 430 based on the gaze direction of the user 410 recognized using the gaze tracking modules 421 , 422 , and 423 , and provides information about the specific object 430 . Detailed information may be displayed, and the user's gaze direction may be determined based on a face direction and a pupil position of the user 410 .

Accordingly, information on a specific object 430 located on the rear side of the display device 420 is displayed on the display device 420 made of a transparent material, but the user can display the information on a specific object (located on the back side of the display device) ( 430), you can feel an optical illusion as if it were directly displayed.

In addition, the display device 420 may display the latest detailed information related to the specific object 430 matching the gaze (direction) of the tracked user 410 . In this case, in the case of another object 440 that does not match the gaze (direction) of the user 410, only current state information may be displayed.

5A to 5B are diagrams for explaining a method of displaying information according to a user's gaze movement of a display apparatus according to an embodiment of the present invention.

First, as shown in FIG. 5A , the display device 520 can display related information such as a refrigerator 530 , an oven 540 , and a microwave oven 550 located at the rear of the display device 520 . have. The display device 520 is positioned between the user 510 and various home appliances, and may include a specific module 521 for tracking the user's 510 gaze in a predetermined area. In other words, the refrigerator 530 among various home appliances (eg, the refrigerator 530 , the oven 540 , the microwave oven 550 , etc.) that the user 510 passes through the display device 520 made of a transparent material. ) can be viewed. Then, the display device 520 tracks the gaze of the user 510 using the specific module 521 for tracking the gaze and finds out that the gaze of the user 510 matches the refrigerator 530 . Detailed information 531 related to the refrigerator 530 may be displayed on the screen. In this case, the detailed information 531 may include device information of the corresponding device, device information usage history, device usage description information, A/S information, and the like. On the other hand, the display device 520 may display only brief status information 541 and 551 indicating whether the home appliances other than the refrigerator 530 are available or on/off, and the like. In addition, the display device 520 may receive information related to various home appliances in real time through a wired or wireless home network.

On the other hand, as shown in FIG. 5B , when the gaze of the user 510 moves to another home appliance (eg, the oven 540), the display device 520 is selected by the gaze of the user. Detailed information 541 mapped to the oven 540 may be displayed. At this time, status information such as on/off status of other home appliances 530 and 550 that do not match the gaze of the user 510 is also displayed. Only 531 and 551 can be displayed on the screen, and various types of information mapped to the respective home appliances can be searched for and received in real time based on a home network.

In this case, the specific home appliance recognized by the eye tracking of the user 510 may correspond to a related information screen (a detailed information screen or a status information screen) displayed on the display device 520 screen 1:1. As such, when it is detected that the user's 510's gaze is moved to another object while detailed information related to the specific object extracted (or recognized) by the gaze tracking is being displayed, the display device 520 is Detailed information related to another object matched according to the moved user's gaze may be displayed. That is, when detailed information related to another object is displayed, detailed information related to a specific object that was previously displayed may be reduced or displayed only as status information indicating a brief current status on the screen.

As described above, the display device 520 according to an embodiment of the present invention can select and provide information on a specific object selected according to the gaze movement of the user 510 in real time without a separate input through user gaze tracking. This has the advantage of being able to differentiate information desired by the user 510 from information not provided by the user 510 in a differentiated manner.

6 is an exemplary diagram illustrating an operation method according to a gesture of a display device according to an embodiment of the present invention. The display device 620 and various home appliances 630 , 640 , and 650 shown in FIG. 6 may be connected in a home network manner.

As shown in FIG. 6 , the display apparatus according to an embodiment of the present invention may perform a specific operation according to a gesture input by a user. In particular, when the microwave oven 650 is selected by the user 610 eye tracking, the display device 620 may display detailed information 651 related to the microwave oven 650 . In this case, when a preset user gesture input is detected while the detailed information 651 is being displayed through the display device 620 , the display device 620 may end the display operation. The user gesture sensed here may include a touch operation, or may include a preset hand shape and operation for terminating the display device 620 .

On the other hand, when a preset first user gesture is detected while the home appliance extracted by eye tracking of the user 610, that is, detailed information 651 related to a specific object is being displayed through the display device 620, , by transmitting an activation signal to the microwave oven 650 , the microwave oven 650 may be on standby in a state for use. Here, the first user gesture may be a preset user hand shape to activate (eg, “ON”) the home appliance extracted by user eye tracking.

When the second user gesture is detected in a state in which the microwave oven 650 is activated by the activation signal, a deactivation signal may be transmitted to the microwave oven 650 . Here, the second user gesture may be a preset user hand shape to deactivate (eg, “OFF” or “standby”) the home appliance extracted by user eye tracking. However, the first user gesture and the second user gesture described above are not limited to a hand shape, and may be an action using a body or a touch by a hand. Also, it may be a user's voice rather than a hand gesture or touch.

7 is an exemplary diagram illustrating a service that can be provided by applying a display device to a kitchen according to an embodiment of the present invention.

As shown in FIG. 7 , the display device according to an embodiment of the present invention is applied and installed in a kitchen to provide services such as managing kitchen appliances and providing recipes.

First, as shown in FIG. 7 , the user may activate the display device 721 [S710]. For example, the display device 721 may be implemented in the form of a transparent display, and the user operates the button 711 formed on the table 711a through a gesture input (or touch input, etc.) to display the display device. can be activated.

Then, the display screen of the display device 721 may be unfolded [S720]. In this case, the display device 721 is a flexible display device that can be freely bent or bent. For example, it can be stored and stored in a specific storage space while being rolled up on a small pulley 722. .

In addition, information related to various home appliances located in the kitchen may be displayed through the display device 721 [S730]. In this case, the information 732 , 733 , and 734 being displayed through the display device 721 may include related information related to home appliances located on the rear side of the display device 721 . Accordingly, the display device 721 may track the user's gaze and display related information related to the home appliance that the user is looking at. In this case, the display device 721 tracks the user's gaze and displays detailed information 732 for a specific object matching the user's gaze, and only brief state information 733 and 734 for other objects. can provide For example, when the user looks at the refrigerator, the display device 721 tracks the user's gaze and displays detailed information related to the refrigerator (eg, internal temperature, antibacterial status, list of ingredients stored in the refrigerator, etc.) on the surface of the refrigerator. You can display it as if it were being displayed directly on the .

In addition, the display device 721 may further display the utilization information for the home appliance through the home network [S740, S750]. For example, when the home appliance that the user watches is a refrigerator, the display device 721 may provide a list of ingredients stored in the refrigerator to the user. It may be registered information. And when a specific ingredient 752 is selected from the ingredient list through a user input (eg, a gesture input, etc.), the display device accesses the Internet through a connected home network to cook using the selected specific ingredient. A method 762 and the like may be provided.

As described above, the present invention has been described with specific matters such as specific components and limited embodiments and drawings, but these are only provided to help a more general understanding of the present invention, and the present invention is limited by the above embodiments Various modifications and variations are possible from these descriptions by those of ordinary skill in the art to which the present invention pertains.

Therefore, the spirit of the present invention should not be limited to the described embodiments, and not only the claims described below, but also all of the claims and all equivalents or equivalent modifications to the claims will be said to belong to the scope of the spirit of the present invention. .

110: image receiving unit 120: eye tracking unit
130: user identification unit 140: control unit
150: storage unit 160: display unit

Claims (20)

In the display device for displaying information using user eye tracking,
an image receiver configured to acquire a surrounding image including the user image of a predetermined region;
a user identification unit that analyzes the user image received through the image receiving unit to recognize and identify the user's face; and
a storage unit for accumulating and storing a history of using a specific object for each user;
When a user is detected, an eye tracking unit for tracking the user's gaze from the user image;
control to display a plurality of objects corresponding to the plurality of home appliances connected to the display device through a home network on a screen, and the specific object matching the tracked user's gaze among the plurality of objects displayed on the screen by extracting and controlling to display detailed information related to the extracted specific object among the plurality of objects,
a control unit for extracting a usage history related to the extracted specific object, and controlling to further provide a personalized customized supplementary service to a specific user based on the extracted usage history; and
a display unit that provides a screen according to the control of the control unit;
consists of,
The user identification unit
Extracting a face image of a user included in the user image through a face recognition program, and comparing the extracted face image with a face image pre-stored in a database to identify a user corresponding to the extracted face image, When it appears that the extracted face image is not stored in the database through the comparison result with the face image, the user corresponding to the extracted face image is registered as a new user,
the storage unit
The new user accumulates and stores the history of using the specific object.
display device.
The method of claim 1,
Detailed information related to the extracted specific object,
Display device comprising at least one of device information, device usage history, device usage description, and A/S information of the home appliance corresponding to the extracted specific object.
The method of claim 1,
the control unit
For at least one object other than the specific object among the plurality of objects, the display device, characterized in that for displaying the current state information of the home appliance corresponding to the at least one object.
The method of claim 1,
the control unit
When it is detected that the user's gaze is moved while the detailed information related to the extracted specific object is displayed, detailed information related to another object matching the moved user's gaze among the plurality of objects is displayed. display device.
The method of claim 1,
the control unit
In a state in which detailed information related to the extracted specific object is displayed, when a preset user gesture is detected, a display operation of the display device is terminated.
The method of claim 1,
the control unit
When a preset first user gesture is detected while the detailed information related to the extracted specific object is displayed, an activation signal is transmitted to the home appliance corresponding to the extracted specific object.
7. The method of claim 6,
the control unit
In a state in which the home appliance is activated by the activation signal, when a preset second user gesture is detected, the display device transmits a deactivation signal to the home appliance.
3. The method of claim 2,
The device usage history includes a set number of recent usage histories for home appliances corresponding to the extracted specific object,
When any one of the recent use histories is selected, the control unit transmits a control signal to the home appliance to enter an operating state corresponding to the selected use histories.
3. The method of claim 2,
The device use history includes a plurality of user-specific usage histories for the home appliance corresponding to the extracted specific object,
When any one of the plurality of users is sensed, the control unit transmits a control signal to the home appliance so that an operation state corresponding to the sensed user's use history is detected.
delete In the information display method using the user gaze tracking of the display device,
obtaining a surrounding image including the user image of a predetermined area;
analyzing the acquired user image to recognize and identify the user's face;
accumulating and storing a history of using a specific object for each user;
tracking a user's gaze from the user image when a user is detected;
displaying a plurality of objects corresponding to the plurality of home appliances connected to the display device through a home network on a screen;
extracting a specific object matching the tracked user's gaze from among a plurality of objects displayed on the screen; and
displaying detailed information related to the extracted specific object among the plurality of objects; and
extracting a usage history related to the extracted specific object, and controlling to further provide a personalized customized supplementary service to a specific user based on the extracted usage history
including,
The step of recognizing and identifying the user's face
A face image of the user included in the user image is extracted through a face recognition program, and a user corresponding to the extracted face image is identified by comparing the extracted face image with a face image pre-stored in a database, If it appears that the extracted face image is not stored in the database through the comparison result with the face image, the user corresponding to the extracted face image is registered as a new user,
The step of accumulating and storing for each user is
The new user accumulates and stores the history of using the specific object.
How to display information.
A computer-readable recording medium in which a program for performing the method of claim 11 is recorded.
A computer program stored in a computer-readable recording medium for performing the method of claim 11 .
delete delete delete delete delete delete delete

Images