US20170060370A1 - Display apparatus and control method thereof - Google Patents
Display apparatus and control method thereof Download PDFInfo
- Publication number
- US20170060370A1 US20170060370A1 US15/348,061 US201615348061A US2017060370A1 US 20170060370 A1 US20170060370 A1 US 20170060370A1 US 201615348061 A US201615348061 A US 201615348061A US 2017060370 A1 US2017060370 A1 US 2017060370A1
- Authority
- US
- United States
- Prior art keywords
- user
- recognition
- recognized
- gaze
- unit
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Abandoned
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
- G06F3/048—Interaction techniques based on graphical user interfaces [GUI]
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
- G06F3/048—Interaction techniques based on graphical user interfaces [GUI]
- G06F3/0484—Interaction techniques based on graphical user interfaces [GUI] for the control of specific functions or operations, e.g. selecting or manipulating an object, an image or a displayed text element, setting a parameter value or selecting a range
-
- G—PHYSICS
- G02—OPTICS
- G02B—OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
- G02B27/00—Optical systems or apparatus not provided for by any of the groups G02B1/00 - G02B26/00, G02B30/00
- G02B27/01—Head-up displays
- G02B27/017—Head mounted
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
- G06F3/011—Arrangements for interaction with the human body, e.g. for user immersion in virtual reality
- G06F3/012—Head tracking input arrangements
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
- G06F3/011—Arrangements for interaction with the human body, e.g. for user immersion in virtual reality
- G06F3/013—Eye tracking input arrangements
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
- G06F3/017—Gesture based interaction, e.g. based on a set of recognized hand gestures
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
- G06F3/03—Arrangements for converting the position or the displacement of a member into a coded form
- G06F3/0304—Detection arrangements using opto-electronic means
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
- G06F3/048—Interaction techniques based on graphical user interfaces [GUI]
- G06F3/0481—Interaction techniques based on graphical user interfaces [GUI] based on specific properties of the displayed interaction object or a metaphor-based environment, e.g. interaction with desktop elements like windows or icons, or assisted by a cursor's changing behaviour or appearance
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
- G06F3/048—Interaction techniques based on graphical user interfaces [GUI]
- G06F3/0481—Interaction techniques based on graphical user interfaces [GUI] based on specific properties of the displayed interaction object or a metaphor-based environment, e.g. interaction with desktop elements like windows or icons, or assisted by a cursor's changing behaviour or appearance
- G06F3/04812—Interaction techniques based on cursor appearance or behaviour, e.g. being affected by the presence of displayed objects
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
- G06F3/048—Interaction techniques based on graphical user interfaces [GUI]
- G06F3/0481—Interaction techniques based on graphical user interfaces [GUI] based on specific properties of the displayed interaction object or a metaphor-based environment, e.g. interaction with desktop elements like windows or icons, or assisted by a cursor's changing behaviour or appearance
- G06F3/04815—Interaction with a metaphor-based environment or interaction object displayed as three-dimensional, e.g. changing the user viewpoint with respect to the environment or object
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/14—Digital output to display device ; Cooperation and interconnection of the display device with other functional units
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/16—Sound input; Sound output
- G06F3/167—Audio in a user interface, e.g. using voice commands for navigating, audio feedback
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
- G06V40/10—Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
- G06V40/16—Human faces, e.g. facial parts, sketches or expressions
- G06V40/179—Human faces, e.g. facial parts, sketches or expressions metadata assisted face recognition
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
- G06V40/10—Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
- G06V40/18—Eye characteristics, e.g. of the iris
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
- H04N21/41—Structure of client; Structure of client peripherals
- H04N21/422—Input-only peripherals, i.e. input devices connected to specially adapted client devices, e.g. global positioning system [GPS]
- H04N21/4227—Providing Remote input by a user located remotely from the client device, e.g. at work
-
- G—PHYSICS
- G02—OPTICS
- G02B—OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
- G02B27/00—Optical systems or apparatus not provided for by any of the groups G02B1/00 - G02B26/00, G02B30/00
- G02B27/01—Head-up displays
- G02B27/0101—Head-up displays characterised by optical features
- G02B2027/014—Head-up displays characterised by optical features comprising information/image processing systems
-
- G—PHYSICS
- G02—OPTICS
- G02B—OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
- G02B27/00—Optical systems or apparatus not provided for by any of the groups G02B1/00 - G02B26/00, G02B30/00
- G02B27/01—Head-up displays
- G02B27/0179—Display position adjusting means not related to the information to be displayed
- G02B2027/0187—Display position adjusting means not related to the information to be displayed slaved to motion of at least a part of the body of the user, e.g. head, eye
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F2203/00—Indexing scheme relating to G06F3/00 - G06F3/048
- G06F2203/048—Indexing scheme relating to G06F3/048
- G06F2203/04803—Split screen, i.e. subdividing the display area or the window area into separate subareas
Definitions
- Embodiments of the present disclosure relate to a display apparatus that improves recognition accuracy of information input by a user and a control method thereof.
- a display apparatus displays stereoscopic images.
- Such flat display devices include a liquid crystal display device (LCD), electroluminescent display device (ELD), field emission display device (FED), plasma display panel (PDP), thin film transistor liquid crystal display device (TFT-LCD), and flexible display device.
- LCD liquid crystal display device
- ELD electroluminescent display device
- FED field emission display device
- PDP plasma display panel
- TFT-LCD thin film transistor liquid crystal display device
- Such display devices include recognition technologies for improved user convenience.
- Examples of the recognition technologies include a motion recognition technology and a voice recognition technology.
- the motion recognition technology is technology for recognizing a user's motion using a motion sensor and determining a user command from the recognized motion.
- the voice recognition technology is technology for recognizing a user's voice using a voice sensor and determining a user command from the recognized voice.
- a motion that is not often performed by the user or a motion having high discrimination may be selected and set using a command although motions that may be performed by the user are restricted. As a result, selection of a motion to set a command may be difficult.
- a display apparatus includes a recognition unit to recognize a gaze of a user, a controller to determine whether the recognized gaze is within a predetermined recognition region and to control entry into an interactive mode upon determining that the recognized gaze is within the predetermined recognition region, and a display unit to display an image corresponding to the interactive mode.
- the display unit may display the predetermined recognition region.
- the controller may display display information of the predetermined region differently from display information before the gaze is recognized.
- the display unit may display a position of the recognized gaze as a cursor.
- the recognition unit may recognize a motion of the user, and the controller may determine an operation command corresponding to the recognized motion and control driving of the display unit based on the determined operation command.
- the recognition unit may recognize a voice of the user, and the controller may determine an operation command corresponding to the recognized voice and control driving of the display unit based on the determined operation command.
- the controller may control display of a plurality of recognition modes for interaction with the user and select one from among the recognition modes based on a position of the recognized gaze.
- the display apparatus may further include an image collection unit to collect an image of the user and a sound collection unit to collect a voice of the user, wherein the controller may control activation of the image collection unit and the sound collection unit based on the selected recognition mode.
- the recognition unit may recognize a face of the user, and the controller may determine whether the recognized user is a preregistered user and, upon determining that the recognized user is the preregistered user, control gaze recognition of the user.
- the controller may control gaze recognition of a user having highest priority based on prestored priorities.
- the controller may determine whether the recognized gaze is within a predetermined end region and, upon determining that the recognized gaze is within the predetermined end region, control the interactive mode to be completed.
- a display apparatus includes a first recognition unit to recognize a gaze of a user, a second recognition unit to recognize a command from the user, a controller to determine whether the recognized gaze is within a predetermined recognition region, to control entry into an interactive mode upon determining that the recognized gaze is within the predetermined recognition region, and to selectively control a plurality of functions based on the recognized gaze and command, and a display unit to display an image corresponding to the interactive mode and an image corresponding to the selectively controlled function.
- the second recognition unit may recognize at least one signal selected from between a motion and voice of the user and recognize an operation command corresponding to the at least one recognized signal.
- the controller may confirm a command corresponding to the recognized motion and control execution of a function corresponding to the confirmed command.
- the controller may confirm a command corresponding to the recognized voice and control execution of a function corresponding to the confirmed command.
- the functions may include at least one selected from among reproduction of an external broadcast signal, reproduction of external content, a reproduction mode, and a screen adjustment mode.
- the display apparatus may further include a communication unit to receive a sound signal transmitted from an external remote controller, wherein the second recognition unit may recognize a voice from the received sound signal.
- a display apparatus includes a tuner to receive an external broadcast signal, a signal processor to separate the broadcast signal into an image signal having image information and a sound signal having sound information and to signal-process the separated image signal and sound signal, a recognition unit to recognize a gaze of a user, a controller to determine whether the recognized gaze is within a predetermined recognition region and to control entry into an interactive mode upon determining that the recognized gaze is within the predetermined recognition region, a display unit to display the signal-processed image signal and an image corresponding to the interactive mode, and a sound output unit to output the signal-processed sound signal.
- the display unit may display the predetermined recognition region, a channel button, and a volume button and display a position of the recognized gaze as a cursor.
- the controller may control the interactive mode to be completed.
- the display apparatus may further include a collection module to collect at least one kind of information selected from between a voice and image of the user, wherein the recognition unit may recognize an operation command corresponding to the at least one kind of information, and, upon entering the interactive mode, the controller may control at least one function based on the recognized command.
- a collection module to collect at least one kind of information selected from between a voice and image of the user, wherein the recognition unit may recognize an operation command corresponding to the at least one kind of information, and, upon entering the interactive mode, the controller may control at least one function based on the recognized command.
- a control method of a display apparatus includes recognizing a gaze of a user, determining whether the recognized gaze is within a predetermined recognition region, entering an interactive mode upon determining that the recognized gaze is within the predetermined recognition region, displaying a plurality of recognition modes for interaction with the user, determining a recognition mode corresponding to a position of the recognized gaze from among the displayed recognition modes, executing the determined recognition mode to recognize a command from the user, and executing a function corresponding to the recognized command.
- the control method may further include displaying the predetermined recognition region as different display information upon entering the interactive mode.
- the entering the interactive mode may include determining whether the gaze is pointed at the predetermined region for a predetermined period of time.
- the executing the determined recognition mode to recognize the command from the user may include recognizing a motion of the user when the determined recognition mode is a motion mode and determining a command corresponding to the recognized motion.
- the executing the determined recognition mode to recognize the command from the user may include recognizing a voice of the user when the determined recognition mode is a voice mode and determining a command corresponding to the recognized voice.
- the control method may further include recognizing a face of the user, determining whether the recognized user is a preregistered user, recognizing a gaze of the user upon determining that the recognized user is the preregistered user.
- the control method may further include determining whether the recognized gaze is within a predetermined end region and completing the interactive mode upon determining that the recognized gaze is within the predetermined end region.
- the control method may further include, when a motion is recognized in a state in which the gaze is within the predetermined region, confirming a command corresponding to the recognized motion and controlling execution of a function corresponding to the confirmed command.
- the control method may further include, when a voice is recognized in a state in which the gaze is within the predetermined region, confirming a command corresponding to the recognized voice and controlling execution of a function corresponding to the confirmed command.
- the control method may further include receiving a voice signal transmitted from an external remote controller and recognizing a command from the received voice signal.
- the control method may further include completing the interactive mode when the gaze of the user is outside the region of the display unit for a predetermined period of time or more.
- the control method may further include displaying a position of the recognized gaze as a cursor.
- the control method may further include completing the interactive mode when a position of the recognized gaze alternately moves inside and outside the predetermined recognition region.
- FIG. 1 is a view illustrating a display apparatus according to an embodiment
- FIGS. 2A-2C, 3A, 3B and 4 are views illustrating display of the display apparatus according to the embodiment
- FIG. 5 is a control block diagram of the display apparatus according to the embodiment.
- FIG. 6 is a detailed view showing the construction of a recognition unit of the display apparatus according to the embodiment.
- FIG. 7 is a view illustrating a popup menu displayed on the display apparatus according to the embodiment.
- FIGS. 8A and 8B are control flowcharts of the display apparatus according to the embodiment.
- FIG. 9 is a control block diagram of a display apparatus according to another embodiment.
- FIG. 10 is a control flowchart of the display apparatus according to the embodiment.
- FIG. 11 is a view illustrating display of the display apparatus according to the embodiment.
- FIG. 12 is a view illustrating a display apparatus according to a further embodiment
- FIG. 13 is a control block diagram of the display apparatus according to the embodiment.
- FIG. 14 is a control flowchart of the display apparatus according to the embodiment.
- FIGS. 15 and 16 are views illustrating display of the display apparatus according to the embodiment.
- FIG. 1 is a view illustrating a display apparatus according to an embodiment. Hereinafter, the construction of the display apparatus will be described with reference to FIGS. 2A to 4 .
- the display apparatus is an image display apparatus, such as a television, monitor, or display device of a mobile communication terminal. As shown in FIG. 1 , the display apparatus 1 includes a main body 100 to display an image and a collection module 200 to collect image information and voice information of a user, which will be described in more detail.
- the main body 100 includes a display unit 110 to display an image and a bezel 120 to cover an image non-display region.
- the display unit 110 includes a display region 110 a to display a main image requested by a user, a recognition region 110 b to enable entry into an interactive mode, and an end region 110 c to enable end of the interactive mode.
- the recognition region 110 b is a region at which a user looks to enter the interactive mode
- the end region 110 c is a region at which the user looks to complete the interactive mode.
- the display unit 110 displays the recognition region 110 b and the end region 110 c as display information distinguished from the display region 110 a such that the user may discriminate therebetween.
- the recognition region 110 b and the end region 110 c may be displayed in the display region 110 a as figures filled with semitransparent colors or contour lines of semitransparent colors.
- the recognition region 110 b and the end region 110 c may be displayed as semitransparent icons.
- the display unit 110 displays the recognition region 110 b and the end region 110 c in the display region 110 a in an overlapping fashion.
- the recognition region 110 b and the end region 110 c may be displayed as opaque symbols (figures, contour lines, icons, etc.) of sizes which do not interfere with a user in viewing the main image displayed in the display region 110 a.
- the display unit 110 may periodically display the recognition region 110 b and the end region 110 c at predetermined time intervals. Alternatively, the display unit 110 may display a message regarding positions of the recognition region 110 b and the end region 110 c and use of an interactive mode at the time of initial operation.
- the display unit 110 operates differently before and after entry into the interactive mode. That is, display unit 110 displays the recognition region 110 b as different display information before and after entry into the interactive mode.
- the recognition region 110 b may be displayed as an empty figure before entry into the interactive mode, and an icon may be displayed in the empty figure of the recognition region 110 b after entry into the interactive mode.
- the recognition region 110 b may be displayed as a figure of a first color before entry into the interactive mode, and the recognition region 110 b may be displayed as a figure of a second color after entry into the interactive mode.
- the display unit 110 may display only the recognition region 110 b without displaying the end region 110 c and, after entry into the interactive mode, may display the end region 110 c.
- the display unit 110 After entry into the interactive mode, the display unit 110 maintains display information according to the entry into the interactive mode to inform a user that the interactive mode is being executed.
- the display of the icon in the figure of the recognition region 110 b or the second color of the figure of the recognition region 110 b may be maintained.
- the display unit 110 After entry into the interactive mode, the display unit 110 displays a plurality of recognition modes used to control operations and functions of the display apparatus.
- a recognition mode for interaction between a user and the display apparatus is displayed.
- the display unit 110 displays a motion recognition button a 1 , a voice recognition button a 2 , and a gaze recognition button a 3 in the display region 110 a .
- the display unit 110 displays the user's gaze as a cursor a 4 .
- the display unit 110 displays an image of content requested by the user. In a case in which screen adjustment information is changed by the user, the display unit 110 displays an image based on the changed screen adjustment information.
- the display unit 110 includes a display panel 111 , a diffusion plate 112 , a light guide plate 113 , a backlight unit 114 , and a chassis 115 .
- the display panel 111 displays an image, such as text, numbers, or arbitrary icons.
- the display unit 110 further includes glass (not shown) and a filter (not shown) to protect the display panel 111 from external impact.
- the glass prevents the filter from being broken by the external impact.
- the filter includes an optical property film, an electromagnetic interference (EMI) shielding film, and an infrared shielding film.
- EMI electromagnetic interference
- the optical property film decreases the brightness of red (R) and green (G) light incident upon the display panel and increases the brightness of blue (B) light, thereby improving optical properties.
- the EMI shielding film shields electromagnetic interference to prevent electromagnetic waves incident upon the display panel from being discharged outside.
- the infrared shielding film shields infrared light emitted from the display panel to prevent more than a predetermined amount of infrared light from being discharged outside such that infrared-based signals, such as signals generated by a remote controller, are normally transmitted to the display panel.
- the diffusion plate 112 is a semitransparent panel to diffuse light emitted from the backlight unit 114 along the surface thereof such that colors and brightness are uniform throughout the screen.
- the diffusion plate 112 improves the brightness of light emitted from the backlight unit 114 and supplies light having such improved brightness to the display panel. That is, the diffusion plate 112 increases the amount of light emitted from light emitting diodes (LEDs) of the backlight unit 114 and uniformly maintains brightness throughout the screen.
- LEDs light emitting diodes
- the light guide plate 113 enables light from the backlight unit 114 to be uniformly incident upon the entire display panel, and the backlight unit 114 emits light from the rear or the side of the display panel 111 .
- the display panel 111 Since the display panel 111 does not emit light, the display panel 111 adjusts the transmission amount and colors of the light emitted from the backlight unit such that an image is displayed on the display panel 111 .
- the chassis 115 is a panel, to which a drive module 130 necessary for image display and sound output is connected.
- Examples of the drive module 130 may include various printed circuit boards to control image display and sound output, an interface device for connection with external equipment, and a power supply to supply power to the respective devices.
- the chassis 115 is made of metal exhibiting high heat dissipation and strength.
- the bezel 120 covers the image non-display region of the display unit 110 to form the external appearance of the display unit 110 .
- An input unit (not shown) having a plurality of buttons may be further disposed at the bezel 120 . That is, an operation command may be input to the display apparatus 1 through the input unit.
- the main body 100 further includes a cover 121 to protect the image non-display region of the display unit 110 and a support member (not shown) to support the display panel, the diffusion plate, the light guide plate, the backlight unit, and the chassis of the display unit 110 in a separated state.
- the support member and the cover 121 may be detachably coupled to the bezel 120 .
- the cover 121 is coupled to the bezel 120 to form a receiving space.
- the display panel, the diffusion plate, the light guide plate, the backlight unit, and the chassis are disposed in the receiving space.
- the collection module 200 is provided at the upper end of the bezel 120 of the main body 100 to collect an image and sound of a user and surroundings of the user and to transmit the collected image and sound to the drive module 130 .
- the collection module 200 may be mounted at a position where user information is easily collected in addition to the upper end of the bezel 120 of the main body 100 .
- the collection module 200 may be provided separately from the display apparatus. That is, the collection module 200 may be provided separately from the display apparatus such that the collection module 200 is connected to the interface device (not shown) provided at the main body 100 .
- the collection module 200 (multimedia over internet protocol; MOIP) includes a housing 210 , an image collection unit 220 , such as a camera, mounted in the housing 210 , a sound collection unit 230 , such as a first microphone, mounted in the housing 210 , and a first microprocessor (not shown) to control driving of the camera and the first microphone and to transmit image signals and sound signals collected by the camera and the first microphone to the drive module 130 .
- MOIP multimedia over internet protocol
- the first microphone may include a plurality of microphones 230 a and 230 b.
- the image collection unit 220 i.e. the camera, collects an image of a user and surroundings of the user and transmits the collected image data to the first microprocessor.
- the sound collection units 230 a and 230 b i.e. the first microphones, collect sound of surroundings of a user including a voice of the user and transmit the collected sound data to the first microprocessor.
- the first microprocessor transmits the received image data and sound data to the drive module 130 .
- the first microprocessor may control the received image data and sound data to be stored in a memory (not shown).
- the collection module 200 may further include a communication unit (not shown), such as Ethernet.
- a communication unit such as Ethernet.
- the first microprocessor of the collection module (MOIP) compresses an image signal and sound signal and transmits the compressed data to the drive module 130 via Ethernet.
- the camera 220 and the first microphone 230 of the collection module may be directly connected to the drive module 130 to directly transmit the image signal and sound signal to the drive module.
- FIG. 5 is a control block diagram of the display apparatus according to the embodiment. Hereinafter, the construction of the display apparatus will be described with reference to FIGS. 6 and 7 .
- the display apparatus 1 is operated based on an interaction signal generated according to user intention and/or a signal from the input unit (not shown).
- the display apparatus 1 includes the image collection unit 220 and the sound collection unit 230 as the collection module to collect a user command, the drive module 130 to control the operation of the display apparatus 1 based on an operation command, and the display unit 110 and a sound output unit 140 as an output module to output operation execution.
- the image collection unit 220 is electrically connected to the drive module 130 to collect an image of a user and transmit a signal corresponding to the collected image to the drive module 130 according to a command from the drive module 130 .
- the sound collection unit 230 is electrically connected to the drive module 130 to collect a voice of a user and transmit a signal corresponding to the collected voice to the drive module 130 according to a command from the drive module 130 .
- the drive module 130 controls the operation of the display unit 110 and/or the sound output unit 140 according to a signal from the input unit (not shown).
- the drive module 130 controls entry into and end of an interactive mode based on information collected by the image collection unit 220 and/or the sound collection unit 230 . After entry into the interactive mode, the drive module 130 executes the interactive mode to control the operation of the display unit 110 and/or the sound output unit 140 .
- the drive module 130 includes a recognition unit 131 , a database 132 , a network interface 133 , a controller 134 , a storage unit 135 , a display drive unit 136 , and a sound drive unit 137 .
- the recognition unit 131 receives a signal corresponding to the information collected by the image collection unit 220 and/or the sound collection unit 230 , recognizes user recognition and an operation command desired by a user based on the received signal, and transmits the recognized operation command to the controller 134 , which will be described with reference to FIG. 6 .
- FIG. 6 is a detailed view showing the construction of the recognition unit of the display apparatus according to the embodiment.
- the recognition unit 131 includes a first recognition unit 131 - 1 to recognize an interactive mode entry command and a second recognition unit 131 - 2 to recognize a function execution and operation command.
- the first recognition unit 131 - 1 includes a face recognition unit 131 a and a gaze recognition unit 131 b.
- the face recognition unit 131 a removes a background region from the image transmitted from the image collection unit 220 to acquire a face image of the user, extracts a face region from the acquired face image of the user using edge and color information, extracts feature points, such as eyes, a nose, and a mouth, from the face region to recognize a face, and compares the recognized face with a preregistered face image to determine whether the recognized face is a face of a preregistered user.
- the recognition unit 131 filters the face image during face recognition to remove noise and performs preprocessing necessary to analyze features of the face image.
- the gaze recognition unit 131 b detects an eye image from the face image of the user and acquires the position of a pupil from the detected eye image to recognize the position of the user's gaze.
- the second recognition unit 131 - 2 includes a motion recognition unit 131 c and a voice recognition unit 131 d.
- the motion recognition unit 131 c removes a background region from the image transmitted from the image collection unit 220 to acquire a body image of the user, extracts face, arm, hand, torso, leg, and foot regions from the acquired body image of the user using edge and color information, extracts feature points from the extracted face, arm, hand, torso, leg, and foot regions to recognize a user motion, searches for a motion matching the recognized motion from among motions stored in the database 132 , and recognizes an operation command corresponding to the matching motion.
- the voice recognition unit 131 d removes noise from the sound transmitted from the sound collection unit 230 to acquire a voice of the user, extracts a feature vector from the acquired voice of the user, compares the extracted feature vector with a language model or phoneme-based sound model of each word stored in the database 132 to extract candidate words or candidate phonemes, recognizes a sentence based on the candidate words or candidate phonemes, searches for a sentence matching the recognized sentence from among sentences stored in the database 132 , and recognizes an operation command corresponding to the searched sentence.
- the voice recognition unit 131 d may recognize the operation command from the candidate words or candidate phonemes.
- the voice recognition unit 131 d may recognize the user using specific voice information.
- the database 132 stores a face image of a preregistered user, feature point information and pupil information of the face image. Also, the database 132 stores an operation command for each motion and an operation command for each voice.
- the pupil information includes information regarding size and position of a pupil and the distance between pupils. A user may be recognized based on the pupil information.
- the database 132 may store an operation command corresponding to a motion for each user and an operation command corresponding to a voice for each user.
- An IP address is assigned to the network interface 133 such that the network interface 133 performs network communication and receives and processes a data packet transmitted from the outside through a network.
- the network interface 133 stores the data packet or transmits the data packet to the controller such that the data packet is reproduced.
- the network interface 133 receives content, such as broadcasts, games, VOD, and broadcast signals, and relevant information. Also, the network interface 133 may receive firmware update information and update files.
- the display apparatus may further include an input interface.
- the input interface may be connected to an external storage device, such as a USB memory, or an optical disc device, such as a DVD or CD, to receive multimedia data and to transmit the received multimedia data to the controller.
- the controller 134 decides whether entry into the interactive mode is to be performed based on the determination as to whether the user recognized by the recognition unit 131 has been registered. In a case in which a plurality of users is recognized by the recognition unit 131 , the controller 134 confirms a user having highest priority from among the recognized users and controls entry into the interactive mode based on the position of the gaze of the confirmed user.
- the controller 134 may determine whether a user's gaze is within a recognition region and, upon determining that the user's gaze is within the recognition region, control entry into the interactive mode based on the gaze of the user pointed at the recognition region.
- the controller 134 determines a position the user looks at based on the position of the user's gaze transmitted from the recognition unit 131 . At this time, the controller 134 determines whether the user's gaze is within the recognition region for a predetermined period of time or more and, upon determining that the user's gaze is within the recognition region for the predetermined period of time or more, controls entry into the interactive mode.
- the controller 134 controls display information of the recognition region of the display unit 110 differently from the display information before entry into the interactive mode.
- the controller 134 controls a plurality of recognition modes to receive an operation command through interaction to be displayed. At this time, the controller 134 compares the position of the user's gaze transmitted from the recognition unit 131 with positions of prestored recognition mode buttons to determine a recognition mode the user wishes to select and controls the image collection unit 220 and the sound collection unit 230 to be turned on/off based on the determined recognition mode.
- the controller 134 controls the sound collection unit 230 to be turned on and the image collection unit 220 to be turned off.
- the controller 134 controls the sound collection unit 230 to be turned off and the image collection unit 220 to be turned on.
- the operation command includes change of output information, such as display or sound, and change of functions.
- the controller 134 controls driving of the respective drive units 136 and 137 based on the operation command transmitted from the recognition unit 131 .
- the controller 134 controls the interactive mode to be completed.
- the controller 134 may control a popup menu to be displayed.
- the popup menu P 1 includes a priority change button, volume control button, screen adjustment button, and end button.
- the controller 134 may determine that the user wishes to change the recognition mode and control the recognition mode buttons to be redisplayed.
- the controller 134 controls a main image signal requested by the user to be displayed and a sound signal requested by the user to be output and controls an image corresponding to the interactive mode to be displayed on the main image in an overlapping fashion during output of the image and the sound.
- the storage unit 135 stores priorities of a plurality of preregistered users, positions of the recognition region and the end region, before entry into the interactive mode, entry into the interactive mode, display information of the recognition region corresponding to execution of the interactive mode, and position information of the recognition mode buttons.
- the display drive unit 136 drives the display unit 110 based on a command from the controller 134
- the sound drive unit 137 drives the sound output unit 140 based on a command from the controller 134 .
- the display unit 110 which is an output module, displays an image in the display region based on a command from the controller 134 in the drive module 130 and displays an image corresponding to execution of the interactive mode.
- the sound output unit 140 which is another output module, outputs sound based on a command from the controller in the drive module.
- the sound output unit 140 may be separated from the display apparatus 1 .
- the sound output unit 140 such as a speaker, may be provided separately from the display apparatus 1 and may be connected to the display apparatus 1 as needed.
- FIGS. 8A and 8B are control flowcharts of the display apparatus according to the embodiment. Control of the display apparatus will be described with reference to FIGS. 1 to 7 .
- the image collection unit 220 is operated to collect an image ( 301 ).
- the display apparatus removes a background region from the collected image to acquire an image of the user, extracts a face region using edge and color information of the acquired image, and extracts feature points, such as eyes, a nose, and a mouth, from the face region to recognize a face ( 302 ).
- the display apparatus confirms the number of recognized users based on the number of recognized faces ( 303 ) and determines whether the number of the confirmed users is plural ( 304 ).
- the recognized face is compared with face images of preregistered users to determine whether the user having the recognized face is one of the preregistered users ( 305 ).
- a message stating that an interactive mode may not be executed is displayed through the display unit 110 ( 306 ).
- a pupil of the recognized face is recognized to track the gaze of the user ( 310 ).
- the display apparatus compares the recognized faces with face images of preregistered users to determine whether the users having the recognized faces are the preregistered users, to confirm the number of the users determined as the preregistered users ( 307 ), and to determine whether the number of the confirmed users is plural ( 308 ).
- the display apparatus Upon determining that a plurality of preregistered users has been recognized, the display apparatus determines a user having highest priority from among the recognized users ( 309 ) and recognizes a pupil of the determined user to track the gaze of the user ( 310 ).
- the display apparatus determines whether the recognized gaze is within the recognition region ( 311 ). Upon determining that the recognized gaze is within the recognition region, the display apparatus determines whether the recognized gaze is pointed at the recognition region for a predetermined period of time or more.
- the display apparatus Upon determining that the recognized gaze is pointed at the recognition region for the predetermined period of time or more, the display apparatus enters an interactive mode ( 312 ). In addition, the display apparatus changes display information of the recognition region in the display unit 110 and displays the changed display information such that entry into the interactive mode is recognized by the user ( 313 ).
- an icon may be displayed in the recognition region 110 b or the color of the recognition region 110 b may be changed.
- the display unit 110 displays a plurality of recognition modes to receive an operation command, confirms the position of the user's gaze ( 314 ), and determines a recognition mode selected by the user based on the confirmed position of the user's gaze ( 315 ).
- the display apparatus determines the position of one of the recognition mode buttons matching the confirmed position of the user's gaze to determine the recognition mode selected by the user.
- the display apparatus displays the position of the user's gaze on the display unit as a cursor in real time such that the user confirms the position of the user's gaze.
- the display apparatus executes the determined recognition mode ( 316 ). Also, the display apparatus recognizes an operation command based on the recognition result according to execution of the recognition mode and operates the respective drive units based on the recognized operation command ( 317 ).
- the display apparatus When motion recognition is selected by a user, the display apparatus operates the image collection unit, removes a background region from an image collected by the image collection unit to acquire a body image of the user, extracts face, arm, hand, torso, leg, and foot regions from the acquired body image of the user using edge and color information, extracts feature points from the extracted face, arm, hand, torso, leg, and foot regions to recognize a user motion, searches for a motion matching the recognized motion from among motions stored in the database 132 , and recognizes an operation command corresponding to the matching motion.
- the display apparatus drives the respective drive units 136 and 137 based on the recognized operation command to display an image of content requested by the user through the display unit 110 and to output sound of content requested by the user through the sound output unit 140 .
- the display apparatus When voice recognition is selected by a user, the display apparatus operates the sound collection unit, removes noise from sound collected by the sound collection unit 230 to acquire a voice of the user, extracts a feature vector from the acquired voice of the user, compares the extracted feature vector with a language model or phoneme-based sound model of each word stored in the database 132 to extract candidate words or candidate phonemes, recognizes a sentence based on the candidate words or candidate phonemes, searches for a sentence matching the recognized sentence from among sentences stored in the database 132 , and recognizes an operation command corresponding to the searched sentence.
- the display apparatus drives the respective drive units 136 and 137 based on the recognized operation command to display an image of content requested by the user through the display unit 110 and to output sound of content requested by the user through the sound output unit 140 .
- the display apparatus When gaze recognition is selected by a user, the display apparatus operates the image collection unit, displays a popup menu through the display unit 110 , recognizes a face and pupil of the user using an image collected by the image collection unit 220 , tracks the user's gaze based on the recognized pupil, confirms the position of the tracked gaze, and perform an operation corresponding to a menu selected from the popup menu by the user.
- the popup menu includes a plurality of menus, and each menu has an operation command button.
- the display apparatus confirms a button having a position matching the position of the user's gaze from among the operation command buttons in the popup menu and performs an operation corresponding to the confirmed button.
- a popup menu P 1 including a priority change button, volume control button, screen adjustment button, and end button is displayed as shown in FIG. 7 .
- an up/down button for volume control is displayed on the display unit 110 .
- volume up button volume is turned up.
- volume down button volume is turned down.
- menu selection through gaze recognition may be determined based on blinking of eyes, time for which the gaze is present on the button, or other action by a user's eye or eyes.
- the display apparatus displays content transmitted through the network interface 133 or the input interface on the display unit. At this time, the display apparatus controls a main image signal requested by the user to be displayed and a sound signal requested by the user to be output and controls an image corresponding to the interactive mode to be displayed on the main image in an overlapping fashion during output of the image and the sound.
- the display apparatus When the user's gaze is within the end region of the display unit 110 for a predetermined period of time or more, the display apparatus completes the interactive mode. Afterwards, when the user's gaze is within the recognition region, the display apparatus determines that the user wishes to change the recognition mode and redisplays the recognition mode buttons.
- the display apparatus may complete the interactive mode.
- FIG. 9 is a control block diagram of a display apparatus according to another embodiment.
- the display apparatus 1 includes an image collection unit 220 and a sound collection unit 230 as a collection module to receive a user command, a drive module 150 to perform operation control based on an operation command, and a display unit 110 and a sound output unit 140 as an output module to output operation execution.
- the image collection unit 220 , the sound collection unit 230 , the display unit 110 , and the sound output unit 140 are identical to those of the previous embodiment, and therefore, a description thereof will be omitted.
- the drive module 150 includes a recognition unit 151 , a database 152 , a network interface 153 , a controller 154 , a storage unit 155 , a display drive unit 156 , and a sound drive unit 157 .
- the recognition unit 151 receives a signal corresponding to information collected by the image collection unit 220 and/or the sound collection unit 230 , recognizes user recognition and an operation command desired by a user based on the received signal, and transmits the recognized operation command to the controller 154 .
- the recognition unit 151 determines whether a user having performed a motion is identical to a user looking at a recognition region and, upon determining that the user having performed a motion is identical to the user looking at the recognition region, recognizes an operation command corresponding to the recognized motion in a state in which the user's gaze is within the recognition region.
- the recognition unit 151 determines whether a user having spoken is identical to a user looking at a recognition region and, upon determining that the user having performed a motion is identical to the user looking at the recognition region, recognizes an operation command corresponding to the recognized voice in a state in which the user's gaze is within the recognition region.
- the recognition unit 151 includes a first recognition unit to recognize the face and gaze of a user and a second recognition unit to recognize a function execution and operation command.
- the first recognition unit and the second recognition unit are identical to the first recognition unit 131 - 1 and the second recognition unit 131 - 2 of the previous embodiment, and therefore, a description thereof will be omitted.
- the database 152 stores a face image and pupil information of each user in a matched state. Also, the database 152 stores voice information and pupil information of each user in a matched state.
- the other construction of the database 152 and the construction of the network interface 153 are identical to that of the database 132 and that of the network interface 133 , and therefore, a description thereof will be omitted.
- the controller 154 transmits an operation command to the image collection unit 220 and the sound collection unit 230 .
- the controller 154 decides whether entry into the interactive mode is to be performed based on the determination as to whether the user recognized by the recognition unit 151 has been registered. In a case in which a plurality of users is recognized by the recognition unit 151 , the controller 154 confirms a user having highest priority from among the recognized users and controls entry into the interactive mode based on the position of the gaze of the confirmed user.
- the controller 154 determines whether the user's gaze is within the recognition region for a predetermined period of time or more and, upon determining that the user's gaze is within the recognition region for the predetermined period of time or more, controls entry into the interactive mode.
- the controller 154 controls a plurality of recognition modes to receive a user command through interaction to be displayed. At this time, the controller 154 compares the position of the user's gaze transmitted from the recognition unit 151 with positions of prestored recognition mode buttons to determine a recognition mode the user wishes to select and controls the image collection unit 220 and the sound collection unit 230 to be turned on/off based on the determined recognition mode.
- the recognition modes include motion recognition and voice recognition.
- the controller 154 receives an operation command corresponding to a first multi-modal mode including a combination of gaze recognition and motion recognition.
- the controller 154 receives an operation command corresponding to a second multi-modal mode including a combination of gaze recognition and voice recognition.
- the controller 154 controls driving of the respective drive units 156 and 157 based on the operation command.
- the controller 154 controls the interactive mode to be completed.
- the controller 154 controls a main image signal requested by the user to be displayed and a sound signal requested by the user to be output.
- the storage unit 155 stores priorities of a plurality of preregistered users, positions of the recognition region and the end region, before entry into the interactive mode, entry into the interactive mode, display information of the recognition region corresponding to execution of the interactive mode, and position information of the recognition mode buttons.
- the display drive unit 156 and the sound drive unit 157 are identical to those of the previous embodiment, and therefore, a description thereof will be omitted.
- FIG. 10 is a control flowchart of the display apparatus according to the embodiment. Control of the display apparatus will be described with reference to FIGS. 9 and 11 .
- the image collection unit 220 When the display apparatus is turned on by a user, the image collection unit 220 is operated to collect an image. The display apparatus recognizes faces based on the collected image and confirms the number of recognized users based on the recognized faces.
- the display apparatus determines whether a plurality of users has been confirmed, determines preregistered users from among the recognized users, and determines whether the number of the preregistered users corresponding to the preregistered users is plural.
- the display apparatus Upon determining that the number of the preregistered users corresponding to the preregistered users is plural, the display apparatus determines a user having highest priority from among the users and recognizes a pupil of the determined user to track the gaze of the user.
- the display apparatus determines whether the recognized gaze is within the recognition region. Upon determining that the recognized gaze is within the recognition region, the display apparatus determines whether the recognized gaze is pointed at the recognition region for a predetermined period of time or more. Upon determining that the recognized gaze is pointed at the recognition region for the predetermined period of time or more, the display apparatus enters an interactive mode ( 401 ). In addition, the display apparatus changes display information of the recognition region in the display unit 110 and displays the changed display information such that entry into the interactive mode is recognized by the user ( 402 ).
- the display unit 110 displays a plurality of recognition modes to receive an operation command, confirms the position of the user's gaze ( 403 ), and determines a recognition mode selected by the user based on the confirmed position of the user's gaze ( 404 ).
- the display apparatus determines the position of one of the recognition mode buttons matching the confirmed position of the user's gaze to determine the recognition mode selected by the user.
- the display apparatus displays the position of the user's gaze in the display unit 110 as a cursor in real time such that the user confirms the position of the user's gaze.
- the display apparatus executes a multi-modal mode including a combination of the determined recognition mode and a gaze recognition mode ( 405 ). Also, the display apparatus recognizes an operation command based on the recognition result according to execution of the multi-modal mode and operates the respective drive units based on the recognized operation command ( 406 ).
- the display apparatus Before executing a first multi-modal mode including a combination of a motion recognition mode and a gaze recognition mode, the display apparatus determines whether a user having performed a motion is identical to a user looking at the recognition region and, upon determining that the user having performed a motion is identical to the user looking at the recognition region, executes an interactive mode based on the first multi-modal mode. Upon determining that the user having performed a motion is not identical to the user looking at the recognition region, a message stating that it is not possible to execute an interactive mode may be displayed.
- the display apparatus determines whether a user having spoken is identical to a user looking at the recognition region and, upon determining that the user having spoken is identical to the user looking at the recognition region, executes an interactive mode based on the second multi-modal mode. Upon determining that the user having spoken is not identical to the user looking at the recognition region, a message stating that it is not possible to execute an interactive mode may be displayed.
- the display apparatus executes a first multi-modal mode including a combination of a motion recognition mode and a gaze recognition mode.
- the display apparatus operates the image collection unit 220 , determines whether a user's gaze is within a recognition region and, upon determining that the user's gaze is within the recognition region, recognizes a user motion in a state in which the user's gaze is within the recognition region, and recognizes an operation command corresponding to the recognized motion.
- the display apparatus removes a background region from an image collected by the image collection unit 220 to acquire a body image of the user, extracts face, arm, hand, torso, leg, and foot regions from the acquired body image of the user using edge and color information, extracts feature points from the extracted face, arm, hand, torso, leg, and foot regions to recognize a user motion, searches for a motion matching the recognized motion from among motions stored in the database 152 , and recognizes an operation command corresponding to the matching motion.
- the display apparatus drives the respective drive units 156 and 157 based on the recognized operation command to display an image of content requested by the user through the display unit 110 and to output sound of content requested by the user through the sound output unit 140 .
- the display apparatus When the first multi-modal mode is executed in a state in which a plurality of functions is displayed in the display region 110 a of the display unit as shown in FIG. 11 , the display apparatus recognizes a user's gaze and motion using the image collection unit.
- the display apparatus recognizes the user's gaze within the recognition region and, at the same time, the one finger stretching motion, and recognizes an operation command corresponding to the one finger stretching motion.
- the display apparatus When the operation command corresponding to the one finger stretching motion is ‘search’, the display apparatus is connected to the Internet via the network interface to perform search.
- the display apparatus executes a second multi-modal mode including a combination of a voice recognition mode and a gaze recognition mode.
- the display apparatus operates the sound collection unit 230 , determines whether a user's gaze is within a recognition region and, upon determining that the user's gaze is within the recognition region, recognizes a voice of the user in a state in which the user's gaze is within the recognition region, and recognizes an operation command corresponding to the recognized voice.
- the display apparatus operates the sound collection unit 230 , removes noise from sound collected by the sound collection unit 230 to acquire a voice of the user, extracts a feature vector from the acquired voice of the user, compares the extracted feature vector with a language model or phoneme-based sound model of each word stored in the database 152 to extract candidate words or candidate phonemes, recognizes a sentence based on the candidate words or candidate phonemes, searches for a sentence matching the recognized sentence from among sentences stored in the database 152 , and recognizes an operation command corresponding to the searched sentence.
- the display apparatus drives the respective drive units 156 and 157 based on the recognized operation command to display an image of content requested by the user through the display unit 110 and to output sound of content requested by the user through the sound output unit 140 .
- the display apparatus When the second multi-modal mode is executed in a state in which a plurality of functions is displayed in the display region 110 a of the display unit as shown in FIG. 11 , the display apparatus recognizes a user's gaze and voice using the image collection unit and the sound collection unit.
- the display apparatus recognizes the user's gaze within the recognition region and, at the same time, the word ‘three’, and recognizes an operation command corresponding to ‘three’.
- the display apparatus When the operation command corresponding to ‘three’ is a ‘movie’, the display apparatus reproduces a movie downloaded through the network interface or a movie stored in an additional storage device or displays a submenu of the movie.
- an image corresponding to the interactive mode is displayed on the main image in an overlapping fashion.
- the display apparatus When the user's gaze is within the end region of the display unit 110 for a predetermined period of time or more, the display apparatus completes the interactive mode. Afterwards, when the user's gaze is within the end region, the display apparatus reperforms entry into the interactive mode and redisplays it.
- FIG. 12 is a view illustrating a display apparatus according to a further embodiment.
- a television will be described as an example of the display apparatus.
- the television includes a main body 100 to display an image, a stand 160 mounted to the bottom of the main body 100 to support the main body 100 , and a remote controller 170 to receive a user command.
- the stand 160 is detachably coupled to the main body 100 .
- the television may further include a bracket.
- the main body 100 may be mounted to a wall using the bracket. In this case, the stand may be removed.
- the main body 100 includes a display unit 110 to display an image and a bezel 120 to cover an image non-display region.
- the construction of the main body 100 is identical to that of the previous embodiment, and therefore, a description thereof will be omitted.
- the main body 100 may further include an input unit (not shown) having a plurality of buttons to receive a television operation command.
- a collection module 200 is provided at the upper end of the bezel 120 of the main body 100 to collect an image and sound of a user and surroundings of the user and to transmit the collected image and sound to a drive module.
- the collection module 200 may be mounted at a position where user information may be easily collected in addition to the upper end of the bezel 120 of the main body 100 .
- the collection module 200 may be provided separately from the television. That is, the collection module 200 may be provided separately from the main body 100 such that the collection module 200 is connected to an interface device (not shown) provided at the main body 100 .
- the collection module 200 is identical to that of the previous embodiment, and therefore, a description thereof will be omitted.
- the remote controller 170 is a device to remotely control television for user convenience.
- the remote controller 170 includes a plurality of buttons 171 , such as function selection buttons, number buttons, power on/off buttons, channel change buttons, and volume control buttons, a second sound collection unit, i.e. a second microphone 172 , to collect a voice signal of a user, a second microprocessor (not shown) to generate an operation signal upon receiving a signal from the buttons 171 and the voice signal, a communication unit 173 to transmit the operation signal generated by the second microprocessor, and a battery to supply driving power to the second microphone, the buttons, the second microprocessor, and the communication unit.
- buttons 171 such as function selection buttons, number buttons, power on/off buttons, channel change buttons, and volume control buttons
- a second sound collection unit i.e. a second microphone 172
- a second microprocessor (not shown) to generate an operation signal upon receiving a signal from the buttons 171 and the voice signal
- a communication unit 173 to transmit the operation signal generated by the second microprocessor
- a battery to supply driving power
- the second sound collection unit 172 may constantly operate.
- the remote controller 170 transmits an operation signal, such as a channel change signal or a volume control signal, to the television in response to user manipulation.
- an operation signal such as a channel change signal or a volume control signal
- the remote controller may further include a plurality of buttons to select and control an Internet function in addition to the broadcast viewing control buttons in order to control a display apparatus supporting the Internet in addition to broadcast viewing.
- FIG. 13 is a control block diagram of the display apparatus, i.e. the television, according to the embodiment.
- the television is operated based on interaction information generated through collection of a user command, a signal from the input unit (not shown) provided at the main body 100 , or a signal from the remote controller 170 .
- the television includes an image collection unit 220 and a sound collection unit 230 as a collection module to collect a user command, the drive module 180 to control the operation of the television based on the user command, and a display unit 110 and a sound output unit 140 as an output module to output operation execution.
- the drive module 180 controls the operation of the display unit 110 and/or the sound output unit 140 based on the signal from the input unit (not shown) or the signal from the remote controller 170 .
- the drive module 180 controls entry into and end of an interactive mode based on information collected by the image collection unit 220 and/or the sound collection unit 230 and, after entry into the interactive mode, executes the interactive mode to control the operation of the display unit 110 and/or the sound output unit 140 .
- the drive module 180 of the television includes a broadcast unit 181 , an interface 182 , a recognition unit 183 , a database 184 , a receiving unit 185 , a controller 186 , a storage unit 187 , and a drive unit 188 .
- the broadcast unit 181 includes a tuner 181 a and a signal processor 181 b.
- the tuner 181 a receives a broadcast signal having a corresponding frequency band according to a tuning control signal from the controller 186 in a wired or wireless fashion to tune the television.
- the signal processor 181 b separates the broadcast signal into an image signal having image data, a sound signal having sound data, and an additional signal having additional data, signal-processes the separated image signal having image data and sound signal having sound data, and transmits the additional data to a GUI generation unit (not shown).
- the additional data includes an electronic program guide (EPG).
- the signal processor 181 b may separate a signal transmitted from a network interface 182 b into an image signal having image data and a sound signal having sound data and signal-process the image signal and the sound signal.
- the interface unit 182 includes an input interface 182 a and a network interface 182 b.
- the input interface 182 a is connected to the collection module 200 to receive an image signal and sound signal collected by the collection module 200 and to transmit the received image signal and sound signal to the recognition unit 183 .
- An IP address is assigned to the network interface 182 b such that the network interface 182 b performs network communication and receives and processes a data packet transmitted from the outside through a network.
- the network interface 182 b stores the data packet or transmits the data packet to the controller 186 such that the data packet is reproduced.
- the recognition unit 183 recognizes a face and pupil of a user, recognizes the user's gaze based on information of the recognized pupil, and recognizes a user command.
- the recognition unit 183 recognizes a voice of the user from the sound signal transmitted from the receiving unit 185 and recognizes an operation command corresponding to the recognized voice.
- the other construction of the recognition unit 183 is identical to that of the previous embodiment, and therefore, a description thereof will be omitted.
- the database 184 is identical to that of the previous embodiment, and therefore, a description thereof will be omitted.
- the receiving unit 185 receives a signal from one of the buttons 171 provided at the remote controller 170 and a sound signal from the second sound collection unit 172 provided at the remote controller 170 .
- the receiving unit 185 transmits the received button signal and sound signal to the recognition unit 183 and the controller 186 .
- the controller 186 controls the image data signal-processed by the signal processor 181 b to be displayed on the display unit 110 and the signal-processed sound to be output through the sound output unit 140 .
- the controller 186 determines a user command based on the button signal from the remote controller 170 and controls driving of the display unit 110 according to the determined user command. Upon receiving an Internet function execution request signal from the remote controller 170 , the controller 186 controls the network interface for Internet connection. When the user's gaze is within the recognition region, the controller 186 controls driving of the collection module to execute the interactive mode.
- controller 186 to control entry into the interactive mode and execution of the interactive mode is identical to that of the previous embodiment, and therefore, a description thereof will be omitted.
- the storage unit 187 is identical to that of the previous embodiment, and therefore, a description thereof will be omitted.
- the drive unit 188 includes a display drive unit 188 a and a sound drive unit 188 b .
- the display drive unit 188 a and the sound drive unit 188 b are identical to those of the previous embodiment, and therefore, a description thereof will be omitted.
- FIG. 14 is a control flowchart of the display apparatus, i.e. the television, according to the embodiment.
- the image collection unit 220 When the television is turned on, the image collection unit 220 is operated to collect an image.
- the television recognizes a face of a user based on the collected image, recognizes a pupil of the user based on the recognized face, and recognizes a gaze corresponding to the position of the pupil ( 501 ).
- the television determines whether the recognized gaze is within the recognition region ( 502 ). Upon determining that the recognized gaze is within the recognition region, the television determines whether the recognized gaze is pointed at the recognition region for a predetermined period of time or more ( 503 ). Upon determining that the recognized gaze is pointed at the recognition region for the predetermined period of time or more, the television enters an interactive mode ( 504 ). In addition, the television changes display information of the recognition region in the display unit 110 and displays the changed display information such that entry into the interactive mode is recognized by the user such as shown in operation 402 of FIG. 10 .
- the television displays a plurality of recognition modes to receive an operation command ( 505 ), confirms the position of the user's gaze, and determines a recognition mode selected by the user based on the confirmed position of the user's gaze ( 506 ).
- the television determines the position of one of the recognition mode buttons matching the confirmed position of the user's gaze to determine the recognition mode selected by the user.
- the television displays the position of the user's gaze in the display unit 110 as a cursor in real time such that the user may confirm the position of the user's gaze.
- the television executes the determined recognition mode ( 507 ). Also, the television recognizes an operation command based on the recognition result according to execution of the recognition mode and operates the respective drive units based on the recognized operation command ( 508 ).
- the television recognizes a user motion, searches for a motion matching the recognized motion from among motions stored in the database 184 , and recognizes an operation command corresponding to the matching motion.
- the television drives the drive unit 188 based on the recognized operation command to display an image of a broadcast or content requested by the user through the display unit 110 and to output sound of a broadcast or content requested by the user through the sound output unit 140 .
- the television operates the first sound collection unit 230 , recognizes a sentence corresponding to a voice in sound collected by the first sound collection unit 230 or the second sound collection unit 172 , searches for a sentence matching the recognized sentence from among sentences stored in the database 184 , and recognizes an operation command corresponding to the searched sentence.
- the television drives the drive unit 188 based on the recognized operation command to display an image of a broadcast or content requested by the user through the display unit 110 and to output sound of a broadcast or content requested by the user through the sound output unit 140 .
- the television When gaze recognition is selected by a user, the television operates the image collection unit, displays a popup menu through the display unit 110 , recognizes a face and pupil of the user using an image collected by the image collection unit 220 , tracks the user's gaze based on the recognized pupil, confirms the position of the tracked gaze, and performs an operation corresponding to a menu item selected by the user from the popup menu.
- the popup menu includes a plurality of menus, and each menu has an operation command button.
- the television confirms a button having a position matching the position of the user's gaze from among the operation command buttons in the popup menu and performs an operation corresponding to the confirmed button, which will be described with reference to FIGS. 15 and 16 .
- a button having a position matching the position of a user's gaze is determined.
- the function includes a broadcast function, Internet function, and communication function with an external device.
- a volume up/down button a 5 for volume control, a channel up/down button a 6 for channel adjustment, and a cursor a 4 indicating the position of the user's gaze are displayed in the display unit 110 as shown in FIG. 16 .
- the television compares the position of the user's gaze with the position of the volume up/down button a 5 and the channel up/down button a 6 to turn the channel/volume up/down.
- menu selection through gaze recognition may be determined based on blinking of eyes or time for which the gaze is present on the button.
- the television displays a button for gaze recognition on a main image displayed in the display unit in an overlapping fashion.
- the television completes the interactive mode. Afterwards, when the user's gaze is within the recognition region, the television determines that the user wishes to change the recognition mode and redisplays the recognition mode buttons.
- the television may complete the interactive mode.
- a user's gaze is tracked to perform entry into an interactive mode, thereby easily achieving entry into the interactive mode and performing more intuitive interaction.
- a multi-modal interactive mode including a combination of face recognition, voice recognition, and gaze recognition is performed, thereby performing a more extended interactive mode and accurately determining a user command.
- functions are correctly performed, thereby improving user convenience.
- the position of a user's gaze is displayed on a display apparatus as a cursor, thereby correctly confirming the position of the user's gaze.
- a user is recognized from a face or pupil of the user, and a gaze, voice, and motion are recognized considering features of the recognized user, thereby improving recognition efficiency.
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- General Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Human Computer Interaction (AREA)
- Multimedia (AREA)
- Health & Medical Sciences (AREA)
- General Health & Medical Sciences (AREA)
- Audiology, Speech & Language Pathology (AREA)
- Optics & Photonics (AREA)
- Signal Processing (AREA)
- Library & Information Science (AREA)
- Oral & Maxillofacial Surgery (AREA)
- Ophthalmology & Optometry (AREA)
- User Interface Of Digital Computer (AREA)
- Position Input By Displaying (AREA)
- Studio Devices (AREA)
- Image Analysis (AREA)
- Controls And Circuits For Display Device (AREA)
- Telephone Function (AREA)
Abstract
Description
- This application is a divisional of U.S. application Ser. No. 14/056,472, filed on Oct. 17, 2013, which claims the benefit of Korean Patent Application No. 10-2012-0116956, filed on Oct. 19, 2012 in the Korean Intellectual Property Office, the disclosures of which are incorporated herein by reference.
- 1. Field
- Embodiments of the present disclosure relate to a display apparatus that improves recognition accuracy of information input by a user and a control method thereof.
- 2. Description of the Related Art
- A display apparatus displays stereoscopic images.
- In recent years, there have been developed flat display devices which have lower weight and volume than a cathode ray tube and may be installed in a wide variety of spaces, thereby easily achieving large-sized image realization, flat structure, and high quality.
- Representative examples of such flat display devices include a liquid crystal display device (LCD), electroluminescent display device (ELD), field emission display device (FED), plasma display panel (PDP), thin film transistor liquid crystal display device (TFT-LCD), and flexible display device.
- Such display devices include recognition technologies for improved user convenience.
- Examples of the recognition technologies include a motion recognition technology and a voice recognition technology. The motion recognition technology is technology for recognizing a user's motion using a motion sensor and determining a user command from the recognized motion. The voice recognition technology is technology for recognizing a user's voice using a voice sensor and determining a user command from the recognized voice.
- In these recognition technologies, a function that is not desired by a user may be executed or user input may not be performed if the motion or the voice is not correctly input. For this reason, operations may have to be repeatedly performed, which provides inconvenience to a user.
- In addition, certain functions may be unintentionally executed due to a motion or voice unconsciously issued by the user.
- Also, in a case in which the motion recognition technology is applied to a display device, a motion that is not often performed by the user or a motion having high discrimination may be selected and set using a command although motions that may be performed by the user are restricted. As a result, selection of a motion to set a command may be difficult.
- It is an aspect of the present disclosure to provide a display apparatus that recognizes a user's gaze to enter an interactive mode and displays the position of the gaze as a cursor and a control method thereof.
- It is another aspect of the present disclosure to provide a display apparatus that recognizes a user's gaze and motion or simultaneously recognizes a user's gaze and voice to control operations and functions and a control method thereof.
- Additional aspects of the disclosure will be set forth in part in the description which follows and, in part, will be apparent from the description, or may be learned by practice of the disclosure.
- In accordance with one aspect of the present disclosure, a display apparatus includes a recognition unit to recognize a gaze of a user, a controller to determine whether the recognized gaze is within a predetermined recognition region and to control entry into an interactive mode upon determining that the recognized gaze is within the predetermined recognition region, and a display unit to display an image corresponding to the interactive mode.
- The display unit may display the predetermined recognition region.
- When the gaze is recognized, the controller may display display information of the predetermined region differently from display information before the gaze is recognized.
- The display unit may display a position of the recognized gaze as a cursor.
- The recognition unit may recognize a motion of the user, and the controller may determine an operation command corresponding to the recognized motion and control driving of the display unit based on the determined operation command.
- The recognition unit may recognize a voice of the user, and the controller may determine an operation command corresponding to the recognized voice and control driving of the display unit based on the determined operation command.
- Upon entering the interactive mode, the controller may control display of a plurality of recognition modes for interaction with the user and select one from among the recognition modes based on a position of the recognized gaze.
- The display apparatus may further include an image collection unit to collect an image of the user and a sound collection unit to collect a voice of the user, wherein the controller may control activation of the image collection unit and the sound collection unit based on the selected recognition mode.
- The recognition unit may recognize a face of the user, and the controller may determine whether the recognized user is a preregistered user and, upon determining that the recognized user is the preregistered user, control gaze recognition of the user.
- In a case in which a plurality of users is recognized, the controller may control gaze recognition of a user having highest priority based on prestored priorities.
- The controller may determine whether the recognized gaze is within a predetermined end region and, upon determining that the recognized gaze is within the predetermined end region, control the interactive mode to be completed.
- In accordance with another aspect of the present disclosure, a display apparatus includes a first recognition unit to recognize a gaze of a user, a second recognition unit to recognize a command from the user, a controller to determine whether the recognized gaze is within a predetermined recognition region, to control entry into an interactive mode upon determining that the recognized gaze is within the predetermined recognition region, and to selectively control a plurality of functions based on the recognized gaze and command, and a display unit to display an image corresponding to the interactive mode and an image corresponding to the selectively controlled function.
- The second recognition unit may recognize at least one signal selected from between a motion and voice of the user and recognize an operation command corresponding to the at least one recognized signal.
- When a motion is recognized in a state in which the gaze is within the predetermined region, the controller may confirm a command corresponding to the recognized motion and control execution of a function corresponding to the confirmed command.
- When a voice is recognized in a state in which the gaze is within the predetermined region, the controller may confirm a command corresponding to the recognized voice and control execution of a function corresponding to the confirmed command.
- The functions may include at least one selected from among reproduction of an external broadcast signal, reproduction of external content, a reproduction mode, and a screen adjustment mode.
- The display apparatus may further include a communication unit to receive a sound signal transmitted from an external remote controller, wherein the second recognition unit may recognize a voice from the received sound signal.
- In accordance with another aspect of the present disclosure, a display apparatus includes a tuner to receive an external broadcast signal, a signal processor to separate the broadcast signal into an image signal having image information and a sound signal having sound information and to signal-process the separated image signal and sound signal, a recognition unit to recognize a gaze of a user, a controller to determine whether the recognized gaze is within a predetermined recognition region and to control entry into an interactive mode upon determining that the recognized gaze is within the predetermined recognition region, a display unit to display the signal-processed image signal and an image corresponding to the interactive mode, and a sound output unit to output the signal-processed sound signal.
- The display unit may display the predetermined recognition region, a channel button, and a volume button and display a position of the recognized gaze as a cursor.
- When the gaze of the user is outside the region of the display unit, the controller may control the interactive mode to be completed.
- The display apparatus may further include a collection module to collect at least one kind of information selected from between a voice and image of the user, wherein the recognition unit may recognize an operation command corresponding to the at least one kind of information, and, upon entering the interactive mode, the controller may control at least one function based on the recognized command.
- In accordance with a further aspect of the present disclosure, a control method of a display apparatus includes recognizing a gaze of a user, determining whether the recognized gaze is within a predetermined recognition region, entering an interactive mode upon determining that the recognized gaze is within the predetermined recognition region, displaying a plurality of recognition modes for interaction with the user, determining a recognition mode corresponding to a position of the recognized gaze from among the displayed recognition modes, executing the determined recognition mode to recognize a command from the user, and executing a function corresponding to the recognized command.
- The control method may further include displaying the predetermined recognition region as different display information upon entering the interactive mode.
- The entering the interactive mode may include determining whether the gaze is pointed at the predetermined region for a predetermined period of time.
- The executing the determined recognition mode to recognize the command from the user may include recognizing a motion of the user when the determined recognition mode is a motion mode and determining a command corresponding to the recognized motion.
- The executing the determined recognition mode to recognize the command from the user may include recognizing a voice of the user when the determined recognition mode is a voice mode and determining a command corresponding to the recognized voice.
- The control method may further include recognizing a face of the user, determining whether the recognized user is a preregistered user, recognizing a gaze of the user upon determining that the recognized user is the preregistered user.
- The control method may further include determining whether the recognized gaze is within a predetermined end region and completing the interactive mode upon determining that the recognized gaze is within the predetermined end region.
- The control method may further include, when a motion is recognized in a state in which the gaze is within the predetermined region, confirming a command corresponding to the recognized motion and controlling execution of a function corresponding to the confirmed command.
- The control method may further include, when a voice is recognized in a state in which the gaze is within the predetermined region, confirming a command corresponding to the recognized voice and controlling execution of a function corresponding to the confirmed command.
- The control method may further include receiving a voice signal transmitted from an external remote controller and recognizing a command from the received voice signal.
- The control method may further include completing the interactive mode when the gaze of the user is outside the region of the display unit for a predetermined period of time or more.
- The control method may further include displaying a position of the recognized gaze as a cursor.
- The control method may further include completing the interactive mode when a position of the recognized gaze alternately moves inside and outside the predetermined recognition region.
- These and/or other aspects of the disclosure will become apparent and more readily appreciated from the following description of the embodiments, taken in conjunction with the accompanying drawings of which:
-
FIG. 1 is a view illustrating a display apparatus according to an embodiment; -
FIGS. 2A-2C, 3A, 3B and 4 are views illustrating display of the display apparatus according to the embodiment; -
FIG. 5 is a control block diagram of the display apparatus according to the embodiment; -
FIG. 6 is a detailed view showing the construction of a recognition unit of the display apparatus according to the embodiment; -
FIG. 7 is a view illustrating a popup menu displayed on the display apparatus according to the embodiment; -
FIGS. 8A and 8B are control flowcharts of the display apparatus according to the embodiment; -
FIG. 9 is a control block diagram of a display apparatus according to another embodiment; -
FIG. 10 is a control flowchart of the display apparatus according to the embodiment; -
FIG. 11 is a view illustrating display of the display apparatus according to the embodiment; -
FIG. 12 is a view illustrating a display apparatus according to a further embodiment; -
FIG. 13 is a control block diagram of the display apparatus according to the embodiment; -
FIG. 14 is a control flowchart of the display apparatus according to the embodiment; and -
FIGS. 15 and 16 are views illustrating display of the display apparatus according to the embodiment. - Reference will now be made in detail to the embodiments of the present disclosure, examples of which are illustrated in the accompanying drawings, wherein like reference numerals refer to like elements throughout.
-
FIG. 1 is a view illustrating a display apparatus according to an embodiment. Hereinafter, the construction of the display apparatus will be described with reference toFIGS. 2A to 4 . - The display apparatus is an image display apparatus, such as a television, monitor, or display device of a mobile communication terminal. As shown in
FIG. 1 , thedisplay apparatus 1 includes amain body 100 to display an image and acollection module 200 to collect image information and voice information of a user, which will be described in more detail. - The
main body 100 includes adisplay unit 110 to display an image and abezel 120 to cover an image non-display region. - The
display unit 110 includes adisplay region 110 a to display a main image requested by a user, arecognition region 110 b to enable entry into an interactive mode, and anend region 110 c to enable end of the interactive mode. - That is, the
recognition region 110 b is a region at which a user looks to enter the interactive mode, and theend region 110 c is a region at which the user looks to complete the interactive mode. - The
display unit 110 displays therecognition region 110 b and theend region 110 c as display information distinguished from thedisplay region 110 a such that the user may discriminate therebetween. - For example, as shown in
FIGS. 2A and 2B , therecognition region 110 b and theend region 110 c may be displayed in thedisplay region 110 a as figures filled with semitransparent colors or contour lines of semitransparent colors. Alternatively, as shown inFIG. 2C , therecognition region 110 b and theend region 110 c may be displayed as semitransparent icons. - At this time, the
display unit 110 displays therecognition region 110 b and theend region 110 c in thedisplay region 110 a in an overlapping fashion. - Also, the
recognition region 110 b and theend region 110 c may be displayed as opaque symbols (figures, contour lines, icons, etc.) of sizes which do not interfere with a user in viewing the main image displayed in thedisplay region 110 a. - The
display unit 110 may periodically display therecognition region 110 b and theend region 110 c at predetermined time intervals. Alternatively, thedisplay unit 110 may display a message regarding positions of therecognition region 110 b and theend region 110 c and use of an interactive mode at the time of initial operation. - In addition, the
display unit 110 operates differently before and after entry into the interactive mode. That is,display unit 110 displays therecognition region 110 b as different display information before and after entry into the interactive mode. - For example, as shown in
FIG. 3A , therecognition region 110 b may be displayed as an empty figure before entry into the interactive mode, and an icon may be displayed in the empty figure of therecognition region 110 b after entry into the interactive mode. Alternatively, as shown inFIG. 3B , therecognition region 110 b may be displayed as a figure of a first color before entry into the interactive mode, and therecognition region 110 b may be displayed as a figure of a second color after entry into the interactive mode. - Also, the
display unit 110 may display only therecognition region 110 b without displaying theend region 110 c and, after entry into the interactive mode, may display theend region 110 c. - After entry into the interactive mode, the
display unit 110 maintains display information according to the entry into the interactive mode to inform a user that the interactive mode is being executed. - For example, the display of the icon in the figure of the
recognition region 110 b or the second color of the figure of therecognition region 110 b may be maintained. - After entry into the interactive mode, the
display unit 110 displays a plurality of recognition modes used to control operations and functions of the display apparatus. - As shown in
FIG. 4 , a recognition mode for interaction between a user and the display apparatus is displayed. As an example, thedisplay unit 110 displays a motion recognition button a1, a voice recognition button a2, and a gaze recognition button a3 in thedisplay region 110 a. At this time, thedisplay unit 110 displays the user's gaze as a cursor a4. - Also, the
display unit 110 displays an image of content requested by the user. In a case in which screen adjustment information is changed by the user, thedisplay unit 110 displays an image based on the changed screen adjustment information. - The
display unit 110 includes adisplay panel 111, adiffusion plate 112, alight guide plate 113, abacklight unit 114, and achassis 115. - The
display panel 111 displays an image, such as text, numbers, or arbitrary icons. - The
display unit 110 further includes glass (not shown) and a filter (not shown) to protect thedisplay panel 111 from external impact. The glass prevents the filter from being broken by the external impact. - The filter includes an optical property film, an electromagnetic interference (EMI) shielding film, and an infrared shielding film.
- The optical property film decreases the brightness of red (R) and green (G) light incident upon the display panel and increases the brightness of blue (B) light, thereby improving optical properties. The EMI shielding film shields electromagnetic interference to prevent electromagnetic waves incident upon the display panel from being discharged outside.
- Also, the infrared shielding film shields infrared light emitted from the display panel to prevent more than a predetermined amount of infrared light from being discharged outside such that infrared-based signals, such as signals generated by a remote controller, are normally transmitted to the display panel.
- The
diffusion plate 112 is a semitransparent panel to diffuse light emitted from thebacklight unit 114 along the surface thereof such that colors and brightness are uniform throughout the screen. Thediffusion plate 112 improves the brightness of light emitted from thebacklight unit 114 and supplies light having such improved brightness to the display panel. That is, thediffusion plate 112 increases the amount of light emitted from light emitting diodes (LEDs) of thebacklight unit 114 and uniformly maintains brightness throughout the screen. - The
light guide plate 113 enables light from thebacklight unit 114 to be uniformly incident upon the entire display panel, and thebacklight unit 114 emits light from the rear or the side of thedisplay panel 111. - Since the
display panel 111 does not emit light, thedisplay panel 111 adjusts the transmission amount and colors of the light emitted from the backlight unit such that an image is displayed on thedisplay panel 111. - The
chassis 115 is a panel, to which adrive module 130 necessary for image display and sound output is connected. - Examples of the
drive module 130 may include various printed circuit boards to control image display and sound output, an interface device for connection with external equipment, and a power supply to supply power to the respective devices. - The
chassis 115 is made of metal exhibiting high heat dissipation and strength. - The
bezel 120 covers the image non-display region of thedisplay unit 110 to form the external appearance of thedisplay unit 110. - An input unit (not shown) having a plurality of buttons may be further disposed at the
bezel 120. That is, an operation command may be input to thedisplay apparatus 1 through the input unit. - The
main body 100 further includes acover 121 to protect the image non-display region of thedisplay unit 110 and a support member (not shown) to support the display panel, the diffusion plate, the light guide plate, the backlight unit, and the chassis of thedisplay unit 110 in a separated state. The support member and thecover 121 may be detachably coupled to thebezel 120. - The
cover 121 is coupled to thebezel 120 to form a receiving space. The display panel, the diffusion plate, the light guide plate, the backlight unit, and the chassis are disposed in the receiving space. - The
collection module 200 is provided at the upper end of thebezel 120 of themain body 100 to collect an image and sound of a user and surroundings of the user and to transmit the collected image and sound to thedrive module 130. - The
collection module 200 may be mounted at a position where user information is easily collected in addition to the upper end of thebezel 120 of themain body 100. - Also, the
collection module 200 may be provided separately from the display apparatus. That is, thecollection module 200 may be provided separately from the display apparatus such that thecollection module 200 is connected to the interface device (not shown) provided at themain body 100. - Hereinafter, the
collection module 200 will be described in more detail. - The collection module 200 (multimedia over internet protocol; MOIP) includes a
housing 210, animage collection unit 220, such as a camera, mounted in thehousing 210, asound collection unit 230, such as a first microphone, mounted in thehousing 210, and a first microprocessor (not shown) to control driving of the camera and the first microphone and to transmit image signals and sound signals collected by the camera and the first microphone to thedrive module 130. - The first microphone may include a plurality of
microphones - The
image collection unit 220, i.e. the camera, collects an image of a user and surroundings of the user and transmits the collected image data to the first microprocessor. - The
sound collection units - The first microprocessor transmits the received image data and sound data to the
drive module 130. In addition, the first microprocessor may control the received image data and sound data to be stored in a memory (not shown). - The
collection module 200 may further include a communication unit (not shown), such as Ethernet. In this case, the first microprocessor of the collection module (MOIP) compresses an image signal and sound signal and transmits the compressed data to thedrive module 130 via Ethernet. - In addition, the
camera 220 and thefirst microphone 230 of the collection module may be directly connected to thedrive module 130 to directly transmit the image signal and sound signal to the drive module. -
FIG. 5 is a control block diagram of the display apparatus according to the embodiment. Hereinafter, the construction of the display apparatus will be described with reference toFIGS. 6 and 7 . - In this embodiment, the
display apparatus 1 is operated based on an interaction signal generated according to user intention and/or a signal from the input unit (not shown). - The
display apparatus 1 includes theimage collection unit 220 and thesound collection unit 230 as the collection module to collect a user command, thedrive module 130 to control the operation of thedisplay apparatus 1 based on an operation command, and thedisplay unit 110 and asound output unit 140 as an output module to output operation execution. - The
image collection unit 220 is electrically connected to thedrive module 130 to collect an image of a user and transmit a signal corresponding to the collected image to thedrive module 130 according to a command from thedrive module 130. Thesound collection unit 230 is electrically connected to thedrive module 130 to collect a voice of a user and transmit a signal corresponding to the collected voice to thedrive module 130 according to a command from thedrive module 130. - The
drive module 130 controls the operation of thedisplay unit 110 and/or thesound output unit 140 according to a signal from the input unit (not shown). - Also, the
drive module 130 controls entry into and end of an interactive mode based on information collected by theimage collection unit 220 and/or thesound collection unit 230. After entry into the interactive mode, thedrive module 130 executes the interactive mode to control the operation of thedisplay unit 110 and/or thesound output unit 140. - The
drive module 130 includes arecognition unit 131, adatabase 132, anetwork interface 133, acontroller 134, astorage unit 135, adisplay drive unit 136, and asound drive unit 137. - The
recognition unit 131 receives a signal corresponding to the information collected by theimage collection unit 220 and/or thesound collection unit 230, recognizes user recognition and an operation command desired by a user based on the received signal, and transmits the recognized operation command to thecontroller 134, which will be described with reference toFIG. 6 . -
FIG. 6 is a detailed view showing the construction of the recognition unit of the display apparatus according to the embodiment. Therecognition unit 131 includes a first recognition unit 131-1 to recognize an interactive mode entry command and a second recognition unit 131-2 to recognize a function execution and operation command. - The first recognition unit 131-1 includes a
face recognition unit 131 a and agaze recognition unit 131 b. - The
face recognition unit 131 a removes a background region from the image transmitted from theimage collection unit 220 to acquire a face image of the user, extracts a face region from the acquired face image of the user using edge and color information, extracts feature points, such as eyes, a nose, and a mouth, from the face region to recognize a face, and compares the recognized face with a preregistered face image to determine whether the recognized face is a face of a preregistered user. - In addition, the
recognition unit 131 filters the face image during face recognition to remove noise and performs preprocessing necessary to analyze features of the face image. - The
gaze recognition unit 131 b detects an eye image from the face image of the user and acquires the position of a pupil from the detected eye image to recognize the position of the user's gaze. - The second recognition unit 131-2 includes a
motion recognition unit 131 c and avoice recognition unit 131 d. - The
motion recognition unit 131 c removes a background region from the image transmitted from theimage collection unit 220 to acquire a body image of the user, extracts face, arm, hand, torso, leg, and foot regions from the acquired body image of the user using edge and color information, extracts feature points from the extracted face, arm, hand, torso, leg, and foot regions to recognize a user motion, searches for a motion matching the recognized motion from among motions stored in thedatabase 132, and recognizes an operation command corresponding to the matching motion. - The
voice recognition unit 131 d removes noise from the sound transmitted from thesound collection unit 230 to acquire a voice of the user, extracts a feature vector from the acquired voice of the user, compares the extracted feature vector with a language model or phoneme-based sound model of each word stored in thedatabase 132 to extract candidate words or candidate phonemes, recognizes a sentence based on the candidate words or candidate phonemes, searches for a sentence matching the recognized sentence from among sentences stored in thedatabase 132, and recognizes an operation command corresponding to the searched sentence. - In addition, the
voice recognition unit 131 d may recognize the operation command from the candidate words or candidate phonemes. - Also, the
voice recognition unit 131 d may recognize the user using specific voice information. - The
database 132 stores a face image of a preregistered user, feature point information and pupil information of the face image. Also, thedatabase 132 stores an operation command for each motion and an operation command for each voice. The pupil information includes information regarding size and position of a pupil and the distance between pupils. A user may be recognized based on the pupil information. - In addition, in a case in which a plurality of users is present, the
database 132 may store an operation command corresponding to a motion for each user and an operation command corresponding to a voice for each user. - An IP address is assigned to the
network interface 133 such that thenetwork interface 133 performs network communication and receives and processes a data packet transmitted from the outside through a network. In a case in which the data packet includes multimedia data, i.e. image and sound data, thenetwork interface 133 stores the data packet or transmits the data packet to the controller such that the data packet is reproduced. - For example, the
network interface 133 receives content, such as broadcasts, games, VOD, and broadcast signals, and relevant information. Also, thenetwork interface 133 may receive firmware update information and update files. - In addition, the display apparatus may further include an input interface. The input interface may be connected to an external storage device, such as a USB memory, or an optical disc device, such as a DVD or CD, to receive multimedia data and to transmit the received multimedia data to the controller.
- When the
display apparatus 1 is turned on or an interactive mode is requested through button input of the input unit (not shown), thecontroller 134 transmits an operation command to theimage collection unit 220 and thesound collection unit 230. - The
controller 134 decides whether entry into the interactive mode is to be performed based on the determination as to whether the user recognized by therecognition unit 131 has been registered. In a case in which a plurality of users is recognized by therecognition unit 131, thecontroller 134 confirms a user having highest priority from among the recognized users and controls entry into the interactive mode based on the position of the gaze of the confirmed user. - Also, in a case in which a plurality of users is present, the
controller 134 may determine whether a user's gaze is within a recognition region and, upon determining that the user's gaze is within the recognition region, control entry into the interactive mode based on the gaze of the user pointed at the recognition region. - The
controller 134 determines a position the user looks at based on the position of the user's gaze transmitted from therecognition unit 131. At this time, thecontroller 134 determines whether the user's gaze is within the recognition region for a predetermined period of time or more and, upon determining that the user's gaze is within the recognition region for the predetermined period of time or more, controls entry into the interactive mode. - After entry into the interactive mode, the
controller 134 controls display information of the recognition region of thedisplay unit 110 differently from the display information before entry into the interactive mode. - After entry into the interactive mode, the
controller 134 controls a plurality of recognition modes to receive an operation command through interaction to be displayed. At this time, thecontroller 134 compares the position of the user's gaze transmitted from therecognition unit 131 with positions of prestored recognition mode buttons to determine a recognition mode the user wishes to select and controls theimage collection unit 220 and thesound collection unit 230 to be turned on/off based on the determined recognition mode. - For example, in a case in which a voice recognition mode is selected from among the recognition modes, the
controller 134 controls thesound collection unit 230 to be turned on and theimage collection unit 220 to be turned off. In a case in which a gaze recognition mode or a motion recognition mode is selected from among the recognition modes, thecontroller 134 controls thesound collection unit 230 to be turned off and theimage collection unit 220 to be turned on. - The operation command includes change of output information, such as display or sound, and change of functions.
- The
controller 134 controls driving of therespective drive units recognition unit 131. - When the user's gaze is within the end region for a predetermined period of time or more, the
controller 134 controls the interactive mode to be completed. - When the user's gaze is within the end region for the predetermined period of time or more, the
controller 134 may control a popup menu to be displayed. As shown inFIG. 7 , the popup menu P1 includes a priority change button, volume control button, screen adjustment button, and end button. - In addition, if the user looks at the recognition region during execution of the recognition mode, the
controller 134 may determine that the user wishes to change the recognition mode and control the recognition mode buttons to be redisplayed. - The
controller 134 controls a main image signal requested by the user to be displayed and a sound signal requested by the user to be output and controls an image corresponding to the interactive mode to be displayed on the main image in an overlapping fashion during output of the image and the sound. - The
storage unit 135 stores priorities of a plurality of preregistered users, positions of the recognition region and the end region, before entry into the interactive mode, entry into the interactive mode, display information of the recognition region corresponding to execution of the interactive mode, and position information of the recognition mode buttons. - The
display drive unit 136 drives thedisplay unit 110 based on a command from thecontroller 134, and thesound drive unit 137 drives thesound output unit 140 based on a command from thecontroller 134. - The
display unit 110, which is an output module, displays an image in the display region based on a command from thecontroller 134 in thedrive module 130 and displays an image corresponding to execution of the interactive mode. - The
sound output unit 140, which is another output module, outputs sound based on a command from the controller in the drive module. Thesound output unit 140 may be separated from thedisplay apparatus 1. - That is, the
sound output unit 140, such as a speaker, may be provided separately from thedisplay apparatus 1 and may be connected to thedisplay apparatus 1 as needed. -
FIGS. 8A and 8B are control flowcharts of the display apparatus according to the embodiment. Control of the display apparatus will be described with reference toFIGS. 1 to 7 . - When the display apparatus is turned on by a user, the
image collection unit 220 is operated to collect an image (301). - The display apparatus removes a background region from the collected image to acquire an image of the user, extracts a face region using edge and color information of the acquired image, and extracts feature points, such as eyes, a nose, and a mouth, from the face region to recognize a face (302).
- Subsequently, the display apparatus confirms the number of recognized users based on the number of recognized faces (303) and determines whether the number of the confirmed users is plural (304).
- When only one user has been confirmed, the recognized face is compared with face images of preregistered users to determine whether the user having the recognized face is one of the preregistered users (305). Upon determining that the user having the recognized face is not one of the preregistered users, a message stating that an interactive mode may not be executed is displayed through the display unit 110 (306). Upon determining that the user having the recognized face is one of the preregistered users, a pupil of the recognized face is recognized to track the gaze of the user (310).
- When the number of recognized users is plural, the display apparatus compares the recognized faces with face images of preregistered users to determine whether the users having the recognized faces are the preregistered users, to confirm the number of the users determined as the preregistered users (307), and to determine whether the number of the confirmed users is plural (308).
- Upon determining that a plurality of preregistered users has been recognized, the display apparatus determines a user having highest priority from among the recognized users (309) and recognizes a pupil of the determined user to track the gaze of the user (310).
- Subsequently, the display apparatus determines whether the recognized gaze is within the recognition region (311). Upon determining that the recognized gaze is within the recognition region, the display apparatus determines whether the recognized gaze is pointed at the recognition region for a predetermined period of time or more.
- Upon determining that the recognized gaze is pointed at the recognition region for the predetermined period of time or more, the display apparatus enters an interactive mode (312). In addition, the display apparatus changes display information of the recognition region in the
display unit 110 and displays the changed display information such that entry into the interactive mode is recognized by the user (313). - For example, as shown in
FIG. 3 , an icon may be displayed in therecognition region 110 b or the color of therecognition region 110 b may be changed. - Subsequently, the
display unit 110 displays a plurality of recognition modes to receive an operation command, confirms the position of the user's gaze (314), and determines a recognition mode selected by the user based on the confirmed position of the user's gaze (315). - That is, the display apparatus determines the position of one of the recognition mode buttons matching the confirmed position of the user's gaze to determine the recognition mode selected by the user.
- At this time, the display apparatus displays the position of the user's gaze on the display unit as a cursor in real time such that the user confirms the position of the user's gaze.
- Subsequently, the display apparatus executes the determined recognition mode (316). Also, the display apparatus recognizes an operation command based on the recognition result according to execution of the recognition mode and operates the respective drive units based on the recognized operation command (317).
- Recognition of an operation command based on selection of each recognition mode will hereinafter be described in more detail.
- (1) When motion recognition is selected by a user, the display apparatus operates the image collection unit, removes a background region from an image collected by the image collection unit to acquire a body image of the user, extracts face, arm, hand, torso, leg, and foot regions from the acquired body image of the user using edge and color information, extracts feature points from the extracted face, arm, hand, torso, leg, and foot regions to recognize a user motion, searches for a motion matching the recognized motion from among motions stored in the
database 132, and recognizes an operation command corresponding to the matching motion. - Subsequently, the display apparatus drives the
respective drive units display unit 110 and to output sound of content requested by the user through thesound output unit 140. - (2) When voice recognition is selected by a user, the display apparatus operates the sound collection unit, removes noise from sound collected by the
sound collection unit 230 to acquire a voice of the user, extracts a feature vector from the acquired voice of the user, compares the extracted feature vector with a language model or phoneme-based sound model of each word stored in thedatabase 132 to extract candidate words or candidate phonemes, recognizes a sentence based on the candidate words or candidate phonemes, searches for a sentence matching the recognized sentence from among sentences stored in thedatabase 132, and recognizes an operation command corresponding to the searched sentence. - Subsequently, the display apparatus drives the
respective drive units display unit 110 and to output sound of content requested by the user through thesound output unit 140. - (3) When gaze recognition is selected by a user, the display apparatus operates the image collection unit, displays a popup menu through the
display unit 110, recognizes a face and pupil of the user using an image collected by theimage collection unit 220, tracks the user's gaze based on the recognized pupil, confirms the position of the tracked gaze, and perform an operation corresponding to a menu selected from the popup menu by the user. - The popup menu includes a plurality of menus, and each menu has an operation command button.
- That is, the display apparatus confirms a button having a position matching the position of the user's gaze from among the operation command buttons in the popup menu and performs an operation corresponding to the confirmed button.
- For example, it is assumed that a popup menu P1 including a priority change button, volume control button, screen adjustment button, and end button is displayed as shown in
FIG. 7 . If the position of the user's gaze is present on the volume control button, an up/down button for volume control is displayed on thedisplay unit 110. When the user's gaze is present on the volume up button, volume is turned up. When the user's gaze is present on the volume down button, volume is turned down. - In addition, menu selection through gaze recognition may be determined based on blinking of eyes, time for which the gaze is present on the button, or other action by a user's eye or eyes.
- The display apparatus displays content transmitted through the
network interface 133 or the input interface on the display unit. At this time, the display apparatus controls a main image signal requested by the user to be displayed and a sound signal requested by the user to be output and controls an image corresponding to the interactive mode to be displayed on the main image in an overlapping fashion during output of the image and the sound. - When the user's gaze is within the end region of the
display unit 110 for a predetermined period of time or more, the display apparatus completes the interactive mode. Afterwards, when the user's gaze is within the recognition region, the display apparatus determines that the user wishes to change the recognition mode and redisplays the recognition mode buttons. - In addition, when the user's gaze is outside the
display unit 110 for a predetermined period of time or more, the display apparatus may complete the interactive mode. -
FIG. 9 is a control block diagram of a display apparatus according to another embodiment. In this embodiment, thedisplay apparatus 1 includes animage collection unit 220 and asound collection unit 230 as a collection module to receive a user command, adrive module 150 to perform operation control based on an operation command, and adisplay unit 110 and asound output unit 140 as an output module to output operation execution. - The
image collection unit 220, thesound collection unit 230, thedisplay unit 110, and thesound output unit 140 are identical to those of the previous embodiment, and therefore, a description thereof will be omitted. - The
drive module 150 includes arecognition unit 151, adatabase 152, anetwork interface 153, acontroller 154, astorage unit 155, adisplay drive unit 156, and asound drive unit 157. - The
recognition unit 151 receives a signal corresponding to information collected by theimage collection unit 220 and/or thesound collection unit 230, recognizes user recognition and an operation command desired by a user based on the received signal, and transmits the recognized operation command to thecontroller 154. - The
recognition unit 151 determines whether a user having performed a motion is identical to a user looking at a recognition region and, upon determining that the user having performed a motion is identical to the user looking at the recognition region, recognizes an operation command corresponding to the recognized motion in a state in which the user's gaze is within the recognition region. - Also, the
recognition unit 151 determines whether a user having spoken is identical to a user looking at a recognition region and, upon determining that the user having performed a motion is identical to the user looking at the recognition region, recognizes an operation command corresponding to the recognized voice in a state in which the user's gaze is within the recognition region. - The
recognition unit 151 includes a first recognition unit to recognize the face and gaze of a user and a second recognition unit to recognize a function execution and operation command. The first recognition unit and the second recognition unit are identical to the first recognition unit 131-1 and the second recognition unit 131-2 of the previous embodiment, and therefore, a description thereof will be omitted. - The
database 152 stores a face image and pupil information of each user in a matched state. Also, thedatabase 152 stores voice information and pupil information of each user in a matched state. - The other construction of the
database 152 and the construction of thenetwork interface 153 are identical to that of thedatabase 132 and that of thenetwork interface 133, and therefore, a description thereof will be omitted. - When the
display apparatus 1 is turned on or an interactive mode is requested through button input of an input unit (not shown), thecontroller 154 transmits an operation command to theimage collection unit 220 and thesound collection unit 230. - The
controller 154 decides whether entry into the interactive mode is to be performed based on the determination as to whether the user recognized by therecognition unit 151 has been registered. In a case in which a plurality of users is recognized by therecognition unit 151, thecontroller 154 confirms a user having highest priority from among the recognized users and controls entry into the interactive mode based on the position of the gaze of the confirmed user. - The
controller 154 determines whether the user's gaze is within the recognition region for a predetermined period of time or more and, upon determining that the user's gaze is within the recognition region for the predetermined period of time or more, controls entry into the interactive mode. - After entry into the interactive mode, the
controller 154 controls a plurality of recognition modes to receive a user command through interaction to be displayed. At this time, thecontroller 154 compares the position of the user's gaze transmitted from therecognition unit 151 with positions of prestored recognition mode buttons to determine a recognition mode the user wishes to select and controls theimage collection unit 220 and thesound collection unit 230 to be turned on/off based on the determined recognition mode. - The recognition modes include motion recognition and voice recognition.
- When the motion recognition is selected, the
controller 154 receives an operation command corresponding to a first multi-modal mode including a combination of gaze recognition and motion recognition. When the voice recognition is selected, thecontroller 154 receives an operation command corresponding to a second multi-modal mode including a combination of gaze recognition and voice recognition. - The
controller 154 controls driving of therespective drive units - When the user's gaze is within an end region for a predetermined period of time or more, the
controller 154 controls the interactive mode to be completed. - The
controller 154 controls a main image signal requested by the user to be displayed and a sound signal requested by the user to be output. - The
storage unit 155 stores priorities of a plurality of preregistered users, positions of the recognition region and the end region, before entry into the interactive mode, entry into the interactive mode, display information of the recognition region corresponding to execution of the interactive mode, and position information of the recognition mode buttons. - The
display drive unit 156 and thesound drive unit 157 are identical to those of the previous embodiment, and therefore, a description thereof will be omitted. -
FIG. 10 is a control flowchart of the display apparatus according to the embodiment. Control of the display apparatus will be described with reference toFIGS. 9 and 11 . - When the display apparatus is turned on by a user, the
image collection unit 220 is operated to collect an image. The display apparatus recognizes faces based on the collected image and confirms the number of recognized users based on the recognized faces. - Subsequently, the display apparatus determines whether a plurality of users has been confirmed, determines preregistered users from among the recognized users, and determines whether the number of the preregistered users corresponding to the preregistered users is plural.
- Upon determining that the number of the preregistered users corresponding to the preregistered users is plural, the display apparatus determines a user having highest priority from among the users and recognizes a pupil of the determined user to track the gaze of the user.
- Subsequently, the display apparatus determines whether the recognized gaze is within the recognition region. Upon determining that the recognized gaze is within the recognition region, the display apparatus determines whether the recognized gaze is pointed at the recognition region for a predetermined period of time or more. Upon determining that the recognized gaze is pointed at the recognition region for the predetermined period of time or more, the display apparatus enters an interactive mode (401). In addition, the display apparatus changes display information of the recognition region in the
display unit 110 and displays the changed display information such that entry into the interactive mode is recognized by the user (402). - The above procedure is identical to that of the previous embodiment.
- Subsequently, the
display unit 110 displays a plurality of recognition modes to receive an operation command, confirms the position of the user's gaze (403), and determines a recognition mode selected by the user based on the confirmed position of the user's gaze (404). - That is, the display apparatus determines the position of one of the recognition mode buttons matching the confirmed position of the user's gaze to determine the recognition mode selected by the user.
- At this time, the display apparatus displays the position of the user's gaze in the
display unit 110 as a cursor in real time such that the user confirms the position of the user's gaze. - Subsequently, the display apparatus executes a multi-modal mode including a combination of the determined recognition mode and a gaze recognition mode (405). Also, the display apparatus recognizes an operation command based on the recognition result according to execution of the multi-modal mode and operates the respective drive units based on the recognized operation command (406).
- Before executing a first multi-modal mode including a combination of a motion recognition mode and a gaze recognition mode, the display apparatus determines whether a user having performed a motion is identical to a user looking at the recognition region and, upon determining that the user having performed a motion is identical to the user looking at the recognition region, executes an interactive mode based on the first multi-modal mode. Upon determining that the user having performed a motion is not identical to the user looking at the recognition region, a message stating that it is not possible to execute an interactive mode may be displayed.
- Also, before executing a second multi-modal mode including a combination of a voice recognition mode and a gaze recognition mode, the display apparatus determines whether a user having spoken is identical to a user looking at the recognition region and, upon determining that the user having spoken is identical to the user looking at the recognition region, executes an interactive mode based on the second multi-modal mode. Upon determining that the user having spoken is not identical to the user looking at the recognition region, a message stating that it is not possible to execute an interactive mode may be displayed.
- Recognition of an operation command based on the multi-modal mode will hereinafter be described in more detail.
- (1) When motion recognition is selected by a user, the display apparatus executes a first multi-modal mode including a combination of a motion recognition mode and a gaze recognition mode. At this time, the display apparatus operates the
image collection unit 220, determines whether a user's gaze is within a recognition region and, upon determining that the user's gaze is within the recognition region, recognizes a user motion in a state in which the user's gaze is within the recognition region, and recognizes an operation command corresponding to the recognized motion. - That is, the display apparatus removes a background region from an image collected by the
image collection unit 220 to acquire a body image of the user, extracts face, arm, hand, torso, leg, and foot regions from the acquired body image of the user using edge and color information, extracts feature points from the extracted face, arm, hand, torso, leg, and foot regions to recognize a user motion, searches for a motion matching the recognized motion from among motions stored in thedatabase 152, and recognizes an operation command corresponding to the matching motion. - Subsequently, the display apparatus drives the
respective drive units display unit 110 and to output sound of content requested by the user through thesound output unit 140. - This example will be described with reference to
FIG. 11 . - When the first multi-modal mode is executed in a state in which a plurality of functions is displayed in the
display region 110 a of the display unit as shown inFIG. 11 , the display apparatus recognizes a user's gaze and motion using the image collection unit. - On the assumption that the user is looking at the
recognition region 110 b and, at the same time, has performed a one finger stretching motion, the display apparatus recognizes the user's gaze within the recognition region and, at the same time, the one finger stretching motion, and recognizes an operation command corresponding to the one finger stretching motion. - When the operation command corresponding to the one finger stretching motion is ‘search’, the display apparatus is connected to the Internet via the network interface to perform search.
- (2) When voice recognition is selected by a user, the display apparatus executes a second multi-modal mode including a combination of a voice recognition mode and a gaze recognition mode. At this time, the display apparatus operates the
sound collection unit 230, determines whether a user's gaze is within a recognition region and, upon determining that the user's gaze is within the recognition region, recognizes a voice of the user in a state in which the user's gaze is within the recognition region, and recognizes an operation command corresponding to the recognized voice. - That is, the display apparatus operates the
sound collection unit 230, removes noise from sound collected by thesound collection unit 230 to acquire a voice of the user, extracts a feature vector from the acquired voice of the user, compares the extracted feature vector with a language model or phoneme-based sound model of each word stored in thedatabase 152 to extract candidate words or candidate phonemes, recognizes a sentence based on the candidate words or candidate phonemes, searches for a sentence matching the recognized sentence from among sentences stored in thedatabase 152, and recognizes an operation command corresponding to the searched sentence. - Subsequently, the display apparatus drives the
respective drive units display unit 110 and to output sound of content requested by the user through thesound output unit 140. - This example will be described with reference to
FIG. 11 . - When the second multi-modal mode is executed in a state in which a plurality of functions is displayed in the
display region 110 a of the display unit as shown inFIG. 11 , the display apparatus recognizes a user's gaze and voice using the image collection unit and the sound collection unit. - On the assumption that the user is looking at the
recognition region 110 b and, at the same time, has said a word ‘three’, the display apparatus recognizes the user's gaze within the recognition region and, at the same time, the word ‘three’, and recognizes an operation command corresponding to ‘three’. - When the operation command corresponding to ‘three’ is a ‘movie’, the display apparatus reproduces a movie downloaded through the network interface or a movie stored in an additional storage device or displays a submenu of the movie.
- When an interactive mode is executed during display of a main image in the display unit, an image corresponding to the interactive mode is displayed on the main image in an overlapping fashion.
- When the user's gaze is within the end region of the
display unit 110 for a predetermined period of time or more, the display apparatus completes the interactive mode. Afterwards, when the user's gaze is within the end region, the display apparatus reperforms entry into the interactive mode and redisplays it. -
FIG. 12 is a view illustrating a display apparatus according to a further embodiment. Hereinafter, a television will be described as an example of the display apparatus. - In this embodiment, the television includes a
main body 100 to display an image, astand 160 mounted to the bottom of themain body 100 to support themain body 100, and aremote controller 170 to receive a user command. - The
stand 160 is detachably coupled to themain body 100. The television may further include a bracket. Themain body 100 may be mounted to a wall using the bracket. In this case, the stand may be removed. - The
main body 100 includes adisplay unit 110 to display an image and abezel 120 to cover an image non-display region. The construction of themain body 100 is identical to that of the previous embodiment, and therefore, a description thereof will be omitted. - The
main body 100 may further include an input unit (not shown) having a plurality of buttons to receive a television operation command. - A
collection module 200 is provided at the upper end of thebezel 120 of themain body 100 to collect an image and sound of a user and surroundings of the user and to transmit the collected image and sound to a drive module. - The
collection module 200 may be mounted at a position where user information may be easily collected in addition to the upper end of thebezel 120 of themain body 100. - Also, the
collection module 200 may be provided separately from the television. That is, thecollection module 200 may be provided separately from themain body 100 such that thecollection module 200 is connected to an interface device (not shown) provided at themain body 100. - The
collection module 200 is identical to that of the previous embodiment, and therefore, a description thereof will be omitted. - The
remote controller 170 is a device to remotely control television for user convenience. - The
remote controller 170 includes a plurality ofbuttons 171, such as function selection buttons, number buttons, power on/off buttons, channel change buttons, and volume control buttons, a second sound collection unit, i.e. asecond microphone 172, to collect a voice signal of a user, a second microprocessor (not shown) to generate an operation signal upon receiving a signal from thebuttons 171 and the voice signal, acommunication unit 173 to transmit the operation signal generated by the second microprocessor, and a battery to supply driving power to the second microphone, the buttons, the second microprocessor, and the communication unit. - The second
sound collection unit 172 may constantly operate. - The
remote controller 170 transmits an operation signal, such as a channel change signal or a volume control signal, to the television in response to user manipulation. As a result, the user may conveniently adjust various functions of the television. - In addition, the remote controller may further include a plurality of buttons to select and control an Internet function in addition to the broadcast viewing control buttons in order to control a display apparatus supporting the Internet in addition to broadcast viewing.
-
FIG. 13 is a control block diagram of the display apparatus, i.e. the television, according to the embodiment. - The television is operated based on interaction information generated through collection of a user command, a signal from the input unit (not shown) provided at the
main body 100, or a signal from theremote controller 170. - The television includes an
image collection unit 220 and asound collection unit 230 as a collection module to collect a user command, thedrive module 180 to control the operation of the television based on the user command, and adisplay unit 110 and asound output unit 140 as an output module to output operation execution. - The
drive module 180 controls the operation of thedisplay unit 110 and/or thesound output unit 140 based on the signal from the input unit (not shown) or the signal from theremote controller 170. - That is, the
drive module 180 controls entry into and end of an interactive mode based on information collected by theimage collection unit 220 and/or thesound collection unit 230 and, after entry into the interactive mode, executes the interactive mode to control the operation of thedisplay unit 110 and/or thesound output unit 140. - The
drive module 180 of the television includes abroadcast unit 181, aninterface 182, arecognition unit 183, adatabase 184, a receivingunit 185, acontroller 186, astorage unit 187, and adrive unit 188. - The
broadcast unit 181 includes atuner 181 a and asignal processor 181 b. - The
tuner 181 a receives a broadcast signal having a corresponding frequency band according to a tuning control signal from thecontroller 186 in a wired or wireless fashion to tune the television. - The
signal processor 181 b separates the broadcast signal into an image signal having image data, a sound signal having sound data, and an additional signal having additional data, signal-processes the separated image signal having image data and sound signal having sound data, and transmits the additional data to a GUI generation unit (not shown). The additional data includes an electronic program guide (EPG). - The
signal processor 181 b may separate a signal transmitted from anetwork interface 182 b into an image signal having image data and a sound signal having sound data and signal-process the image signal and the sound signal. - The
interface unit 182 includes aninput interface 182 a and anetwork interface 182 b. - The
input interface 182 a is connected to thecollection module 200 to receive an image signal and sound signal collected by thecollection module 200 and to transmit the received image signal and sound signal to therecognition unit 183. - An IP address is assigned to the
network interface 182 b such that thenetwork interface 182 b performs network communication and receives and processes a data packet transmitted from the outside through a network. In a case in which the data packet includes multimedia data, i.e. image and sound data, thenetwork interface 182 b stores the data packet or transmits the data packet to thecontroller 186 such that the data packet is reproduced. - The
recognition unit 183 recognizes a face and pupil of a user, recognizes the user's gaze based on information of the recognized pupil, and recognizes a user command. - The
recognition unit 183 recognizes a voice of the user from the sound signal transmitted from the receivingunit 185 and recognizes an operation command corresponding to the recognized voice. - The other construction of the
recognition unit 183 is identical to that of the previous embodiment, and therefore, a description thereof will be omitted. - The
database 184 is identical to that of the previous embodiment, and therefore, a description thereof will be omitted. - The receiving
unit 185 receives a signal from one of thebuttons 171 provided at theremote controller 170 and a sound signal from the secondsound collection unit 172 provided at theremote controller 170. - The receiving
unit 185 transmits the received button signal and sound signal to therecognition unit 183 and thecontroller 186. - The
controller 186 controls the image data signal-processed by thesignal processor 181 b to be displayed on thedisplay unit 110 and the signal-processed sound to be output through thesound output unit 140. - The
controller 186 determines a user command based on the button signal from theremote controller 170 and controls driving of thedisplay unit 110 according to the determined user command. Upon receiving an Internet function execution request signal from theremote controller 170, thecontroller 186 controls the network interface for Internet connection. When the user's gaze is within the recognition region, thecontroller 186 controls driving of the collection module to execute the interactive mode. - The construction of the
controller 186 to control entry into the interactive mode and execution of the interactive mode is identical to that of the previous embodiment, and therefore, a description thereof will be omitted. - The
storage unit 187 is identical to that of the previous embodiment, and therefore, a description thereof will be omitted. - The
drive unit 188 includes adisplay drive unit 188 a and asound drive unit 188 b. Thedisplay drive unit 188 a and thesound drive unit 188 b are identical to those of the previous embodiment, and therefore, a description thereof will be omitted. -
FIG. 14 is a control flowchart of the display apparatus, i.e. the television, according to the embodiment. - When the television is turned on, the
image collection unit 220 is operated to collect an image. The television recognizes a face of a user based on the collected image, recognizes a pupil of the user based on the recognized face, and recognizes a gaze corresponding to the position of the pupil (501). - Subsequently, the television determines whether the recognized gaze is within the recognition region (502). Upon determining that the recognized gaze is within the recognition region, the television determines whether the recognized gaze is pointed at the recognition region for a predetermined period of time or more (503). Upon determining that the recognized gaze is pointed at the recognition region for the predetermined period of time or more, the television enters an interactive mode (504). In addition, the television changes display information of the recognition region in the
display unit 110 and displays the changed display information such that entry into the interactive mode is recognized by the user such as shown inoperation 402 ofFIG. 10 . - The above procedure is identical to that of the previous embodiment.
- Subsequently, the television displays a plurality of recognition modes to receive an operation command (505), confirms the position of the user's gaze, and determines a recognition mode selected by the user based on the confirmed position of the user's gaze (506).
- That is, the television determines the position of one of the recognition mode buttons matching the confirmed position of the user's gaze to determine the recognition mode selected by the user.
- At this time, the television displays the position of the user's gaze in the
display unit 110 as a cursor in real time such that the user may confirm the position of the user's gaze. - Subsequently, the television executes the determined recognition mode (507). Also, the television recognizes an operation command based on the recognition result according to execution of the recognition mode and operates the respective drive units based on the recognized operation command (508).
- Recognition of an operation command based on selection of each recognition mode will hereinafter be described in more detail.
- (1) When motion recognition is selected by a user, the television recognizes a user motion, searches for a motion matching the recognized motion from among motions stored in the
database 184, and recognizes an operation command corresponding to the matching motion. - Subsequently, the television drives the
drive unit 188 based on the recognized operation command to display an image of a broadcast or content requested by the user through thedisplay unit 110 and to output sound of a broadcast or content requested by the user through thesound output unit 140. - (2) When voice recognition is selected by a user, the television operates the first
sound collection unit 230, recognizes a sentence corresponding to a voice in sound collected by the firstsound collection unit 230 or the secondsound collection unit 172, searches for a sentence matching the recognized sentence from among sentences stored in thedatabase 184, and recognizes an operation command corresponding to the searched sentence. - Subsequently, the television drives the
drive unit 188 based on the recognized operation command to display an image of a broadcast or content requested by the user through thedisplay unit 110 and to output sound of a broadcast or content requested by the user through thesound output unit 140. - (3) When gaze recognition is selected by a user, the television operates the image collection unit, displays a popup menu through the
display unit 110, recognizes a face and pupil of the user using an image collected by theimage collection unit 220, tracks the user's gaze based on the recognized pupil, confirms the position of the tracked gaze, and performs an operation corresponding to a menu item selected by the user from the popup menu. - The popup menu includes a plurality of menus, and each menu has an operation command button.
- That is, the television confirms a button having a position matching the position of the user's gaze from among the operation command buttons in the popup menu and performs an operation corresponding to the confirmed button, which will be described with reference to
FIGS. 15 and 16 . - In a case in which a popup menu P2 including a function change button, channel/volume control button, recognition mode change button, screen adjustment button, and end button is displayed in the
display unit 110 as shown inFIG. 15 , a button having a position matching the position of a user's gaze is determined. The function includes a broadcast function, Internet function, and communication function with an external device. - If the position of the user's gaze is present on the channel/volume control button, a volume up/down button a5 for volume control, a channel up/down button a6 for channel adjustment, and a cursor a4 indicating the position of the user's gaze are displayed in the
display unit 110 as shown inFIG. 16 . - Subsequently, the television compares the position of the user's gaze with the position of the volume up/down button a5 and the channel up/down button a6 to turn the channel/volume up/down.
- In addition, menu selection through gaze recognition may be determined based on blinking of eyes or time for which the gaze is present on the button.
- At this time, the television displays a button for gaze recognition on a main image displayed in the display unit in an overlapping fashion.
- When the user's gaze is within the end region of the
display unit 110 for a predetermined period of time or more, the television completes the interactive mode. Afterwards, when the user's gaze is within the recognition region, the television determines that the user wishes to change the recognition mode and redisplays the recognition mode buttons. - In addition, when the user's gaze is outside the
display unit 110 for a predetermined period of time or more, the television may complete the interactive mode. - In accordance with an aspect of the present disclosure as described above, a user's gaze is tracked to perform entry into an interactive mode, thereby easily achieving entry into the interactive mode and performing more intuitive interaction.
- Also, a multi-modal interactive mode including a combination of face recognition, voice recognition, and gaze recognition is performed, thereby performing a more extended interactive mode and accurately determining a user command. As a result, functions are correctly performed, thereby improving user convenience.
- In addition, the position of a user's gaze is displayed on a display apparatus as a cursor, thereby correctly confirming the position of the user's gaze.
- Additionally, a user is recognized from a face or pupil of the user, and a gaze, voice, and motion are recognized considering features of the recognized user, thereby improving recognition efficiency.
- Although a few embodiments of the present disclosure have been shown and described, it would be appreciated by those skilled in the art that changes may be made in these embodiments without departing from the principles and spirit of the invention, the scope of which is defined in the claims and their equivalents.
Claims (10)
Priority Applications (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US15/348,061 US20170060370A1 (en) | 2012-10-19 | 2016-11-10 | Display apparatus and control method thereof |
Applications Claiming Priority (4)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
KR1020120116956A KR102095765B1 (en) | 2012-10-19 | 2012-10-19 | Display apparatus and method for controlling the same |
KR10-2012-0116956 | 2012-10-19 | ||
US14/056,472 US9524023B2 (en) | 2012-10-19 | 2013-10-17 | Display apparatus and control method thereof |
US15/348,061 US20170060370A1 (en) | 2012-10-19 | 2016-11-10 | Display apparatus and control method thereof |
Related Parent Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US14/056,472 Division US9524023B2 (en) | 2012-10-19 | 2013-10-17 | Display apparatus and control method thereof |
Publications (1)
Publication Number | Publication Date |
---|---|
US20170060370A1 true US20170060370A1 (en) | 2017-03-02 |
Family
ID=49518656
Family Applications (2)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US14/056,472 Active 2034-03-02 US9524023B2 (en) | 2012-10-19 | 2013-10-17 | Display apparatus and control method thereof |
US15/348,061 Abandoned US20170060370A1 (en) | 2012-10-19 | 2016-11-10 | Display apparatus and control method thereof |
Family Applications Before (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US14/056,472 Active 2034-03-02 US9524023B2 (en) | 2012-10-19 | 2013-10-17 | Display apparatus and control method thereof |
Country Status (9)
Country | Link |
---|---|
US (2) | US9524023B2 (en) |
EP (2) | EP3425485B1 (en) |
JP (1) | JP6271935B2 (en) |
KR (1) | KR102095765B1 (en) |
CN (1) | CN103826145B (en) |
IN (1) | IN2015DN03276A (en) |
MX (1) | MX346605B (en) |
RU (1) | RU2617927C2 (en) |
WO (1) | WO2014061916A1 (en) |
Cited By (4)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US10540009B2 (en) | 2015-02-25 | 2020-01-21 | Kyocera Corporation | Wearable device, control method, and control program |
US10962785B2 (en) | 2016-12-19 | 2021-03-30 | Samsung Electronics Co., Ltd. | Electronic device, mobile electronic device, and display device, which are controlled on basis of user gaze and input |
US11470239B2 (en) * | 2019-07-31 | 2022-10-11 | Canon Kabushiki Kaisha | Electronic device for receiving line of sight input, method of controlling electronic device, and non-transitory computer readable medium |
US11949943B2 (en) * | 2018-07-16 | 2024-04-02 | Arris Enterprises Llc | Gaze-responsive advertisement |
Families Citing this family (39)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
JP2015056141A (en) | 2013-09-13 | 2015-03-23 | ソニー株式会社 | Information processing device and information processing method |
CN104598009A (en) * | 2013-10-30 | 2015-05-06 | 鸿富锦精密工业(武汉)有限公司 | Screen button control method and system |
JP5884811B2 (en) * | 2013-11-18 | 2016-03-15 | コニカミノルタ株式会社 | AR display device, AR display control device, printing condition setting system, printing system, printing setting display method and program |
KR102193029B1 (en) * | 2014-05-09 | 2020-12-18 | 삼성전자주식회사 | Display apparatus and method for performing videotelephony using the same |
KR102337682B1 (en) | 2014-10-01 | 2021-12-09 | 삼성전자주식회사 | Display apparatus and Method for controlling thereof |
KR102289904B1 (en) * | 2015-01-23 | 2021-08-18 | 삼성디스플레이 주식회사 | Display apparatus |
KR20160116162A (en) * | 2015-03-26 | 2016-10-07 | 삼성디스플레이 주식회사 | Display apparatus |
CN105630148A (en) * | 2015-08-07 | 2016-06-01 | 宇龙计算机通信科技(深圳)有限公司 | Terminal display method, terminal display apparatus and terminal |
JP2019003228A (en) * | 2017-06-09 | 2019-01-10 | 富士通株式会社 | Equipment cooperation system, equipment cooperation device, equipment cooperation method, and equipment cooperation program |
KR102203720B1 (en) * | 2017-06-26 | 2021-01-15 | 에스케이텔레콤 주식회사 | Method and apparatus for speech recognition |
US10732708B1 (en) * | 2017-11-21 | 2020-08-04 | Amazon Technologies, Inc. | Disambiguation of virtual reality information using multi-modal data including speech |
US11232645B1 (en) | 2017-11-21 | 2022-01-25 | Amazon Technologies, Inc. | Virtual spaces as a platform |
US10521946B1 (en) | 2017-11-21 | 2019-12-31 | Amazon Technologies, Inc. | Processing speech to drive animations on avatars |
CN114860187A (en) * | 2018-01-03 | 2022-08-05 | 腾讯科技(深圳)有限公司 | Intelligent voice equipment control method and device, computer equipment and storage medium |
US10981067B2 (en) | 2018-02-06 | 2021-04-20 | Gree, Inc. | Game processing system, method of processing game, and storage medium storing program for processing game |
JP6908573B2 (en) | 2018-02-06 | 2021-07-28 | グリー株式会社 | Game processing system, game processing method, and game processing program |
JP6850769B2 (en) * | 2018-07-05 | 2021-03-31 | グリー株式会社 | Game processing system, game processing method, and game processing program |
CN116328317A (en) | 2018-02-06 | 2023-06-27 | 日本聚逸株式会社 | Application processing system, application processing method, and application processing program |
JP6855416B2 (en) * | 2018-07-06 | 2021-04-07 | グリー株式会社 | Game processing system, game processing method, and game processing program |
EP3520868A1 (en) | 2018-02-06 | 2019-08-07 | Gree, Inc. | Game processing system, method of processing game, and program for processing game |
CN108509856A (en) * | 2018-03-06 | 2018-09-07 | 深圳市沃特沃德股份有限公司 | Audio regulation method, device and stereo set |
US11169668B2 (en) * | 2018-05-16 | 2021-11-09 | Google Llc | Selecting an input mode for a virtual assistant |
CN108897589B (en) * | 2018-05-31 | 2020-10-27 | 刘国华 | Human-computer interaction method and device in display equipment, computer equipment and storage medium |
JP6524320B2 (en) * | 2018-09-12 | 2019-06-05 | 株式会社東芝 | Glasses-type wearable terminal and method using the same |
CN109683705A (en) * | 2018-11-30 | 2019-04-26 | 北京七鑫易维信息技术有限公司 | The methods, devices and systems of eyeball fixes control interactive controls |
US20220028380A1 (en) * | 2018-12-11 | 2022-01-27 | Lg Electronics Inc. | Display device |
US20220050580A1 (en) * | 2019-01-28 | 2022-02-17 | Sony Group Corporation | Information processing apparatus, information processing method, and program |
CN109788256A (en) * | 2019-01-30 | 2019-05-21 | 温州大学 | A kind of display system of multi-modality images |
CN110211251A (en) * | 2019-04-26 | 2019-09-06 | 珠海格力电器股份有限公司 | A kind of face identification method, device, storage medium and recognition of face terminal |
CN112423190A (en) * | 2019-08-20 | 2021-02-26 | 苹果公司 | Audio-based feedback for head-mounted devices |
US11112875B1 (en) * | 2020-03-20 | 2021-09-07 | Huawei Technologies Co., Ltd. | Methods and systems for controlling a device using hand gestures in multi-user environment |
US11698677B1 (en) * | 2020-06-29 | 2023-07-11 | Apple Inc. | Presenting a notification based on an engagement score and an interruption priority value |
CN112102828A (en) * | 2020-09-04 | 2020-12-18 | 杭州中软安人网络通信股份有限公司 | Voice control method and system for automatically broadcasting content on large screen |
US11762458B2 (en) * | 2021-02-15 | 2023-09-19 | Sony Group Corporation | Media display device control based on eye gaze |
JP7039753B1 (en) | 2021-07-01 | 2022-03-22 | 株式会社あかつき | Game server, game program, information processing method |
JP7089128B1 (en) | 2021-07-01 | 2022-06-21 | 株式会社あかつき | Game server, game program, information processing method |
JP7141556B1 (en) | 2022-03-09 | 2022-09-22 | 株式会社あかつき | Game server, game program, information processing method |
JP7248853B2 (en) * | 2022-03-09 | 2023-03-29 | 株式会社あかつき | Game server, game program, information processing method |
JP7387932B2 (en) * | 2022-03-09 | 2023-11-28 | 株式会社あかつき | Game server, game program, information processing method |
Citations (16)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20060061962A1 (en) * | 2004-09-22 | 2006-03-23 | Intel Corporation | Retractable and extendable camera and microphone array computer system |
US20060129408A1 (en) * | 2004-12-14 | 2006-06-15 | Jia-Lin Shen | A remote control device and method with speech control |
US20080013262A1 (en) * | 2006-06-30 | 2008-01-17 | Logitech Europe S.A. | Computer monitor with detachable module |
US20090079813A1 (en) * | 2007-09-24 | 2009-03-26 | Gesturetek, Inc. | Enhanced Interface for Voice and Video Communications |
US20090122021A1 (en) * | 2007-11-09 | 2009-05-14 | Chen-Yu Liu | Touch screen display with electric field shielding layer |
US20090217211A1 (en) * | 2008-02-27 | 2009-08-27 | Gesturetek, Inc. | Enhanced input using recognized gestures |
US20090278764A1 (en) * | 2007-08-29 | 2009-11-12 | Nintendo Co., Ltd. | Imaging apparatus |
US20120046101A1 (en) * | 2002-07-27 | 2012-02-23 | Sony Computer Entertainment Inc. | Apparatus for image and sound capture in a game environment |
US20120162437A1 (en) * | 2010-12-24 | 2012-06-28 | Kabushiki Kaisha Toshiba | Electronic apparatus |
WO2012083415A1 (en) * | 2010-11-15 | 2012-06-28 | Tandemlaunch Technologies Inc. | System and method for interacting with and analyzing media on a display using eye gaze tracking |
US20120201050A1 (en) * | 2011-02-08 | 2012-08-09 | Samsung Electronics Co., Ltd. | Backlight unit and display apparatus having the same |
US20120327327A1 (en) * | 2011-06-27 | 2012-12-27 | Wonseok Joo | Mobile terminal |
US20130027302A1 (en) * | 2011-07-25 | 2013-01-31 | Kyocera Corporation | Electronic device, electronic document control program, and electronic document control method |
US20130044055A1 (en) * | 2011-08-20 | 2013-02-21 | Amit Vishram Karmarkar | Method and system of user authentication with bioresponse data |
US20130307771A1 (en) * | 2012-05-18 | 2013-11-21 | Microsoft Corporation | Interaction and management of devices using gaze detection |
US8922480B1 (en) * | 2010-03-05 | 2014-12-30 | Amazon Technologies, Inc. | Viewer-based device control |
Family Cites Families (32)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
JPH0981309A (en) * | 1995-09-13 | 1997-03-28 | Toshiba Corp | Input device |
DE19736995B4 (en) * | 1997-08-26 | 2009-05-07 | Fraunhofer-Gesellschaft zur Förderung der angewandten Forschung e.V. | Device for determining a fixation point |
US7216351B1 (en) * | 1999-04-07 | 2007-05-08 | International Business Machines Corporation | Systems and methods for synchronizing multi-modal interactions |
JP2001070293A (en) * | 1999-09-06 | 2001-03-21 | Toshiba Corp | Radio-diagnostic device |
JP2001350578A (en) * | 2000-06-06 | 2001-12-21 | Sensor:Kk | Device and method for input control by visual line and recording medium with control program to be inputted by visual line recorded thereon |
US20030038754A1 (en) * | 2001-08-22 | 2003-02-27 | Mikael Goldstein | Method and apparatus for gaze responsive text presentation in RSVP display |
JP2005117185A (en) * | 2003-10-03 | 2005-04-28 | Canon Inc | Digital television set |
MXPA06014097A (en) * | 2004-06-04 | 2007-03-07 | Qualcomm Inc | High data rate interface apparatus and method. |
JP4872241B2 (en) * | 2005-05-31 | 2012-02-08 | 船井電機株式会社 | TV receiver |
JP2007160473A (en) * | 2005-12-15 | 2007-06-28 | Fujitsu Ltd | Interactive object identifying method in robot and robot |
DE102006011288A1 (en) * | 2006-03-10 | 2007-09-13 | Siemens Ag | Method for selecting functions using a user interface and user interface |
JP2007324866A (en) * | 2006-05-31 | 2007-12-13 | Funai Electric Co Ltd | Electronic apparatus and television receiver |
DE102008025124A1 (en) * | 2008-05-26 | 2009-12-03 | Volkswagen Ag | Display system operating method for e.g. passenger car, involves generating control signal based on operating information and adjusted mode of operation to control function of display system |
KR101427686B1 (en) * | 2008-06-09 | 2014-08-12 | 삼성전자주식회사 | The method for selecting program and the apparatus thereof |
US8700332B2 (en) * | 2008-11-10 | 2014-04-15 | Volkswagen Ag | Operating device for a motor vehicle |
KR20100064177A (en) * | 2008-12-04 | 2010-06-14 | 삼성전자주식회사 | Electronic device and method for displaying |
WO2010101697A2 (en) * | 2009-02-06 | 2010-09-10 | Oculis Labs, Inc. | Video-based privacy supporting system |
US20110025689A1 (en) * | 2009-07-29 | 2011-02-03 | Microsoft Corporation | Auto-Generating A Visual Representation |
US20110084897A1 (en) * | 2009-10-13 | 2011-04-14 | Sony Ericsson Mobile Communications Ab | Electronic device |
KR101626159B1 (en) * | 2009-11-25 | 2016-05-31 | 엘지전자 주식회사 | User adaptive display device and method thereof |
US20110154014A1 (en) * | 2009-12-18 | 2011-06-23 | Sony Ericsson Mobile Communications Ab | Data exchange for mobile devices |
US8977972B2 (en) * | 2009-12-31 | 2015-03-10 | Intel Corporation | Using multi-modal input to control multiple objects on a display |
KR101815020B1 (en) * | 2010-08-26 | 2018-01-31 | 삼성전자주식회사 | Apparatus and Method for Controlling Interface |
JP2012053545A (en) * | 2010-08-31 | 2012-03-15 | Canon Inc | Image processing system, and method for controlling the same |
JP5583531B2 (en) * | 2010-09-10 | 2014-09-03 | 日立コンシューマエレクトロニクス株式会社 | Video display device |
CN103329066B (en) * | 2011-01-19 | 2017-03-29 | 惠普发展公司,有限责任合伙企业 | For the method and system of multi-mode gesture control |
CN103339026A (en) * | 2011-01-31 | 2013-10-02 | 三洋电机株式会社 | Electric bicycle |
KR101191640B1 (en) * | 2011-02-23 | 2012-10-17 | 동국대학교 산학협력단 | Apparatus and method for providing information to user |
US8643680B2 (en) * | 2011-04-08 | 2014-02-04 | Amazon Technologies, Inc. | Gaze-based content display |
US9727132B2 (en) * | 2011-07-01 | 2017-08-08 | Microsoft Technology Licensing, Llc | Multi-visor: managing applications in augmented reality environments |
US9594500B2 (en) * | 2012-06-27 | 2017-03-14 | sigmund lindsay clements | Touch Free hygienic display control panel for a smart toilet |
KR102081930B1 (en) * | 2013-03-21 | 2020-02-26 | 엘지전자 주식회사 | Display device detecting gaze location and method for controlling thereof |
-
2012
- 2012-10-19 KR KR1020120116956A patent/KR102095765B1/en active IP Right Grant
-
2013
- 2013-09-06 IN IN3276DEN2015 patent/IN2015DN03276A/en unknown
- 2013-09-06 MX MX2015004841A patent/MX346605B/en active IP Right Grant
- 2013-09-06 RU RU2015114537A patent/RU2617927C2/en active
- 2013-09-06 WO PCT/KR2013/008073 patent/WO2014061916A1/en active Application Filing
- 2013-10-08 JP JP2013210769A patent/JP6271935B2/en not_active Expired - Fee Related
- 2013-10-15 EP EP18191156.1A patent/EP3425485B1/en active Active
- 2013-10-15 EP EP13188743.2A patent/EP2735938B1/en active Active
- 2013-10-17 CN CN201310486543.4A patent/CN103826145B/en active Active
- 2013-10-17 US US14/056,472 patent/US9524023B2/en active Active
-
2016
- 2016-11-10 US US15/348,061 patent/US20170060370A1/en not_active Abandoned
Patent Citations (16)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20120046101A1 (en) * | 2002-07-27 | 2012-02-23 | Sony Computer Entertainment Inc. | Apparatus for image and sound capture in a game environment |
US20060061962A1 (en) * | 2004-09-22 | 2006-03-23 | Intel Corporation | Retractable and extendable camera and microphone array computer system |
US20060129408A1 (en) * | 2004-12-14 | 2006-06-15 | Jia-Lin Shen | A remote control device and method with speech control |
US20080013262A1 (en) * | 2006-06-30 | 2008-01-17 | Logitech Europe S.A. | Computer monitor with detachable module |
US20090278764A1 (en) * | 2007-08-29 | 2009-11-12 | Nintendo Co., Ltd. | Imaging apparatus |
US20090079813A1 (en) * | 2007-09-24 | 2009-03-26 | Gesturetek, Inc. | Enhanced Interface for Voice and Video Communications |
US20090122021A1 (en) * | 2007-11-09 | 2009-05-14 | Chen-Yu Liu | Touch screen display with electric field shielding layer |
US20090217211A1 (en) * | 2008-02-27 | 2009-08-27 | Gesturetek, Inc. | Enhanced input using recognized gestures |
US8922480B1 (en) * | 2010-03-05 | 2014-12-30 | Amazon Technologies, Inc. | Viewer-based device control |
WO2012083415A1 (en) * | 2010-11-15 | 2012-06-28 | Tandemlaunch Technologies Inc. | System and method for interacting with and analyzing media on a display using eye gaze tracking |
US20120162437A1 (en) * | 2010-12-24 | 2012-06-28 | Kabushiki Kaisha Toshiba | Electronic apparatus |
US20120201050A1 (en) * | 2011-02-08 | 2012-08-09 | Samsung Electronics Co., Ltd. | Backlight unit and display apparatus having the same |
US20120327327A1 (en) * | 2011-06-27 | 2012-12-27 | Wonseok Joo | Mobile terminal |
US20130027302A1 (en) * | 2011-07-25 | 2013-01-31 | Kyocera Corporation | Electronic device, electronic document control program, and electronic document control method |
US20130044055A1 (en) * | 2011-08-20 | 2013-02-21 | Amit Vishram Karmarkar | Method and system of user authentication with bioresponse data |
US20130307771A1 (en) * | 2012-05-18 | 2013-11-21 | Microsoft Corporation | Interaction and management of devices using gaze detection |
Cited By (4)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US10540009B2 (en) | 2015-02-25 | 2020-01-21 | Kyocera Corporation | Wearable device, control method, and control program |
US10962785B2 (en) | 2016-12-19 | 2021-03-30 | Samsung Electronics Co., Ltd. | Electronic device, mobile electronic device, and display device, which are controlled on basis of user gaze and input |
US11949943B2 (en) * | 2018-07-16 | 2024-04-02 | Arris Enterprises Llc | Gaze-responsive advertisement |
US11470239B2 (en) * | 2019-07-31 | 2022-10-11 | Canon Kabushiki Kaisha | Electronic device for receiving line of sight input, method of controlling electronic device, and non-transitory computer readable medium |
Also Published As
Publication number | Publication date |
---|---|
EP2735938A2 (en) | 2014-05-28 |
CN103826145B (en) | 2019-02-15 |
EP3425485A3 (en) | 2019-02-13 |
MX2015004841A (en) | 2015-07-21 |
US20140111420A1 (en) | 2014-04-24 |
JP2014086085A (en) | 2014-05-12 |
WO2014061916A1 (en) | 2014-04-24 |
EP2735938A3 (en) | 2015-01-07 |
KR20140050484A (en) | 2014-04-29 |
EP2735938B1 (en) | 2018-08-29 |
CN103826145A (en) | 2014-05-28 |
JP6271935B2 (en) | 2018-01-31 |
IN2015DN03276A (en) | 2015-10-09 |
EP3425485B1 (en) | 2020-12-02 |
KR102095765B1 (en) | 2020-04-01 |
MX346605B (en) | 2017-03-24 |
US9524023B2 (en) | 2016-12-20 |
RU2015114537A (en) | 2016-11-10 |
EP3425485A2 (en) | 2019-01-09 |
RU2617927C2 (en) | 2017-04-28 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
US9524023B2 (en) | Display apparatus and control method thereof | |
US8823741B2 (en) | Transparent display apparatus and method for operating the same | |
US8823663B2 (en) | Transparent display apparatus and method for operating the same | |
EP3143477B1 (en) | System and method for providing haptic feedback to assist in capturing images | |
TWI423112B (en) | Portable virtual human-machine interaction device and method therewith | |
US20120256854A1 (en) | Transparent display apparatus and method for operating the same | |
EP2991064A2 (en) | Method for removing image sticking in display device | |
KR101079598B1 (en) | Display apparatus and control method thereof | |
TW201344597A (en) | Control method and controller for display device and multimedia system | |
US11183151B2 (en) | Display apparatus and controlling method thereof | |
EP2428875A2 (en) | Image display apparatus and method for operating the same | |
JPWO2010035477A1 (en) | User interface device, user interface method, and recording medium | |
CN109391730A (en) | Mobile terminal and its control method | |
KR102431712B1 (en) | Electronic apparatus, method for controlling thereof and computer program product thereof | |
US11240466B2 (en) | Display device, mobile device, video calling method performed by the display device, and video calling method performed by the mobile device | |
CN112473121B (en) | Display device and avoidance ball display method based on limb identification | |
EP3479883B1 (en) | Method for touch control in mobile real-time simulation game | |
KR20150008769A (en) | Image display apparatus, and method for operating the same | |
EP2500815B1 (en) | Transparent display apparatus and method for operating the same | |
KR20170046947A (en) | Mobile terminal and method for controlling the same | |
KR20200092158A (en) | Electronic apparatus and control method of the electronic apparatus | |
KR20140051804A (en) | Display apparatus and method of controlling the smae | |
CN114860363A (en) | Content item display method and device and electronic equipment | |
CN113687902A (en) | Resource display method and device, computer equipment and storage medium | |
KR20110138851A (en) | Controlling method a memu list and image displaying device thereof |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
STPP | Information on status: patent application and granting procedure in general |
Free format text: NON FINAL ACTION MAILED |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: RESPONSE TO NON-FINAL OFFICE ACTION ENTERED AND FORWARDED TO EXAMINER |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: FINAL REJECTION MAILED |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: DOCKETED NEW CASE - READY FOR EXAMINATION |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: NON FINAL ACTION MAILED |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: RESPONSE TO NON-FINAL OFFICE ACTION ENTERED AND FORWARDED TO EXAMINER |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: NON FINAL ACTION MAILED |
|
STCB | Information on status: application discontinuation |
Free format text: ABANDONED -- FAILURE TO RESPOND TO AN OFFICE ACTION |