KR102480568B1 - A device and method for displaying a user interface(ui) of virtual input device based on motion rocognition - Google Patents

Abstract

The present invention relates to a device capable of displaying a UI of a virtual input device desired by a user through motion recognition for a user's gesture. The display device of the present invention includes a display unit for displaying an image, a storage unit for storing a template linking different virtual input devices for each user's gesture, and a user interface (UI) for generating a user interface (UI) of the virtual input device. unit, a camera that captures the user's gesture and the captured user's gesture is compared with the user's gesture stored in the template of the storage unit, and the user interface (UI) of the virtual input device corresponding to the matched user gesture is displayed on the display unit It may be configured to include a control unit for controlling a user interface (UI) unit to display.

Description

Apparatus and method for displaying a user interface (UI) of a virtual input device based on motion recognition

The present invention relates to a display device and a method for displaying a virtual input device based on motion recognition, and more particularly, a virtual input device according to a user's gesture is templated to recognize a user's gesture to display and manipulate a specific virtual input device. It relates to a display device that does.

Recently, a command input device using gesture recognition capable of recognizing a user's motion and executing a specific command in conjunction with the recognized motion has been developed.

As a conventional command input method using gesture recognition, a method in which a user inputs a command by recognizing a user gesture after displaying a virtual input device through voice input or a remote control is disclosed in Korean Patent Publication No. 10-2012-0096849. there is.

Such a conventional command input method using gestures basically uses voice food technology to display a virtual input device or uses an input device such as a real remote control to switch to a virtual input mode, and then recognizes the gesture through a camera. User commands can be entered.

In general, users prefer command input through simple input. In contrast, the conventional command input method using gestures requires a complex technical solution to additionally apply voice recognition technology to apply gesture recognition technology, and voice recognition technology in an environment where a real input device, for example, a remote control, is not found. If this is not applied, it can lead to very difficult situations.

An object of the present invention is to solve the above-mentioned conventional problems, and not only can display user interfaces (UI) of various virtual input devices only by recognizing gestures, but also user interfaces (UI) of virtual input devices displayed by recognizing finger movements. ) to provide a method and apparatus for displaying a virtual input device UI based on motion recognition, through which a user command can be input.

Another object of the present invention is to provide consistent user command input by resetting the position of a finger on a displayed virtual input device even if the position of a user's finger changes when the user's gaze returns to the original position after temporarily leaving the screen. An object of the present invention is to provide a method and apparatus for displaying a virtual input device UI based on motion recognition.

Another object of the present invention is to provide a method and apparatus for displaying a virtual input device UI based on motion recognition capable of supporting voice guidance for the user's command input convenience even when the visually impaired or the user's eyes are out of sight. .

Another object of the present invention is a virtual input based on motion recognition that can generate a sound effect similar to a manipulation sound generated when a physical input device is operated when a user inputs through a user interface (UI) of a displayed virtual input device. It is to provide a method and device for displaying a device UI.

An apparatus for displaying a user interface (UI) of a virtual input device based on motion recognition according to an embodiment for solving the above-described problems of the present invention is a display unit displaying an image and a virtual input different for each user's gesture. A storage unit for storing a template associated with a device, a user interface (UI) unit for generating a user interface (UI) of the virtual input device, a camera for capturing a user's gesture, and a template for the captured user's gesture It can be configured to include a control unit that controls the user interface (UI) unit so that the user interface (UI) of the virtual input device corresponding to the user gesture registered in and displayed on the display unit is compared with the user gesture that is matched.

The virtual input device may include a remote control, a touch remote control, a keyboard, and a mouse.

The user gesture may include a grip gesture of a physical input device.

The device may further include a motion calculation unit recognizing a user's finger movement and a user's face direction or gaze in order to operate the displayed virtual input device UI.

The controller may display a pointer on the UI of the virtual input device based on the motion of the user's finger calculated by the motion calculator.

The motion calculation unit may determine the movement direction of the user's finger, the amount of movement, the location, and convert the size of the pointer of the virtual input device UI to the actual finger.

The storage unit may store at least one of a manipulation sound of the virtual input device according to the movement of the user's finger, a voice explaining function execution details, and a voice explaining the name of a function button of the virtual input device where the user's finger is located. there is.

The motion calculation unit may determine whether the user gazes or not based on a comparison between the distribution and concentration of the gaze recognition data within a predetermined time with a predefined threshold value.

The control unit may reset the user's previous finger position when it is determined that the user's gaze is in a 'not looking' state.

The control unit may switch to a voice guidance mode when it is determined that the user's eyes are in a 'non-looking' state.

The control unit can delete, add, or change function buttons on the UI of the virtual input device according to a user command.

A method of displaying a UI of a virtual input device based on motion recognition according to an embodiment of the present invention includes the steps of storing a template in which different virtual input devices are linked for each user's gesture, photographing the user's gesture, and Comparing the captured user's gesture with the user's gesture registered in the template of the storage unit, and displaying the UI of the virtual input device corresponding to the matched user's gesture as a result of the comparison.

According to the present invention, a user can simply display user interfaces (UIs) of various virtual input devices using only gestures.

In addition, the user can register and use the user interface of his or her own virtual input device, so it is convenient to use.

In addition, even when a visually impaired person or a user's gaze is out of the front, the user can use the user interface of the virtual input device more conveniently by performing voice guidance together.

In addition, even if an environment changes while the user is using the displayed user interface of the virtual input device, the consistency of the command input of the user interface of the virtual input device can be maintained.

In addition, even when the user uses the user interface of the virtual input device displayed on the display unit through a gesture, sound effects similar to those of using a physical input device can be provided.

In addition, since the user interface of the software-based virtual input device can be applied, it is possible to conveniently perform updates for the addition of new functions.

1 is a block diagram showing a device for displaying a user interface (UI) of a virtual input device based on motion recognition according to an embodiment of the present invention;
2 is a diagram showing a template linking a user gesture and a virtual input device;
3 is a diagram showing a use state of a device displaying a user interface (UI) of a virtual input device based on motion recognition according to the present invention;
4 is a diagram for explaining editing of a user interface (UI) of a virtual input device displayed according to the present invention; and
5 is a flowchart illustrating a method of displaying a user interface (UI) of a virtual input device based on motion recognition according to an embodiment of the present invention.

Hereinafter, the present invention will be described in detail with reference to the accompanying drawings. In the following embodiments, only configurations directly related to the spirit of the present invention will be described, and descriptions of other configurations will be omitted. However, in implementing a device or system to which the spirit of the present invention is applied, it is revealed that the omitted configuration does not mean that the description is unnecessary. Like reference numbers expressed in the drawings indicate like elements.

Apparatus 100 for displaying a user interface (UI) of a virtual input device based on motion recognition according to the present invention is a television (TV) equipped with a camera, a personal computer (PC), a smartphone, a laptop computer, a PDA, a game machine etc., and will be briefly referred to as a 'display device' hereinafter.

As shown in FIG. 1, the display apparatus 100 according to the present invention includes a storage unit 110, a user interface (UI) unit 120, a camera 130, a control unit 140, a display unit 160, and a voice An output unit 170 may be included. In addition to the components described above, the display device 100 includes an image receiving unit (not shown) including a tuner, a wired/wireless communication unit (not shown), a timing control unit (not shown), an image processing unit (not shown), and an audio processing unit (not shown). It may include more components such as, etc., and are omitted for brief description.

The storage unit 110 includes, for example, a template DB of a virtual input device associated with a user gesture, a user personal information DB, a sound effect DB according to finger movements, a voice DB for explaining button names on the virtual input device UI, and a finger movement According to the input command execution content explanation voice DB, etc. may be pre-stored. Of course, the storage unit 110 may include various information such as image data, graphic data, and OS (operating system) other than the data listed above.

2 is a diagram showing a template of a virtual input device associated with a user gesture. If the user's gesture is a keyboard typing gesture with both hands, a virtual keyboard is used, and if the user's gesture is a gesture of gripping a mouse with one hand, a virtual mouse is displayed. , A gesture of gripping the remote control with one hand represents an example of configuring a template by linking the remote control.

The user personal information DB registers and stores virtual input devices composed of frequently used buttons for each user. The user may initially edit the user interface (UI) of the virtual input device configured as a basis, delete unnecessary buttons by dragging and dropping, or add and store necessary buttons.

The sound effect DB according to the movement of the finger recognizes the movement of the finger and stores a typing sound in the case of a virtual keyboard, a click sound in the case of a virtual mouse, and a button touch sound of a virtual remote control as a user setting or default.

Button name description on the virtual input device UI The voice DB may store voices explaining the names of the buttons located when the user's finger is positioned on the button of each virtual input device UI. For example, when the user's finger position is placed on the 'record' button, a message 'located on the record button' may be stored.

The voice DB for explaining the execution contents of commands entered according to finger movements can store voices that explain user commands according to finger movements. For example, if you drag and drop the 'record' button to the trash to create your own virtual input device, you can define and save the message 'You put the record button in the trash to delete' in advance. there is.

The storage unit 110 may be a flash memory type, a hard disk type, a multimedia card micro type, a card type memory (eg SD or XD memory, etc.), RAM (RAM, Random Access Memory) SRAM (Static Random Access Memory), ROM (ROM, Read-Only Memory), EEPROM (Electrically Erasable Programmable Read-Only Memory), PROM (Programmable Read-Only Memory) magnetic memory, magnetic disk , an optical disk, and at least one type of storage medium.

The user interface (UI) unit 120 may generate and display a virtual input device user interface corresponding to the recognized user gesture. A virtual input device user interface (UI) refers to an image capable of user input having various buttons such as a virtual keyboard, a virtual remote control, a touch remote control, and a virtual mouse. A user may input a predetermined command of a corresponding button by clicking, double-clicking, or touching the button.

The camera 130 may generate an image by photographing a user's hand motion (gesture), face, and eyes. The camera 130 may include a webcam, a pan tilt camera, a digital camera, an analog camera, a camcorder, and the like. The camera 130 may be separately installed or may be built into the display device 100 . The camera 130 may be installed only on the front side, or a 2D or 3D image may be captured using several cameras.

The control unit 140 includes each component of the display device 100, for example, the storage unit 110, the user interface (UI) unit 120, the camera 130, the display unit 160, and the audio output unit 170. ) can be controlled.

The control unit 110 compares the user's gesture image captured by the camera 130 with the gesture stored in the storage unit 110, and generates a virtual input device user interface (UI) corresponding to the matched gesture to display the user interface. The (UI) unit 120 may be controlled.

The controller 140 includes application specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), and micro-controllers. , may include microprocessors.

The motion calculation unit 150 may calculate the user's face direction, gaze direction, and finger movement based on the user's image captured by the camera. The motion calculation unit 150 may determine whether the user gazes or not based on a comparison between the distribution and concentration of the gaze recognition data within a predetermined time with a predefined threshold value. The motion calculation unit 150 may receive user motion (finger motion) captured by the camera 130 and calculate the position of the corresponding finger, the amount of movement of the finger, the direction of movement of the finger, and the size of the virtual input device UI for the actual finger. there is. The controller 140 may execute various commands using the finger position, finger movement amount, finger movement direction, and size conversion value of the virtual input device UI for the actual finger calculated by the motion calculator 150 . The motion calculator 150 may be implemented as software, a program, or hardware including software.

The display unit 160 may display an image processed by an image processing unit (not shown) or a virtual input device UI generated by the UI unit 120 so that viewers can see it. The display unit 160 may display one still image frame by vertically arranging a plurality of scanned horizontal scan lines. The display unit 160 includes a display panel (not shown) for displaying the image, and the display panel (not shown) includes a liquid crystal panel including a liquid crystal layer, an organic light emitting panel including a light emitting layer made of organic materials, or plasma. A display panel and the like may be included. The display unit 160 may include a flexible panel.

The audio output unit 170 may be implemented as a speaker and may output the audio signal processed by the audio processing unit (not shown) as sound. The voice processing unit may receive the voice signal corresponding to the finger movement calculated by the motion calculation unit 150 from the storage unit 110 and process it into a signal outputtable by the audio output unit 170 . Based on the finger movements calculated by the motion calculation unit 150, the voice output unit 170 outputs operation sounds of the virtual input device according to the user's finger movements, voices explaining function execution details, and the user's fingers. A voice explaining the name of the function button of the virtual input device located there may be output.

A method of displaying a user interface (UI) of a virtual input device based on motion recognition according to the present invention will be described using a display device (TV) as an example.

As shown in FIG. 3, when a TV user makes a gesture of gripping a remote control while watching TV, the camera captures and recognizes the gesture of the TV user. At this time, the recognized user's gesture of holding the remote control is compared with a gesture predefined in the template for each gesture of the storage unit 110, and the virtual remote control UI 164 is used as an input device for the matching gesture on the display unit ( 160). The mapping of the virtual UI to the input device for each gesture is based on the general characteristics of the user's gesture when using a remote control, keyboard, and mouse. For example, as shown in FIG. 2, a gesture of spreading both hands, such as when typing on a keyboard, is mapped to a keyboard, a gesture of wrapping a mouse and bending an index finger, a gesture of supporting a mouse and a remote control, and slightly raising a thumb is mapped to a display of a remote control. .

In addition, when the user makes a gesture of holding the remote controller, two types of virtual remote controllers may be displayed for the user's convenience so that the user can select a general type virtual remote control and a touch type virtual remote control according to the user's preference. Then, when the user makes a gesture of moving his or her thumb, the general remote control is finally displayed, and when the user takes a gesture of the index finger, which is mainly used when using the touch method, the touch type remote control can be finally displayed. Of course, at this time, it is also possible to reduce the virtual remote control UI display step through finger gesture recognition without the grip gesture step.

In addition, the user's grip gesture recognition rate may be increased by additionally registering a hand shape when the user holds the physical remote controller in the storage unit 110 . In particular, the virtual remote control can be displayed in a half tone, and as time passes after the grip and finger gestures are recognized and displayed, when no new information is additionally obtained from the user's gesture or finger movement, the transparency gradually increases. It can be designed to fade and disappear from the screen after a certain period of time.

As shown in FIG. 4 , after the virtual remote control is displayed on the screen, the motion of the user's finger may be detected and the position of the detected finger may be displayed on the virtual remote control UI. In this way, if the position of the detected finger is displayed, it is possible to help the user adjust the amount of movement of the finger to press the button of the remote control while viewing the virtual remote control on the TV screen.

In addition, after displaying the virtual remote control on the TV screen, when the user makes a gesture of touching or pressing a button of the virtual remote control, a corresponding sound effect is generated so that the user feels that the user actually operates the button of the virtual remote control. can Of course, the user may edit the desired sound effect as needed, or it may be set by default.

If, as a result of recognizing the direction of the user's face or gaze, if the virtual remote control user is recognized as not directly looking at the TV screen by the camera, the user displays the virtual remote control on the screen to use the virtual remote control and operates the virtual remote control. When a finger movement is taken, the function of automatically guiding the name of the button where the finger is located by voice is automatically activated, so that the remote control can be conveniently used without directly looking at the virtual remote control. That is, finger movement, face direction, and whether or not gaze is watching TV can be simultaneously detected/analyzed.

However, the finger movement uses continuous frame data, and the direction of the face and whether to gaze at the TV screen can be determined based on continuous or non-continuous frame data according to options. This is because, in the case of a visually impaired person, even if the direction of the face is toward the TV, it may not coincide with the gaze of the TV screen. Whether or not to gaze at the TV screen is determined based on data acquired using an infrared camera.

Also, if the user temporarily moves, the position and gesture of a finger on the virtual remote control displayed on the TV screen may change. In this case, even if the user makes a gesture for manipulating the remote control again in a state where the virtual remote controller is displayed, the finger position may be recognized as a different finger position from the initially displayed finger position, causing an erroneous operation. Therefore, as a result of recognizing the user's face direction or gaze, if the virtual remote control user is recognized as not directly looking at the TV screen by the camera, the initially displayed finger position is reset, and the finger after the gaze returns to the front position can be set.

FIG. 4 shows an example of editing with a user's preferred virtual remote controller 164 using the initial virtual remote controller 162 . When a user wants to register a button that performs a specific function on the remote control, for example, a microphone button for voice conversation, as a user's simple gesture, double-click on the corresponding button on the virtual remote control displayed on the screen to use the gesture instead of that button. It is moved to the registration screen, and each user can register/delete and use his or her own preferred simple gesture. In addition, simple gestures registered for each user can be selected and used through the virtual remote control after checking the registered list through the favorite gesture function. In addition, each user can hide only the remote control buttons that he or she frequently uses, and the rest of the buttons can be hidden by drag and drop using the trash bin function in the virtual remote control, or can be brought out and displayed again if necessary. As a result, the user-registered virtual remote control having only the buttons required by the user is convenient to use because of its good intuition. In particular, the user can set the number, position, and size of virtual remote control buttons by scaling them up or down according to the user's preference and eyesight.

In addition, it is possible to display the initial appearance of the remote control at the time of TV launch by putting a 'Home' button. In addition, detailed functions of the TV OSD menu can be assigned to any button of the virtual remote control in the form of a shortcut key, and detailed functions of the TV OSD menu can be executed through a double-click gesture.

Hereinafter, a method of displaying a user interface (UI) of a virtual input device based on motion recognition according to the present invention will be described in detail with reference to FIG. 5 .

Step S210, a template in which different virtual input devices are linked (mapped) for each user's gesture in advance is stored in a storage unit. Of course, in the storage unit 110, in addition to the virtual input device template for each user's gesture, voice data describing the button name or button function of the virtual input device UI, voice data describing the action executed by the gesture, and virtual input A button operation sound of the device UI may be stored in advance.

In step S220, the user's gesture, face direction, and gaze are photographed using the camera 130.

In step S230 , the user gesture captured by the camera 130 is compared with user gestures registered in the template of the storage unit 110 .

In step S240, the UI of the virtual input device corresponding to the user gesture matched as a result of the comparison is displayed.

In step S250, the user's finger movement and gaze are recognized.

In step S260, the location of the user's finger is displayed as, for example, a finger-shaped pointer on the displayed UI of the virtual input device. Then, the user may input a command by moving a finger and clicking or touching a desired button by a gesture using the UI of the virtual remote controller.

In step S270, when the user's gaze leaves the TV screen, it is automatically changed to a voice mode that explains the function or name of a predetermined button located on the pointer, and a predetermined voice is output. Of course, it can also be set to always perform voice explanation in advance regardless of gaze.

The above embodiments are merely illustrative, and various modifications and equivalent other embodiments are possible to those skilled in the art. Therefore, the true technical protection scope of the present invention should be determined by the technical spirit of the invention described in the claims below.

100: display device
110: storage unit
120: UI unit
130: camera
140: control unit
150: motion calculation unit
160: display unit
170: audio output unit

Claims (16)

In the display device,
a display unit displaying an image;
with a camera;
Identifying a gesture corresponding to a user's hand based on an image obtained through the camera;
Identifying whether the identified gesture is a gesture corresponding to both hands or a gesture corresponding to one hand among a plurality of predefined gestures;
If the identified gesture is a gesture corresponding to both hands, controlling the display unit to display a first user interface (UI) corresponding to a first virtual input device that can be input with both hands;
If the identified gesture is a gesture corresponding to the one hand, controlling the display unit to display a second UI corresponding to a second virtual input device that can be input with one hand;
While a screen without the first UI and the second UI is displayed on the display unit, the first UI according to a gesture corresponding to both hands and the second UI according to a gesture corresponding to one hand are displayed, respectively. A controller for controlling the display unit
A display device comprising a. According to claim 1,
The first virtual input device includes a keyboard,
The display device, characterized in that the second virtual input device includes at least one of a remote control, a touch remote control, or a mouse. According to claim 1,
The display device, characterized in that the gesture corresponding to the one hand includes a grip gesture of a physical input device. According to claim 1,
The control unit recognizes a user's finger movement for operating a first UI corresponding to the first virtual input device. According to claim 1,
The display device, characterized in that the control unit displays a pointer to move on the second UI corresponding to the second virtual input device according to the user's finger movement. According to claim 5,
The display apparatus according to claim 1 , wherein the control unit identifies at least one of a movement direction, a movement amount, or a location of the finger according to the finger movement. According to claim 4,
Further comprising a storage unit for storing the plurality of predefined gestures,
The control unit transmits at least one of a manipulation sound of the first virtual input device according to the finger movement, a voice explaining function execution details, and a voice explaining the name of a function button of the first virtual input device to the storage unit. A display device characterized in that for storing. According to claim 1,
The controller may delete, add, or delete a function button on the first or second UI corresponding to the first or second virtual input device according to a user command corresponding to a gesture corresponding to both hands or a gesture corresponding to one hand. A display device characterized in that for changing. In the control method of the display device,
identifying a gesture corresponding to the user's hand based on the obtained image;
identifying whether the identified gesture is a gesture corresponding to both hands or one hand among a plurality of predefined gestures;
When the identified gesture is a gesture corresponding to both hands, a first user interface (UI) corresponding to a first virtual input device that can be input with both hands is displayed, and the identified gesture is a gesture corresponding to the one hand. If so, displaying a second UI corresponding to a second virtual input device capable of being input with one hand;
Displaying the first UI according to a gesture corresponding to both hands and the second UI according to a gesture corresponding to one hand while a screen without the first UI and the second UI is displayed, respectively
Control method of a display device comprising a. According to claim 9,
The first virtual input device includes a keyboard,
Wherein the second virtual input device includes at least one of a remote control, a touch remote control, and a mouse. According to claim 9,
The control method of a display device, characterized in that the gesture corresponding to the one hand includes a grip gesture of a physical input device. According to claim 9,
The step of identifying whether the gesture corresponds to both hands or one hand includes recognizing a user's finger movement for operating a first UI corresponding to the first virtual input device. Control method of a display device characterized by. According to claim 9,
The displaying of the second UI corresponding to the second virtual input device may include displaying a pointer to move on the second UI corresponding to the second virtual input device according to the movement of the user's finger. A control method of a display device to be. According to claim 13,
The displaying of the pointer to move further comprises identifying at least one of a moving direction, a moving amount, or a position of the finger according to the finger movement. According to claim 12,
The step of recognizing the user's finger movement may include a manipulation sound of the first virtual input device according to the finger movement, a voice explaining function execution details, or a voice explaining the name of a function button of the first virtual input device. A control method of a display device comprising the step of storing at least one. According to claim 9,
The displaying of the first or second UI corresponding to the first or second virtual input device may include the first or second UI according to a user command corresponding to a gesture corresponding to both hands or a gesture corresponding to one hand. A control method of a display device comprising the step of deleting, adding or changing a function button on a first or second UI corresponding to a virtual input device.

Images