KR20180107288A - Method and system for providing user interface, and non-transitory computer-readable recording medium - Google Patents

Abstract

According to an aspect of the present invention, there is provided a method for providing a user interface, comprising: (a) recognizing gesture and biometric information obtained respectively from a gesture input means and a biometric information input means; (b) Or an object and a command corresponding to the recognized gesture or the recognized biometric information with reference to the correspondence between the biometric information and the predetermined object or command and specifies the user with reference to the recognized biometric information And (c) determining that the specified command is to be performed on the specified object within a scope of authority granted to the specified user.

Description

[0001] METHOD AND SYSTEM FOR PROVIDING USER INTERFACE, AND NON-TRANSITORY COMPUTER [0002] READABLE RECORDING MEDIUM [

The present invention relates to a method, system and non-transitory computer readable recording medium for providing a user interface.

2. Description of the Related Art In recent years, mobile terminal devices, including smart phones, have received high-level security information, and accordingly, personalization technology or security technology that can be applied to mobile terminal devices is increasing in demand and interest.

In this regard, a technique called Touch ID (that is, recognizing a fingerprint of a user by using RF (Radio Frequency) within a predetermined area) introduced by Apple, Inc. or the like And a technique of visually recognizing the shape of a fingerprint using an optical device such as an RGB camera have been introduced.

However, the prior art disclosed so far only concerns the recognition of the biometric information such as the fingerprint on the mobile terminal device, and the user can easily control the mobile terminal device based on the biometric information recognized on the mobile terminal device There is no suggestion to provide a user interface that can help to do this.

Accordingly, the present inventor has developed a technique capable of providing a convenient and extended user interface to a user by recognizing a gesture in addition to biometric information and making full use of the gesture.

It is an object of the present invention to solve all the problems described above.

In addition, the present invention recognizes the gesture and the biometric information acquired from the gesture input means and the biometric information input means, and recognizes the gesture and the biometric information by referring to the corresponding relationship between the predetermined gesture or biometric information and the predetermined object or command, Identifies an object and a command corresponding to the recognized gesture or the recognized biometric information, identifies the user with reference to the recognized biometric information, identifies the user within the authority range given to the specified user, It is another object of the present invention to recognize both the gesture and the biometric information and to provide a more convenient and extended user interface to the user based on the recognition result.

In order to accomplish the above object, a representative structure of the present invention is as follows.

According to an aspect of the present invention, there is provided a method for providing a user interface, comprising: (a) recognizing gesture and biometric information obtained respectively from a gesture input means and a biometric information input means; (b) Or an object and a command corresponding to the recognized gesture or the recognized biometric information with reference to the correspondence between the biometric information and the predetermined object or command and specifies the user with reference to the recognized biometric information And (c) determining that the specified command is to be performed on the specified object within a scope of authority granted to the specified user.

According to another aspect of the present invention, there is provided a system for providing a user interface, comprising: a gesture recognition unit for recognizing a gesture acquired from a gesture input unit; a biometric information recognition unit for recognizing biometric information acquired from the biometric information input unit; And specifying an object and a command corresponding to the recognized gesture or the recognized biometric information with reference to a correspondence relationship between the predetermined gesture or biometric information and the predetermined object or command, And a command execution unit for specifying a user and determining that the specified command is to be executed for the specified object within a scope of authority granted to the specified user.

In addition, there is provided another method, system, and non-transitory computer readable recording medium for recording a computer program for carrying out the method for implementing the present invention.

According to the present invention, in addition to merely recognizing biometric information and confirming the identity of the user, the gesture and biometric information are recognized and utilized together, thereby achieving various controls for the mobile terminal device.

In addition, according to the present invention, an object or a command is associated with an inputable gesture or biometric information, so that when a user simply inputs a gesture and biometric information, an instruction corresponding to the gesture and biometric information can be performed Is achieved.

1 is a diagram illustrating an exemplary internal configuration of a user interface providing system according to an embodiment of the present invention.
FIGS. 2 to 4 are views illustrating exemplary operations of a user interface according to an exemplary embodiment of the present invention. Referring to FIG.
FIG. 5 is a diagram illustrating a process of providing visual feedback according to an exemplary embodiment of the present invention. Referring to FIG.
FIG. 6 is a diagram illustrating an exemplary operation of a user interface according to an exemplary embodiment of the present invention. Referring to FIG.
FIG. 7 is a diagram illustrating a configuration of providing a user interface using integrated technique means capable of receiving both a gesture and biometric information according to an embodiment of the present invention. Referring to FIG.
FIG. 8 is a diagram illustrating a configuration in which various commands are executed corresponding to various gestures according to an embodiment of the present invention.

The following detailed description of the invention refers to the accompanying drawings, which illustrate, by way of illustration, specific embodiments in which the invention may be practiced. These embodiments are described in sufficient detail to enable those skilled in the art to practice the invention. It should be understood that the various embodiments of the present invention are different, but need not be mutually exclusive. For example, certain features, structures, and characteristics described herein may be implemented in other embodiments without departing from the spirit and scope of the invention in connection with an embodiment. It is also to be understood that the position or arrangement of the individual components within each disclosed embodiment may be varied without departing from the spirit and scope of the invention. The following detailed description is, therefore, not to be taken in a limiting sense, and the scope of the present invention is to be limited only by the appended claims, along with the full scope of equivalents to which such claims are entitled, if properly explained. In the drawings, like reference numerals refer to the same or similar functions throughout the several views.

Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings, so that those skilled in the art can easily carry out the present invention.

Configuration of user interface providing system

Hereinafter, an internal configuration of a user interface providing system that performs an important function for the implementation of the present invention and a function of each component will be described.

According to one embodiment of the present invention there is provided a computer system having memory means such as a personal computer (e.g., a desktop computer, a notebook computer, etc.), a server, a workstation, a PDA, a web pad, a mobile phone, a smart phone, The present invention can be applied to any user interface providing system of the present invention as long as it is equipped with computing capability and the user terminal apparatus itself to be described later may be a user interface providing system. In particular, the user interface providing system according to an embodiment of the present invention may include a gesture input unit that can receive various gestures from a user using a touch screen, an infrared sensor, an acceleration sensor, a camera, A biometric information input means for inputting biometric information from a user by using an iris recognition sensor, a speech recognition sensor, a pulse recognition sensor, an EEG sensor, a temperature recognition sensor, and the like. And display means capable of displaying the display screen.

1 is a diagram illustrating an exemplary internal configuration of a user interface providing system according to an embodiment of the present invention.

Referring to FIG. 1, a user interface providing system 100 according to an embodiment of the present invention includes a gesture recognition unit 110, a biometric information recognition unit 120, an instruction execution unit 130, a communication unit 140, And a control unit 150. The gesture recognizing unit 110, the biometric information recognizing unit 120, the instruction executing unit 130, the communication unit 140 and the control unit 150 may be connected to the external system Lt; / RTI > shown in FIG. These program modules may be included in the user interface providing system 100 in the form of an operating system, an application program module, and other program modules, and may be physically stored in various well-known storage devices. These program modules may also be stored in a remote storage device capable of communicating with the user interface providing system 100. These program modules include, but are not limited to, routines, subroutines, programs, objects, components, data structures, etc. that perform particular tasks or implement particular abstract data types as described below in accordance with the present invention.

First, according to an embodiment of the present invention, the gesture recognition unit 110 may perform a function of recognizing a gesture inputted from a user with reference to a user operation input from a predetermined gesture input means. As will be described later, the gesture according to the present invention may correspond to at least one object or command.

For example, a gesture input from a user may include a touch input to an arbitrary point on the touch screen, a touch input (i.e., a swipe or drag) that extends from a first point to a second point on the touch screen, , Touch release, flicking, pinch, and the like. As another example, rotating or shaking a user terminal device having an acceleration sensor or a gyro sensor may be included in the gesture described in the present invention. However, it should be noted that the gesture according to the present invention is not necessarily limited to those listed above, and may be changed within a range that can achieve the object of the present invention.

Next, the biometric information recognition unit 120 can perform a function of recognizing the biometric information input from the predetermined biometric information input means. As described later, the biometric information according to the present invention may correspond to at least one object or command.

For example, biometric information input from a user includes fingerprint information obtained from a fingerprint recognition sensor, iris information acquired from an iris recognition sensor, vein information acquired from a vein recognition sensor, audio information acquired from a speech recognition sensor, Pulse information obtained from a sensor, brain wave information obtained from an EEG sensor, temperature information obtained from a temperature recognition sensor, and the like. However, it should be noted that the biometric information according to the present invention is not necessarily limited to those listed above, but may be modified within a range that can achieve the object of the present invention.

Meanwhile, according to an embodiment of the present invention, the gesture and the biometric information may be input sequentially or simultaneously. For example, the biometric information may be recognized after the gesture is recognized first, or the gesture may be recognized after the biometric information is recognized first.

Further, according to an embodiment of the present invention, when the gesture input means and the biometric information input means are integrally formed, the gesture and the biometric information may be inputted at a time.

For example, the gesture may be input through a touch screen, and the biometric information may be input through a fingerprint recognition sensor provided on a home button separate from the touch screen.

As another example, both the gesture and the biometric information can be input through an integrated technique such as a fingerprint on display (FOD) or a trace fingerprint sensor.

Next, according to one embodiment of the present invention, the command execution unit 130 refers to the correspondence relation between a predetermined gesture or biometric information and a predetermined object or command, It is possible to identify an object and an instruction corresponding to the information, and identify the user with reference to the recognized biometric information.

Also, according to an embodiment of the present invention, the command execution unit 130 performs a function of determining that the specified command is to be performed on the specified object within the authority granted to the specified user .

Specifically, in accordance with an embodiment of the present invention, each of the various gestures that can be input from the user can be pre-set to correspond to an object or command, and when referring to the predefined correspondence, Quot; object " or " command " For example, a gesture that touches one of a plurality of icons displayed on the touch screen may be pre-set to correspond to selecting an object indicated by the icon. As another example, a gesture (i.e., swipe or drag) that continuously touches from one point to another point on the touch screen may be pre-set to correspond to a save or load command.

Basically, according to an embodiment of the present invention, biometric information that can be input from a user can be utilized as a reference for identifying a user and specifying a user. In addition, according to an embodiment of the present invention, each of various pieces of biometric information that can be input from a user can be previously set to correspond to an object or a command, and referring to the predetermined correspondence relationship, The corresponding object or command can be specified. For example, the fingerprint information corresponding to the index finger may be previously set to correspond to the object A, and the fingerprint information corresponding to the thumb may be previously set to correspond to the object B. For example, the fingerprint information corresponding to the index finger may be previously set to correspond to an instruction to perform settlement using the credit card A, and the fingerprint information corresponding to the thumb can be set using the credit card B It can be pre-set to correspond to an instruction to perform.

FIGS. 2 to 4 are views illustrating exemplary operations of a user interface according to an exemplary embodiment of the present invention. Referring to FIG.

2 (a), when the user A touches the content A displayed on the touch screen 210 of the user terminal device with the index finger and drags the finger in the direction of the fingerprint recognition sensor 220 of the user terminal device The user interface providing system 100 according to the embodiment of the present invention recognizes the above gesture and the biometric information when the index finger is positioned on the fingerprint recognition sensor 220 after performing the gesture (i.e., pulling) To determine the user A, the content A and the command to store, and to determine that the command to store the content A in the given storage space is to be performed to the user A of the cloud server.

2 (b), after the user B places the index finger on the fingerprint recognition sensor 220 of the user terminal, the index finger is detected by the fingerprint recognition sensor 220 on the touch screen 210 The user interface providing system 100 according to the embodiment of the present invention recognizes the above gesture and biometric information and transmits the user B, the content B, and the fetch command The content B stored in the storage space allocated to the user B of the cloud server in a state corresponding to the fingerprint information of the index finger of the user B is loaded to the user terminal device, It may be determined that an instruction to be displayed on the screen 210 is to be performed.

3 (a), when the user A moves the index finger from the top to the bottom with the index finger placed on the fingerprint sensor 320 of the user terminal, The user interface providing system 100 according to the example can identify the user A, the content A, and the storing instruction by recognizing the above gesture and biometric information, and can detect the user A in the storage space given to the user A in the cloud server It may be determined that an instruction to store the content A corresponding to the finger is to be performed.

3 (b), when the user B moves the index finger from the bottom to the top with the index finger placed on the fingerprint sensor 320 of the user terminal, The user interface providing system 100 according to the example can identify the user B, the content B, and the fetch command by recognizing the above gesture and biometric information, and can detect the user B in the storage space given to the user B in the cloud server It is possible to determine that an instruction to fetch the content B stored in correspondence with the finger to the user terminal device is performed.

4, when the user A touches the content A displayed on the touch screen 410 with the index finger for a predetermined time and touches the content B with the thumb for a predetermined time or more, the index finger and thumb The contents A and the contents B can correspond to the fingers, respectively. 4, when the user A inputs a predetermined gesture using the index finger, a predetermined command can be performed on the content A corresponding to the index finger, and when the user A uses the thumb When a predetermined gesture is input, a predetermined command can be performed on the finger B corresponding to the thumb.

FIG. 5 is a diagram illustrating a process of providing visual feedback according to an exemplary embodiment of the present invention. Referring to FIG.

Referring to FIG. 5, when a user touches a specific object displayed on the touch screen 510 of the user terminal device, an arrow shape that can induce the user to input a gesture corresponding to the "pull" On the touch screen 510. [0053] FIG.

FIG. 6 is a diagram illustrating an exemplary operation of a user interface according to an exemplary embodiment of the present invention. Referring to FIG.

6, when a user touches an arbitrary area on the touch screen 610 of the user terminal device with the index finger, the user touches the specific area 631 and 632 among the plurality of areas displayed on the touch screen 610 When the index finger is positioned on the fingerprint recognition sensor 620 after a gesture (i.e., pulling) of dragging in the direction of the fingerprint recognition sensor 620 of the user terminal device is performed, the user interface according to the embodiment of the present invention The system 100 recognizes the above gesture and biometric information and can determine that an instruction corresponding to the specific area is performed. For example, if the above drag gesture passes through a block 631 indicated by the number "7 ", payment can be made with a seven-month installment (see Fig. 6A) (632) marked ", the settlement can be made with a three-month installment (see Fig. 6 (b)).

FIG. 7 is a diagram illustrating a configuration of providing a user interface using integrated technique means capable of receiving both a gesture and biometric information according to an embodiment of the present invention. Referring to FIG.

7A, when the user touches the first graphic element 711 on the fingerprint recognition integrated touch screen 710, information on the position where the above touch operation has been performed, Fingerprint information can be obtained and when the user performs an operation (for example, a drag operation or the like) to move the above touch to the second graphic elements 712 to 715 on the fingerprint recognition integrated touch screen 710, (E.g., file transfer, file deletion, file sharing, cloud sharing) corresponding to the second graphic element 712 to 715 with respect to the object corresponding to the first graphic element 711, Storage, application execution, etc.) can be executed.

Next, referring to FIG. 7B, fingerprint information can be obtained only when a user touches a specific area (hereinafter, fingerprint recognition area) on the fingerprint recognition integrated touch screen 720. That is, whether the fingerprint recognition function is activated or not may be determined depending on the area where the touch operation is input to the fingerprint recognition integrated touchscreen 720. [

7B, if the user touches the first graphic element 721 on the fingerprint recognition integrated touch screen 720, information on the position where the above touch operation is performed is obtained If the user performs an operation (for example, a drag operation or the like) to move the above touch to the fingerprint recognition area 722, the fingerprint information of the user can be obtained through the above fingerprint recognition area 722 The first graphic element 721 or the instruction corresponding to the fingerprint recognition area 722 with respect to the object corresponding to the fingerprint recognition area 722 on the assumption that the user has the necessary authority, (E.g., file transfer, file deletion, file sharing, cloud storage, application execution, etc.).

Hereinafter, various embodiments that can be assumed in addition to the embodiment of FIG. 7 will be described.

First, according to an embodiment of the present invention, when a user performs an operation of touching a first graphic element on a fingerprint-integrated touch screen and the touch operation is not released (that is, when the user continues to touch the fingerprint- The fingerprint information of the user can be obtained through the fingerprint recognition area when the user moves to the fingerprint recognition area and performs the fingerprint recognition function.

According to an embodiment of the present invention, the fingerprint recognition area performing the fingerprint recognition function on the fingerprint-integrated touch screen can be temporarily displayed only on the basis of the touch operation of the user.

For example, the fingerprint recognition area may be displayed on the fingerprint recognition integrated touch screen only when the user inputs a touch operation corresponding to a predetermined gesture on the fingerprint recognition integrated touch screen.

In another example, the fingerprint recognition area may be displayed on the fingerprint integrated touch screen only when the user touches a predetermined graphic element on the fingerprint integrated touch screen. On the contrary, predetermined graphic elements may be displayed on the fingerprint recognition integrated touch screen only when the user touches the fingerprint recognition area on the fingerprint recognition integrated touch screen.

In another example, the fingerprint recognition area can be displayed on the fingerprint recognition integrated touch screen only while the user maintains the touch state for a predetermined graphic element on the fingerprint recognition integrated touch screen, The fingerprint recognition area displayed on the fingerprint recognition integrated touch screen may disappear.

According to an embodiment of the present invention, the display size of the fingerprint recognition area, the display format of the fingerprint recognition area, and the display format of the fingerprint recognition area may be changed according to the type or function of the graphic element touched by the user or the object corresponding to the graphic element on the fingerprint- , Display color, display method, display time, and the like.

Meanwhile, in accordance with an embodiment of the present invention, auditory feedback or tactile feedback may be provided in association with touching a predetermined graphic element displayed on a fingerprint-integrated touch screen by a user. In this case, The intensity, period (frequency), pattern, providing method, etc. of the auditory feedback or tactile feedback may be changed depending on the type of the graphic element to be touched or the type or function of the object corresponding to the graphic element.

Meanwhile, FIG. 8 is a diagram illustrating a configuration in which various commands are executed corresponding to various gestures according to an embodiment of the present invention.

Referring to FIG. 8, a user may input biometric information (i.e., fingerprint information) after performing various gestures, or may perform various commands by performing various gestures after the user inputs biometric information. Specifically, when the user directly touches the fingerprint recognition sensor to input fingerprint information (see (a) of FIG. 8) without performing any gesture, the user touches the fingerprint recognition sensor after performing a gesture to shake his / her hand In the case of inputting the fingerprint information (refer to FIG. 8B), it is assumed that the user touches the fingerprint recognition sensor after performing the gesture reversing the hand to input fingerprint information (refer to FIG. 8C) Different commands may be executed corresponding to each of the above three cases. In addition, when the user directly inputs the fingerprint information by touching the fingerprint recognition sensor without performing any gesture (see (d) of FIG. 8), the user touches the fingerprint recognition sensor to input fingerprint information, (FIG. 8 (e)), a case where the user touches the fingerprint recognition sensor to input the fingerprint information and then performs a gesture to reverse the hand (see FIG. 8 (f)), and the like In the above three cases, different commands can be executed corresponding to each of them.

Next, the communication unit 140 according to an embodiment of the present invention performs a function for allowing the user interface providing system 100 to communicate with an external device.

The control unit 150 controls the flow of data between the gesture recognition unit 110, the biometric information recognition unit 120, the command execution unit 130, and the communication unit 140. [ . That is, the control unit 150 controls the flow of data between the components of the user interface providing system from the outside, thereby controlling the gesture recognition unit 110, the biometric information recognition unit 120, the instruction execution unit 130, (140), respectively.

The embodiments of the present invention described above can be implemented in the form of program instructions that can be executed through various computer components and recorded in a non-transitory computer readable recording medium. The non-transitory computer readable medium may include program instructions, data files, data structures, etc., either alone or in combination. The program instructions recorded on the non-transitory computer-readable recording medium may be those specially designed and constructed for the present invention or may be those known to those skilled in the computer software arts. Examples of non-transitory computer-readable media include magnetic media such as hard disks, floppy disks and magnetic tape, optical recording media such as CD-ROMs, DVDs, magneto-optical media such as floppy disks magneto-optical media), and hardware devices specifically configured to store and execute program instructions such as ROM, RAM, flash memory, and the like. Examples of program instructions include machine language code such as those generated by a compiler, as well as high-level language code that can be executed by a computer using an interpreter or the like. The hardware device may be configured to operate as one or more software modules for performing the processing according to the present invention, and vice versa.

While the present invention has been particularly shown and described with reference to exemplary embodiments thereof, it is to be understood that the invention is not limited to the disclosed exemplary embodiments, but, on the contrary, Those skilled in the art will appreciate that various modifications, additions and substitutions are possible, without departing from the scope and spirit of the invention as disclosed in the accompanying claims.

Therefore, the spirit of the present invention should not be construed as being limited to the above-described embodiments, and all of the equivalents or equivalents of the claims, as well as the following claims, I will say.

100: User interface providing system
110: Gesture recognition unit
120: Biometric information recognition unit
130: Command execution unit
140:
150:
210, 310, 410, 510: touch screen
220, 320, 420, 520: Fingerprint sensor

Claims (1)

A method for providing a user interface.

Images