CN114489353A

CN114489353A - Input method and device based on hand and fingerprint sensor

Info

Publication number: CN114489353A
Application number: CN202210039381.9A
Authority: CN
Inventors: 冯建江; 周杰; 刘宗健
Original assignee: Tsinghua University
Current assignee: Tsinghua University
Priority date: 2022-01-13
Filing date: 2022-01-13
Publication date: 2022-05-13

Abstract

The invention discloses an input method and device based on a hand and a fingerprint sensor, wherein the method comprises the following steps: acquiring a plurality of sample images, the sample images including images of different positions of each finger region; testing the identifiability of a plurality of sample images, wherein the test result is that at least five independent parts of each finger are used for inputting and reading in fingerprints; based on the test result, acquiring an input first matching fingerprint by using a fingerprint sensor according to a preset virtual keyboard mode to store the first matching fingerprint into a fingerprint database, wherein the first matching fingerprint comprises images or characteristic templates of different finger positions; and acquiring a read-in second matching fingerprint by using the fingerprint sensor, performing matching operation on the second matching fingerprint and the first matching fingerprint stored in the fingerprint database to obtain a matching result, and displaying the matching result in real time through intelligent display equipment based on the matching result to form visual feedback. The invention is more stable and universal in principle, has better feasibility in operation and higher typing speed, and is more convenient for users.

Description

Input method and device based on hand and fingerprint sensor

Technical Field

The invention relates to the field of human-computer interaction, in particular to an input method and device based on a hand and a fingerprint sensor.

Background

With the gradual breakthrough of intelligent devices, related hardware devices also need to be breakthrough, wherein the typing keyboard puts higher requirements on the designed technology due to the relatively complex operability. Wearable equipment is present main design orientation, and it is more convenient than handheld device freely, more adapts to new life style. However, the limited input conditions of the wearable device can limit the implementation of the functions of the wearable device, for example, currently, smart glasses designed in mainstream usually do not support text input, voice input is inconvenient in many social environments, and in addition, limited touch gestures can only solve the most basic requirements, and are blank in the function of typing input.

Since human beings have innate perception functions for their limbs and can perform simple and effective interaction even without visual feedback, in recent years, a lot of palms and arms based on a body are designed as inputtable interfaces, and interaction efficiency is improved. Such as OmniTouch, Skinput, each use body-mounted sensors to sense body touch and provide visual feedback through a user interface projected on the body surface. These methods do not really disengage from the keyboard itself, and do not take advantage of body touch. Kitty and Mobile Lorm Glove use a Glove or palm with electronic contacts to detect finger touches, but the user needs to wear the Glove, which is not convenient enough in the interactive experience.

The PalmType detects the clicks of fingers at different positions of the palm through a sensor worn on the wrist, and designs an improved QWERTY layout mapping on the whole palm of a non-dominant hand, and the fingers of the dominant hand are used as a stylus for inputting. This is novel in design and takes advantage of proprioception well, thus optimizing QWERTY 39% faster than current touchpad keyboards. However, the wearable infrared sensor still has three obvious defects:

1. the positioning and pressing detection of the fingers are not accurate based on the positioning method, and the positioning is not accurate if the wrist moves to different degrees;

2. the range of the finger area is limited, each finger corresponds to 5 letters, and therefore, false touch is easily caused, and the input efficiency is reduced. On the basis of no visual feedback, people can hardly guarantee to directly press the target position;

3. during typing, the non-dominant hand needs to be completely extended and closed, and the finger used as the input stylus is repeatedly used without stop, so that the finger is easy to fatigue, the inconvenience of input is caused, and the user experience is not free enough.

Therefore, a new input mode needs to be found, and new improvements are made on the basis of keeping the advantage of using proprioception of the original wearable device.

Disclosure of Invention

The present invention is directed to solving, at least to some extent, one of the technical problems in the related art.

Therefore, the invention aims to improve the existing input mode, solve the defects of the prior art on the basis of keeping the advantage of using proprioception of the original wearable device, and provide an input method based on a hand and a fingerprint sensor.

Another object of the present invention is to provide an input device based on a hand and a fingerprint sensor.

In order to achieve the above object, the present invention provides an input method based on a hand and a fingerprint sensor, comprising the following steps:

obtaining a plurality of sample images, the sample images including images of different locations of each finger region;

testing the identifiability of the sample images based on the sample images to obtain a test result; the test result is that at least five independent parts of each finger are used for inputting and reading in fingerprints;

based on the test result, acquiring an input first matching fingerprint by using a fingerprint sensor according to a preset virtual keyboard mode so as to store the first matching fingerprint into a fingerprint database; wherein the first matching fingerprint comprises images or feature templates of different finger positions;

and acquiring a read-in second matching fingerprint by using the fingerprint sensor, performing matching operation on the second matching fingerprint and the first matching fingerprint stored in the fingerprint database to obtain a matching result, and displaying the matching result in real time through intelligent display equipment to form visual feedback based on the matching result.

According to the input method based on the hand and the fingerprint sensor, sample images of different positions of each finger and palm area are obtained; testing the identifiability of a plurality of sample images, wherein the test result is that at least five independent parts of each finger are used for inputting and reading in fingerprints; based on the test result, acquiring an input first matching fingerprint by using a fingerprint sensor according to a preset virtual keyboard mode to store the first matching fingerprint into a fingerprint database, wherein the first matching fingerprint comprises images or characteristic templates of different finger positions; and acquiring a read-in second matching fingerprint by using the fingerprint sensor, performing matching operation on the second matching fingerprint and the first matching fingerprint stored in the fingerprint database to obtain a matching result, and displaying the matching result in real time through intelligent display equipment based on the matching result to form visual feedback. The invention is more stable and universal in principle, has better feasibility in operation and higher typing speed, and is more convenient for users.

In addition, the input method based on the hand and the fingerprint sensor according to the above embodiment of the present invention may further have the following additional technical features:

further, the images of different locations of the single finger region include: images of the finger tip, finger pad, first joint, side of the finger, middle finger pad, and second joint.

Further, the acquiring and inputting a first matching fingerprint by using a fingerprint sensor according to a preset virtual keyboard mode to store the first matching fingerprint in a fingerprint database includes:

selecting a standard keyboard mode or a custom keyboard mode in the setting; wherein, the first and the second end of the pipe are connected with each other,

in the standard keyboard mode, according to position prompts given by the intelligent display equipment, a fingerprint sensor is used for inputting images or characteristic templates of different finger positions for multiple times so as to store the images or characteristic templates into a fingerprint database to finish inputting; wherein the different finger positions correspond to different symbols or functions on the virtual keyboard;

in the self-defined keyboard mode, the fingerprint sensor is used for inputting the image or the characteristic template of the corresponding self-defined position according to the sequentially appeared letters or symbols so as to be stored in the fingerprint database to finish inputting.

Further, the acquiring, by the fingerprint sensor, a second matching fingerprint, and performing matching operation with the first matching fingerprint stored in the fingerprint database to obtain a matching result includes:

and after the registration of the first matching fingerprint is finished, starting input, continuously acquiring a read-in image or characteristic template by using the fingerprint sensor, performing multiple matching operation on the read-in image or characteristic template and the registered image or characteristic template in the fingerprint database, mapping a matching result to a keyboard symbol or function corresponding to the part when the matching is successful and a preset position is triggered, and displaying the result in real time through intelligent display equipment.

Further, fingerprint matching based on a single sensor includes at least: a single thumb-worn fingerprint sensor, a single index finger-worn fingerprint sensor and a palm-fixed fingerprint sensor; wherein the content of the first and second substances,

the single thumb-worn fingerprint sensor comprising: presetting various keyboard layouts, taking a preset part of each finger as an input position, wearing a fingerprint sensor by a single thumb, performing matching operation in real time after each part is touched, and outputting corresponding letters or other instructions; the preset multiple keyboard layouts at least comprise a first keyboard layout, a second keyboard layout and a third keyboard layout.

Further, the first keyboard layout comprises:

the layout is carried out according to the sequence of letters, the fingertips of two hands oppositely form a first virtual keyboard, the middle finger pad, the first joint, the finger pad and the fingertips are selected as positions for inputting letters, the layout of the fingers in the first three rows corresponds to inputting 8 letters, the layout of the fingers in the fourth row corresponds to the remaining two letters y and z, and the finger side parts of the two index fingers are respectively used as a delete key and a space key;

pressing the thumb pad of the other hand by the thumb wearing the fingerprint sensor to trigger the SHIFT key of the keyboard; if the SHIFT key is pressed to display permanent switching, two index fingers in the first row correspond to Arabic numerals, and the positions of other fingers define punctuation marks by user; pressing the SHIFT key again returns to the alphabetical setting; while holding the SHIFT key and other alphabet displays the temporary switch.

Further, the second keyboard layout comprises:

laid out according to PalmType, the fingertips of the two hands oppose each other to form a second virtual keyboard, and the letters q and p in the first row are removed and placed at the two small thumbs in the fourth row, the letter g in the second row is placed on the finger side of the middle finger in the left hand, the two index fingers represent the first row, the two middle fingers represent the second row, and the two ring fingers represent the third row.

Further, the third keyboard layout comprises:

laying out according to the nine-square lattice key, and laying out all keyboards on the same hand to form a third virtual keyboard, wherein 8 letter positions on the third virtual keyboard correspond to fingertips and finger bellies of four fingers of a typing hand, the finger side of an index finger and the finger side of a middle finger respectively correspond to a delete key and a space key, and a first joint of the index finger corresponds to a plurality of symbol keys;

and pressing the letters at the same position by continuous pressing and interval preset time to print the corresponding letters or symbols.

Further, the single index finger wears a fingerprint sensor comprising:

according to the palm type layout, all keyboards are laid out in the same hand to form a fourth virtual keyboard, five letters on the right side of each row in the fourth virtual keyboard respectively correspond to two joints and three finger bellies of each finger, and other letters correspond to different areas of a palm;

and (4) laying out according to the Sudoku, laying out all keyboards in the same hand to form a fifth virtual keyboard, and setting custom selection preset positions on keys in the fifth virtual keyboard.

In order to achieve the above object, another aspect of the present invention provides an input device based on a hand and a fingerprint sensor, comprising:

a sample acquisition module for acquiring a plurality of sample images, the sample images including images of different positions of each finger region;

the sample testing module is used for testing the identifiability of the sample images to obtain a testing result based on the sample images; the test result is that at least five independent parts of each finger are used for inputting and reading in fingerprints;

the fingerprint input module is used for acquiring input first matched fingerprints by using a fingerprint sensor according to a preset virtual keyboard mode based on the test result so as to store the first matched fingerprints into a fingerprint database; wherein the first matching fingerprint comprises images or feature templates of different finger positions;

and the fingerprint matching module is used for acquiring a read-in second matching fingerprint by using the fingerprint sensor, performing matching operation on the second matching fingerprint and the first matching fingerprint stored in the fingerprint database to obtain a matching result, and displaying the matching result in real time through intelligent display equipment based on the matching result to form visual feedback.

According to the input device based on the hand and the fingerprint sensor, sample images of different positions of each finger area are obtained; testing the identifiability of a plurality of sample images, wherein the test result is that at least five independent parts of each finger are used for inputting and reading in fingerprints; based on the test result, acquiring an input first matching fingerprint by using a fingerprint sensor according to a preset virtual keyboard mode to store the first matching fingerprint into a fingerprint database, wherein the first matching fingerprint comprises images or characteristic templates of different finger positions; and acquiring a read-in second matching fingerprint by using the fingerprint sensor, performing matching operation on the second matching fingerprint and the first matching fingerprint stored in the fingerprint database to obtain a matching result, and displaying the matching result in real time through intelligent display equipment based on the matching result to form visual feedback. The invention is more stable and universal in principle, has better feasibility in operation and higher typing speed, and is more convenient for users.

The invention has the beneficial effects that:

1. the input principle based on positioning in all the previous methods is eliminated, and the method based on line matching is changed. The method utilizes the special lines on each hand, has extremely high specificity and accuracy, and does not depend on the positions of any other mobile equipment;

2. the available input range is expanded. Such as a thumb-worn sensor, almost any position that can be touched by the thumb can be assigned a new symbol or function. The proprioception of a person is further utilized, particularly, the perception of fingers is more efficient, and the space is fully utilized, for example, the sensor can be supported to press two fingers simultaneously to endow more possible symbols or functions;

3. the gesture of the hand is not required to be limited too much, all fingers are fully utilized, input operation can be performed in various environments, and the convenience degree and the degree of freedom of interaction are greatly improved;

4. the natural semantic redundant information is used for correcting fuzzy input and error input, so that the input accuracy is further improved;

5. the input function is further enriched by rationalizing the key design. For example, the two symbol sharing keys are realized by the SHIFT key, so that the arabic numerals and punctuation marks can be input through the sharing keys. For the conditions that special people or fingers are injured and the like, the keys are possibly not enough, and the problems can be solved through self-defining keys and sharing keys.

Additional aspects and advantages of the invention will be set forth in part in the description which follows and, in part, will be obvious from the description, or may be learned by practice of the invention.

Drawings

The foregoing and/or additional aspects and advantages of the present invention will become apparent and readily appreciated from the following description of the embodiments, taken in conjunction with the accompanying drawings of which:

FIG. 1 is a flow chart of a hand and fingerprint sensor based input method according to an embodiment of the invention;

FIG. 2 is a schematic diagram of images of different positions of a finger region according to an embodiment of the invention;

FIG. 3 is a schematic diagram of an alphabetically arranged virtual keyboard according to one embodiment of the invention;

FIG. 4 is a schematic diagram of a conventional QWERTY-type keyboard;

FIG. 5 is a schematic diagram of arranging virtual keyboards in keyboard order according to one embodiment of the invention;

FIG. 6 is a schematic diagram of a conventional nine-grid keyboard;

FIG. 7 is a schematic diagram of a virtual keyboard arranged in Sudoku order, according to one embodiment of the invention;

FIG. 8 is a schematic diagram of a single-handed virtual keyboard arranged in keyboard order according to one embodiment of the present invention;

FIG. 9 is a schematic diagram of a virtual keyboard arranged in Sudoku order, according to another embodiment of the invention;

FIG. 10 is a general flowchart illustrating the use of a virtual keyboard according to an embodiment of the present invention;

FIG. 11 is a flowchart illustrating registration for use of a virtual keyboard according to an embodiment of the present invention;

FIG. 12 is a schematic diagram illustrating an input flow for a virtual keyboard according to an embodiment of the present invention;

FIG. 13 is a diagram illustrating an input device based on a hand and a fingerprint sensor according to an embodiment of the present invention.

Detailed Description

It should be noted that the embodiments and features of the embodiments in the present application may be combined with each other without conflict. The present invention will be described in detail below with reference to the accompanying drawings in conjunction with embodiments.

In order to make the technical solutions of the present invention better understood, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.

An input method and apparatus based on a hand and a fingerprint sensor according to an embodiment of the present invention will be described with reference to the accompanying drawings, and first, an input method based on a hand and a fingerprint sensor according to an embodiment of the present invention will be described with reference to the accompanying drawings.

Fig. 1 is a flowchart of an input method based on a hand and a fingerprint sensor according to an embodiment of the present invention.

As shown in fig. 1, the input method based on the hand and the fingerprint sensor comprises the following steps:

in step S1, a plurality of sample images are acquired, the sample images including images of different positions of each finger region.

Specifically, the basic idea that most interactive keyboards are based on positioning input is abandoned, and the idea based on image matching is used for input operation. The invention is inspired by fingerprint matching, the classic fingerprint matching algorithm is mainly applied to inquiry of on-site fingerprints, registration and unlocking of fingerprints and the like, and the permanence and the particularity of the fingerprints and other lines can be more widely applied in the field of man-machine interaction, thereby further bringing convenience to the man-machine interaction. The reason for the feasibility of the method is that the human hand has unique and natural grains, and the grains at the finger joints and the grains at the finger backs can be used as recognizable positions except fingerprints and palm grains.

Step S2, testing the identifiability of the sample images based on the sample images to obtain a test result; the test result is that each finger has at least five independent parts for recording and reading in the fingerprint.

Specifically, the present invention tests the identifiability of images at different positions of the finger region, as shown in fig. 2, in order, the finger tip (a), the finger pad (b), the first joint (c), the finger side (d), the middle finger pad (e), and the second joint (f). The experimental result shows that the recognition performance of the part of the finger pad (b) is the best, the finger tip (a), the finger side (d) and the first joint (c) are arranged next, the middle finger pad (e) can also be recognized, but certain probability matching failure exists, and the recognizability of the second joint (f) is poor. The main reason is that the parts (a) - (d) are provided with partial fingerprint information more or less, so that effective fingerprint characteristics can be well extracted through an algorithm and more accurate matching is carried out; the middle finger belly (e) has clear lines and is smooth, and the identification operation can be carried out; due to the convex structure of the second joint (f), the registered image or the characteristic template of each time cannot clearly restore real grains, and the fingers can be folded to different degrees when bent, which brings difficulty to identification. Thus, based on this algorithm, at least five separate sites per finger are available for input.

Because the fingerprint sensor can acquire images, unlike a physical keyboard, the present invention supports the sensor to simultaneously press multiple finger positions, such as the thumb against the index finger, middle finger, and the like, and may assign new symbols or functions to represent the SHIFT key, space, ENTER key, and the like. The key design is specially suitable for the input principle based on matching, and the functions of the virtual keyboard can be further enriched.

In addition, the palm area can also be used for good identification, because palm prints have similar identifiability with fingerprints, and PalmRC has research and shows that there are 9 landmarks which can be obviously sensed on the palm based on the proprioception ability of the user. The joints on the back of the fingers are also provided with grains, so that the finger can be used for instruction operation with low accuracy requirements such as starting or ending and the like, and corresponds to the posture of making a fist when the thumb wears a sensor.

Step S3, based on the test result, acquiring an input first matching fingerprint by using a fingerprint sensor according to a preset virtual keyboard mode to store the first matching fingerprint in a fingerprint database; wherein the first matching fingerprint comprises images or feature templates of different finger positions.

Specifically, based on the principle, the keyboard can be designed in different operation designs and keyboard layouts, and the keyboard is more flexible and changeable in design compared with the positioning principle and the like, and is suitable for the requirements of different crowds. When a certain position can not be used because of the problems of poor texture quality, injury, disability and the like, the problem can be solved through the design of a user-defined key, a user can enter the operation according to the convenient key design of the user when registering, in addition, the shared key can be realized through the SHIFT key, and the key is possible to be sufficiently used. Several operation design ideas based on single sensor are proposed below, and the corresponding keyboard layout designs are discussed respectively.

As an example, a single thumb is worn on a sensor, which is inspired by preliminary experiments in the basic principle, with five effective parts ((a) - (e)) of each finger as input positions, and a single thumb is worn on a small fingerprint sensor, performs matching operation in real time after touching each part, and outputs corresponding letters or other instructions. A total of 9 fingers 45 independent positions can be operated, which is sufficient for keyboard input, except for the thumb wearing the sensor. In consideration of input efficiency and convenience of input, fingertips, finger bellies and finger sides should be considered first in the layout.

There are unlimited possibilities for the key arrangement, and specific layout designs include, but are not limited to, the following three layouts:

(1) the layout is carried out according to the order of letters, and two opposite fingertips of hands are imagined to form a virtual keyboard. The opposite of the selection fingertips forms a keyboard, but not the other possible reason is that this facilitates thumb-to-thumb presses to all positions. The middle pad (e), the first joint (c), the pad (b) and the fingertip (a) are selected as the positions where the letters can be input, as shown in fig. 3, a layout in which 8 letters can be input in each row is formed, and the little finger in the fourth row corresponds to the remaining two letters y and z, which takes into account the relative inconvenience of the thumb in pressing the little finger at another position. The finger side parts (d) of the two index fingers are very easy to press, and are respectively used as a delete key and a space key.

Pressing the thumb pad of the other hand with the thumb wearing the sensor (b) triggers the SHIFT key which is equivalent to a switching keyboard, if only pressing the SHIFT key, the permanent switching is realized, at the moment, the first row of two forefingers have 10 effective positions ((a) - (e)), which exactly correspond to 10 Arabic numerals, and the positions of other fingers can customize common punctuation marks. Holding down the SHIFT key again will return to the alphabetic setting, thereby implementing the share key. If the SHIFT key and other letters are pressed at the same time, the function of temporary switching is realized, and the functions of letter case switching and the like can be realized.

(2) The present layout migrates the familiar QWERTY layout to four fingers for ease of learning. As shown in fig. 4, for this familiar QWERTY structure, there are three rows of letters, corresponding to 10, 9, and 7 letters, respectively, and those familiar with the QWERTY layout are essentially familiar with the relative positions of the letters, and therefore can accurately type words blind without looking at the keyboard. Based on this, it is necessary to map the relative position of the layout letters to different positions of the finger.

This layout envisions opposing fingertips of both hands, constituting a virtual keyboard. Since only 8 positions of a row are available, which is not sufficient compared to the QWERTY, and the letters of different rows in the QWERTY are not strictly aligned, in order to minimize the learning cost and facilitate the man-machine interaction, as shown in fig. 5, the layout design is subjected to preliminary experiments, in which the letters q and p of the first row are removed and placed at the two small thumbs of the fourth row, the letter g of the second row is placed at the finger side of the middle finger of the left hand, and the relative positions other than this substantially conform to the human instinctive sense, the two index fingers represent the first row, the two middle fingers represent the second row, and the two ring fingers represent the third row. In accordance with method (1), the delete key and the space key are still arranged on the finger sides (d) of the two index fingers.

The SHIFT key is used as a switching key to still press the other thumb pad (b), permanent switching is realized when only the thumb is pressed, at the moment, 10 positions of the letter position in the first row and the side of the index finger respectively correspond to 10 Arabic numerals, and common symbols are customized in other positions. And the temporary switching function is realized when the thumb and other positions are pressed simultaneously.

(3) On the premise that some people are more familiar with the layout of the nine-square grid, the layout of the reduced keys can be well applied to the thought. The general layout of the squared figure is shown in fig. 6, in which there are 8 positions corresponding to a plurality of letters, besides, it should include deletion and space keys. This low key count requirement will make the layout simpler and easier to learn, and for convenience all keyboard layouts are arranged in the same hand, i.e. the sensor-worn hand, to achieve the goal of one-handed typing.

In this layout the 8 letter positions would correspond to the tips of the four fingers of the typewriter (a) and the pad (b), as shown in fig. 7, in the normal layout order, the letters a, b, c correspond to the tip of the index finger (a), then the middle, ring, little finger (a), then the pad (b), and so on. The index finger side and the middle finger side will correspond to the delete key and the space key, respectively. The first joint (c) of the index finger corresponds to a plurality of symbol keys.

When a specific letter is selected in actual typing, the letter is selected by repeatedly pressing the same position for many times. For example, if the letter a is to be typed out, the index finger tip (a) needs to be pressed once, and if the letter b is to be typed out, the index finger tip (a) needs to be pressed twice in quick succession. If the index finger tip (a) is pressed twice with a certain interval, aa two letters are typed. Three consecutive compressions of the first joint (c) of the index finger will type a symbol? Four consecutive presses will type a symbol! . By the continuous pressing and the adjustment of the interval time, a desired letter or symbol can be typed.

The SHIFT key is used as a switching key to still press the finger pad (b) of the other thumb, so that the permanent switching is realized when only the thumb is pressed, and the temporary switching function is realized by pressing the thumb and other letters. After permanent switching, the letter positions will correspond to Arabic numerals, the first joints (c) of the middle and ring fingers are added to the previous 8 positions to form 10 Arabic numeral positions, and other positions define common symbols.

It can be understood that the three layouts have advantages and disadvantages and are suitable for people with different habits. The three layouts are compared, and the third layout is the most convenient to use because the target of one-handed typing is realized. The first and second layouts are suitable for persons unfamiliar and familiar with the QWERTY layout, respectively.

Further, the comparison between the indicators of easy learning, user experience, functionality, speed, accuracy and the like and the PalmType method is carried out:

1. the second layout of the idea is equivalent to PalmType learning easiness, is the migration of variant QWERTY, continues the user experience capability and effectively reduces the learning cost based on the prior capability; the ease of learning of other ideas is not comparable because everyone has different typing habits. More possible layouts can provide more humanized choices, and the learning cost can be reduced to the greatest extent by selecting the proper layout;

2. the method of the invention can effectively utilize the sensing force of the finger tips, and needs to use different finger positions to find the sensor of the thumb during typing, rather than using a certain finger to repeatedly find the position on the flattened palm, the idea of using the finger as a pen touch is probably convenient during writing, but the typing is not convenient. From the perspective, the invention is more humanized, does not need a fixed posture for inputting, particularly, realizes a third idea, even basically realizes a typing task with one hand, and is more free and convenient when people can type in different environments such as walking, class and the like, so that the user experience is better;

3. thanks to a new input principle, compared with the palm type, the method and the device make full use of limited input positions, enrich the functions of the keyboard by introducing the SHIFT shared key and simultaneously pressing the design of a plurality of fingers, and have no Arabic numerals and punctuation marks in the palm type, so that the method and the device are more comprehensive in functions;

4. WPM is used in PalmType to measure input speed, and corrected error rate and uncorrected error rate are used to measure input accuracy. The invention has obvious advantages on the basis that an experimenter selects a layout suitable for the experimenter and has fully trained and learned. Positioning-based methods will give a challenge to the accuracy of the input, while matching-based will not present this problem; the input speed is greatly improved by fully utilizing all fingers instead of a single finger; the problem of palm type densely arranged letters causing false touches will also be solved, as finger-based proprioception does not present such errors. The number of keys is greatly reduced by one-hand typing, the input accuracy can be improved during use, and the keyboard is more convenient and flexible.

As yet another example, a single index finger wearing a sensor, which continues the PalmType design, a single index finger of a dominant hand or other finger other than the thumb wearing a sensor acting as a stylus, and the entire palm of another non-dominant hand acting as a virtual keyboard, perform input operations.

Layout designs include, but are not limited to, the following two designs:

(1) the palm layout imitating the palm type is designed, five letters on the right side of each line in the palm type correspond to two joints and three finger bellies of each finger respectively, and other letters correspond to different areas of the palm. Since the matching method is still adopted instead of the positioning method, only the relative position relationship of letters and the identifiability of textures can be considered, and different parts of fingers can provide accurate matching results, which can be divided more finely, but the position relationship of the QWERTY layout is also sacrificed.

In this arrangement, as shown in fig. 8, the letters of each finger are divided more finely, and since the sensor is worn by the index finger of the other hand, both finger sides of the finger can be pressed easily, and thus 5 positions of the first joint (c), the pad (b), the tip (a), and the two finger side portions (d) (d') of the finger can be used. In addition, a plurality of finger joints can be pressed at the same time, and new symbol definitions can be given. The thumb of the hand serving as the keyboard can also be utilized, and the two finger sides (d) (d'), the finger pad (b) and the finger tip (a) of the thumb head can correspond to the up-down, left-right and functional keys respectively.

Touching the thumb of the same hand with the sensor-worn index finger will trigger the SHIFT key, and in the permanently switched mode, the five valid positions ((a) (b) (c) (d) (d')) of the index finger and middle finger, respectively, correspond to 10 arabic numerals, and the other positions define symbols. If the thumb of the hand wearing the sensor and other letters of the hand serving as the keyboard are pressed at the same time, the temporary switching function can be realized, and the purposes of adjusting the letter case and the like are realized. The temporary switching function is feasible, firstly, the operation can be realized by pressing the positions simultaneously, in addition, different from a physical keyboard, the fingerprint sensor acquires the image or the characteristic template and matches the image or the characteristic template, the sensor can be supported to press a plurality of positions simultaneously, the corresponding correct registered image or the characteristic template is matched, and new symbols or functions are given.

(2) The layout of the nine-square grids can be designed when the finger is worn by the user, and the finger-worn sensor is suitable for people with different habits. The key setting can still self-define and select 9 main positions. Because the area of the palm is very big, if the layout is in the palm, the distance between the keys is far, and the keys are not easy to quickly press to a specific position and are accurately identified during quick typing. The standard keyboard squared layout here still migrates to a different location on the fingers.

As shown in fig. 9, the 9 positions correspond to the first joints (c), the pad (b) and the tip (a) of the index finger, the middle finger and the ring finger, respectively, in accordance with the layout of fig. 6. Like the nine-square lattice of the idea 1, selection of specific letters can be realized by continuously pressing and controlling the pressing interval time. The SHIFT key is still at the thumb position of the sensor-wearing hand, and the permanent switching can be realized by pressing the thumb, or the temporary switching function can be realized by simultaneously pressing the SHIFT and the letter.

It should be understood that both of the above arrangements are designed with the index finger as the stylus and the other hand as the keyboard. The two schemes aim at people with different habits, the first layout has the same defect in accuracy as the palm type, the identification of partial positions is possibly poor, and the problem is solved if a palm area has an area with higher identification efficiency for user-defined layout; in addition, the range of this layout operation is more concentrated and the posture of the hand serving as the keyboard is not limited. In general, the idea is to sacrifice the learning easiness of the original layout design by the matching precision; the second layout is that the nine-square lattice based on the recognition algorithm is relatively more stable, the accuracy can be ensured when the keyboard is pressed, the probability of mistaken touch is reduced, and the keyboard is simpler and is easier to learn and operate. But this layout will not embody the advantage of the matching method over the positioning method more precisely.

It should be noted that, in the design of the virtual keyboard with a single sensor, there are many other diversified design ideas besides the above two main ideas. For example, the sensor can be fixed at the palm of the hand, and the finger of the other hand touches different parts of the sensor for inputting, but the layout of 26 keys is difficult to arrange in consideration of convenience, and is more suitable for the layout of the nine-square grid.

And step S4, acquiring the read-in second matching fingerprint by using the fingerprint sensor, performing matching operation on the second matching fingerprint and the first matching fingerprint stored in the fingerprint database to obtain a matching result, and displaying the matching result in real time through intelligent display equipment to form visual feedback based on the matching result.

Specifically, visual feedback is formed through real-time display of intelligent display equipment based on matching results, the intelligent display equipment comprises but is not limited to intelligent glasses and VR helmets, and the visual feedback is finally formed to complete input human-computer interaction.

It should be appreciated that the present invention can utilize a VR headset or smart eyewear device, a fingerprint sensor connected to a computer, and a user's hand as a prototype of a virtual keyboard design wearable analog keyboard. The fingerprint sensor worn on the finger can acquire the image or the characteristic template, and the image or the characteristic template is matched with the stored registered image or the stored characteristic template to obtain a result, and the result is displayed in real time through the intelligent glasses to form visual feedback. This visual feedback facilitates the user typing in a natural posture without the need to look down at one's hands.

Further, the general flow, the registration flow and the input flow of the virtual keyboard of the present invention are shown in fig. 10, 11 and 12, respectively. Firstly, a user starts the equipment, selects a keyboard mode in the setting, enters a registration interface of a start key if the equipment is not registered, a sensor is started, the user needs to input images or characteristic templates for starting or closing the keyboard for many times according to requirements, for example, in a keyboard mode of carrying the sensor by a thumb, lines at a knuckle of a finger obtained by the user through a fist-making posture are obtained by the sensor and are stored;

then, a user selects a standard keyboard or a self-defined keyboard in the setting, the user needs to input images or feature templates of different finger positions for many times according to position prompts given by the intelligent glasses in the standard keyboard mode, the positions correspond to different symbols or functions on the virtual keyboard, and the user can select to input again or finish inputting after registering every time; in the user-defined keyboard mode, a user needs to input an image or a feature template of a corresponding user-defined position according to sequentially appearing letters or symbols;

after finishing the fingerprint registration of all keyboard symbols or functions, the user can start the input, the sensor can continuously acquire the read-in images or characteristic templates and carry out multiple matching operation with the registered images or characteristic templates in the library. Specific sampled feature templates include, but are not limited to, SIFT, SURF, fingerprint minutiae, and dense feature points, and specific matching methods include, but are not limited to, point matching, image matching techniques. And when the matching is successful and the effective position is triggered, mapping the matching result to the keyboard symbol or function corresponding to the part, and displaying the result in real time through the intelligent glasses.

Through the steps, a plurality of sample images are obtained, wherein the sample images comprise images of different positions of each finger area; testing the identifiability of a plurality of sample images, wherein the test result is that at least five independent parts of each finger are used for inputting and reading in fingerprints; based on the test result, acquiring an input first matching fingerprint by using a fingerprint sensor according to a preset virtual keyboard mode to store the first matching fingerprint into a fingerprint database, wherein the first matching fingerprint comprises images or characteristic templates of different finger positions; and acquiring a read-in second matching fingerprint by using the fingerprint sensor, performing matching operation on the second matching fingerprint and the first matching fingerprint stored in the fingerprint database to obtain a matching result, and displaying the matching result in real time through intelligent display equipment based on the matching result to form visual feedback. The invention is more stable and universal in principle, has better feasibility in operation and higher typing speed, and is more convenient for users.

It should be noted that there are various input method implementation manners based on the hand and the fingerprint sensor, but no matter how the specific implementation manner is, as long as the method solves the deficiency of the existing virtual keyboard, the method is a solution to the problems in the prior art and has corresponding effects.

In order to implement the above embodiment, as shown in fig. 13, the present embodiment further provides an input device 10 based on a hand and a fingerprint sensor, where the input device 10 includes: the system comprises a sample acquisition module 100, a sample testing module 200, a fingerprint input module 300 and a fingerprint matching module 400.

A sample acquisition module 100 for acquiring a plurality of sample images, the sample images including images of different positions of each finger region;

the sample testing module 200 is configured to test identifiability of the plurality of sample images based on the plurality of sample images to obtain a testing result; the test result is that each finger has at least five independent parts for inputting and reading in the fingerprint;

the fingerprint input module 300 is used for acquiring input first matched fingerprints by using a fingerprint sensor according to a preset virtual keyboard mode based on a test result so as to store the first matched fingerprints into a fingerprint database; wherein the first matching fingerprint comprises images or feature templates of different finger positions;

and the fingerprint matching module 400 is used for acquiring the read-in second matching fingerprint by using the fingerprint sensor, performing matching operation on the second matching fingerprint and the first matching fingerprint stored in the fingerprint database to obtain a matching result, and displaying the matching result in real time through an intelligent display device to form visual feedback based on the matching result.

According to the input device based on the hand and the fingerprint sensor, a plurality of sample images are obtained, wherein the sample images comprise images of different positions of each finger area; testing the identifiability of a plurality of sample images, wherein the test result is that at least five independent parts of each finger are used for inputting and reading in fingerprints; based on the test result, acquiring an input first matching fingerprint by using a fingerprint sensor according to a preset virtual keyboard mode to store the first matching fingerprint into a fingerprint database, wherein the first matching fingerprint comprises images or characteristic templates of different finger positions; and acquiring a read-in second matching fingerprint by using the fingerprint sensor, performing matching operation on the second matching fingerprint and the first matching fingerprint stored in the fingerprint database to obtain a matching result, and displaying the matching result in real time through intelligent display equipment based on the matching result to form visual feedback. The invention is more stable and universal in principle, has better feasibility in operation and higher typing speed, and is more convenient for users.

It should be noted that the foregoing explanation of the embodiment of the input method based on the hand and the fingerprint sensor is also applicable to the input device based on the hand and the fingerprint sensor of this embodiment, and will not be described herein again.

Furthermore, the terms "first", "second" and "first" are used for descriptive purposes only and are not to be construed as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one such feature. In the description of the present invention, "a plurality" means at least two, e.g., two, three, etc., unless explicitly specified otherwise.

In the description herein, references to the description of the term "one embodiment," "some embodiments," "an example," "a specific example," or "some examples," etc., mean that a particular feature, structure, material, or characteristic described in connection with the embodiment or example is included in at least one embodiment or example of the invention. In this specification, the schematic representations of the terms used above are not necessarily intended to refer to the same embodiment or example. Furthermore, the particular features, structures, materials, or characteristics described may be combined in any suitable manner in any one or more embodiments or examples. Furthermore, various embodiments or examples and features of different embodiments or examples described in this specification can be combined and combined by one skilled in the art without contradiction.

Although embodiments of the present invention have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting the present invention, and that variations, modifications, substitutions and alterations can be made to the above embodiments by those of ordinary skill in the art within the scope of the present invention.

Claims

1. An input method based on a hand and a fingerprint sensor is characterized by comprising the following steps:

based on the test result, acquiring an input first matching fingerprint by using a fingerprint sensor according to a preset virtual keyboard mode so as to store the first matching fingerprint into a fingerprint database; wherein the first matching fingerprint comprises images or feature templates of different finger locations;

2. The method of claim 1, wherein the images of different locations of the single finger region comprise: images of the finger tip, finger pad, first joint, side of the finger, middle finger pad, and second joint.

3. The method according to claim 2, wherein the acquiring the first matching entered fingerprint by using the fingerprint sensor according to the preset virtual keyboard mode for storing in the fingerprint database comprises:

selecting a standard keyboard mode or a custom keyboard mode in the setting; wherein the content of the first and second substances,

4. The method of claim 3, wherein the obtaining of the second matching fingerprint by the fingerprint sensor and the matching operation with the first matching fingerprint stored in the fingerprint database to obtain the matching result comprises:

5. The method of claim 4, wherein fingerprint matching based on a single sensor comprises at least: a single thumb-worn fingerprint sensor, a single index finger-worn fingerprint sensor and a palm-fixed fingerprint sensor; wherein the content of the first and second substances,

6. The method of claim 5, wherein the first keyboard layout comprises:

the thumb wearing the fingerprint sensor presses the thumb pad of the other hand to trigger the SHIFT key of the keyboard; if the SHIFT key is pressed to display permanent switching, two index fingers in the first row correspond to Arabic numerals, and the positions of other fingers define punctuation marks by user; pressing the SHIFT key again returns to the alphabetical setting; while holding the SHIFT key and other alphabet displays the temporary switch.

7. The method of claim 5, wherein the second keyboard layout comprises:

8. The method of claim 5, wherein the third keyboard layout comprises:

9. The method of claim 5, wherein the single index finger-worn fingerprint sensor comprises:

and (4) carrying out layout according to the Sudoku, and laying all the keyboards in the same hand to form a fifth virtual keyboard, wherein the keys in the fifth virtual keyboard are provided with custom-selected preset positions.

10. An input device based on a hand and fingerprint sensor, comprising:

the fingerprint input module is used for acquiring input first matched fingerprints by using a fingerprint sensor according to a preset virtual keyboard mode based on the test result so as to store the first matched fingerprints into a fingerprint database; wherein the first matching fingerprint comprises images or feature templates of different finger locations;