WO2021128414A1

WO2021128414A1 - Wearable device and input method thereof

Info

Publication number: WO2021128414A1
Application number: PCT/CN2019/130475
Authority: WO
Inventors: 张超; 刘超
Original assignee: 歌尔股份有限公司
Priority date: 2019-12-25
Filing date: 2019-12-31
Publication date: 2021-07-01
Also published as: CN111007942A

Abstract

A wearable device and an input method thereof. Preset characters and preset areas of a hand are superimposed and displayed by means of a display lens (1) so that a user can view the preset characters displayed in the preset areas by means of the display lens; and subsequently, a preset character in a preset area is selected according to the preset area to which a preset feature of a finger of a user points, so as to perform an operation corresponding to the preset character. The hand is directly used as an input method, so that no additional devices are needed, thus avoiding disadvantages such as poor convenience of wired connection and low reliability of wireless connection, achieving high portability and reliability, expanding the use scope and use scenarios of the wearable device, and improving practicability and popularity.

Description

Wearable equipment and its input method

This application claims the priority of a Chinese patent application filed with the Chinese Patent Office on December 25, 2019, the application number is 201911360493.9, and the invention title is "a wearable device and its input method", the entire content of which is incorporated herein by reference Applying.

Technical field

This application relates to the technical field of wearable devices, in particular to a wearable device and an input method thereof.

Background technique

AR (Augmented Reality) glasses are a new type of glasses for "augmented reality" applications. Most AR glasses have diversified functions such as display, taking pictures, video calls, processing text messages, emails, games and entertainment. On the basis of seeing the real scene, it can see virtual information and even interact with it at the same time. It is a new form of intelligent hardware products in the future.

The external input devices (hereinafter referred to as peripherals) of the existing AR glasses usually include, for example, a keyboard, a handle, a stylus, and a ring. These peripherals can be connected to the AR glasses in a wired way, but this kind of connection will bring inconveniences in use, storage, and carrying; these peripherals can also be connected to the AR glasses in a wireless way, such as Bluetooth , WiFi, etc. Although this method solves some of the inconvenience caused by wired connection, it has performance requirements for the battery module, and because it is wirelessly connected, it is prone to loss of peripherals, and once the battery is consumed In the case of exhaustion or loss of peripherals, the use of most of the functions of AR glasses will be restricted, or even cannot be used normally, and the reliability is low.

Therefore, how to provide a solution to the above technical problems is a problem that needs to be solved by those skilled in the art at present.

Summary of the invention

The purpose of this application is to provide a wearable device and its input method, without the shortcomings of poor wired connection and low reliability of wireless connection, and high portability and reliability; in addition, it can also expand the use of wearable devices Scope and usage scenarios, high practicability and popularity.

In order to solve the above technical problems, the present application provides an input method for a wearable device, the wearable device includes a display lens and an image acquisition device for taking a picture of the front of the display lens to obtain a front image. The input method include:

Identifying the preset area of the user's hand from the front image;

Superimposing and displaying preset characters to the preset area through the display lens;

Identifying the preset feature of the input finger from the front image;

It is determined whether the preset feature satisfies the input confirmation condition, and if so, an operation corresponding to the preset character corresponding to the preset area where the preset feature is located is performed.

Preferably, when the preset character corresponding to the preset area where the preset feature is located is a text character,

Performing the operation corresponding to the preset character corresponding to the preset area where the preset feature is located includes:

The text is constructed according to the text characters.

Preferably, after composing the text according to the text characters, the method further includes:

Judging whether the user's hand is performing a preset action based on the front image;

If yes, execute the functional operation corresponding to the preset action.

Preferably, when the preset character corresponding to the preset area where the preset feature is located is a functional character,

Perform functional operations corresponding to the functional characters.

Preferably, the function characters include confirmation characters and/or deletion characters;

Perform functional operations corresponding to the functional characters, including:

Perform a confirmation operation and/or delete operation corresponding to the function character.

Preferably, the predetermined area includes the knuckles and/or the palm of a finger, and the predetermined feature is a nail.

Preferably, determining whether the preset feature satisfies the input confirmation condition includes:

It is determined whether the preset feature stops moving and the duration of the stop movement exceeds a duration threshold.

It is determined whether the distance between the preset feature and the preset area where it is located is less than a distance threshold.

Preferably, determining whether the distance between the preset feature and the preset area of its location is less than a distance threshold includes:

Acquiring the first depth information of the preset feature and the second depth information of the preset area where it is located;

Performing a difference between the second depth information and the first depth information to obtain a difference;

It is determined whether the difference is less than the distance threshold.

In order to solve the above technical problems, the present application also provides a wearable device, the wearable device includes a display lens, and further includes:

An image acquisition device for taking a picture of the front of the display lens to obtain a front image;

Optical machine for projecting preset characters on the display lens based on the control of the processor;

The processor is used to implement the steps of the above-mentioned input method of the wearable device when the computer program is executed.

This application provides an input method for a wearable device. In this method, preset characters and a preset area of the hand can be superimposed and displayed through the display lens, so that the user can see the preset displayed in the preset area through the display lens. Set a character, and subsequently select a preset character in the preset area according to which preset area the user points to by the preset feature of the finger, and then perform an operation corresponding to the preset character. It can be seen that this method directly uses the hand as an input means without additional peripherals, and does not have the shortcomings of poor wired connection and low reliability of wireless connection, and has high portability and reliability; in addition, it can also expand wearable devices. The scope of use and usage scenarios of the, the practicability and popularity are high.

The application also provides a wearable device, which has the same beneficial effects as the aforementioned input method.

Description of the drawings

In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following will briefly introduce the prior art and the drawings needed in the embodiments. Obviously, the drawings in the following description are only some of the present application. Embodiments, for those of ordinary skill in the art, without creative work, other drawings can be obtained based on these drawings.

FIG. 1 is a process flowchart of a wearable device input method provided by this application;

Figure 2 is a schematic structural diagram of a wearable device provided by this application;

FIG. 3 is a diagram of the correspondence between preset areas of hands and preset characters provided by this application;

4 is a diagram of the positional relationship between the hand and the wearable device when the wearable device provided in this application performs data input;

Figure 5 is a schematic diagram of a right-hand input provided by this application;

FIG. 6 is a schematic diagram of a user making a fist provided by this application;

FIG. 7 is a schematic structural diagram of another wearable device provided by this application.

Detailed ways

The core of this application is to provide a wearable device and its input method. There are no shortcomings such as poor convenience of wired connection and low reliability of wireless connection, and high portability and reliability; in addition, the use of wearable devices can be expanded Scope and usage scenarios, high practicability and popularity.

In order to make the purpose, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be described clearly and completely in conjunction with the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments It is a part of the embodiments of the present application, but not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by those of ordinary skill in the art without creative work shall fall within the protection scope of this application.

In this application, the wearable device has a function of superimposing virtual and real scenes. The wearable device here can be, but not limited to, AR glasses. Based on this, in order to solve the problems of poor convenience when using wired connection and low reliability when using wireless connection in the peripherals of wearable devices in the prior art, the design idea of this application is to divide the user's hand into fingerprint lines in advance. Multiple areas to establish the correspondence between each area and characters. In the subsequent use process, only the preset area of the user's hand needs to be recognized, and each preset character and the preset area are superimposed and displayed, and the user can determine the user through the preset characteristics of the fingers of the other hand Which character to select and perform the operation corresponding to that character.

Please refer to FIG. 1. FIG. 1 is a process flowchart of a wearable device input method provided by this application.

The wearable device includes a display lens and an image acquisition device for taking a picture of the front of the display lens to obtain a front image.

The input method includes:

S11: Identify the preset area of the user's hand from the front image;

S12: Superimpose the preset characters to the preset area through the display lens;

S13: Identify the preset feature of the input finger from the front image;

S14: Judge whether the preset feature satisfies the input confirmation condition, if yes, go to S15;

S15: Perform an operation corresponding to the preset character corresponding to the preset area where the preset feature is located.

Please refer to FIG. 2, which is a schematic structural diagram of a wearable device provided by this application. The wearable device is equipped with a display lens and an image acquisition device, and also includes an optical machine and a processor. The image acquisition device here can be a camera, and the camera can be an existing camera on the wearable device, and the camera will be in real time or periodically The front of the display lens is photographed to obtain a front image, and the processor performs image recognition on the front image to identify the preset area and preset features. In addition, when the wearable device is working, the processor can project virtual images and other information into the coupling area of the display lens through an optical machine, and the display lens couples the projected light in the coupling area into the glass substrate of the display lens. , The light is transmitted to the coupling-out area of the display lens through the principle of "total reflection", and then released on the display lens. At the same time, the outside world can also be seen through the display lens, so that virtual and real scenes can be superimposed.

Specifically, please refer to FIG. 3, which is a diagram of the correspondence between preset areas of hands and preset characters provided by this application. In practical applications, you can first divide the opponent's preset areas such as fingerprint lines and hand shapes, where the preset areas can be the knuckles and palms of the fingers, where the knuckles of the fingers can be the thumb, index finger, One or more knuckles of the middle finger, ring finger, and little finger. In addition, functions are assigned to each preset area, that is, preset characters are defined for each preset area. Among them, the character here can be a single character or a string, where a single character can be a number, letter, arithmetic symbol, punctuation mark, etc., and the string can be English (for example, delete) or Chinese (for example, confirm, delete) Or expressions, etc., this application is not particularly limited here.

In addition, a preset area can correspond to one character or multiple characters. For example, in FIG. 3, one knuckle corresponds to multiple characters. Specifically, it divides the five fingers, especially the four fingers from the index finger to the little finger, into 3*4=12 different preset areas by the inter-phalangeal lines in the palm of the hand. At the same time, the processor will divide each finger into 3*4=12 different preset areas. For the location of the preset area, apply the preset characters to the hand according to the algorithm that has been set similar to the Jiugongge input method. For example, when the left hand is used as the method of the Jiugongge input method, the preset area of the index finger is from left to right, Assign the value 123/, respectively. ! , ABC, DEF, in the same way, according to the usual input method, assign the remaining finger segments to the remaining other letters or values. According to different input methods, such as numbers, commonly used symbols, etc., you can change the assignment algorithm corresponding to the preset area according to actual needs. The addition of this new type of finger nine-grid input method will make wearable devices more viscous and more practical in future smart hardware products, making wearable devices more popular in the future.

After establishing the corresponding relationship between the preset area of the hand and the preset characters, the user can use one hand (e.g. the left hand) to input text such as password input and text message reply while using the wearable device. ) Extends to the front of the display lens. Please refer to FIG. 4 for details. FIG. 4 is a diagram of the positional relationship between the hand and the wearable device when the wearable device provided by this application performs data input. At this time, the processor performs image recognition on the front image collected by the image collecting device to identify the preset area of the hand. After recognizing the preset area of the hand, the processor will project the virtual preset character on the display lens according to the correspondence between the preset area and the preset character, and since the user can see the hand through the display lens, standing on the user From the angle of view, the user will see the preset characters displayed in the preset area of the user's hand.

It should be noted that even if the hand moves during use, because the image acquisition device collects the front image in real time or periodically (the cycle is usually very small), the processor will re-accurately identify the prediction from the moved front image. Set the area, and superimpose the preset characters to the preset area through the display lens, realize the purpose of moving the preset characters with the preset area of the hand, and ensure the real-time matching of the preset characters and the preset area.

Please refer to FIG. 5, which is a schematic diagram of a right-hand input provided by this application.

After the user sees the preset character displayed in the preset area, when he wants to select which character, he can move the finger of the other hand (for example, the right hand) (referred to as the input finger in this application) to the preset area corresponding to the character In order to accurately identify the preset area pointed by the input finger, a preset feature can be set on the input finger. The preset feature here requires that other areas of the finger can be distinguished to avoid misrecognition. After the preset feature of the input finger is recognized through the front image, it is determined whether the preset feature meets the input confirmation condition, and if so, the operation corresponding to the preset character corresponding to the preset area where the preset feature is located is performed. Here, the input confirmation condition is used to determine whether the user wants to select the preset character at the location of the preset feature.

In addition, considering that some preset characters are directly input as text characters, the processor will directly receive such characters and form text based on such characters, such as the password or Chinese character information when inputting a password; some preset characters are used for Performing the corresponding function does not constitute the text itself. Such preset characters are used to implement a certain function through it, such as a character used to indicate "confirmation" after entering a password. It can be seen that, depending on the type of preset characters, the operations to be performed by the processor are also different, and the specific operations to be performed can be customized in advance according to the preset characters.

In summary, this application provides an input method for a wearable device. In this method, the preset characters and the preset area of the hand can be superimposed and displayed through the display lens, so that the user can see the display on the preset through the display lens. The preset characters of the area are subsequently selected according to which preset area the user points to by the preset characteristics of the finger, and then the operation corresponding to the preset characters is performed. It can be seen that this method directly uses the hand as an input means without additional peripherals. It does not have the shortcomings of poor convenience of wired connection and low reliability of wireless connection. It has high portability and reliability, and is more integrated. It makes the interaction between people and the virtual scene stronger, more immersive and interesting; in addition, it can also expand the use range and use scenarios of wearable devices, with high practicability and popularity.

On the basis of the above embodiment:

As a preferred embodiment, when the preset character corresponding to the preset area where the preset feature is located is a text character,

Perform operations corresponding to the preset characters corresponding to the preset area where the preset feature is located, including:

The text is constructed based on the characters of the text.

Specifically, the text characters here are usually numbers, letters, punctuation marks, etc. When inputting Chinese characters or passwords, text characters can be used to form Chinese characters, passwords and other texts, such as passwords directly composed of numbers and letters, or Chinese characters obtained by spelling through letters, etc. The processor is determining preset features When the selected preset character is a text character, the text is formed directly according to the text character. Wherein, when composing the text, the processor may specifically determine whether the character is directly used as the text according to the current scene (for example, the current scene is the password input mode or the short message reply mode, etc.) or to obtain the text after performing other subsequent processing based on the character.

In addition, taking the current need for Chinese character reply as an example, what kind of processing the processor performs after receiving the letter is also related to the keyboard mode of the current preset character and the corresponding input method. For example, in Figures 3 and 5, the preset characters are projected onto the preset area in a keyboard manner similar to Jiugongge. Correspondingly, the processor will also use the input method similar to Jiugongge for text character processing during processing. The processor will obtain the corresponding Chinese characters according to the letters entered by the user and combined with the input method, and display the corresponding Chinese characters in other preset areas, such as the palm. Also, because the same pinyin may correspond to multiple Chinese characters, the processor can select multiple The Chinese characters are displayed on the palm so that the user can select the Chinese characters he needs.

Take the current need to enter a password and the password is composed of numbers and letters as an example. Please refer to Figure 5. When selecting letters, you can directly use the letters as text and display the letters. When you need to switch between letters and numbers, you can select 123/ Corresponding to the preset area, after the processor determines that the preset feature of the input finger selects the preset area, it switches the letters in the preset area to numbers so that subsequent users can select numbers, and after selecting numbers, the processor directly changes The number is used as text and the number is displayed.

It can be seen that this method directly uses the hand as an input means without additional peripherals. It does not have the shortcomings of poor convenience of wired connection and low reliability of wireless connection. It has high portability and reliability, and is more integrated. It makes the interaction between people and the virtual scene stronger, more immersive and interesting; in addition, it can also expand the use range and use scenarios of wearable devices, with high practicability and popularity.

As a preferred embodiment, after composing the text according to the text characters, it further includes:

Judge whether the user's hand is performing a preset action based on the front image;

If yes, perform the functional operation corresponding to the preset action.

Specifically, after the user has entered the text, operations such as confirmation, sending, and deletion may be performed. In order to realize this function, in this embodiment, considering that the number of preset areas on hand is limited, in order to reduce the occupation of the preset area, the corresponding function can also be realized by the user performing a preset action in this embodiment. Among them, the functional operations here can be confirmation operations, sending operations, deletion operations, etc., and the preset actions and the functional operations corresponding to the preset actions can be defined according to actual needs.

Please refer to FIG. 6, which is a schematic diagram of a user making a fist provided by this application. In practical applications, you can set the handshake as a confirmation operation. For example, after the user enters the password, the user can shake hands (either left-handed or right-handed). When the processor determines that the user's hands are shaking based on the front image, the password input is considered complete at this time. Or, the user can shake hands after editing the reply text, and the processor can send the reply text when it is determined based on the front image that the user's hand is shaking hands.

It can be seen that this method can reduce the occupation of the preset area of the functional operation opponent, and realize the expansion of more functions.

As a preferred embodiment, when the preset character corresponding to the preset area where the preset feature is located is a functional character,

Perform the function operation corresponding to the function character.

In addition to using the above embodiments to confirm, send, delete and other operations after text input, you can also set functional characters to implement these functions. Taking Figure 3 and Figure 5 as an example, you can combine the knuckle or palm of your thumb with the function Corresponding to characters, where the functional characters here may include confirmation characters and/or deletion characters; accordingly, performing functional operations corresponding to the functional characters includes: performing confirmation operations and/or deletion operations corresponding to the functional characters.

Of course, the functional characters here may also include other characters, which are not specifically limited in this application.

As a preferred embodiment, the preset area includes the knuckles and/or palms of the fingers, and the preset feature is nails.

Specifically, since there are deep fingerprint lines between the various knuckles of the finger, the knuckles and the palm can be accurately identified by combining the fingerprint lines and the shape of the palm. In addition, the nail on the finger is clearly distinguished from other parts of the finger. Therefore, the nail can be selected as a preset feature, and the processor can recognize the nail according to the shape of the nail. Of course, the preset area and the preset feature may also be other areas, which are not specifically limited in this application, and are determined according to actual conditions.

As a preferred embodiment, determining whether the preset feature meets the input confirmation condition includes:

Determine whether the preset feature stops moving and the duration of the stop movement exceeds the duration threshold.

The applicant considers that when the user wants to select a preset area, the preset feature must point to the preset area, and the preset feature may pass through other preset areas during the movement. In order to distinguish, in this embodiment , The preset feature of the input finger can be kept at the position of the preset area to be selected for a period of time to distinguish it from the input finger passing through other preset areas briefly during the movement.

Specifically, the processor will determine whether the preset feature stops moving, if it is, then determine whether the duration of the stop movement exceeds the duration threshold, if so, it will determine that the preset feature satisfies the input confirmation condition, and the preset location of the preset feature at this time The area is the preset area to be selected by the user. The duration threshold here can be, but is not limited to, 2s, and this embodiment does not specifically limit the duration threshold, and it is determined according to actual conditions. It can be seen that this judgment method is simple and highly reliable.

Determine whether the distance between the preset feature and the preset area where it is located is less than the distance threshold.

The applicant considers that when a user wants to select a preset area, he can also make a selection by inputting a finger to approach or touch the preset area. Therefore, in this embodiment, the processor can also determine whether the distance between the preset feature and the preset area where it is located is less than the distance threshold. If so, it determines that the preset feature satisfies the input confirmation condition. The preset area at the position is the preset area to be selected by the user. The setting of the distance threshold here is also related to the selection of the preset feature. For example, when the preset feature is a nail, even if the input finger touches the preset area, the nail is separated from the preset area by a preset distance. In this embodiment, the distance threshold is not particularly limited, and it is determined according to actual conditions. It can be seen that this judgment method is simple and highly reliable. As a preferred embodiment, determining whether the distance between the preset feature and the preset area of the location is less than the distance threshold includes:

Perform a difference between the second depth information and the first depth information to obtain a difference;

Determine whether the difference is less than the distance threshold.

Specifically, in order to obtain the distance between the preset feature and the preset area where it is located, the first depth information D1 of the preset feature may be acquired through the image acquisition device, and the location of the preset feature may also be acquired through the image acquisition device. For the second depth information D2 of the preset area, the second depth information D2-first depth information D1=△D. If △D is less than the distance threshold, it means that the preset feature is close to the preset area at this time. The preset area where the preset feature is located is the preset area to be selected by the user. It can be seen that this judgment method is simple and accurate.

Please refer to FIG. 7. FIG. 7 is a schematic structural diagram of another wearable device provided by this application. The wearable device includes a display lens 1, and further includes:

The image acquisition device 2 is used to take a picture of the front of the display lens 1 to obtain a front image;

The light machine 3 is used to project preset characters on the display lens 1 based on the control of the processor 4;

The processor 4 is used to implement the steps of the above-mentioned input method of the wearable device when executing the computer program.

For the introduction of the wearable device provided by this application, please refer to the foregoing method embodiment, and this application will not repeat it here.

It should be noted that in this specification, the terms "include", "include" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements not only includes those elements , But also includes other elements that are not explicitly listed, or elements inherent to the process, method, article, or equipment. If there are no more restrictions, the element defined by the sentence "including a..." does not exclude the existence of other same elements in the process, method, article, or equipment that includes the element.

The foregoing description of the disclosed embodiments enables those skilled in the art to implement or use this application. Various modifications to these embodiments will be obvious to those skilled in the art, and the general principles defined herein can be implemented in other embodiments without departing from the spirit or scope of the application. Therefore, this application will not be limited to the embodiments shown in this document, but should conform to the widest scope consistent with the principles and novel features disclosed in this document.

Claims

An input method for a wearable device, characterized in that the wearable device includes a display lens and an image acquisition device for taking a picture of the front of the display lens to obtain a front image, characterized in that the input method includes:

Identifying the preset area of the user's hand from the front image;

Superimposing and displaying preset characters to the preset area through the display lens;

Identifying the preset feature of the input finger from the front image;

It is determined whether the preset feature satisfies the input confirmation condition, and if so, an operation corresponding to the preset character corresponding to the preset area where the preset feature is located is performed.
5. The input method of a wearable device according to claim 1, wherein when the preset character corresponding to the preset area where the preset feature is located is a text character,

Performing the operation corresponding to the preset character corresponding to the preset area where the preset feature is located includes:

The text is constructed according to the text characters.
3. The input method of a wearable device according to claim 2, wherein after forming the text according to the text characters, the method further comprises:

Judging whether the user's hand is performing a preset action based on the front image;

If yes, execute the functional operation corresponding to the preset action.
The input method of the wearable device according to claim 1, wherein when the preset character corresponding to the preset area where the preset feature is located is a functional character,

Performing the operation corresponding to the preset character corresponding to the preset area where the preset feature is located includes:

Perform functional operations corresponding to the functional characters.
The input method of the wearable device according to claim 4, wherein the function characters include confirmation characters and/or deletion characters;

Perform functional operations corresponding to the functional characters, including:

Perform a confirmation operation and/or delete operation corresponding to the function character.
The input method of the wearable device according to claim 1, wherein the preset area includes the knuckles and/or the palm of a finger, and the preset feature is a nail.
8. The input method of a wearable device according to any one of claims 1 to 6, wherein determining whether the preset feature meets an input confirmation condition comprises:

It is determined whether the preset feature stops moving and the duration of the stop movement exceeds a duration threshold.
8. The input method of a wearable device according to any one of claims 1 to 6, wherein determining whether the preset feature meets an input confirmation condition comprises:

It is determined whether the distance between the preset feature and the preset area where it is located is less than a distance threshold.
8. The input method of a wearable device according to claim 8, wherein determining whether the distance between the preset feature and the preset area of the location is less than a distance threshold comprises:

Acquiring the first depth information of the preset feature and the second depth information of the preset area where it is located;

Performing a difference between the second depth information and the first depth information to obtain a difference;

It is determined whether the difference is less than the distance threshold.
A wearable device, the wearable device includes a display lens, and is characterized in that it further includes:

An image acquisition device for taking a picture of the front of the display lens to obtain a front image;

Optical machine for projecting preset characters on the display lens based on the control of the processor;

The processor is configured to implement the steps of the input method of the wearable device according to any one of claims 1 to 9 when the computer program is executed.