CN117289801A

CN117289801A - Gesture interaction method and device, electronic equipment and storage medium

Info

Publication number: CN117289801A
Application number: CN202311280881.2A
Authority: CN
Inventors: 杨天翼; 尹子硕; 陈昊芝
Original assignee: Beijing Positive Negative Infinite Technology Co ltd
Current assignee: Beijing Positive Negative Infinite Technology Co ltd
Priority date: 2023-09-28
Filing date: 2023-09-28
Publication date: 2023-12-26

Abstract

The embodiment of the application provides a gesture interaction method, a gesture interaction device, electronic equipment and a storage medium, and relates to the technical field of virtual reality. The method comprises the following steps: collecting hand images of a user, and determining virtual hand position information according to the hand images; determining first position information of the first endpoint according to the virtual hand position information; the first endpoint is associated with a position of the virtual hand; if the first end point is detected to enter the interaction hot zone of the target area, determining target elements from at least two interaction elements based on the distances between the first end point and the at least two interaction elements in the target area respectively; and responding to the user gesture, and executing the interaction instruction of the user gesture for the target element. According to the embodiment of the application, the positioning accuracy is improved, and meanwhile, the user experience is improved.

Description

Gesture interaction method and device, electronic equipment and storage medium

Technical Field

The application relates to the technical field of virtual reality, in particular to a gesture interaction method, a gesture interaction device, electronic equipment and a storage medium.

Background

VR (Virtual Reality) refers to a Virtual environment created through computer technology simulation. In some virtual scenarios, a user needs to perform text input, and in consideration of inconvenience in carrying a physical keyboard and easiness in environmental influence of recognition accuracy of voice input, a method of performing text input through a virtual keyboard has been developed.

At present, a text input method based on a virtual keyboard aims by sending a ray from a control end to select keys in the virtual keyboard, and when the direction of the ray changes slightly, the other end of the ray also generates larger deviation, so that the accuracy of the keys positioned by the ray is easily affected by the stability of the control end, and the small movement of the control end can lead to larger deviation of the keys selected by the ray, thereby leading to lower accuracy of key positioning.

Disclosure of Invention

The embodiment of the application provides a gesture interaction method, a gesture interaction device, electronic equipment and a storage medium, which can solve the problem of low accuracy of key positioning in the prior art.

The technical scheme is as follows:

according to an aspect of the embodiments of the present application, there is provided a gesture interaction method, including:

collecting hand images of a user, and determining virtual hand position information according to the hand images;

determining first position information of the first endpoint according to the virtual hand position information; the first endpoint is associated with a position of the virtual hand;

if the first endpoint is detected to enter an interaction hot zone of a target area, determining a target element from at least two interaction elements in the target area based on the distance between the first endpoint and the at least two interaction elements respectively;

In response to a user gesture, an interaction instruction of the user gesture for the target element is executed.

Optionally, the determining the first position information of the first endpoint according to the virtual hand position information includes:

determining second position information of a hand root node from the virtual hand position information;

determining first location information of the first endpoint based on second location information of the hand root node; the relative position between the first endpoint and the hand root node remains unchanged.

Optionally, the determining a target element from the at least two interaction elements further includes:

determining a second endpoint based on the location of the target element;

displaying an indicator pointing from the first endpoint to the second endpoint.

Optionally, the determining, based on the distance between the first endpoint and at least two interaction elements in the target area, the target element from the at least two interaction elements includes:

determining a distance between the first endpoint and at least two interactive elements in the target region;

determining a current nearest element nearest to the first endpoint in a current period and a current distance between the current nearest element and the first endpoint based on a distance between the first endpoint and at least two interactive elements in the target area;

If the current distance is detected to be smaller than the previous distance and the difference value between the current distance and the previous distance is not smaller than the preset buffer distance, taking the current nearest element as the target element;

otherwise, taking the last nearest element as the target element;

the last distance is a distance between a last nearest element and the first endpoint, and the last nearest element is an interaction element nearest to the first endpoint in a last period.

Optionally, the detecting that the first endpoint enters the interaction hot zone of the target area includes:

if the first endpoint is detected to collide with the collision box of the target area, determining that the first endpoint enters an interaction hot zone of the target area;

or (b)

Determining an interaction distance between the first endpoint and the target region;

and if the interaction distance is smaller than a preset distance threshold, determining that the first endpoint enters an interaction hot zone of the target area.

Optionally, the target area includes a virtual keyboard;

the responding to the user gesture, executing the interaction instruction of the user gesture aiming at the target element, comprising the following steps:

determining the user gesture based on the virtual hand position information;

If a first trigger instruction aiming at the key is determined based on the gesture of the user, determining the key type of the target element;

if the key type is a character key, displaying the character corresponding to the target element in a display area;

and if the key type is an instruction key, executing the instruction corresponding to the target element.

Optionally, the determining the first trigger instruction for the key based on the gesture of the user includes:

and if the user gesture is detected to be in accordance with a first preset gesture and the duration is smaller than the preset duration, generating a first trigger instruction.

Optionally, the method further comprises:

if a second trigger instruction for the control column is determined based on the user gesture, displaying the control column of the target element at the relevant position of the target element;

the control field of the target element includes at least one interactive control for operating the target element.

Optionally, the determining the second trigger instruction for the control bar based on the user gesture includes any one of the following:

detecting that the user gesture accords with a second preset gesture;

or if the gesture of the user accords with the first preset gesture and the duration is longer than the preset duration, generating a second trigger instruction.

Optionally, the target area includes a text box;

determining the user gesture based on the virtual hand position information;

if the fact that the second endpoint of the indicator moves in the text box within the time period of the user gesture maintaining the first preset gesture is detected, determining a starting position point and an ending position point corresponding to the second endpoint;

and determining a selected area based on a starting position point and an ending position point corresponding to the second endpoint, and taking at least one element included in the selected area as the target element.

Optionally, the selecting the at least one element included in the selected area as the target element further includes:

if the user gesture is detected to be a release gesture, displaying a control column of the target element at the relevant position of the target element;

Optionally, the method further comprises:

if an adjustment instruction for the variable area is determined based on the user gesture, adjusting the variable area;

Wherein the variable region comprises the virtual keyboard or any sub-region of the virtual keyboard.

Optionally, the determining, based on the user gesture, an adjustment instruction for the variable region includes:

displaying a corresponding position adjustment control on the variable region;

generating an adjustment instruction for the variable region based on the operation for the position adjustment control;

and responding to the adjustment instruction, and adjusting the position of the variable area.

Optionally, the adjustment instruction includes a cut instruction for the virtual keyboard:

the adjusting the position of the variable region in response to the adjustment instruction includes:

and splitting the virtual keyboard into at least two sub-areas based on a preset splitting boundary.

Optionally, the adjustment instruction includes a merge instruction for at least two sub-regions;

after the position of the variable area is adjusted in response to the adjustment instruction, the method further comprises:

and displaying the virtual keyboard.

According to another aspect of an embodiment of the present application, there is provided a gesture interaction device, including:

the virtual hand position determining module is used for acquiring hand images of a user and determining virtual hand position information according to the hand images;

The first endpoint determining module is used for determining first position information of the first endpoint according to the virtual hand position information; the first endpoint is associated with a position of the virtual hand;

the target element determining module is used for determining target elements from at least two interactive elements in the target area based on the distances between the first end point and the at least two interactive elements in the target area respectively if the first end point is detected to enter the interactive hot zone of the target area;

and the interaction module is used for responding to the user gesture and executing an interaction instruction of the user gesture aiming at the target element.

According to another aspect of the embodiments of the present application, there is provided an electronic device including a memory, a processor, and a computer program stored on the memory and executable on the processor, the processor implementing the steps of any of the gesture interaction methods described above when executing the program.

According to a further aspect of embodiments of the present application, there is provided a computer readable storage medium having stored thereon a computer program which, when executed by a processor, implements the steps of any of the gesture interaction methods described above.

The beneficial effects that technical scheme that this application embodiment provided brought are:

the first endpoint used for representing the position of the virtual hand of the user is determined, the target element is determined by calculating the distance between the first endpoint and at least two interaction elements, the positioning of the target element is not affected by the stability of the control end, the positioning accuracy is higher, the user does not need to consume time and attention to keep the stability of the control end, and the user experience is improved.

By responding to the user gesture, executing the interaction instruction of the user gesture for the target element, decoupling the selection of the target element from the interaction operation of the user on the target element, and avoiding the problem of false touch.

Further, the flexibility of the size design of the elements in the target area is increased, the size of the target area and the size of the elements in the target area can be designed to be smaller, the occupation of the display space is reduced, meanwhile, the movement range of the hands of the user can be reduced, and the experience of the user is further improved.

Drawings

In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings that are required to be used in the description of the embodiments of the present application will be briefly described below.

Fig. 1 is a schematic flow chart of a gesture interaction method provided in an embodiment of the present application;

fig. 2 is a schematic diagram of an input method virtual interface provided in an embodiment of the present application;

FIG. 3 is a schematic diagram of a distance calculation according to an embodiment of the present disclosure;

fig. 4 is a schematic diagram of a virtual hand key node according to an embodiment of the present application;

FIG. 5 is a schematic view of an indicator according to an embodiment of the present application;

FIG. 6 is a schematic diagram of a target element selection process according to an embodiment of the present disclosure;

FIG. 7 is a schematic diagram of coordinate distance calculation according to an embodiment of the present disclosure;

FIG. 8 is a schematic diagram of a text box interaction process according to an embodiment of the present application;

FIG. 9 is a schematic diagram of another input method virtual interface according to an embodiment of the present disclosure;

FIG. 10 is a schematic diagram of another input method virtual interface according to an embodiment of the present disclosure;

FIG. 11 is a schematic diagram of another input method virtual interface according to an embodiment of the present disclosure;

fig. 12 is a schematic structural diagram of a gesture interaction device according to an embodiment of the present application;

fig. 13 is a schematic structural diagram of an electronic device according to an embodiment of the present application.

Detailed Description

Embodiments of the present application are described below with reference to the drawings in the present application. It should be understood that the embodiments described below with reference to the drawings are exemplary descriptions for explaining the technical solutions of the embodiments of the present application, and the technical solutions of the embodiments of the present application are not limited.

As used herein, the singular forms "a", "an" and "the" are intended to include the plural forms as well, unless expressly stated otherwise, as understood by those skilled in the art. It will be further understood that the terms "comprises" and "comprising," when used in this application, specify the presence of stated features, information, data, steps, operations, elements, and/or components, but do not preclude the presence or addition of other features, information, data, steps, operations, elements, components, and/or groups thereof, all of which may be included in the present application. It will be understood that when an element is referred to as being "connected" or "coupled" to another element, it can be directly connected or coupled to the other element or intervening elements may be present. Further, "connected" or "coupled" as used herein may include wirelessly connected or wirelessly coupled. The term "and/or" as used herein indicates that at least one of the items defined by the term, e.g., "a and/or B" may be implemented as "a", or as "B", or as "a and B".

For the purpose of making the objects, technical solutions and advantages of the present application more apparent, the embodiments of the present application will be described in further detail below with reference to the accompanying drawings.

VR refers to a virtual environment created through computer technology simulation. The method utilizes the devices such as HMD (Head Mounted Display) devices, head-mounted display devices, sensors and the like to enable a user to feel a virtual world in an immersive manner, and interact with a virtual scene through interaction devices.

In some virtual scenarios, a user needs to perform text input, and in consideration of inconvenience in carrying a physical keyboard and easiness in environmental influence of recognition accuracy of voice input, a method of performing text input through a virtual keyboard has been developed.

Currently, a text input method based on a virtual keyboard aims by sending a ray from a control end to select keys in the virtual keyboard, wherein the ray can be a shape (such as a touch pen) presented by the control end held by a hand of a user or can be a ray emitted by the control end through ray tracing on the hand.

When the direction of the ray generates tiny change, the other end of the ray also generates larger deviation, so that the accuracy of the ray positioning keys is easily affected by the stability of the control end, and the tiny movement of the control end can lead to larger deviation of the ray selected keys, thereby leading to lower accuracy of the key positioning.

In addition, the size of the virtual keyboard and the size of the keys need to be designed to be large in consideration of the fact that the movement amplitude of the other end of the ray is large, so that more display space is occupied; the space between each key is larger, so that the user consumes more physical power in moving between different keys, and the inconvenience of the user in operation is increased; meanwhile, when the keys are selected in a positioning mode, the control end is required to be very stable in order to avoid offset, so that a user needs to consume more time and attention, the difficulty of user operation is improved, and the user experience is reduced.

Another text input method based on a virtual keyboard is to use index finger to stamp each key to input through simulating the form and interaction mode of a physical button, wherein the virtual key can only provide visual or auditory feedback and lacks tactile feedback; and because the hand shields the vision of the key, the user can not accurately sense the distance relation between the key and the finger, so that the accuracy of the operation of clicking the button is low. In addition, at least the upper arm and forearm muscles are required to be adjusted in the action of lifting the hand to be poked in the air, so that fatigue is easily caused by long-time operation, and the user experience is reduced.

The gesture interaction method, device, electronic equipment and storage medium aim to solve the technical problems in the prior art.

The technical solutions of the embodiments of the present application and technical effects produced by the technical solutions of the present application are described below by describing several exemplary embodiments. It should be noted that the following embodiments may be referred to, or combined with each other, and the description will not be repeated for the same terms, similar features, similar implementation steps, and the like in different embodiments.

Fig. 1 is a flow chart of a gesture interaction method provided in an embodiment of the present application, as shown in fig. 1, where the method includes:

step S110, collecting hand images of a user, and determining virtual hand position information according to the hand images.

Specifically, the hand image of the user may be an image including the hand of the user, the hand image of the user may be acquired by the camera, for example, the hand motion of the user may be captured by the camera, and one frame of image in the video data captured by the camera is taken as the hand image. The camera may be disposed in the HMD device, or may be disposed separately, which is not limited in this embodiment of the present application.

After the hand image is acquired, gesture recognition can be performed on the hand image, so that position information of a plurality of key nodes (such as finger key nodes, palm key nodes and the like) on the hand can be obtained and used as virtual hand position information. The virtual hand position information may include three-dimensional coordinates of a plurality of key nodes on the hand.

Alternatively, the user may issue a preset command to enter the text input mode, and when a text input mode trigger is detected, the input method virtual interface may be displayed at a position at a preset operation distance from the user, for example, at a position 0.5 m from the center of the user's chest, based on the spatial coordinates of the user's body and head.

Step S120, determining first position information of a first endpoint according to the virtual hand position information; the first endpoint is associated with a position of the virtual hand.

Specifically, after the virtual hand position information is determined, the first position information of the first endpoint may be determined according to the virtual hand position information. The first position information of the first endpoint may include coordinates of the first endpoint and a direction of the first endpoint, where the first endpoint is associated with a position of the virtual hand, i.e. the position of the first endpoint may change with a change of the position of the virtual hand, where the first endpoint may be understood as an abstract representation of the virtual hand in the virtual scene.

Optionally, the first endpoint may be used to characterize a position at which the directional gesture of the virtual hand is directed. When a user uses a directional gesture, the location at which the gesture is directed may be understood as an object that the user wishes to select or manipulate.

In step S130, if the first endpoint is detected to enter the interaction zone of the target area, the target element is determined from the at least two interaction elements based on the distances between the first endpoint and the at least two interaction elements in the target area, respectively.

Specifically, the target area may be an area in the virtual scene, which needs to be interacted by the user, the target area may include at least one element, the element may be an interactive element and a non-interactive element, and the interactive element may be an element capable of interacting with the user, for example, a key; the non-interactive element may be an element that does not interact with the user, such as a decorative element.

The target area may include a virtual keyboard, or may include other interactable areas in the input method virtual interface. Fig. 2 is a schematic diagram of an input method virtual interface provided in an embodiment of the present application, where, as shown in fig. 2, the virtual interface may include a text box and a virtual keyboard, and the target area may include the text box or the virtual keyboard.

Optionally, the input method virtual interface may further include a candidate list, and the recommended character strings may be obtained by performing association, automatic completion or intelligent recommendation according to the selected characters, and the recommended character strings are sequentially arranged in the candidate list according to the recommendation sequence.

The interaction zone of the target area is used for representing an effective area capable of interacting with the target area, the interaction zone can be a space range under a three-dimensional coordinate system, and when the interaction behavior of the user occurs in the interaction zone of the target area, the target area can respond to the interaction behavior of the user; when the user's interaction behavior occurs outside of the interaction zone of the target area, the target area will not respond to the user's interaction behavior.

The first endpoint may represent a position of a virtual hand of the user, when the first endpoint is detected to enter an interaction zone of the target area, the gesture of the user is indicated to be capable of interacting with the target area, a target element may be determined from at least two interaction elements based on a distance between the first endpoint and the at least two interaction elements, and the target element may be an interaction element closest to the first endpoint in the at least two interaction elements.

Fig. 3 is a schematic diagram of distance calculation provided in the embodiment of the present application, as shown in fig. 3, the interaction element may include a key "F" and a key "G", and a distance between a center of the key "F" or the key "G" and the first endpoint may be used as a distance between the two elements and the first endpoint, respectively.

Alternatively, the distance between the first endpoint and each interactive element in the target region may be calculated, and the interactive element closest to the first endpoint is taken as the target element.

Alternatively, at least two interactive elements in the adjacent region may be selected from the target region based on the adjacent region of the first end point (for example, a spherical region centered on the first end point and having a preset length as a radius), and then a distance between the first end point and the selected at least two interactive elements may be calculated, and the interactive element closest to the first end point may be taken as the target element.

In the embodiment of the application, the first endpoint used for representing the position of the virtual hand of the user is determined, and the target element is determined by calculating the distance between the first endpoint and at least two interaction elements, wherein the target element is the closest interaction element to the first endpoint, namely, the possibility that the target element is the element with which the user wants to interact is maximum. Compared with the method for selecting through ray positioning in the prior art, in the embodiment of the application, the selection of the target element is automatically determined based on the distance between the target element and the first end point, but not through ray aiming of the control end, the positioning of the target element is not affected by the stability of the control end, the positioning accuracy is higher, the user does not need to consume time and attention to keep the stability of the control end, and the user experience is improved.

Moreover, the positioning of the target element is realized through the calculation of the distance between the points, unlike the intersection point of one ray and one interface in the prior art, the problem that the movement amplitude of the other end of the ray is larger is not required to be considered, the size design of the element in the target area is more flexible, the size of the target area and the element in the target area can be designed to be smaller, the occupation of the display space is reduced, the movement amplitude of the hand of the user is reduced, and the experience of the user is further improved.

In step S140, in response to the user gesture, an interaction instruction of the user gesture with respect to the target element is performed.

Specifically, after the target element is determined, the user may be made aware of the currently selected element by highlighting the target element, for example, by adjusting color or transparency, or the like. Then, the user gesture can be identified, the interaction instruction corresponding to the user gesture is determined, and the interaction instruction is executed on the target element.

In combination with the example given in fig. 2, when the target area is a virtual keyboard and the target element is a character key, if a trigger gesture corresponding to the click command is identified, the click command to the character key is triggered and executed, and the character corresponding to the character key is input into the text box.

The relationship between the trigger gesture and the interaction instruction may be preset, for example, a pinch gesture of a thumb and other fingers is used as a trigger gesture of the click instruction. In addition, for different target areas, the same triggering gesture can correspond to different interaction instructions, for example, when the target area is a virtual keyboard, a pinch gesture of a thumb and other fingers is used for triggering a certain key in the keyboard; when the target area is a text box, the pinch gesture of the thumb and other fingers is used for triggering a certain text element in the selected text box.

Alternatively, a pinch gesture of a thumb and other fingers may be used as the trigger gesture, where the trigger gesture may be a pinch of the thumb and any other finger, for example, a pinch of the thumb and index finger, or a pinch of the thumb and index finger; the triggering gesture may also be a pinch of the thumb with a plurality of other fingers, for example, a pinch of the thumb with the index and middle fingers together.

When the pinch gesture is used as a trigger gesture, the pinch action can give the user a tactile sensation, so that the user can obtain tactile feedback when interacting with the virtual element, i.e. the user can feel that the virtual element is operated through the tactile sensation.

Compared with the interactive mode of simulating entity buttons in the prior art, the mode of using the index finger to stamp each key for inputting is adopted, and the key selection and the click operation are completed simultaneously. Because the problem of false touch does not exist, the size design of the elements in the target area is not required to be limited, the size of the target area and the size of the elements in the target area can be designed to be smaller, the occupation of the display space is reduced, the movement amplitude of the hands of a user can be reduced, and the experience of the user is further improved.

In the embodiment of the application, the first endpoint used for representing the position of the virtual hand of the user is determined, the target element is determined by calculating the distance between the first endpoint and at least two interactive elements, the positioning of the target element is not influenced by the stability of the control end, the positioning accuracy is higher, the user does not need to consume time and attention to keep the stability of the control end, and the user experience is improved.

As an alternative embodiment, determining the first position information of the first endpoint according to the virtual hand position information includes:

determining first location information of the first endpoint based on the second location information of the hand root node; the relative position between the first endpoint and the root node of the hand remains unchanged.

Specifically, the virtual hand position information may include position information of each key node on the hand, from which second position information of a virtual hand root node may be determined, and fig. 4 is a schematic diagram of a virtual hand key node provided in the embodiment of the present application, as shown in fig. 4, a plurality of virtual hand key nodes (black dots in fig. 4) represent tree-shaped distribution using one key node as a root node, where the root node is a hand root node. The second position information of the hand root node may include coordinates of the hand root node and a direction of the hand root node.

After determining the second position information of the hand root node, the first position information of the first endpoint may be determined based on the second position information of the hand root node, where a relative position between the first endpoint and the hand root node may remain unchanged, such that the first endpoint may change with a change in the hand root node.

It should be noted that, in fig. 4, the relative positional relationship between the first endpoint (white dots in fig. 4) and the hand root node may be represented as a line (dashed lines in fig. 4) between the first endpoint and the hand root node, and this line may be regarded as a vector in the three-dimensional coordinate system, and the magnitude and direction of this vector may represent the relative position between the first endpoint and the hand root node. Also, when the user uses a pinch finger of the thumb and other fingers as a directional gesture, the direction of this vector may represent the direction in which the directional gesture is directed.

Optionally, the hand root node may be located in an area of the palm, for example, the center of the palm is taken as the hand root node, when the user makes different gestures, the positions of the key nodes on the fingers change, but the positions of the palm portions are basically unchanged, by setting the hand root node in the area of the palm, the positions of the hand root node cannot change along with the changes of the different gestures, that is, the positions of the hand root node may more accurately represent the positions of the virtual hands, and the first endpoint determined based on the hand root node may change along with the changes of the positions of the virtual hands.

Optionally, determining the first location information of the first endpoint based on the second location information of the hand root node includes:

the pose of the hand root node is used as second position information, the pose of the first endpoint is used as first position information, the pose of the hand root node can be converted into a corresponding transformation matrix, the transformation matrix of the first endpoint is determined based on the transformation matrix of the hand root node, and the transformation matrix of the first endpoint is converted into the pose of the first endpoint.

The transformation matrix for the first endpoint may be determined based on the following formula:

M′_C＝M′_P*M_C

wherein M '_C is the transformation matrix of the first endpoint, M' _P is the transformation matrix of the hand root node, and M_C is the initial transformation matrix of the first endpoint.

As an alternative embodiment, determining the target element from the at least two interactive elements further comprises:

determining a second endpoint based on the location of the target element;

displaying an indicator pointing from a first endpoint to a second endpoint.

Specifically, after the target element is determined, the second endpoint may be determined based on the position of the target element, for example, taking the center of the target element as the second endpoint. After determining the first endpoint and the second endpoint, an indicator pointing from the first endpoint to the second endpoint may be determined based on positions corresponding to the first endpoint and the second endpoint, and the indicator may be displayed based on a grid surface model with directivity, where the shape of the grid surface model may include a straight line, a triangle, a cone, a pyramid, an arrow, and the like.

The pointer may be understood as a visual representation of the pointing object element from the virtual hand, with the pointer updated when the first endpoint and the second endpoint are updated. Through displaying the indicator pointing to the second endpoint from the first endpoint, the user can perceive the relative position between the hand of the user and the virtual element through visual information such as the length, the inclination angle and the endpoint position of the indicator, so that the user is helped to select the virtual element more accurately, the accuracy of positioning the virtual element is improved, more directional information is provided for the user, and convenience and experience of interactive operation of the user are improved.

Fig. 5 is a schematic diagram of an indicator provided in an embodiment of the present application, where, as shown in fig. 5, the shape of the indicator in fig. 5 is triangle, and in fig. 5, the first end point is the tail of the indicator, and the second end point is the head of the indicator, and the user can more clearly perceive the selected direction through the shape with the directivity function.

As an alternative embodiment, determining the target element from the at least two interaction elements based on the distances between the first endpoint and the at least two interaction elements in the target region, respectively, includes:

determining a current nearest element nearest to the first endpoint in the current period and a current distance between the current nearest element and the first endpoint based on the distance between the first endpoint and at least two interactive elements in the target area;

if the current distance is detected to be smaller than the previous distance and the difference value between the current distance and the previous distance is not smaller than the preset buffer distance, taking the current nearest element as a target element;

otherwise, taking the last nearest element as a target element;

the last distance is the distance between the last nearest element and the first endpoint, and the last nearest element is the interaction element nearest to the first endpoint in the last period.

Specifically, each preset period is spaced, the distance between the first endpoint and at least two interactive elements in the target area is calculated, in the last period, the interactive element closest to the first endpoint can be used as the last nearest element, and the distance between the last nearest element and the first endpoint can be used as the last distance.

In the current period, the interactive element closest to the first endpoint may be taken as the current closest element, and the distance between the current closest element and the first endpoint may be taken as the current distance.

Since the last nearest element is the nearest interactive element of the last period from the first endpoint, that is, the last nearest element is the target element, after determining the current nearest element and the current distance, the current distance and the last distance can be compared, if the current distance is smaller than the last distance, the current nearest element is closer to the first endpoint, but the target element is not updated to the current nearest element at this time, and on the premise that the current distance is smaller than the last distance, the difference between the current distance and the last distance and the size of the preset buffer distance also need to be judged, if the difference between the current distance and the last distance is not smaller than the preset buffer distance, the obvious movement of the hand of the user is indicated, that is, the user subjectively moves to switch the selected object at this time, frequent switching of the target element caused by the shake of the non-subjective intention of the hand of the user is avoided, misoperation caused by the shake of the hand of the user is reduced, and the positioning accuracy of the virtual element is improved.

Fig. 6 is a schematic diagram of a target element selection process provided in the embodiment of the present application, as shown in fig. 6, where the preset buffer distance is 0.02, and in the first period, the element distance between the first endpoint and the key F is 0.10, and the element distance between the first endpoint and the key G is 0.12, the key F is taken as the target element, i.e. the user wishes to select "F"; in the second period, the element distance between the first end point and the key F is 0.11, and the element distance between the first end point and the key G is 0.10, wherein the first end point is closer to the key G at this time, but the difference value of the two element distances is 0.01 and is smaller than the preset buffer distance 0.02, so that the fact that the hand of the user possibly shakes with non-subjective intention is indicated, and the key F is still used as a target element at this time; in the third period, the element distance between the first endpoint and the key F is 0.12, the element distance between the first endpoint and the key G is 0.10, the difference value of the two element distances is not smaller than the preset buffer distance 0.02, and the fact that the user subjectively moves the hand to select the key G is indicated, and then the target element is updated to be the key G.

As an alternative embodiment, detecting that the first endpoint enters the interaction zone of the target region includes:

if the collision of the first end point and the collision box of the target area is detected, determining that the first end point enters an interaction hot zone of the target area;

or (b)

and if the interaction distance is smaller than the preset distance threshold, determining that the first terminal enters the interaction hot zone of the target area.

Specifically, to detect whether the first endpoint enters the interaction zone of the target area, any of the following manners may be used:

the collision box corresponding to the target area can be set, and the collision box is used for judging whether the input of the user collides with the target area or not, and the shape of the collision box can be cuboid, cube and the like. When the position of the first endpoint is located at the edge or inside of the collision cell of the target region, it is determined that the first endpoint collides with the collision cell of the target region, i.e., the first endpoint enters the interaction zone of the target region.

The interaction distance between the first endpoint and the target region may also be calculated, for example, the interaction distance between the first endpoint and the center of the target region is calculated, and when the interaction distance is smaller than a preset distance threshold value, it is determined that the first endpoint enters the interaction hot zone of the target region.

As an alternative embodiment, the target area comprises a virtual keyboard;

in response to the user gesture, executing an interaction instruction of the user gesture for the target element, including:

determining a user gesture based on the virtual hand position information;

Specifically, the target area may be a virtual keyboard, and the target element may be a key on the virtual keyboard.

The process of the user interacting with the virtual keyboard through gestures may include: after the virtual hand position information of the user is obtained, gesture recognition can be performed on three-dimensional coordinates of a plurality of key nodes in the virtual hand position information to obtain corresponding user gestures, and corresponding interaction instructions are determined based on the user gestures. If the interaction instruction corresponding to the gesture of the user is determined to be the first trigger instruction, the first trigger instruction may be executed, and the first trigger instruction may be a click instruction for the key.

When a click instruction aiming at the target element, namely a first trigger instruction is executed, the key type of the target element can be determined; if the key type of the target key is a character key, displaying the character corresponding to the target element in a display area, wherein the display area can comprise a candidate list or a text box; if the key type of the target key is an instruction key, such as an enter key or a delete key, an instruction corresponding to the instruction key may be executed.

As an optional embodiment, determining the first trigger instruction for the key based on the gesture of the user includes:

and if the detected gesture of the user accords with the first preset gesture and the duration is smaller than the preset duration, generating a first trigger instruction.

Specifically, when it is detected that the gesture of the user accords with the first preset gesture and the duration of the first preset gesture kept by the user is smaller than the preset duration, a first trigger instruction may be generated.

The first preset gesture may be a gesture of pinching the thumb with other fingers, and may be pinching the thumb with any one of the other fingers, for example, pinching the thumb with an index finger, or pinching the thumb with a middle finger; the thumb may be pinched with a plurality of other fingers, for example, the thumb is pinched with the index finger and the middle finger, which is not limited in the setting of the first preset gesture in the embodiment of the present application.

Alternatively, the type of user gesture may be determined based on the coordinate distance between the thumb tip and the other finger tips. Fig. 7 is a schematic diagram of coordinate distance calculation provided in the embodiments of the present application, and as shown in fig. 7, it may be determined whether a user makes a gesture of pinching the thumb and the index finger by calculating the distance between the thumb tip and the index finger tip.

The following description will be made with reference to an example assuming that a thumb and an index finger pinch as a first gesture, a thumb and a middle finger pinch as a second gesture, and a thumb and an index finger and a middle finger pinch together as a third gesture. Thus, a distance between the coordinates of the thumb tip and the coordinates of the index finger tip may be calculated, and when the distance is less than a preset threshold, determined as the first gesture; calculating the distance between the coordinates of the thumb tip and the coordinates of the middle finger tip, and determining the distance as a second gesture when the distance is smaller than a preset threshold value; and calculating the distance between the coordinates of the thumb tip and the coordinates of the index finger tip and the distance between the coordinates of the thumb tip and the coordinates of the middle finger tip, and determining the third gesture when the two distances are smaller than a preset threshold value.

In addition, different gestures may be set to correspond to different trigger instructions corresponding to the same target element, e.g., a first gesture corresponds to a lower case instruction, a second gesture corresponds to an upper case instruction, and a third gesture corresponds to a click instruction. Different instructions are executed by setting different gestures for the same target element, and a user can realize different interactive operations by changing the gestures, so that the interactive operations of the user are enriched, and the user experience is improved.

As an alternative embodiment, the method further comprises:

if a second trigger instruction aiming at the control column is determined based on the user gesture, displaying the control column of the target element at the relevant position of the target element;

Specifically, based on the user gesture, a corresponding interaction instruction is determined, and when the interaction instruction corresponding to the user gesture is a second trigger instruction for the control column, the second trigger instruction may be executed, and the second trigger instruction may be a display instruction for the control column.

When the display instruction for the control bar, namely the second trigger instruction is executed, the relevant position of the target element can be determined, wherein the relevant position can be the position adjacent to the target element, for example, the vertex on the outsourcing rectangle of the key corresponding to the target element; and display the control column of the target element in the relevant position, the control column is used for operating the target element, the control column can include at least one interactive control used for operating the target element, the interactive control can include the control of switching the capital letter, control of switching the lowercase letter, etc., the concrete function of the interactive control can be set up according to the actual application scene, this embodiment of the application does not limit.

Through the control bar of the target element, the interaction control in the control bar is utilized to operate the target element, so that the interaction operation between the user and the virtual element is enriched, the convenience of the user operation is improved, and the user experience is further improved.

As an optional embodiment, determining the second trigger instruction for the control bar based on the user gesture includes any one of the following:

detecting that the gesture of the user accords with a second preset gesture;

or if the gesture of the user is detected to be in accordance with the first preset gesture and the duration is longer than the preset duration, generating a second trigger instruction.

Specifically, when it is detected that the user gesture corresponds to the second preset gesture or it is detected that the user gesture corresponds to the first preset gesture and the duration of the first preset gesture kept by the user is longer than the preset duration, the second trigger instruction may be generated.

The second preset gesture may be a gesture different from the first preset gesture, for example, when the first preset gesture is a thumb and index finger pinching, the second preset gesture may be a thumb and middle finger pinching gesture, that is, when the user makes a thumb and middle finger pinching gesture, or when the user continuously pinches the thumb and index finger, the second trigger instruction may be generated, so that flexibility and richness of interaction between the user and the virtual element through the gesture are increased.

As an alternative embodiment, the target area comprises a text box;

determining a user gesture based on the virtual hand position information;

if the fact that the second endpoint of the indicator moves in the text box within the time period of keeping the first preset gesture by the user gesture is detected, determining a starting position point and an ending position point corresponding to the second endpoint;

and determining a selected area based on the starting position point and the ending position point corresponding to the second endpoint, and taking at least one element included in the selected area as a target element.

Specifically, the target area may include a text box, and an element in the text box may be a character in the text box, for example, when the text is a kanji, the element may be a kanji; when the text is english, the element may be a letter.

The process of the user interacting with the text box through gestures may include: after the virtual hand position information of the user is obtained, gesture recognition can be performed on three-dimensional coordinates of a plurality of key nodes in the virtual hand position information, and corresponding user gestures are obtained. When the gesture of the user is detected to be in accordance with the first preset gesture, the target element (namely one character) is selected by the user; on the basis, the user can keep the gesture of the first preset gesture, and meanwhile move the hand, so that the second endpoint of the indicator moves in the text box, and a starting position point and an ending position point of the movement of the second endpoint are obtained.

And taking the area between the starting position point and the ending position point as a selected area, and updating the target element as at least one element in the selected area, so that a user can simultaneously select a plurality of characters, and the interactive operation between the user and the text box is enriched.

As an alternative embodiment, the selecting area includes at least one element as the target element, and then further includes:

Specifically, after the user selects at least one character, the selected character may be highlighted by changing a font color, a background color, or the like.

After detecting that the user gesture is a release gesture, a character editing mode may be entered, where the release gesture may be for a first preset gesture, for example, when the first preset gesture is a gesture in which a thumb and an index finger pinch, that is, a coordinate distance between the thumb fingertip and the index finger fingertip is smaller than a preset threshold, the release gesture may be a gesture in which a thumb and an index finger pinch are released, that is, a coordinate distance between the thumb fingertip and the index finger fingertip is larger than a preset threshold.

When entering the character editing mode, the relevant position of the target element can be determined, wherein the relevant position can be the position adjacent to the target element, such as the vertex on the outsourcing rectangle of the corresponding area of the target element; and display the control column of the target element in the relevant position, the control column is used for operating the target element, the control column can include at least one interactive control used for operating the target element, the interactive control can include copying the control of the character, searching for the control of the character, deleting the control of the character, etc., the concrete function of the interactive control can be set up according to the actual application scene, this is not limited to the embodiment of the application.

FIG. 8 is a schematic diagram of a text box interaction process provided in an embodiment of the present application, where, as shown in FIG. 8, a user makes a first preset gesture (e.g. pinch thumb and index finger), selects a character "parent" (i.e. a target element) and highlights, and keeps the gesture moving a distance to the left, and the target element is updated to "we are parent" and highlights; when the user makes a release gesture (for example, releases the thumb and the index finger), a control bar is displayed near the target element, wherein the control bar comprises three interactive controls for copying, searching and deleting, and the operations of copying, searching and deleting the target element can be respectively performed through the interactive controls.

As an alternative embodiment, the method further comprises:

wherein the variable region comprises a virtual keyboard or any sub-region of a virtual keyboard.

Specifically, after the user gesture is identified, an interaction instruction corresponding to the user gesture may be determined, and if an adjustment instruction for the variable area is determined, the variable area may be adjusted according to the adjustment instruction. The variable area may include the whole virtual keyboard before the segmentation, or may include any sub-area obtained after the segmentation. When a text box and a candidate list are included in the virtual interface, the variable region may also include a text box or a candidate list.

As an alternative embodiment, determining an adjustment instruction for the variable region based on the user gesture includes:

displaying a corresponding position adjustment control on the variable region;

the position of the variable region is adjusted in response to the adjustment instruction.

Specifically, for each variable region, a corresponding position adjustment control may be set for the variable region, where the position adjustment control is used to adjust the position of the variable region.

After the adjustment instruction is triggered, a position adjustment control corresponding to the variable region can be displayed, and a user generates an adjustment instruction for the variable region by operating the position adjustment control so as to adjust the position of the variable region based on the adjustment instruction.

Fig. 9 is a schematic diagram of an input method virtual interface provided in an embodiment of the present application, where, as shown in fig. 9, the virtual interface may include a text box, a candidate list, and a virtual keyboard, where, a position adjustment control is set on both sides and below the virtual interface, and a user may adjust positions of each area by operating the movement control, for example, the user may move or rotate the virtual keyboard by moving or rotating the position adjustment control below. For another example, the user may adjust the relative positions between the text box, the candidate list, and the virtual keyboard by moving position adjustment controls on both sides.

As an alternative embodiment, the adjustment instruction includes a cut instruction for the virtual keyboard:

adjusting the position of the variable region in response to the adjustment instruction

Specifically, after the user gesture is identified, an interaction instruction corresponding to the user gesture may be determined, and when determining a segmentation instruction for the virtual keyboard, the virtual keyboard may be segmented based on a preset segmentation boundary, so as to obtain at least two sub-regions. For example, an axis of the virtual keyboard in the horizontal direction may be used as a slicing boundary, thereby slicing the virtual keyboard into two sub-areas of the same size.

Optionally, for the movement control located below, the user may generate a splitting instruction for the virtual keyboard by locating two fingers in the position adjustment control, and moving the two fingers towards the outside along the opposite directions on the level, that is, the distance between the two fingers is further and further, until the distance is greater than the preset splitting distance, and based on the splitting instruction, obtain two sub-areas according to the preset splitting boundary.

Fig. 10 is a schematic diagram of an input method virtual interface provided in the embodiment of the present application, as shown in fig. 10, in which a virtual keyboard is split into two sub-areas, and for each sub-area, a corresponding position adjustment control may be set, and by moving or rotating the position adjustment control, a corresponding movement instruction or rotation instruction may be generated, so that a variable area is moved or rotated, so that the position of each sub-area may adapt to comfortable operation ranges of different hands, and user experience is further improved.

Fig. 11 is a schematic diagram of an input method virtual interface provided in the embodiment of the present application, as shown in fig. 11, when a virtual hand and a keyboard area overlap in a field of view, the relative position and content details of the virtual hand model or the keyboard area can be more clearly seen and perceived by a user by changing the transparency of the virtual hand model or the keyboard area.

As an alternative embodiment, the adjustment instruction comprises a merge instruction for at least two sub-regions;

after adjusting the position of the variable region in response to the adjustment instruction, the method further includes:

and displaying the virtual keyboard.

Specifically, for the virtual keyboard after segmentation, the user can also generate a merging instruction for each sub-region by operating a position adjustment control corresponding to at least one sub-region, and after merging at least two sub-regions based on the merging instruction, a preset segmentation boundary can be eliminated, and the virtual keyboard is displayed, so that an original virtual keyboard which is not segmented is restored.

Optionally, the user may generate a merging instruction by moving the position adjustment control of any sub-region, and merge the two sub-regions when detecting that the distance between the two sub-regions is smaller than the preset merging distance.

Optionally, the user may generate the merging instruction by respectively positioning two fingers on the position adjustment controls of the two sub-areas, and merging the two sub-areas by moving the two fingers toward the inner side in the horizontally opposite directions, that is, the distance between the two fingers is closer and closer until the distance is smaller than the preset merging distance.

Fig. 12 is a schematic structural diagram of a gesture interaction device provided in an embodiment of the present application, where, as shown in fig. 12, the device includes:

the virtual hand position determining module 210 is configured to collect a hand image of a user, and determine virtual hand position information according to the hand image;

a first endpoint determining module 220, configured to determine first location information of a first endpoint according to the virtual hand location information; the first endpoint is associated with a position of the virtual hand;

a target element determining module 230, configured to determine, if the first endpoint is detected to enter an interaction zone of a target area, a target element from at least two interaction elements in the target area based on distances between the first endpoint and the at least two interaction elements, respectively;

and the interaction module 240 is configured to respond to a user gesture and execute an interaction instruction of the user gesture for the target element.

As an alternative embodiment, the first endpoint determination module is specifically configured to:

As an alternative embodiment, the apparatus further comprises an indicator module for:

determining a second endpoint based on the location of the target element;

As an alternative embodiment, the target element determining module is specifically configured to:

Otherwise, taking the last nearest element as the target element;

or (b)

As an alternative embodiment, the target area includes a virtual keyboard;

the interaction module comprises:

the gesture recognition module is used for determining the gesture of the user based on the virtual hand position information;

the first trigger instruction module is used for determining the key type of the target element if a first trigger instruction for the key is determined based on the gesture of the user;

As an alternative embodiment, the first trigger instruction module is specifically configured to:

As an alternative embodiment, the apparatus further comprises a second trigger instruction module, configured to:

As an alternative embodiment, the second trigger instruction module is specifically configured to:

detecting that the user gesture accords with a second preset gesture;

As an alternative embodiment, the target area includes a text box;

the interaction module is specifically used for:

determining the user gesture based on the virtual hand position information;

As an alternative embodiment, the apparatus further comprises a character editing module for:

As an alternative embodiment, the apparatus further comprises an adjustment module for:

As an alternative embodiment, the adjustment module comprises:

the position adjustment control display module is used for displaying the corresponding position adjustment control on the variable area;

An adjustment instruction generation module for generating an adjustment instruction for the variable region based on an operation for the position adjustment control;

and the adjustment sub-module is used for responding to the adjustment instruction and adjusting the position of the variable area.

the adjusting sub-module is specifically used for:

the apparatus further comprises a reduction template for:

and displaying the virtual keyboard.

The apparatus of the embodiments of the present application may perform the method provided by the embodiments of the present application, and implementation principles of the method are similar, and actions performed by each module in the apparatus of each embodiment of the present application correspond to steps in the method of each embodiment of the present application, and detailed functional descriptions of each module of the apparatus may be referred to in the corresponding method shown in the foregoing, which is not repeated herein.

The embodiment of the application provides an electronic device, which comprises a memory, a processor and a computer program stored on the memory, wherein the processor executes the computer program to realize the steps of the gesture interaction method, and compared with the related technology, the method can realize the steps of the gesture interaction method: the first endpoint used for representing the position of the virtual hand of the user is determined, the target element is determined by calculating the distance between the first endpoint and at least two interaction elements, the positioning of the target element is not affected by the stability of the control end, the positioning accuracy is higher, the user does not need to consume time and attention to keep the stability of the control end, and the user experience is improved. By responding to the user gesture, executing the interaction instruction of the user gesture for the target element, decoupling the selection of the target element from the interaction operation of the user on the target element, and avoiding the problem of false touch. Further, the flexibility of the size design of the elements in the target area is increased, the size of the target area and the size of the elements in the target area can be designed to be smaller, the occupation of the display space is reduced, meanwhile, the movement range of the hands of the user can be reduced, and the experience of the user is further improved.

In an alternative embodiment, there is provided an electronic device, as shown in fig. 13, the electronic device 4000 shown in fig. 13 includes: a processor 4001 and a memory 4003. Wherein the processor 4001 is coupled to the memory 4003, such as via a bus 4002. Optionally, the electronic device 4000 may further comprise a transceiver 4004, the transceiver 4004 may be used for data interaction between the electronic device and other electronic devices, such as transmission of data and/or reception of data, etc. It should be noted that, in practical applications, the transceiver 4004 is not limited to one, and the structure of the electronic device 4000 is not limited to the embodiment of the present application.

The processor 4001 may be a CPU (Central Processing Unit ), general purpose processor, DSP (Digital Signal Processor, data signal processor), ASIC (Application Specific Integrated Circuit ), FPGA (Field Programmable Gate Array, field programmable gate array) or other programmable logic device, transistor logic device, hardware components, or any combination thereof. Which may implement or perform the various exemplary logic blocks, modules, and circuits described in connection with this disclosure. The processor 4001 may also be a combination that implements computing functionality, e.g., comprising one or more microprocessor combinations, a combination of a DSP and a microprocessor, etc.

Bus 4002 may include a path to transfer information between the aforementioned components. Bus 4002 may be a PCI (Peripheral Component Interconnect, peripheral component interconnect standard) bus or an EISA (Extended Industry Standard Architecture ) bus, or the like. The bus 4002 can be divided into an address bus, a data bus, a control bus, and the like. For ease of illustration, only one thick line is shown in fig. 13, but not only one bus or one type of bus.

Memory 4003 may be, but is not limited to, ROM (Read Only Memory) or other type of static storage device that can store static information and instructions, RAM (Random Access Memory ) or other type of dynamic storage device that can store information and instructions, EEPROM (Electrically Erasable Programmable Read Only Memory ), CD-ROM (Compact Disc Read Only Memory, compact disc Read Only Memory) or other optical disk storage, optical disk storage (including compact discs, laser discs, optical discs, digital versatile discs, blu-ray discs, etc.), magnetic disk storage media, other magnetic storage devices, or any other medium that can be used to carry or store a computer program and that can be Read by a computer.

The memory 4003 is used for storing a computer program that executes an embodiment of the present application, and is controlled to be executed by the processor 4001. The processor 4001 is configured to execute a computer program stored in the memory 4003 to realize the steps shown in the foregoing method embodiment.

Embodiments of the present application provide a computer readable storage medium having a computer program stored thereon, where the computer program, when executed by a processor, may implement the steps and corresponding content of the foregoing method embodiments.

The terms "first," "second," "third," "fourth," "1," "2," and the like in the description and in the claims of this application and in the above-described figures, if any, are used for distinguishing between similar objects and not necessarily for describing a particular sequential or chronological order. It is to be understood that the data so used may be interchanged where appropriate such that the embodiments of the present application described herein may be implemented in other sequences than those illustrated or otherwise described.

It should be understood that, although the flowcharts of the embodiments of the present application indicate the respective operation steps by arrows, the order of implementation of these steps is not limited to the order indicated by the arrows. In some implementations of embodiments of the present application, the implementation steps in the flowcharts may be performed in other orders as desired, unless explicitly stated herein. Furthermore, some or all of the steps in the flowcharts may include multiple sub-steps or multiple stages based on the actual implementation scenario. Some or all of these sub-steps or phases may be performed at the same time, or each of these sub-steps or phases may be performed at different times, respectively. In the case of different execution time, the execution sequence of the sub-steps or stages may be flexibly configured according to the requirement, which is not limited in the embodiment of the present application.

The foregoing is merely an optional implementation manner of the implementation scenario of the application, and it should be noted that, for those skilled in the art, other similar implementation manners based on the technical ideas of the application are adopted without departing from the technical ideas of the application, and also belong to the protection scope of the embodiments of the application.

Claims

1. A method of gesture interaction, comprising:

2. The method of gesture interaction of claim 1, wherein determining the first location information of the first endpoint from the virtual hand location information comprises:

3. The gesture interaction method according to claim 1, wherein the determining a target element from the at least two interaction elements further comprises:

determining a second endpoint based on the location of the target element;

4. The gesture interaction method according to claim 1, wherein the determining a target element from at least two interaction elements in the target area based on distances between the first endpoint and the at least two interaction elements, respectively, includes:

otherwise, taking the last nearest element as the target element;

5. The gesture interaction method according to claim 1, wherein the detecting that the first endpoint enters an interaction zone of a target area comprises:

or (b)

6. The gesture interaction method of any of claims 1-5, wherein the target area comprises a virtual keyboard;

Determining the user gesture based on the virtual hand position information;

7. The gesture interaction method of claim 6, wherein the determining a first trigger instruction for a key based on a user gesture comprises:

8. The gesture interaction method of claim 7, further comprising:

9. The gesture interaction method of claim 8, wherein the determining a second trigger instruction for a control bar based on the user gesture comprises any one of:

Detecting that the user gesture accords with a second preset gesture;

10. The gesture interaction method of claim 3, wherein the target area comprises a text box;

determining the user gesture based on the virtual hand position information;

11. The gesture interaction method of claim 10, wherein the selecting the at least one element included in the selected region as the target element further comprises:

12. The gesture interaction method of claim 6, further comprising:

13. The gesture interaction method of claim 12, wherein the determining an adjustment instruction for the variable region based on the user gesture comprises:

displaying a corresponding position adjustment control on the variable region;

14. The gesture interaction method of claim 13, wherein the adjustment instruction comprises a cut instruction for the virtual keyboard:

15. The gesture interaction method of claim 14, wherein the adjustment instruction comprises a merge instruction for at least two sub-regions;

and displaying the virtual keyboard.

16. A gesture interaction device, comprising:

17. An electronic device comprising a memory, a processor and a computer program stored on the memory, characterized in that the processor executes the computer program to implement the steps of the method of any one of claims 1 to 15.

18. A computer readable storage medium, on which a computer program is stored, characterized in that the computer program, when being executed by a processor, implements the steps of the method of any of claims 1 to 15.