WO2018028003A1

WO2018028003A1 - Array type input method and system for use in three-dimensional space

Info

Publication number: WO2018028003A1
Application number: PCT/CN2016/095986
Authority: WO
Inventors: 李小波; 朱郁丛; 杨小冬
Original assignee: 北京英梅吉科技有限公司; 李小波; 朱郁丛; 杨小冬
Priority date: 2016-08-10
Filing date: 2016-08-19
Publication date: 2018-02-15
Also published as: CN106293127B; CN106293127A

Abstract

Disclosed in the present invention are an array type input method and system for use in a three-dimensional space, comprising: receiving an input request from a user; responding to the input request, and configuring a display layout of an optional input element in the three-dimensional space according to the type of an element to be inputted; displaying the optional input element in the three-dimensional space in front of the user in the form of a three-dimensional array; using a three-dimensional space positioning technology to detect an input means medium of the user, and positioning a moving position of the user in the three-dimensional array by using said input means in real-time, detecting a three-dimensional coordinate at each position, thus taking the optional input element corresponding to each three-dimensional coordinate as an input element of the user, until it is detected that the user confirms that an operation is completed; transmitting the input element of the user to a subsequent operating system, thereby completing the input operation. The present invention distributes numbers, letters or symbols, etc. at different layers of depth in the three-dimensional space, solving the problem whereby inputting in a three-dimensional space is inconvenient, and allowing for a user to input in a three-dimensional space in a more organized and efficient manner.

Description

Array input method and system for use in three-dimensional space

Technical field

The present invention relates to the field of virtual three-dimensional space processing technologies, and in particular, to an array input method and system for use in a three-dimensional space.

Background technique

With the development of VR (Virtual Reality), AR (Augmented Reality) and MR (Mixed Reality) technologies, the input operation frequency of users in virtual three-dimensional space also increases rapidly, so in virtual three-dimensional space. It is especially important to complete the input quickly and efficiently.

At this stage, in the (virtual) three-dimensional space, the input mode is mostly in a planar or two-dimensional manner. In some AR scenarios, although there are small input logics with in-depth information, these operations are feasible for light operations or non-large input operations, but when users need to input a long list of passwords, texts, etc., It seems very inefficient, and it can't make full use of the depth information in 3D space, and the security of information is not improved.

Summary of the invention

It is an object of the invention to at least address one of the technical drawbacks.

To this end, the object of the present invention is to provide an array input method and system for use in a three-dimensional space, which can realize the distribution of numbers, letters or symbols in three-dimensional space on different depth layers, thereby greatly reducing each layer. The number of elements solves the problem of inconvenient input in three-dimensional space, making the input of the user in three-dimensional space more organized and efficient.

In order to achieve the above object, an embodiment of an aspect of the present invention provides an array input method for use in a three-dimensional space, comprising the following steps:

Step S1: Receive an input request of a user, where the input request includes: a type of an element to be input;

Step S2, in response to the input request, and setting an alternate input element to display a layout in a three-dimensional space according to the type of the element to be input, wherein the display layout includes a number of layers of the candidate input element in three-dimensional space And arranging, each of the candidate input elements having unique three-dimensional coordinates in the three-dimensional space;

Step S3, displaying the candidate input elements in a three-dimensional array in a three-dimensional space in front of the user;

Step S4, using a three-dimensional spatial positioning technology to detect a user's input means medium, and real-time positioning the user with the input hand A moving position of the segment in the three-dimensional array, detecting three-dimensional coordinates at each position, and then using an alternate input element corresponding to each of the three-dimensional coordinates as an input element of the user until the user's confirmation completion operation is detected;

In step S5, the input element of the user is transmitted to a subsequent operating system to complete an input operation.

Further, the type of the input element includes at least one or a combination of the following: numbers, letters, symbols, and operation control keys.

Further, in the step S2, setting the candidate input element to display the layout in the three-dimensional space according to the type of the element to be input includes:

When the types of the input elements include: numbers, letters, symbols, and operation control keys, the operation control keys, numbers, letters, and symbols are sequentially arranged in a manner that is close to and far from the user, each type The input elements are arranged in one or more layers;

When the type of the input element includes: a number and an operation control key, and the user requests to input a password, sequentially arranging the operation control key and the number in a manner that is close to and far from the user, wherein the The operation control keys are distributed in the same layer, the numbers are arranged in multiple layers, and the same numbers are repeatedly arranged in different layers;

When the type of the input element includes: a letter and an operation control key, the operation control key and the letter are sequentially arranged in a manner that is close to and far from the user, wherein the operation control keys are distributed on the same layer. The letters are arranged in a plurality of layers, and the same letters are repeatedly arranged in different layers.

Further, the selection of the display layout of the candidate input element in three-dimensional space includes the following two methods:

1) The system is pre-configured, and the configuration is automatically selected according to the type of the element to be input in the input request;

2) Configured by the user at the input request.

Further, in the step S4, the three-dimensional spatial positioning technology includes at least one of the following: a time-of-flight TOF positioning technology, a binocular camera positioning technology, an ultrasonic positioning technology, an optical positioning technology, a kinect sensing positioning technology, and a dynamic controller Ps. Move positioning technology, lighthouse positioning technology, laser radar positioning technology, ultra-wideband UWB positioning technology, synchronous positioning and mapping SLAM technology, Virtuix omni positioning technology, structured light positioning technology.

Further, in the step S4, the user input means medium includes: a hand input, an input by an external control device, and input by using a keyboard and a mouse.

Further, in the step S4,

First, the three-dimensional spatial positioning technology is used to acquire the coordinates of the x-y-axis plane of the current input position of the user in the three-dimensional array parallel to the user's perspective;

Then, the three-dimensional spatial positioning technology is used to detect the movement of the user in the three-dimensional array by input means, and acquire coordinates on the z-axis depth information of the user perpendicular to the user's perspective, thereby obtaining the user's The three-dimensional coordinates of the current input position;

Finally, according to the correspondence relationship of the candidate input elements corresponding to each three-dimensional coordinate configured in the step S2, the candidate input element corresponding to the three-dimensional coordinates is used as an input element of the user;

In the step S3, each input element is located, the input element is highlighted to prompt the user, and after receiving the confirmation operation of the user, the input element is transmitted to the subsequent operating system.

Another embodiment of the present invention provides an array input system for use in a three-dimensional space, comprising: a communication module, configured to receive an input request of a user, wherein the input request includes: a type of an element to be input; three-dimensional array generation And a display module, the three-dimensional array generation and display module is connected to the communication module, configured to respond to the input request, and set an alternate input element to display a layout in a three-dimensional space according to the type of the element to be input. The alternative input elements are displayed in a three-dimensional array in a three-dimensional array in front of the user, wherein the display layout comprises a number of layers and an arrangement of the candidate input elements in three-dimensional space, each of the candidate input elements The three-dimensional space has a unique three-dimensional coordinate; the three-dimensional spatial positioning module is connected to the three-dimensional array generation and display module, and is configured to detect a user input medium by using a three-dimensional spatial positioning technology, and real-time Positioning the user to detect the position of movement in the three-dimensional array by the input means, detecting each position 3D coordinates, the alternate input element corresponding to each of the three-dimensional coordinates is used as an input element of the user until the confirmation of the user is completed; the communication module is further configured to transmit the input element of the user to a subsequent operating system , complete the input operation.

Further, the type of the input element includes at least one or a combination of the following: numbers, letters, symbols, and operation control keys,

The three-dimensional array generation and display module sets a display layout of the candidate input elements in three-dimensional space, including the following forms:

When the type of the input element includes: a number and an operation control key and the user requests to input a password, the operation control key and the number are sequentially arranged in a manner that is close to and far from the user, wherein the operation control The keys are distributed in the same layer, the numbers are arranged in multiple layers, and the same numbers are repeatedly arranged in different layers;

2) Configured by the user at the input request.

Further, the three-dimensional spatial positioning technology adopted by the three-dimensional spatial positioning module includes at least one of the following: a time-of-flight TOF positioning technology, a binocular camera positioning technology, an ultrasonic positioning technology, an optical positioning technology, a kinect sensing positioning technology, and a dynamic controller Ps move. Positioning technology, lighthouse positioning technology, laser radar positioning technology, ultra-wideband UWB positioning technology, synchronous positioning and mapping SLAM technology, Virtuix omni positioning technology, structured light positioning technology.

Further, the three-dimensional spatial positioning module acquires the coordinates of the xy-axis plane of the current input position of the user in the three-dimensional array parallel to the user's perspective, and then uses the three-dimensional spatial positioning technology to detect The user acquires coordinates on the z-axis depth information of the user perpendicular to the user's perspective by the movement of the input means in the three-dimensional array, thereby obtaining the three-dimensional coordinates of the current input position of the user; Corresponding relationship of the candidate input elements corresponding to each three-dimensional coordinate, the candidate input element corresponding to the three-dimensional coordinates is used as an input element of the user,

The three-dimensional spatial positioning module locates an input element, highlights the input element to prompt the user, and after receiving the confirmation operation of the user, transmits the input element to the subsequent operating system.

Further, the input means medium of the user includes: hand input, input by using an external control device, input by using a keyboard and a mouse;

The array input system for use in a three-dimensional space is mounted on a virtual reality VR device or an augmented reality AR and a mixed reality MR device, and communicates with the virtual reality VR device or an operating system within the AR and MR devices.

The array input method and system for three-dimensional space according to an embodiment of the present invention fully utilizes the depth dimension information of the (virtual) three-dimensional space, and the elements such as numbers, letters or symbols having a large number of originals are input according to convenience. The rules are distributed on different depth layers, which greatly reduces the number of elements in each layer, solves the problem of inconvenient input in three-dimensional space, and makes the input of users in three-dimensional space more organized and efficient. Moreover, by using the three-dimensional spatial positioning technology to accurately locate the user's input, the user's input elements can be accurately located. Displaying the numerical arrangement through the three-dimensional array can greatly improve the security of the user input information with respect to the two-dimensional planar display form.

The additional aspects and advantages of the invention will be set forth in part in the description which follows.

DRAWINGS

1 is a flow chart of an array input method for use in a three-dimensional space, in accordance with one embodiment of the present invention;

2 is a flow chart of an array input method for use in three-dimensional space, in accordance with another embodiment of the present invention;

3 is a schematic diagram of a three-dimensional array according to an embodiment of the present invention;

4 is a schematic diagram of a three-layer numeric keypad array in accordance with an embodiment of the present invention;

5 is a schematic diagram of a three-layer English alphabet keyboard array according to an embodiment of the present invention;

6 is a schematic diagram of a three-dimensional digital array for selecting a first layer of "5" elements in accordance with an embodiment of the present invention;

7 is a schematic diagram of a three-dimensional digital array point-selecting a second layer "0" element according to an embodiment of the invention;

Figure 8 is a block diagram of an array input system for use in a three dimensional space in accordance with an embodiment of the present invention.

detailed description

The embodiments of the present invention are described in detail below, and the examples of the embodiments are illustrated in the drawings, wherein the same or similar reference numerals are used to refer to the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the drawings are intended to be illustrative of the invention and are not to be construed as limiting.

In the present invention, the terms "installation", "connected", "connected", "fixed" and the like shall be understood broadly, and may be either a fixed connection or a detachable connection, unless explicitly stated and defined otherwise. , or connected integrally; may be mechanical connection or electrical connection; may be directly connected, or may be indirectly connected through an intermediate medium, and may be internal communication between the two elements. For those skilled in the art, the specific meanings of the above terms in the present invention can be understood on a case-by-case basis.

The present invention provides an array input method and system for use in a (virtual) three-dimensional space, by using depth information of a three-dimensional space to display alternate input elements in a three-dimensional array form in a three-dimensional space in front of the user. Forms are relatively flat and two-dimensional in the way of elements, which can make full use of depth information and improve the security of information input by users.

As shown in FIG. 1, the method for inputting an array in a three-dimensional space according to an embodiment of the present invention includes the following steps:

In step S1, an input request of the user is received.

When the user wishes to enter information, the input mode of the device is triggered to send an input request. Wherein, the input request includes: a type of the element to be input. For example, the user may specify in the input request that the elements to be input include: a combination of letters and numbers or a combination of letters, numbers, and symbols. In this step, the device may provide a selection prompt box for displaying the element to be input, and the user selects according to the need, and sends the input request after confirmation.

Step S2, in response to the input request, and setting the candidate input element to display the layout in three-dimensional space according to the type of the element to be input.

Specifically, after receiving the input request of the user, the response to the input request is initiated, and the type of the element to be input recorded in the input request is acquired. The candidate input elements are displayed in a three-dimensional space according to the type of the element to be input.

In one embodiment of the invention, one of the following means of implementing the above display layout is as follows:

(1) rendering a corresponding three-dimensional picture in a virtual picture to realize three-dimensional display;

(2) An optical light guide lens is placed a few centimeters in front of the eye, and a three-dimensional effect is achieved by striking a corresponding three-dimensional image on the lens.

It should be noted that the means for realizing the three-dimensional display layout is not limited to the above two types, and the user mode can also be adopted. The purpose of three-dimensional display of input elements can be achieved, and will not be described here.

The display layout includes the number of layers and the arrangement of the candidate input elements in the three-dimensional space, and the three-dimensional display means is used to arrange the candidate input elements in front of the user in a three-dimensional array. In addition, each candidate input element has unique three-dimensional coordinates in three-dimensional space for subsequent positioning and matching in three-dimensional space.

As shown in FIG. 3, numbers, letters, and symbols are arranged in order from the user to the near and far, wherein the number of letters can be arranged in multiple layers. Of course, FIG. 3 is only an example purpose, and the order, layer number and position of different types of elements can be set by the system and the user according to needs and user habits.

In one embodiment of the invention, the type of input element includes at least one or a combination of the following: numbers, letters, symbols, and operational control keys. The operation control keys are, for example, a backspace key, a newline key, and a confirmation key. It should be noted that the type of the input element is not limited to the above, and may also include its user type, and the type of the input element may be added and deleted as needed.

Specifically, setting the candidate input element to display the layout in three-dimensional space according to the type of the element to be input includes:

(1) When the types of input elements include: numbers, letters, symbols, and operation control keys, the operation control keys, numbers, letters, and symbols are sequentially arranged in a manner that is close to and far from the user, each type of The input elements are arranged in one or more layers.

(2) When the type of the input element includes: the number and the operation control key, the operation control key and the numerical layer are arranged in order according to the distance from the user, wherein the operation control keys are distributed on the same layer, and the number can be The distribution is arranged in one or more layers, and the same numbers are repeatedly arranged in different layers. The order in which the operation controls and numbers are arranged between the different layers is set by the system or user. For example, the operation control key is on the first layer, the number is on the next layer; or the number is on the front, and the operation control key is on the last layer, which can be set according to the user's habits.

For example, when a user needs to input a password, the numbers are arranged in multiple layers, which can improve the security of the input information. Referring to Fig. 4, 10 numbers of 0-9 are displayed in multiple layers, and the same numbers are repeatedly arranged in a plurality of layers, and different layers are distinguished. When using a multi-layer arrangement, the user can switch between multiple layers of numbers so that the surrounding person cannot infer the password entered by the user from the relative displacement of the hand.

Take the input of a 6-digit password as an example. For a normal two-dimensional numeric password, the user's safety factor is 1/10^6. However, with the present invention, input is performed in a three-dimensional array, and since the different layers are distinguished, the same 6-digit password has a security level of 1/30^6. It can be seen that the safety factor is increased by 729 times compared to the two-dimensional input. When the number of input passwords is more or the number of layers is more, the safety factor can be increased more.

When the user only needs to input numbers in daily, the digital single layer can be arranged, referring to the first layer of FIG. The user can specify whether he or she is a password input in the input request, or the system automatically infers that the user performs password input according to the function currently used by the user, thereby configuring the corresponding digital display layout.

(3) When the type of the input element includes: letters and operation control keys, the operation control keys and letters are arranged in layers according to the distance from the user in the distance, wherein the operation control keys are distributed in the same layer, and the letter distribution In a multi-layer arrangement, the same letters are repeatedly arranged in different layers. The order in which the operation control keys and letters are arranged between different layers is set by the system or the user. For example, referring to FIG. 5, the operation control key is on the first layer, the letters are on the next layer; or the letter is on the front, and the operation control key is on the last layer, which can be set according to the user's habits. For example, 26 English letters can be arranged in two layers, each layer is arranged with 13 letters, and the specific arrangement of each letter in each layer is arbitrarily set. In addition, 26 English letters can also be arranged in 3 layers, with 8 letter elements arranged in the first two layers and 10 letter elements arranged in the third layer. Of course, the above is only a preferred embodiment, the three-dimensional array is not limited to the number of layers and the number of elements per layer, the actual form depends on the specific input requirements.

In one embodiment of the invention, the selection of the display layout of the candidate input elements in three dimensions includes the following two methods:

2) Configured by the user in the input request.

It should be noted that the candidate input elements display the layout in a three-dimensional space, and the layering rules are determined according to the user's input content and the convenience of the operation, and the layered layer can be selected according to one's preference and each layer is to be input. The arrangement of the content.

In step S3, the candidate input elements are displayed in a three-dimensional space in front of the user in a three-dimensional array.

In this step, a three-dimensional array in a (virtual) three-dimensional space is generated at a suitable location in the user's field of view, the three-dimensional array including all of the alternate input elements for viewing and selection by the user.

Step S4, using three-dimensional spatial positioning technology to detect the user's input means medium, and real-time positioning the user's moving position in the three-dimensional array by the input means, detecting the three-dimensional coordinates at each position, and then selecting each three-dimensional coordinate correspondingly The input element is used as the user's input element until the user's confirmation is detected to complete the operation.

The numbers, letters or symbols to be input are arranged in a three-dimensional space as required, and displayed in a three-dimensional space in front of the user. When the user is ready to perform an input operation, the user's input means medium is detected by a three-dimensional spatial positioning technique. In an embodiment of the invention, the user's input means medium includes: hand input, input by an external control device, input using a keyboard and mouse. That is, the user can move in the three-dimensional array space by means of a hand, a control device, a keyboard, a mouse, etc., and complete the selection confirmation operation. In addition, the user can use one-hand or single-control device input means to select elements in the three-dimensional array with two- or two-control devices.

The three-dimensional spatial positioning technology is used to spatially locate the moving position of the user in the three-dimensional space, and the user is positioned in real time to move the position of the input means in the three-dimensional array, and the three-dimensional coordinates at each position are detected. According to step S2, each candidate input element has a unique three-dimensional coordinate in three-dimensional space. Comparing the detected three-dimensional coordinates of the moving position with the three-dimensional coordinates of the candidate input elements, determining the candidate input elements selected by the user, and then using the alternate input elements corresponding to each three-dimensional coordinate as the input elements of the user until The user's confirmation completion operation was detected.

In an embodiment of the present invention, the three-dimensional spatial positioning technology includes at least one of the following: a time-of-flight TOF positioning technology, a binocular camera positioning technology, an ultrasonic positioning technology, an optical positioning technology, a kinect sensing positioning technology, and a dynamic controller Ps move positioning. Technology, lighthouse positioning technology, laser radar positioning technology, ultra-wideband UWB positioning technology, synchronous positioning and mapping SLAM technology, Virtuix omni positioning technology, structured light positioning technology.

It should be noted that the three-dimensional spatial positioning technology used in the present invention is not limited to the above examples, and may be a user-type positioning technology, as long as the positioning in the three-dimensional space can be implemented, and details are not described herein again.

In step S5, the input element of the user is transmitted to the subsequent operating system to complete the input operation.

The flight time TOF positioning technology is taken as an example to illustrate the positioning process:

2 is a flow chart of an array input method for use in three-dimensional space in accordance with another embodiment of the present invention.

In step S201, the number of layers and the arrangement of the elements to be input are determined as required.

The type of input element is specified by the user according to the purpose of his input, that is, input of numbers, letters or symbols, or numbers, letters, symbols, and the like.

A three-dimensional array in (virtual) three-dimensional space, ie an array containing z-axis depth information, is generated at a suitable location in the user's field of view, according to explicit input requirements. The z-axis depth information is in one-to-one correspondence with the depth information of the corresponding operation means.

Step S202, displaying a corresponding three-dimensional array.

Step S203, positioning the input means medium using a three-dimensional spatial positioning technique.

The three-dimensional spatial positioning technology is used to acquire the input means medium of the user, for example, by means of a hand, a control device, a keyboard and a mouse.

Step S204, using the input means to move the selection in the three-dimensional array.

After learning the input means medium of the user, the three-dimensional spatial positioning technology is used to acquire the position information of the user in the three-dimensional array by using the input means, and the position information is mapped to the corresponding array element position in the generated three-dimensional space array, and the corresponding position is given. Feedback.

First, the coordinates of the x-y axis plane of the user's current input position in the three-dimensional array parallel to the user's perspective are obtained by using the three-dimensional spatial positioning technology.

Then, the three-dimensional spatial positioning technology is used to detect the movement of the user in the three-dimensional array by the input means, and the coordinates on the z-axis depth information perpendicular to the user's perspective are obtained, thereby obtaining the three-dimensional coordinates of the current input position of the user. Specifically, the position change information is mapped in real time to the three-dimensional coordinates of the candidate input elements in the three-dimensional array with respect to the relative movement of the input means such as the hand or the control device. When moving to the trigger range of an alternate input element in the 3D array, the operation of selecting the element at the corresponding position is implemented.

Finally, corresponding to the corresponding relationship of the candidate input elements corresponding to each three-dimensional coordinate of the configuration, the three-dimensional coordinates are corresponding The alternate input element acts as the user's input element.

The TOF positioning technology is used as a spatial positioning scheme. The gesture input is used as a case input means, and the operation process of the present invention is described by inputting a 0-9 password as a specific case requirement.

First, after the user specifies that he or she wants to enter a numeric password of 0-9, the requirement is explicitly entered in the input request. According to the requirement, a three-dimensional array including digital and operation control keys is generated in a three-dimensional space, and the number and order of the layers can be determined according to the user, for example, set to three layers.

A three-dimensional array with depth information is then generated and presented in the (virtual) three-dimensional space in the user's field of view. The specific generation method is: each array element (that is, an array block with a single number or operation control key) has a unique three-dimensional coordinate, including: two-dimensional coordinate information and depth information of the z-axis. In addition, for the three-layer arrangement, the array shape of the second layer and the third layer will perform some deformation and perspective in the two-dimensional space according to the principle of perspective, so that the visual effect also has a three-dimensional array feeling. Here, the z-axis depth information is in one-to-one correspondence with the depth information of the subsequent input means. After the three-dimensional array is established, the TOF positioning technology is used to first locate the user's input means.

Specifically, in the embodiment, the user uses a gesture as an input means. The medium of the input means is located by the TOF positioning technique, that is, the position of the specific hand. The TOF continuously transmits a light pulse to the medium of the input means, that is, the user's hand through a unique camera, and then receives the light pulse returned from the hand with the sensor, and obtains the depth information of the hand by detecting the round-trip time of the light pulse. The TOF camera can simultaneously obtain the depth information of the entire image in the field of view, that is, the z-axis depth information of the hand.

According to the present invention, the TOF camera is used as a reference, and the relative position of the TOF camera and the generated three-dimensional array and the TOF camera is determined to be unchanged. In this way, a three-dimensional coordinate system with reference to the TOF camera and the three-dimensional array is formed, and the X, Y-axis and Z-axis coordinates of the fixed three-dimensional array are pre-defined. At the same time, the depth information of the hand and the x and y axis coordinates in this coordinate system are given by the TOF camera. The hand moves within this coordinate system relative to the invariant three-dimensional array, so that the relative movement coordinates of the hand in three-dimensional space can be calculated. Since the z-axis depth information and the depth information of the three-dimensional array are one-to-one correspondence. Therefore, by acquiring the relative movement of the hand on the z-axis, it can also be reflected on the z-axis of the invariant three-dimensional array depth information. Based on this, it is possible to realize the calibration and acquisition of the medium of the input means by using the TOF positioning technology: the three-dimensional coordinates of the hand.

After determining the specific location of the hand, the user moving the hand can move the selection in the established three-layer digital array. When the TOF detects the hand, the initial position of the hand is initialized and corresponds to the initial position in the generated three-dimensional array. Thus, it is only necessary to detect the relative movement of the hand to determine the specific position of the hand in the three-dimensional space. As shown in FIG. 6, when the hand is detected, if the three-dimensional coordinate of the hand position at this time is detected as V1, the three-dimensional coordinates of the number "5" of the first layer of the generated three-layer digital array are determined by the judgment. Corresponding to V0, it is preliminarily judged that the input element currently selected by the user is 5. At this time, the "5" of the first layer will be highlighted to prompt the user for the correspondence between the position of the hand and the three-dimensional array.

After seeing the highlighted prompt, if the user confirms that it is correct, it is completed in the three-dimensional array space by moving his own hand. Choose Confirm. For example, if the user wants to select the "0" element of the second layer, the hand needs to be moved down. At this point, the TOF camera will monitor the position of the hand and give feedback in real time. During the movement, it will pass the “8” of the first layer, and it will also highlight it, indicating that it has been selected. As the hand continues to move down, the button will change from highlight to normal when the response range of the first layer "8" is removed; the hand continues to move down, and the "0" element of the first layer will Highlight response. At this time, the hand will stop moving downward and move toward the extension of the z-axis. At this time, the TOF camera will capture the three-dimensional information of the hand, and the depth information of the hand is detected to increase, when compared with the depth information of the three-dimensional array element. When the relative displacement is generated, after the hand moves to the three-dimensional coordinate response range of the "0" element of the second layer, the "0" element of the second layer becomes highlighted, and thus the moving point selection operation is completed. As shown in Fig. 7, when the hand is selected to the "0" of the second layer, it is necessary to confirm the trigger. The triggering method of the specific input means may be arbitrary and is not limited. For example, the index finger is selected by hand as a trigger method. Thus, the "0" of the second layer is input to the corresponding register.

In step S205, the confirmation operation is completed according to the standard of the input means.

When a confirmation operation is performed by an input means such as a hand or a control device, the corresponding element is triggered to complete the input operation.

Step S206, obtaining corresponding input requirements and transmitting to the operating system.

Each time an input element is located, the input element is highlighted to prompt the user, and after receiving the user's confirmation operation, the input element is transmitted to the subsequent operating system.

As shown in FIG. 8, an embodiment of the present invention further provides an array input system for use in a three-dimensional space, comprising: a communication module 1, a three-dimensional array generation and display module 2, and a three-dimensional spatial positioning module 3.

Specifically, the communication module 1 is configured to receive an input request of a user.

When the user wishes to input information, the input mode of the device is triggered and an input request is sent to the communication module 1 of the input system of the present invention. Wherein, the input request includes: a type of the element to be input. For example, the user may specify in the input request that the elements to be input include: a combination of letters and numbers or a combination of letters, numbers, and symbols. In this step, the device may provide a selection prompt box for displaying the element to be input, and the user selects according to the need, and sends the input request after confirmation.

The three-dimensional array generation and display module 2 is connected to the communication module 1 for responding to the input request, and setting the candidate input element to display the layout in three-dimensional space according to the type of the element to be input, and the candidate input element is in front of the user in a three-dimensional array form. The three-dimensional space is displayed.

Specifically, the three-dimensional array generation and display module 2 sets the display layout of the candidate input elements in the three-dimensional space according to the type of the element to be input, including:

(2) When the type of the input element includes: a number and an operation control key and the user requests to input a password, the operation control keys and the numbers are sequentially arranged in a manner that is close to and far from the user, wherein the operation control keys are distributed in In the same layer, numbers can be arranged in one or more layers, and the same numbers are repeatedly arranged in different layers. The order in which the operation controls and numbers are arranged between the different layers is set by the system or user. For example, the operation control key is on the first layer, the number is on the next layer; or the number is on the front, and the operation control key is on the last layer, which can be set according to the user's habits.

(3) When the type of the input element includes: letters and operation control keys, the operation control keys and letters are arranged in layers according to the distance from the user in the distance, wherein the operation control keys are distributed in the same layer, and the letter distribution In a multi-layer arrangement, the same letters are repeatedly arranged in different layers. The order in which the operation control keys and letters are arranged between different layers is set by the system or the user.

2) Configured by the user in the input request.

The three-dimensional array generation and display module 2 further displays the alternate input elements in a three-dimensional array in front of the user in a three-dimensional space.

Specifically, the three-dimensional array generation and display module 2 is generated at a suitable position of the user's visual field of view. A three-dimensional array in (virtual) three-dimensional space that includes all of the alternate input elements for viewing and selection by the user.

The three-dimensional spatial positioning module 3 is connected to the three-dimensional array generating and displaying module 2, and is configured to detect the input means medium of the user by using the three-dimensional spatial positioning technology, and locate the moving position of the user in the three-dimensional array by the input means in real time, and detect each position. The three-dimensional coordinates, the alternate input element corresponding to each three-dimensional coordinate is taken as the input element of the user until the confirmation of the user is completed. Then, the communication module 1 transmits the input element of the user to the subsequent operating system to complete the input operation.

The three-dimensional array generation and display module 2 displays the numbers, letters or symbols to be input in the three-dimensional space as required, and displays them in a three-dimensional space in front of the user. When the user is ready to perform an input operation, the user's input means medium is detected by the three-dimensional spatial positioning module 3. In an embodiment of the invention, the user's input means medium includes: hand input, input by an external control device, input using a keyboard and mouse. That is, the user can move in the three-dimensional array space by means of a hand, a control device, a keyboard, a mouse, etc., and complete the selection confirmation operation. In addition, the user can use one-hand or single-control device input means to select elements in the three-dimensional array with two- or two-control devices.

The three-dimensional spatial positioning module 3 uses the three-dimensional spatial positioning technology to spatially locate the moving position of the user in the three-dimensional space, and locates the moving position of the user in the three-dimensional array by the input means in real time, and detects the three-dimensional coordinates at each position. According to each candidate input element, there is a unique three-dimensional coordinate in three-dimensional space. Comparing the detected three-dimensional coordinates of the moving position with the three-dimensional coordinates of the candidate input elements, determining the candidate input elements selected by the user, and then using the alternate input elements corresponding to each three-dimensional coordinate as the input elements of the user until The user's confirmation completion operation was detected.

In an embodiment of the present invention, the three-dimensional spatial positioning technology adopted by the three-dimensional spatial positioning module 3 includes at least one of the following: a time-of-flight TOF positioning technology, a binocular camera positioning technology, an ultrasonic positioning technology, an optical positioning technology, and a kinect sensing positioning technology. Dynamic controller Ps move positioning technology, lighthouse Light house positioning technology, laser radar positioning technology, ultra-wideband UWB positioning technology, synchronous positioning and mapping SLAM technology, Virtuix omni positioning technology, structured light positioning technology.

In an embodiment of the present invention, the array input system for three-dimensional space provided by the present invention is installed on a virtual reality VR device or an augmented reality AR and a mixed reality MR device, and a virtual reality VR device or an AR and MR device. The internal operating system communicates.

It should be noted that the device to which the present invention is applicable is not limited to the above examples, and the three-dimensional array arrangement input provided by the present invention can be applied as long as there is a requirement for three-dimensional display and input, and details are not described herein again.

The embodiments described above are only intended to describe the preferred embodiments of the present invention, and are not intended to limit the scope of the present invention, and various embodiments of the present invention may be made by those skilled in the art without departing from the spirit of the invention. Modifications and improvements are intended to fall within the scope of the invention as defined by the appended claims.

Industrial applicability

The array input method and system for three-dimensional space of the present invention fully utilizes the depth dimension information of the (virtual) three-dimensional space, and distributes elements such as numbers, letters or symbols that are originally large in number according to the rules of input convenience. Different depth layers, thus greatly reducing the number of elements per layer, solving the problem of inconvenient input in three-dimensional space, making the input of the user in three-dimensional space more organized and efficient. Moreover, by using the three-dimensional spatial positioning technology to accurately locate the user's input, the user's input elements can be accurately located. Displaying the numerical arrangement through the three-dimensional array can greatly improve the security of the user input information with respect to the two-dimensional planar display form.

Claims

An array input method for use in a three-dimensional space, comprising the steps of:

Step S1: Receive an input request of a user, where the input request includes: a type of an element to be input;

Step S2, in response to the input request, and setting an alternate input element to display a layout in a three-dimensional space according to the type of the element to be input, wherein the display layout includes a number of layers of the candidate input element in three-dimensional space And arranging, each of the candidate input elements having unique three-dimensional coordinates in the three-dimensional space;

Step S3, displaying the candidate input elements in a three-dimensional array in a three-dimensional space in front of the user;

Step S4, using three-dimensional spatial positioning technology to detect the user's input means medium, and real-time positioning the user's moving position in the three-dimensional array by the input means, detecting the three-dimensional coordinates at each position, and then each of the three-dimensional coordinates The corresponding candidate input element is used as an input element of the user until the confirmation of the user's confirmation is completed;

In step S5, the input element of the user is transmitted to a subsequent operating system to complete an input operation.
The array input method for use in a three-dimensional space according to claim 1, wherein the type of the input element includes at least one or a combination of the following: numbers, letters, symbols, and operation control keys.

In the step S2, setting the candidate input element to display the layout in the three-dimensional space according to the type of the element to be input includes:

When the types of the input elements include: numbers, letters, symbols, and operation control keys, the operation control keys, numbers, letters, and symbols are sequentially arranged in a manner that is close to and far from the user, each type The input elements are arranged in one or more layers;

When the type of the input element includes: a number and an operation control key, and the user requests to input a password, sequentially arranging the operation control key and the number in a manner that is close to and far from the user, wherein the The operation control keys are distributed in the same layer, the numbers are arranged in multiple layers, and the same numbers are repeatedly arranged in different layers;

When the type of the input element includes: a letter and an operation control key, the operation control key and the letter are sequentially arranged in a manner that is close to and far from the user, wherein the operation control keys are distributed on the same layer. The letters are arranged in a plurality of layers, and the same letters are repeatedly arranged in different layers.
The method for inputting an array in a three-dimensional space according to claim 1, wherein the selection of the display layout of the candidate input element in the three-dimensional space comprises the following two methods:

1) The system is pre-configured, and the configuration is automatically selected according to the type of the element to be input in the input request;

2) Configured by the user at the input request.
The method for inputting an array in a three-dimensional space according to claim 1, wherein in the step S4, the three-dimensional spatial positioning technology comprises at least one of the following: a time-of-flight TOF positioning technology, a binocular camera. Positioning Technology, ultrasonic positioning technology, optical positioning technology, kinect somatosensory positioning technology, dynamic controller Ps move positioning technology, lighthouse Light house positioning technology, laser radar positioning technology, ultra-wideband UWB positioning technology, synchronous positioning and mapping SLAM technology, Virtuix omni Positioning technology, structured light positioning technology.
The method for inputting an array in a three-dimensional space according to claim 1, wherein in the step S4, the input means medium of the user comprises: hand input, input by an external control device, and use of a keyboard. Mouse input.
The array input method for three-dimensional space according to claim 1, wherein in said step S4,

First, the three-dimensional spatial positioning technology is used to acquire the coordinates of the x-y-axis plane of the current input position of the user in the three-dimensional array parallel to the user's perspective;

Then, the three-dimensional spatial positioning technology is used to detect the movement of the user in the three-dimensional array by input means, and acquire coordinates on the z-axis depth information of the user perpendicular to the user's perspective, thereby obtaining the user's The three-dimensional coordinates of the current input position;

Finally, according to the correspondence relationship of the candidate input elements corresponding to each three-dimensional coordinate configured in the step S2, the candidate input element corresponding to the three-dimensional coordinates is used as an input element of the user;

In the step S3, each input element is located, the input element is highlighted to prompt the user, and after receiving the confirmation operation of the user, the input element is transmitted to the subsequent operating system.
An array input system for use in a three-dimensional space, comprising:

a communication module, configured to receive an input request of a user, where the input request includes: a type of an element to be input;

a three-dimensional array generation and display module, the three-dimensional array generation and display module being connected to the communication module, configured to respond to the input request, and set an alternate input element to display a layout in a three-dimensional space according to the type of the element to be input And displaying the candidate input elements in a three-dimensional array in a three-dimensional space in front of the user, wherein the display layout comprises a number of layers and an arrangement of the candidate input elements in three-dimensional space, each of the preparations The selected input element has unique three-dimensional coordinates in the three-dimensional space;

a three-dimensional spatial positioning module, the three-dimensional spatial positioning module is connected to the three-dimensional array generation and display module, and is configured to detect a user input means medium by using a three-dimensional spatial positioning technology, and locate a user in the three-dimensional array by using the input means in real time. Moving position, detecting three-dimensional coordinates at each position, using an alternate input element corresponding to each of the three-dimensional coordinates as an input element of the user until the confirmation of the user is completed;

The communication module is further configured to transmit the input element of the user to a subsequent operating system to complete an input operation.
The array input system for use in a three-dimensional space according to claim 7, wherein the type of the input element comprises at least one or a combination of the following: numbers, letters, symbols, and operation control keys.

The three-dimensional array generation and display module sets a display layout of the candidate input elements in three-dimensional space, including the following forms:

When the types of the input elements include: numbers, letters, symbols, and operation control keys, the operation control keys, numbers, letters, and symbols are sequentially arranged in a manner that is close to and far from the user, each type The input elements are arranged in one or more layers;

When the type of the input element includes: a number and an operation control key and the user requests to input a password, the operation control key and the number are sequentially arranged in a manner that is close to and far from the user, wherein the operation control The keys are distributed in the same layer, the numbers are arranged in multiple layers, and the same numbers are repeatedly arranged in different layers;

When the type of the input element includes: a letter and an operation control key, the operation control key and the letter are sequentially arranged in a manner that is close to and far from the user, wherein the operation control keys are distributed on the same layer. The letters are arranged in a plurality of layers, and the same letters are repeatedly arranged in different layers.
The array input system for use in a three-dimensional space according to claim 7, wherein the selection of the display layout of the candidate input element in the three-dimensional space comprises the following two methods:

1) The system is pre-configured, and the configuration is automatically selected according to the type of the element to be input in the input request;

2) Configured by the user at the input request.
The array input system for use in a three-dimensional space according to claim 7, wherein the three-dimensional spatial positioning technology adopted by the three-dimensional spatial positioning module comprises at least one of the following: a time-of-flight TOF positioning technology, and binocular camera positioning. Technology, ultrasonic positioning technology, optical positioning technology, kinect somatosensory positioning technology, dynamic controller Ps move positioning technology, lighthouse Light house positioning technology, laser radar positioning technology, ultra-wideband UWB positioning technology, synchronous positioning and mapping SLAM technology, Virtuix omni Positioning technology, structured light positioning technology.
The array input system for use in a three-dimensional space according to claim 7, wherein the three-dimensional spatial positioning module acquires a current input position of a user in the three-dimensional array by using the three-dimensional spatial positioning technology Coordinates of the xy-axis plane in which the user's perspective is parallel; then the three-dimensional spatial positioning technique is used to detect the movement of the user in the three-dimensional array by input means, and the z-axis depth information perpendicular to the user's perspective is obtained. The upper coordinate, and then the three-dimensional coordinates of the current input position of the user; finally, according to the correspondence relationship of the alternate input elements corresponding to each three-dimensional coordinate of the configuration, the candidate input element corresponding to the three-dimensional coordinate is used as the input element of the user ,

The three-dimensional spatial positioning module locates an input element, highlights the input element to prompt the user, and after receiving the confirmation operation of the user, transmits the input element to the subsequent operating system.
The array input system for use in a three-dimensional space according to claim 7, wherein the input means medium of the user comprises: hand input, input by an external control device, input by using a keyboard and a mouse; The array input system in three-dimensional space is installed on virtual reality VR equipment or augmented reality AR and mixed reality MR equipment. The virtual reality VR device or the operating system within the AR and MR devices communicate.