WO2022237055A1 - 虚拟键盘交互方法及系统 - Google Patents

虚拟键盘交互方法及系统 Download PDF

Info

Publication number
WO2022237055A1
WO2022237055A1 PCT/CN2021/121388 CN2021121388W WO2022237055A1 WO 2022237055 A1 WO2022237055 A1 WO 2022237055A1 CN 2021121388 W CN2021121388 W CN 2021121388W WO 2022237055 A1 WO2022237055 A1 WO 2022237055A1
Authority
WO
WIPO (PCT)
Prior art keywords
virtual keyboard
image data
preset
fingertip
touch area
Prior art date
Application number
PCT/CN2021/121388
Other languages
English (en)
French (fr)
Inventor
吴涛
Original Assignee
青岛小鸟看看科技有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 青岛小鸟看看科技有限公司 filed Critical 青岛小鸟看看科技有限公司
Priority to US17/816,413 priority Critical patent/US20220365655A1/en
Publication of WO2022237055A1 publication Critical patent/WO2022237055A1/zh

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/048—Interaction techniques based on graphical user interfaces [GUI]
    • G06F3/0487—Interaction techniques based on graphical user interfaces [GUI] using specific features provided by the input device, e.g. functions controlled by the rotation of a mouse with dual sensing arrangements, or of the nature of the input device, e.g. tap gestures based on pressure sensed by a digitiser
    • G06F3/0488—Interaction techniques based on graphical user interfaces [GUI] using specific features provided by the input device, e.g. functions controlled by the rotation of a mouse with dual sensing arrangements, or of the nature of the input device, e.g. tap gestures based on pressure sensed by a digitiser using a touch-screen or digitiser, e.g. input of commands through traced gestures
    • G06F3/04886—Interaction techniques based on graphical user interfaces [GUI] using specific features provided by the input device, e.g. functions controlled by the rotation of a mouse with dual sensing arrangements, or of the nature of the input device, e.g. tap gestures based on pressure sensed by a digitiser using a touch-screen or digitiser, e.g. input of commands through traced gestures by partitioning the display area of the touch-screen or the surface of the digitising tablet into independently controllable areas, e.g. virtual keyboards or menus

Definitions

  • the present application relates to the technical field of virtual keyboards, and more specifically, to a virtual keyboard interaction method and system.
  • the keyboard has been inherited as one of the most common input devices for users to interact with artificial reality systems.
  • the virtual keyboard is mainly rendered and projected directly in front of the user's viewing angle by a virtual reality head-mounted display device (HMD), but the existing virtual keyboard lacks the characteristics that make the physical keyboard successful. For example, users cannot place their hands on the virtual keyboard, there are no landmarks to orient the user's hands, and/or no tactile feedback to indicate that a key has been activated, affecting the user's input experience.
  • HMD virtual reality head-mounted display device
  • the existing method of user participation relying on external devices for example, the user uses a handle controller to call the characters on each key as a method of selecting these keys, which is slow and difficult to input, the efficiency of input characters is not high enough, and the degree of intelligence is low And other issues.
  • the purpose of this application is to provide a virtual keyboard interaction method and system to solve the problems of slow input, low efficiency, and affecting user experience in current artificial reality systems.
  • the virtual keyboard interaction method provided by the present application includes a pre-trained fingertip detection model; through the fingertip detection model, the three-dimensional space position coordinates of all fingertips on the image data to be detected relative to the preset reference position are obtained; based on the three-dimensional space position coordinates, Determine the touch area corresponding to the fingertip; when the touch area overlaps with the preset sensing area of the virtual keyboard, obtain the volume information of the touch area falling into the sensing area; Whether the virtual keyboard where the sensing area corresponding to the control area is located is triggered.
  • the process of pre-training the fingertip detection model includes: collecting image data of sample finger movements; annotating fingertip information on the image data to obtain annotated image data; training neural network based on the annotated image data. network model until the neural network model converges within a preset range to form a fingertip detection model.
  • the fingertips In addition, in at least one exemplary embodiment, during the process of acquiring the three-dimensional position coordinates of all fingertips on the image data to be detected relative to the preset reference position, when the image data to be detected is acquired by a depth camera, the fingertips
  • the detection model directly obtains the three-dimensional space position coordinates of the fingertip relative to the depth camera; when the image data to be detected is obtained through the visible light camera, the image position information of the fingertip on the two image data to be detected is respectively obtained through the fingertip detection model, and passed
  • the triangular stereo imaging principle is used to obtain the three-dimensional space position coordinates of the fingertip relative to the visible light camera.
  • the process of determining the touch area corresponding to the fingertip includes: based on the three-dimensional space position coordinates of the fingertip as the center, a spherical area with a preset distance as the radius, as the contact area of the fingertip corresponding touch area.
  • the preset distance is 2mm-7mm.
  • the preset reference position is the coordinate origin position of the camera that collects the image data to be detected.
  • the process of judging whether the virtual keyboard where the sensing area corresponding to the touch area is located is triggered includes: obtaining the ratio of the volume information to the volume value of the sensing area, To obtain the probability distribution of the overlapping area between the touch area and the sensing area; determine whether the probability distribution is greater than a preset threshold, and when the probability distribution is greater than the preset threshold, it means that the key of the virtual keyboard where the sensing area is located is selected.
  • the key after the key is selected once, it also includes: judging whether the probability distribution of the key in the image data to be detected is greater than the preset threshold; When the probability distribution of several frames is greater than the preset threshold, the character corresponding to the key is triggered; otherwise, the character corresponding to the key is not triggered.
  • the preset threshold is 0.75.
  • the preset number of frames includes 3 frames.
  • the perception area includes a three-dimensional space area in a camera coordinate system for acquiring the image data to be detected, and each of the virtual keys is assigned a three-dimensional space area.
  • a virtual keyboard interaction system including: a fingertip detection model training unit, configured to pre-train the fingertip detection model; a three-dimensional space position coordinate acquisition unit, configured to acquire the fingertip detection model The three-dimensional space position coordinates of all fingertips on the image data to be detected relative to the preset reference position; the touch area determination unit is set to determine the touch area corresponding to the fingertip based on the three-dimensional space position coordinates; the volume information acquisition unit is set In order to obtain the volume information of the touch area falling into the sensing area when the touch area overlaps with the preset virtual keyboard sensing area; the virtual keyboard triggers the judging unit, which is set to judge and touch based on the volume information and preset rules. Whether the virtual keyboard where the perception area corresponding to the area is located is triggered.
  • an electronic device including the virtual keyboard interaction system described in the foregoing embodiments, or including a memory and a processor, the memory is configured to store computer instructions, and the processor is configured to read from The computer instruction is called in the memory to execute the virtual keyboard interaction method described in any one of the above embodiments.
  • a computer-readable storage medium on which a computer program is stored, and when the computer program is executed by a processor, the virtual keyboard interaction method described in any one of the above-mentioned embodiments is implemented.
  • the pre-trained fingertip detection model uses the above-mentioned virtual keyboard interaction method and system to obtain the three-dimensional space position coordinates of all fingertips on the image data to be detected relative to the preset reference position; based on the three-dimensional space position coordinates, determine the corresponding fingertips The touch area; when the touch area overlaps with the preset perception area of the virtual keyboard, the volume information of the touch area falling into the perception area is obtained; based on the volume information and preset rules, the perception corresponding to the touch area is judged Whether the virtual keyboard where the region is located is triggered or not can enable the user to interact with the virtual keyboard conveniently and quickly, improve the accuracy of character input, and achieve a more satisfactory user experience.
  • Fig. 1 is the flowchart of the virtual keyboard interaction method according to the embodiment of the present application.
  • Fig. 2 is a schematic block diagram of a virtual keyboard interaction system according to an embodiment of the present application.
  • Fig. 1 shows the flow of a virtual keyboard interaction method according to an embodiment of the present application.
  • the virtual keyboard interaction method of the embodiment of the present application includes:
  • S120 Using the fingertip detection model, obtain the three-dimensional space position coordinates of all fingertips on the image data to be detected relative to a preset reference position;
  • S130 Determine a touch area corresponding to the fingertip based on the three-dimensional space position coordinates
  • S150 Based on the volume information and preset rules, determine whether the virtual keyboard where the sensing area corresponding to the touch area is located is triggered.
  • the process of pre-training the fingertip detection model in step S110 includes: S111: collecting image data of sample finger movements; S112: labeling fingertip information on the image data to obtain labeled image data; S113: training based on labeled image data Neural network model until the neural network model converges within a preset range to form a fingertip detection model.
  • the corresponding fingertip detection models are also divided into two categories.
  • the virtual reality display device HMD
  • the virtual reality display device has a built-in depth camera
  • the built-in visible light camera of the HMD in the interactive scene of the virtual keyboard provided according to the application, at least two visible light cameras can be used to collect about 3 million finger movement data About a piece of (visible light) image data, and then mark the 10 fingertip information of the left and right hands on the (visible light) image data, and train the convolutional neural network model through the marked image data after marking to obtain the corresponding fingertip detection model .
  • the fingertips are directly obtained through the fingertip detection model
  • the three-dimensional space position coordinates of the relative depth camera when the image data to be detected is acquired by a visible light camera, the image position information of the fingertips on the two image data to be detected are obtained respectively through the fingertip detection model, and through the triangular stereoscopic imaging principle, obtain The three-dimensional space position coordinates of the fingertip relative to the visible light camera.
  • the above-mentioned preset reference position can be the coordinate origin position of the camera that collects the image data to be detected.
  • step S120 the three-dimensional space position coordinates of all fingertips of the two hands of the user at the current moment are sequentially obtained, and then a touch area is assigned to the three-dimensional space position coordinates of all fingertips.
  • the three-dimensional space position coordinates of the fingertips can be used as the center of the circle, A spherical area with a preset distance as a radius serves as a touch area corresponding to the fingertip.
  • the preset distance can be set to 2mm-7mm, and generally the spherical area that can be set to be 5mm away from the three-dimensional space position coordinates of the fingertip is the touch area.
  • the perception area of the virtual keyboard can be set as the three-dimensional space area under the HMD built-in camera coordinate system, that is, each virtual key is assigned a three-dimensional space area.
  • the virtual The volume of the perception area of the keyboard keys is set to 15mm*15mm*15mm (length*width*height).
  • each button of the virtual keyboard has a corresponding three-dimensional space relative to the HMD camera coordinate system Under the sensing area, when the fingertip area (touch area) falls into the cube area of the key with a certain probability, the key character corresponding to the area is input into the virtual reality content.
  • step S150 based on the volume information and preset rules, the process of judging whether the virtual keyboard where the sensing area corresponding to the touch area is located is triggered includes: obtaining the ratio of the volume information to the volume value of the sensing area to obtain the ratio of the touch area and The probability distribution of the overlapping area of the sensing area; judging whether the probability distribution is greater than a preset threshold, and when the probability distribution is greater than the preset threshold, it means that the key of the virtual keyboard where the sensing area is located is selected.
  • the volume information T of the touch area falling into the perception area for example, calculate the probability distribution of the button being selected and input by the user, the probability distribution is T/(15mm*15mm*15mm), when the probability distribution is greater than the preset threshold, it means the perception The key of the virtual keyboard where the zone is located is selected.
  • the preset threshold can be set to 0.75, and the range of the above-mentioned sensing area, the range of the preset threshold, and the range of the touch area can all be set and adjusted according to the size of the specific virtual keyboard or user experience, and It is not specifically limited to the above-mentioned respective data.
  • the button after the button is judged to be selected once, it may also include: judging whether the probability distribution of a preset number of frames of the button in the image data to be detected is greater than a preset threshold; When the probability distribution of the consecutive preset number of frames is greater than the preset threshold, the character corresponding to the key is triggered; otherwise, the character corresponding to the key is not triggered.
  • the above-mentioned consecutive preset number of frames can be set to 3 consecutive frames. If the images of the current 3 consecutive frames of the button are all detected as having a selected probability distribution greater than 0.75, then the character corresponding to the button is input to the current virtual reality content.
  • the virtual keyboard interaction method provided by the present application can conveniently and quickly realize the intelligent interaction between the user and the virtual keyboard, and the key input speed is fast and the accuracy is high, so as to achieve a faster and more satisfying user experience.
  • FIG. 2 shows a schematic logic of a virtual keyboard interaction system according to an embodiment of the present application.
  • the virtual keyboard interaction system 200 of the embodiment of the present application includes:
  • the fingertip detection model training unit 210 is configured as a pre-trained fingertip detection model
  • the three-dimensional space position coordinate acquisition unit 220 is configured to obtain the three-dimensional space position coordinates of all fingertips on the image data to be detected relative to the preset reference position through the fingertip detection model;
  • the touch area determination unit 230 is configured to determine the touch area corresponding to the fingertip based on the three-dimensional space position coordinates
  • the volume information acquiring unit 240 is configured to acquire the volume information that the touch area falls into the sensing area when the touch area overlaps with the preset sensing area of the virtual keyboard;
  • the virtual keyboard trigger judging unit 250 is configured to judge whether the virtual keyboard where the sensing area corresponding to the touch area is located is triggered based on volume information and preset rules.
  • the device may include a virtual keyboard interaction system 200 as shown in FIG. 2 , or the device may include a memory and a processor. Instructions, the processor is configured to call the computer instructions from the memory to execute any virtual keyboard interaction method provided in the above method embodiments.
  • the embodiment of the present application also provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, any virtual keyboard interaction method provided in the above-mentioned method embodiments is implemented.
  • a virtual keyboard is rendered and presented by the HMD at the preset position of the user's visual angle, and the user can use the left and right hands to simulate the user's use of the physical keyboard.
  • the finger confirms the operation input of the virtual key; based on HMD rendering, a virtual keyboard located in front of the user's eye view is generated, and the movement information of the user's left and right fingers is detected in real time based on the convolutional neural network model, and the position in the 3D space close to the perceived position of the virtual keyboard is detected.
  • the motion path information of the fingertips tracks the path distribution of the user's fingers on the virtual keys during the motion process, and obtains parameters associated with user input interactions.

Abstract

本申请提供了一种虚拟键盘交互方法及系统，包括预训练手指尖检测模型；通过手指尖检测模型，获取待检测图像数据上的所有手指尖相对预设基准位置的三维空间位置坐标；基于三维空间位置坐标，确定与手指尖对应的触控区域；当触控区域与预设的虚拟键盘的感知区域存在重叠时，获取触控区域落入感知区域的体积信息；基于体积信息及预设规则，判断与触控区域对应的感知区域所在的虚拟键盘是否触发。利用上述实施例能够提高字符输入效率及准确度，提高用户体验。

Description

虚拟键盘交互方法及系统
交叉引用
本申请要求于2021年05月10日提交中国国家知识产权局、申请号为202110505160.1、公开名称“虚拟键盘交互方法及系统”的中国专利申请的优先权，其全部内容通过引用结合在本申请中。
技术领域
本申请涉及虚拟键盘技术领域，更为具体地，涉及一种虚拟键盘交互方法及系统。
背景技术
随着计算机游戏、健康与安全、工业和教育等领域的发展，人工现实系统在这些领域中的应用变得越来越普遍。例如，人工现实系统正被整合到移动设备、游戏机、个人电脑、电影院和主题公园等，人工现实是在呈现给用户之前以某种方式调整现实的一种形式，其可包括例如虚拟现实(VR)、增强现实(AR)、混合现实(MR)或其某些组合和/或衍生物。
随着人工现实在各行各业的普及和发展，键盘作为一种用户和人工现实系统交互最为普遍的输入设备之一被继承下来。通常，虚拟键盘主要通过虚拟现实头戴式显示器设备(HMD)被渲染投影在用户视角的正前方，但是现有的虚拟键盘缺乏促成物理键盘成功的特性。例如，用户不能将他们的手放在虚拟键盘上，没有地标来确定用户的手的方向，和/或没有触觉反馈来指示键已被激活，影响用户的输入体验。
另外，现有的依靠外部设备实现用户参与方式，例如用户通过一种手柄控制器调用各个键上的字符作为选择这些键的方法，存在输入缓慢且困难，输入字符效率不够高、智能化程度低等问题。
发明内容
鉴于上述问题，本申请的目的是提供一种虚拟键盘交互方法及系统，以 解决目前人工现实系统存在的输入缓慢、效率低、影响用户体验等问题。
本申请提供的虚拟键盘交互方法，包括预训练手指尖检测模型；通过手指尖检测模型，获取待检测图像数据上的所有手指尖相对预设基准位置的三维空间位置坐标；基于三维空间位置坐标，确定与手指尖对应的触控区域；当触控区域与预设的虚拟键盘的感知区域存在重叠时，获取触控区域落入感知区域的体积信息；基于体积信息及预设规则，判断与触控区域对应的感知区域所在的虚拟键盘是否触发。
此外，在至少一个示例性实施例中，预训练手指尖检测模型的过程包括：采集样本手指运动的图像数据；在图像数据上标注手指尖信息，以获取标注图像数据；基于标注图像数据训练神经网络模型，直至神经网络模型收敛在预设范围内，形成手指尖检测模型。
此外，在至少一个示例性实施例中，在获取待检测图像数据上的所有手指尖相对预设基准位置的三维空间位置坐标的过程中，当待检测图像数据通过深度相机获取时，通过手指尖检测模型直接获取手指尖相对深度相机的三维空间位置坐标；当待检测图像数据通过可见光相机获取时，通过手指尖检测模型分别获取两个待检测图像数据上的手指尖的图像位置信息，并通过三角立体成像原理，获取手指尖相对可见光相机的三维空间位置坐标。
此外，在至少一个示例性实施例中，确定与手指尖对应的触控区域的过程包括：基于以手指尖的三维空间位置坐标为圆心，以预设距离为半径的球形区域，作为与手指尖对应的触控区域。
此外，在至少一个示例性实施例中，预设距离为2mm-7mm。
此外，在至少一个示例性实施例中，预设基准位置为采集待检测图像数据的相机的坐标原点位置。
此外，在至少一个示例性实施例中，基于体积信息及预设规则，判断与触控区域对应的感知区域所在的虚拟键盘是否触发的过程包括：获取体积信息与感知区域的体积值之比，以获取触控区域与感知区域的重叠区域的概率分布；判断概率分布是否大于预设阈值，当概率分布大于预设阈值时，表示感知区域所在的虚拟键盘的按键被选择。
此外，在至少一个示例性实施例中，在按键被选择一次后，还包括：判断按键在待检测图像数据中连续预设个数帧的概率分布是否均大于预设阈 值；当按键在连续预设个数帧的概率分布均大于预设阈值时，触发按键对应的字符；否则，不触发按键对应的字符。
此外，在至少一个示例性实施例中，预设阈值为0.75。
此外，在至少一个示例性实施例中，预设个数帧包括3帧。
此外，在至少一个示例性实施例中，所述感知区域包括用于获取所述待检测图像数据的相机坐标系下的三维空间区域，且每一个所述虚拟按键均分配有一块三维空间区域。
根据本申请的另一方面，提供一种虚拟键盘交互系统，包括：手指尖检测模型训练单元，设置为预训练手指尖检测模型；三维空间位置坐标获取单元，设置为通过手指尖检测模型，获取待检测图像数据上的所有手指尖相对预设基准位置的三维空间位置坐标；触控区域确定单元，设置为基于三维空间位置坐标，确定与手指尖对应的触控区域；体积信息获取单元，设置为当触控区域与预设的虚拟键盘的感知区域存在重叠时，获取触控区域落入感知区域的体积信息；虚拟键盘触发判断单元，设置为基于体积信息及预设规则，判断与触控区域对应的感知区域所在的虚拟键盘是否触发。
根据本申请的另一方面，提供一种电子装置，包括前述实施例中所述的虚拟键盘交互系统，或者包括存储器和处理器，所述存储器设置为存储计算机指令，所述处理器设置为从所述存储器中调用所述计算机指令，以执行上述任一项实施例中所述的虚拟键盘交互方法。
根据本申请的另一方面，提供一种计算机可读存储介质，其上存储有计算机程序，所述计算机程序在被处理器执行时实现上述任一项实施例中所述的虚拟键盘交互方法。
利用上述虚拟键盘交互方法及系统，通过预训练的手指尖检测模型，获取待检测图像数据上的所有手指尖相对预设基准位置的三维空间位置坐标；基于三维空间位置坐标，确定与手指尖对应的触控区域；当触控区域与预设的虚拟键盘的感知区域存在重叠时，获取触控区域落入感知区域的体积信息；基于体积信息及预设规则，判断与触控区域对应的感知区域所在的虚拟键盘是否触发，能够使得用户便捷、快速地与虚拟键盘交互，并提高字符输入的准确度，实现更令人满意的用户体验。
为了实现上述以及相关目的，本申请的一个或多个方面包括后面将详细说明的特征。下面的说明以及附图详细说明了本申请的某些示例性方面。然而，这些方面指示的仅仅是可使用本申请的原理的各种方式中的一些方式。此外，本申请旨在包括所有这些方面以及它们的等同物。
附图说明
通过参考以下结合附图的说明，并且随着对本申请的更全面理解，本申请的其它目的及结果将更加明白及易于理解。在附图中：
图1为根据本申请实施例的虚拟键盘交互方法的流程图；
图2为根据本申请实施例的虚拟键盘交互系统的方框示意图。
在所有附图中相同的标号指示相似或相应的特征或功能。
具体实施方式
在下面的描述中，出于说明的目的，为了提供对一个或多个实施例的全面理解，阐述了许多具体细节。然而，很明显，也可以在没有这些具体细节的情况下实现这些实施例。在其它例子中，为了便于描述一个或多个实施例，公知的结构和设备以方框图的形式示出。
在本申请的描述中，需要理解的是，术语“中心”、“纵向”、“横向”、“长度”、“宽度”、“厚度”、“上”、“下”、“前”、“后”、“左”、“右”、“竖直”、“水平”、“顶”、“底”“内”、“外”、“顺时针”、“逆时针”、“轴向”、“径向”、“周向”等指示的方位或位置关系为基于附图所示的方位或位置关系，仅是为了便于描述本申请和简化描述，而不是指示或暗示所指的装置或元件必须具有特定的方位、以特定的方位构造和操作，因此不能理解为对本申请的限制。
为详细描述本申请的虚拟键盘交互方法及系统，以下将结合附图对本申请的具体实施例进行详细描述。
图1示出了根据本申请实施例的虚拟键盘交互方法的流程。
如图1所示，本申请实施例的虚拟键盘交互方法，包括：
S110：预训练手指尖检测模型；
S120：通过手指尖检测模型，获取待检测图像数据上的所有手指尖相对预设基准位置的三维空间位置坐标；
S130：基于三维空间位置坐标，确定与手指尖对应的触控区域；
S140：当触控区域与预设的虚拟键盘的感知区域存在重叠时，获取触控区域落入感知区域的体积信息；
S150：基于体积信息及预设规则，判断与触控区域对应的感知区域所在的虚拟键盘是否触发。
具体地，步骤S110中预训练手指尖检测模型的过程包括：S111：采集样本手指运动的图像数据；S112：在图像数据上标注手指尖信息，以获取标注图像数据；S113：基于标注图像数据训练神经网络模型，直至神经网络模型收敛在预设范围内，形成手指尖检测模型。
其中，针对不同的图像数据的采集方式，对应的手指尖检测模型也分为两个类别，如果虚拟现实显示设备(HMD)内置的为深度相机，在根据本申请提供的虚拟键盘的交互场景中，采集HMD内的深度相机的手指运动数据大概300万张左右的图像数据，然后在图像数据上标注左、右手的10个手指尖信息，并通过标注后的标注图像数据训练卷积神经网络模型，获得对应的高精度的手指尖检测模型；另外，当HMD内置的为可见光相机时，在根据本申请提供的虚拟键盘的交互场景中，可通过至少两个可见光相机采集手指运动数据大概300万张左右的(可见光)图像数据，然后在(可见光)图像数据上标注左、右手的10个手指尖信息，并通过标注后的标注图像数据训练卷积神经网络模型，获得对应的手指尖检测模型。
在通过手指尖检测模型获取待检测图像数据上的所有手指尖相对预设基准位置的三维空间位置坐标的过程中，当待检测图像数据通过深度相机获取时，通过手指尖检测模型直接获取手指尖相对深度相机的三维空间位置坐标；当待检测图像数据通过可见光相机获取时，通过手指尖检测模型分别获取两个待检测图像数据上的手指尖的图像位置信息，并通过三角立体成像原理，获取手指尖相对可见光相机的三维空间位置坐标。
可知，上述预设基准位置可采用采集待检测图像数据的相机的坐标原点位置。
根据步骤S120依次获取当前时刻用户两个手的所有手指尖的三维空间位置坐标，然后对所有手指尖的三维空间位置坐标赋予一个触控区域，具体可以以手指尖的三维空间位置坐标为圆心，以预设距离为半径的球形区域，作 为与手指尖对应的触控区域。
作为具体示例，预设距离可设置为2mm-7mm，通常可设置为距离手指尖的三维空间位置坐标5mm的球形区域为触控区域。
需要说明的是，虚拟键盘的感知区域可设置为HMD内置的相机坐标系下的三维空间区域，即每一个虚拟按键的按键均分配一块三维空间区域，为了提高手指触控的灵敏度，可将虚拟键盘的按键的感知区域的体积设置为15mm*15mm*15mm(长*宽*高)，按照虚拟键盘的体积参数，该虚拟键盘的每一个按键相对HMD相机坐标系下均有一块对应的三维空间下的感知区域，当手指尖区域(触控区域)落入该按键的正方体区域一定的概率下，该区域对应的按键字符被输入到虚拟现实内容中。
在步骤S150中，基于体积信息及预设规则，判断与触控区域对应的感知区域所在的虚拟键盘是否触发的过程包括：获取体积信息与感知区域的体积值之比，以获取触控区域与感知区域的重叠区域的概率分布；判断概率分布是否大于预设阈值，当概率分布大于预设阈值时，表示感知区域所在的虚拟键盘的按键被选择。
具体地，对每一个手指尖的触控区域进行检测和分析，确认是否有触控区域落入虚拟键盘上的一些按键的感知区域中，当存在触控区域落入感知区域内时，进一步判断触控区域落入感知区域内的体积信息T，例如，计算该按键被用户选择输入的概率分布，概率分布为T/(15mm*15mm*15mm)，当概率分布大于预设阈值时，表示感知区域所在的虚拟键盘的按键被选择。
作为具体示例，该预设阈值可设置为0.75，上述感知区域的范围、预设阈值的范围以及触控区域的范围，均可根据具体的虚拟键盘的尺寸或者用户体验进行设定及调整，并不具体限制为上述各数据。
在本申请的一个具体实施方式中，在按键被判断为选择一次后，还可以包括：判断按键在待检测图像数据中连续预设个数帧的概率分布是否均大于预设阈值；当按键在连续预设个数帧的概率分布均大于预设阈值时，触发按键对应的字符；否则，不触发按键对应的字符。
上述连续预设个数帧可设置为连续3帧，如果当前按键的连续3帧的图像均检测为被选择的概率分布大于0.75，则把该按键对应的字符输入给当前的虚拟现实内容。
本申请提供的虚拟键盘交互方法，能够方便、快捷的实现用户与虚拟键盘之间的智能交互，按键的输入速度快、准确度高，从而达到更快更令用户满意的体验效果。
与上述虚拟键盘交互方法相对应，本申请还提供一种虚拟键盘交互系统。具体地，图2示出了根据本申请实施例的虚拟键盘交互系统的示意逻辑。
如图2所示，本申请实施例的虚拟键盘交互系统200，包括：
手指尖检测模型训练单元210，设置为预训练手指尖检测模型；
三维空间位置坐标获取单元220，设置为通过手指尖检测模型，获取待检测图像数据上的所有手指尖相对预设基准位置的三维空间位置坐标；
触控区域确定单元230，设置为基于三维空间位置坐标，确定与手指尖对应的触控区域；
体积信息获取单元240，设置为当触控区域与预设的虚拟键盘的感知区域存在重叠时，获取触控区域落入感知区域的体积信息；
虚拟键盘触发判断单元250，设置为基于体积信息及预设规则，判断与触控区域对应的感知区域所在的虚拟键盘是否触发。
需要说明的是，上述虚拟键盘交互系统的实施例可参考虚拟键盘交互方法实施例中的描述，此处不再一一赘述。
本申请实施例中还提供了一种电子装置，该装置中可以包括如图2所示的虚拟键盘交互系统200，或者，该装置中可以包括储器和处理器，所述存储器设置为存储计算机指令，所述处理器设置为从所述存储器中调用所述计算机指令，以执行上述方法实施例中提供的任一种虚拟键盘交互方法。
本申请实施例中还提供了一种计算机可读存储介质，其上存储有计算机程序，所述计算机程序在被处理器执行时实现根据上述方法实施例中提供的任一种虚拟键盘交互方法。
通过上述本申请提供的虚拟键盘交互方法及系统，通过HMD在用户眼前视角的预设位置上渲染呈现一个虚拟键盘，用户可以通过左右两只手模拟用户使用物理键盘方式，通过用户左右手的10个手指对虚拟键进行确认操作输入；基于HMD渲染生成具有位于用户眼睛视角前面的虚拟键盘，通过基于卷积神经网络模型实时检测用户左右手指运动信息，检测接近虚拟键盘的感知位置的3D空间中的手指尖的运动路径信息，追踪用户手指在运动过程中的虚 拟键上的路径分布，获取与用户输入交互相关联的参数，这些参数可用于计算用户打算选择各个键的概率，分析各个键在一定时间序列上的概率分布，确认用户真正选择的键，最终把选择键对应的字符输入给虚拟现实，提高虚拟键盘按键输入的用户体验。
如上参照附图以示例的方式描述根据本申请的虚拟键盘交互方法及系统。但是，本领域技术人员应当理解，对于上述本申请所提出的虚拟键盘交互方法及系统，还可以在不脱离本申请内容的基础上做出各种改进。因此，本申请的保护范围应当由所附的权利要求书的内容确定。

Claims (14)

  1. 一种虚拟键盘交互方法，包括：
    预训练手指尖检测模型；
    通过所述手指尖检测模型，获取待检测图像数据上的所有手指尖相对预设基准位置的三维空间位置坐标；
    基于所述三维空间位置坐标，确定与所述手指尖对应的触控区域；
    当所述触控区域与预设的虚拟键盘的感知区域存在重叠时，获取所述触控区域落入所述感知区域的体积信息；
    基于所述体积信息及预设规则，判断与所述触控区域对应的感知区域所在的虚拟键盘是否触发。
  2. 如权利要求1所述的虚拟键盘交互方法，其中，所述预训练手指尖检测模型的过程包括：
    采集样本手指运动的图像数据；
    在所述图像数据上标注手指尖信息，以获取标注图像数据；
    基于所述标注图像数据训练神经网络模型，直至所述神经网络模型收敛在预设范围内，形成手指尖检测模型。
  3. 如权利要求1所述的虚拟键盘交互方法，其中，在获取待检测图像数据上的所有手指尖相对预设基准位置的三维空间位置坐标的过程中，
    当所述待检测图像数据通过深度相机获取时，通过所述手指尖检测模型直接获取所述手指尖相对所述深度相机的三维空间位置坐标；
    当所述待检测图像数据通过可见光相机获取时，通过所述手指尖检测模型分别获取两个待检测图像数据上的手指尖的图像位置信息，并通过三角立体成像原理，获取所述手指尖相对所述可见光相机的三维空间位置坐标。
  4. 如权利要求1所述的虚拟键盘交互方法，其中，确定与所述手指尖对应的触控区域的过程包括：
    基于以所述手指尖的三维空间位置坐标为圆心，以预设距离为半径的球 形区域，作为与所述手指尖对应的触控区域。
  5. 如权利要求4所述的虚拟键盘交互方法，其中，
    所述预设距离为2mm-7mm。
  6. 如权利要求4所述的虚拟键盘交互方法，其中，
    所述预设基准位置为采集所述待检测图像数据的相机的坐标原点位置。
  7. 如权利要求1所述的虚拟键盘交互方法，其中，基于所述体积信息及预设规则，判断与所述触控区域对应的感知区域所在的虚拟键盘是否触发的过程包括：
    获取所述体积信息与所述感知区域的体积值之比，以获取所述触控区域与所述感知区域的重叠区域的概率分布；
    判断所述概率分布是否大于预设阈值，当所述概率分布大于所述预设阈值时，表示所述感知区域所在的虚拟键盘的按键被选择。
  8. 如权利要求7所述的虚拟键盘交互方法，其中，在所述按键被选择一次后，还包括：
    判断所述按键在所述待检测图像数据中连续预设个数帧的概率分布是否均大于所述预设阈值；
    当所述按键在连续所述预设个数帧的概率分布均大于所述预设阈值时，触发所述按键对应的字符；否则，不触发所述按键对应的字符。
  9. 如权利要求7所述的虚拟键盘交互方法，其中，
    所述预设阈值为0.75。
  10. 如权利要求7所述的虚拟键盘交互方法，其中，
    所述预设个数帧包括3帧。
  11. 如权利要求1所述的虚拟键盘交互方法，其中，所述感知区域包括 用于获取所述待检测图像数据的相机坐标系下的三维空间区域，且每一个所述虚拟按键均分配有一块三维空间区域。
  12. 一种虚拟键盘交互系统，包括：
    手指尖检测模型训练单元，设置为预训练手指尖检测模型；
    三维空间位置坐标获取单元，设置为通过所述手指尖检测模型，获取待检测图像数据上的所有手指尖相对预设基准位置的三维空间位置坐标；
    触控区域确定单元，设置为基于所述三维空间位置坐标，确定与所述手指尖对应的触控区域；
    体积信息获取单元，设置为当所述触控区域与预设的虚拟键盘的感知区域存在重叠时，获取所述触控区域落入所述感知区域的体积信息；
    虚拟键盘触发判断单元，设置为基于所述体积信息及预设规则，判断与所述触控区域对应的感知区域所在的虚拟键盘是否触发。
  13. 一种电子装置，包括如权利要求12所述的系统；或者，
    包括存储器和处理器，所述存储器设置为存储计算机指令，所述处理器设置为从所述存储器中调用所述计算机指令，以执行如权利要求1-11中任一项所述的方法。
  14. 一种计算机可读存储介质，其上存储有计算机程序，所述计算机程序在被处理器执行时实现根据权利要求1-11中任一项所述的方法。
PCT/CN2021/121388 2021-05-10 2021-09-28 虚拟键盘交互方法及系统 WO2022237055A1 (zh)

Priority Applications (1)

Application Number Priority Date Filing Date Title
US17/816,413 US20220365655A1 (en) 2021-05-10 2022-07-30 Virtual Keyboard Interaction Method and System

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN202110505160.1 2021-05-10
CN202110505160.1A CN113238705A (zh) 2021-05-10 2021-05-10 虚拟键盘交互方法及系统

Related Child Applications (1)

Application Number Title Priority Date Filing Date
US17/816,413 Continuation US20220365655A1 (en) 2021-05-10 2022-07-30 Virtual Keyboard Interaction Method and System

Publications (1)

Publication Number Publication Date
WO2022237055A1 true WO2022237055A1 (zh) 2022-11-17

Family

ID=77133230

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2021/121388 WO2022237055A1 (zh) 2021-05-10 2021-09-28 虚拟键盘交互方法及系统

Country Status (2)

Country Link
CN (1) CN113238705A (zh)
WO (1) WO2022237055A1 (zh)

Families Citing this family (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN113238705A (zh) * 2021-05-10 2021-08-10 青岛小鸟看看科技有限公司 虚拟键盘交互方法及系统
WO2023087158A1 (en) * 2021-11-16 2023-05-25 Huawei Technologies Co., Ltd. Method of generating haptic output and electronic apparatus configured to generate haptic output using said method
CN114690900B (zh) * 2022-03-16 2023-07-18 中数元宇数字科技(上海)有限公司 一种虚拟场景中的输入识别方法、设备及存储介质

Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN106371599A (zh) * 2016-09-08 2017-02-01 清华大学 深度图像中的高精度指尖定位方法及装置
CN108536288A (zh) * 2018-03-27 2018-09-14 网易（杭州）网络有限公司 虚拟现实中交互操作的处理方法和装置
CN109933190A (zh) * 2019-02-02 2019-06-25 青岛小鸟看看科技有限公司 一种头戴显示设备及其交互方法
CN113238705A (zh) * 2021-05-10 2021-08-10 青岛小鸟看看科技有限公司 虚拟键盘交互方法及系统

Family Cites Families (10)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US9122395B2 (en) * 2012-05-29 2015-09-01 Garett Engle Method of capturing system input by relative finger positioning
KR101865655B1 (ko) * 2014-06-26 2018-06-11 한국과학기술원 증강현실 상호 작용 서비스 제공 장치 및 방법
CN107168541A (zh) * 2017-04-07 2017-09-15 北京小鸟看看科技有限公司 一种输入的实现方法和装置
CN107563494B (zh) * 2017-08-01 2020-08-18 华南理工大学 一种基于卷积神经网络和热图的第一视角指尖检测方法
CN110163048B (zh) * 2018-07-10 2023-06-02 腾讯科技（深圳）有限公司 手部关键点的识别模型训练方法、识别方法及设备
CN109683667A (zh) * 2018-12-25 2019-04-26 上海萃钛智能科技有限公司 一种头戴式计算机及其信息输入方法
CN109858524B (zh) * 2019-01-04 2020-10-16 北京达佳互联信息技术有限公司 手势识别方法、装置、电子设备及存储介质
US11029845B2 (en) * 2019-07-11 2021-06-08 Microsoft Technology Licensing, Llc Virtual keyboard engagement
CN112541375A (zh) * 2019-09-20 2021-03-23 阿里巴巴集团控股有限公司 一种手部关键点识别方法及装置
CN112651298A (zh) * 2020-11-27 2021-04-13 深圳点猫科技有限公司 基于手指关节定位的点读方法、装置、系统及介质

Patent Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN106371599A (zh) * 2016-09-08 2017-02-01 清华大学 深度图像中的高精度指尖定位方法及装置
CN108536288A (zh) * 2018-03-27 2018-09-14 网易（杭州）网络有限公司 虚拟现实中交互操作的处理方法和装置
CN109933190A (zh) * 2019-02-02 2019-06-25 青岛小鸟看看科技有限公司 一种头戴显示设备及其交互方法
CN113238705A (zh) * 2021-05-10 2021-08-10 青岛小鸟看看科技有限公司 虚拟键盘交互方法及系统

Also Published As

Publication number Publication date
CN113238705A (zh) 2021-08-10

Similar Documents

Publication Publication Date Title
WO2022237055A1 (zh) 虚拟键盘交互方法及系统
US10511778B2 (en) Method and apparatus for push interaction
US8395620B2 (en) Method and system for tracking of a subject
CN103713737B (zh) 用于智能眼镜的虚拟键盘系统
CN104471511B (zh) 识别指点手势的装置、用户接口和方法
CN105279795B (zh) 基于3d标记的增强现实系统
TW201814438A (zh) 基於虛擬實境場景的輸入方法及裝置
US20140092013A1 (en) Video-based image control system
CN104364733A (zh) 注视位置检测装置、注视位置检测方法和注视位置检测程序
CN110456905A (zh) 定位跟踪方法、装置、系统及电子设备
WO2022174594A1 (zh) 基于多相机的裸手追踪显示方法、装置及系统
TWI825004B (zh) 輸入方法、裝置、設備、系統和電腦儲存媒體
CN105929946B (zh) 一种基于虚拟界面的自然交互方法
Zhang Applications of Google MediaPipe Pose Estimation Using a Single Camera
KR20230055352A (ko) 컨텐츠 제공 및 사용자 평가 방법, 그리고 컨텐츠 제공 및 사용자 평가 시스템
US20220365655A1 (en) Virtual Keyboard Interaction Method and System
Koutlemanis et al. A steerable multitouch display for surface computing and its evaluation
US20230030260A1 (en) Systems and methods for improved player interaction using augmented reality
Nai et al. Learning End-User Customized Mid-Air Hand Gestures Using a Depth Image Sensor
WO2016057997A1 (en) Support based 3d navigation
Zhu et al. Gesture recognition and application research of kinect
CN117122890A (zh) 用于激活屏幕内容的选择性导航或放大的方法和系统
van Wezel Gesture-based interaction concepts for mobile augmented reality applications
Comite Computer Vision and Human-Computer Interaction: artificial vision techniques and use cases with creating interfaces and interaction models
Derry Evaluating Head Gestures for Panning 2-D Spatial Information

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 21941621

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE