WO2024043765A1

WO2024043765A1 - Method and device for displaying virtual keyboard on head mounted display (hmd) device

Info

Publication number: WO2024043765A1
Application number: PCT/KR2023/012684
Authority: WO
Inventors: 배종혁; 고태헌; 문춘경; 엄홍열; 이승준; 한훈
Original assignee: 삼성전자 주식회사
Priority date: 2022-08-25
Filing date: 2023-08-25
Publication date: 2024-02-29

Abstract

A method of a head mounted display (HMD) device displaying a virtual keyboard, according to an embodiment of the present disclosure, may comprise a step for identifying selection of a keyboard input area. A method of a head mounted display (HMD) device displaying a virtual keyboard, according to an embodiment of the present disclosure, may comprise a step for identifying hands in the field of view (FOV) of a display on the basis of at least one sensor. A method of a head mounted display (HMD) device displaying a virtual keyboard, according to an embodiment of the present disclosure, may comprise a step for displaying the virtual keyboard on the display on the basis of the identification. In a method of a head mounted display (HMD) device displaying a virtual keyboard, according to an embodiment of the present disclosure, if hands are identified in the FOV of the display, the virtual keyboard may be displayed on the basis of a location of the identified hands. In a method of a head mounted display (HMD) device displaying a virtual keyboard, according to an embodiment of the present disclosure, if hands are not identified in the FOV of the display, the virtual keyboard may be displayed on the basis of a location of the selected keyboard input area.

Description

Method and device for displaying virtual keyboard on HMD (HEAD MOUNTED DISPLAY) device

With the development of augmented reality or virtual reality technology, HMD devices are widely used as a tool to provide augmented reality or virtual reality experiences. Due to the limited space of HMD devices, various user interfaces that can replace the physical user interface are applied, a representative example being a virtual keyboard.

A virtual keyboard is a concept distinct from a physical keyboard and refers to a virtual keyboard implemented through software. In cases where it is difficult to connect and use a physical keyboard, such as an HMD device, a virtual keyboard can be a useful user input method.

A method of displaying a virtual keyboard by a head mounted display (HMD) device according to an embodiment of the present disclosure may include identifying a keyboard input area selection. A method of displaying a virtual keyboard by a head mounted display (HMD) device according to an embodiment of the present disclosure may include identifying a hand in the field of view (FOV) of the display based on at least one sensor. there is. A method of displaying a virtual keyboard by a head mounted display (HMD) device according to an embodiment of the present disclosure may include displaying a virtual keyboard on a display based on identification. A method for a head mounted display (HMD) device to display a virtual keyboard according to an embodiment of the present disclosure may display a virtual keyboard based on the position of the identified hand when a hand is identified in the FOV of the display. A method for an HMD (head mounted display) device to display a virtual keyboard according to an embodiment of the present disclosure may display a virtual keyboard based on the location of the selected keyboard input area when a hand is not identified in the FOV of the display. there is.

A head mounted display (HMD) device that displays a virtual keyboard according to an embodiment of the present disclosure includes a display; Ministry of Communications; a storage unit that stores a program including at least one instruction; and at least one processor executing at least one instruction stored in the storage unit. At least one processor may identify the keyboard input area selection by executing at least one instruction. At least one processor may identify a hand in a field of view (FOV) of a display based on at least one sensor by executing at least one instruction. At least one processor may display a virtual keyboard on the display based on the identification by executing at least one instruction. When a hand is identified in the FOV of the display by executing at least one instruction, the at least one processor may display a virtual keyboard based on the location of the identified hand. At least one processor may display a virtual keyboard based on the location of the selected keyboard input area when a hand is not identified in the FOV of the display by executing at least one instruction.

Meanwhile, according to one embodiment of the present disclosure, a computer-readable recording medium on which a program for executing the above-described method is recorded is provided.

1A and 1B are diagrams illustrating examples of electronic devices according to an embodiment of the present disclosure.

Figure 2 shows a block diagram of an electronic device according to an embodiment of the present disclosure.

FIG. 3 is a diagram illustrating a method by which an electronic device identifies a plane and determines a virtual input area according to an embodiment of the present disclosure.

FIGS. 4A and 4B are diagrams for explaining a method of displaying a virtual keyboard on the display of an electronic device and interacting with a user, according to an embodiment of the present disclosure.

FIGS. 5A and 5B are diagrams for explaining a method of displaying a virtual keyboard on the display of an electronic device and interacting with a user, according to an embodiment of the present disclosure.

FIGS. 6A and 6B are diagrams illustrating a method by which an electronic device displays a virtual keyboard and interacts with a user according to an embodiment of the present disclosure.

Figure 7 is a flowchart of a method for an electronic device to display a virtual keyboard according to an embodiment of the present disclosure.

FIG. 8 is a diagram illustrating a method by which an electronic device provides a virtual keyboard based on the position and angle of the hand, according to an embodiment of the present disclosure.

FIG. 9 is a diagram illustrating a method by which an electronic device provides a virtual keyboard based on the position and angle of the hand, according to an embodiment of the present disclosure.

FIG. 10 is a diagram illustrating a method for an electronic device to determine a virtual input area according to an embodiment of the present disclosure.

FIG. 11 is a diagram illustrating a method by which an electronic device displays a virtual keyboard based on the surrounding environment according to an embodiment of the present disclosure.

FIG. 12 is a diagram illustrating a method for an electronic device to display feedback about typing according to an embodiment of the present disclosure.

FIG. 13 is a diagram illustrating a method by which an electronic device determines an input method based on the position of a hand, according to an embodiment of the present disclosure.

Below, with reference to the attached drawings, embodiments of the present disclosure will be described in detail so that those skilled in the art can easily implement the present disclosure. In describing the embodiments, description of technical content that is well known in the technical field to which this disclosure belongs and that is not directly related to this disclosure will be omitted. This is to convey the gist of the present disclosure more clearly without obscuring it by omitting unnecessary explanation. For the same reason, some components are exaggerated, omitted, or schematically shown in the accompanying drawings. Additionally, the size of each component does not entirely reflect its actual size.

However, the present disclosure may be implemented in many different forms and is not limited to the embodiments described herein. In order to clearly explain the present disclosure in the drawings, parts that are not related to the description are omitted, and similar parts are given similar reference numerals throughout the specification.

The advantages and features of the present disclosure and methods for achieving them will become clear by referring to the embodiments described in detail below along with the accompanying drawings. However, the present disclosure is not limited to the embodiments disclosed below and may be implemented in various different forms. The present embodiments are merely intended to ensure that the present disclosure is complete, and those of ordinary skill in the art to which the present disclosure pertains are not limited to the present disclosure. It is provided to fully inform the scope of the present disclosure, and the present disclosure is defined only by the scope of the claims.

At this time, it will be understood that each block of the processing flow diagram diagrams and combinations of the flow diagram diagrams can be performed by computer program instructions. These computer program instructions can be mounted on a processor of a general-purpose computer, special-purpose computer, or other programmable data processing equipment, so that the instructions performed through the processor of the computer or other programmable data processing equipment are described in the flow chart block(s). It creates the means to perform functions. These computer program instructions may also be stored in computer-usable or computer-readable memory that can be directed to a computer or other programmable data processing equipment to implement a function in a particular manner, so that the computer-usable or computer-readable memory It is also possible to produce manufactured items containing instruction means that perform the functions described in the flowchart block(s). Computer program instructions can also be mounted on a computer or other programmable data processing equipment, so that a series of operational steps are performed on the computer or other programmable data processing equipment to create a process that is executed by the computer, thereby generating a process that is executed by the computer or other programmable data processing equipment. Instructions that perform processing equipment may also provide steps for executing the functions described in the flow diagram block(s).

Additionally, each block may represent a module, segment, or portion of code that includes one or more executable instructions for executing specified logical function(s). Additionally, it should be noted that in some alternative execution examples it is possible for the functions mentioned in the blocks to occur out of order. For example, it is possible for two blocks shown in succession to be performed substantially at the same time, or it is possible for the blocks to be performed in reverse order depending on the corresponding function.

At this time, the term '~unit' used in this embodiment refers to software or hardware components such as FPGA or ASIC, and the '~unit' performs certain roles. However, '~part' is not limited to software or hardware. The '~ part' may be configured to reside in an addressable storage medium and may be configured to reproduce on one or more processors. Therefore, as an example, '~ part' refers to components such as software components, object-oriented software components, class components, and task components, processes, functions, properties, and procedures. , subroutines, segments of program code, drivers, firmware, microcode, circuitry, data, databases, data structures, tables, arrays, and variables. The functions provided within the components and 'parts' may be combined into a smaller number of components and 'parts' or may be further separated into additional components and 'parts'. Additionally, components and 'parts' may be implemented to regenerate one or more CPUs within a device or a secure multimedia card. Also, in the embodiment, ‘~ part’ may include one or more processors.

In the present disclosure, augmented reality or AR (Augmented Reality) means showing a virtual image together in a physical environment space of the real world or showing a real object and a virtual object image together. In AR-dedicated electronic devices such as AR glasses, real objects shown through a see-through display and virtual object images displayed on the device's display are shown together. In mobile devices, real objects and virtual objects captured by a camera are displayed together on the display. Unlike virtual reality (VR), where images, backgrounds, surroundings, and objects are all virtual images, in augmented reality, the background, surroundings, and objects are real objects, and only the added information is virtual. Augmented reality can provide additional information while further increasing the effect of reality, so it can be applied in various industrial fields.

In this disclosure, ‘Virtual Reality (VR)’ means showing a virtual image as if it were real.

In addition, 'Virtual Reality Device or VR device' refers to a device that can express 'virtual reality', and is not only a virtual reality device in the form of glasses that the user generally wears on the face, It includes Head Mounted Display Apparatus (HMD) and Virtual Reality Helmet worn on the head.

When using a virtual keyboard, it is difficult to know the position and alignment of the user's hands while looking at the screen because virtual graphics are projected onto a plane that is difficult for the user to physically recognize. Due to this, the user must input while looking directly at the virtual keyboard. However, due to the nature of the HMD device, where the virtual screen is placed in front of the user, the user must look straight down to view the virtual keyboard. Due to the nature of the HMD device, the center of gravity is at the user's face. If the movement of the field of view is repeated for a long period of time, it can cause fatigue and dizziness, and looking down while wearing the HMD device can increase the burden on the user's neck.

Therefore, there is a need for technology that allows the virtual keyboard to be used while facing the front.

The electronic device 1000 according to an embodiment of the present disclosure may be an augmented reality device or a virtual reality device, but is not limited thereto, and is a head mounted display (HMD) device that provides a virtual keyboard and allows text input using the virtual keyboard. It can include any device that performs the acquisition function.

Referring to FIG. 1A , the electronic device 1000 is a glasses-type display device and may include a glasses-type body configured to be worn by a user.

The eyeglass-shaped body may include a frame (or rim) 110 and a support portion 190, and the support portion 190 extends from the frame 110 and may be used to seat the augmented reality device on the user's head. . The support portion 190 may include

temples

190L and 190R and a nose support portion (not shown). The

temples

190L and 190R extend from the frame 110 and may be used to secure the electronic device 1000 to the user's head at the side portion of the glasses-shaped body. The nose support unit (not shown) extends from the frame 110 and may be used to seat the electronic device 1000 on the user's nose, and may include, for example, the bridge of the nose and the nose of glasses, but is not limited thereto. No.

Additionally, a lens unit 1350 and a wave guide 1320 may be located in the frame 110. The lens unit 1350 may include a lens unit 1350L for the left eye and a lens unit 1350R for the right eye. Additionally, the wave guide 1320 may be configured to receive projected light from an input area and output at least a portion of the input light from an output area. This wave guide 1320 may include a wave guide 1320L for the left eye and a wave guide 1320R for the right eye.

The left eye lens unit (1350L) and the left eye wave guide (1320L) can be placed in a position corresponding to the user's left eye, and the right eye lens unit (1350R) and the right eye wave guide (1320R) can be placed in a position corresponding to the user's right eye. It can be placed in any location. For example, the left eye lens unit 1350L and the left eye wave guide 1320L may be attached to each other, or the right eye lens unit 1350R and the right eye wave guide 1320R may be attached to each other, but are not limited thereto. .

Additionally, the optical engine 1310 of the projector that projects display light containing an image may include an optical engine 1310L for the left eye and an optical engine 1310R for the right eye. The left eye optical engine 1310L and the right eye optical engine 1310R may be located on both sides of the electronic device 1000. Alternatively, one optical engine 1310 may be included in the central portion around the nose support 192 of the electronic device 1000. Light emitted from the optical engine 120 may be displayed through the wave guide 1320.

A user interface (UI) for interaction between the HMD display device and the user may include the user's gaze or the user's motion (gesture) information. Such information may be obtained based on sensing results obtained by sensors included in the electronic device 1000.

Referring to FIG. 1B, a sensor unit 1100 capable of sensing the external environment or the user's movement may be disposed on the front of the electronic device 1000. The sensor unit 1100 may include a distance sensor 1110, an image sensor 1120, a gyro sensor 1130, and an acceleration sensor 1140.

The distance sensor 1110 may include at least one of an ultrasonic sensor, an infrared sensor, a LIDAR (LIght Detection And Ranging) sensor, or a RADAR (Radio Detection And Ranging) sensor.

The ultrasonic sensor measures the distance based on the time taken from transmitting ultrasonic waves to receiving reflected waves or the wavelength of the received reflected waves. Infrared sensors use electromagnetic waves with longer wavelengths than visible light, and infrared rays are safe because they have long wavelengths and low energy. A short-distance measurement sensor using infrared rays can measure distance according to the intensity of the reflected infrared rays by radiating infrared rays from the sensor. Infrared sensors are also called ToF (time of flight) sensors because they calculate distance by measuring the time it takes for light to reflect and return to the subject. The light emitted from the sensor reflects and returns at different times depending on the curvature of the subject, and this time difference can be used to measure the distance to the subject, 3D scan, or height measurement.

The LiDAR sensor has a narrow detection range, but uses a laser with strong straight-line motion, so there is a low probability of distortion occurring, so it has higher precision than cameras or radars, and the error range is only in the mm unit. In addition, LIDAR has a higher horizontal azimuth than cameras or radars, so it can accurately measure altitude and direction, has an excellent ability to detect small objects, and can detect temperature or material distribution. A radar sensor can determine the existence, direction, distance, and speed of an object by measuring the reflected wave that returns after the irradiated electromagnetic waves hit the object. Since it uses radio waves, it has low sensitivity to the surrounding environment, enabling stable measurements.

The image sensor 1120 typically has a camera and can photograph an object to identify its presence, location, and size. The image sensor 1120 can obtain an image frame, such as a still image or video, when an application requiring a shooting function is executed. The image captured through the image sensor 1120 may be processed through the processor 1800 or a separate image processor (not shown). The image sensor 1120 may include, for example, at least one of a rotatable RGB camera module or a plurality of depth camera modules, but is not limited thereto.

The electronic device 1000 may implement a VST (video see-through) function by displaying (playing) an external image captured using the image sensor 1120 on an internal display in real time. Unlike OST (optical see-through), which displays a virtual image projected on a display (lens) over the external environment directly perceived by the user's eyes through a see-through display, VST displays images captured through a camera (or vision sensor). A display method that displays images overlapping virtual content may also be referred to as pass through. The VST function can be an effective means of recognizing the external environment in a blackout HMD device where the external view is blocked. The image sensor 1120 may be composed of a plurality of image sensors.

The gyro sensor 1130 is a sensor that measures angular velocity and refers to a gyroscope implemented in chip form by applying MEMS technology. The gyro sensor 1130 can detect the rotation and rotation speed of the electronic device 1000.

The acceleration sensor 1140 is a sensor that measures dynamic force, such as a change in speed (acceleration) or a change in momentum (impact), and can detect the motion state of the electronic device 1000 in detail.

The gyro sensor 1130 and the acceleration sensor 1140 may detect movement of the electronic device 1000 or movement of the user.

More accurate sensing by using information about the movement of the object sensed using the distance sensor 1110 and the image sensor 1120 and the movement of the electronic device 1000 sensed using the gyro sensor and acceleration sensor 1140. results can be obtained.

Referring to FIG. 2, the electronic device 1000 according to an embodiment of the present disclosure includes a sensor unit 1100, an input unit 1200, a display unit 1300, an output unit 1400, a communication interface 1600, and a storage unit. It may include a unit 1700 and a processor 1800.

The sensor unit 1100 can sense the environment around the electronic device 1000. The sensor unit 1100 includes a distance sensor 1110, an image sensor 1120, a gyro sensor 1130, an acceleration sensor 1140, and an eye tracking sensor 1150, as well as a temperature sensor, a gyro sensor, and a distance sensor (LADAR, LIDAR). ), an infrared sensor, and a UWB sensor. Since the distance sensor 1110, the image sensor 1120, the gyro sensor 1130, and the acceleration sensor 1140 have been described above with reference to FIG. 1B, detailed descriptions will be omitted.

The gaze tracking sensor 1150 can detect and track the gaze of the user wearing the electronic device 1000. The gaze tracking sensor 1150 may be installed in a direction toward the user's eyes and can detect the gaze direction of the user's left eye and the gaze direction of the user's right eye. Detecting the user's gaze direction may include obtaining gaze information related to the user's gaze.

The input unit 1200 refers to a means for a user to input data to control the electronic device 1000, and may include a microphone 1210 and a button unit 1220.

The microphone 1210 receives external acoustic signals and processes them into electrical voice data. For example, the microphone 1210 may receive an acoustic signal from an external device or a speaker. Various noise removal algorithms can be used to remove noise generated in the process of receiving an external acoustic signal through the microphone 1210. The microphone 1210 may receive a user's voice input for controlling the electronic device 1000.

The button unit 1220 includes a key pad, a dome switch, and a touch pad (contact capacitance type, pressure resistance membrane type, infrared detection type, surface ultrasonic conduction type, integral tension measurement type, It may include, but is not limited to, at least one of a piezo effect type, etc.), a jog wheel, or a jog switch.

The display unit 1300 displays and outputs information processed by the electronic device 1000. For example, the display unit 1300 may display a user interface for photographing the surroundings of the electronic device 1000 and information related to services provided based on the captured images of the surroundings of the electronic device 1000.

When the display unit 1300 and the touch pad form a layered structure to form a touch screen, the display unit 1300 can also be used as an input device. The display unit 1300 includes a liquid crystal display unit, a thin film transistor-liquid crystal display unit, an organic light-emitting diode, a flexible display unit, 3 It may include at least one of a 3D display unit and an electrophoretic display unit.

According to one embodiment, the display unit 1300 may provide AR images and VR images. As shown in FIG. 1A, the display unit 1300 may include an optical engine 1380. When the electronic device 1000 is a glasses-type device, the display unit 1300 may include a left display unit and a right display unit.

The output unit 1400 is for outputting audio signals or haptic signals.

The speaker 1410 may output audio data received from the communication interface 1600 or stored in the storage unit 1700. A plurality of speakers may output three-dimensional sound signals. For example, the plurality of speakers may be implemented as at least one of a right channel speaker, a left channel speaker, and a mono channel speaker. Additionally, the plurality of speakers may output sound signals related to functions performed by the electronic device 1000 (eg, notification sound, guidance voice, guide voice).

The haptic module 1420 may output haptic signals related to functions performed in the electronic device 1000. Haptic signals can be distinguished based on intensity or pattern.

The communication interface 1600 can transmit and receive data for providing services related to the electronic device 1000 with an external device (not shown) and a server (not shown). The communication interface 1600 may include one or more components that enable communication between the electronic device 1000 and a server device (not shown), and between the electronic device 1000 and an external device (not shown). For example, the communication interface 1600 may include a short-range communication unit and a broadcast reception unit.

The short-range wireless communication unit includes a Bluetooth communication unit, BLE (Bluetooth Low Energy) communication unit, Near Field Communication unit, WLAN (Wi-Fi) communication unit, Zigbee communication unit, and infrared (IrDA) communication unit. Data Association) communication department, WFD (Wi-Fi Direct) communication department, UWB (ultra wideband) communication department, Ant+ communication department, etc., but is not limited thereto.

The broadcast receiver receives broadcast signals and/or broadcast-related information from the outside through a broadcast channel. Broadcast channels may include satellite channels and terrestrial channels. Depending on the implementation example, the electronic device 1000 may not include a broadcast reception unit.

The communication interface 1600 can obtain content from an external device (not shown). The communication interface 1600 can acquire content from an external device (not shown) through wired or wireless communication. Here, external devices (not shown) include server devices, mobile terminals, wearable devices (e.g., watches, bands, glasses, masks, etc.), home appliances (e.g., TVs, desktop PCs, laptops, DVD devices, washing machines, refrigerators, etc.) ), etc., but is not limited thereto. Content may include multimedia files, video files, and audio files.

The storage unit 1700 can store programs to be executed by the processor 1800, which will be described later, and can store data input to or output from the electronic device 1000.

The storage unit 1700 may include at least one of internal memory (not shown) and external memory (not shown). Built-in memory includes, for example, volatile memory (e.g., DRAM (Dynamic RAM), SRAM (Static RAM), SDRAM (Synchronous Dynamic RAM), etc.), non-volatile memory (e.g., OTPROM (One Time Programmable ROM), etc. ), PROM (Programmable ROM), EPROM (Erasable and Programmable ROM), EEPROM (Electrically Erasable and Programmable ROM), Mask ROM, Flash ROM, etc.), hard disk drive (HDD), or solid state drive (SSD). It can be included. According to one embodiment, the processor 1800 may load commands or data received from at least one of the non-volatile memory or other components into the volatile memory and process them. Additionally, the processor 1800 may store data received or generated from other components in non-volatile memory. External memory includes, for example, at least one of CF (Compact Flash), SD (Secure Digital), Micro-SD (Micro Secure Digital), Mini-SD (Mini Secure Digital), xD (extreme Digital), and Memory Stick. It can be included.

Programs stored in the storage unit 1700 may be classified into a plurality of modules according to their functions, and may include, for example, an identification module 1710 and an image processing module 1720.

The processor 1800 controls the overall operation of the electronic device 1000. For example, the processor 1800 executes programs stored in the storage unit 1700, thereby controlling the sensor unit 1100, the input unit 1200, the display unit 1300, the output unit 1400, and the communication interface 1600. and the storage unit 1700, etc. can be controlled overall.

The processor 1800 can display a virtual keyboard and obtain virtual keyboard input by executing the identification module 1710 and the image processing module 1720 stored in the storage unit 1700.

According to one embodiment, the electronic device 1000 may include a plurality of processors 1800, and the identification module 1710 and the image processing module 1720 may be executed by the plurality of processors 1800.

The processor 1800 can identify the front plane of the electronic device 1000, identify the hand within the FOV of the display, and determine the virtual input area by executing the identification module 1710 stored in the storage unit 1700. .

The processor 1800 displays a hand placement guide in the virtual input area by executing the image processing module 1720 stored in the storage unit 1700, and when a hand is identified within the FOV of the display, the identified hand is displayed as a VST. and the virtual keyboard can be displayed. According to one embodiment of the present disclosure, the virtual keyboard module 1723 can determine and display a specific key of the virtual keyboard or the shape, color, angle, and position of the virtual keyboard.

According to an embodiment of the present disclosure, when the hand placement guide is not an image, the hand placement guide module 1721 may be configured as a separate module from the image processing module 1720.

In the electronic device 1000 that does not include a physical keyboard or is not connected to a physical keyboard, a situation may occur in which text input through the keyboard is required. The electronic device 1000 according to an embodiment of the present disclosure may provide a virtual keyboard to interact with the user in a situation where keyboard input is required, for example, when a keyboard input area is selected while an application is running. In this specification, a situation requiring keyboard input may be referred to as a keyboard input mode.

Referring to FIG. 3, the electronic device 1000 according to an embodiment of the present disclosure may obtain spatial information around the electronic device 1000 using the distance sensor 1110 or the image sensor 1120. Spatial information around the electronic device 1000 may include information on the presence, shape, distance, location, and movement of nearby objects obtained using the image sensor 1120 or the distance sensor 1110.

Based on the acquired spatial information, the electronic device 1000 can identify the plane 3100 on the front of the electronic device 1000 and obtain information about the identified plane 3100. The front plane 3100 of the electronic device 1000 may be identified based on object recognition of the captured front image. Information about the plane may include information about the existence, location, size, and shape of the plane.

The electronic device 1000 may determine a virtual input area 3200 where the user's hand will be located within the plane 3100 based on information about the plane 3100. The virtual input area 3200 may be determined to have a predetermined margin (3400) from the edge of the identified plane (3100), and the virtual input area (3200) and the margin (3400) are of the virtual keyboard to be placed. It may be determined based on attributes (e.g., shape, size, type, etc.). The free space 3400 may be determined by considering the space in the plane space where the user's hand can be comfortably positioned, and can be determined by most users based on demographic information (e.g., average body size, joint angle, etc.). can be determined as a distance that can be used comfortably.

Once the virtual input area 3200 is determined, the electronic device 1000 may output an affordance (action inducer) that induces the user to place his or her hand on the virtual input area 3200. For example, the electronic device 1000 may output a text guide and/or a hand-shaped graphic guide 3300 on the display 1300 instructing to place a hand on the determined virtual input area 3200. Alternatively, the electronic device 1000 may output an audio guide instructing to place a hand on the virtual input area 3200 to the speaker 1410.

The field of view (FOV) of a sensor refers to the area where sensing is possible, and is generally defined as an angle. Mechanical sensors can rotate 360 degrees and have a field of view of 360 degrees, and fixed sensors can expand their field of view by arranging multiple sensors and combining the sensing results of each sensor. In this specification, a sensor is a means of identifying an external object and obtaining information, and is a distance sensor 1110 included in the sensor unit 1100 or a plurality of distance sensors constituting the distance sensor 1100, an image sensor ( 1120) or a plurality of image sensors constituting the image sensor 1120, or a combination thereof.

The FOV of the display refers to the viewing angle of the display that can be viewed at once. Glasses-type AR displays have an FOV of approximately 50 degrees, and blackout-type VR displays have an FOV of approximately 110 degrees. Therefore, the FOV of the sensor has a wider range than the FOV of the display.

Referring to FIG. 4A, the user is wearing the electronic device 1000, which is an HMD device, and looking at the plane where the hand, which is a virtual keyboard input means, is placed. Accordingly, it is assumed that the user's hand 1000 is located within the FOV of the sensor and the FOV of the display 1300.

According to one embodiment of the present disclosure, the user's hands are identified in the FOV of the sensor in a keyboard input mode (e.g., the user selects a keyboard input area of an application), and the user's hands are within the FOV of the display 1300. In this case, the electronic device 1000 switches the display to VST mode (or referred to as pass through mode) and displays a virtual display on the user's hand 2000 indicated as VST in the FOV of the display 1300 looking at the user's hand. It can be displayed by matching the keyboard 3000. At this time, the electronic device 1000 arranges and displays the keyboard input area 3500 around the virtual keyboard 3000 so that the user can check whether the keyboard input is performed properly. The keyboard input area 3500 according to an embodiment of the present disclosure is locked to the virtual keyboard 3000, so that when the position of the virtual keyboard 3000 changes, the position of the keyboard input area 3500 will also change accordingly. You can.

Referring to FIG. 4B, the display 1300 of the electronic device 1000 may include an application display area 3600 and a virtual keyboard display area, and the application display area 3600 may include an application and keyboard input area 3500. A virtual keyboard 3000 and a hand 2000 may be displayed in the virtual keyboard display area. According to one embodiment of the present disclosure, the keyboard input area 3500 may be displayed in a separate area other than the application display area 3600.

When the electronic device 1000 according to an embodiment of the present disclosure detects activation of the keyboard input area 3500 or selection of the keyboard input area 3500, it displays a virtual keyboard for text input and uses the displayed virtual keyboard. Through this, you can interact with the user and obtain text input. The electronic device 1000 may display a keyboard input area 3500 around the virtual keyboard 3000 to allow the user to check whether the text corresponding to the key being input is properly entered.

When activation of the keyboard input area 3500 is detected while running an application, the electronic device 1000 uses a sensor to identify whether a plane space 3100 exists within the FOV of the sensor, and the virtual device 1000 determined within the identified plane space A guide 3300 for positioning the user's hand in the input area 3200 may be output. When the user's hand 2000 is detected in the flat space 3100 within the FOV of the sensor and the detected hand is located within the FOV of the display 1300, the electronic device 1000 creates a virtual keyboard placement area on the display 1300. The virtual keyboard 3000 and the user's hand 2000 can be displayed in the determined virtual keyboard placement area.

If the electronic device 1000 according to an embodiment of the present disclosure is a device composed of a transparent display such as AR glass, the user can directly see his or her hand located within the FOV of the electronic device 1000 through the transparent display, so that the user The step of displaying the hand 2000 on the display may be omitted. When the electronic device 1000 according to an embodiment of the present disclosure is a blackout HMD device, the electronic device 1000 acquires an image of the user's hand 2000 using a camera module 1130 that photographs the outside, The acquired image can be displayed in VST mode.

As shown in FIG. 4B, the electronic device 1000 displays a virtual keyboard 3000 and a hand 2000 on the display 1300, and an application display area 3600 for displaying applications requiring text input and a keyboard input are provided around the virtual keyboard 3000. The area 3500 can be arranged and displayed.

5A and 5B are diagrams for explaining a method of displaying a virtual keyboard on the display of an electronic device and interacting with a user, according to an embodiment of the present disclosure.

In FIGS. 5A and 5B , descriptions similar to or overlapping with FIGS. 4A and 4B may be briefly described or omitted.

In the case of an HMD device, the load due to its own weight can be a burden on the user's neck, and the effect is even greater when the user bends his or her neck to look down. When using a physical keyboard, most users often use the keyboard while looking at the display in front of them rather than looking at the keyboard. Therefore, it may be more convenient for the user to look straight ahead even when using the virtual keyboard, and the user experience of the HMD device may be improved.

Referring to FIG. 5A, while wearing the electronic device 1000, which is an HMD device, the user is looking straight ahead rather than at the plane where the hands are to input the virtual keyboard. Accordingly, it is assumed that the user's hand 1000 is not located within the FOV of the display 1300.

As described above, since the FOV of the sensor has a wider range than the FOV of the display, the sensor can sense the user's hand 1000 even if the user's hand 1000 is not located within the FOV of the display 1300.

According to one embodiment of the present disclosure, the user's hands are identified in the sensor's FOV in a keyboard input mode (e.g., the user selects a keyboard input area of an application) and the user's hands are not within the FOV of the display 1300. If not, the electronic device 1000 may display the virtual keyboard 3000 in the FOV of the display 1300 facing the front. At this time, since the user's hand 2000 is not displayed in the FOV of the display 1300, the electronic device 1000 may display the virtual keyboard 3000 around the keyboard input area 3500. The virtual keyboard 3000 according to an embodiment of the present disclosure is locked to the keyboard input area 3500, and when the position of the keyboard input area 3500 changes, the position of the virtual keyboard 3000 may also change accordingly. You can.

Referring to FIG. 5B, the display 1300 of the electronic device 1000 may include an application display area 3600 and a virtual keyboard display area. An application and keyboard input area 3500 may be displayed in the application display area 3600, and a virtual keyboard 3000 may be displayed in the virtual keyboard display area. According to one embodiment of the present disclosure, the keyboard input area 3500 may be displayed in a separate area other than the application display area 3600.

Assume that a message is received while playing a video in the electronic device 1000, and the message is displayed as a pop-up in the messenger application (3600). The user can select the keyboard input area 3500 for conversation with the other party, and the electronic device 1000, which has identified that the keyboard input area 3500 has been selected, can display the virtual keyboard 3000 for text input. You can interact with the user and obtain text input through the displayed virtual keyboard. The electronic device 1000 may display a virtual keyboard 3000 around the keyboard input area 3500 to allow the user to check whether the text corresponding to the input key is properly entered.

More specifically, when activation of the keyboard input area 3500 is detected while the application is running, the electronic device 1000 uses a sensor to identify whether a flat space 3100 exists within the sensor's FOV and enters the identified space. A guide 3300 to position the user's hand can be output. When the user's hand 2000 is detected in the flat space 3100 within the FOV of the sensor and the detected hand is located outside the FOV of the display 1300, the electronic device 1000 creates a virtual keyboard placement area on the display 1300. can be determined and the virtual keyboard 3000 can be displayed in the determined virtual keyboard placement area.

At this time, the user can input text by typing on the front plane 3100 with his or her fingers. Therefore, since the (flat) spatial area with which the user interacts and the spatial area of the virtual keyboard displayed on the display 1300 are different from each other, in this case, the virtual keyboard 3000 displayed on the display 1300 may be referred to as a keyboard navigator. .

The keyboard navigator is a graphical user interface (GUI) that helps recognize the user's hand position or finger movement (tap) that is not displayed on the display. Like a virtual keyboard, the keys corresponding to the position of each finger are displayed. By adding specific marks, the location of each finger can be specified. Alternatively, the electronic device 1000 may display a virtual hand image in the keyboard navigator on the display 1300 and display the displayed hand image in conjunction with the actual hand movement. For example, when the user is identified as having tapped the left index finger on the flat area 3100, the electronic device 1000 displays the virtual hand displayed on the keyboard navigator 3000 as having also tapped the left index finger or displays the virtual hand as tapping the left index finger or corresponding to the left index finger. You can mark the marker to change.

Referring to FIG. 6A, the electronic device 1000 can identify the user's hand 2000 using a sensor, and displays a virtual keyboard in the FOV of the display 1300 based on the position of the identified user's hand 2000. (3000) can be displayed.

The electronic device 1000 can arrange the virtual keyboard 3000 according to (by mapping) the position of the identified user's hand 2000. At this time, the electronic device 1000 may identify the tip of the finger capable of typing on the virtual keyboard 3000 and display the position on the keyboard corresponding to the finger differently from other parts. For example, the electronic device 1000 inserts a marker at the tip of the user's finger, or performs visual processing on the keys 2001 to 2010 corresponding to the position where the user's finger is predicted to be tapped on the virtual keyboard. You can.

The user's hand 2000 exists within the FOV of the display 1300 of the electronic device 1000, and the electronic device 1000 is an HMD device (e.g., augmented reality-AR-device) consisting of a display 1300 made of a transparent material. In this case, the user can directly see and recognize his or her hands through the transparent display. Accordingly, the electronic device 1000 can identify whether a hand is within the sensor's FOV and display the virtual keyboard 3000 based on the identified hand position.

The user's hand 2000 exists within the FOV of the display 1300 of the electronic device 1000, and the electronic device 1000 is a blackout HMD device (e.g., a virtual HMD device) in which light from the outside is blocked for the user's immersive experience. In the case of a reality-VR-device, the user cannot directly see his or her hands. Accordingly, the electronic device 1000 displays the image of the hand acquired through the camera module 1130 that photographs the outside as VST (video see-through) on the display, and operates the virtual keyboard 3000 based on the position of the hand. It can be displayed.

When the user's hand 2000 is located outside the FOV of the display 1300 of the electronic device 1000, the electronic device 1000 displays the virtual keyboard 3000 on the display 1300 and uses the virtual hand image as the virtual keyboard. It can be displayed overlapping with . Alternatively, when the user's hand 2000 is located outside the FOV of the display 1300 of the electronic device 1000, the electronic device 1000 may display only the virtual keyboard 3000 on the display 1300.

Referring to FIG. 6B, when the user's finger is identified as tapping the keyboard arrangement area 3200, the electronic device 1000 may determine that a key corresponding to the finger has been input. Additionally, the electronic device 1000 may display keys determined to have been entered with greater emphasis.

For example, when trying to input 'QWERTY' in Figure 6b, when the user taps the keyboard layout area 3200 in order starting from the left little finger 2001, the fingers corresponding to the user's

tap

2001, 2002, 2003, 2004, 2005, and 2006 are displayed in bold in that order, and 'Q', 'W', 'E', 'R', 'T', and 'Y' are displayed in the text input window 3300 of the display 1300. They are displayed in order.

In the description of FIG. 7, parts similar to or overlapping with FIGS. 3 to 6B may be briefly described or omitted.

In step S701, the electronic device 1000 may identify that the keyboard input area 3500 is selected.

When the keyboard input area 3500 is selected, the electronic device 1000 according to an embodiment of the present disclosure may identify a plane on which to place the user's hands for inputting the virtual keyboard 3000. The electronic device 1000 may obtain information about the plane 3100 within the sensing area, that is, within the sensor FOV, using at least one sensor. Information about the plane may include information about the existence, location, size, and shape of the plane.

If a plane exists within the sensor FOV, the electronic device 1000 according to an embodiment of the present disclosure may determine a virtual input area 3200 in which to place the user's hand on the identified plane. The virtual input area 3200 may be determined to have a predetermined margin (3400) from the edge of the identified plane (3100), and the virtual input area (3200) and the margin (3400) are of the virtual keyboard to be placed. It may be determined based on attributes (e.g., shape, size, type, etc.). The free space 3400 may be determined by considering the space in the plane space where the user's hand can be comfortably positioned, and can be determined by most users based on demographic information (e.g., average body size, joint angle, etc.). can be determined as a distance that can be used comfortably.

According to an embodiment of the present disclosure, even if a plane is not identified within the sensing area or a plane is identified, the virtual keyboard cannot be input if the hand is not identified in the identified plane, so the electronic device 1000 performs a virtual keyboard display procedure. can be terminated.

In step S702, the electronic device 1000 may determine whether the hand 2000 is identified in the FOV of the display 1300 based on at least one sensor.

Generally, the sensor's FOV has a wider range than the display's FOV. When the user wearing the electronic device 1000, which is an HMD device, is looking in the direction of the hand, the hand 2000 can be identified in the FOV of the display 1300, and the user wearing the electronic device 1000, which is an HMD device, can be identified. When looking straight ahead, the hand 2000 may not be identified in the FOV of the display 1300.

In step S703, if the hand 2000 is identified in the FOV of the display 1300 as a result of the determination in step S702, the electronic device 1000 may display the virtual keyboard 3000 based on the position of the identified hand 2000. there is.

According to an embodiment of the present disclosure, when the user's hand is within the FOV of the display 1300, the electronic device 1000 switches the display to VST mode (or, referred to as pass through mode), and moves the user's hand to the FOV of the display 1300. The virtual keyboard 3000 can be displayed by matching it on the user's hand 2000 displayed as a VST in the FOV of the display 1300 being viewed. At this time, the electronic device 1000 arranges and displays the keyboard input area 3500 around the virtual keyboard 3000 so that the user can check whether the keyboard input is performed properly. The keyboard input area 3500 according to an embodiment of the present disclosure is locked to the virtual keyboard 3000, so that when the position of the virtual keyboard 3000 changes, the position of the keyboard input area 3500 will also change accordingly. You can.

If the electronic device 1000 according to an embodiment of the present disclosure is a device configured with a transparent display, the user can directly see his or her hand located within the FOV of the electronic device 1000 through the transparent display, so the user's hand ( The step of displaying 2000) on the display can be omitted.

In step S704, the electronic device 1000 displays the virtual keyboard 3000 based on the position of the keyboard input area 3500 when the hand 2000 is not identified in the FOV of the display 1300 as a result of the determination in step S702. You can.

As mentioned above, in the case of an HMD device, the load due to its own weight can be a burden on the user's neck, and the burden is even greater when the user bends his or her neck to look down. When using a physical keyboard, most users often use the keyboard while looking at the display in front of them rather than looking at the keyboard. Therefore, it may be more convenient for the user to look straight ahead even when using the virtual keyboard, and the user experience of the HMD device may be improved. When a user wearing the electronic device 1000, which is an HMD device, is looking straight ahead, the hand 2000 may not be identified in the FOV of the display 1300.

Even if the user's hand 1000 is not located within the FOV of the display 1300, the FOV of the sensor is wider than the FOV of the display, so even if the user's hand 1000 is not located within the FOV of the display 1300, the sensor detects the user's hand. can be sensed.

When the user's hand is not within the FOV of the display 1300, the electronic device 1000 may display the virtual keyboard 3000 in the FOV of the display 1300 facing the front. At this time, since the user's hand 2000 is not displayed in the FOV of the display 1300, the electronic device 1000 may display the virtual keyboard 3000 around the keyboard input area 3500.

The virtual keyboard 3000 according to an embodiment of the present disclosure is locked to the keyboard input area 3500, and when the position of the keyboard input area 3500 changes, the position of the virtual keyboard 3000 may also change accordingly. You can.

In step S705, the electronic device 1000 may obtain a virtual keyboard input based on the movement of the finger corresponding to the virtual keyboard 3000.

Sensors included in the electronic device 1000 can sense the movement of surrounding objects. Therefore, by sensing the movement of the user's finger (object) identified in the FOV of the sensor, a virtual keyboard input corresponding to the user's finger movement can be obtained.

Referring to FIG. 8, the electronic device 1000 may provide a separate virtual keyboard based on the position and angle of the user's hand. According to an embodiment of the present disclosure, the electronic device 1000 may determine whether to use a separate virtual keyboard based on user settings. A user may try to use a virtual keyboard depending on the type of physical keyboard he or she prefers. Accordingly, the electronic device 1000 may be configured to set in advance the type of virtual keyboard to be provided to the user.

According to an embodiment of the present disclosure, at least one sensor included in the sensor unit 1100 of the electronic device 1000 can identify the user's hand 2000 and provides information about the position, shape, and movement of the hand. Information can be obtained. The electronic device 1000 may enter or analyze the virtual input area 3200 by quantifying the angle of the hand at which it is positioned.

For example, the electronic device 1000 includes a 1-1 vector in the direction of the left index finger, a 1-2 vector in the direction of the right index finger, and a second vector in a direction horizontal to both hands (or the horizontal axis of the virtual input area). Based on this, the left virtual keyboard (3000-1) and the right virtual keyboard (3000-2) among the detachable keyboards can be displayed. At this time, the reference point of the left virtual keyboard (3000-1) may be the 'F' key where the left index finger related to the 1-1 vector is located, and the reference point of the right virtual keyboard (3000-2) may be the 1-1 vector. 2 This could be the 'J' key where the index finger of the right hand related to the vector will be located.

According to one embodiment of the present disclosure, the left virtual keyboard 3000-1 and the right virtual keyboard 3000-2 may be displayed on the user's left and right hands, respectively.

Referring to FIG. 9, the electronic device 1000 may determine the virtual keyboard display position based on the position and angle of the user's hand.

For example, the electronic device 1000 determines the angle of the virtual keyboard 3000 based on the third vector connecting the tip of the index finger of the left hand and the index finger of the right hand, and uses the virtual keyboard 3000 based on the determined angle. It can be displayed. At this time, the reference point of the virtual keyboard may be the 'F' key where the left index finger is located and the 'J' key where the right index finger is located.

Referring to FIG. 10 , the electronic device 1000 may determine the virtual input area 3200 based on the presence and location of other objects within the identified plane 3100.

For example, when the first object 4000-1 and the second object 4000-2 exist in the identified plane 3100, the electronic device 1000 provides information about the plane 3100 and information about the objects. The virtual input area 3200 can be determined based on the information. The virtual input area 3200 has a predetermined margin 3400 from the edge of the identified plane 3100, and additionally has a first margin 3400-1 from the first object 4000-1. It may be determined to have a second free space 3400-2 from the second object 4000-2.

Referring to FIG. 11, the electronic device 1000 may display the virtual keyboard 3000 based on the background on which the virtual keyboard 3000 is displayed.

For example, the electronic device 1000 acquires the background color of the identified plane 3100 using the image sensor 1120, or acquires the background color of the image displayed as the background of the virtual keyboard 1300 on the display 1300. It can be obtained. The electronic device 1000 can display the virtual keyboard 3000 in contrast with the obtained background color, so that the virtual keyboard 3000 appears more visually emphasized. Displaying the virtual keyboard 3000 in contrast means determining the color of the virtual keyboard 3000 to be a complementary color that contrasts with the background color, or determining the brightness of the virtual keyboard 3000 and the brightness of the background color to be as different as possible. It may include, but is not limited to, various color contrast methods may be used.

Alternatively, when the background color is gradient as shown in FIG. 11, the virtual keyboard 3000 may be displayed as a color or light/dark gradient.

Referring to FIG. 12 , the electronic device 1000 may provide visual feedback for typing corresponding to the virtual keyboard 3000. For example, virtual keyboard typing accuracy can be improved by providing the user with the accuracy of typing at the current fingertip position.

In the left figure of Figure 12, if the current user's right index finger (2007-1) is located between the two virtual keys 'U' and 'I', tapping the right index finger (2007-1) at the current position will result in 'U' The accuracy for the key is predicted to be 40%, and the accuracy for the 'I' key is predicted to be 60%, so there is almost a half chance of making a typo.

The electronic device 1000 according to an embodiment of the present disclosure may visually display keys corresponding to the current finger position differently depending on accuracy. In the left picture of FIG. 12, the user can see the accuracy when tapping his finger (2007-1) at the current location (2000-1), so he moves his finger (2007-2) to input 'U'. Accuracy for the 'U' key can be increased to 80%.

The electronic device 1000 according to an embodiment of the present disclosure may display the typing accuracy for each key by varying color or transparency, or may place a predetermined marker (for example, a marker) on a key with a higher probability (priority). For example, a border) can be displayed.

Referring to FIG. 13, the electronic device 1000 can identify that the user's hand has left the virtual input area while using the virtual keyboard. In this case, the user's left hand can be used as a new input interface. When it is identified that the user's hand (2000-3) has left the first virtual input area (3200-3) while using the virtual keyboard (3000), the electronic device (1000) The distance between the first virtual input areas 3200-3 can be measured. If the measured distance is less than or equal to a predetermined threshold, the electronic device 1000 determines a second virtual input area 3200-4 corresponding to the user's hand 2000-4 that has left, and the second virtual input area ( 3200-4), the movement of the user's hand can be used as mouse input. For example, the electronic device 1000 may display the pointer 3700 at a predetermined position within the application display area 3600, and the user may move the user's hand within the second virtual input area 3200-4. The electronic device 1000 can be controlled through virtual mouse input corresponding to .

A method of displaying a virtual keyboard by an HMD device according to an embodiment of the present disclosure may include obtaining information about a plane within a sensing area of at least one sensor. A method of displaying a virtual keyboard by an HMD device according to an embodiment of the present disclosure may include identifying a hand on a plane.

A method of displaying a virtual keyboard by an HMD device according to an embodiment of the present disclosure may include determining a virtual input area in a plane based on information about the plane.

The method of displaying a virtual keyboard by an HMD device according to an embodiment of the present disclosure may further include outputting a hand placement guide corresponding to a virtual input area.

A method of displaying a virtual keyboard by an HMD device according to an embodiment of the present disclosure may include, when a hand is identified in the FOV of the display, displaying a real-time captured image of the hand along with the virtual keyboard.

A method of displaying a virtual keyboard by an HMD device according to an embodiment of the present disclosure further includes the steps of image processing keys corresponding to the user's fingers or keys corresponding to virtual keyboard input obtained in response to the movement of the finger. It can be included.

The method of displaying a virtual keyboard by an HMD device according to an embodiment of the present disclosure may further include determining the type of virtual keyboard or arrangement of the virtual keyboard based on position information of the user's hand.

In an HMD device that displays a virtual keyboard according to an embodiment of the present disclosure, at least one processor may obtain information about a plane within a sensing area of at least one sensor. In an HMD device that displays a virtual keyboard according to an embodiment of the present disclosure, at least one processor may identify a hand on a plane.

In the HMD device displaying a virtual keyboard according to an embodiment of the present disclosure, at least one processor may determine a virtual input area in the plane based on information about the plane.

In the HMD device displaying a virtual keyboard according to an embodiment of the present disclosure, at least one processor may output a hand placement guide corresponding to the virtual input area.

In an HMD device that displays a virtual keyboard according to an embodiment of the present disclosure, when a hand is identified in the FOV of the display, at least one processor may display a real-time captured image of the hand along with the virtual keyboard.

In the HMD device displaying a virtual keyboard according to an embodiment of the present disclosure, at least one processor is configured to image process keys corresponding to the user's fingers or keys corresponding to virtual keyboard input acquired in response to the movement of the finger. You can.

In an HMD device that displays a virtual keyboard according to an embodiment of the present disclosure, at least one processor may determine the type of virtual keyboard or arrangement of the virtual keyboard based on position information of the user's hand.

A storage medium that can be read by a device may be provided in the form of a non-transitory storage medium. Here, 'non-transitory storage medium' simply means that it is a tangible device and does not contain signals (e.g. electromagnetic waves). This term refers to cases where data is semi-permanently stored in a storage medium and temporary storage media. It does not distinguish between cases where it is stored as . For example, a 'non-transitory storage medium' may include a buffer where data is temporarily stored.

According to one embodiment, methods according to various embodiments disclosed in this document may be provided and included in a computer program product. Computer program products are commodities and can be traded between sellers and buyers. A computer program product may be distributed in the form of a machine-readable storage medium (e.g. compact disc read only memory (CD-ROM)) or through an application store or between two user devices (e.g. smartphones). It may be distributed in person or online (e.g., downloaded or uploaded). In the case of online distribution, at least a portion of the computer program product (e.g., a downloadable app) is stored on a machine-readable storage medium, such as the memory of a manufacturer's server, an application store's server, or a relay server. It can be temporarily stored or created temporarily.

Claims

In a method for a head mounted display (HMD) device to display a virtual keyboard,

identifying a keyboard input area selection;

Identifying a hand in a field of view (FOV) of a display, based on at least one sensor; and

Comprising: displaying a virtual keyboard on the display based on the identification,

Displaying the virtual keyboard;

When a hand is identified in the FOV of the display, display the virtual keyboard based on the location of the identified hand,

When a hand is not identified in the FOV of the display, displaying the virtual keyboard based on the location of the selected keyboard input area.
The method of claim 1, wherein

Obtaining information about a plane within a sensing area of at least one sensor; and

The method further comprising: identifying a hand in the plane.
The method of claim 2, wherein

The method further comprising: determining a virtual input area in the plane based on information about the plane.
The method of claim 3, wherein

The method further comprising outputting a hand placement guide corresponding to the virtual input area.
According to any one of claims 1 to 4,

Displaying the virtual keyboard;

When a hand is identified in the FOV of the display, displaying a real-time captured image of the hand together with the virtual keyboard.
According to any one of claims 1 to 5,

The method further includes image processing a key corresponding to a user's finger or a key corresponding to a virtual keyboard input obtained in response to movement of the finger.
The method according to any one of claims 1 to 6,

The method further includes determining the type of the virtual keyboard or the arrangement of the virtual keyboard based on the position information of the user's hand.
In a head mounted display (HMD) device that displays a virtual keyboard,

display(1300);

a storage unit 1700 that stores a program including at least one instruction; and

At least one processor 1800 that executes at least one instruction stored in the storage unit,

The at least one processor executes the at least one instruction,

Identifies the keyboard input area selection,

Based on at least one sensor, identify a hand in a field of view (FOV) of the display,

Displaying a virtual keyboard on the display based on the identification,

The at least one processor,

When a hand is identified in the FOV of the display, display the virtual keyboard based on the location of the identified hand,

When a hand is not identified in the FOV of the display, the HMD device displays the virtual keyboard based on the location of the selected keyboard input area.
The method of claim 8, wherein the at least one processor:

Obtain information about a plane within the sensing area of at least one sensor,

An HMD device that identifies a hand in the plane.
The method of claim 9, wherein the at least one processor:

An HMD device that determines a virtual input area in the plane based on information about the plane.
11. The method of claim 10, wherein the at least one processor:

An HMD device that outputs a hand placement guide corresponding to the virtual input area.
The method of any one of claims 8 to 11, wherein the at least one processor:

An HMD device that displays a real-time captured image of the hand together with the virtual keyboard when a hand is identified in the FOV of the display.
The method of any one of claims 8 to 12, wherein the at least one processor:

An HMD device that processes images of keys corresponding to the user's fingers or keys corresponding to virtual keyboard input obtained in response to movement of the fingers.
The method of any one of claims 8 to 13, wherein the at least one processor:

An HMD device that determines the type of the virtual keyboard or the arrangement of the virtual keyboard based on the position information of the user's hand.
A computer-readable recording medium on which a program for performing the method of any one of claims 1 to 7 is recorded on a computer.