JP2022092745A - Operation method using gesture in extended reality and head-mounted display system - Google Patents

Abstract

To provide an operation method using gesture in extended reality (XR) and a head-mounted display (HMD) system.SOLUTION: In a method, a first gesture is identified in a first image. The first gesture corresponds to a hand of a user. A virtual hand and a first interactive object arranged in an interactive area are displayed in response to a result of identifying the first gesture. The virtual hand performs the first gesture. A second gesture is identified in a second image. The second gesture corresponds to the hand of the user, and it is different from the first gesture. The second gesture mutually operates with the first interactive object in the interactive area. The virtual hand and a second interactive object are displayed on a display in response to a result of identifying the second gesture. The virtual hand performs the second gesture. Accordingly, intuitive gesture control is provided.SELECTED DRAWING: Figure 2

Description

The present disclosure generally relates to virtual simulations, in particular gesture-based manipulation methods and head-mounted display systems in augmented reality (XR).

Extended reality (XR) technology for simulating sensations, perceptions and / or environments such as virtual reality (VR), augmented reality (AR), mixed reality (MR), etc. Has been popular recently. The techniques described above can be applied in multiple areas such as gaming, military training, healthcare and remote work. In XR, when a user wears a head-mounted display (HMD), the user can use his or her hand to make gestures and further trigger specific functions. The function may be related to the control of hardware or software. Therefore, it is easy for the user to control the head-mounted display system with his / her own hands.

Some gestures may not be intuitive to trigger a function. Accordingly, the present disclosure is directed to gesture-based operating methods and head-mounted display systems in XRs that provide intuitive gesture control.

In an exemplary embodiment, the gesture-based operation method in XR includes, but is not limited to, the following steps. The first gesture is identified in the first image. The first gesture corresponds to the user's hand. The virtual hand and the first interactive object located in the interactive area are displayed in response to the identified result of the first gesture. The virtual hand performs the first gesture. The second gesture is identified in the second image. The second gesture corresponds to the hand of the user and is different from the first gesture. The second gesture interacts with the first interactive object in the interactive area. The virtual hand and the second interactive object are displayed on the display in response to the identified results of the first gesture and the second gesture. The virtual hand makes the second gesture. The number of virtual hands can be one or two. The virtual hand can be a full-body or half-body avatar's hand in XR.

In an exemplary embodiment, the head-mounted display system includes, but is not limited to, an image capture device, a display, and a processor. The image capture device captures an image. The processor is coupled to the image capture device and the display. The processor is configured to perform the next step. The processor identifies a first gesture in a first image captured by the image capture device. The first gesture corresponds to the user's hand. The processor displays a virtual hand and a first interactive object located in the interactive area of the display in response to the identified result of the first gesture. The virtual hand makes the first gesture. The processor identifies a second gesture in a second image captured by the image capture device. The second gesture corresponds to the hand of the user, and unlike the first gesture, the second gesture interacts with a first interactive object in the interactive area. The processor displays the virtual hand and the second interactive object on the display in response to the identified results of the first gesture and the second gesture. The virtual hand makes the second gesture. The number of virtual hands can be one or two. The virtual hand can be a full-body or half-body avatar's hand in XR.

In light of the above, according to the method of operation and the head-mounted display system, two consecutive gestures can be identified in two images, and the combination of gestures can trigger the display to display different interactive objects. In addition, one interactive object is provided to further interact with the virtual hand. Therefore, a convenient and interesting way to control a head-mounted display system is provided.

It should be understood that this summary does not include all aspects and embodiments of the present disclosure, nor does it imply any limitation or limitation to any method, and is disclosed herein. The invention is to be understood by those skilled in the art to include obvious improvements and modifications to it.

It is a block diagram which shows the head-mounted display system by an exemplary embodiment of this disclosure. It is a flowchart which shows the operation method by a gesture in augmented reality (XR) by an exemplary embodiment of this disclosure. It is a schematic diagram which shows the prediction of the gesture classifier by an exemplary embodiment of this disclosure. , It is a schematic diagram which shows the trigger of an interactive object by a gesture by an exemplary embodiment of this disclosure. , It is a schematic diagram which shows the trigger of an interactive object by a gesture by an exemplary embodiment of this disclosure. , It is a schematic diagram which shows the trigger of an interactive object by a gesture by an exemplary embodiment of this disclosure.

Preferred embodiments of the present disclosure are referred to in detail, examples of which are shown in the accompanying drawings. Wherever possible, the drawings and description refer to the same or similar members using the same reference numbers.

FIG. 1 is a block diagram showing a head-mounted display system 100 according to an exemplary embodiment of the present disclosure. With reference to FIG. 1, the head-mounted display (HMD) system 100 includes, but is not limited to, a memory 110, a display 120, an image capture device 130, and a processor. The HMD system 100 is applied to XR or other reality simulation related techniques.

The memory 110 can be any type of fixed or movable random access memory (RAM), read-only memory (ROM), flash memory, similar devices, or a combination of the above devices. The memory 110 records program code, device configuration, buffer data, or persistent data (images, gesture classifiers, predefined gestures, settings, etc.), which will be described later.

The display 120 can be an LCD, an LED display, or an OLED display.

The image capture device 130 can be a monochrome camera or a camera such as a color camera, a deep camera, a video recorder, or an image capture device capable of capturing other images.

The processor 150 is coupled to the memory 110, the display 120, and the image capture device 130. Processor 150 is configured to load program code stored in memory 110 and perform the procedures of the exemplary embodiments of the present disclosure.

In some embodiments, the processor 150 is a CPU (central processing unit), microprocessor, microcontroller, graphics processing unit (GPU), digital signal processing (DSP) chip, field programmable gate array (FPGA). Can be done. The function of the processor 150 may be performed by an independent electronic device or an integrated circuit (IC), and the operation of the processor 150 may be performed by software.

In one embodiment, the HMD or digital eyeglasses include a memory 110, a display 120, an image capture device 130, and a processor 150. In some embodiments, the processor 150 may not be located in the same device as the display 120 and / or the image capture device 130. However, the device with the display 120, the image capture device 130, and the processor 150, respectively, is a communication transceiver or physical with compatible communication techniques such as Bluetooth®, Wi-Fi, and IR radio communication. Can receive data from each other, including more typical transmission lines. For example, the processor 150 can be located inside the HMD, while the image capture device 130 can be located outside the HMD. In another example, the processor 150 can be located inside the computing device, while the display 120 can be located outside the computing device.

In order to better understand the operating process provided in one or more embodiments of the present disclosure, some embodiments are exemplified below to illustrate the head-mounted display system 100 in detail. The devices and modules of the head-mounted display system 100 are applied to the following embodiments, and the gesture operation method in the XR provided herein will be described. Each step of the method can be adjusted according to the actual implementation situation, and is not limited to the one described here.

FIG. 2 is a flowchart showing a gesture-based operation method in augmented reality (XR) according to an exemplary embodiment of the present disclosure. Referring to FIG. 2, processor 150 can identify the first gesture in the first image captured by the image capture device 130 (step S210). Specifically, the first gesture is a predefined gesture such as palm up, palm down, waving, fist gesture, and so on. The first gesture corresponds to the user's hand. First, the processor 150 can identify the hand of the image. The processor 150 can then identify gestures made by the user in the first image and compare whether the identified gesture is a predefined first gesture.

In one embodiment, the processor 150 identifies the joint of the user's hand from the first image and predicts the gesture of the user's hand based on the first image and the identified joint of the user's hand via the gesture classifier. can do. Specifically, the position of the joints of the hand is related to the gesture of the hand. In addition, image contours, sizes, textures, shapes, and other features are related to hand gestures. The designer prepares many images containing predefined gestures as training samples, and uses the training samples to use machine learning algorithms (deep learning, artificial neural networks (ANN), which are composed of gesture recognition functions. Gesture classifiers can be trained by support vector machines (SVMs, etc.). In addition, these training samples identify the wrist joint, which becomes another training sample for training the same gesture classifier or another gesture classifier. The trained gesture classifier can then be used to determine the gestures made on the input image.

In some embodiments, the processor 150 predicts a gesture based solely on the first image, without the identified joints of the user's hand, and then based on the identified joints of the user's hand. You can confirm the gesture. For example, FIG. 3 is a schematic diagram showing the prediction of a gesture classifier according to an exemplary embodiment of the present disclosure. With reference to FIG. 3, when the image OM including the gesture is input to the gesture classifier, the feature is extracted from the image OM (step S301, that is, feature extraction). For example, in step S301, processor 150 performs a convolution calculation with a filter containing the pixel values of the image OM and outputs a feature map using the corresponding kernel. Features can be textures, corners, edges, or shapes. Next, the processor 150 can classify the features extracted from step S301 such as the feature map (step S302, that is, classification). In addition, one gesture classifier may be composed of one or more labels (that is, one or more gestures of the present embodiment). The gesture classifier can output the determined gesture.

After only one or more gestures are determined based on the image OM, the image OM identifying the wrist joint J is input to the same or another gesture classifier. Similarly, the processor 150 can execute feature extraction (step S301) and classification (step S302) on the image OM that identifies the wrist joint J, and output the determined gesture. The determined gesture is then used to check the correctness of the first determined gesture. For example, if both determined gestures are the same, the processor 150 can determine the gesture. If the determined gestures are different, the processor 150 can determine the gesture in another image.

In one embodiment, the processor 150 can further identify the user's right and left hands. This means that the processor 150 is aware of which hand is performing the gesture or is captured by the image capture device 130 (ie, the hand is the field of view (FOV) of the image capture device 130). Placed inside). In some embodiments, processor 150 may define different predefined gestures or the same predefined gestures for the user's right and left hands, respectively. For example, one function is triggered by a thumbs-up gesture with the right or left hand. In another example, another function is triggered by a gesture that raises the index finger with the right hand, and the same function is triggered by a gesture that raises the little finger with the left hand.

It should be noted that there are many gesture recognition algorithms such as 3D model-based algorithms, skeleton-based algorithms, appearance-based models, and electromyogram-based models. These algorithms can be implemented according to actual requirements.

The processor 150 can display the virtual hand and the first interactive object placed in the interactive area on the display 120 in response to the identified result of the first gesture (step S230). Specifically, when the identified result is that the gesture of the first image is the same as the first gesture, the virtual hand corresponding to the user's hand performs the first gesture. The processor 150 can display a virtual hand performing the first gesture on the display 120, whereby the user can recognize whether he / she is performing the correct gesture. However, even if the identified gesture of the first image is not the first gesture, the processor 150 can display the identified gesture on the display 120. Further, the first gesture is used to trigger the display 120 to display the first interactive object. This means that the first interactive object can be hidden from the display 120 until the user makes the first gesture. The first interactive object is an image, video, virtual ball, or other virtual object. The first interactive object is located in the interactive area of the virtual hand or the avatar's hand. This means that the fingers, palms, or other parts of the virtual hand can interact with any object located in the interactive area. For example, in the interactive area, a finger can touch a virtual key or a palm can hold a virtual ball. The shape and position of the interactive area can be changed according to actual requirements. Further, the number of virtual hands can be one or two. The virtual hand can be a full-body or half-body avatar's hand in XR.

In one embodiment, the first interactive object notifies the user that it can interact and causes the user to try another gesture. That is, the first interactive object is associated with hints for subsequent gestures. For example, the first interactive object is a virtual ball, and the user can try to hold or grab the virtual ball.

Processor 150 can identify the second gesture in the second image (step S250). Specifically, the second gesture is another predefined gesture, such as a palm-up, palm-down, crossed finger, or fist gesture, but different from the first gesture. Is. The second gesture also corresponds to the user's hand. Processor 150 can identify gestures made by the user in the second image and compare whether the identified gesture is a predefined second gesture.

In one embodiment, as described in detail in step S210, the processor 150 identifies the joint of the user's hand from the second image and via the gesture classifier the second image and the identified user's hand joint. The gesture of the user's hand can be predicted based on. In some embodiments, as detailed in step S210, the processor 150 predicts the gesture, simply based on the second image, without the identified joints of the user's hand, and then the user's. Predicted gestures can be determined based on the identified joints of the hand.

The processor 150 can display the virtual hand and the second interactive object on the display 120 in response to the identified result of the second gesture (step S270). Specifically, if the identified result is that the gesture of the second image is the same as the second gesture, the virtual hand corresponding to the user performs the second gesture. The processor 150 can display a virtual hand performing the second gesture on the display 120, and the hand having the second gesture can interact with the first interactive object in the interactive area. For example, a virtual hand grabs a virtual ball. In some embodiments, an animation of the transformation of the first interactive object can be displayed on the display 120. For example, a virtual ball is crushed. However, even if the identified gesture of the second image is not the second gesture, the processor 150 can display the identified gesture on the display 120. In addition, the first interactive object can be hidden due to the wrong gesture.

In addition, the combination of the first and second gestures is used to trigger the display 120 to show the second interactive object and hide the first interactive object. This means that the second interactive object can be hidden from the display 120 from the time the user performs the first gesture until the user performs the second gesture. If a third gesture different from the second gesture is identified in the second image after the first gesture is identified in the first image, the first interactive object is still displayed on the display 120 and the second interactive object is visible. Not done. The second interactive object can be an image, video, menu, or other virtual object. On the other hand, since the second gesture is identified, the first interactive object, which is a hint of the second gesture, does not need to be displayed. Therefore, the first interactive object can help the user intuitively combine the first and second gestures.

For example, FIGS. 4A and 4B are schematics showing the triggering of an interactive object by gestures according to an exemplary embodiment of the present disclosure. With reference to FIG. 4A, the palm-up gesture defined as the first gesture is identified in the first image with the left hand at the first time point. A virtual left-hand LH with a palm-up gesture and a virtual ball io1 (ie, a first interactive object) are displayed on the display 120. With reference to FIG. 4B, the fist gesture defined as the second gesture is identified in the second image with the left hand at the second time point. A virtual left-handed LH with a fist gesture and a main menu io2 (ie, a second interactive object) are displayed on the display 120. The main menu io2 includes a plurality of icons such as a friend list, a map, and an app store icon.

In one embodiment, the second interactive object includes a first menu and a second menu. The second menu is different from the first menu. The processor 150 displays the first menu on the display 120 when the right hand is identified, and displays the second menu on the display 120 when the left hand is identified. This is because if the combination of the first and second gestures is done by the right hand, the first menu is displayed on the display 120, but if the combination of the first and second gestures is done by the left hand, the display. It means that the second menu is displayed at 120.

For example, as shown in FIG. 4B, the second menu is the main menu io2. 5A and 5B are schematics showing the triggering of an interactive object by gestures based on an exemplary embodiment of the present disclosure. With reference to FIG. 5A, the palm-up gesture defined as the first gesture is identified in the first image with the right hand at the third time point. A virtual right-handed RH (ie, a first interactive object) with a palm-up gesture and a virtual ball io3 is displayed on the display 120. With reference to FIG. 5B, the fist gesture defined as the second gesture is identified in the second image with the left hand at the fourth time point. A virtual right-handed RH with a first gesture and a quick setting menu io4 (ie, a second interactive object or a first menu) are displayed on the display 120. The quick setting menu io4 includes a plurality of icons such as camera on / off, specific action by virtual hand, and messaging icon.

In one embodiment, if the second gesture is detected, the processor 150 can further hide the first interactive object on the display 120. This means that the first interactive object is not displayed without further instruction for subsequent gestures. Therefore, only the second interactive object is displayed on the display 120. Taking FIGS. 5A and 5B as an example, the virtual ball io3 is hidden after the fist gesture is identified.

In another embodiment, the second interactive object is displayed on the display 120 and the identification result of the first gesture and the second gesture is confirmed (that is, the combination of the first gesture and the second gesture is performed by the user. Case), the processor 150 can hide the first and second interactive objects on the display 120. Therefore, the menu can be turned off by gesture.

For example, FIGS. 6A and 6B are schematics showing a gesture-based triggering of an interactive object according to an exemplary embodiment of the present disclosure. With reference to FIG. 6A, the quick setting menu io4 is displayed on the display 120. The virtual ball io3 is displayed for a palm-up gesture in the right hand RH. With reference to FIG. 6B, both the virtual ball io3 and the quick setting menu io4 are hidden due to the fist gesture with the right hand RH.

The first interactive object, the second interactive object, and the gesture of FIGS. 4A to 6B can be changed according to an actual request, and the embodiment is not limited thereto.

In summary, the above exemplary embodiments have shown gesture-based operation methods and head-mounted display systems in XR. The combination of gestures is identified by the two images and is used to display the second interactive object on the display. In addition, the first interactive object can be displayed after the first gesture has been identified to further encourage the second gesture to be performed. Therefore, intuitive gesture control is provided.

As will be apparent to those skilled in the art, various modifications and variations may be made to the structure of the present disclosure without departing from the scope or spirit of the present disclosure. In the above-mentioned viewpoint, the present disclosure is intended to include amendments and modifications of the present disclosure, provided that they are within the scope of the claims described below and their equality.

Gesture operation methods and head-mounted display systems can be applied to augmented reality.

100 Head-mounted display system 110 Memory 120 Display 130 Image capture device 150 Processors S210-S270, S301-S302 Step OM image
J Joint LH Virtual Left Hand io1 Virtual Ball io2 Main Menu RH Virtual Right Hand io3 Virtual Ball io4 Quick Setting Menu

Claims (9)

In the first image, the step of identifying the first gesture corresponding to the user's hand,
In response to the identified result of the first gesture, the virtual hand and the first interactive object placed in the interactive area are displayed, and the virtual hand performs the first gesture.
In the second image, a second gesture that corresponds to the user's hand and is different from the first gesture is identified, wherein the second gesture interacts with the first interactive object in the interactive region.
A step of displaying the virtual hand and the second interactive object in response to the identified result of the second gesture, and the virtual hand performing the second gesture.
Gesture-based operation methods in augmented reality (XR), including. An image capture device that captures images and
With the display
Combined with the image capture device and the display
In the first image captured by the image capture device, the first gesture corresponding to the user's hand is identified.
In response to the identified result of the first gesture, the virtual hand and the first interactive object placed in the interactive area are displayed on the display, and the virtual hand performs the first gesture.
In the second image captured by the image capture device, a second gesture that corresponds to the user's hand and is different from the first gesture is identified, and the second gesture interacts with the first interactive object in the interactive area. Acts and
The virtual hand and the second interactive object are displayed on the display in response to the identified result of the second gesture, and the virtual hand performs the second gesture.
With a processor configured to
Head-mounted display system with. The step of identifying the first gesture or the second gesture is
Identifying the joint of the user's hand from the first image or the second image,
The gesture classifier predicts the gesture of the user's hand based on the first image or the second image and the identified joints of the user's hand, and the gesture classifier is trained by a machine learning algorithm. When,
The operation method according to claim 1, comprising the above.
Or,
The processor further
The joint of the user's hand is identified from the first image or the second image, and the joint is identified.
The gesture classifier predicts the gesture of the user's hand based on the first image or the second image and the identified joints of the user's hand, and the gesture classifier is trained by a machine learning algorithm. ,
The head-mounted display system according to claim 2. The step of predicting the gesture of the user's hand is
Predicting gestures simply based on the first or second image, without the identified joints of the user's hand.
Establishing the predicted gesture based on the identified joints of the user's hand,
The operation method according to claim 3, which includes
Or,
The processor further
Predicting gestures simply based on the first or second image, without the identified joints of the user's hand.
Establishing a predicted gesture based on the identified joint of the user's hand,
The head-mounted display system according to claim 3. The operation method according to claim 1, further comprising a step of hiding the first interactive object in response to the identified result of the second gesture.
Or,
The processor further
Hide the first interactive object in response to the identified result of the second gesture.
The head-mounted display system according to claim 2. The operation method according to claim 1, further comprising a step of hiding the first interactive object and the second interactive object in response to the identified result of the second gesture.
Or,
The processor further
Hiding the first interactive object and the second interactive object in response to the identified result of the second gesture.
The head-mounted display system according to claim 2. The step of identifying the first gesture or the second gesture is
The step of displaying the second interactive object, which comprises identifying one of the user's right and left hands, is
Displaying the first menu in response to the identification of the right hand,
The operation method according to claim 1, wherein a second menu different from the first menu is displayed in response to the identification of the left hand, and the second interactive object includes the first menu and the second menu. ,
Or,
The processor further
Identify one of the user's right and left hands
In response to the identification of the right hand, the first menu is displayed on the display.
In response to the identification of the left hand, the display displays the second menu different from the first menu, and the second interactive object includes the first menu and the second menu.
The head-mounted display system according to claim 2. The first menu corresponds to the quick setting menu, and the second menu corresponds to the main menu.
The operation method according to claim 7.
Or,
The head-mounted display system according to claim 7. The first gesture is a gesture with the palm facing up, and the second gesture is a fist gesture.
The operation method according to claim 1.
Or,
The head-mounted display system according to claim 2.

Images