TW202221474A - Operating method by gestures in extended reality and head-mounted display system - Google Patents

Abstract

An operating method by gestures in extended reality (XR) and a head-mounted display (HMD) system are provided. In the method, a first gesture is identified in a first image. The first gesture corresponds to a user’s hand. A virtual hand and a first interactive object located on an interactive area are presented in response to an identified result of the first gesture. The virtual hand makes the first gesture. A second gesture is identified in a second image. The second gesture corresponds to the user’s hand and is different from the first gesture. The second gesture interacts with the first interactive object in the interactive area. The virtual hand and a second interactive object are presented on the display in response to an identified result of the second gesture. The virtual hand makes the second gesture. Accordingly, intuitive gesture control is provided.

Description

Gesture-based operation method and head-mounted display system in extended reality

The present invention relates to a virtual simulation, and more particularly, to an operation method and a head-mounted display system through gestures in extended reality (XR).

Extended reality (XR) technologies for simulating sensations, perceptions and/or environments are popular today, such as virtual reality (VR), augmented reality (AR), and mixed reality; MR). The aforementioned techniques can be applied in a variety of fields, such as gaming, military training, healthcare, teleworking, and the like.

In XR, when a user wears a head-mounted display (HMD), the user can use his/her hands to make gestures and further trigger specific functions. The functions may be related to hardware or software control. It is easy for the user to control the head mounted display system using his/her hands.

Some gestures may not be intuitive enough to trigger functions. In view of this, embodiments of the present invention provide an operation method and a head-mounted display system through gestures in XR, so as to provide intuitive gesture control.

The operation method through gestures in XR according to the embodiment of the present invention includes but is not limited to the following steps. The first gesture is recognized in the first image. The first gesture corresponds to the user's hand. In response to the recognition result of the first gesture, the virtual hand and the first interactive object located on the interactive area are presented. The virtual hand makes the first gesture. The second gesture is recognized in the second image. The second gesture corresponds to the user's hand and is different from the first gesture. The second gesture interacts with the first interactive object in the interactive area. In response to the recognition results of the first gesture and the second gesture, the virtual hand and the second interactive object are displayed on the display. The virtual hand makes a second gesture. The number of virtual hands can be one or two. The virtual hand can be a full-body or half-body avatar's hand in XR.

The head-mounted display system of the embodiment of the present invention includes, but is not limited to, an image sensor, a display, and a processor. The image sensor captures the image. The processor is coupled to the image sensor and the display. The processor is configured to perform the following steps. The processor recognizes the first gesture in the first image captured by the image sensor. The first gesture corresponds to the user's hand. In response to the recognition result of the first gesture, the processor presents the virtual hand and the first interactive object on the interactive area on the display. The virtual hand makes the first gesture. The processor recognizes the second gesture in the second image captured by the image sensor. The second gesture corresponds to the user's hand and is different from the first gesture, and the second gesture interacts with the first interactive object in the interaction area. The processor presents the virtual hand and the second interactive object on the display in response to the recognition results of the first gesture and the second gesture. The virtual hand makes a second gesture. The number of virtual hands can be one or two. The virtual hand can be a full-body or half-body avatar's hand in XR.

Based on the above, according to the operation method and head-mounted display system in XR by gestures, two consecutive gestures are recognized in two images, and the combination of gestures can trigger the display to present different interactive objects. Additionally, interactive objects are provided to further interact with the virtual hand. Thereby, a convenient and fun way to control the head mounted display system is provided.

In order to make the above-mentioned features and advantages of the present invention more obvious and easy to understand, the following embodiments are given and described in detail with the accompanying drawings as follows.

Reference will now be made in detail to the preferred embodiments of the present invention, examples of which are illustrated in the accompanying drawings. Wherever possible, the same reference numerals are used in the drawings and the description to refer to the same or like parts.

1 is a block diagram illustrating a head mounted display system 100 in accordance with one of the exemplary embodiments of the present invention. Referring to FIG. 1 , a head mounted display (HMD) system 100 includes, but is not limited to, a memory 110 , a display 120 , an image sensor 130 and a processor. The HMD system 100 is suitable for use with XR or other reality simulation related technologies.

The memory 110 may be any type of fixed or removable random-access memory (RAM), read-only memory (ROM), flash memory, similar devices, or the above. combination. Memory 110 records code, device configuration, buffer data, or persistent data (eg, images, gesture classifiers, predefined gestures, or settings), which will be described later.

Display 120 may be an LCD, LED display or OLED display.

The image sensor 130 may be a camera (eg, a monochrome camera or a color camera), a depth camera, a video recorder, or other image sensor capable of capturing images.

The processor 150 is coupled to the memory 110 , the display 120 and the image sensor 130 . The processor 150 is configured to load the program code stored in the memory 110 to perform the procedures of the exemplary embodiment of the present invention.

In some embodiments, the processor 150 may be a central processing unit (CPU), microprocessor, microcontroller, graphics processing unit (GPU), digital signal processing (DSP) ) chip, field programmable gate array (field-programmable gate array; FPGA). The functions of the processor 150 may also be implemented by an independent electronic device or an integrated circuit (IC), and the operations of the processor 150 may also be implemented by software.

In one embodiment, the HMD or digital glasses includes a memory 110 , a display 120 , an image sensor 130 and a processor 150 . In some embodiments, processor 150 may not be installed in the same device as display 120 and/or image sensor 130 . However, the devices equipped with the display 120, the image sensor 130, and the processor 150, respectively, may further include a communication transceiver or physical transmission line with compatible communication technologies such as Bluetooth, Wi-Fi, and IR wireless communication to transmit or receive data from each other. For example, the processor 150 may be installed in the HMD, while the image sensor 130 is installed outside the HMD. For another example, the processor 150 may be installed in the computing device, while the display 120 is installed outside the computing device.

In order to better understand the operational procedures provided in one or more embodiments of the present invention, several embodiments will be exemplified below to explain the head mounted display system 100 in detail. The devices and modules in the system 100 are applied in the following embodiments to explain the operation methods provided herein through gestures in XR. Each step of the method can be adjusted according to the actual implementation and should not be limited to what is described herein.

2 is a flowchart illustrating a method of operation through gestures in Extended Reality (XR) in accordance with one of the exemplary embodiments of the present invention. Referring to FIG. 2 , the processor 150 may recognize the first gesture in the first image captured by the image sensor 130 (step S210 ). Specifically, the first gesture is a predefined gesture, such as a palm up, palm down, waving or fist gesture. The first gesture corresponds to the user's hand. First, the processor 150 can recognize the hand in the image. Then, the processor 150 may recognize the gesture made by the user's hand in the first image, and compare whether the recognized gesture is a predefined first gesture.

In one embodiment, the processor 150 may identify the joints of the user's hand from the first image, and predict the gesture of the user's hand based on the first image and the identified joints of the user's hand through the gesture classifier. Specifically, the positions of the hand joints are related to gestures. In addition, outlines, sizes, textures, shapes, and other features in images are associated with gestures. Designers can prepare a large number of images containing predefined gestures as training samples, and use the training samples through a machine learning algorithm (such as deep learning, artificial neural network (ANN) or support vector machine configured with gesture recognition function) (support vector machine; SVM)) to train a gesture classifier. Furthermore, hand joints are identified in these training samples and would be another training sample used to train the same gesture classifier or another gesture classifier. The trained gesture classifier can then be used to determine which gesture was made in the input image.

In some embodiments, the processor 150 may predict the gesture based only on the first image and without the identified joints of the user's hand, and then confirm the predicted based on the identified joints of the user's hand gesture. For example, FIG. 3 is a schematic diagram illustrating predictions of a gesture classifier in accordance with one of the exemplary embodiments of the present invention. Referring to FIG. 3 , if an image OM containing a gesture is input into the gesture classifier, features will be extracted from the image OM (step S301 , ie, feature extraction). For example, in step S301, the processor 150 performs a convolution calculation on the filter including the pixel value of the image OM and the corresponding kernel to output a feature map. Features can be textures, corners, edges, or shapes. Next, the processor 150 may classify the features (eg, feature maps) extracted from step S301 (step S302 , ie, classify). It should be noted that a gesture classifier may be configured with one or more labels (ie, in this embodiment, one or more gestures). The gesture classifier may output the determined gesture.

After determining one or more gestures based only on the image OM, the image OM with the identified hand joint J will be input into the same or another gesture classifier. Similarly, the processor 150 may perform feature extraction (step S301 ) and classification (step S302 ) on the image OM with the identified hand joint J to output the determined gesture. The gestures determined first are used to check the correctness of the gestures determined first. For example, if the two determined gestures are the same, the processor 150 may confirm the gesture. If the determined gesture is different, the processor 150 may determine the gesture in another image.

In one embodiment, the processor 150 may further identify the user's right and left hands. This means that the processor 150 knows which hand is making the gesture or captured by the image sensor 130 (ie, the hand is within the field of view (FOV) of the image sensor 130). In some embodiments, the processor 150 may define different predefined gestures or the same predefined gesture for the user's right and left hands, respectively. For example, a function would be triggered by a right or left thumb up gesture. For another example, another function will be triggered by the index finger up gesture of the right hand, but the same function will be triggered by the little finger up gesture of the left hand.

It should be noted that a large number of gesture recognition algorithms still exist, eg, 3D model-based algorithms, bone-based algorithms, appearance-based models, or electromyography-based models. Those algorithms can be implemented for actual requirements.

The processor 150 may present the virtual hand and the first interactive object on the interactive area on the display 120 in response to the recognition result of the first gesture (step S230 ). Specifically, if the recognition result is that the gesture in the first image is the same as the first gesture, the virtual hand corresponding to the user's hand will make the first gesture. The processor 150 may present the virtual hand making the first gesture on the display 120 so that the user may know whether he/she made the correct gesture. However, if the recognized gesture of the first image is not the first gesture, the processor 150 may still present the recognized gesture on the display 120 . In addition, the first gesture is used to trigger the display 120 to present the first interactive object. This means that the first interactive object may not be presented on the display 120 until the user makes the first gesture. The first interactive object may be an image, a video, a virtual ball or other virtual objects. The first interactive object is located on the interactive area of the virtual hand or the hand of the avatar. This means that the fingers, palm or other parts of the virtual hand may be able to interact with any object located in the interaction area. For example, a finger can touch a virtual key, or a palm can grasp a virtual ball in the interactive area. It should be noted that the shape and orientation of the interactive area can be modified based on actual requirements. Also, the number of virtual hands may be one or two. The virtual hand can be a full-body or half-body avatar's hand in XR.

In one embodiment, the first interactive object is used to notify the user that the interaction is available, and cause the user to attempt another gesture. That is to say, the first interactive object is related to the prompt of the subsequent gesture. For example, the first interactive object is a virtual ball, and the user can try to grasp or grab the virtual ball.

The processor 150 may recognize the second gesture in the second image (step S250). Specifically, the second gesture is another predefined gesture, such as a palm up, palm down, interdigitated or fist gesture, but is different from the first gesture. The second gesture also corresponds to the user's hand. The processor 150 may recognize the gesture made by the user's hand in the second image, and compare whether the recognized gesture is a predefined second gesture.

In one embodiment, as mentioned in detail in step S210 , the processor 150 may recognize the joints of the user's hand from the second image, and use a gesture classifier based on the second image and the recognized motion of the user's hand. joints to predict the gestures of the user's hand. In some embodiments, as mentioned in detail in step S210, the processor 150 may predict the gesture based only on the second image and without the identified joints of the user's hand, and then based on the identified usage the joints of the user's hand to confirm the predicted gesture.

The processor 150 may present the virtual hand and the second interactive object on the display 120 in response to the recognition result of the second gesture (step S270 ). Specifically, if the recognition result is that the gesture in the second image is the same as the second gesture, the virtual hand corresponding to the user will make the second gesture. The processor 150 can present the virtual hand making the second gesture on the display 120, and the hand with the second gesture can interact with the first interactive object in the interactive area. For example, a virtual hand grabs a virtual ball. In some embodiments, an animation of the deformation of the first interactive object may be presented on the display 120 . For example, squeeze the virtual ball. However, if the recognized gesture of the second image is not the second gesture, the processor 150 may still present the recognized gesture on the display 120 . In addition, the first interactive object can be hidden due to wrong gestures.

Furthermore, the combination of the first gesture and the second gesture is used to trigger the display 120 to present the second interactive object but hide the first interactive object. This means that the second interactive object may not be presented on the display 120 until the user makes the first gesture and then the second gesture. If a third gesture different from the second gesture is recognized in the second image after the first gesture has been recognized in the first image, the first interactive object is still presented on the display 120 and the second interaction will not be presented object. The second interactive object may be an image, a video, a menu or other virtual objects. On the other hand, since the second gesture is recognized, it is not necessary to present the first interactive object (which is a prompt for the second gesture). Therefore, the first interactive object can indicate that the user intuitively performs the combination of the first gesture and the second gesture.

For example, FIGS. 4A and 4B are schematic diagrams illustrating triggering of interactive objects through gestures in accordance with one of the exemplary embodiments of the present invention. Referring to FIG. 4A , a palm-up gesture of the left hand is recognized in the first image at a first time point, which is defined as a first gesture. A virtual left hand LH with a palm up gesture and a virtual ball io1 (ie, the first interactive object) will be presented on the display 120 . Referring to FIG. 4B , the fist gesture of the left hand is recognized in the second image at the second time point, which is defined as the second gesture. The virtual left hand LH with the fist gesture and the main menu io2 (ie, the second interactive object) will be presented on the display 120 . The main menu io2 contains several icons, such as those for the friends list, maps, and the app store.

In one embodiment, the second interactive object includes a first menu and a second menu. The second menu is different from the first menu. The processor 150 may present a first menu on the display 120 if a right hand is recognized, and a second menu on the display 120 if a left hand is recognized. This means that if the combination of the first gesture and the second gesture is made by the right hand, then the first menu will be presented on the display 120 . However, if the combination of the first gesture and the second gesture is made by the left hand, then a second menu will be presented on the display 120 .

For example, the second menu is the main menu io2 as shown in FIG. 4B . 5A and 5B are schematic diagrams illustrating triggering of interactive objects through gestures in accordance with one of the exemplary embodiments of the present invention. Referring to FIG. 5A , a palm-up gesture of the right hand is recognized in the first image at a third time point, which is defined as a first gesture. A virtual right hand RH with a palm-up gesture and a virtual ball io3 (ie, the first interactive object) will be presented on the display 120 . Referring to FIG. 5B , at the fourth time point, the fist gesture of the left hand is recognized in the second image, which is defined as the second gesture. A virtual right hand RH with a fist gesture and a quick setting menu io4 (ie, the second interactive object or the first menu) will be presented on the display 120 . The quick settings menu io4 contains icons such as those for turning the camera on/off, performing certain movements on the virtual hand, and messaging.

In one embodiment, the processor 150 may further hide the first interactive object on the display 120 if the second gesture is detected. This means that subsequent gestures do not need to be additionally indicated, and the first interactive object will not be visible. Therefore, only the second interactive object is presented on the display 120 . Taking FIG. 5A and FIG. 5B as an example, after the fist gesture is recognized, the virtual ball io3 is hidden.

In another embodiment, if the second interactive object has been presented on the display 120 and the recognition results of the first gesture and the second gesture are confirmed (ie, the combination of the first gesture and the second gesture is made by the user) , the processor 150 can hide the first interactive object and the second interactive object on the display 120 . Therefore, the menu can be closed by a gesture.

For example, FIGS. 6A and 6B are schematic diagrams illustrating triggering of interactive objects through gestures in accordance with one of the exemplary embodiments of the present invention. Referring to FIG. 6A , a quick settings menu io4 has been presented on the display 120 . The virtual ball io3 is presented due to the palm-up gesture of the right hand RH. Referring to FIG. 6B , both the virtual ball io3 and the quick setting menu io4 are hidden due to the fist gesture of the right hand RH.

It should be noted that the first interactive object, the second interactive object, the first gesture, and the second gesture in FIGS. 4A to 6B may be modified based on actual requirements, and the embodiment is not limited thereto.

To sum up, in the operation method by gestures in XR and the head-mounted display system of the embodiments of the present invention, the gesture combination is recognized in the two images, and the gesture combination is used to present the second interactive object on the display. In addition, after the first gesture is recognized, the first interactive object may be displayed to further prompt to make the second gesture. Thereby, intuitive gesture control can be provided.

Although the present invention has been disclosed above by the embodiments, it is not intended to limit the present invention. Anyone with ordinary knowledge in the technical field can make some changes and modifications without departing from the spirit and scope of the present invention. Therefore, The protection scope of the present invention shall be determined by the scope of the appended patent application.

100: Head Mounted Display Systems 110: Memory 120: Display 130: Image sensor 150: Processor io1, io3: virtual ball io2: main menu io4: Quick Setup Menu J: Identified hand joints LH: virtual left hand OM: Video RH: virtual right hand S210, S230, S250, S270, S301, S302: Steps

1 is a block diagram illustrating a head mounted display system in accordance with one of the exemplary embodiments of the present invention. 2 is a flowchart illustrating a method of operation through gestures in Extended Reality (XR) in accordance with one of the exemplary embodiments of the present invention. 3 is a schematic diagram illustrating predictions of a gesture classifier in accordance with one of the exemplary embodiments of the present invention. 4A and 4B are schematic diagrams illustrating triggering of interactive objects through gestures in accordance with one of the exemplary embodiments of the present invention. 5A and 5B are schematic diagrams illustrating triggering of interactive objects through gestures in accordance with one of the exemplary embodiments of the present invention. 6A and 6B are schematic diagrams illustrating triggering of interactive objects through gestures in accordance with one of the exemplary embodiments of the present invention.

S210~S270: Steps

Claims (16)

An operation method through gestures in extended reality (XR), including: identifying a first gesture in a first image, wherein the first gesture corresponds to a user's hand; In response to a recognition result of the first gesture, a virtual hand and a first interactive object located on an interactive area are presented, wherein the virtual hand makes the first gesture; Recognizing a second gesture in a second image, wherein the second gesture corresponds to the user's hand and is different from the first gesture, and the second gesture is the same as in the interaction area The first interactive object interacts with; In response to the recognition result of the second gesture, the virtual hand and a second interactive object are presented, wherein the virtual hand makes the second gesture. The method for operating through gestures in extended reality according to claim 1, wherein the step of recognizing the first gesture or the second gesture comprises: Identify the joints of the user's hand from the first image or the second image; and Predicting the gesture of the user's hand based on the first image or the second image and the identified joints of the user's hand by a gesture classifier, wherein the machine learning algorithm is used to train the gesture Gesture classifier. The method for operating through gestures in extended reality according to claim 2, wherein the step of predicting the gesture of the user's hand comprises: predicting the gesture based only on the first image or the second image and without the identified joints of the user's hand; and The predicted gesture is confirmed based on the identified joints of the user's hand. The operation method through gestures in the extended reality as described in claim 1, further comprising: The first interactive object is hidden in response to the recognition result of the second gesture. The operation method through gestures in the extended reality as described in claim 1, further comprising: The first interactive object and the second interactive object are hidden in response to the recognition result of the second gesture. The method for operating through gestures in extended reality according to claim 1, wherein the step of recognizing the first gesture or the second gesture comprises: Identifying one of the user's right hand and left hand and presenting the second interactive object includes: presenting a first menu in response to identifying the right hand; and A second menu is presented in response to recognizing the left hand, wherein the second menu is different from the first menu, and the second interactive object includes the first menu and the second menu. The operation method through gestures in extended reality according to claim 6, wherein the first menu corresponds to a quick setting menu, and the second menu corresponds to a main menu. The method for operating through gestures in extended reality according to claim 1, wherein the first gesture is a palm-up gesture, and the second gesture is a fist-clenching gesture. A head-mounted display system, comprising: an image sensor for capturing images; a display; and a processor, coupled to the image sensor and the display, and configured to: identifying a first gesture in a first image captured by the image sensor, wherein the first gesture corresponds to a user's hand; In response to the recognition result of the first gesture, a virtual hand and a first interactive object located on an interactive area are presented on the display, wherein the virtual hand makes the first gesture; Recognizing a second gesture in a second image captured by the image sensor, wherein the second gesture corresponds to the user's hand and is different from the first gesture, and the The second gesture interacts with the first interactive object in the interactive area; In response to the recognition result of the second gesture, the virtual hand and a second interactive object are presented on the display, wherein the virtual hand makes the second gesture. The head mounted display system of claim 9, wherein the processor is further configured to: Identify the joints of the user's hand from the first image or the second image; and Predicting the gesture of the user's hand based on the first image or the second image and the identified joints of the user's hand by a gesture classifier, wherein the machine learning algorithm is used to train the gesture Gesture classifier. The head mounted display system of claim 10, wherein the processor is further configured to: predicting the gesture based only on the first image or the second image and without the identified joints of the user's hand; and The predicted gesture is confirmed based on the identified joints of the user's hand. The head mounted display system of claim 9, wherein the processor is further configured to: The first interactive object is hidden in response to the recognition result of the second gesture. The head mounted display system of claim 9, wherein the processor is further configured to: The first interactive object and the second interactive object are hidden in response to the recognition result of the second gesture. The head mounted display system of claim 9, wherein the processor is further configured to: Identifying one of the user's right and left hands and presenting the second interactive object includes: presenting a first menu on the display in response to recognizing the right hand; and presenting a second menu on the display in response to recognizing the left hand, wherein the second menu is different from the first menu, and the second interactive object includes the first menu and the second menu menu. The head mounted display system of claim 14, wherein the first menu corresponds to a quick settings menu and the second menu corresponds to a main menu. The head mounted display system of claim 9, wherein the first gesture is a palm up gesture and the second gesture is a fist gesture.

Images