TW202340912A - Virtual system controllable by hand gesture - Google Patents

Abstract

A virtual system controllable by hand gesture includes a head-mounted displaying device and a terminal device. The terminal device is connected to a cloud server to obtain a three-dimensional virtual space and a plurality of three-dimensional objects, which are displayed on the head-mounted displaying device. The terminal device stores a plurality of hand gestures and operation instructions. The terminal device captures the operating image of a non-contact operation of the user corresponding to the three-dimensional virtual space or objects. The terminal device extracts the hand image through image calculation and marks the framework image thereof, which is then compared with each hand gesture, thereby obtaining matching hand gesture and operation instruction, which is fed back to the three-dimensional virtual space or objects, achieving the movement in the three-dimensional virtual space or non-contact operation of the three-dimensional objects.

Description

Virtual system with hand gesture control

The present invention relates to the field of human-machine interface, and in particular, to a virtual system with hand gesture control.

The full name of virtual reality in English is Virtual Reality, usually referred to as VR. When experiencing virtual reality, a head-mounted display (HMD) is used to completely cover the visual field of the eyes. The user cannot see the real environment and is completely immersed in the environment presented by the head-mounted display. In the virtual world, most users use controllers to interact with content in the virtual world.

The controller is directly installed on the user's hand or held by the user in a hand-held manner. This type of controller requires the user to wear a specially designed data glove to detect the user's hand movements for further identification. , the sensing methods of the controller include: accelerometer, gyroscope, flexible resistor, infrared ranging and muscle physiological electrical signals, etc.

The characteristic of using a controller to monitor the user's hand movements is that it can detect more precise and detailed hand movements. However, because of this, its calculation and processing method is also more complex and prone to calculation delays, resulting in delayed recognition responses.

Moreover, it is difficult for the type of controller to fully conform to the usage habits of each user, and it can easily cause discomfort to the user when wearing it and cause burden when using it. Therefore, how to improve the current inconvenience of using virtual reality is very important. .

In order to solve the above problems, the present invention provides a virtual system with hand gesture control, which can control the three-dimensional virtual space through non-contact hand image control to achieve controller-free and non-contact control effects.

An embodiment of the present invention provides a virtual system with hand gesture control, which includes: a head-mounted display, which is worn on the user's head. The head-mounted display has a screen, and the screen corresponds to the user's head. Eyes; and a terminal device installed on a head-mounted display. The terminal device has a communication module, a display module, a database, a photography module and a control module. The communication module is connected to a cloud server. The cloud server is provided with a three-dimensional virtual space, and the three-dimensional virtual space has a plurality of three-dimensional objects. The communication module can transmit the three-dimensional virtual space and each of the three-dimensional objects to the display module; the display module is coupled to the screen, and the display module The group can display the three-dimensional virtual space with each three-dimensional object on the screen; the database contains a plurality of hand postures and an operation command corresponding to each hand posture; the photography module provides the user with the ability to take pictures corresponding to the three-dimensional virtual space or one of the three-dimensional objects. An operation image of the contact control; the control module receives the operation image from the photography module, the control module extracts a hand image through an image calculation process, and marks a skeleton image with the hand image, and the control module converts the skeleton image Compare with various hand postures existing in the database to obtain one of the matching hand postures and corresponding operation instructions. The control module will feedback the obtained operation instructions to the three-dimensional virtual space or three-dimensional objects to operate in the three-dimensional virtual space. Move around or perform contactless manipulation of three-dimensional objects.

Through the above, the present invention can capture the user's operation images corresponding to the three-dimensional virtual space or three-dimensional objects, analyze the operation images to obtain the corresponding operation instructions, and use the operation instructions to feedback control the three-dimensional virtual space or three-dimensional objects presented on the head-mounted display. Objects; thereby achieving a contactless control effect and improving the inconvenience of conventionally requiring a controller to control virtual space.

In order to facilitate the explanation of the central idea of the present invention expressed in the above summary column, specific embodiments are hereby expressed. Various objects in the embodiments are drawn according to proportions, sizes, deformations or displacements suitable for illustration, rather than according to the proportions of actual components, and are explained first.

Referring to FIGS. 1 to 7 , the present invention provides a virtual system 100 with hand gesture control, which includes:

The present invention provides a virtual system 100 with hand gesture control, which includes:

The head-mounted display 10 is worn on the head of the user 1. The head-mounted display 10 has a screen 11. The screen 11 corresponds to the eyes of the user 1. The screen 11 can display a three-dimensional image.

A terminal device 20, which is installed on the head-mounted display 10. The terminal device 20 has a communication module 21, a display module 22, a database 23, a photography module 24 and a control module 25. The communication module 21 is connected to a cloud server 2 through the Internet. The cloud server 2 is provided with a three-dimensional virtual space 3. The three-dimensional virtual space 3 has a plurality of three-dimensional objects 4. The communication module 21 can transmit the three-dimensional virtual space 3 and each three-dimensional object 4. As for the display module 22, the display module 22 can be coupled with the screen 11. The display module 22 can transmit and display the three-dimensional virtual space 3 together with each three-dimensional object 4 on the screen 11; wherein, the three-dimensional virtual space 3 can be a tourist attraction, Hotels, restaurants, stores, car showrooms, etc., the three-dimensional objects 4 correspond to the type settings of the three-dimensional virtual space 3, but the present invention is not limited to this.

The database 23 stores a plurality of hand postures 231 and an operation command corresponding to each hand posture 231; in the embodiment of the present invention, the operation command corresponding to the hand posture 231 can be (1) the hand posture 231 is with the index finger pointing upward. The corresponding operation command is to move up; (2) the hand posture 231 is the index finger downward, the corresponding operation command is to move down; (3) the hand posture 231 is to make a fist and the corresponding operation command of the finger face photography module 24 is to return; (4) ) The hand posture 231 is a push palm, and the corresponding operation command is continue or forward; (5) the hand posture 231 is an OK gesture, and the corresponding operation command is click; (6) the hand posture 231 is a fist, and the palm facial photography module 24 corresponds The operation command is stop; (7) The hand posture 231 is the thumb to the left, and the operation command is move left; (8) The hand posture 231 is the thumb to the right, and the operation command is move right, as shown in Figure 2; but The present invention is not limited to this.

The photography module 24 provides an operation image of the non-contact control performed by the user 1 corresponding to the three-dimensional virtual space 3 or one of the three-dimensional objects 4; the control module 25 receives the operation image from the photography module 24, and the control module 25 transmits the operation image through An image operation process extracts a hand image, and marks a skeleton image from the hand image. The control module 25 compares the skeleton image with each hand posture 231 stored in the database 23 to obtain one of the matching hand postures. 231 and the corresponding operation instructions, the control module 25 feedbacks the obtained operation instructions to the three-dimensional virtual space 3 or the three-dimensional object 4 to move in the three-dimensional virtual space 3 or perform contactless control of the three-dimensional object 4; in the implementation of the present invention In this example, the image processing is computer vision combined with deep learning.

Furthermore, the control module 25 marks the skeleton image I on the hand image H through a data labeling program (LabelImg). The skeleton image I has a plurality of fulcrums P and a plurality of struts S, and each fulcrum P corresponds to each knuckle of the hand. and the junction of the hand and the wrist. Each pole S is connected between two adjacent points P. Each pole P has an exclusive code, as shown in Figure 3; and the control module 25 can determine the relationship between each pole P and each other. position relationship to obtain one of the matching hand gestures 231 from the database 23.

In addition, the control module 25 will receive the dynamic operation image transmitted by the photography module 24, and the control module 25 will divide the operation image into a plurality of stage operation images according to time sequence, and the control module 25 will perform image calculation on each stage operation image. Process, and generate a plurality of hand images H marked with the skeleton image I according to the time sequence, and obtain the operation instructions from the database 23 according to the time sequence, and the control module 25 will feedback the obtained operation instructions to the three-dimensional virtual space 3 or the three-dimensional object 4 , to move within the three-dimensional virtual space 3 or perform contactless control of the three-dimensional object 4 .

The terminal device 20 further has an editing module 26, which is coupled to the database 23. The editing module 26 can display each hand gesture 231 and the corresponding operation command in the database 23 on the display interface. The module 26 can select one of the hand gestures 231 for editing to change the selected hand gesture 231 and the corresponding operation command.

Furthermore, the editing module 26 can enable the photography module 24 to capture a sample image, and send the sample image to the control module 25 for image calculation processing to generate a hand image H marked with the skeleton image I. The editing module 26 can The hand image H marked with the skeleton image I is replaced with one of the hand postures 231 in the database 23 or set to correspond to one of the operation instructions.

The terminal device 20 further has a financial module 27, which is coupled to the control module 25. The financial module 27 has a checkout option. The checkout option is displayed on the display interface in conjunction with the three-dimensional virtual space 3. The control module 25 can obtain The operation command feedback responds to the selected three-dimensional item 4 and clicks the checkout option to perform a contactless checkout operation on the selected three-dimensional item 4, wherein the financial module 27 can calculate and generate a consumption amount, and The consumption amount is presented on the screen 11 through the display module 22 to provide the user 1 with the operation image generated by the corresponding checkout operation and obtain the corresponding operation instructions to pay the consumption amount.

For example, please refer to Figures 1, 4 and 5. When the three-dimensional virtual space 3 displayed on the screen 11 is a shopping mall, and the three-dimensional items 4 are the products sold in the mall, the user 1 can 3 to select items and consume in the shopping mode. When the user 1 selects items in the three-dimensional virtual space 3, the financial module 27 will calculate the amount of each selected three-dimensional item 4 to obtain the consumption amount. And the consumption funds and checkout options will be displayed on the screen 11, and the user 1 can perform checkout operations through the contactless operation financial module 27.

The communication module 21 can obtain a map information 5 from the cloud server 2, and the map information 5 corresponds to the three-dimensional virtual space 3; the terminal device 20 further has a positioning module 28, and the positioning module 28 is coupled with the control module 25. The positioning module The group 28 can obtain the position of the user 1 in the three-dimensional virtual space 3 to generate a position signal; the positioning module 28 can transmit the position signal to the control module 25, and the control module 25 can generate the position signal through graphical processing. A position graphic 281 is marked on the map information 5, as shown in Figures 1 and 5; the control module 25 displays the map information 5 with the position graphic 281 on the screen 11 through the display module 22 in conjunction with the three-dimensional virtual space 3.

Furthermore, the terminal device 20 further has a navigation module 29. The navigation module 29 has a query unit 291 and a guidance unit 292. The query unit 291 can present the map information 5. The map information 5 has plural area information 6. The query unit 291 provides the user 1 to select one of the area information 6 using the operation instruction feedback generated by operating the image. The guidance unit 292 receives the selected area information 6 and calculates and plans to generate a virtual route information L. The user 1 is in the three-dimensional virtual space. 3 can move to the selected area information 6 according to the virtual route information L.

For example, please refer to Figure 1 and Figure 5 to Figure 7. When the three-dimensional virtual space 3 displayed on the screen 11 is a shopping mall, the map information 5 corresponds to the shopping mall, and the regional information 6 corresponds to the shopping mall. Fresh food area, clothing area, home appliance area, stationery area, food area, etc. When the user 1 is in the food area, the positioning module 28 can generate the corresponding position signal, and the position signal generated by the positioning module 28 is transmitted to the control module. Group 25: The control module 25 processes the location signal through graphical processing to generate a location pattern 281 marked in the area information 6 corresponding to the fresh food area on the map information 5. The user 1 can select the clothing area of the area information 6 through the query unit 291. , and the guidance unit 292 will perform path planning from the food area to the clothing area to generate corresponding virtual route information L. In this way, the user 1 can move to the selected location in the three-dimensional virtual space 3 according to the virtual route information L. Area information 6 clothing area.

Based on the above, the present invention can achieve the following effects:

1. The virtual system 100 with hand gesture control of the present invention can capture the operation image of the user 1 corresponding to the three-dimensional virtual space 3 or the three-dimensional object 4, and analyze the operation image to obtain the corresponding operation instruction, which is controlled by the operation instruction feedback. The three-dimensional virtual space 3 or three-dimensional objects 4 presented by the head-mounted display 10 can achieve contactless and controller-less control effects.

2. The virtual system 100 with hand posture control of the present invention can provide the user 1 to create or modify the hand posture 231 and operation instructions by himself to adapt to the operating habits of different users 1, thereby improving the breadth of use and convenience.

3. The virtual system 100 with hand gesture control of the present invention can provide positioning and navigation in the three-dimensional virtual space 3 to avoid the problem of getting lost; and through the virtual route information L, it can quickly and accurately move to the desired area information. 6. Provide convenience of use.

The above embodiments are only used to illustrate the present invention and are not intended to limit the scope of the present invention. All modifications or changes that do not violate the spirit of the present invention fall within the scope of the invention.

1:User 2:Cloud server 3: Three-dimensional virtual space 4: Three-dimensional items 5: Map information 6:Regional information 100: Virtual system with hand gesture control 10:Head mounted display 11:Screen 20:Terminal device 21:Communication module 22:Display module 23:Database 231:Hand posture 24:Photography module 5:Control module 26:Edit module 27:Financial module 28: Positioning module 281: Position graphics 29:Navigation module 291: Query unit 292: Guidance unit I: skeleton image P: fulcrum S: support rod H:Hand image L:Virtual route information

Figure 1 is an architectural block diagram of the present invention. Figure 2 is a schematic diagram of hand postures and corresponding operation instructions of the present invention. Figure 3 is a hand image marked with a skeleton image according to the present invention. FIG. 4 is a schematic diagram (1) of an embodiment of the present invention, showing a user wearing a head-mounted display for contactless control. Figure 5 is a schematic diagram (2) of the embodiment of the present invention, showing that map information is displayed on the screen in conjunction with the three-dimensional virtual space. Figure 6 is a schematic diagram (3) of the embodiment of the present invention, showing that the user responds to the query unit through operating instructions. Figure 7 is a schematic diagram (4) of the embodiment of the present invention, showing that the guidance unit generates virtual route information from map information.

2:Cloud server

3: Three-dimensional virtual space

4: Three-dimensional items

5: Map information

6:Regional information

100: Virtual system with hand gesture control

10:Head mounted display

20:Terminal device

21:Communication module

22:Display module

23:Database

24:Photography module

25:Control module

26:Edit module

27:Financial module

28: Positioning module

29:Navigation module

291: Query unit

292: Guidance unit

Claims (10)

A virtual system with hand gesture control, which includes: A head-mounted display, which is worn on the user's head and has a screen corresponding to the user's eyes; and A terminal device installed on the head-mounted display. The terminal device has a communication module, a display module, a database, a photography module and a control module. The communication module is connected to a cloud server. The cloud server is provided with a three-dimensional virtual space. The three-dimensional virtual space has a plurality of three-dimensional objects. The communication module can transmit the three-dimensional virtual space and each of the three-dimensional objects to the display module; the display module and the screen Coupled, the display module can display the three-dimensional virtual space with each of the three-dimensional objects on the screen; the database contains a plurality of hand postures and operation instructions corresponding to each hand posture; the photography module provides shooting use An operation image corresponding to the non-contact control of the three-dimensional virtual space or one of the three-dimensional objects; the control module receives the operation image from the photography module, and the control module extracts the operation image through an image calculation process to extract a hand image, and a skeleton image is marked by the hand image, and the control module compares the skeleton image with each hand posture existing in the database to obtain one of the matching hand postures and the corresponding operation. Instructions: The control module feedbacks the obtained operation instructions to the three-dimensional virtual space or the three-dimensional object, so as to move within the three-dimensional virtual space or perform contactless control of the three-dimensional object. The virtual system with hand posture control as described in claim 1, wherein the skeleton image has a plurality of fulcrums and a plurality of struts, each of which corresponds to each knuckle of the hand and the junction of the hand and the wrist, and each The rod is connected between two adjacent points, and each point has a unique code. The control module determines the positional relationship between the two points to obtain a matching hand posture from the database. The virtual system with hand gesture control as described in claim 1, wherein the camera module can dynamically track the operation image of the user's non-contact control of the three-dimensional virtual space or one of the three-dimensional objects. The virtual system with hand gesture control as described in claim 3, wherein the operation image is a dynamic image, the control module divides the operation image into a plurality of stage operation images according to time sequence, and the control module divides each stage into Operate the image to perform the image operation processing, and generate a plurality of hand images marked with the skeleton image according to the time sequence, and obtain the operation instructions from the database according to the time sequence, and the control module will obtain the operation instructions Instruction feedback responds to the three-dimensional virtual space or the three-dimensional object to move within the three-dimensional virtual space or perform contactless manipulation of the three-dimensional object. The virtual system with hand gesture control as described in claim 1 or 4, wherein the terminal device further has an editing module, the editing module is coupled to the database, and the editing module can edit the database Each of the hand postures and the corresponding operation instructions are displayed on the screen of the head-mounted display. The editing module can select one of the hand postures for editing to change the selected hand posture and the corresponding operation instructions. The virtual system with hand gesture control as described in claim 5, wherein the editing module is further coupled to the photography module and the control module, and the editing module can enable the photography module to capture a sample image and transmit the sample image to the control module to generate the hand image marked with the skeleton image through the image calculation processing. The editing module can replace the hand image marked with the skeleton image in the database. One of the hand postures or settings corresponds to one of the operation instructions. The virtual system with hand gesture control as described in claim 5, wherein the terminal device further has a financial module coupled to the control module, the financial module has a checkout option, and the checkout Options are displayed on the display interface in conjunction with the three-dimensional virtual space, and the control module can feedback the obtained operation instructions to the selected three-dimensional item and click the checkout option to respond to the selected three-dimensional item. When items undergo contactless checkout operations, the financial module can calculate a consumption amount and present the consumption amount on the screen through the display module to provide the user with the operation image generated by the corresponding checkout operation. , obtain the corresponding operation instructions to pay the consumption amount. The virtual system with hand gesture control as described in claim 5, wherein the communication module can obtain a map information from the cloud server, and the map information corresponds to the three-dimensional virtual space; the terminal device has a positioning module , the positioning module can obtain the user's position in the three-dimensional virtual space to generate a position signal; the positioning module can transmit the position signal to the control module, and the control module can transmit the position signal through Graphical processing generates a location graphic and marks it on the map information. The control module displays the map information with the location graphic on the screen through the display module in conjunction with the three-dimensional virtual space. The virtual system with hand gesture control as described in claim 8, wherein the terminal device has a navigation module, the navigation module has a query unit and a guidance unit, and the query unit can present the map information, The map information has plural area information, and the query unit provides the user with feedback on the operation instructions generated by the operation image to select one of the area information. The guidance unit receives the selected area information and calculates and plans to generate a Virtual route information, the user can move to the selected area information in the three-dimensional virtual space based on the virtual route information. The virtual system with hand gesture control as described in claim 1 or 2, wherein the control module marks the skeleton image on the hand image through a data labeling program (LabelImg); the image operation processing is computer vision ( Computer Vision) combined with deep learning.

Images