TWI824859B - Virtual shopping gesture control system - Google Patents

Abstract

A virtual shopping gesture control system, which is operated on a terminal device connected to a head-mounted display, includes a database, a virtual module, a positioning module, a camera module, a processing module, and a financial module. The virtual module obtains the virtual spatial map of the shopping mall from the database to be displayed through the head-mounted display. The positioning module changes the location on the virtual spatial map in the shopping mall according to the control of the user. The camera module takes pictures of the user's hand. The processing module performs image calculations on the hand image for comparison with the hand gestures in the database, so as to obtain the matching hand gesture and the corresponding gesture operation instructions, and provides a control feedback to select virtual commodities or change the location on the virtual spatial map in the shopping mall.

Description

Virtual shopping gesture control system

The invention relates to the field related to online shopping, and in particular to a virtual shopping gesture control system.

With the rapid development of global information technology, the future will be an information age with the Internet and knowledge economy as the core. Modern people's lifestyles tend to be networked and intelligent, and shopping methods have gradually changed from the previous single on-site shopping to physical shopping. With the combination of experience and online shopping, there is no need to go to shopping malls or mass merchandisers to select items one by one when purchasing the required items. Instead, online shopping is the current mainstream of consumption. Online shopping is to connect to the terminal through a computer or smart mobile device and log in to the shopping website. You can choose products that meet your needs and have them delivered to your home in a short time. The consumption model of quickly shopping and receiving goods without going out provides modern people with a very convenient shopping environment.

In order to increase the authenticity of online shopping, related technology uses virtual reality to simulate a three-dimensional and highly realistic virtual shopping 3D space using computer technology. When the user wears a special display device (VR glasses) to enter the virtual shopping 3D space, After entering the space, users can use hand-held controllers to shuttle or interact in the virtual shopping 3D space environment.

However, it is impossible to interact with the virtual shopping 3D space without a hand-held controller, and there is still a gap in control between a hand-held controller and direct hand control. Users cannot interact with intuitive control, resulting in Inconvenience of use.

In order to solve the above problems, the present invention provides a virtual shopping gesture control system, which uses gesture control to perform contactless virtual control to provide intuitive control effects.

One embodiment of the present invention provides a virtual shopping gesture control system, which is installed on a server and operates on a terminal device as an application. The terminal device is connected to a head-mounted display, and the head-mounted display is worn by the user. Head, the virtual shopping gesture control system includes: a database, which stores a shopping mall virtual space scene map, a plurality of virtual products, a plurality of hand postures and a gesture operation command corresponding to each hand posture, wherein each virtual product is set in In the shopping mall virtual space scene graph, each virtual product has a label information. The label information includes a color label, a product name information and a product price information; a virtual module is coupled with the database, and the virtual module is obtained from the database. Read the shopping mall virtual space scene map, present the shopping mall virtual space scene map on the terminal device, and display it through the head-mounted display; a positioning module coupled with the virtual module, the positioning module uses the positioning sense of the terminal device The detection function is used for positioning, and the shopping mall virtual space scene map is imported, and the location of the shopping mall virtual space scene map is changed with the user's control; a photography module, which is coupled with the virtual module, and the photography module runs on the terminal device The photography function captures an image of the user's hand; a processing module is coupled to the database, positioning module and photography module. The processing module receives the hand image obtained by the photography module and performs image calculation processing. A skeleton feature is marked from the hand image, and the processing module compares the skeleton feature with each hand posture in the database to obtain one of the matching hand postures and the corresponding gesture operation command. The processing module will obtain The gesture operation command feedback control is used to select the virtual product or change the location of the scene map in the virtual space of the shopping mall.

Through the above, the present invention uses gesture control for non-contact virtual control to provide an intuitive control effect, thereby improving the inconvenience of conventional control that requires a controller.

In order to facilitate the explanation of the central idea of the present invention expressed in the above summary column, specific embodiments are hereby expressed. Various objects in the embodiments are drawn according to proportions, sizes, deformations or displacements suitable for illustration, rather than according to the proportions of actual components.

The directional terms mentioned in the present invention, such as "up", "down", "front", "back", "left", "right", "inside", "outside", "side", etc., are only diagrams. The direction of the formula; therefore, the direction terms used are used to illustrate and understand the present invention, rather than to limit the present invention, and are explained first.

Referring to Figures 1 to 5, the present invention provides a virtual shopping gesture control system 100, which is set up on a server and operates as an application on a terminal device 1. The terminal device 1 is connected to a head-mounted display 2. The head-mounted display 2 is worn on the head of the user 3; in the embodiment of the present invention, the terminal device 1 is a smart mobile device; the head-mounted display 2 has a hole 4 through which the camera lens 5 of the terminal device 1 can pass. Hole 4 is exposed.

The virtual shopping gesture control system 100 of the present invention includes:

A database 10, which stores a shopping mall virtual space scene map M, a plurality of virtual products V, a plurality of hand postures H and a gesture operation command corresponding to each hand posture H; each virtual product V is set in the shopping mall virtual space scene map M In the shopping mall virtual space scene graph M, there is a product area M1 and a checkout area M2. The product area M1 is equipped with each virtual product V, as shown in Figure 2; where each virtual product V has a label information, and the label information includes A color label, a product name information and a product price information, and the color label is used to represent the characteristic representative color of the appearance of the virtual product V.

A virtual module 20 is coupled to the database 10. The virtual module 20 reads the shopping mall virtual space scene map M from the database 10, and presents the shopping mall virtual space scene map M to the terminal device 1, and displays the shopping mall virtual space scene map M through the head-mounted device 1. Display 2 displays, as shown in Figure 1 and Figure 2.

A positioning module 30, which is coupled to the virtual module 20. The positioning module 30 uses the positioning sensing function of the terminal device 1 for positioning, and imports the shopping mall virtual space scene map M, and changes the location in the shopping mall with the control of the user 3. The location of the virtual space scene graph M.

A photography module 40, which is coupled to the virtual module 20. The photography module 40 runs the photography function of the photography lens 5 of the terminal device 1 and captures a hand image P of the user 3, as shown in Figures 1 and 3, The hand image P of the user 3 is used to select the virtual product V or change the location of the scene map M in the virtual space of the mall; in the embodiment of the present invention, the photography module 40 can control the photography function of the terminal device 1 to Dynamically track the hand image P of user 3 to achieve the effect of obtaining changes in the hand image P in real time.

A processing module 50, which is coupled to the database 10, the positioning module 30 and the photography module 40. The processing module 50 receives the hand image P obtained by the photography module 40 and performs image calculation processing. P marks a skeleton feature S (as shown in FIG. 4 ), and the processing module 50 compares the skeleton feature S with each hand posture H in the database 10 to obtain one of the matching hand postures H and the corresponding gesture. Operation instructions, the processing module 50 will feedback control the obtained gesture operation instructions to select the virtual product V or change the location of the scene map M in the virtual space of the mall.

Furthermore, the skeleton feature S has a plurality of feature points F and a plurality of feature lines L. Each feature point F corresponds to each knuckle of the hand and the junction of the hand and the wrist. Each feature line L is connected between two adjacent feature points F. time, as shown in Figure 4; wherein, each feature point F has an exclusive code, and the processing module 50 determines the positional relationship between each feature point F based on the exclusive code of each feature point F, so as to obtain a match from the database 10 One of the hand postures H.

For example: the skeleton feature S marked by the hand image P of the user 3 determines that the thumb of the user 3 fits the other fingers, which corresponds to the hand posture H and the gesture operation command of the database 10 as grasping; The skeleton feature S marked by the hand image P of the user 3 determines that the thumb of the user 3 is separated from the other fingers, which corresponds to the hand posture H and the gesture operation command of the database 10 to put down; the present invention does not limit the hand The shape of the head posture H.

A financial module 60, which is coupled to the database 10 and the processing module 50. The financial module 60 is arranged in the checkout area M2. The financial module 60 performs operations on the selected virtual commodity V located in the checkout area M2. The analysis and processing are as shown in Figures 1 and 5; among them, the financial module 60 performs image detection analysis and complex color tone testing on the selected virtual commodity V to obtain a color value of the virtual commodity V. The value is obtained from the database 10 by the matching color tag, and the corresponding tag information is obtained from the color tag, and the tag information records the product price information corresponding to the virtual product V. The financial module 60 is based on the Calculate a consumption amount based on the above product price information and provide the payment consumption amount.

In the embodiment of the present invention, the financial module 60 will capture the image of the virtual commodity V. When the financial module 60 acquires the image of the virtual commodity V, it may be caused by the different ambient light and angle of the shopping mall virtual space scene map M. The color values generated by the analysis of the image of the same virtual product V are different, so through different hue tests, it can be ensured that the image of the same virtual product V still maintains accurate recognition in different environments. Each hue test is red, blue or green. One or a combination of these hues is used as a color recognition test.

Furthermore, the color value of virtual product V can represent the characteristic representative color of the appearance of virtual product V. The color value of virtual product V is an HSV value. That is to say, the color value of each virtual product V includes hue, saturation and lightness. , and the color value can describe the color of the appearance of the virtual product V at a point in the cylindrical coordinate system.

In addition, when the financial module 60 cannot identify the virtual commodity V, the financial module 60 will perform color and brightness correction image processing on the image of the virtual commodity V, and the financial module 60 will analyze the corrected image of the virtual commodity V again. To obtain the color value of virtual product V.

An editing module 70, which is coupled to the database 10 and the photography module 40. The editing module 70 can display each hand posture H and the corresponding gesture operation command in the database 10 on the head-mounted display through the terminal device 1 2. The editing module 70 can select one of the hand postures H for editing, so as to change the selected hand posture H and the corresponding gesture operation command.

Furthermore, the editing module 70 can further activate the photography function of the terminal device 1 to capture a sample image, and send the sample image to the editing module 70 for image calculation processing to generate a hand image P marked with the skeleton feature S, and the editing module The group 70 can replace one of the hand postures H in the database 10 with the hand image P marked with the skeleton feature S, and set it to correspond to one of the operation instructions, or add it to the hand posture H and set the corresponding operation instruction. Therefore, , the user 3 can change the hand posture H and the corresponding operation instructions according to the operation habits.

Based on the above, the present invention can achieve the following effects:

1. The virtual shopping gesture control system 100 of the present invention can use gesture control to perform contactless virtual control to provide intuitive control effects.

2. The virtual shopping gesture control system 100 of the present invention identifies the virtual product V through image processing, analyzes the main color characteristics of the virtual product V, and uses the color as the label of the virtual product V, so as to provide fast and accurate processing of the product during subsequent checkout operations. be recognized, thereby reducing the computational burden of image processing, improving accuracy and fast checkout efficiency, and providing a convenient virtual shopping effect.

3. The virtual shopping gesture control system 100 of the present invention uses an intelligent image recognition method and identifies the color characteristics of the product. Even if the product is skewed within a certain angle, it can achieve a recognition rate of 99%; thereby, the probability of errors is greatly reduced. and shorten the checkout time for virtual goods V.

4. The virtual shopping gesture control system 100 of the present invention uses color as the main identification judgment. The financial module 60 can be recognized normally even when shadows, reflected light, tilt, bends, etc. appear on the image of the virtual commodity V. In the stacking of several images of virtual goods V, only the main color features of some of the images of virtual goods V are exposed, and comparisons can still be made from the fragmented images of virtual goods V.

The above embodiments are only used to illustrate the present invention and are not intended to limit the scope of the present invention. All modifications or changes that do not violate the spirit of the present invention fall within the scope of the invention.

1:Terminal device 2:Head mounted display 3:User 4:hole 5:Photography lens 100:Virtual shopping gesture control system 10:Database 20:Virtual module 30: Positioning module 40:Photography module 50: Processing module 60:Financial module 70:Edit Module H:Hand posture M: Shopping mall virtual space scene map M1: Commodity area M2: Checkout area V: virtual goods P: hand image S: skeleton features F: Feature points L: Characteristic line

Figure 1 is a block diagram of the system architecture of the present invention. Figure 2 is a schematic diagram (1) of an embodiment of the present invention, showing the connection between a terminal device and a head-mounted display. Figure 3 is a schematic diagram (2) of the embodiment of the present invention, showing hand image analysis. Figure 4 is a schematic diagram (3) of the embodiment of the present invention, showing the user wirelessly selecting virtual goods. Figure 5 is intention (4) of the embodiment of the present invention, showing the checkout of virtual goods.

1:Terminal device

2:Head mounted display

100:Virtual shopping gesture control system

10:Database

20:Virtual module

30: Positioning module

40:Photography module

50: Processing module

60:Financial module

70:Edit Module

Claims (9)

A virtual shopping gesture control system is set up on a server and operates on a terminal device as an application. The terminal device is connected to a head-mounted display. The head-mounted display is worn on the user's head. The virtual shopping The gesture control system includes: a database that stores a shopping mall virtual space scene map, a plurality of virtual products, a plurality of hand gestures and a gesture operation command corresponding to each hand gesture, wherein each virtual product is set in the shopping mall virtual In the spatial scene graph, each virtual product has a tag information, and each tag information includes a color tag, a product name information and a product price information; a virtual module is coupled to the database, and the virtual module The mall virtual space scene map is read from the database, and the mall virtual space scene map is presented to the terminal device and displayed through the head-mounted display; a positioning module coupled to the virtual module, The positioning module uses the positioning sensing function of the terminal device for positioning, and imports the virtual space scene map of the shopping mall, and changes the location of the virtual space scene map of the shopping mall with the user's control; a photography module, which is connected to the virtual space scene map of the shopping mall. The virtual module is coupled, and the photography module runs the photography function of the terminal device and captures an image of the user's hand; a processing module is coupled to the database, the positioning module and the photography module, The processing module receives the hand image obtained by the photography module and performs image calculation processing. The hand image is marked with a skeleton feature. The processing module combines the skeleton feature with each hand in the database. The gestures are compared to obtain one of the matching hand postures and the corresponding gesture operation command. The processing module feedback controls the acquired gesture operation command to select the virtual product or change the virtual product located in the mall virtual space. The location of the scene graph; and A financial module coupled to the database and the processing module; the financial module captures the image of the selected virtual product and performs image detection analysis and complex tone testing to obtain the image of the virtual product A color value, the financial module obtains the matching color tag from the database based on the color value, obtains the corresponding tag information from the color tag, and the tag information record contains the corresponding virtual product The financial module calculates a consumption amount based on the commodity price information and provides payment for the consumption amount. The virtual shopping gesture control system as described in claim 1, wherein each of the hue tests uses one of red, blue or green or a combination thereof as a color recognition test. The virtual shopping gesture control system as described in claim 2, wherein when the virtual product cannot be recognized, the processing module performs color and brightness correction image processing on the product image, and the processing module corrects the virtual product. The virtual product is parsed again to obtain the color value of the virtual product. The virtual shopping gesture control system of claim 3, wherein the color value of the virtual product can represent the characteristic representative color of the appearance of the virtual product. The virtual shopping gesture control system of claim 4, wherein the color value of the virtual product is an HSV value. The virtual shopping gesture control system as described in claim 1, wherein the skeleton feature has a plurality of feature points and a plurality of feature lines, each feature point corresponding to each knuckle of the hand and the junction of the hand and the wrist, and each feature line Connected between two adjacent feature points, each feature point has an exclusive code. The processing module determines the positional relationship between the feature points based on the exclusive code of each feature point to obtain a match from the database. One of the hand postures. The virtual shopping gesture control system as described in claim 1 or 6, wherein the photography model The group can control the photography function of the terminal device to dynamically track the hand image of the user. The virtual shopping gesture control system as described in claim 7, wherein there is an editing module coupled to the database and the photography module, and the editing module can convert each hand gesture in the database And the corresponding gesture operation instructions are displayed on the head-mounted display through the terminal device. The editing module can select one of the hand postures for editing to change the selected hand posture and the corresponding gesture operation instructions. The virtual shopping gesture control system as described in claim 8, wherein the shopping mall virtual space scene map has a commodity area and a checkout area, and the commodity area is equipped with each of the virtual commodities; the financial module is installed in the checkout area area, the financial module performs analysis and processing on the selected virtual commodity located in the checkout area.

Images