WO2016179819A1

WO2016179819A1 - Human-computer interaction method and system for trying on virtual hand accessory

Info

Publication number: WO2016179819A1
Application number: PCT/CN2015/078927
Authority: WO
Inventors: 周谆
Original assignee: 周谆
Priority date: 2015-05-14
Filing date: 2015-05-14
Publication date: 2016-11-17

Abstract

A human-computer interaction method and system for trying on a virtual hand accessory, achieving the technical effect of improving the accessory purchasing experience of a customer. The method comprises: importing a three-dimensional hand model and a three-dimensional accessory model respectively (S21); displaying the three-dimensional accessory model at a designated position of the three-dimensional hand model (S22); receiving depth information, acquired by a depth sensor in a perception space thereof, about a hand (S23); and calling, based on the depth information about the hand, a physical engine to adjust displaying of the three-dimensional hand model and the three-dimensional accessory model (S24). A three-dimensional accessory model and a three-dimensional hand model of one hand of a customer are constructed, and the three-dimensional accessory model is worn on the three-dimensional hand model; by means of relative movement between a wearing hand and an interaction hand, in conjunction with a physical engine, change of the position or angle of the three-dimensional accessory model can be virtually controlled, and adjustment of the positions and angles of the hand and an accessory in true try-on is simulated; and when the customer cannot truly try on the accessory, the effect of truly trying on the accessory is achieved.

Description

Human-computer interaction method and system for hand virtual jewelry fitting

Technical field

The present invention relates to the field of virtual reality technology, and in particular, to a human-computer interaction method and system for hand-made virtual jewelry trials.

Background technique

When purchasing jewelry, in addition to observing the style of the jewelry, the customer needs to try on the jewelry to determine whether the size of the jewelry is appropriate and whether the wearing effect is satisfactory.

technical problem

For valuable jewelry, there is a safety problem when the jewelry is taken out of the counter for the customer to try on; therefore, more expensive jewelry is not stored at the counter, but a picture or video of the jewelry is displayed for the customer to watch, and the customer intends to purchase At the time, the customer will go to the storage place of the jewelry, which is usually a special place where the security of the safe is more strict. This not only reduces the customer's purchasing experience, but also increases the labor cost of the merchant.

Or, even if it is not a valuable jewelry, because the merchant's jewelry style is numerous, it is impossible to put it on the counter, and it can only be used by customers for pictures or videos; especially in the current Internet, the sales of jewelry sold through the Internet. There are more ways. In this sales method, it is usually a picture showing the jewelry, or a wearing effect picture showing the wearing of the model. The customer uses the image data and some size information to measure whether to buy the jewelry, and the jewelry purchased in this way usually There are problems of improper wearing and unsatisfactory results, which all reduce the customer's buying experience.

Problem solution

Technical solution

The object of the present invention is to provide a human-computer interaction method and system for hand-made virtual jewelry trials, which realizes the technical effect of improving the purchase experience of customers purchasing jewelry.

The object of the invention is achieved by the following technical solutions:

A human-computer interaction method for hand virtual jewelry fitting is proposed, which comprises: respectively introducing a hand three-dimensional model and a three-dimensional model of the ornament; displaying the three-dimensional model of the accessory on the designated position of the hand three-dimensional model; Receiving depth information of the hand acquired by the depth sensor in the sensing space thereof; and calling the physics engine to adjust the display of the hand three-dimensional model and the jewelry three-dimensional model based on the depth information of the hand.

A man-machine interaction system for hand virtual jewelry fitting is proposed, comprising a depth sensor, an importing unit, a display unit, a physics engine and a control unit; the importing unit is configured to import a hand three-dimensional model and a three-dimensional model of the ornament; a depth sensor for acquiring depth information of the hand in the sensing space thereof; the control unit, configured to display the three-dimensional model of the accessory on a specified position of the three-dimensional model of the hand; and receiving and based on the hand Depth information, invoking the physics engine to adjust display of the hand three-dimensional model and the accessory three-dimensional model; the physics engine for simulating friction and collision between the hand three-dimensional model and the accessory three-dimensional model The display unit is configured to display the three-dimensional model of the hand and the three-dimensional model of the accessory.

Advantageous effects of the invention

Beneficial effect

The beneficial effects or advantages of the technical solution provided by the present invention are: using a hand virtual jewelry fitting method, in an intelligent master control device such as a computer, constructing a jewelry and a virtual hand 3D model and a 3D model of the jewelry, in the customer When you want to try on a certain accessory, import and display the 3D model of the jewelry and the 3D model of the hand of the customer's hand, and control the virtual jewelry to be worn on the virtual hand, and then use the depth sensor to obtain the depth of the customer's hand. Information, based on the depth information of the hand, obtains the movement parameters of the hand 3D model or the 3D model of the ornament, drives the 3D model of the hand and the 3D model movement of the ornament; when constructing the 3D model of the hand and the 3D model of the ornament, setting the 3D model of the hand and The parameters of the 3D model of the jewelry, such as material, weight, resistance, collision mechanism, etc., and set the collider on the 3D model of the hand and the 3D model of the ornament. For the 3D model of the ornament, the force is applied to the 3D model of the ornament according to the moving parameter, so that the 3D ornament is made. The model moves relative to the hand 3D model and calls the physics engine simulation The collision and friction between the movement and the accessories during the movement, so that the movement between the three-dimensional models of the virtual world conforms to the physical laws of the real world, thereby realizing the effect of adjusting the display of the three-dimensional model of the hand and the three-dimensional model of the jewelry, for example, according to the hand The action adjusts the display angle of the virtual hand and the virtual jewelry, so that the customer can understand whether the size of the jewelry and the hand is appropriate, whether the effect of the jewelry worn on the hand is good or not, etc., because the physical engine is used to simulate the hand and the jewelry. The collision and friction between the three-dimensional model of the hand and the relative movement of the three-dimensional model of the jewelry are more realistic and realistic, and the effect of the customer's real try-on jewelry is generated. When you can actually wear jewelry, you can improve the purchasing experience of customers trying on jewelry.

Brief description of the drawing

DRAWINGS

1 is a schematic view of a depth sensor sensing space;

2 is a flowchart of a method for human-computer interaction of a hand virtual jewelry trial according to an embodiment of the present invention;

3 is a schematic diagram of relative movement of a wearing hand and an interactive hand according to an embodiment of the present invention;

4 is a schematic diagram of relative movement of a wearing hand and an interactive hand according to an embodiment of the present invention;

FIG. 5 is a flowchart of still another method for human-computer interaction of a hand virtual jewelry trial according to an embodiment of the present invention; FIG.

FIG. 6 is a flowchart of still another method for human-computer interaction of a hand virtual jewelry trial according to an embodiment of the present invention; FIG.

FIG. 7 is a block diagram of a human-computer interaction system for hand virtual jewelry trials according to an embodiment of the present invention.

Invention embodiment

Embodiments of the invention

The present invention constructs a three-dimensional model of a jewelry and a three-dimensional model of a hand by using a virtual try-on method by using a human-computer interaction method and system for hand-made virtual jewelry fitting, that is, constructing a virtual jewelry and a virtual hand. The virtual hand is used to wear the virtual jewelry, and the depth information of the hand is obtained through the depth sensor. Based on the depth information, the physical engine is invoked to adjust the display relationship of the virtual hand wearing the virtual jewelry, which is the same as the real hand wearing the real jewelry. Effect.

The current depth sensor can obtain information such as bone information, spatial three-dimensional coordinates and speed of the hand, including the palm, the wrist and each finger. For example, the kinect sensor can acquire the hand at a frequency of several tens of frames per second. The information, while the LeapMotion sensor can be acquired at a frequency of more than one hundred frames per second.

As shown in FIG. 1 , the depth sensor 1 is placed on the front end of the display interface 21 of a device such as a computer, and the perceptible range forms a sensing space 4 in front of the display interface that can sense the palm, the wrist, and each finger. When the hand is operated in this perceptual space, the three-dimensional coordinates and speed information (including the velocity and speed direction) of the palm, wrist and each finger can be sensed by the depth sensor, and the depth sensor calculates the bone information supply of the hand based on the information. Use; bone information includes palm, wrist, finger And the bone size, speed, direction and other information of each finger section. Of course, in the case where the depth sensor is wirelessly connected to a device such as a computer, the depth sensor is not necessarily limited to being placed around the device such as a computer, but is placed according to the spatial position of the actual hand operation, and it is not necessary to limit the hand to a fixed position. Gesture operation. The display interface can be, but is not limited to, a television screen, a projection screen, a computer screen, a head mounted 3D display system, and the like.

The depth information acquired by the depth sensor mentioned in the embodiment of the present invention generally refers to the bone information of the hand calculated by the depth sensor after acquiring the hand depth information.

The accessories mentioned in the embodiments of the present invention include, but are not limited to, rings, watches, bracelets, bracelets and the like.

The technical solutions provided by the embodiments of the present invention are described in detail below with reference to the accompanying drawings.

FIG. 2 is a flowchart of a method for human-computer interaction of a virtual jewelry try-on according to an embodiment of the present invention, where the method includes the following steps:

Step S21: respectively importing a three-dimensional model of the hand and a three-dimensional model of the jewelry;

Before importing the 3D model of the hand and the 3D model of the ornament, we must first construct the 3D model of the hand and the 3D model of the ornament.

The construction of the three-dimensional model of the jewelry can be carried out by any existing method for constructing the three-dimensional model; it can be constructed according to the actual existing jewelry, or can be constructed according to the jewelry design concept or drawing; in the Internet e-commerce, Customers can even create and build 3D models of their own jewelry and upload them to suppliers.

For multiple accessories, it is necessary to construct a three-dimensional model of multiple ornaments, and store all the three-dimensional models of the jewelry in a three-dimensional model library. When importing the three-dimensional model of the jewelry, the three-dimensional model of the target jewelry is selected from the three-dimensional model library of the jewelry and imported. The target three-dimensional model of the jewelry; the customer can use the existing methods, such as mouse point drag, or gesture control, etc., to select the three-dimensional model of the Italian jewelry from the jewelry three-dimensional model library.

Since the customer is constantly changing, the customer's hand 3D model is constructed in real time. As shown in Figure 1, in the sensing space 4 of the depth sensor 1, the customer's hand information is acquired by the depth sensor, including the hand. Bone information (including wrists, palms, and fingers) that includes bone width information and bone length information, ie the width of the wrist bone, the width and length of the palm bone, the width and length of each finger (including the individual knuckles) ) and other information, based on the depth sensor acquired these bones The skeleton information can construct the 3D model of the client's hand in real time. Of course, a hand 3D model library can also be created for the hand 3D model, and the constructed 3D model of the client hand can be stored, when the customer tries to wear the jewelry next time. It is no longer necessary to rebuild the 3D model of the hand, which is especially suitable for old customers and enhance the customer's buying experience.

In order to make the hand 3D model more suitable for the customer's hand characteristics, when constructing the 3D model of the hand, the 3D model of the hand is constructed in two steps. First, the hand model is built, and then the bone is built in the hand model. The model processes the hand model and the bone model so that the depth sensor can capture the depth information of the hand in the perceptual space, process the corresponding bone information, and associate the bone information with the skeleton model to construct the bone information. The three-dimensional model of the hand. Specifically, the bone information is used to drive the skeleton model, so that the hand in the display interface can be synchronized with the actual hand motion, including the displacement, rotation, finger click, telescopic, etc. of the hand; The information stretches or shrinks the size of the bone model, making the bone model more in line with the personal characteristics of the customer's hand bone.

After importing the 3D model of the hand and the 3D model of the jewelry, they are displayed on the display interface.

Step S22: displaying the three-dimensional model of the ornament on the designated position of the three-dimensional model of the hand;

In order to simulate the situation of hand-wearing jewelry in the real situation, after importing the three-dimensional model of the ornament and the three-dimensional model of the hand and displaying it, the situation of the hand-trimmed jewelry is simulated, so that the three-dimensional model of the ornament is displayed at the designated position of the hand three-dimensional model. .

A specific implementation manner is to select a position on the hand three-dimensional model that needs to be worn, and display the three-dimensional model of the jewelry at this position. For example, selecting one of the fingers of the hand three-dimensional model displays the three-dimensional model of the ring on the finger portion; or, after selecting the wrist portion in the three-dimensional model of the hand, displaying a three-dimensional model of the jewelry such as a bracelet or a watch on the wrist On the part. The depth sensor can obtain the bone information of the wrist, the palm and each finger. In addition to the information such as the width and length of the bone, including the position information of the wrist, the palm and each finger, the three-dimensional model of the wearing hand is selected to wear the three-dimensional jewelry. After the specific position of the model, the specific three-dimensional coordinates of the wearing position are determined, and the three-dimensional model of the jewelry can be displayed at this position; the specific ring, bracelet, watch and other accessories can adopt the center point position of the three-dimensional model and The way the wrist bones or the center of the finger bones coincide with each other to achieve the effect of attaching the ring and bracelet to the finger or wrist.

Although the 3D model of the jewelry is displayed on the 3D model of the hand, the vision of the hand wearing jewelry can be realized. Sense effect, but there is no actual contact or collision between the 3D model of the hand and the 3D model of the ornament. The 3D model of the ornament floats on the 3D model of the hand, making it difficult for the user to feel the real wearing feeling. When the three-dimensional model of the hand and the three-dimensional model of the ornament are displayed, it is easy to occur that the three-dimensional model of the hand coincides with the edge of the three-dimensional model of the ornament.

In order to make the effect of wearing the virtual model between the virtual models more realistic and realistic, the physics engine is used to simulate the collision and friction between the three-dimensional model of the hand and the three-dimensional model of the jewelry, so as to realize the effect of the virtual wearing of the jewelry by the customer. Bring a realistic interactive experience. Physical engines such as Nvidia's PhysX, etc., use CPU, GPU and other resources to accurately simulate the simulation effects of gravity, friction, collision, etc. of objects in three-dimensional space.

After constructing the 3D model of the hand and the 3D model of the ornament, the attributes of the 3D model of the hand and the 3D model of the ornament are respectively set. The setting of the property includes setting the weight, resistance, collision mechanism, material, environment variable, background sound and the background sound of the model. Lights, etc.; and set up collision devices on the 3D model of the hand and the 3D model of the jewelry, set the collision conditions, establish the parameters of the 3D model of the hand and the 3D model of the jewelry; after completing the above settings, import the 3D model and accessories of the hand. The 3D model displays the 3D model of the ornament at a specified position on the 3D model of the hand.

After displaying the 3D model of the jewelry on the 3D model of the hand, the details of the 3D model of the jewelry, such as the circumference of the ring, the length of the watch strap, etc., can be subtle based on the bone information, that is, according to the width of the client's hand. Adjustment, so that the three-dimensional model of the jewelry can be just nested at the designated position of the hand.

Step S23: receiving depth information of the hand acquired by the depth sensor in the sensing space thereof;

When the customer tries on the jewelry, it is not only to try to see if the jewelry is suitable, but also to see if the jewelry and the hand are aesthetically pleasing. When the real try-on, the angle of the hand or the jewelry is adjusted, and the jewelry is worn from all angles. effect.

In response to this, after the hand 3D model and the 3D model of the ornament are imported, and the 3D model of the ornament is displayed at the designated position of the 3D model of the hand, the depth sensor acquires the depth information of the hand in real time, including the 3D coordinates of the hand and Speed and other information, the customer's hand in the depth sensor's perception space, like the real wear jewelry to adjust the angle of the hand, in order to achieve the virtual system simulation of the real wear test.

Step S24: Calling the physics engine to adjust the three-dimensional model of the hand and the three-dimensional ornament based on the depth information of the hand The display of the model.

The depth sensor acquires the depth information of the hand in its perceptual space, including the bone information of the palm, the wrist and each finger, and the bone information can reflect the characteristic information of the hand; the depth information also includes the three-dimensional coordinates of the palm, the wrist and each finger and Velocity information, which can accurately capture the position, posture and motion of the hand; in the sensing space of the depth sensor, acquire the depth information generated by the movement, flipping, etc. of the hand at a certain frequency, based on the depth information and the above characteristic information The reaction to the display interface is to perform the corresponding action for the hand of the virtual simulation.

Based on the depth information acquired by the depth sensor, after constructing the 3D model of the hand based on the bone information, the motion information of the hand is acquired at a certain frequency, thereby controlling the motion of the 3D model of the hand in the display interface, so that the motion can be made within the sensing space. The hand feature information, the motion information and the hand 3D model feature information and the motion information in the display interface are in one-to-one correspondence, thereby simulating the effect of the real hand motion.

At the same time, since the collider is set in the 3D model of the hand and the 3D model of the ornament, and the related settings are set respectively, the physics engine is called to simulate the collision and friction between the 3D model of the hand and the 3D model of the ornament, so that the 3D model of the hand is made. Real contact with the three-dimensional model of the jewelry, as in reality, the three-dimensional model of the jewelry acts on the specified position of the three-dimensional model of the hand by applying external force, gravity, friction and other factors, providing the customer with a virtual trial wear. User experience with real try-on.

For example, according to the movement of the hand, the display angle of the three-dimensional model of the hand and the three-dimensional model of the ornament is adjusted, so that the customer can understand whether the effect of the jewelry worn on the hand is good or not, and the effect of the customer actually wearing the jewelry is generated, and the customer cannot be true. When you try on accessories, you can improve the purchasing experience of customers trying on jewelry.

The customer can set whether the jewelry is displayed on the left or right hand, or on the finger or on the wrist, as shown in Figure 1. The customer can select the three-dimensional model of the jewelry from the three-dimensional model library of the jewelry, or replace the three-dimensional model of the jewelry during the virtual fitting process; the selection method can adopt an existing selection method such as a keyboard, a mouse or a touch screen, or can be controlled by gestures. Selecting and replacing the three-dimensional model of the jewelry; when selecting by gesture, specifically, receiving the depth information of the hand acquired by the depth sensor in the sensing space thereof, and analyzing the gesture corresponding to the hand based on the depth information of the hand, for example, the palm is sensing A space waving in one direction indicates that the next three-dimensional model of the ornament is displayed, and waving in the other direction indicates that the three-dimensional model of the previous ornament is displayed.

A specific embodiment in which the physics engine is invoked to adjust the display of the hand 3D model and the 3D model of the jewelry is According to the change of the hand depth information, the display of the 3D model of the hand is synchronized, so that the hand 3D model is synchronized with the movement of the hand of the customer in the sensing space, and then the physics engine is invoked to simulate the friction between the 3D model of the hand and the 3D model of the accessory. Collision makes the three-dimensional model of the jewelry synchronously displayed by the frictional force and the three-dimensional model of the hand, including synchronous movement, synchronous rotation, etc., thus realizing the effect of wearing the jewelry in the hand.

Specifically, when both hands of the client are in the sensing space of the depth sensor, the corresponding hand 3D model can be distinguished as the wearing hand 3D model and the interactive hand 3D model, as shown in FIG. 3 and FIG. 4, the left hand is Wear your hands and your right hand is an interactive hand.

Then, the following situations are simulated: the left hand wears the jewelry, and the right hand adjusts the position or the angle of the jewelry. The following specific methods are proposed in the embodiment of the present invention; of course, the hand and the interactive hand can also be used according to the actual hand habit of the customer. Make settings, not to repeat them here.

Based on the depth information of the hand, the physics engine is called to adjust the display of the hand 3D model and the 3D model of the jewelry, which may be the steps shown in FIG. 5:

Step S51: determining whether the state of wearing the three-dimensional model of the hand changes based on the depth information of the hand;

In the case of real wear, the wearing hand is usually kept still, and the position and angle of the jewelry are adjusted by the other hand. Correspondingly, in the virtual system, if the state of wearing the hand three-dimensional model remains unchanged, that is, the depth information of the hand of the wearing hand does not change, reflecting the state in which the wearing hand of the customer remains stationary, indicating the customer The position or angle of the jewelry will be adjusted soon. then

Step S52: determining, according to the depth information of the hand, whether there is relative motion between the wearing hand three-dimensional model and the interactive hand three-dimensional model;

When there is relative motion between the wearing hand three-dimensional model and the interactive hand three-dimensional model, that is, the relative movement between the wearing hand and the interactive hand is reflected; and in the state in which the wearing hand remains stationary, if the wearing hand and the interactive hand are in relative motion, It is inevitable that the interaction hand moves, that is, the depth information of the hand of the interaction hand changes; and the movement of the interaction hand corresponds to the adjustment of the position or angle of the jewelry, then

Step S53: Applying force to the three-dimensional model of the jewelry, and calling the physics engine to simulate the friction and collision between the three-dimensional model of the hand and the three-dimensional model of the ornament, so that the three-dimensional model of the ornament moves relative to the three-dimensional model of the hand.

When the real wear is performed, the different movements of the interactive hand correspond to the different adjustments of the jewelry, and when the interactive hand three-dimensional model occurs, the position of the three-dimensional model of the accessory needs to be adjusted according to the different state of motion, and displayed. For example, moving the position of the three-dimensional model of the jewelry, or adjusting the angle of the three-dimensional model of the jewelry, and the like.

Based on the depth information of the hand, a specific implementation for determining whether there is relative motion between the wearing hand 3D model and the interactive hand 3D model is as shown in FIG. 3, based on the hand depth information, including the interactive hand (corresponding to the right hand) and the wearing hand ( Corresponding to the depth information of the left hand, the direction information of the finger or the wrist in the three-dimensional model of the wearing hand is obtained, and it is determined whether the three-dimensional model of the interactive hand is translationally moved relative to the direction of the finger or the wrist in the three-dimensional model of the wearing hand, which corresponds to the relative hand of the interactive hand. The finger or wrist direction shifts and moves; if it is judged that the interactive hand 3D model is moved relative to the direction of the finger or wrist in the 3D model of the wearing hand, the action is correspondingly adjusted to adjust the jewelry 3D model in the wearing hand 3D model finger Or the displacement on the wrist and display it; corresponding to the actual wearing situation, adjust the position of the ring on the finger, adjust the position of the watch on the wrist, etc.; the translational movement includes the translation of the entire hand, and the translation of only the finger; When implemented, the force is applied to the three-dimensional model of the ornament, so that the three-dimensional model of the jewelry is relatively The three-dimensional model moves in the direction of the finger or wrist. At the same time, the physics engine is invoked to simulate the friction and collision between the three-dimensional model of the ornament and the three-dimensional model of the hand, so that the three-dimensional model of the accessory combines the applied force, friction and collision to simulate the real wearing. The situation is translated and moved, which makes the virtual try-on and the real try-on experience more consistent, and improves the customer's purchase experience.

Based on the depth information of the hand, another specific implementation for determining whether there is relative motion between the wearing hand 3D model and the interactive hand 3D model is as shown in FIG. 4, based on the hand depth information, including the interactive hand (corresponding to the right hand) 3D model. Wearing the depth information of the hand (corresponding to the left hand), obtaining the direction information of the finger or the wrist in the three-dimensional model of the wearing hand, and determining whether the three-dimensional model of the interactive hand is rotated relative to the direction of the finger or the wrist in the three-dimensional model of the wearing hand, which corresponds to rotation Wrist or jewelry on the finger; if it is judged that the interactive hand 3D model is rotated relative to the direction of the finger or wrist in the 3D model of the wearing hand, the action is corresponding to the adjustment of the jewelry 3D model in the wearing hand 3D model in the finger or The angle on the wrist is displayed; corresponding to the actual wearing situation, the angle of the ring on the finger is adjusted, and the angle of the watch on the wrist is adjusted; in the specific implementation, the force is applied on the three-dimensional model of the ornament, so that the three-dimensional model of the accessory is relatively The three-dimensional model of the hand rotates around the finger or wrist, and at the same time, the caller The engine simulates the friction and collision between the three-dimensional model of the ornament and the three-dimensional model of the hand, so that the three-dimensional model of the accessory combines the force, friction and collision factors, and simulates the real wearing situation for vertical movement, so that the virtual try-on and the real try-on The experience is more consistent and improves the customer's buying experience.

Before the virtual wear jewelry, the customer can only choose to import the three-dimensional model of the jewelry, and adjust the display of the three-dimensional model of the jewelry through operations of the input device such as a mouse or a keyboard, such as flipping, zooming, and the like.

The customer can also control the flipping or zooming of the 3D model of the jewelry based on the depth sensor based on the gesture, as shown in Figure 6:

Step S61: receiving depth information of the hand acquired by the depth sensor in the sensing space thereof;

Step S62: analyzing, according to the depth information of the hand, a gesture corresponding to the hand;

Step S63: Control selection, rotation, movement or zooming of the three-dimensional model of the ornament based on the gesture.

Based on the depth information of the hand, the three-dimensional coordinates and velocity information of the hand are obtained; and the displacement of the hand is analyzed according to the three-dimensional coordinates of the hand, and the moving direction of the hand is analyzed according to the speed information of the hand.

The depth sensor collects the depth information of the hand change in the sensing space. Therefore, the three-dimensional coordinates in the acquired depth information form the motion displacement of the hand on the time axis, and the speed direction and the velocity in the velocity information are in the space. The direction of movement and speed of movement of the hand are constructed.

Based on the movement of the hand, the gesture corresponding to the hand movement can be calculated; specifically, combined with the movement displacement of the hand and the movement direction and the movement speed, a gesture model of the hand occurring in the space can be constructed, and different gesture models correspond to different gestures. Human-computer interaction instructions, for example, selecting a three-dimensional model of the ornament from the three-dimensional model library of the jewelry, flipping the three-dimensional model of the ornament, and enlarging or reducing the three-dimensional model of the ornament, and the corresponding relationship between the specific gesture and the human-computer interaction command can be determined according to the actual situation of the game. This program is not limited.

When the virtual try-on jewelry, the customer adjusts the size of the three-dimensional model of the jewelry by the above method, for example, using a mouse or a keyboard to input parameters to change the size of the three-dimensional model of the jewelry, and for example, the distance generated by the relative movement of the two fingers of the interactive hand. The size changes to adjust the size of the 3D model of the jewelry, etc.; the advantage of this is that the customer can customize the size of the jewelry suitable for his hand according to the size of the hand, not limited to the size of the existing jewelry, thereby improving the customer's Purchase experience. Moreover, the user can complete the selection and purchase process of the jewelry by himself, thereby reducing the labor cost of the jewelry merchant.

Based on the human-computer interaction method of the virtual jewelry trial, the embodiment of the present invention further provides a human-computer interaction system for virtual jewelry trial, as shown in FIG. 7, including a depth sensor 1, a display unit 2, an import unit 3, and a physics. The engine 4 and the main control unit 5; the depth sensor is used to acquire the depth information of the hand in the sensing space; the importing unit is used to respectively introduce the hand three-dimensional model and the three-dimensional model of the jewelry; the main control unit displays the three-dimensional model of the jewelry on the hand The specified position of the 3D model, and the receiving depth sensor is obtained in its perceptual space Taking the depth information of the hand, based on the depth information of the hand, calling the physics engine to adjust the display of the hand 3D model and the 3D model of the ornament; the display unit displays the 3D model of the hand and the 3D model of the ornament; the physics engine is used to simulate the 3D of the hand Friction and collision between the model and the 3D model of the ornament.

The above system further comprises a building unit 5 and a setting unit 7; the building unit is used for constructing a three-dimensional model of the jewelry of the accessory and a three-dimensional model of the hand of the client. Specifically, the constructed three-dimensional hand model includes a hand model and a bone model in the hand model, and the control unit receives the depth information of the hand acquired by the depth sensor in the sensing space thereof, and is based on the depth information analysis of the hand. The skeleton information of the hand, the construction unit associates the bone information with the skeleton model, and constructs a three-dimensional model of the hand of the hand; wherein the bone information includes bone width information and bone length information of the wrist, the palm, and each finger; specifically, the building unit Based on the bone information, the size of the skeleton model is stretched or contracted, and after the three-dimensional model of the ornament is displayed at a specified position of the hand three-dimensional model, the stoneware of the three-dimensional model of the ornament is adjusted based on the skeleton information, so that the three-dimensional model of the ornament is more realistic. The sleeve is attached to the 3D model of the hand.

The setting unit 7 sets the three-dimensional model of the hand and the attributes of the three-dimensional model of the ornament, respectively sets the collider on the hand three-dimensional model and the three-dimensional model of the ornament, sets the collision condition of the collider, and establishes the three-dimensional model of the hand and the three-dimensional model of the ornament. The parameters of the real hand wearing jewelry are based on these settings when simulating the customer's try-on jewelry.

Corresponding to the three-dimensional model of the jewelry, the above system further comprises a three-dimensional model library 6 for storing the three-dimensional model of the jewelry for the customer, for the customer to select the three-dimensional model of the jewelry from the three-dimensional model library of the jewelry for virtual trial wear.

The customer can select the three-dimensional model of the jewelry from the three-dimensional model library of the jewelry by means of gesture control, and can also select, zoom, move or rotate the three-dimensional model of the jewelry after the import unit imports the three-dimensional model of the jewelry based on the gesture control manner; After receiving the depth information of the hand acquired by the depth sensor in the sensing space, analyzing the gesture corresponding to the hand based on the depth information of the hand, and controlling the zooming, moving, rotating or different jewelry of the three-dimensional model of the jewelry based on the difference of the gesture Switching between 3D models, etc.

After importing the 3D model of the hand and the 3D model of the jewelry, the main control unit wears the 3D model of the ornament on the 3D model of the hand, specifically: selecting the position of the 3D model of the accessory to be worn on the 3D model of the hand; The model is shown at this location.

To simulate the wearing hand of the jewelry and the interactive hand of adjusting the position and angle of the jewelry in the scene of the real try-on jewelry, the three-dimensional model of the hand is constructed to include the three-dimensional model of the wearing hand and the three-dimensional model of the interactive hand.

Then simulate the following situation of real hand wearing jewelry: the left hand wears the jewelry, and the right hand adjusts the position or angle of the jewelry, which can correspond to the wearing hand remaining, and the state of the interactive hand relative to the wearing hand changes; specifically, based on the acquired The depth information of the hand determines whether the state of the wearing hand 3D model changes; if not, based on the acquired depth information of the hand, it is determined whether there is relative motion between the wearing hand 3D model and the interactive hand 3D model; The model applies force and calls the physics engine to simulate the collision and friction between the three-dimensional model of the ornament and the three-dimensional model of the hand, so that the three-dimensional model of the ornament moves relative to the three-dimensional model of the hand.

Corresponding to the position of the interactive hand to adjust the jewelry, the three-dimensional model of the wearing hand and the three-dimensional model of the interactive hand have relative motion, and the moving hand 3D model can be moved relative to the finger or wrist direction of the three-dimensional model of the hand, and the finger or wrist in the three-dimensional model of the hand is worn. The direction information is obtained based on the depth information of the hand, including the interactive hand (corresponding to the right hand) and the wearing hand (corresponding to the left hand); if it is judged that the interactive hand 3D model is translational movement relative to the direction of the finger or wrist in the wearing hand 3D model The action is corresponding to adjusting the displacement of the three-dimensional model of the jewelry on the finger or the wrist in the three-dimensional model of the wearing hand, and displaying it; corresponding to the actual wearing situation, adjusting the position of the ring on the finger, adjusting the wristwatch on the wrist Positioning, etc.; translational movement includes translation of the entire hand, and translation of only the finger; in specific implementation, a force is applied on the three-dimensional model of the ornament, so that the three-dimensional model of the ornament is moved relative to the finger or wrist of the hand three-dimensional model, and at the same time, The physics engine simulates between the 3D model of the ornament and the 3D model of the hand. Friction and collision make the three-dimensional model of the jewelry combined with the applied force, friction and collision factors to simulate the real wearing situation for translational movement, which makes the virtual try-on and the real try-on experience more consistent, and improves the customer's purchasing experience.

Corresponding to the angle of the interactive hand adjustment ornament, the three-dimensional model of the wearing hand and the three-dimensional model of the interactive hand have relative motion, and the three-dimensional model of the interactive hand can be rotated relative to the direction of the finger or the wrist in the three-dimensional model of the wearing hand. The hand three-dimensional model is rotated relative to the direction of the finger or the wrist in the three-dimensional model of the wearing hand, and the action is correspondingly adjusted to adjust the angle of the three-dimensional model of the accessory on the finger or the wrist in the three-dimensional model of the wearing hand, and is displayed; corresponding to the actual wearing In the case, it is to adjust the angle of the ring on the finger, adjust the angle of the watch on the wrist, etc.; in the specific implementation, apply on the three-dimensional model of the jewelry The force causes the three-dimensional model of the ornament to move vertically relative to the finger or wrist direction of the hand three-dimensional model. At the same time, the physics engine is invoked to simulate the friction and collision between the three-dimensional model of the ornament and the three-dimensional model of the hand, so that the three-dimensional model of the accessory combines the applied force and friction. As well as collision and other factors, the simulation of the real wearing situation to rotate, making the virtual try-on and the real try-on experience more consistent, improve the customer's purchase experience.

When the customer only places the wearing hand in the sensing space of the depth sensor, the customer can view the fitting effect from different angles by flipping or moving the wearing hand. The most direct way is that according to the change of the hand depth information, Synchronize the display of the 3D model of the hand and the 3D model of the ornament; that is, synchronize the display of the 3D model of the ornament with the display of the 3D model of the hand, and then call the physics engine to simulate the friction and collision between the 3D model of the hand and the 3D model of the accessory. The three-dimensional model of the jewelry is synchronously displayed by the frictional force and the three-dimensional model of the hand, including synchronous movement, synchronous rotation, etc. The effect of the reaction is that when the customer adjusts the position and angle of the wearing hand, the accessory moves following the movement of the wearing hand. In order to observe the wearing effect of the jewelry and the hand at different angles.

The working method of the human-computer interaction system tried by the specific virtual jewelry has been detailed in the human-computer interaction method of the virtual jewelry trial, and will not be described here.

The human-computer interaction method and system for virtual jewelry trials proposed in the embodiments of the present invention construct a three-dimensional model of the jewelry and a three-dimensional model of the hand of the client, and wear the three-dimensional model of the jewelry on the three-dimensional model of the hand, and adopt the jewelry. The virtual try-on method of the hand realizes the effect of the real try-on jewelry; and realizes the change of the position or angle of the virtual control jewelry three-dimensional model by wearing the relative motion of the hand and the interactive hand, and simulates the real hand-adjusting adjustment of the hand And the position and angle of the jewelry, when the customer can not really try on the jewelry, through the virtual try-on, to achieve the real try-on effect, while introducing the physics engine to simulate the collision and friction between the hand 3D model and the 3D model of the jewelry, Make the virtual wear more realistic, and enable the customer to produce a sense of experience and experience consistent with the real try-on.

Claims

The human-computer interaction method for the hand virtual jewelry trial is characterized by comprising:

Import the hand 3D model and the 3D model of the jewelry separately;

Displaying the three-dimensional model of the ornament on a specified position of the three-dimensional model of the hand;

Receiving depth information of the hand acquired by the depth sensor in its perceptual space;

Based on the depth information of the hand, the physics engine is invoked to adjust the display of the three-dimensional model of the hand and the three-dimensional model of the accessory.

The human-computer interaction method of the hand virtual jewelry fitting according to claim 1, wherein before the manual introduction and display of the three-dimensional model of the hand and the three-dimensional model of the ornament, the method comprises:

Constructing the hand three-dimensional model and the three-dimensional model of the jewelry respectively;

Setting the three-dimensional model of the hand and the attributes of the three-dimensional model of the accessory;

Providing a collider on the hand three-dimensional model and the jewelry three-dimensional model respectively;

Setting a collision condition of the collider, and establishing parameters of the hand three-dimensional model and the three-dimensional model of the accessory.

The human-computer interaction method for hand virtual jewelry fitting according to claim 2, wherein the hand three-dimensional model comprises a hand model and a skeleton model built in the hand model;

The three-dimensional model of the hand is constructed, specifically:

Receiving depth information of the hand acquired by the depth sensor in its perceptual space;

Obtaining bone information of the hand based on the obtained depth information of the hand; wherein the bone information includes bone width information and bone length information of the wrist, the palm, and each finger;

A three-dimensional model of the hand of the hand is constructed by associating the bone information with the skeleton model.

The human-computer interaction method of the hand virtual jewelry fitting according to claim 3, wherein the associating the skeleton information with the skeleton model is specifically:

Stretches or shrinks the size of the bone model based on bone information.

The human-computer interaction method of the hand virtual jewelry fitting according to claim 3, wherein after the three-dimensional model of the accessory is displayed on the designated position of the hand three-dimensional model, the method further includes:

The size of the three-dimensional model of the ornament is adjusted based on the bone information.

The human-computer interaction method of the hand virtual jewelry fitting according to claim 2, wherein the calling physics engine adjusts the display of the hand three-dimensional model and the three-dimensional model of the accessory, specifically:

Synchronizing the display of the three-dimensional model of the hand according to the change of the depth information of the hand;

The physics engine is invoked to simulate friction and collision between the hand three-dimensional model and the jewelry three-dimensional model, so that the jewelry three-dimensional model is displayed synchronously with the hand three-dimensional model.

The human-computer interaction method for hand virtual jewelry fitting according to claim 1, wherein the hand three-dimensional model comprises a three-dimensional model of wearing a hand and a three-dimensional model of an interactive hand;

Then, based on the depth information of the hand, the physics engine is invoked to adjust the display of the three-dimensional model of the hand and the three-dimensional model of the accessory, specifically:

Determining whether the state of the three-dimensional model of the wearing hand changes based on the depth information of the hand; if not,

Determining, according to the depth information of the hand, whether the three-dimensional model of the wearing hand and the three-dimensional model of the interactive hand have mutual motion;

If yes, applying a force to the three-dimensional model of the ornament, and calling the physics engine to simulate friction and collision between the three-dimensional model of the hand and the three-dimensional model of the accessory, so that the three-dimensional model of the accessory is compared with the three-dimensional model of the hand mobile.

The human-computer interaction method of the hand virtual jewelry fitting according to claim 7, wherein the determining whether the three-dimensional model of the wearing hand and the three-dimensional model of the interactive hand exist according to the depth information of the hand Relative movement, specifically:

Obtaining a finger or a hand in the three-dimensional model of the wearing hand based on the depth information of the hand Direction information of the wrist;

Determining whether the interactive hand three-dimensional model is translationally moved relative to a direction of a finger or a wrist in the three-dimensional model of the wearing hand; if yes,

The calling physics engine adjusts the display of the three-dimensional model of the hand and the three-dimensional model of the accessory, specifically: applying a force to the three-dimensional model of the accessory, and calling the physics engine to simulate the three-dimensional model of the hand and the The friction and collision between the three-dimensional models of the jewelry causes the three-dimensional model of the ornament to move in translation relative to the direction on the finger or wrist in the three-dimensional model of the wearing hand.

The human-computer interaction method of the hand virtual jewelry fitting according to claim 7, wherein the determining whether the three-dimensional model of the wearing hand and the three-dimensional model of the interactive hand exist according to the depth information of the hand Relative movement, specifically:

Obtaining direction information of a finger or a wrist in the three-dimensional model of the wearing hand based on the depth information of the hand;

Determining whether the interactive hand three-dimensional model is rotated relative to a finger or a wrist in the wearing hand three-dimensional model; if yes,

The calling physics engine adjusts the display of the three-dimensional model of the hand and the three-dimensional model of the accessory, specifically: applying a force to the three-dimensional model of the accessory, and calling the physics engine to simulate the three-dimensional model of the hand and the accessory The friction and collision between the three-dimensional models causes the three-dimensional model of the ornament to rotate relative to the finger or wrist in the three-dimensional model of the wearing hand.

The method for human-computer interaction of the hand-crafted virtual accessory according to claim 1, wherein after the introduction of the three-dimensional model of the accessory, the method further comprises:

Receiving depth information of the hand acquired by the depth sensor in its sensing space;

And analyzing the gesture corresponding to the hand based on the depth information of the hand;

The selection, scaling, movement or rotation of the three-dimensional model of the ornament is controlled based on the gesture.

A human-computer interaction system in which a hand virtual jewelry is tried, characterized in that it comprises a depth sensor, an introduction unit, a display unit, a physics engine and a control unit;

The introduction unit is configured to introduce a three-dimensional model of the hand and a three-dimensional model of the jewelry;

The depth sensor is configured to acquire depth information of the hand in the sensing space thereof;

The control unit is configured to display the three-dimensional model of the accessory on a specified position of the three-dimensional model of the hand; and receive and based on the depth information of the hand, invoke the physics engine to adjust the three-dimensional model and the hand The display of the three-dimensional model of the ornament;

The physics engine is configured to simulate friction and collision between the three-dimensional model of the hand and the three-dimensional model of the accessory;

The display unit is configured to display the three-dimensional model of the hand and the three-dimensional model of the accessory.

The human-computer interaction system of the hand virtual jewelry fitting according to claim 11, wherein the system further comprises a building unit and a setting unit; the building unit is configured to import the hand in the importing unit Before the three-dimensional model and the three-dimensional model of the ornament, constructing the three-dimensional model of the hand and the three-dimensional model of the accessory;

The setting unit sets the three-dimensional model of the hand and the attribute of the three-dimensional model of the accessory, and respectively sets a collider on the hand three-dimensional model and the three-dimensional model of the accessory, and sets a collision condition of the collider, And establishing parameters of the hand three-dimensional model and the three-dimensional model of the accessory.

The human-computer interaction system of the hand virtual jewelry fitting according to claim 12, wherein the hand three-dimensional model constructed by the building unit comprises a hand model and a skeleton model in the hand model, The control unit receives the depth information of the hand acquired by the depth sensor in the sensing space thereof, and obtains the bone information of the hand based on the acquired depth information of the hand; the building unit associates the skeleton information with the The skeleton model constructs a three-dimensional model of the hand of the hand; wherein the bone information includes bone width information and bone length information of the wrist, the palm, and each finger.

The human-computer interaction system of the hand virtual jewelry fitting according to claim 13, wherein the building unit associates the skeleton information with the skeleton model, specifically:

The building unit stretches or contracts the skeleton model based on the bone information size.

The human-computer interaction system of the hand virtual jewelry fitting according to claim 13, wherein the control unit displays the three-dimensional model of the accessory on the designated position of the hand three-dimensional model, the control The unit is further configured to: adjust a size of the three-dimensional model of the accessory based on the bone information.

The human-computer interaction system of the hand virtual jewelry fitting according to claim 12, wherein the control unit calls the physics engine to adjust the display of the hand three-dimensional model and the three-dimensional model of the accessory, specifically:

Synchronizing the display of the three-dimensional model of the hand according to the change of the depth information of the hand;

The physics engine is invoked to simulate friction and collision between the hand three-dimensional model and the jewelry three-dimensional model, so that the jewelry three-dimensional model is displayed synchronously with the hand three-dimensional model.

The human-computer interaction system of the hand virtual jewelry fitting according to claim 12, wherein the hand three-dimensional model constructed by the building unit comprises a three-dimensional model of wearing a hand and a three-dimensional model of an interactive hand;

Then, the control unit calls the physics engine to adjust the display of the three-dimensional model of the hand and the three-dimensional model of the accessory based on the depth information of the hand, specifically:

Determining whether the state of the three-dimensional model of the wearing hand changes based on the depth information of the hand; if not,

Determining, according to the depth information of the hand, whether the three-dimensional model of the wearing hand and the three-dimensional model of the interactive hand have mutual motion;

If yes, applying a force to the three-dimensional model of the ornament, and calling the physics engine to simulate friction and collision between the three-dimensional model of the hand and the three-dimensional model of the accessory, so that the three-dimensional model of the accessory is compared with the three-dimensional model of the hand mobile.

The human-computer interaction system of the hand virtual jewelry fitting according to claim 17, wherein the control unit determines the three-dimensional model of the wearing hand and the three-dimensional model of the interactive hand based on the depth information of the hand Whether there is relative movement, specifically: Obtaining direction information of a finger or a wrist in the three-dimensional model of the wearing hand based on the depth information of the hand;

Determining whether the interactive hand three-dimensional model is translationally moved relative to a direction of a finger or a wrist in the three-dimensional model of the wearing hand; if yes,

Applying force to the three-dimensional model of the ornament, and calling the physics engine to simulate friction and collision between the three-dimensional model of the hand and the three-dimensional model of the accessory, so that the three-dimensional model of the accessory is opposite to the finger in the three-dimensional model of the wearing hand Or move in the direction of the wrist.

The human-computer interaction system of the hand virtual jewelry fitting according to claim 17, wherein the control unit determines the three-dimensional model of the wearing hand and the three-dimensional model of the interactive hand based on the depth information of the hand Whether there is relative motion, specifically: obtaining direction information of a finger or a wrist in the three-dimensional model of the wearing hand based on the depth information of the hand;

Determining whether the interactive hand three-dimensional model is rotated relative to a finger or a wrist in the wearing hand three-dimensional model; if yes,

Applying force to the three-dimensional model of the ornament, and calling the physics engine to simulate friction and collision between the three-dimensional model of the hand and the three-dimensional model of the accessory, so that the three-dimensional model of the accessory is opposite to the finger in the three-dimensional model of the wearing hand Or the wrist rotates.

The human-computer interaction system of the virtual hand-stitched hand-picked according to claim 11, wherein after the importing unit imports the three-dimensional model of the accessory, the control unit is further configured to:

Receiving depth information of the hand acquired by the depth sensor in its sensing space;

And analyzing the gesture corresponding to the hand based on the depth information of the hand;

The selection, scaling, movement or rotation of the three-dimensional model of the ornament is controlled based on the gesture.