CN104360729A - Multi-interactive method and device based on Kinect and Unity 3D - Google Patents

Abstract

The invention relates to a multi-interaction method based on Kinect and Unity3D, comprising: S1: adjusting camera parameters in Unity3D to be consistent with the effective detection range of Kinect; S2: using Kinect to determine user coordinates and ground equations; S3: determining virtual simulation coordinates according to relative positions , and register the virtual model; S4: Design interactive gestures and voice; S5: Determine the displacement animation and multimedia effects of the Unity3D control model; S6: Fusion and display the images obtained by the camera in Unity3D and the image obtained by the Kinect camera. The present invention utilizes the support of Kinect for voice recognition and the triggering method of adding virtual model three-dimensional registration to the positioning of the human body skeleton, provides more interactive modes for the user through the recognition function of body movements, improves the user experience, and utilizes the three-dimensional engine of Unity3D to The model pose is automatically processed, which greatly simplifies the steps required for 3D registration. The invention also discloses a multi-interaction device based on Kinect and Unity3D.

Description

Multi-interaction method and device based on Kinect and Unity3D

technical field

The invention relates to the technical field of computer augmented reality, in particular to a multi-interaction method and device based on Kinect and Unity3D. the

Background technique

Augmented Reality (Augmented Reality) was first proposed in the 1990s, and now it has been widely used in many aspects such as medical care, education, industry, and commerce. A more general definition of augmented reality was proposed by Ronald Azuma of the University of North Carolina in 1997, including three main aspects: Combines real and virtual, Interactive in real time, Registered in 3D ). The technology superimposes a virtual scene on top of the real one on the screen and allows participants to interact with the virtual scene. At present, the implementation process of augmented reality is generally as follows: 1) Obtain the scene image through the image acquisition device; 2) Identify and track the calibration image or text in the scene, calculate its deformation and calculate its displacement and rotation matrix; 3) According to the position and rotation matrix of the calibration image , registering the position information of the corresponding virtual model in the three-dimensional space; 4) fusing the virtual model and the real scene, and displaying it on the screen. the

However, the current commonly used technologies have the following defects: 1) The interaction method is simplistic, and the registration of the virtual model can only be triggered by the calibration image or text, and after registration, the model can only be translated and rotated, and the model can only follow the movement of the calibration object , less interaction methods and more restrictions; 2) The 3D registration algorithm is cumbersome, it is necessary to determine the position and posture of the model according to the feature point coordinate system, and then convert it to the camera coordinate system, and finally integrate the virtual model and the real scene and align it according to the coordinates of the display screen. It displays. It can be seen that the current technology requires more calculation steps in the three-dimensional registration stage of the virtual model, and the operation is not simple and automatic enough. the

Contents of the invention

The technical problem to be solved by the present invention is, aiming at the deficiencies in the prior art, how to use Kinect to support voice recognition and increase the trigger mode of virtual model three-dimensional registration for human skeleton positioning, and provide users with more information through the recognition function of body movements. How to interact, improve the user experience, and how to use Unity3D's 3D engine to automatically process the model pose, greatly simplifying the key issues of the steps required for 3D registration. the

For this purpose, the present invention proposes a kind of multi-interaction method based on Kinect and Unity3D, comprising:

S1: Adjust the camera parameters in Unity3D to be consistent with the effective detection range of Kinect;

S2: Use Kinect to determine user coordinates and ground equations;

S3: Determine the virtual simulation coordinates according to the relative position, and register the virtual model;

S4: Design interactive gestures and voice;

S5: Determine the displacement animation and multimedia effects of the Unity3D control model;

S6: Fusion and displaying the picture acquired by the camera in Unity3D and the image acquired by the camera of Kinect. the

Further, the step S1 further includes: placing the Kinect at a preset position in the real scene, and adjusting the real scene to be within the effective detection range of the Kinect. the

Further, the step S1 further includes: adjusting the Camera Field of view and Clipping Planes parameters in Unity3D. the

Further, said step S2 further includes:

S21: Use the SkeletonFrame.FloorClipPlane function to determine the plane equation representing the ground, wherein the plane equation in the Kinect coordinate system is: Ax+By+Cz+D=0, (A, B, C) is the The plane normal vector of the plane equation, the plane equation under the Unity3D coordinate system is: y+E=0, (0,1,0) is the plane normal vector of the plane equation;

S22: Rotate (A, B, C) to coincide with (0, 1, 0), and complete the registration of the Kinect coordinate system and the Unity3D coordinate system. the

Further, the registration of the Kinect coordinate system and the Unity3D coordinate system further includes: when any point (k ₁ , k ₂ , k ₃ ) in the Kinect coordinate system is converted to the Unity3D coordinate system, the rotation angle around the X axis is -arctan (B/C), the rotation angle around the Z axis is arctan(A/B), and the rotation radius is The coordinates after rotation are: (k ₁ cosα-(k ₂ cosβ-k ₃ sinβ)sinα, k ₁ sinα+(k ₂ cosβ-k ₃ sinβ)cosα, k ₂ sinβ+k ₃ cosβ), where α=arctan (A/B), β=-arctan (B/C).

Further, said step S6 further includes:

S61: Perform sampling or difference operation on two images;

S62: traverse the two images after the operation, and compare the depth values of the points corresponding to the pixel points of the target image in the two images;

S63: Set the color value of the corresponding point of the target image to the color value of the pixel with a smaller depth value. the

Further, the registered virtual model can also trigger a default model by the user moving to a special location. the

Further, the registered virtual model can also trigger the registration of the corresponding model through the voice of the user. the

For this purpose, the present invention proposes a kind of multi-interaction device based on Kinect and Unity3D, comprising:

The adjustment module is used to adjust the camera parameters in Unity3D to be consistent with the effective detection range of Kinect;

Determine the coordinates and ground equation module, used to determine user coordinates and ground equations using Kinect;

The virtual model registration module is used to determine the virtual simulation coordinates according to the relative position, and register the virtual model;

Design module for designing interactive gestures and voice;

Determine the effect module, which is used to determine the displacement animation and multimedia effects of the Unity3D control model;

The image fusion module is used to fuse and display the images obtained by the camera in Unity3D and the images obtained by the camera of Kinect. the

A kind of multi-interaction method based on Kinect and Unity3D disclosed by the present invention firstly simplifies the conversion between the real scene coordinate system and the virtual scene coordinate system by setting the position and attribute of the camera in Unity3D; The corresponding coordinates in , and the plane equation representing the ground, and then the three-dimensional registration coordinates can be determined according to the relative positional relationship between the virtual model to be registered and the ground and the user. The trigger registration mechanism is more flexible and can be triggered when the user moves to a specific position , and can also be triggered using the speech recognition module. Once again, the interaction mode after the model registration is enriched, and the model can be interacted with through body movements and voice operations; finally, the Transform component and the Mecanim animation system in Unity3D are used to simplify the realization of virtual model displacement changes and animation effects. The invention also discloses a multi-interaction device based on Kinect and Unity3D. the

Description of drawings

The features and advantages of the present invention will be more clearly understood by referring to the accompanying drawings, which are schematic and should not be construed as limiting the present invention in any way, in the accompanying drawings:

Fig. 1 shows a kind of flow chart of steps based on the multi-interaction method of Kinect and Unity3D in the embodiment of the present invention;

Fig. 2 shows a kind of structural diagram based on Kinect and Unity3D multi-interaction device in the embodiment of the present invention. the

Detailed ways

Embodiments of the present invention will be described in detail below with reference to the accompanying drawings. the

As shown in Figure 1, the present invention provides a kind of multi-interaction method based on Kinect and Unity3D, comprises specific following steps:

Step S1: Adjust the camera parameters in Unity3D to be consistent with the effective detection range of Kinect. Specifically, place the Kinect to the preset position of the real scene, and adjust the real scene to be within the effective detection range of the Kinect, wherein the effective range refers to 1.2-3.6 meters from the camera, 57 degrees horizontally, and 43 degrees vertically. the

Furthermore, in the coordinate system of the data returned by Kinect, the origin is the Kinect sensor, so the camera in Unity3D is placed at the coordinate origin to facilitate the calculation of virtual model coordinates during 3D registration. Adjust the Field of view and Clipping Planes parameters of the camera in Unity3D, and set the Field of view, Clipping Planes and other parameters to the same value as the effective range of Kinect. the

Step S2: Use Kinect to determine user coordinates and ground equations. the

Specifically, use the SkeletonFrame.FloorClipPlane function to determine the plane equation representing the ground, wherein the plane equation in the Kinect coordinate system is: Ax+By+Cz+D=0, (A, B, C) is the plane method of the plane equation Vector, the plane equation in the Unity3D coordinate system is: y+E=0, (0, 1, 0) is the plane normal vector of the plane equation; rotate (A, B, C) to (0, 1, 0) Coincidentally, complete the registration of the Kinect coordinate system and the Unity3D coordinate system. the

Furthermore, the registration between the Kinect coordinate system and the Unity3D coordinate system further includes: when any point (k ₁ , k ₂ , k ₃ ) in the Kinect coordinate system is converted to the Unity3D coordinate system, the rotation angle around the X axis is -arctan(B /C), the rotation angle around the Z axis is arctan(A/B), and the rotation radius is The coordinates after rotation are: (k ₁ cosα-(k ₂ cosβ-k ₃ sinβ)sinα, k ₁ sinα+(k ₂ cosβ-k ₃ sinβ)cosα, k ₂ sinβ+k ₃ cosβ), where α=arctan (A/B), β=-arctan (B/C).

Step S3: Determine the virtual simulation coordinates according to the relative position, and register the virtual model. the

Specifically, according to the conversion formula for converting the Kinect coordinate system to the Unity3D coordinate system in the above steps, the Skeleton Point returned by the API in the Kinect SDK is converted to the Unity3D coordinate system. In the Unity3D coordinate system, locate the model to the required coordinates according to the ground height, user coordinates and the relative position of the virtual model and the user; or select the default model for registration, or add words related to the model to the voice library for registration, among which, several Words correspond to a model, which is recognized by the Kinect Speech module. After the user speaks a word in the speech database, a three-dimensional model corresponding to the word is registered in the scene. the

Step S4: Design interactive gestures and voice. the

Specifically, design interaction poses and determine the set of body movements for each operation. For example: using the arm to hover means selecting an object or clicking a button; moving the arm means sliding the mouse or panning the model; moving the hands away and approaching means zooming the model; rotating the ball with both hands means rotating the model, etc., and simple interaction is realized through voice. For example: display and disappear of the model, play and pause of multimedia, etc. the

Step S5: Determine the displacement animation and multimedia effects of the Unity3D control model. the

Specifically, corresponding operations are performed on the model according to the user's body movements. Use the Transform component in the GameObject object in the Unity3D SDK to translate, rotate, and scale the model; use the Mecanim animation system to control the model to follow, run, guide and other designed interactive actions for the user; use Audio components and Movie Textures Components control multimedia effects. the

Step S6: Fusion and displaying the picture acquired by the camera in Unity3D and the image acquired by the camera of Kinect. the

Specifically, step S6 further includes:

Step S61: Sampling or difference operation is performed on the two images to scale them to the size of the target image. the

Step S62: traverse the two images after the operation, and compare the depth values of the points corresponding to the pixel points of the target image in the two images;

Step S63: Set the color value of the corresponding point of the target image to the color value of the pixel with a smaller depth value. the

A multi-interaction method based on Kinect and Unity3D disclosed by the invention is an augmented reality technology with simple three-dimensional registration operation. Utilize Kinect's support for speech recognition and human skeleton positioning to increase the trigger method of 3D registration of virtual models, provide users with more interactive methods through the recognition function of body movements, and improve user experience; use Unity3D's 3D engine to model The pose is automatically processed, which greatly simplifies the steps required for 3D registration. That is to say, the integrated use of somatosensory interactive equipment and 3D game engine simplifies the 3D registration process, increases the trigger method of 3D registration, enriches the user interaction channels, and improves the user operation experience. the

As shown in Figure 2, the present invention provides a kind of multi-interaction device 10 based on Kinect and Unity3D, comprising: adjustment module 101, determination coordinate and ground equation module 102, virtual model registration module 103, design module 104, determination effect module 105 and an image fusion module 106 . the

Specifically, the adjustment module 101 is used to adjust the camera parameters in Unity3D to be consistent with the effective detection range of Kinect; the determination coordinate and ground equation module 102 is used to determine the user coordinates and the ground equation using Kinect; the virtual model registration module 103 is used to determine the virtual model according to the relative position. Simulate the coordinates and register the virtual model; the design module 104 is used to design interactive gestures and voice; the determination effect module 105 is used to determine the displacement animation and multimedia effects of the Unity3D control model; the image fusion module 106 is used to obtain the pictures obtained by the camera in Unity3D Fusion and display with the image acquired by the camera of Kinect. the

A kind of multi-interaction method based on Kinect and Unity3D disclosed by the present invention firstly simplifies the conversion between the real scene coordinate system and the virtual scene coordinate system by setting the position and attribute of the camera in Unity3D; The corresponding coordinates in , and the plane equation representing the ground, and then the three-dimensional registration coordinates can be determined according to the relative positional relationship between the virtual model to be registered and the ground and the user. The trigger registration mechanism is more flexible and can be triggered when the user moves to a specific position , and can also be triggered using the speech recognition module. Once again, the interaction mode after the model registration is enriched, and the model can be interacted with through body movements and voice operations; finally, the Transform component and the Mecanim animation system in Unity3D are used to simplify the realization of virtual model displacement changes and animation effects. The invention also discloses a multi-interaction device based on Kinect and Unity3D. the

The above embodiments are only used to illustrate the present invention, but not to limit the present invention. Those of ordinary skill in the relevant technical field can make various changes and modifications without departing from the spirit and scope of the present invention. Therefore, all Equivalent technical solutions also belong to the category of the present invention, and the scope of patent protection of the present invention should be defined by the claims. the

Although the embodiments of the present invention have been described in conjunction with the accompanying drawings, those skilled in the art can make various modifications and variations without departing from the spirit and scope of the present invention. within the bounds of the requirements. the

Claims (9)

1. based on many exchange methods of Kinect and Unity3D, it is characterized in that, comprise concrete following steps:

S1: in adjustment Unity3D, camera parameters is consistent with Kinect valid analysing range;

S2: utilize Kinect to determine user coordinates and ground equation;

S3: according to relative position determination virtual analog coordinate, and register dummy model;

S4: design mutual posture and voice;

S5: determine Unity3D Controlling model displacement animation and Multimedia;

S6: the image that the picture get the video camera in Unity3D and the camera of Kinect obtain merges and shows.

2. the method for claim 1, is characterized in that, described step S1 comprises further: place the predeterminated position of Kinect to reality scene, and adjustment reality scene is in Kinect valid analysing range.

3. the method for claim 1, is characterized in that, described step S1 comprises further: video camera Field of view and Clipping Planes parameter in adjustment Unity3D.

4. the method for claim 1, is characterized in that, described step S2 comprises further:

S21: use SkeletonFrame.FloorClipPlane function to determine the plane equation representing ground, wherein, described plane equation under described Kinect coordinate system is: Ax+By+Cz+D=0, the planar process vector that (A, B, C) is described plane equation, described plane equation under described Unity3D coordinate system is: y+E=0, the planar process vector that (0,1,0) is described plane equation;

S22: (A, B, C) is rotated and overlaps to (0,1,0), complete the registration of Kinect coordinate system and Unity3D coordinate system.

5. method as claimed in claim 4, it is characterized in that, the registration of described Kinect coordinate system and Unity3D coordinate system comprises further: arbitrfary point (k under Kinect coordinate system _1,k _2,k ₃) to Unity3D ordinate transform time, need be-arctan (B/C) around the X-axis anglec of rotation, be arctan (A/B) around the Z axis anglec of rotation, radius of turn be rotation recoil is designated as: (k ₁cos α-(k ₂cos β-k ₃sin β) sin α, k ₁sin α+(k ₂cos β-k ₃sin β) cos α, k ₂sin β+k ₃cos β), wherein, α=arctan (A/B), β=-arctan (B/C).

6. the method for claim 1, is characterized in that, described step S6 comprises further:

S61: two width images are sampled or difference operation;

S62: two width images after operation are traveled through, compares the depth value with object image slices vegetarian refreshments corresponding point in two width images;

S63: object image corresponding point color value is set to the color value of depth value compared with statuette vegetarian refreshments.

7. the method for claim 1, is characterized in that, described registration dummy model can also move to by user the mode that specific position triggers default models.

8. the method for claim 1, is characterized in that, described registration dummy model can also trigger the mode of corresponding model registration by user speech.

9., based on many interactive devices of Kinect and Unity3D, it is characterized in that, comprising:

Adjusting module, consistent with Kinect valid analysing range for adjusting camera parameters in Unity3D;

Determining coordinate and ground equation module, determining user coordinates and ground equation for utilizing Kinect;

Dummy model Registering modules, for according to relative position determination virtual analog coordinate, and registers dummy model;

Design module, for designing mutual posture and voice;

Determine effects module, for determining Unity3D Controlling model displacement animation and Multimedia;

Image co-registration module, the image that the camera for the picture that gets the video camera in Unity3D and Kinect obtains merges and shows.