CN104360729B - Many exchange methods and device based on Kinect and Unity3D - Google Patents

Abstract

The invention relates to a multi-interaction method based on Kinect and Unity3D, comprising: S1: adjusting camera parameters in Unity3D to be consistent with the effective detection range of Kinect; S2: using Kinect to determine user coordinates and ground equations; S3: determining virtual simulation coordinates according to relative positions , and register the virtual model; S4: Design interactive gestures and voice; S5: Determine the displacement animation and multimedia effects of the Unity3D control model; S6: Fusion and display the images obtained by the camera in Unity3D and the image obtained by the Kinect camera. The present invention utilizes the support of Kinect for voice recognition and the triggering method of adding virtual model three-dimensional registration to the positioning of the human body skeleton, provides more interactive modes for the user through the recognition function of body movements, improves the user experience, and utilizes the three-dimensional engine of Unity3D to The model pose is automatically processed, which greatly simplifies the steps required for 3D registration. The invention also discloses a multi-interaction device based on Kinect and Unity3D.

Description

Multi-interaction method and device based on Kinect and Unity3D

technical field

The invention relates to the technical field of computer augmented reality, in particular to a multi-interaction method and device based on Kinect and Unity3D.

Background technique

Augmented Reality technology (Augmented Reality) was first proposed in the 1990s, and now it has been widely used in many aspects such as medical care, education, industry, and commerce. A more general definition of augmented reality was proposed by Ronald Azuma of the University of North Carolina in 1997, including three main aspects: Combines real and virtual, Interactive in real time, Registered in 3D ). The technology superimposes a virtual scene on top of the real one on the screen and allows participants to interact with the virtual scene. At present, the implementation process of augmented reality is generally as follows: 1) Obtain the scene image through the image acquisition device; 2) Identify and track the calibration image or text in the scene, calculate its deformation and calculate its displacement and rotation matrix; 3) According to the position and rotation matrix of the calibration image , registering the position information of the corresponding virtual model in the three-dimensional space; 4) fusing the virtual model and the real scene, and displaying it on the screen.

However, the current commonly used technologies have the following defects: 1) The interaction method is simplistic, and the registration of the virtual model can only be triggered by the calibration image or text, and after registration, the model can only be translated and rotated, and the model can only follow the movement of the calibration object , less interaction methods and more restrictions; 2) The 3D registration algorithm is cumbersome, it is necessary to determine the position and posture of the model according to the feature point coordinate system, and then convert it to the camera coordinate system, and finally integrate the virtual model and the real scene and align it according to the coordinates of the display screen. It displays. It can be seen that the current technology requires more calculation steps in the three-dimensional registration stage of the virtual model, and the operation is not simple and automatic enough.

Contents of the invention

The technical problem to be solved by the present invention is, aiming at the deficiencies in the prior art, how to use Kinect to support voice recognition and increase the trigger mode of virtual model three-dimensional registration for human skeleton positioning, and provide users with more information through the recognition function of body movements. How to interact, improve the user experience, and how to use Unity3D's 3D engine to automatically process the model pose, greatly simplifying the key issues of the steps required for 3D registration.

For this purpose, the present invention proposes a kind of multi-interaction method based on Kinect and Unity3D, comprising:

S1: Adjust the camera parameters in Unity3D to be consistent with the effective detection range of Kinect;

S2: Use Kinect to determine user coordinates and ground equations;

S3: Determine the virtual simulation coordinates according to the relative position, and register the virtual model;

S4: Design interactive gestures and voice;

S5: Determine the displacement animation and multimedia effects of the Unity3D control model;

S6: Fusion and displaying the picture acquired by the camera in Unity3D and the image acquired by the camera of Kinect.

Further, the step S1 further includes: placing the Kinect at a preset position in the real scene, and adjusting the real scene to be within the effective detection range of the Kinect.

Further, the step S1 further includes: adjusting Field of view and Clipping Planes parameters of the camera in Unity3D.

Further, the step S2 further includes:

S21: Use the SkeletonFrame.FloorClipPlane function to determine the plane equation representing the ground, wherein the plane equation in the Kinect coordinate system is: Ax+By+Cz+D=0, (A, B, C) is the The plane normal vector of plane equation, described plane equation under described Unity3D coordinate system is: y+E=0, (0,1,0) is the plane normal vector of described plane equation;

S22: Rotate (A, B, C) to coincide with (0, 1, 0), and complete the registration of the Kinect coordinate system and the Unity3D coordinate system.

Further, the registration of the Kinect coordinate system and the Unity3D coordinate system further includes: when any point (k ₁ , k ₂ , k ₃ ) in the Kinect coordinate system is converted to the Unity3D coordinate system, the rotation angle around the X axis is -arctan (B/C), the rotation angle around the Z axis is arctan(A/B), and the rotation radius is The coordinates after rotation are: (k ₁ cosα-(k ₂ cosβ-k ₃ sinβ)sinα, k ₁ sinα+(k ₂ cosβ-k ₃ sinβ)cosα, k ₂ sinβ+k ₃ cosβ), where α=arctan (A/B), β=-arctan (B/C).

Further, the step S6 further includes:

S61: Perform sampling or difference operation on two images;

S62: traverse the two images after the operation, and compare the depth values of the points corresponding to the pixels of the target image in the two images;

S63: Set the color value of the corresponding point of the target image to the color value of the pixel with a smaller depth value.

Further, the registered virtual model can also trigger a default model by the user moving to a special location.

Further, the registered virtual model can also trigger the registration of the corresponding model through the voice of the user.

For this purpose, the present invention proposes a kind of multi-interaction device based on Kinect and Unity3D, comprising:

The adjustment module is used to adjust the camera parameters in Unity3D to be consistent with the effective detection range of Kinect;

Determine the coordinates and ground equation module, used to determine user coordinates and ground equations using Kinect;

The virtual model registration module is used to determine the virtual simulation coordinates according to the relative position and register the virtual model;

Design module for designing interactive gestures and voice;

Determine the effect module, which is used to determine the displacement animation and multimedia effects of the Unity3D control model;

The image fusion module is used to fuse and display the images obtained by the camera in Unity3D and the images obtained by the camera of Kinect.

A kind of multi-interaction method based on Kinect and Unity3D disclosed by the present invention firstly simplifies the conversion between the real scene coordinate system and the virtual scene coordinate system by setting the position and attribute of the camera in Unity3D; The corresponding coordinates in , and the plane equation representing the ground, and then the three-dimensional registration coordinates can be determined according to the relative positional relationship between the virtual model to be registered and the ground and the user. The trigger registration mechanism is more flexible and can be triggered when the user moves to a specific position , and can also be triggered using the speech recognition module. Once again, the interaction mode after the model registration is enriched, and the model can be interacted with through body movements and voice operations; finally, the Transform component and the Mecanim animation system in Unity3D are used to simplify the realization of virtual model displacement changes and animation effects. The invention also discloses a multi-interaction device based on Kinect and Unity3D.

Description of drawings

The features and advantages of the present invention will be more clearly understood by referring to the accompanying drawings, which are schematic and should not be construed as limiting the invention in any way. In the accompanying drawings:

Fig. 1 shows a kind of flow chart of steps based on the multi-interaction method of Kinect and Unity3D in the embodiment of the present invention;

FIG. 2 shows a structural diagram of a multi-interaction device based on Kinect and Unity3D in an embodiment of the present invention.

detailed description

Embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

As shown in Figure 1, the present invention provides a kind of multi-interaction method based on Kinect and Unity3D, comprises specific following steps:

Step S1: Adjust the camera parameters in Unity3D to be consistent with the effective detection range of Kinect. Specifically, place the Kinect to the preset position of the real scene, and adjust the real scene to be within the effective detection range of the Kinect, wherein the effective range refers to 1.2-3.6 meters from the camera, 57 degrees horizontally, and 43 degrees vertically.

Furthermore, in the coordinate system of the data returned by Kinect, the origin is the Kinect sensor, so the camera in Unity3D is placed at the coordinate origin to facilitate the calculation of virtual model coordinates during 3D registration. Adjust the Field of view and Clipping Planes parameters of the camera in Unity3D, and set the Field of view, Clipping Planes and other parameters to the same value as the effective range of Kinect.

Step S2: Use Kinect to determine user coordinates and ground equations.

Specifically, use the SkeletonFrame.FloorClipPlane function to determine the plane equation representing the ground, wherein the plane equation in the Kinect coordinate system is: Ax+By+Cz+D=0, (A, B, C) is the plane method of the plane equation Vector, the plane equation in the Unity3D coordinate system is: y+E=0, (0, 1, 0) is the plane normal vector of the plane equation; rotate (A, B, C) to (0, 1, 0) Coincidentally, complete the registration of the Kinect coordinate system and the Unity3D coordinate system.

Furthermore, the registration between the Kinect coordinate system and the Unity3D coordinate system further includes: when any point (k ₁ , k ₂ , k ₃ ) in the Kinect coordinate system is converted to the Unity3D coordinate system, the rotation angle around the X axis is -arctan(B /C), the rotation angle around the Z axis is arctan(A/B), and the rotation radius is The coordinates after rotation are: (k ₁ cosα-(k ₂ cosβ-k ₃ sinβ)sinα, k ₁ sinα+(k ₂ cosβ-k ₃ sinβ)cosα, k ₂ sinβ+k ₃ cosβ), where α=arctan (A/B), β=-arctan (B/C).

Step S3: Determine the virtual simulation coordinates according to the relative position, and register the virtual model.

Specifically, according to the conversion formula for converting the Kinect coordinate system to the Unity3D coordinate system in the above steps, the Skeleton Point returned by the API in the Kinect SDK is converted to the Unity3D coordinate system. In the Unity3D coordinate system, locate the model to the required coordinates according to the ground height, user coordinates and the relative position of the virtual model and the user; or select the default model for registration, or add words related to the model to the voice library for registration, among which, several Words correspond to a model, through Kinect Speech module speech recognition. After the user speaks a word in the speech database, a three-dimensional model corresponding to the word is registered in the scene.

Step S4: Design interactive gestures and voice.

Specifically, design interaction poses and determine the set of body movements for each operation. For example: using the arm to hover means selecting an object or clicking a button; moving the arm means sliding the mouse or panning the model; moving the hands away and approaching means zooming the model; rotating the ball with both hands means rotating the model, etc., and simple interaction is realized through voice. For example: display and disappear of the model, play and pause of multimedia, etc.

Step S5: Determine the displacement animation and multimedia effects of the Unity3D control model.

Specifically, corresponding operations are performed on the model according to the user's body movements. Use the Transform component in the GameObject object in the Unity3D SDK to translate, rotate, and scale the model; use the Mecanim animation system to control the model to follow, run, guide and other designed interactive actions for the user; use the Audio component and MovieTextures component Control multimedia effects.

Step S6: Fusion and displaying the picture acquired by the camera in Unity3D and the image acquired by the camera of Kinect.

Specifically, step S6 further includes:

Step S61: Sampling or difference operation is performed on the two images to scale them to the size of the target image.

Step S62: traverse the two images after operation, and compare the depth values of the points corresponding to the pixels of the target image in the two images;

Step S63: Set the color value of the corresponding point of the target image to the color value of the pixel with a smaller depth value.

A multi-interaction method based on Kinect and Unity3D disclosed by the invention is an augmented reality technology with simple three-dimensional registration operation. Utilize Kinect's support for speech recognition and human skeleton positioning to increase the trigger method of 3D registration of virtual models, provide users with more interactive methods through the recognition function of body movements, and improve user experience; use Unity3D's 3D engine to model The pose is automatically processed, which greatly simplifies the steps required for 3D registration. That is to say, the integrated use of somatosensory interactive equipment and 3D game engine simplifies the 3D registration process, increases the trigger method of 3D registration, enriches the user interaction channels, and improves the user operation experience.

As shown in Figure 2, the present invention provides a kind of multi-interaction device 10 based on Kinect and Unity3D, comprising: adjustment module 101, determination coordinate and ground equation module 102, virtual model registration module 103, design module 104, determination effect module 105 and an image fusion module 106 .

Specifically, the adjustment module 101 is used to adjust the camera parameters in Unity3D to be consistent with the effective detection range of Kinect; the determination coordinate and ground equation module 102 is used to determine the user coordinates and the ground equation using Kinect; the virtual model registration module 103 is used to determine the virtual model according to the relative position. Simulate the coordinates and register the virtual model; the design module 104 is used to design interactive gestures and voice; the determination effect module 105 is used to determine the displacement animation and multimedia effects of the Unity3D control model; the image fusion module 106 is used to obtain the pictures obtained by the camera in Unity3D Fusion and display with the image acquired by the camera of Kinect.

A kind of multi-interaction method based on Kinect and Unity3D disclosed by the present invention firstly simplifies the conversion between the real scene coordinate system and the virtual scene coordinate system by setting the position and attribute of the camera in Unity3D; The corresponding coordinates in , and the plane equation representing the ground, and then the three-dimensional registration coordinates can be determined according to the relative positional relationship between the virtual model to be registered and the ground and the user. The trigger registration mechanism is more flexible and can be triggered when the user moves to a specific position , and can also be triggered using the speech recognition module. Once again, the interaction mode after the model registration is enriched, and the model can be interacted with through body movements and voice operations; finally, the Transform component and the Mecanim animation system in Unity3D are used to simplify the realization of virtual model displacement changes and animation effects. The invention also discloses a multi-interaction device based on Kinect and Unity3D.

The above embodiments are only used to illustrate the present invention, but not to limit the present invention. Those of ordinary skill in the relevant technical field can make various changes and modifications without departing from the spirit and scope of the present invention. Therefore, all Equivalent technical solutions also belong to the category of the present invention, and the scope of patent protection of the present invention should be defined by the claims.

Although the embodiments of the present invention have been described in conjunction with the accompanying drawings, those skilled in the art can make various modifications and variations without departing from the spirit and scope of the present invention. within the bounds of the requirements.

Claims (9)

1. a kind of many exchange methods based on Kinect and Unity3D, it is characterised in that including specific following steps：

S1：Adjust camera parameters in Unity3D consistent with Kinect valid analysing ranges；

S2：User coordinates and ground equation are determined using Kinect；

S3：Virtual analog coordinate is determined according to relative position, and registers dummy model；

S4：Design interaction posture and voice；

S5：Determine Unity3D Controlling model displacement animations and Multimedia；

S6：The picture got to the video camera in Unity3D is merged and shown with the image that Kinect camera is obtained Show.

2. the method as described in claim 1, it is characterised in that the step S1 further comprises：Kinect is placed to reality The predeterminated position of scene, adjustment reality scene is in Kinect valid analysing ranges.

3. the method as described in claim 1, it is characterised in that the step S1 further comprises：Adjust in Unity3D and image Machine Field of view and Clipping Planes parameters.

4. the method as described in claim 1, it is characterised in that the step S2 further comprises：

S21：Determine to represent the plane equation on ground, i.e., described ground using SkeletonFrame.FloorClipPlane functions Equation, wherein, the plane equation under the Kinect coordinate systems is：Ax+By+Cz+D=0, (A, B, C) is described flat The plane normal vector of face equation, the plane equation under the Unity3D coordinate systems is：Y+E=0, (0,1,0) is described The plane normal vector of plane equation；

S22：(A, B, C) rotation to (0,1,0) is overlapped, Kinect coordinate systems are completed registering with Unity3D coordinate systems.

5. method as claimed in claim 4, it is characterised in that the Kinect coordinate systems are registering with Unity3D coordinate systems Further comprise：Arbitrfary point (k under Kinect coordinate systems₁, k₂, k₃) to Unity3D coordinate systems change when, need to be around the X-axis anglec of rotation For-arctan (B/C), the anglec of rotation is arctan (A/B) about the z axis, and radius of turn isCoordinate after rotation For：(k₁cosα-(k₂cosβ-k₃Sin β) sin α,

k₁sinα+(k₂cosβ-k₃Sin β) cos α,

k₂sinβ+k₃Cos β), wherein, α=arctan (A/B), β=- arctan (B/C).

6. the method as described in claim 1, it is characterised in that the step S6 further comprises：

S61：Two images are sampled or difference operation；

S62：Two images after operation are traveled through, compare the depth with purpose image slices vegetarian refreshments corresponding points in two images Value；

S63：Purpose image corresponding points color value is set to color value of the depth value compared with statuette vegetarian refreshments.

7. the method as described in claim 1, it is characterised in that the registration dummy model can also be moved to spy by user The mode of different location triggered default models.

8. the method as described in claim 1, it is characterised in that the registration dummy model can also pass through user's speech trigger The mode of correspondence model registration.

9. a kind of many interactive devices based on Kinect and Unity3D, it is characterised in that including：

Adjusting module, it is consistent with Kinect valid analysing ranges for adjusting camera parameters in Unity3D；

Coordinate and ground equation module are determined, for determining user coordinates and ground equation using Kinect；

Dummy model Registering modules, for determining virtual analog coordinate according to relative position, and register dummy model；

Module is designed, for designing interactive posture and voice；

Effects module is determined, for determining Unity3D Controlling model displacement animations and Multimedia；

Image co-registration module, the figure that picture and Kinect camera for being got to the video camera in Unity3D are obtained As being merged and being shown.