CN115328304A - 2D-3D fused virtual reality interaction method and device - Google Patents
2D-3D fused virtual reality interaction method and device Download PDFInfo
- Publication number
- CN115328304A CN115328304A CN202210917515.2A CN202210917515A CN115328304A CN 115328304 A CN115328304 A CN 115328304A CN 202210917515 A CN202210917515 A CN 202210917515A CN 115328304 A CN115328304 A CN 115328304A
- Authority
- CN
- China
- Prior art keywords
- mobile phone
- virtual
- handle
- model
- screen
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
- G06F3/011—Arrangements for interaction with the human body, e.g. for user immersion in virtual reality
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
- G06F3/016—Input arrangements with force or tactile feedback as computer generated output to the user
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T17/00—Three dimensional [3D] modelling, e.g. data description of 3D objects
Landscapes
- Engineering & Computer Science (AREA)
- General Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Human Computer Interaction (AREA)
- Computer Graphics (AREA)
- Geometry (AREA)
- Software Systems (AREA)
- User Interface Of Digital Computer (AREA)
Abstract
The invention provides 2D-3D fused virtual reality interaction equipment and a method, aiming at solving the technical problems that the stability is poor and accurate input cannot be realized when a handle is used for interacting with a virtual object in a VR (virtual reality) environment at present, and the technical problems that simple 2D input can be carried out when a mobile phone is used for interacting with the virtual object, no tactile feedback exists, and large-range and full-angle space tracking cannot be provided. During interaction, the handle provides tactile feedback and large-range spatial 3D input, and moves, rotates and zooms the virtual object; the handle can also be used as a mobile phone tracker to track the position of the mobile phone in real time in the VR environment; the handset provides virtual graphical user input, precise tangible 2D input and its basic functions.
Description
Technical Field
The invention relates to the field of virtual reality interaction, in particular to a 2D-3D fused virtual reality interaction method and device.
Background
In virtual reality interaction, a handle is usually used for performing intuitive 3D operation on an object, the handle can provide the functions of target selection, interaction triggering, stable 3D tracking, object operation and the like, for example, when the object is operated, a user can temporarily bind a virtual object with the handle by pressing a specific key, and then the object is selected, moved and rotated through the handle; when it is desired to introduce a new device as an interactive device, the handle may act as a tracker for the new device.
The handle is generally operated by a single hand, so that the interaction efficiency is high, but the stability is poor; and the handle is operated in direct space 3D, and can not achieve the same precision as the alignment task completed by tangible 2D equipment. Although some handgrips provide touchpad functionality, their limited panel size has significant disadvantages compared to cell phones.
Meanwhile, mobile phones have become indispensable devices for people's daily life, which have abundant sensors and various usage modes, such as making phone calls, typing, photographing, etc., have high-definition screens and multi-touch recognition, and can perform virtual graphic user input and precise tangible 2D input, i.e., while displaying detailed images and text, a finger slides on the screen to provide efficient and precise tangible 2D input. At the same time, familiar gesture input modes and wireless connections can facilitate the development of utilizing cell phones to interact with virtual content. However, the mobile phone lacks an independent space tracking function, and needs additional equipment input, and in addition, the mobile phone lacks a physical button, and cannot provide physical feedback, and the mobile phone cannot be normally used in a VR environment, and simple functions (such as receiving and making a call, replying information and the like) of the mobile phone need to be experienced by taking off a head mounted display, so that virtual reality experience is seriously influenced.
In conclusion, the handle can perform intuitive 3D operation and spatial tracking, but the operation stability is poor, and accurate input cannot be provided. The cell phone has multiple sensors and a high resolution screen, can make virtual graphical user input and precise tangible 2D input, but lacks independent spatial tracking and physical buttons.
A number of documents have demonstrated that in a VR environment, a cell phone can perform its basic functions as an accurate 6-dof input device. In the document "Photoroller" published by Fabrice Matulic et al, visual Representations of sensors for precision Touch Input with Mobile Phones in VR (CHI' 2021: p.1-13) ", a 6-degree-of-freedom virtual reality tracker is used for tracking a Mobile phone, a mirror is fixed above a Mobile phone screen, and a front camera of the Mobile phone is reflected, so that a top view of the thumb of an operator is obtained. However, the virtual hand effect generated by the design is larger than the real hand effect, and no physical feedback exists, so that the operation immersion feeling is poor, and the design utilizes the front camera of the mobile phone to obtain the finger information, and the camera of the mobile phone can not be used any more, so that the effect of virtual physical space fusion can not be generated, and the function is single. In the published literature, "TrackCap: enclosing tablets for 3D Interaction on Mobile Head-Mounted display" (CHI' 2019), which is published by Peter Mohr et al, a Mobile phone is used as an accurate 6-degree-of-freedom input device, and the self-made tracking device and a Mobile phone camera are used for realizing the space tracking of the Mobile phone. However, this design cannot rotate the mobile phone due to the limitation of the camera of the mobile phone, and the operation feeling is deteriorated.
Disclosure of Invention
The invention provides 2D-3D fused virtual reality interaction equipment and a method, aiming at solving the technical problems that the stability is poor and the accurate input cannot be realized when a handle is used for interacting with a virtual object in a VR environment at present, and the technical problems that the simple 2D input can be only carried out when a mobile phone is used for interacting with the virtual object, the tactile feedback is not available, and the large-range and full-angle space tracking cannot be provided.
The invention has the following inventive concept:
the mobile phone and the handle are connected into a whole through the connecting structure, and the mobile phone and the handle are designed through a software method to realize communication, so that various interaction modes are generated. The invention provides a mobile phone screen mirroring method for solving the problem of image transmission delay, namely mobile phone mirroring software loaded on a mobile phone and screen acquisition software loaded on a computer are used for capturing and acquiring mobile phone screen information, and the mobile phone screen information is used as a texture map and is transmitted to the surface of a mobile phone model in a VR virtual environment; the invention provides a hand region visualization method for enhancing operation immersion, namely, a depth camera is used for collecting colors and depth values of a hand region and transmitting the colors and the depth values to a computer, the visualized hand region is rendered in a point cloud format in a three-dimensional virtual space in real time, the operation immersion is enhanced, and real-time communication between a mobile phone and a VR system is realized through TCP/IP communication during interaction. During interaction, the handle provides tactile feedback and large-range spatial 3D input, and moves, rotates and zooms the virtual object; the handle can also be used as a mobile phone tracker to track the position of the mobile phone in real time in the VR environment; the handset provides virtual graphical user input, precise tangible 2D input and its basic functions.
The invention designs two devices into a whole in both aspects of appearance and function setting, and can generate a series of new functions and new effects while realizing the basic functions, such as: the real-time viewing of the physical world in the virtual environment can be realized by utilizing the function of a mobile phone camera; when the user operates the object, the interaction equipment is moved under the condition that the position of the user is not changed, so that all directions of the virtual object can be seen, the workload is reduced, and the operation efficiency is improved.
The technical scheme of the invention is as follows:
the virtual reality interaction method based on 2D-3D fusion is characterized by comprising the following steps:
step 1: establishing a system communication framework:
a distributed development mode is adopted, data processing, transferring and storing are realized by a server, and a VR helmet and a mobile phone are used as clients; when the VR helmet and the mobile phone work, the control instruction is sent to the server through the TCP/IP communication channel; the VR helmet and the mobile phone carry out data transmission by taking the server as an interaction medium; the TCP/IP communication mechanism supports the synchronization between the mobile phone and the VR helmet in the virtual space; the VR helmet, the mobile phone and the server work in parallel;
step 2: creating a three-dimensional scene in a VR virtual environment:
creating a three-dimensional object model, importing the three-dimensional object model into a VR virtual environment, and converting a model coordinate system where the three-dimensional object model is located into a world coordinate system of the VR virtual environment;
and 3, step 3: the mobile phone is physically combined with the handle:
the mobile phone and the handle are connected into a whole through the connecting structure;
and 4, step 4: and (3) virtual and real model calibration fusion:
creating a three-dimensional mobile phone model of the mobile phone in scene creation software, converting the local coordinate system center of the three-dimensional mobile phone model into the local central coordinate system of the handle, and enabling the pose relationship between the handle virtual model and the three-dimensional mobile phone model in the VR virtual environment to be consistent with the poses of the mobile phone and the handle in the physical world;
and 5: mobile phone screen mirroring:
the method comprises the steps that mobile phone mirror image software loaded on a mobile phone is used for projecting a mobile phone screen onto a window of a server, screen acquisition software loaded on a computer is used for acquiring mobile phone screen information, the mobile phone screen information is transmitted to a Virtual Reality (VR) virtual environment through data flow, and the acquired physical mobile phone screen is used as a texture map and is transmitted to the surface of a three-dimensional mobile phone model in the VR virtual environment in real time;
and 6: the method comprises the following steps of arranging a depth camera in front of a VR helmet, and calibrating the depth camera with the VR camera:
6.1, arranging a depth camera for capturing the environment of the hand area of the operator on the VR helmet, and connecting the depth camera and the VR helmet to the server;
6.2, converting the coordinate value of the depth camera in the camera coordinate system into a coordinate value in a VR helmet coordinate system, so that the point cloud information of the hand of the operator can be displayed in the VR virtual environment;
and 7: hand region segmentation and visualization:
7.1, acquiring a point cloud data set of a current area by using a depth camera, and segmenting the hand area of an operator from a background;
7.2, converting the foreground color image from RGB into HSV color space, applying skin color probability detection, taking skin color pixels obtained from the foreground detection hand area as the final hand area, obtaining the color and depth value of the calibrated hand area according to external parameters obtained when the depth camera is calibrated, uploading the color and depth value to a computer, and finally rendering the visual hand area in a three-dimensional virtual space in a point cloud format in real time so that an operator can see the hand area in a point cloud form;
and step 8: interaction:
the handle is used as a mobile phone tracker, and 2D input and/or basic functions of the mobile phone are/is executed through the mobile phone;
extensive spatial input and 3D tracking is performed by the handle.
Further, one or more of the following interaction modes are employed in step 8:
interaction mode 1: the handle is used as a tracker of the mobile phone, an additional physical button is provided for the mobile phone, and the mobile phone is operated under the condition that a VR helmet is not removed in a VR environment, so that the basic functions of the mobile phone are realized;
interaction mode 2: designing various virtual buttons on a screen of the mobile phone, and clicking the virtual buttons on the screen of the mobile phone to control the virtual objects after the virtual objects are selected by rays emitted by a handle;
interaction mode 3: displaying a 2D object on a mobile phone screen as a 3D virtual object in a VR virtual environment, selecting the 3D virtual object by a handle emission ray, and operating the 3D virtual object; similarly, the mobile phone is moved, so that after a 3D virtual object in the VR virtual environment collides with a ray emitted by the three-dimensional mobile phone model, the 3D virtual object is displayed in a 2D form on a screen of the physical mobile phone;
interaction mode 4: the virtual object is selected by the aid of the handle emitting rays, an object with the same shape appears on the screen of the mobile phone at the same time, and the object in the VR virtual environment is driven to move, rotate and zoom by operating the object on the screen of the mobile phone;
interaction mode 5: on the premise of not removing the VR helmet, a camera of the mobile phone is used for seeing the physical world in real time, and the AR and the VR virtual scene are fused;
interaction mode 6: when the virtual object is operated, the mobile phone is moved, and the operator sees all directions of the virtual object;
interaction mode 7:
the interactive equipment is held by hands, one hand operates a handle, a virtual object is selected by the handle and large-range space 3D input is provided, the other hand slides on a screen of the mobile phone to provide 2D input, and the virtual object can be moved, rotated and zoomed through the handle and the mobile phone;
interaction mode 8:
drawing or writing on the screen of the mobile phone, and displaying in a 3D form in a VR environment in real time.
Further, the connecting structure in the step 2 comprises a connecting structure body, and a sliding groove and a curved hollow groove, wherein the sliding groove is arranged on the connecting structure body and used for installing the mobile phone, and the curved hollow groove is used for installing the handle.
A2D-3D fused virtual reality interaction device comprises a handle; it is characterized in that: the mobile phone is connected with the handle into a whole; after the three-dimensional mobile phone model of the mobile phone and the handle virtual model of the handle are led into a VR virtual environment, the pose relationship between the three-dimensional mobile phone model and the handle virtual model is consistent with the pose relationship between the mobile phone and the handle in the physical world; the screen of the mobile phone can be used as a texture map and transmitted to the surface of a three-dimensional mobile phone model in a VR virtual environment in real time; the mobile phone is matched with the handle, and the virtual reality interaction is realized by using the interaction method, wherein during the interaction: the handle is used as a mobile phone tracker and is used for large-range space input and 3D tracking; the mobile phone is used for carrying out 2D input and/or realizing the basic functions of the mobile phone when the VR helmet is not taken off.
Furthermore, the mobile phone and the handle are connected into a whole through a connecting structure; the connecting structure comprises a connecting main body, a sliding groove and a curved surface hollow groove, wherein the sliding groove is arranged on the connecting main body and used for installing the mobile phone, and the curved surface hollow groove is used for installing the handle.
The invention has the beneficial effects that:
1. in the physical world, the mobile phone and the handle are connected into a whole by using the connecting structure, the communication between the mobile phone and the handle is realized by software design, the operation sense of reality is enhanced, the handle can provide large-range space input and 3D tracking, the mobile phone can perform basic functions such as virtual graphic user input, accurate tangible 2D input, mobile phone photographing and the like, and the mobile phone and the handle are matched with each other to give full play to respective advantages. And carrying out data transmission by using a client-server parallel interaction mode, and carrying out mixed 2D-3D operation in the virtual world.
2. The invention supports the interactive mode of two-hand operation, is beneficial to enhancing the stability of operation and improving the coordination between people and the VR system.
3. The invention supports the interaction mode of multi-device input, can improve the operation efficiency and enrich the types of the interaction modes.
4. The invention utilizes the depth camera to capture the hand environment, the matching of the virtual model and the real physical model and the real-time feedback of the mobile phone screen, and the seamless fusion among the user, the real environment and the virtual environment under various information fusion systems, thereby realizing natural and harmonious human-computer interaction and leading the user to obtain better tactile enhancement experience.
5. After the mobile phone and the handle are combined, various new interaction modes can be realized while respective basic functions are realized, such as 3D space drawing and operation are performed by using a mobile phone screen, multiple-degree-of-freedom fusion operation objects of the mobile phone and the handle, virtual physical space fusion and the like.
Drawings
Fig. 1 is a communication framework diagram of an interactive device according to the present invention (arrows in the diagram represent the flow of data).
Fig. 2 is a physical structure diagram of the combination of the mobile phone and the handle in the interactive device and a registration diagram of a virtual scene, where a is a schematic diagram of the combination of the mobile phone and the handle in a real physical world, b is a schematic diagram of a mobile phone model and a handle model in a virtual environment, and c is a schematic diagram of a connection structure for connecting the mobile phone and the handle.
Fig. 3 a-h are schematic diagrams of eight interaction modes supported by the interaction device of the present invention, respectively.
Fig. 4 is a schematic diagram of an application of the interactive device of the present invention in an entire VR system.
Detailed Description
The invention is further described below with reference to the accompanying drawings.
Referring to fig. 1 to 4, the 2D-3D fusion virtual reality interaction method of the present embodiment includes the following specific steps:
step 1: establishing a system communication framework:
as shown in fig. 1, a distributed development mode is adopted, a computer is used as a server 101 and is responsible for processing, transferring and storing data, and a VR helmet and a mobile phone are used as clients 102 and 103; when the two clients, namely the VR headset and the mobile phone, work, the control instruction is sent to the server 101 through the TCP/IP communication channel. The VR helmet is connected with the server 101 through a data line for data transmission, and the mobile phone is connected with the server 101 through a wireless connection for data transmission; the VR helmet and the mobile phone carry out data transmission by taking a server as an interactive medium; the clients 102 and 103 and the server 101 work in parallel, and receive and transmit data by taking a TCP/IP communication channel as a medium. Wherein the TCP/IP communication mechanism supports synchronization between the handset and the controller VR headset in virtual space.
Step 2: creating a three-dimensional scene in the VR:
2.1, a three-dimensional object model with any shape and size is created by using three-dimensional modeling software, then the three-dimensional object model is led into a VR virtual environment, a VR helmet is connected with a computer through a USB data line, and a three-dimensional scene is displayed in the VR helmet.
2.2, converting a coordinate system:
2.1, after the drawn three-dimensional object model is introduced into the VR virtual environment, because a model coordinate system where the three-dimensional object model is located is not consistent with a world coordinate system of the VR virtual environment, the model coordinate system needs to be converted into the world coordinate system of the VR virtual environment; setting the world coordinate system of the VR virtual environment as O c X c Y c Z c The model coordinate system is O m X m Y m Z m The two conversion relationships are as follows:
in the formula, R represents a rotation matrix, and T represents a translation vector.
If the three-dimensional modeling software adopted in the step 2.1 is 3dMax and the imported VR virtual environment is unity,
and step 3: the mobile phone is physically combined with the handle:
combining the real shapes and sizes of the mobile phone 201 and the handle 202, a connecting structure shown in a figure a in fig. 2 is designed, the connecting structure is provided with a sliding groove 204 for installing the mobile phone and a curved hollow groove for installing the handle 202, the mobile phone 201 is slid into the sliding groove 204 to be fixed, and the handle 202 is placed in the curved hollow groove, so that the mobile phone 210 and the handle 202 are connected into a whole through a connecting structure.
And 4, step 4: and (3) calibrating and fusing a virtual model and a real model:
the method is characterized in that a three-dimensional mobile phone model is directly created in scene creation software, but the pose relationship between a handle virtual model and the three-dimensional mobile phone model is not consistent with the poses of a mobile phone and a handle in a physical world, so that virtual and real model calibration is needed, namely a visual tracking method is utilized to convert a local coordinate system of a geometric center of the three-dimensional mobile phone model into a local central coordinate system of the handle 202, and the method comprises the following steps:
calculating a transformation matrix from a local central coordinate system of the three-dimensional mobile phone model to a local central coordinate system of the handle 202, and setting the local central coordinate of the handle 202 as O p (X p ，Y p ，Z p ) The local center coordinate of the three-dimensional mobile phone model is O s (X s ，Y s ，Z s ) The relationship between the two is as follows:
in the formula, R represents a rotation matrix, and T represents a translation vector.
After the virtual-real model is calibrated, the three-dimensional mobile phone model and the handle virtual model are bound into a whole and consistent with the poses of the mobile phone and the handle in the physical world, wherein the tracking principle between the VR helmet and the handle adopted by the invention is an active optical positioning tracking scheme based on infrared light generated during the development of virtual reality equipment of the VR helmet, the infrared light emitted by infrared light on the handle is observed by a camera of the VR helmet, a primary pose is calculated by detecting a light spot according to a multi-view geometric principle, and then key data generated in an IMU (Inertial Measurement Unit) of the handle is fused to position and track the handle. The handle 202 provides a ready real-time spatial tracking function for the handset 210 in the interactive device of the present invention, and the virtual interactive device moves and rotates with the interactive device in the physical environment to achieve visual and tactile feedback consistent with the real interactive device, as shown in the right diagram of fig. 2, and the dashed lines represent the interactive devices (three-dimensional handset model and handle virtual model) in the VR virtual environment.
And 5: mobile phone screen mirroring:
the method comprises the steps that mobile phone mirror image software (existing open software) loaded on a mobile phone is used for projecting a mobile phone screen onto a computer window through wireless communication, screen acquisition software (existing open software) loaded on a computer (server 101) is used for acquiring mobile phone screen information, the mobile phone screen information is transmitted to a Virtual Reality (VR) environment through data flow, and the acquired physical mobile phone screen is used as a texture map and is transmitted to the surface of a three-dimensional mobile phone model in the VR virtual environment in real time.
Step 6: set up the depth camera before the VR helmet to with the calibration of depth camera and VR camera:
6.1, arranging a depth camera 402 for capturing the environment of the hand area of an operator in front of a VR helmet 401, and connecting the depth camera 402 and the VR helmet 401 to the same computer (a server 101) through a USB data line;
6.2, calibration of depth camera 402 with VR headset 401:
the depth camera 402 obtains the operator hand point cloud in the camera coordinate system, and presents a data point in the VR headset coordinate system, so the depth camera 402 and the VR headset 401 need to be calibrated, that is, the camera coordinate system of the depth camera 402 is converted into the VR headset coordinate system by using the Iterative Closest Point (ICP), so as to achieve the effect of displaying the operator hand point cloud information in the VR virtual scene, where the conversion relationship is as follows:
R＝R r (R l )T
T＝T r -RT l
where R represents a rotation matrix between the depth camera 402 and the VR headset 401, T represents a translation matrix between the depth camera 402 and the VR headset 401, and R represents r And T r Respectively representing the rotation matrix and translation vector of the VR headset 401; r l And T l Representing the rotation matrix and translation vector, respectively, of the depth camera 402.
And 7: hand region segmentation and visualization:
7.1, acquiring a point cloud data set of the current area by using a depth camera 402, and removing point clouds outside a hand area, thereby segmenting the hand area of an operator from a background;
and 7.2, converting the foreground color image from RGB into HSV color space, applying skin color probability detection, and taking skin color pixels obtained from the foreground detection hand area as the final hand area. According to external parameters obtained in the calibration process of the infrared cameras in the depth camera 402 and the VR helmet 401, the color and the depth value of the calibrated hand area are obtained and uploaded to a computer, and finally, the visual hand area is rendered in a three-dimensional virtual space in a point cloud format in real time, so that an operator can see the real hand in a point cloud mode.
And 8: interaction:
the basic functions of the mobile phone are realized in the VR virtual environment: the method comprises the steps that the content of a mobile phone screen is collected in real time through screen mirror image software loaded on a real mobile phone 201, textures are transmitted to a three-dimensional virtual mobile phone model in a VR virtual environment, and chatting and real-time observation of a physical world in the VR virtual environment are achieved through basic software such as a keyboard and a camera of the real mobile phone;
operating the virtual object: one hand 203 operates the handle 202 to enable rays to collide with the three-dimensional object model in the VR virtual environment, a trigger key of the handle 202 is pressed, the three-dimensional object model is temporarily bound with the handle 202 (if the trigger key on the handle is released, the selection of the object is cancelled), and the three-dimensional object model moves along with the movement of the handle 202; meanwhile, the other hand performs 2D input through the mobile phone, firstly selects a moving, rotating or zooming mode, and then a finger slides or clicks a corresponding button on a screen of the real mobile phone to control the virtual object. Wherein: providing input in two degrees of freedom in the X, Y axes based on a planar coordinate system, the third degree of freedom being defined as orthogonal to the other two degrees of freedom: z = X × Y and is controlled by two fingers scrolling up/down on the handset touchscreen.
As shown in fig. 3, the invention supports eight interaction modes, which all require two hands to hold the interaction device, specifically:
interaction mode 1:
the basic functions of the mobile phone are realized in the VR environment: as shown in a in fig. 3, the basic functions of the mobile phone are implemented in the VR virtual environment, wherein the handle 202 serves as a tracker of the mobile phone 201 and provides additional physical buttons for the real mobile phone, and by using the mobile phone screen mirroring and hand area visualization technology, the operator can normally use the mobile phone 201 by observing the change of the mobile phone screen interface in the virtual environment without removing the VR helmet.
Interaction mode 2:
as shown in b in fig. 3, a variety of virtual buttons are designed on the screen of the real mobile phone 201, and after the handle 202 emits a ray to select a virtual object, an operator can control the virtual object by clicking the virtual button on the screen of the mobile phone 201.
Interaction mode 3:
2D-3D content transmission: as shown in the diagram c in fig. 3, the content conversion between the mobile phone and the virtual space is realized. Dragging the 2D object on the screen of the real mobile phone 201 into a VR space, so as to display a corresponding 3D virtual object in a virtual environment, and emitting rays through a handle 202 to select the 3D virtual object so as to operate the 3D virtual object; similarly, the real mobile phone is moved, so that after the 3D virtual object in the VR virtual environment collides with the ray emitted by the three-dimensional mobile phone model, the 3D virtual object is displayed in a 2D form on the screen of the real and virtual mobile phones 201.
Interaction mode 4:
the tangible input of the mobile phone screen is as follows: as shown in d in fig. 3, a handle 202 emits a ray to select a virtual object, an object with the same shape appears on the screen of the mobile phone 201 at the same time, and the object on the screen of the real mobile phone 201 is operated, so that the object in the VR virtual environment is driven to move, rotate and zoom.
Interaction mode 5:
enhancing virtualization: as shown in e in fig. 3, on the premise that the VR helmet is not removed, the physical world is seen in real time by using the camera of the mobile phone 201, and the AR and the VR virtual scene are fused.
Interaction mode 6:
the second virtual perspective: as shown in fig. 3 f, when operating the virtual object, the operator can see the directions of the virtual object by moving the real cellular phone 201.
Interaction mode 7:
asymmetric operation: as shown in g in fig. 3, the interactive device is held by two hands, the handle 202 is operated by one hand, the virtual object is selected and a large-scale space 3D input is provided by using the handle 202, the finger of the other hand slides on the screen of the real mobile phone 201 to provide an accurate 2D input, and the virtual object can be moved, rotated and zoomed through the handle 202 and the mobile phone 201. In fig. 3, the gesture indicated by reference numeral 308 is a fixed gesture for realizing the scaling of the virtual object by two fingers, the gesture indicated by reference numeral 304 is a gesture for realizing the movement or rotation of the virtual object along the X and Y axes by one finger, and the gesture indicated by reference numeral 305 is a gesture for realizing the movement or rotation of the virtual object along the Z axis by two fingers.
Interaction mode 8:
spatially tangible input: as shown in fig. 3 h, the operator can draw or write a word (e.g., draw curve 307) on the screen of the mobile phone 201, and display it in the VR environment in 3D (shown as virtual curve 306).
Claims (5)
- The 2D-3D fused virtual reality interaction method is characterized by comprising the following steps:step 1: establishing a system communication framework:a distributed development mode is adopted, data processing, transferring and storing are realized by a server, and a VR helmet and a mobile phone are used as clients; when the VR helmet and the mobile phone work, the control instruction is sent to the server through the TCP/IP communication channel; the VR helmet and the mobile phone carry out data transmission by taking the server as an interaction medium; the TCP/IP communication mechanism supports synchronization between the mobile phone and the VR helmet in the virtual space; the VR helmet, the mobile phone and the server work in parallel;step 2: creating a three-dimensional scene in a VR virtual environment:creating a three-dimensional object model, importing the three-dimensional object model into a VR virtual environment, and converting a model coordinate system where the three-dimensional object model is located into a world coordinate system of the VR virtual environment;and step 3: the mobile phone is physically combined with the handle:the mobile phone and the handle are connected into a whole through the connecting structure;and 4, step 4: and (3) virtual and real model calibration fusion:creating a three-dimensional mobile phone model of the mobile phone in scene creation software, converting the local coordinate system center of the three-dimensional mobile phone model into the local central coordinate system of the handle, and enabling the pose relationship between the handle virtual model and the three-dimensional mobile phone model in the VR virtual environment to be consistent with the poses of the mobile phone and the handle in the physical world;and 5: mobile phone screen mirroring:the method comprises the steps that mobile phone mirror image software loaded on a mobile phone is used for projecting a mobile phone screen onto a window of a server, screen acquisition software loaded on a computer is used for acquiring mobile phone screen information, the mobile phone screen information is transmitted to a Virtual Reality (VR) virtual environment through data flow, and the acquired physical mobile phone screen is used as a texture map and is transmitted to the surface of a three-dimensional mobile phone model in the VR virtual environment in real time;step 6: the method comprises the following steps of arranging a depth camera in front of a VR helmet, and calibrating the depth camera with the VR camera:6.1, arranging a depth camera for capturing the environment of the hand area of the operator on the VR helmet, and connecting the depth camera and the VR helmet to the server;6.2, converting the coordinate value of the depth camera in the camera coordinate system into a coordinate value in a VR helmet coordinate system, so that the point cloud information of the hand of the operator can be displayed in the VR virtual environment;and 7: hand region segmentation and visualization:7.1, acquiring a point cloud data set of a current area by using a depth camera, and segmenting the hand area of an operator from a background;7.2, converting the foreground color image from RGB into HSV color space, applying skin color probability detection, using skin color pixels obtained from the foreground detection hand area as the final hand area, obtaining the color and depth value of the calibrated hand area according to external parameters obtained when the depth camera is calibrated, uploading the color and depth value to a computer, and finally rendering the visualized hand area in a three-dimensional virtual space in a point cloud format in real time to enable an operator to see the hand area in a point cloud form;and 8: interaction:the handle is used as a mobile phone tracker, and 2D input and/or basic functions of the mobile phone are/is executed through the mobile phone;extensive spatial input and 3D tracking is performed by the handle.
- 2. The 2D-3D fused virtual reality interaction method according to claim 1, wherein: one or more of the following interaction modes are adopted in the step 8:interaction mode 1: the handle is used as a tracker of the mobile phone, an additional physical button is provided for the mobile phone, and the mobile phone is operated under the condition that a VR helmet is not removed in a VR environment, so that the basic functions of the mobile phone are realized;interaction mode 2: designing various virtual buttons on a screen of the mobile phone, and clicking the virtual buttons on the screen of the mobile phone to control the virtual objects after the virtual objects are selected by rays emitted by a handle;interaction mode 3: displaying a 2D object on a mobile phone screen as a 3D virtual object in a VR virtual environment, selecting the 3D virtual object by a handle emission ray, and operating the 3D virtual object; similarly, the mobile phone is moved, so that after a 3D virtual object in the VR virtual environment collides with a ray emitted by the three-dimensional mobile phone model, the 3D virtual object is displayed in a 2D form on a screen of the physical mobile phone;interaction mode 4: the virtual object is selected by the aid of the handle emitting rays, an object with the same shape appears on the screen of the mobile phone at the same time, and the object in the VR virtual environment is driven to move, rotate and zoom by operating the object on the screen of the mobile phone;interaction mode 5: on the premise of not removing the VR helmet, a camera of the mobile phone is used for seeing the physical world in real time, and the AR and the VR virtual scene are fused;interaction mode 6: when the virtual object is operated, the mobile phone is moved, and an operator sees all directions of the virtual object;interaction mode 7:the interactive equipment is held by hands, one hand operates a handle, a virtual object is selected by the handle and large-range space 3D input is provided, the other hand slides on a screen of the mobile phone to provide 2D input, and the virtual object can be moved, rotated and zoomed through the handle and the mobile phone;interaction mode 8:drawing or writing on the screen of the mobile phone, and displaying in a 3D form in a VR environment in real time.
- 3. The 2D-3D fused virtual reality interaction method according to claim 1 or 2, wherein: the connecting structure in the step 2 comprises a connecting structure body, a sliding groove and a curved surface hollow groove, wherein the sliding groove is arranged on the connecting structure body and used for installing the mobile phone, and the curved surface hollow groove is used for installing the handle.
- 4. A2D-3D fused virtual reality interaction device comprises a handle; the method is characterized in that: the mobile phone is connected with the handle into a whole; after the three-dimensional mobile phone model of the mobile phone and the handle virtual model of the handle are introduced into the VR virtual environment, the pose relationship between the three-dimensional mobile phone model and the handle virtual model is consistent with the pose relationship between the mobile phone and the handle in the physical world; the screen of the mobile phone can be used as a texture map to be transmitted to the surface of a three-dimensional mobile phone model in a VR virtual environment in real time; the mobile phone is matched with the handle, and the interaction method of claim 1 or 2 is used for realizing virtual reality interaction, wherein during interaction: the handle is used as a mobile phone tracker and is used for large-range space input and 3D tracking; the mobile phone is used for carrying out 2D input and/or realizing the basic functions of the mobile phone when the VR helmet is not removed.
- 5. The 2D-3D fused virtual reality interaction device of claim 4, wherein: the mobile phone and the handle are connected into a whole through a connecting structure; the connecting structure comprises a connecting main body, a sliding groove and a curved surface hollow groove, wherein the sliding groove is arranged on the connecting main body and used for installing the mobile phone, and the curved surface hollow groove is used for installing the handle.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210917515.2A CN115328304A (en) | 2022-08-01 | 2022-08-01 | 2D-3D fused virtual reality interaction method and device |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210917515.2A CN115328304A (en) | 2022-08-01 | 2022-08-01 | 2D-3D fused virtual reality interaction method and device |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN115328304A true CN115328304A (en) | 2022-11-11 |
Family
ID=83919375
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202210917515.2A Pending CN115328304A (en) | 2022-08-01 | 2022-08-01 | 2D-3D fused virtual reality interaction method and device |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN115328304A (en) |
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115830112A (en) * | 2022-11-25 | 2023-03-21 | 之江实验室 | Mixed reality interaction method and system based on handheld real object |
| CN117608410A (en) * | 2024-01-17 | 2024-02-27 | 山东五纬数字科技有限公司 | 3D virtual digital person interaction system and method |
| CN117608410B (en) * | 2024-01-17 | 2024-05-31 | 山东五纬数字科技有限公司 | 3D virtual digital person interaction system and method |
-
2022
- 2022-08-01 CN CN202210917515.2A patent/CN115328304A/en active Pending
Cited By (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115830112A (en) * | 2022-11-25 | 2023-03-21 | 之江实验室 | Mixed reality interaction method and system based on handheld real object |
| CN115830112B (en) * | 2022-11-25 | 2023-09-22 | 之江实验室 | Mixed reality interaction method and system based on handheld real object |
| CN117608410A (en) * | 2024-01-17 | 2024-02-27 | 山东五纬数字科技有限公司 | 3D virtual digital person interaction system and method |
| CN117608410B (en) * | 2024-01-17 | 2024-05-31 | 山东五纬数字科技有限公司 | 3D virtual digital person interaction system and method |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| Grossman et al. | Multi-finger gestural interaction with 3d volumetric displays | |
| CN108762482B (en) | Data interaction method and system between large screen and augmented reality glasses | |
| CN106575354B (en) | Virtualization of tangible interface objects | |
| JP3997566B2 (en) | Drawing apparatus and drawing method | |
| CN110163942B (en) | Image data processing method and device | |
| Millette et al. | DualCAD: integrating augmented reality with a desktop GUI and smartphone interaction | |
| JP4513830B2 (en) | Drawing apparatus and drawing method | |
| CN104915979A (en) | System capable of realizing immersive virtual reality across mobile platforms | |
| CN103793060A (en) | User interaction system and method | |
| CN102193631A (en) | Wearable three-dimensional gesture interaction system and using method thereof | |
| WO2005069170A1 (en) | Image file list display device | |
| Mohr et al. | Mixed reality light fields for interactive remote assistance | |
| CN104656893A (en) | Remote interaction control system and method for physical information space | |
| Englmeier et al. | A tangible spherical proxy for object manipulation in augmented reality | |
| CN104081307A (en) | Image processing apparatus, image processing method, and program | |
| CN109952552A (en) | Visual cues system | |
| CN112198962A (en) | Method for interacting with virtual reality equipment and virtual reality equipment | |
| CN113672099A (en) | Electronic equipment and interaction method thereof | |
| Thomas et al. | Spatial augmented reality—A tool for 3D data visualization | |
| CN113961107B (en) | Screen-oriented augmented reality interaction method, device and storage medium | |
| CN115328304A (en) | 2D-3D fused virtual reality interaction method and device | |
| CN106971370B (en) | System and method for taking picture as background picture plane vector diagram | |
| CN111467803A (en) | In-game display control method and device, storage medium, and electronic device | |
| CN109102571B (en) | Virtual image control method, device, equipment and storage medium thereof | |
| CN113849112A (en) | Augmented reality interaction method and device suitable for power grid regulation and control and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |