TWI571767B - Rear-screen three-dimension interactive system and method - Google Patents

Description

Back-screen three-dimensional interactive system and method

The present invention relates to a system and method for interacting with a three-dimensional virtual reality, and more particularly to a system and method capable of interacting with a three-dimensional virtual reality image from the back of the screen.

With the advancement of three-dimension computer graphics technology, today's computer technology has enabled the average consumer to display 3D models of objects on notebook computers. With the advancement of technology, 3D computer graphics technology has been widely used in various Industry, including animation, video games and computer-aided design (CAD).

However, for today's 3D display technology, if the operator wants to interact with the 3D model, there are still many limitations from the perspective of technology and control devices, and the operator perceives How to perform good space coupling between the perceived visual position and the model operating position is also a bottleneck in the development of today's 3D interactive technology.

Another problem encountered is that today's displays are still limited to a two-dimension display. Although 3D models are designed to exist in three-dimensional space, they are essentially two-dimensional displays, and it is still difficult to express these. The stereoscopic version of the 3D model. Therefore, in order to make the model feel like a "pop-up" display screen, today's 3D display technology usually presents two offset images of the same image to the left and right eyes of the viewer, so that the viewer produces the image. Stereoscopic, but with this technology, the operator has to wear extra head-mounted equipment.

Another technique for expressing 3D stereoscopic perception through a 2D display is to dynamically and correspondingly change the observed object by using the "motion parallax" effect, which tracks the relative displacement of the viewer's observation point (eye). The position of the (viewed object), thereby simulating the effect of 3D stereoscopic perception.

Another limitation comes from the input device. Today's input devices are also 2D essential, allowing accurate 2D input, but not suitable for 3D input, because the input device itself has a limited degree of freedom (DoF). The input in the third dimension cannot be accepted, and in order to overcome the limitation of freedom, controller input devices with three or more degrees of freedom are developed to enhance the practicality in 3D interaction.

However, the biggest problem encountered in the 3D interactive system is still the problem of space coupling between the control input and the visual output space, that is, the dynamic interaction and relative position change between the 3D model and the human hand. In the operation of the human body, the human eye first observes the position of the physical object, and then guides the hand through the sensing system to perform an action to touch or hold the physical object. This process is called hand-eye coordination.

However, the current 3D motion sensor can only handle the spatial coupling of the front-screen 3D interactive system with the motion sensor in front of the screen. It can't handle the rear-screen configuration. The spatial coupling of the 3D interactive system, and the application of the conventional technology to the 3D motion sensor, is limited to the application of augmented reality.

As a result of the job, the applicant, based on the shortcomings of the conventional technology, has carefully tested and researched, and has a perseverance spirit, and finally conceived the case of the back-screen three-dimensional interactive system and the party. The method is a 3D interactive system based on a back-screen configuration, which can perform good spatial coupling to the back-screen 3D interactive system, and can overcome the above disadvantages. The following is a brief description of the present invention.

In view of the shortcomings in the prior art, the present invention proposes a virtual reality three-dimensional interactive system on a rear-screen architecture, which is simplified and simpler than the conventional technology. A minimally configurable combination of components forms a complete rear-screen three-dimensional interactive system, with spatially coupled calculations that allow the operator to truly feel the kinesthetic vision and gestures Drive virtual objects for rotation, displacement, motion, and more.

The system of the present invention can be widely and practically applied in the fields of animation, video game, industrial design, and computer-aided design (CAD), for example, in the design verification test (DVT) of the manufacturing industry. It is used in the design review stage of 3D modeling in the animation industry.

The present invention provides a three-dimensional interactive system, which is applied to a virtual reality technology, comprising: a computer device; a display device electrically connected to the computer device and having a display for displaying a three-dimensional image of an object a face, a front side of one of the operators, and a back side of the front side, the display surface being located on the front side and facing the operator; a motion sensing device electrically connected to the computer device and disposed on the surface a first distance between the back side and the display device for sensing the operation of the operator on the back side; and an image sensing device electrically connected to the computer device and disposed on the back The front side is spaced apart from the display device by a second distance for sensing visually one of the visual movements of the operator.

The present invention further provides a three-dimensional interactive system, which is applied to a virtual reality technology, comprising: a computer device for processing digital data; a display device electrically connected to the computer device and having a display surface displaying a three-dimensional image of an object, a front side facing one of the operators, and a back side opposite to the front side, the display surface being located on the front side and facing the operator; and a motion sensing device The computer device is electrically connected to the back side and spaced apart from the display device for sensing a movement of the operator on the back side, wherein the operator changes by observing the three-dimensional image In response to the action, the motion sensing device receives the motion, and the computer device adjusts the displayed three-dimensional image to thereby enable the operator to experience an interactive experience with the object.

The present invention further provides a three-dimensional interactive method, which is applied to a virtual reality technology, comprising: displaying a three-dimensional spatial stereoscopic image of a virtual reality to an operator through a display device, the content of the image being simulated on the display device a hypothetical state of the virtual space on the back side, the operator perceiving the virtual reality through the image; the operator performs a hand-based operation on the back side according to the content of the virtual reality; Sensing the operation action and correspondingly adjusting the display content of the display device to the virtual reality; and the operator changes the operation action according to the adjusted feedback of the virtual reality.

100‧‧‧Back screen three-dimensional interactive system

110‧‧‧motion sensor

120‧‧‧ screen

130‧‧‧Portable computer equipment

140‧‧‧ Operator

150‧‧‧Virtual Teapot

160‧‧‧Hands

160”‧‧‧Virtual Hand

300‧‧‧ screen

310‧‧‧Table

320‧‧‧Virtual Teapot

330‧‧‧

340‧‧‧ wall

350‧‧‧ motion sensor

360‧‧‧Hands

360"‧‧‧Virtual Hand

400‧‧‧Back screen three-dimensional kinesthetic interactive system

410‧‧‧ motion sensor

420‧‧‧ screen

430‧‧‧Portable computer device

440‧‧‧Operator

450‧‧‧Cap marks

460‧‧‧Image sensor

700‧‧‧Back screen three-dimensional kinesthetic interactive system

710‧‧‧ motion sensor

720‧‧‧ screen

730‧‧‧Portable computer device

750‧‧ visual movement mark

760‧‧‧Image sensor

B‧‧‧ rear

F‧‧‧ ahead

Behind R‧‧‧

W‧‧‧Width

H‧‧‧ Height

D‧‧‧Distance

O‧‧‧ origin

x _V , y _V , z _V ‧‧‧ Location of virtual vision

x _A , y _A , z _A ‧‧‧ Real visual location

r , t , n ‧ ‧ the coordinate value of any point on the near plane

l , b , n ‧ ‧ the coordinate value of any point on the near plane

r , b , n ‧‧ ‧ the coordinate value of any point on the near plane

r , t , n ‧ ‧ the coordinate value of any point on the near plane

X‧‧‧Cart coordinates X-axis

Y‧‧‧Cart coordinates Y-axis

Z‧‧‧Cart coordinates Z-axis

Step 810‧‧‧ Displaying a three-dimensional spatial stereoscopic image of the virtual reality through the display device, the content of the image simulating a hypothetical state of the virtual space on the back side of the display device, the operator perceiving the virtual reality through the image territory

Step 820‧‧‧ The operator performs a hand-based operation on the back side according to the content of the virtual reality

Step 830‧‧‧ sensing the operation from the back side and correspondingly adjusting the display content of the display device to the virtual reality

Step 840‧‧‧ The operator changes the operation according to the adjusted feedback of the virtual reality

Figure 1 is a schematic diagram showing the system of the back-screen three-dimensional interactive system of the present invention.

Figure 2 is a schematic diagram showing the operation of the back-screen three-dimensional interactive system of the present invention.

Figures 3(a) and 3(b) show schematic diagrams of the actual operation of the back-screen three-dimensional interactive system of the present invention.

Figure 4 is a schematic diagram showing the system of the back-screen three-dimensional kinesthetic interactive system of the present invention.

Figures 5(a) through 5(c) show schematic diagrams of the spatial coupling relationship of the kinesthetic vision system of the present invention.

Figure 6 is a schematic diagram showing the geometric relationship between the frustum and the near plane of the kinesthetic vision system of the present invention.

Figure 7 is a schematic diagram showing the actual system of the back-screen three-dimensional kinesthetic interactive system of the present invention.

Figure 8 is a flow chart showing the three-dimensional kinesthetic interaction method of the back screen of the present invention.

The present invention will be fully understood by the following examples, which can be understood by those skilled in the art, but the implementation of the present invention may not be limited by the following embodiments; the drawings of the present invention are The limitations on size, size and scale are not included, and the size, size and scale of the actual implementation of the invention are not limited by the drawings of the invention.

The term "preferred" as used herein is non-exclusive and should be understood to mean "preferably, but not limited to," an open term, and does not have a limiting meaning, and does not exclude other features or steps; as described or recited in any specification or claim. Any of the steps may be performed in any order, and are not limited to the order described in the claims; the scope of the present invention should be determined only by the accompanying claims and their equivalents, and should not be determined by the embodiments of the embodiments; The term "comprising" and its variations appearing in the specification and claims are an open term and are not intended to be limiting, and do not exclude other features or steps.

Please refer to FIG. 1 and FIG. 2 together. FIG. 1 is a schematic diagram showing the system of the back-screen three-dimensional interactive system of the present invention, and FIG. 2 is a schematic diagram showing the operation of the back-screen three-dimensional interactive system of the present invention, as shown in FIG. Shown, the back-screen three-dimensional interactive system (rear-screen) proposed by the invention The 3D interactive system 100 includes a motion sensor 110 and a portable computer device 130 having a screen 120. The operator 140 is located at the front F of the screen 120 of the portable computer device 130, and the operator 140 views the display of the screen 120. The portable computer device 130 is operated by the content, and the motion sensor 110 is disposed behind the screen 120 of the portable computer device 130 and separated from the screen 120 by a certain distance. The motion sensor 110 is a sensor that can sense and detect. The action or trajectory of a human finger, hand or other finger is measured, tracked, and recorded, and provided to the portable computer device 130 as a sensor for inputting information, such as a motion controller manufactured by Leap Motion.

The above-mentioned back-screen three-dimensional interactive system can be directly applied to the virtual reality technology to construct an interaction in real time between the virtual reality and the operator, as shown in FIG. 2, the screen 120 shows a simple virtual reality. In this virtual reality, there is a virtual stereo teapot 150. The operator expects to enter the virtual reality environment to play the stereo teapot 150 through the back-screen three-dimensional interactive system of the present invention. When the operator 140 needs to see the situation on the screen 120, he naturally and naturally moves his hand 160 to the rear B of the screen 120, and touches the teapot 150 that looks like the rear of the screen 120. When the hand 160 of the operator 140 enters the range of the screen 120, the system will immediately display on the screen 120 a same proportion and size as the operator 140 hand 160 through the complete sensing of the motion sensor 110. The virtual hand 160" of the posture, the gesture, and the position, movement, and the like of the virtual hand 160" completely compare the real hand 160 of the operator 140, and the operator 140 can observe the virtual The position of the hand 160 is adjusted to the position of the hand 160" until the virtual hand 160" touches the virtual teapot 150.

The virtual hand 160" described above is constructed in a virtual reality environment, and The corresponding configuration is the same size and position as the real hand 160 behind the screen. In this way, the operator 140 can almost feel that it is actually putting his hand into the virtual reality in the screen, and with The virtual objects interact directly, and all objects in the virtual reality are simulated with the correct stereoscopic perspective dimensions, and the virtual visual coordinates and real visual coordinates are converted accordingly.

Please refer to FIG. 3(a) and FIG. 3(b) together. Figures 3(a) and 3(b) show the actual operation scenario diagram of the back-screen three-dimensional interactive system of the present invention, similar to the disclosure of FIG. The screens of the 3(a) and 3(b) diagrams 300 also show a simple virtual reality, but the virtual reality constructed by the 3(a) and 3(b) diagrams is virtual compared to the second diagram. The reality is more detailed, and the virtual reality content of the 3(a) and 3(b) diagrams includes the virtual teapot 320 placed on the table 310, and the table 310 is also interspersed with some messy debris 330, and the table 310 is supported by On the wall surface 340, a motion sensor 350 is placed near the position where the table 310 and the wall surface 340 meet.

The operator sees the situation on the screen 300 according to himself, naturally and naturally moves his hand 360 to the rear of the screen 300, and touches the teapot 320 that appears to be behind the screen 300, when the operator's hand When 360 enters the range of the screen 300, the system will instantly display a virtual hand 360" having the same scale, size, posture, and gesture as the operator's hand 360 on the screen 300, and the virtual hand 360" The position action and the like, the operator's real hand 360 can be completely instantly compared, and the operator can adjust the position of the real hand 360 by observing the position of the virtual hand 360", and can freely perform various kinds with the virtual teapot 320. Interaction, for example: 360° rotation of the virtual viewing of the teapot 320 and so on.

In the above-mentioned back-screen three-dimensional interactive system, the visual vision location of the virtual reality displayed on the entire screen does not move as the operator moves. The location changes accordingly, that is, the perceived visual location and the manipulating locations of model lack space coupling, so if the operator's viewing position changes, the system screen displays The position of the model operation does not change, that is, the operator does not feel the kinesthetic from the display of the screen, so the present invention further adds the kinesthetic vision system to the above-mentioned back-screen three-dimensional interaction. system.

4 is a schematic diagram of a system of a back-screen three-dimensional kinesthetic interactive system of the present invention. The rear-screen 3D kinesthetic interactive system 400 of the present invention includes a motion sensor 410 having a screen 420. The portable computer device 430 and the image sensor 460 are located at the front F of the screen 420 of the portable computer device 430. The operator 240 operates the portable computer device 430 by observing the display content of the screen 420. The motion sensor 410 is disposed behind the screen 420 of the portable computer device 430. The motion sensor 410 is an action or trajectory that can sense, detect, track, and record human fingers, hands, or other fingers. And providing the portable computer device 430 as a sensor for inputting information, for example, a motion controller manufactured by Leap Motion.

In order to track the visual movement of the real human eye, and correspondingly change the visual position of the model displayed by the screen and the visual perspective of the displayed virtual reality, in this embodiment, further behind the operator 440, the screen An image sensor 460 is disposed on the front F of the 420, for example, a webcam, etc., and the image sensor 460 is mounted on the rear of the operator 440 through the stand 470, and the height is close to the head, and is connected to the screen 420 and the operator. The distance between the 440 is a certain distance. In order to increase the recognition rate of the head, the head of the operator 440 is worn with a cap mark 450 to represent the visual movement of the real human eye by tracking the movement of the head.

5(a) to 5(c) are diagrams showing the spatial coupling relationship of the kinesthetic vision system of the present invention. In order to construct a spatially coupled image, the present invention constructs a suitable virtual situation by synchronizing the positions of real vision and virtual vision. The spatial coupling relationship can be instantly reflected, and the operator 440 can feel the kinesthetic from the screen 420. As shown in the fifth (a) to (c) diagrams, the object is gazing during the simultaneous movement of the virtual vision and the real vision. The relative displacement, called "motion parallax", will be presented in virtual reality in virtual depth. The geometric relationship between virtual vision and real vision is as follows: Where x _V , y _V , z _V represent the position of the virtual vision, x _A , y _A , z _A represent the position of the real vision, the coordinate origin O is the surface of the center of the screen, and W _V represents the width of the near plane. W _A represents the width of the screen, H _V represents the height of the near plane, H _A represents the height of the screen, D _V represents the distance from the origin O of the virtual vision to the center of the near plane, and D _A represents the origin of the coordinates from the real vision to The distance between the center of the screen.

Figure 6 is a schematic diagram showing the geometric relationship between the truncated cone and the near plane of the kinesthetic vision system of the present invention, in order to simulate the true visual view of the truncated cone shape through the virtual frustum (frustum), Know the position of the operator's eye relative to the monitor for a fiustum calibration, in Figure 6, where r , l , t , b , n represent any near-plane on the near-plane truncated cone On the positional parameter relative to the real visual coordinate, the parameter f is the distance from any near plane to the z- axis direction. In this embodiment, it is set to infinity. When the eye moves, the above parameters will change and need to be taken. Enter the following projection matrix formula: Through the above formulas (1) to (4), the kinesthetic vision can be brought into the back-screen three-dimensional interactive system of the present invention, and become a back-screen three-dimensional kinesthetic interactive system, through the back-screen three-dimensional kinesthetic interaction. In system 400, operator 440 will be able to very clearly perceive the sensitive kinesthetic vision presented in the virtual reality.

FIG. 7 is a schematic diagram showing the actual system of the back-screen three-dimensional kinesthetic interaction system of the present invention. The back-screen three-dimensional kinesthetic interaction system 700 of FIG. 7 includes a motion sensor 710, a portable computer device 730 having a screen 720, Components such as the visual movement marker 750 and the image sensor 760.

Figure 8 is a flow chart showing the three-dimensional kinesthetic interaction method of the back screen of the present invention. In summary, the present invention also provides a back-screen three-dimensional kinesthetic interaction method, which mainly includes the following series of steps: Step 810: Display a three-dimensional spatial stereoscopic image of the virtual reality through the display device, and the image is The content simulates a hypothetical state of the virtual space on the back side of the display device, the operator perceives the virtual reality through the image; step 820: the operator performs hand-based on the back side according to the content of the virtual reality Operation operation; step 830: sensing the operation action from the back surface and correspondingly adjusting the display content of the display device to the virtual reality; and step 840: the operator changes the operation according to the adjusted feedback of the virtual reality action.

The above embodiments of the present invention may be combined or substituted with each other in any way, thereby deriving more embodiments, without departing from the scope of the present invention. Further embodiments of the present invention are provided as follows:

Embodiment 1: A three-dimensional interactive system is applied to a virtual reality technology, comprising: a computer device; a display device electrically connected to the computer device and having a three-dimensional image for displaying an object a display surface, a front side facing one of the operators, and a back side opposite to the front side, the display surface being located on the front side and facing the operator; a motion sensing device electrically connected to the computer device and disposed on the display surface The back side is spaced apart from the display device by a first distance for sensing an action of the operator on the back side; and an image sensing device electrically connected to the computer device and disposed on the back side The front side is spaced apart from the display device by a second distance for sensing visually one of the visual movements of the operator.

Embodiment 2: The system of embodiment 1, further comprising: a visual movement marker disposed on a head of the operator, the image sensing device sensing movement of the visual movement marker to track the visual movement .

Embodiment 3: The system of Embodiment 1, wherein the operator changes the action by observing the three-dimensional image, the motion sensing device receives the action and the image sensing device receives the visual movement, and the computer device adjusts the The three-dimensional image is displayed thereby enabling the operator to experience an interactive experience with the object.

Embodiment 4: The system as described in Embodiment 1, the display device, the motion sensing device, and the image sensing device are electrically connected to the computer device through one of a wired communication technology and a wireless communication technology. The communication technology is selected from one of Bluetooth communication technology, one Wi-Fi communication, one 3G communication technology, one 4G communication technology and one combination thereof.

Embodiment 5: The system of Embodiment 1, wherein the computer device is one selected from the group consisting of a notebook computer, a desktop computer, a tablet computer, a tablet phone, and a smart phone.

Embodiment 6: The system of Embodiment 1, wherein the motion sensing device is selected from one of a motion controller and an infrared motion sensor.

Embodiment 7: The system of Embodiment 1, wherein the image sensing device is selected from the group consisting of a network camera, a digital camera, and a video camera.

Embodiment 8 is a virtual reality technology, comprising: a computer device for processing digital data; a display device electrically connected to the computer device and having a display surface displaying a three-dimensional image of an object, a front side facing one of the operators, and a back side opposite to the front side, the display surface being located on the front side and facing the operator; and a motion sensing device The computer device is electrically connected to the back side and spaced apart from the display device for sensing a movement of the operator on the back side, wherein the operator changes by observing the three-dimensional image In response to the action, the motion sensing device receives the motion, and the computer device adjusts the displayed three-dimensional image to thereby enable the operator to experience an interactive experience with the object.

Embodiment 9: A three-dimensional interactive method is applied to a virtual reality technology, comprising: displaying, by a display device, a third-dimensional stereoscopic image of a virtual reality to an operator, where the content of the image is simulated on the display device a hypothetical state of the virtual space on the back side, the operator perceiving the virtual reality through the image; the operator performs a hand-based operation on the back side according to the content of the virtual reality; Sensing the operation action and correspondingly adjusting the display content of the display device to the virtual reality; and the operator is based on the adjusted The feedback of the virtual reality changes the action.

Embodiment 10: The method of Embodiment 9, the virtual reality further includes an object, the operator changes the operation action by observing the image, and the display content of the virtual reality is changed according to the operation. The action is adjusted to enable the operator to experience an interactive experience with the object.

The embodiments of the present invention may be combined or substituted with each other in any way, thereby deriving more embodiments, without departing from the scope of the present invention, and the scope of the present invention is defined by the scope of the present invention. The record is subject to change.

100‧‧‧Back screen three-dimensional interactive system

110‧‧‧motion sensor

120‧‧‧ screen

130‧‧‧Portable computer equipment

140‧‧‧ Operator

160‧‧‧Hands

B‧‧‧ rear

F‧‧‧ ahead

Claims (9)

A three-dimensional interactive system is applied to a virtual reality technology to generate a kinesthetic vision for an operator, comprising: a computer device; a display device electrically connected to the computer device and having a display for containing a display surface of one of the three-dimensional virtual reality images of the object, a front side facing the operator, and a back side opposite to the front side, the display surface being located on the front side and facing the operator, the eyes of the operator facing the Displaying a surface and feeling a real vision; a motion sensing device electrically connected to the computer device and disposed on the back side and spaced apart from the display device by a first distance for sensing the operator Acting on one of the back sides; and an image sensing device electrically connected to the computer device and disposed on the front side and spaced apart from the display device by a second distance, the image sensing device facing the display The device and the operator sense a virtual vision and are used to sense the operator's head movement to represent visual movement of the operator in the real vision, the computer Aligning with the motion sensed by the motion sensing device, calculating a motion parallax between the real vision and the virtual vision through a spatial coupling technique, and instantly representing the virtual reality image on the three-dimensional virtual reality image, causing the operator to When one of the action and the visual movement is performed, spatial kinesthesia can be instantly sensed from the three-dimensional virtual reality image, and the motion is changed according to the kinesthetic three-dimensional virtual reality image. The system of claim 1, further comprising: a visual movement marker disposed on a head of the operator, the image sensing device The movement of the visual movement marker is measured to track the visual movement. The system of claim 1, wherein the operator changes the action by observing the three-dimensional virtual reality image, the motion sensing device receives the action, and the image sensing device receives the visual movement, the computer device The displayed three-dimensional virtual reality image is adjusted to enable the operator to interact with the object. The system of claim 1, wherein the display device, the motion sensing device, and the image sensing device are electrically connected to the computer device through a wired communication technology and a wireless communication technology, the wireless communication The technology is selected from one of Bluetooth communication technology, one Wi-Fi communication, one 3G communication technology, one 4G communication technology and one combination thereof. The system of claim 1, wherein the computer device is selected from the group consisting of a notebook computer, a desktop computer, a tablet computer, a tablet phone, and a smart phone. The system of claim 1, wherein the motion sensing device is selected from one of a motion controller and an infrared motion sensor. The system of claim 1, wherein the image sensing device is selected from the group consisting of a webcam, a digital camera, and a video camera. A three-dimensional interactive method is applied to a virtual reality technology, comprising: displaying a three-dimensional virtual reality image to an operator through a display device, the content of the image simulating a hypothesis of a virtual space on the back side of the display device State, the operator's eyes are transparent Observing the three-dimensional virtual reality image to perceive the hypothetical state and feeling a true vision; the operator performs a hand-based operation on the back surface according to the three-dimensional virtual reality image, and performs the real vision a visual movement; sensing a virtual vision comprising the operator and the display device from a front side of the display device; sensing the operation action and the visual movement from the back side, calculating the real vision through the spatial coupling technique The motion parallax between the virtual visions is instantly adjusted on the three-dimensional virtual reality image; and the operator can instantly feel from the three-dimensional virtual reality image when performing one of the operation actions and the visual movement The kinesthetic in space changes the operational action according to the adjusted three-dimensional virtual reality image. The method of claim 8, wherein the three-dimensional virtual reality image further includes an object, and the operator changes the operation action by observing the image, and the display content of the three-dimensional virtual reality image is changed according to the change This operational action is adjusted to enable the operator to experience an interactive experience with the object.