TWI603288B - Method using 3d geometry data for virtual reality image presentation and control in 3d space - Google Patents

Description

Method for rendering 3D geometric data for rendering and control of virtual reality images in 3D space

The present invention relates to the field of 3D photographic presentation, which uses virtual reality technology to visualize high quality photo images, which also employs 3D modeling techniques (3D). The advantage of modeling technology to provide geometry data for physical measurement or control, and will be used for augmented reality applications, which can also be used for instant Real time application stereoscopic display system.

A virtual reality uses a set of photographic images to visualize solid objects viewed from different viewing angles. It provides high quality photographic images for rendering applications. However, due to the limited number of photographic photographs, the viewing angle is limited by the number of discrete positions of the photographing position, resulting in a non-smooth animation. Moreover, the photographic image does not contain geometric data. They cannot be precisely aligned when presented and cannot be used for any physical related application, ie for measurement or for control.

3D modeling is another solution for presenting physical objects. It has geometric information that can be used to include physical applications such as augmented reality. However, in order to obtain accurate geometric data and for high-quality rendering, it is presented along with texture mapping techniques, and it takes a lot of effort to extract geometric data and store a large number of material images. Moreover, it is not easy to perform photo-realistic rendering with a low performance personal computing device.

Therefore, high image quality, immersive virtual reality rendering is required for commercial applications, and geometric information needs to be incorporated for physical augmented reality applications, especially for desktop personal computers (desktop PCs). Or a mobile device such as a tablet PC or a smart phone. In order to provide a high-quality viewing experience and physical information, one way is to combine the advantages of virtual reality and 3D modeling to provide cost-effective use of computing devices that are readily available. Solutions and quality requirements. The present invention achieves these goals and can be implemented with existing computing devices and mechanical systems.

One aspect of the present invention describes a method of combining a set of photo frames with a set of geometric information and a 3D space (3D space) under a viewing window of an operating device. A systematic way of presenting 2D images, which also describes the related parameters of the image frame and the viewing transformation of the physical object in the 3D rendering space. The mathematical relationship between.

Another embodiment of the present invention describes a system that includes a computer-control mechanical system to automatically capture photographic images at different viewing angles, and also describes a variety of optical scanning hard An optical scanning hardware or a 3D geometric data scanner based on a photo taking camera that obtains 3D geometric data via a silhouette or a referencing mat or a reference stripe. 3D geometry data scanning subsystem.

Another embodiment of the present invention describes a software system and a client viewing device for implementing the present invention, wherein the software system consists of a workstation and a storage system (storage). System) and a remote server, also describes a software program for integrating 2D photographic photos and scanned 3D geometric data to manually or automatically generate a set of control parameters. , which describes the use of images and geometric data to download And a software program that performs viewing, measuring, and controlling a photo realistic solid object.

Another embodiment of the invention describes an extended application of a hardware and software system for implementing stereoscopic display and control functions.

The above described features and advantages of the invention will be apparent from the following description.

100‧‧‧ method

102‧‧‧ physical objects

104‧‧‧ Viewers

106‧‧‧View window

108, 224, 284 ‧ ‧ photographic images

110, 192, 204, 226‧‧‧3D grid

120‧‧‧Specific examples

121‧‧‧Image Capture System

122‧‧‧Rotating platform

123‧‧‧ camera

124‧‧‧ cantilever

126‧‧‧ computer system

128‧‧‧Scan subsystem

130‧‧‧Internet Road

132‧‧‧Client device

134‧‧‧Network storage system

140‧‧‧block diagram

142, 144, 146, 148, 150, 152, 154, 182, 184, 186, 188, 190, 264, 266, 268, 269, 270, 272, 274 ‧ ‧

160‧‧‧2D image capture system

162‧‧‧Computer controlled rotating machinery

164‧‧‧Specific naming rules

180‧‧‧3D Geometric Data Acquisition System

200‧‧‧Parameter Matching System

202‧‧‧Photographs

220‧‧‧Parameter pairing software program

222, 228, 230, 232‧‧‧ functions

236‧‧‧ physical object spindle

234, 316, 318‧‧ reference points

238‧‧‧ Direction

240‧‧‧ arithmetic mechanism

242, 252‧‧‧

254‧‧‧

260‧‧‧File System

262‧‧‧ root directory

280‧‧‧Viewing program

282‧‧‧Operation window

286‧‧‧ operation keys

288‧‧‧ function keys

300‧‧‧ Stereo system

302‧‧‧ Left eye

304‧‧‧ right eye

306‧‧‧left stereo image

308‧‧‧right stereo image

310‧‧‧ objects

312‧‧‧left group

314‧‧‧ right group

Figure 1: Rendering virtual reality images (real objects, viewing windows, high-resolution images, and 3D mesh and viewer relationships) in 3D space in 3D modeling data.

Figure 2: A specific embodiment of a 3D virtual reality system (mechanical image and 3D data capture system, integrated computer, data and program server, and client viewing device).

Figure 3: Block diagram of the data capture, integration, and viewing system (processing of data, data to be stored, integration of applications, and viewers).

Figure 4: Photographic image capture system (mechanical system for capturing images and workflow for generating image files).

Figure 5: Capture 3D modeling data via a photographic camera or 3D scanner (mechanical system of a photographic camera or 3D scanner and workflow to generate 3D geometric data).

Figure 6: Embedding a 3D data system diagram (embed 3D geometry data one by one for 2D images to specify 6 degrees of freedom on the image and to create the required reference photo for performing automated processing).

Figure 7: Adjusting the photo parameters by adjusting the scale, adjusting the position and rotation (the user interface for adjusting the 6 variables or their corresponding data for each photo, there are three main adjustment steps to be implemented).

Figure 8: Automatic parameter generation method for all photos (the user interface for adjusting 6 variables or their corresponding data for each photo, there are three main adjustment steps to be implemented).

Figure 9: File system for image production, 3D data and data graph creation, and flowchart of the viewing program (generated data files and corresponding image production and geometry data files, and used to visualize downloaded images and data) View program flow chart).

Figure 10: Viewer presented and controlled in 3D (viewer function and control for end users, and data source structure for high resolution rendering and morphing techniques for smooth operation).

Figure 11: Follow the stereo system (using the same system to capture two sets of corresponding photographic images, the photos of which match the specifications for stereoscopic display and control).

1 depicts a method 100 of mapping a 2D photographic image 108 onto a 2D viewing window 106 in a projected manner.

The 2D photographic image 108 and the 3D mesh 110 are paired using a matrix transformation of six degrees of freedom for one physical object 102. Here, the physical object is exemplified by a mug, but any other entity such as a shoe, a light bulb, or the like may be used in other embodiments not shown. Objects to replace. A 3D grid 110 can be constructed from geometric parameters generated by a 3D scanner.

Viewer 104 views and controls the image interactively. 2D images can be scaled by the scale s, translated via the screen coordinates (x, y), and rotated by the ω angle plus the (θ, φ) angle. A whole set of photos at each row and column position will be These parameters are indicated.

2 illustrates a specific embodiment 120 of a computer system 126 that includes mechanical control, image processing, and data composition. A photo capture system 121 is composed of a controlled rotating platform 122 and a plurality of arms 124 having movement in the φ direction and tilt control by lens zooming. The camera 123 captures an image of the physical object 102 at different positions (θ, φ).

This includes a 3D scanning subsystem 128 (hardware or software enhanced) for capturing 3D geometric data, which can be constructed as a 3D mesh 110 (shown in Figure 1). If a photogrammetry utilizing the silhouette of the 2D photographic image 108 (shown in Figure 1) is used for 3D modeling, the scanning subsystem 128 can be used with the camera 123 Replace.

The computer system 126 integrates the 2D photographic image 108 and the 3D mesh 110 and transmits them to a remote server and a network storage system connected to the Internet path 130 via an Internet network 130. 134.

View and control 2D photographic images 108 and 3D mesh interactively using an Internet connected client device 132 with viewing and control software such as a computer, tablet, smart phone, etc. 110.

Figure 3 depicts a block diagram 140 to show how data is captured, processed, and stored, and then used by the viewer of the client.

In block 142, the 2D photographic image is captured one by one at each viewing position and is pre-processed to selectively remove the image background or be compressed to have a hierarchical pixel resolution The JPEG format of the historical pixel resolution, transparency information, and then further stored in a 2D photographic image file as shown in block 144.

In block 146, the 3D geometric data is scanned at different viewing positions, such as, but not limited to, a 3D modeling data scan. But after filtering processes to obtain reliable data, they are further integrated into a set of mesh points in a global coordinate system, such as the entity shown in block 148. A solid object file, or a 3D mesh.

In block 150, a composing system will process the 2D photographic image archive and the physical object archive for high image quality, photo-realistic virtual reality presentation, and physical amplification. The physical augmented reality application matches the 3D geometric parameters of the 3D mesh with the corresponding 2D photographic image parameters of the 2D photographic image in the 2D image file, and then the 2D photographic image file and the 3D mesh can be achieved. pair. The integration results are stored under a file structure such as an application and data folder as shown in block 152, at different resolution levels, physical object files. And storing the photographic image under the data map to store the corresponding parameters as (example) Such as but not limited to) xml file structure.

A viewing program as shown in block 154 will be run under a client device to decode the pairing parameters and interactively present high quality photographic images to the end user, and can be Further control and measurement of a 3D grid is provided for specific applications such as real world.

4 illustrates a 2D image capture system 160 that will capture a photographic image of a physical object located in a computer controlled rotating mechanics 162.

The physical object will be viewed from different perspectives by at least one camera moving horizontally and vertically around a physical object having a fixed rotation axis. In the present embodiment, the physical object is controlled to rotate the rotating machine 162 via a computer, for example, via 5 different photographic cameras having different viewing angles (eg, lower, lower right, right, upper right, and upper) relative to the physical object at 8 Different horizontal orientations (eg 0°, 45°, 90°, 135°, 180°, 225°, 270°, and 315°) capture images at the highest allowable resolution to form 40 different image archives. These image files are then stored one by one in a specific naming convention 164. However, in other embodiments not shown, it is also possible to take fewer or more images of a physical object.

It is worth noting that the image file can be pre-processed to remove unwanted background images, add transparency information or lower lower resolution and store in a single root directory (root Directory) for future integration and viewing processing.

FIG. 5 depicts a 3D geometric data capture system 180 for obtaining physical object geometry data. In fact, it can be a separate system or a subsystem of the image capture system as shown in FIG.

The 3D geometric data capture system 180 will use a certain wavelength of visible optics camera, a laser beam or an invisible infrared and reflection capturing system. To obtain the depth data of each object as shown in block 182, or simply to grab the silhouette of the 2D photographic image 108.

The 3D geometry data will be measured to remove unreliable noise data, such as a computing routine for filtering unreliable data, as shown in block 184, and then The final node position in the 3D global coordinate system is computed with statistically more accurate data, such as the program of statistically computed geometry data shown in block 186.

Next, as shown in block 188, the geometric data will be compared and merged with a set of global data sets, and the results will be stored in a standard solid object file as shown in block 190. in.

With all the necessary geometric data and parameters obtained from the physical objects from repeated measurements and data operations at many key locations, a set of final 3D meshes 192 can be constructed from a plurality of 3D geometric parameters.

Figure 6 depicts a parameter pairing system 200 that will be used to pair 2D images and 3D geometric data.

Since the photographic image will be stored in each photographic photo 202 of each view, the 3D geometric parameters of the 3D mesh 204 and the corresponding 2D photographic image parameters of the 2D photographic image/photograph 202 must be paired so as to be identical The presentation space views the photographic photo 202 and the 3D mesh 204.

It is well known that any physical object can be represented in six degrees of freedom. Therefore, a reference point (x, y, z) in the 3D space and an azimuth (θ, φ, ω) of the object can be used to represent the association between a photographic image and 3D geometric data ( Correlated relation).

Therefore, it is necessary to assign a set of six parameters to each photographic photo 202 and tie them together for future presentation and control functions. For example, in this embodiment, the photographic photo 202 can be named Framei, j.jpg, and includes M rows and N columns, and its reference point 206 can be labeled as (x _i,j ,y _i,j , z _{i, j} ). In this way, the six parameters of the 3D geometric data can be labeled as (x _0,0 , y _0,0 , z _0,0 , θ _0,0 , φ _0,0 , ω _0,0 ), and the photographic photos The six parameters of 202 can be labeled as (x _i,j , y _i,j , z _i,j , θ _i,j ,φ _i,j ,ω _i,j ), where i=1, 2... M, and J = 1, 2...N.

FIG. 7 illustrates a parameters matching software program 220 that can be used to pair these parameters with each photographic photo.

The pairing software program 220 has functions 222 of downloading the original 2D photographic image and the 3D geometry of the 3D mesh 226 and storing the integrated data.

The pairing software program 220 is designed to interact with the user via a photographic image 224 and a 3D grid 226 in any of the 2D image photos that appear in the photo selection function 230.

Since the mouse cursor on a computer screen can only be moved in two degrees of freedom, the user can perform parameter pairing manually. The mouse cursor can be used to control the solid object body axis 236, the value of θ and/or φ can be controlled via the tip of the moving solid object spindle 236, while the solid object spindle 236 is rotated about the direction 238. Then you can control the value of ω.

Next, you can pan the reference point 234 on the screen to control the value of x and / or y, and then use the mouse wheel to control the size of the 3D mesh, which is equivalent to adjusting the scale of the object to adjust The projected position of z. It is worth noting that in this embodiment, in order to manually pair the 2D photographic image 224 and the 3D mesh 226, the six parameters (x, y, z, θ, φ, ω) are adjusted. However, in other embodiments not shown herein, it may of course not be necessary to adjust all six parameters when not necessary.

In contrast, an auto computing matching process 228 for assisting pairing parameters is further provided, which can further programmatically pair parameters for a single photo or for multiple photos, and will Detailed in Figure 8.

It is worth noting that the manual pairing processing function 232 can be further replaced by a direct operation using the automatic operation pairing processing function 228 when performing all of the capture processing. Automatic matching of a 2D photographic image file with a 3D mesh is a programmatic automatic pairing when a 3D geometry scan mechanism provides a parameter relationship between a 2D photographic image and a 3D mesh. The parameters of the 2D photographic image and the 3D geometry of the 3D mesh.

FIG. 8 depicts a computing scheme 240 developed to generate parameters that are automatically paired with all of the perspectives of each photographic photo.

By applying the Quaternion technology, it is possible to calculate any 3D vector v representing the reference point and the main axis so as to rotate a rotation angle around a rotating unit axis n (rotating) After the angle θ, the new vector r in the 3D space is obtained.

In this way, it is possible to calculate the rotation unit axis n by using any one of the two photographs of the same column and the known rotation angle by using the parameters. Once the rotational unit axis n is calculated, any other reference point and major axis in each photo of the same column 242 can be calculated and thus automatically paired with its parameters.

The same operation can also be performed on a single line 254 but an image of a different column 252 in the vertical direction. Repeat the same process to complete the calculation for all photos.

In theory, it is only necessary to manually pair three photos to be able to calculate the rotation unit axis in the horizontal and vertical directions, and the human resources for finding the pairing parameters are extremely saved. However, in practice, the rotating trajectory of the camera may not be on a perfect circular path, and the tilting angle and zoom lens may be in a non-linear manner. Projecting a photographic image (non-linear way) requires 5 or 7 more manually paired photos to achieve a more reliable data. There is also provided a visual adjustment of the viewing pairing operation to perform fine adjustments.

9 illustrates a file system 260 constructed under an Internet server to provide an end user with a viewing mechanism for viewing high resolution photographic images with 3D geometric data attached to their client devices. (viewing mechanism).

Viewing programs, instant and high-resolution image data, geometry data, accessory data, and rendered data maps are all stored in a root directory 262 to ensure no cross-domain access. (cross domain access) problem.

As shown in block 264, the viewing program accessed via the end user will download all necessary program routines, referred to herein as "Viewers," and then automatically obtain the instant map as indicated by block 266. Geometric data like images and 3D meshes. Next, as shown in block 268, an interactive operation for viewing the high resolution image and the 3D mesh can be performed to obtain a high resolution image as shown in block 269. In addition, according to amplification For real-world applications, functional operations as shown in block 270 may be further utilized, such as for necessary 3D measurements (as shown in block 272) or 3D control functions (as shown in block 274), and the like.

The above program can be implemented in a client device with a 3D operating environment such as OpenGL or WebGL or other 3D environments.

FIG. 10 depicts a client viewing program 280 developed to implement the functionality described in FIG.

The viewing program 280 can be a WebGL-enabled browser-based HTML5 viewing program for a Windows platform computer system 126 (shown in Figure 2), such as a table. A super-type personal computer, a mobile device or any device capable of displaying an operation window 282, or a native program of an OpenGL ES supporting a mobile device.

The program has an operational button 286 to interactively achieve zoom, pan and rotate functions of the photographic image in view, with a slider controller for viewing high quality photographic images or viewing 3D The wire frame of the model, even in a different way of transparency, is used to view both.

In order to visualize the smoothness of the 2D photographic image in the 3D space, the angular distortion of the 2D photographic image 284 can also be achieved by changing the angle of 0 < Δθ < θ _increment and/or the angle of 0 < Δφ < φ. (angular morphing).

In addition, function keys 288 can also be provided depending on the application to achieve measurement and application control, as well as any other desired functionality.

Figure 11 illustrates that this system is more capable of being applied to a stereo system 300 to view objects in a more realistic sense in response to human being's eyes perception.

The viewing window is divided into two to provide a left stereo image 306 and a right stereo image 308 for the left eye 302 and the right eye 304, respectively.

Two sets of images and pairing parameters are taken taking into account different perspectives of the same object 310. They will be independently set to the left group 312 and the right group 314. In this embodiment, the left side group 312 and the right side group 314 are named, for example, FrameLi, j.jpg and FrameRi, j.jpg, respectively, and their reference points 316 and 318 can be respectively labeled as (x _i,j ,y _{i , j} , z _i,j )L and (x _i,j ,y _i,j ,z _i,j )R. In this way, the six parameters corresponding to the 3D geometric data of the left group 312 and the right group 314 can be respectively labeled as (x _0,0 , y _0,0 , z _0,0 , θ _0,0 , φ _{0, 0} , ω _0,0 )L and (x _0,0 , y _0,0 , z _0,0 , θ _0,0 , φ _0,0 , ω _0,0 )R, while the left group 312 and the right group 314 The six parameters can be labeled as (x _i,j ,y _i,j ,z _i,j ,θ _i,j ,φ _i,j ,ω _i,j )L and (x _i,j ,y _{i, j} , z _i,j , θ _i,j ,φ _i,j ,ω _i,j )R, where i=1, 2...M, and J=1, 2...N.

However, it is also possible to use images of the same column but different rows of a single set of 2D images. This will not be very accurate in the simulation of viewing angle and distance, but will provide sufficient depth feeling to the average viewer.

The viewing window can also be applied to a television, a movie screen or even a new wearable gadget with view glasses.

Although the present invention has been disclosed in the above embodiments, it is not intended to limit the present invention, and those skilled in the art can make some modifications and refinements without departing from the spirit and scope of the present invention. The scope of the invention is defined by the scope of the appended claims.

102‧‧‧ physical objects

120‧‧‧Specific examples

121‧‧‧Image Capture System

122‧‧‧Rotating platform

123‧‧‧ camera

124‧‧‧ cantilever

126‧‧‧ computer system

128‧‧‧Scan subsystem

130‧‧‧Internet Road

132‧‧‧Client device

134‧‧‧Network storage system

Claims (14)

A method of pairing a 2D photographic image file and a 3D mesh using a matrix transformation via six degrees of freedom for a solid object, wherein the 3D geometry of the 3D mesh is for high image quality, imaginary virtual Real-world rendering and physical augmented reality applications in conjunction with 2D photographic image parameters of the 2D photographic image archive, using the matrix transformation for the six degrees of freedom of the physical object is performed manually, and the manual mode Included in at least one of the following steps: manually moving a tip of a spindle of the solid object to control a value of θ; manually moving the tip to control a value of φ; manually rotating the spindle to control an ω value; A size adjustment ratio of the 3D mesh is paired with the 2D photographic image parameters selected from the 2D photographic image file until the entire set of 2D photographic image parameters of the 2D photographic image file are paired Up to now, wherein not less than 3 2D photographic images in the 2D photographic image file are manually selected from the horizontal and vertical directions of the spindle for manual operation with the aid of calculation A method of pairing a 2D photographic image file and a 3D mesh as described in claim 1, wherein the 2D photographic image in the 2D photographic image file is selectively processed in at least one of the following steps : Removes its image background; compresses it into a JPEG format with hierarchical pixel resolution, transparency information; and stores it in a 2D photographic image archive. A method of pairing a 2D photographic image file and a 3D mesh as described in claim 1, wherein the 3D geometric parameter is generated from a visible optical camera of a certain wavelength, a laser beam or an invisible infrared light and reflected light. The capture system is obtained via the silhouette of the 2D photographic image in the 2D photographic image file or the depth data of the object geometry of the physical object. A method for pairing a 2D photographic image file and a 3D mesh according to claim 1, wherein the high image quality and the imaginary virtual reality presentation are constructed under an internet server. A file system for providing an end user with a viewing mechanism for viewing high resolution photographic images and 3D meshes, and the physical augmented reality application is a 3D measurement or 3D control function. A method of pairing a 2D photographic image file and a 3D mesh as described in claim 4, wherein the high resolution photographic image and the 3D mesh are viewed together in a different transparency. A method for pairing a 2D photographic image file and a 3D mesh as described in claim 4, wherein the high image quality, imaginary virtual reality presentation is further utilized to have separate for the left eye and the right eye A stereoscopic system of the left side stereoscopic image and the right side stereoscopic image viewing window. A method for pairing a 2D photographic image file and a 3D mesh as described in claim 4, wherein the high image quality, imaginary virtual reality is presented in the θ direction and the The angular distortion of at least one of the φ directions is further utilized to present a smoothness of the 2D photographic image in the 2D photographic image file in a 3D space. A method of pairing a 2D photographic image file and a 3D mesh using a matrix transformation via six degrees of freedom for a solid object, wherein the 3D geometry of the 3D mesh is for high image quality, imaginary virtual Real-world rendering and physical augmented reality applications in conjunction with 2D photographic image parameters of the 2D photographic image archive, using the matrix transformation for the six degrees of freedom of the physical object is performed in an automated manner, and the automated manner In the case where a 3D geometric scanning mechanism provides a parameter relationship between the 2D photographic image in the 2D photographic image file and the 3D mesh, the 2D photographic image parameters and the 3D geometry are automatically paired programmatically. parameter. A method of pairing a 2D photographic image file and a 3D mesh as described in claim 8 wherein the 2D photographic image in the 2D photographic image file is selectively processed in at least one of the following steps : Removes its image background; compresses it into a JPEG format with hierarchical pixel resolution, transparency information; and stores it in a 2D photographic image archive. A method of pairing a 2D photographic image file and a 3D mesh as described in claim 8 wherein the 3D geometric parameter is generated from a visible optical camera of a certain wavelength, a laser beam or an invisible infrared light and reflected light. The capture system is obtained via the silhouette of the 2D photographic image in the 2D photographic image file or the depth data of the object geometry of the physical object. Pairing a 2D photographic image file and a 3D mesh as described in claim 8 The method, wherein the high image quality, immersive virtual reality presentation is a file system constructed under an internet server for providing an end user with viewing high resolution photographic images and 3D meshes. A viewing mechanism, and the physical augmented reality application is a 3D measurement or 3D control function. A method of pairing a 2D photographic image file and a 3D mesh as described in claim 11 wherein the high resolution photographic image and the 3D mesh are viewed together in a different transparency. A method for pairing a 2D photographic image file and a 3D mesh as described in claim 11 wherein the high image quality, imaginary virtual reality presentation is further utilized to have separate for the left eye and the right eye A stereoscopic system of the left side stereoscopic image and the right side stereoscopic image viewing window. A method for pairing a 2D photographic image file and a 3D mesh according to claim 11, wherein the high image quality, imaginary virtual reality is presented in the θ direction and the The angular distortion of at least one of the φ directions is further utilized to present a smoothness of the 2D photographic image in the 2D photographic image file in a 3D space.