KR101588935B1 - A method using 3d geometry data for virtual reality image presentation and control in 3d space - Google Patents

Abstract

This method, with a set of key geometry parameters for each frame of photo images, is associated with 3D modeling data of the object, using a collection of photographic images of the object captured at different viewing angles, In real-time, high-quality, photo-realistic 3D images. It also provides 3D geometry data for physical applications using an automatic or manual photo-imaging system and imported 3D modeling data of the same object with the same hardware system or from an independent 3D geometry scanning system, The automatic or automatic software tool compiles this information into a complete package of files, which is represented by a viewing program with a 3D environment. It can also be extended to a stereoscopic system and can provide real-time physical manipulation capabilities, high quality, and realistic visual effects.

Description

TECHNICAL FIELD [0001] The present invention relates to a virtual reality image presentation method and a 3D geometry data presentation method,

The present invention relates generally to the field of 3D photographic presentation. Virtual reality technology is used to represent high quality photo images. It provides geometric data for physical measurement or control, takes advantage of 3D modeling techniques, and can be used for augmented reality applications. It can also be extended to a stereoscopic display system for real-time applications.

The virtual reality uses a set of photo images to represent a stereoscopic object seen at different view angles. This provides high quality photo images for presentation applications. However, if the number of photo frames is limited, the viewing angle is limited to the number of several image pickup positions, resulting in a non-smooth animation. The photo image also does not include geometry (dimension) data. Photo images can not be precisely aligned within a presentation and therefore can not be used in physics-related applications for measurement or control.

3D modeling is another approach that represents a three-dimensional object. It has geometry information and can be used in a physical application that includes an augmented reality. However, it is very expensive to obtain geometry data and store vast amounts of texture images in order to obtain accurate geometry data and to represent texture mapping techniques for good quality presentations. In addition, it is difficult to render photo-realistic rendering in real time with low-performance personal computing devices.

There is a need to create photo-realistic virtual reality presentations of high quality images for commercial applications, particularly for mobile devices such as desktop personal computers or tablet PCs and smart phones, and to provide augmented reality applications of geometric data for physical measurement or control There is a need to include geometry information for provision to Combining the advantages of two different approaches, virtual reality and 3D modeling, is one way to provide an economical solution and meet the quality requirements for available computing devices, in order to provide both high quality viewing performance and physical information . The present invention seeks to achieve these goals and may be implemented in existing computing devices and mechanical systems.

In the field of 3D photographic presentation, the present invention provides the geometric data for physical measurement or control, which takes advantage of the 3D modeling technique and can be used for augmented reality applications And provides a technology that can be extended to a stereoscopic display system for real-time applications.

In accordance with one aspect of the invention, a method of combining a set of port frames with a set of geometry information is described, and a systematic manner of representing 2D photos in a 3D space in a viewing window of a computing device is described. The mathematical relationship between the image frame related parameters and the viewing transformation of the stereoscopic object in the 3D presentation space is described.

According to another aspect of the present invention, a system comprising a computer-controlled mechanical system for automatically capturing photo images with different view angles is described. A 3D geometry data scanning subsystem based on various optical scanning hardware, or a photo imaging camera for extracting 3D geometry data by a silhouette or referencing mat or stripe is described.

In accordance with another aspect of the present invention, a software system comprising a workstation, a storage system, a remote server, and a client viewing device for implementing the present invention is described. A software program is described that compares 2D photo frames with scanned 3D geometry data to manually or automatically generate a set of control ring parameters. A software program for loading image and geometry data, and for viewing, measuring, and controlling photo-realistic stereoscopic objects is described.

In accordance with another aspect of the present invention, an extension of hardware and software systems to stereoscopic display and control functions is described.

In the field of 3D photographic presentation, the present invention provides the geometric data for physical measurement or control, which takes advantage of the 3D modeling technique and can be used for augmented reality applications And provides a technology that can be extended to a stereoscopic display system for real-time applications.

BRIEF DESCRIPTION OF THE DRAWINGS While the invention may be better understood by reference to the following description with reference to the accompanying drawings, it is to be understood that the detailed description covers a more abstract system which is not limited by the visual drawings.
Figure 1: An illustration of a virtual reality image presentation in 3D space with 3D modeling data,
It shows the relationship between the real object, the view window, the high-resolution image and the 3D mesh, and the viewer (observer).
2 is a diagram illustrating an example of construction of a 3D virtual reality system,
Mechanical image and 3D data capturing systems, composing computers, data and program servers, and client viewing devices.
Figure 3: Block diagram of data capturing, composing and viewing system,
A process in which data is captured, data is stored, a composition program, and a viewing program.
Figure 4: Photo image capturing system,
A mechanical system for photo capturing, and a workflow of the generated image file.
5 is an explanatory diagram of 3D modeling data capturing by a photo camera or a 3D scanner;
The mechanical system of the photocamera or 3D scanner and the generated 3D geometry data.
Figure 6: Embedding 3D data system diagram,
3D geometry data for assigning 6 degrees of freedom variable to images and frame-by-frame embedding of 2D photos.
It shows the reference frames needed to perform an automatic process.
Fig. 7 is an explanatory diagram of adjustment of frame parameters by scaling, translating and rotating;
A user interface (three main coordination procedures implemented) for adjusting six variables or corresponding data for each frame.
Figure 8: An illustration of automatic parameters generating all frames,
A user interface (three main coordination procedures implemented) for adjusting six variables or corresponding data for each frame.
Figure 9: Flowchart of the configuration and viewing of the file system for imaging, 3D data and profiling,
The generated data file and the corresponding imaging and geometry data files.
Lt; RTI ID = 0.0 > flowchart < / RTI > illustrating image and data loading.
Figure 10: An example of a viewing program with 3D presentation and control,
Represents end user viewing program functions and controls.
A data resource structure for high-resolution presentations, and a morphing technique for smooth operation.
Figure 11: Schematic representation of an extension to a stereoscopic system,
The same system is used to obtain two sets of photo images with frames conforming to the specification for stereoscopic display and control.

Reference will now be made in detail to specific embodiments of the invention. These embodiments are illustrated in the accompanying drawings. While the present invention has been described with reference to these specific embodiments, it should be understood that they are not intended to limit the invention to these embodiments. In fact, it is intended to cover alternatives, modifications, and equivalents falling within the spirit and scope of the present invention as defined by the appended claims. The following detailed description, numerous specific details, are set forth in order to provide a more thorough understanding of the present invention. The present invention may be practiced without some or all of these specific details. On the other hand, well-known process operations have not been described in detail in order to not obscure the present invention. In all the following embodiments, the same or similar components are denoted by the same reference numerals even if they are described in different embodiments.

Referring to Figure 1, a method 100 for a 2D photo image 108 having a projected mapping on a 2D view window 106 is illustrated.

The 2D photo image 108 is matched with the 3D mesh 110 by using a matrix transformation having six degrees of freedom with respect to the three-dimensional object 102 as a three-dimensional object. Here, the three-dimensional object is exemplified as a mug for explanatory purposes, but may be another three-dimensional object such as a shoe, a bulb, etc. instead of the other unillustrated embodiment. The geometry (dimension) parameters generated from the 3D scanner can build the 3D mesh 110.

A viewer (observer) 104 interactively views and controls the images. The 2D ports are zoomed in on a scale s indicated by a set of frames within each column row and row position and are paned by screen coordinates x, y , ω angle, plus (θ, φ) angles.

Referring to Figure 2, an implementation 120 comprises a computer system 126 for mechanical control, image processing, and data composition (synthesis). The photo capturing system 121 comprises a multiarm arm 124 having a controlled rotating platform 122 and cameras 123 and is configured to capture images at different (?,?) Positions of the three- With the zooming and tilting of the lens.

(Hardware or software enhanced) 3D scanner subsystem 128 is included for capturing 3D geometry data to be built into the 3D mesh 110 (shown in FIG. 1). If photogrammetry is used for the silhouette of the 2D photo image 108 (shown in FIG. 1) for 3D modeling, the scanner subsystem 128 may be replaced with cameras 123.

The computer system 126 compiles the 2D photo image 108 and the 3D mesh 110 and transmits them to the remote server and the network storage system 134 linked to the Internet network 130 via the Internet network 130).

Client device 132 includes viewing and control software and interactively views and controls 2D photo image 108 and 3D mesh 110, such as a PC, tablet PC, smart phone, etc., .

Referring to FIG. 3, a block diagram 140 shows how data is captured, processed, stored, and then consumed by the viewer on the client side.

At block 142, 2D photo images are captured per frame at each view location, which are preprocessed to remove the image background if necessary, or compressed into a JPEG format with hierarchical pixel resolution, transparency information, Is stored in the 2D photo image file, as shown in Fig.

At block 146. 3D geometry data is scanned at different view positions, for example, but not exclusively, by 3D modeling data scans. However, in order to obtain reliable data, after a filtering process, they are further compiled into a set of meshes having a global coordinate system, such as a solid object file as shown at block 148, known as a 3D mesh.

At block 150, the compositing system processes the 2D photo image file and the stereoscopic object file to provide photo-realistic virtual reality presentation of high quality images and geometric data for physical measurement or control to the augmented reality application, To conform to the 2D photo image parameters of the 2D photo images in the corresponding 2D image file, and then cause the 2D photo image file to be matched with the 3D mesh. The resulting results are stored in a file structure, such as an application and a data folder, as shown in block 152, which stores a profile that stores photo images, stereoscopic object files, and corresponding parameters at different resolution levels, But not limited to) an xml file structure.

As shown in block 154, the viewing program is executed on the client device to decode the matching parameters, interactively with the end user to represent the photo image with high quality, and to control the 3D mesh for a particular application, such as augmented reality, Additional measurements may be provided.

Referring to FIG. 4, the 2D photo capturing system 160 images the photo images of the three-dimensional object located in the computer-controlled rotation mechanism 162.

The three-dimensional object has a fixed rotation axis, and at least one camera moves horizontally and vertically around the three-dimensional object, and is viewed from different view-ins. In this embodiment, the three-dimensional object is rotated by the rotation of the computer-controlled rotating mechanism 162, for example, at 0 属, 45 属, 90 属, 135 属, 180 属, 225 属, 270 属, and 315 属, The pictures are picked up at the highest resolution possible by the five different photo cameras with different view angles, e.g., bottom side, bottom right side, right side, top right and top side, and eight different horizontal orientations for the three- 40 different image files, and the image files are stored frame by frame with a specific naming convention (164). However, in another embodiment, which is not illustrated, it is also possible to capture fewer or more photos for a three-dimensional object.

It should be noted that the image files may be preprocessed to remove unwanted background images, added transparency information, switched to a hierarchical lower resolution, and stored under a single root directory, for future composing or viewing processes.

Referring to FIG. 5, a 3D geometry data capturing system 180 is used to obtain the geometry (dimension) data of the stereoscopic object. This may be an independent system, or a subsystem, for the photographic capturing system physically as described in FIG.

The 3D geometry data capturing system 180 may be implemented by obtaining depth data 182 of each object geometry or simply by imaging the silhouette of the 2D photo image 108 to obtain an optical camera, Invisible infrared and reflective capturing systems.

3D geometry data is processed by measurement after unreliable noise data is first removed, such as computing routine 184, which filters unreliable data, and then statistically computes geometry data 186, And calculates the statistically more accurate data as the end node position in the 3D global coordinate system, as shown in FIG.

The geometry data 186 are compared and merged into a global data set 188, stored and stored in the standard three-dimensional object file 190.

The final 3D mesh 192 can be constructed from a plurality of 3D geometry parameters by repeated measurement and data operations from a number of key locations to obtain all the necessary geometry data and parameters for the 3D object.

Referring to FIG. 6, the parameter matching system 200 is created to match 2D photos with 3D geometry data.

Since the photo images are stored in respective photo frames 202 with respective view angles, the 3D geometry parameters of the 3D mesh 204 are matched with the 2D photo image parameters of the corresponding 2D photo images / frames 202, Should be visible in the same presentation space.

It is known that any three-dimensional object can be represented in six degrees of freedom. To represent the correlation between the photo image and the 3D geometry data, the orientation angle (?,?,?) Of the reference point and the object in the 3D space (x, y, z) can be selected.

Thus, for each photo frame 202, it is necessary to allocate a set of six parameters and bundle them together for future presentation and control functions. In this embodiment, for example, but not exclusively, the photo frame 202 is represented by a frame i, j.jpg and consists of M columns (columns) and N rows (rows) The reference point 206 may be represented by (xi, j, yi, j, zi, j). As a result, the six parameters of the 3D geometry data can be represented by (x0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0), and the six parameters of the photo frames 202 (where i = 1, 2, ..., M, and J = 1, 2, ..., N), where x i, j, y i, j, z i, j, θ i, j, φ i, j, .

Referring to FIG. 7, a parameter matching software program 220 may be used to match these parameters with each of the photo frames.

The matching software program 220 has functions 222 for loading original 2D photo images and 3D geometry parameters of the 3D mesh 226 and for storing the compassed data.

The matching software program 220 is designed to interact with the user by displaying both the photo image 224 and either side of the 3D mesh 226 in any of the 2D image frames as shown in the frame selection 230. [

The mouse cursor on the computer screen can only be moved in two degrees of freedom, so the user can do parameter matching manually. This allows the stereoscopic object body axis 236 to be controlled by rotating the stereoscopic object body axis 236 to control the value of? By moving the tip of the axis to control the values of? And / or? have.

The reference point 234 can then be panned on the screen to control the x, y values and the mouse wheel can be used to control the size of the 3D mesh, z position. It should be noted that, in this embodiment, to manually match the 2D image frame 224 to the 3D mesh 226, all six parameters (x, y, z,?,?,?) Are adjusted. However, in other embodiments not illustrated, it is of course possible not to adjust all six parameters, if not necessary.

On the other hand, there is further provided an autocomputing matching process 228 to assist in matching the parameters, which may further programmatically match the parameters for a single frame or for a plurality of frames, 8.

It should be noted that the manual matching process 232 may be further substituted by direct computation by using the autocomputing matching process 228 while performing the capturing process together. Automatically matching a 2D photo image file with a 3D mesh means that while the 3D geometry scan mechanism can provide a relationship of parameters between the 2D photo images and the 3D mesh, the parameters of the 2D photo images are converted to 3D geometry parameters To automatically match them programmatically.

Referring to FIG. 8, an operation scheme 240 has been developed to automatically generate a parameter match for all viewing angles in each photo frame.

By applying the quaternion (quaternion) technique, any 3D vector v representing the reference point and body axis can be computed to obtain a new vector r in 3D space after rotating at the rotation angle? About the rotation unit axis n have.

Thus, by using the parameters to compute the rotation unit axis n, any two frames of the same row with known rotation angles can be used. Once known, any other reference point and body axis of each frame of the same row 242 may be computed, thus automatically matching the parameters.

For image frames in a single column 254 of different rows 252, the same operation can be made in the vertical direction. By repeating the same process, all frames can be calculated.

In theory, only three passively matched frames are needed to compute the horizontal and vertical rotation unit axes, thus avoiding the tremendous effort to find the matching parameters. However, in actual practice, the camera trajectory may not be located in a perfect circular path, so that tilting angles and zoom lenses can project photo images in a non-linear manner, so to obtain more reliable data, More or fewer passively matched frames may be needed. A visual adjustment for reviewing the matching operation is also provided for fine adjustment.

Referring to FIG. 9, a file system is built in the Internet server 260 in order to provide the end user with a viewing mechanism for viewing high resolution photo images plus 3D geometry data at the client device.

Viewer programs, real-time high-resolution image data, geometry data, ancillary data, and presentation profiles are all stored under root directory 262 that is free of cross-domain access problems.

The viewer program accessed by the end user loads all necessary program routines and the viewer named herein, as shown in block 264, and then automatically loads the real-time image of the 3D mesh, as shown in block 266 And geometry data. Next, in order to obtain high resolution images, as shown in block 269, an interactive operation is possible to show high resolution images and 3D meshes, as shown in block 268. [ In addition, depending on the augmented reality application for the required 3D measurement, as shown in block 272, or the 3D control function as shown in block 274, the functional operation as shown in block 270 is further possible.

The program may be executed on a client device having a 3D operating environment, such as OpenGL or WebGL, or any other 3D environment.

Referring to FIG. 10, in order to execute the functions described in FIG. 9, a client-side viewing program 280 has been developed.

The viewing program 280 may include a WebGL-enabled browser-based HTML5 for Windows platform for a computer system 126 (shown in Figure 2), such as a desktop computer, a mobile device, or any device capable of representing an operation window 282 A viewing program, or a native program on an OpenGL ES enabled mobile device.

The program has operation buttons 286 for performing zoom, pan and rotation functions when viewing a photo image interactively, to view the photo image with high transparency, high quality, or 3D mesh, or both It has a slider controller for viewing.

To further illustrate the smoothness of the 2D photo images within the 3D space, angular morphing of the 2D photo image (s) 284 is further performed by varying the angle 0 < [theta] < theta increment and / or the angle 0 & can do.

Depending on the application, a function button 288 for performing measurement, application control, and other necessary functions is further provided.

Referring to FIG. 11, the system can be further extended to a stereoscopic system 300 to show objects in a more realistic sense due to the perception of the human eye.

The viewing window is two independent ones for both the left stereo gram 306 and the right stereo gram 308, which provide images to the left eye 302 and the right eye 304, respectively.

The two sets of images and matching parameters are obtained in consideration of the view angle difference for the same object 310. The set of left one 312 and right one 314 are independently present. In this embodiment, the left one 312 and the right one 314 may be named as frame Li, j.jpg and frame Ri, j.jpg, respectively (but not exclusively) Reference points 316 and 318 may be represented by (xi, j, yi, j, zi, j) R and (xi, j, yi, j, zi, j) L, respectively. As a result, the six parameters of the 3D geometry data for the left one 312 and the right one 314 are (x0,0, y0,0, z0,0, 0,0,0,0,0,0,0,0,0,0,0) And the six parameters of the left one 312 and the right one 314 may be represented by (x0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0) L, j, yi, j, zi, j,? i, j,? i, j, j) L (where i = 1, 2 ... M, and J = 1, 2 ... N).

However, it is also possible to use a single set of 2D photos with different column indices for the same row of images. This is not very accurate in viewing angle and distance simulations, but it will provide a sufficient depth sense for average viewers.

The view window can also be applied to a new wearable device having a TV, movie screen, or even view glasses.

Although specific embodiments of the invention have been described, it will be appreciated by those of ordinary skill in the art that there will be alternative embodiments that are equivalent to the described embodiments. Accordingly, it is to be understood that the invention is not to be limited by the specific embodiments shown, but is only intended to fall within the scope of the appended claims.

The present invention is generally applicable to the 3D photo presentation field.

The present invention relates to a 2D photo capturing system and a 2D photo capturing system using the same. The present invention relates to a 2D photo capturing system, and more particularly, A 3D geometry data capturing system 200 a parameter matching system 220 a parameter matching software program 240 an operation scheme 260 an internet server 280 a client side viewing program 300 a stereoscopic system

Claims (10)

A method of matching a 2D photo image file to a 3D mesh by using a matrix transformation having six degrees of freedom for a three-dimensional object,
In order to provide a photo-realistic virtual reality presentation of the best possible image quality captured by the camera and to provide geometric data for physical measurements or to provide control for an augmented reality application, It is configured to correspond to the 2D photo image parameters of the file.
Wherein the 2D photo image file is matched to a 3D mesh. The method according to claim 1,
The 2D photo images in the 2D photo image file,
(1) image processing to remove that image background;
(2) image processing in a JPEG format having hierarchical pixel resolution and transparency information; And
(3) the processing to be performed in the 2D photo image file,
Lt; RTI ID = 0.0 &gt;
Wherein the 2D photo image file is matched to a 3D mesh. The method according to claim 1,
The 3D geometry parameters are generated from a visible optical camera, laser beam or invisible infrared and reflective capturing system of a specific wavelength by obtaining the depth data of each object geometry of the silhouette or 2D object of the 2D photo images in the 2D photo image file Can be
Wherein the 2D photo image file is matched to a 3D mesh. The method according to claim 1,
Parameter matching having six degrees of freedom for the three-dimensional object is performed manually or automatically
Wherein the 2D photo image file is matched to a 3D mesh. The method of claim 4,
What is done manually
(1) manually moving the tip of the axis of the three-dimensional object to control the value of the horizontal angle?;
(2) manually moving the tip of the axis of the three-dimensional object to control the value of the vertical angle?;
(3) manually rotating the body axis of the three-dimensional object to control the value of the rotational angle [omega]; And
(4) using the mouse wheel to control the size of the 3D mesh, which is equivalent to the object's scale and projected z-position
At least one of &lt; RTI ID = 0.0 &gt;
here,
At least three manually matched frames are used to calculate the unit axes rotating in the horizontal and vertical directions
Wherein the 2D photo image file is matched to a 3D mesh. The method of claim 4,
What is done automatically is to automatically match the 2D photo image parameters with the 3D geometry parameters programmatically while the 2D geometry scan mechanism provides the parameter relationship between the 2D photo images in the 2D photo image file and the 3D mesh To do.
Wherein the 2D photo image file is matched to a 3D mesh. The method according to claim 1,
A photo-realistic virtual reality presentation device of the highest possible image quality captured by the camera,
An internet server that is provided to provide end users with photo images of the best possible image quality captured by the camera and a viewing mechanism for viewing the 3D mesh and to provide physical augmented reality applications for 3D measurement or 3D control functions You are connected to a file system
Wherein the 2D photo image file is matched to a 3D mesh. The method of claim 7,
The 3D meshes and the photo images of the best possible image quality captured by the camera are displayed together with different transparency
Wherein the 2D photo image file is matched to a 3D mesh. The method of claim 7,
The photo-realistic virtual reality presentation of the best possible image quality captured by the camera is applied to the left and right stereograms of the stereoscopic system, respectively.
Wherein the 2D photo image file is matched to a 3D mesh. The method of claim 7,
The photo-realistic virtual reality presentation of the highest possible image quality captured by the camera is achieved through angular morphing of 2D photo images in at least one of the horizontal angle? Direction and the vertical angle? Direction, Expands to show the smoothness of the 2D photo image within the file
Wherein the 2D photo image file is matched to a 3D mesh.

Images