KR101340598B1 - Method for generating a movie-based, multi-viewpoint virtual reality and panoramic viewer using 3d surface tile array texture mapping - Google Patents

Abstract

The present invention relates to a method for generating a video-based multi-viewpoint viewer and a panoramic virtual reality viewer using 3D tile array texture mapping. According to the present invention, the method for generating a video-based multi-viewpoint viewer and a panoramic virtual reality viewer using 3D tile array texture mapping realizes an image-based appreciation viewer and an application.

Description

Method for generating a movie-based, multi-viewpoint virtual reality and panoramic viewer using 3D surface tile array texture mapping}

The present invention relates to a method for producing a video-based multi-view and panoramic virtual reality viewer, and more particularly, to texture mapping an image or an image in a tile method on a digital three-dimensional plane in order to produce an interactive viewing virtual reality viewer. The present invention relates to a method of manufacturing a video-based multi-view and panoramic virtual reality viewer using a three-dimensional flat tile array texture mapping to move and rotate a plane according to a user's manipulation to view an image or a video as a multi-view and panoramic view.

Just like viewing 3D objects in reality, there are two types of methods for implementing an online Internet service or an independently operating application that provide interactive viewing to a point of view in virtual reality: object, image, and video.

As shown in FIG. 1, the object method embodies an object made of a polygon mesh in which an object is shaped as it is in 3D digital space, and implements a camera that can adjust the angle as the viewer's eye, and thus the viewpoint of the camera. When you rotate the object or move the camera, you can see the 3D object as if it were a real space.

As the digital space, a virtual reality digital space such as OpneGL (Open Graphics Library) and DirectX (Microsoft DirectX) is used, and the digital space has three-dimensional axes of x, y, and z, so that the spatial configuration is possible. .

However, the object method has a limitation in appreciating the high quality as the reality, and online connection quality and the capacity of the application to bring the three-dimensional object having the number and capacity of heavy points to the virtual space and calculate it by the lighting operation like the reality The hardware specifications of viewers and viewers are different, which makes it difficult to enjoy the same experience in a low specification environment.

Overcoming these shortcomings, an image-type virtual reality viewer has been proposed for viewing in high quality. As shown in FIG. 2, a plurality of images can be connected and shown according to a user's operation, and as shown in FIG. It works in each web browser by Java Script method.

The image-based operation principle is the same as the frame-based technique of making a motion picture in a cartoon movie by the stop motion technique, and the photo-based high-quality image that accurately and realistically simulates the photographs even with low specification access of the initial Internet network. It contributed to 360 degree view of the object. It is mainly implemented as a viewing angle corresponding to one plane in 360 degree direction, but now it provides image-based multi-view multi-angle object by providing an array that enables 360-degree omnidirectional viewing of multiple planes beyond the multi-view of one plane. Appreciation of is possible.

To do this, the object is photographed at an angle on the rotating disk, and the N number of images are changed according to the direction or button of the user's mouse to make the object look as if it is viewed from multiple angles. This method is based on image and driven with high quality at relatively low specification. However, the difference with the real world is that the images that are members are still images without dynamic movement different from reality.

Since then, the proposed method is an image-based appreciation method that provides a multi-view by the image taken by multi-view by replacing the image with the development of the image technology. Using this method, the dynamic image can be rotated according to user manipulation. However, the limitation of this technology is to find and connect the frames in the correct image. Due to the technical limitation, a delay of time is not made immediately. It is a method of occlusion of image frames by algorithm, but since the capacity of each scene is different and the specifications of frame calculation are different, it is impossible to find and connect the occlusion without any real delay.

On the other hand, starting from the MPEG2 (Moving Picture Expert Group 2) broadcast generation, several moving images accurately interlock frames to minimize the delay, intersect transparency, or use the morphing technique to make the occlusion correct. Much research has been done on the technology.

In addition, the field has been continuously studied to the current generation of digital broadcasting using MPEG4 (Moving Picture Expert Group 4), but it is difficult to smoothly and immediately cross into one video stream with minimal delay. This acts as a limitation in retrieving images with sophisticated sound sources, and the download capacity and number of images are increased by multi-view N images more than the image viewer, so it is not possible to provide downloads smoothly in low-level Internet access. There is this.

The present invention has been made to improve the above-mentioned conventional problems, and at least one plane image in a virtual space using images or computer graphics simultaneously taken in multiple angles implemented in a general computer environment including a mobile environment. 3D planar tile array texture mapping that creates a single image by processing texture mapping in a tile method, and then allows users to view a plurality of angular images implemented on tiles of the next point without delay by moving a plane according to a user's manipulation. The purpose of the present invention is to provide a video based multi-view and panoramic virtual reality viewer production method.

In order to achieve the object of the present invention, a video-based multi-view and panoramic virtual reality viewer manufacturing method using 3D planar tile array texture mapping according to the present invention includes: 1) using at least one camera to create an object in a real space; Photographing multiple angles simultaneously and generating an image and an image for each angle of a multi-view pointed at various angles by centering the object; 2) generating an image and an image for each angle of the panoramic view, which simultaneously photographs the multi-angle direction at one point in the real space using at least one or more cameras and looks around at a point in a 360-degree direction; 3) at least one or more images and images for each angle of the object in the real space at the same time or at least one or more in the digital virtual space to texture-mapped the images and images for each angle by simultaneously photographing the multi-angle direction at a point in the real space; Creating a plane object and editing data coordinates required for a texture mapping process; 4) occluding and editing the angle-specific image or the angle-specific image in a tile format to at least one plane object created in a digital virtual space to be a single image or image file; And 5) texture mapping the image file or the image file edited on the plurality of tiles to at least one plane object to generate a multi-view and panoramic virtual reality viewer.

The present invention uses 1-1) computer graphics software to create a computer graphics (CG) space, at least one camera supported by the software using the at least one camera in the computer graphics space at the same time taking multiple angles of images and images Generating a; And 2-1) creating a computer graphics (CG) space using computer graphics software, and simultaneously using the at least one camera supported by the software to take multiple angle directions at one point of the computer graphics to capture images and images for each angle. Generating; further comprising.

On the screen of the multi-view and panoramic virtual reality viewer of the present invention, a slide bar for changing the view of the plane facing the virtual reality viewer from side to side or an arrow for changing up, down, left, and right is displayed on the screen.

6) When the user changes the viewpoint of the plane facing the virtual reality viewer up, down, left, and right, it is determined whether the image file or the image file exists at the viewpoint of the changed plane that the image file or image file exists If the plane is moved or rotated to the corresponding position so that the user to view the viewpoint, and if the image file or image file does not exist, maintaining the position of the current output plane; It features.

The present invention configured as described above can realize the advanced lighting of the reality even in the low-end mobile device as high quality as the real world, the user can do the endless navigation as a video while looking around a single object In addition to providing a point viewer, there is an advantage in that an image-based viewing viewer and application capable of spatially exploring a panoramic view centered on one moving line can be implemented.

Unlike the existing virtual reality viewer, the present invention dynamically changes the viewpoint while watching the motion of an object as it is, so that more accurate shapes, motions, sound sources, and the like can be searched without delay with realistic quality. And smart TVs, such as high-quality multi-view and panoramic viewer can be implemented.

According to the present invention, virtual reality is implemented at the same time without loading an additional solution in a flash player which is a plug-in of a web browser on each user's PC online, and an independent application executable file according to an operating system (Android, IOS) in a mobile device. It can be manufactured as a low cost, but can be implemented in a lightweight but multiple devices to achieve a unified appreciation and minimum solution cost.

According to the present invention, by texture-mapping a plurality of images and videos in a tile format, the number of downloads can be reduced even in a mobile and smart TV environment, thereby achieving an economic effect.

1 illustrates a difference between reality and object-based virtual reality.
2 is a diagram illustrating an example of implementing virtual reality using a plurality of image files.
3 is an exemplary view showing a difference between a conventional image viewer and a video based multi-view and panoramic virtual reality viewer using 3D planar tile array texture mapping according to the present invention.
4 and 5 are diagrams for explaining a driving principle of a video-based multi-view and panoramic virtual reality viewer using 3D planar tile array texture mapping according to the present invention.
6 is a view showing the arrangement of the camera and the service type according to the present invention.
7 is a diagram illustrating an example of an array of tiles providing three views according to the present invention.
8 is a flowchart illustrating an operation state of a video-based multi-view and panoramic virtual reality viewer according to the present invention.
9 and 10 are diagrams illustrating screen states of a video-based multi-view and panoramic virtual reality viewer according to the present invention.

Hereinafter, exemplary embodiments of the present invention will be described in detail with reference to the accompanying drawings.

3 is an exemplary view showing a difference between a conventional image viewer and a video-based multi-view and panoramic virtual reality viewer using a 3D planar tile array texture mapping according to the present invention, and FIGS. 4 and 5 are views according to the present invention. FIG. 6 is a view illustrating a driving principle of a video-based multi-view and panoramic virtual reality viewer using 3D planar tile array texture mapping. FIG. 6 is a diagram illustrating an arrangement and service types of a camera according to the present invention. FIG. 8 is a diagram illustrating an example of a tile array providing three viewpoints according to the present invention. FIG. 8 is a flowchart illustrating an operation state of a video based multiview and a panoramic virtual reality viewer according to the present invention.

First, as shown in FIGS. 3 to 8, a method of manufacturing a video-based multi-view and panoramic virtual reality viewer using 3D planar tile array texture mapping according to the present invention includes an object in a real space using at least one camera. (1) to create a multi-angle image and image of a multi-view by viewing the multiple angles at the same time to view the object at various angles, and at the same time using the at least one camera at a point in the real space (2) generating an image and a video for each angle of a panoramic view looking around at a point in a 360-degree direction by taking a picture, and the image and video for each angle or simultaneously photographing the object in the real space in multiple angles; To texture-map images and images by angle by simultaneously shooting multiple angle directions from one point (3) generating at least one plane object in a digital virtual space, editing data coordinates required for a texture mapping process, and performing the angle-specific image on at least one plane object created in the digital virtual space. Or (4) combining and editing the angle-specific image in a tile format so as to be one image or image file, and texture-mapping the image file or image file edited on the plurality of tiles to at least one or more planar objects. Generating a point and panoramic virtual reality viewer (5).

In addition, the present invention is to create a computer graphics (CG) space using computer graphics software, using the at least one or more cameras supported by the software at the same time to shoot an object in the computer graphics space in multiple angles to generate images and images for each angle Step (1-1) and the computer graphics software to create a computer graphics (CG) space, using the at least one or more cameras supported by the software at the same time to shoot a multi-angle direction at a point in the computer graphics image by angle And generating the image (2-1).

The step (1), as shown in Type 01 of Figure 6, using the at least one or more cameras at the same time to shoot the object of the real space in a multi-angle to take the source data of the video-based multi-view and panoramic virtual reality viewer Creates images and images for each angle used.

The step (2), as shown in Type 02 of Figure 6 by using at least one or more cameras at the same point in the real space to take a multi-angle direction at the same time as if looking around the surroundings as a reality based on the video-based multi-view and panorama virtual Generates images and images for each angle used as source data of the reality viewer.

In step (1-1), in addition to step (1), the computer graphics (CG) space may be generated using computer graphics software, and at least one or more cameras supported by the software as shown in Type 01 of FIG. By using various angles of objects in the computer graphics space, images are taken at the same time to generate angle-specific images and images used as source data for video-based multi-view and panoramic virtual reality viewers.

The step (2-1), in addition to the step (2) to create a computer graphics (CG) space using computer graphics software, as shown in Type 02 of Figure 6 at least one or more cameras supported by the software At the same time, the multi-angle direction is taken at the same point in the computer graphics as if it is looking around, creating images and images for each angle used as source data of video-based multi-view and panoramic virtual reality viewers.

In the step (1-1) and the step (2-1), computer graphics may be generated by the development of computer graphics (CG) technology to generate images and images of a multi-view and a panoramic view from a camera viewpoint of a virtual world. Outputs images and images of the desired scene from a camera in space (OpenGL, DirectX, etc.). Unlike shooting in a real space, the camera has a high degree of freedom of movement, and the difficulty of shooting a reality such as invasion of privacy and difficulty of shooting can be improved. For example, an image and an image having multiple angles can be calculated with respect to one animated three-dimensional data.

In the step (3), at least one plane object is generated in a digital virtual space for texture mapping the angle-specific images and images generated in the steps (1, 2, 1-1, 2-1). And, in order to specify the color of the shape as in the real world or to output it on the surface of the object as an image, the image can be combined with the shape using u, v, w coordinates in addition to the x, y, z coordinates to have the same color as the real world. Run texture mapping to ensure that

Here, in order to implement virtual reality, a plane having at least three points or more in a digital space must be implemented. With the development of digital technology, there are many ways to create virtual graphic shapes like VRML. Diverse from the most basic methods of creating graphical shapes through open source, like OpenGL, there are a variety of game engines that improve visualization performance with more real-world logic operations in the commercially developed DirectX specification. In recent years, various WebGL and Web3D algorithms that implement stereoscopic functions in mobile and web have been developed with open source and paid code.In addition to the traditional CPU technology, the performance has been improved by the GPU technology which is developed more recently. It can be implemented in space.

The open source web 3D used for the texture mapping technology includes, for example, Google's Papervision3D, Away3D, Alternativa3D, sandy3D, etc., but it can be used for commercial purposes without any logo exposure. You can import them together and develop them commercially.

In the step (4), the generated angle-specific image or the angle-specific image is occluded and edited in a tile format so that the generated angle-specific image or the angle-specific image is in one or more plane objects created in a digital virtual space, and thus one image. Produce to be. At this time, a plurality of images having the number of simultaneous frames are tiled into one image, and at least 640x480 and six images can be occluded as shown in FIG. 7 at the current 2048x1024 tolerance value.

(5) generating a multi-view and panoramic virtual reality viewer by texture mapping the image file or the image file edited on the plurality of tiles to at least one plane object.

In the step (5), a texture mapping process of image files or image files edited in the plurality of tiles to at least one plane object. For example, any authoring tool capable of producing a plane in a virtual space such as Unity3D, a commercial authoring tool, is applicable to the present invention. However, the application size of the image material applicable to the plane provided by the development environment of the authoring tool is different depending on the authoring environment. 2048 ㅧ 2048 is allowed in the applicable Away3D of the present invention, and the minimum allowable size is 2048 ㅧ 1024 at the present time because the web browser allows the minimum allowable 2048 ㅧ 1024 of the image material. In consideration of the development situation and specifications, it may be allowed to be expanded to 2048 ㅧ 1024 or more. The tile type mapping type that can be implemented according to the present invention to the technical specification is as shown in FIG. 7.

Meanwhile, as illustrated in FIGS. 9 and 10, a slide bar for changing the viewpoint of a plane facing the virtual reality viewer to the left and right or an arrow for changing to the top, bottom, left and right, are displayed on the screen of the multi-view and panoramic virtual reality viewer. do.

The step (6) is when the user changes the viewpoint of the plane facing the virtual reality viewer while the multi-view and panoramic virtual reality viewer is displayed on the screen (S10) as shown in FIG. 8 (S20). In operation S30, the image file or the image file is determined whether the image file or the image file exists at the viewpoint of the changed plane, and when the image file or the image file exists, the user views the viewpoint by moving or rotating the plane to the corresponding position. If there is no corresponding image file or image file (S40), the position of the currently output plane is maintained (S10).

The present invention thus constructed provides a technique for constructing a virtual space, creating a plane in the virtual space, mapping a multi-faceted image to a single image interlocked with the generated plane, and providing various viewpoints of the virtual space. It should be possible to call up multiple planes at the same time for instant viewing and manipulation.

That is, by moving the three-dimensional plane in which a plurality of images are reproduced in accordance with the operation of the user's viewer it is possible to implement an image or image-based virtual reality viewer that exactly matches the voice and frame. As the moving digital plane moves at an instantaneous speed of 30 to 12 frames, the user does not feel the moving speed of the plane.

The image-based virtual reality viewer provided in this way provides the advanced visual quality of reality in terms of visualization quality, and achieves angles and scene transitions without any interruption like reality, and the image that the user sees fits with one sound source. Because it is a single image, it is possible to realize a multi-view viewer of sophisticated objects more than a sound source by feeling a continuous transition without a bouncing part. As shown in FIG. 6, the camera is a camera type 01 system, which is a camera type in which an object is viewed by rotating the object like a satellite (this is called a multi-view method).

In addition, the type 02 method shown in FIG. 6, which uses a bundle of a plurality of cameras as a material rather than the result of photographing an object as a satellite, is viewed as a 360-degree direction in a plurality of directions (this is a panoramic view method). In the present invention, the present invention can produce an appreciation viewer capable of appreciating both methods as in the real world.

The present invention improves the problem of the case by implementing the two-dimensional file driving method proposed in the conventional analog method in a digital three-dimensional method, two implementation methods of multi-view (Type 01) and panoramic view (Type 02) All is possible.

9 and 10 are diagrams illustrating screen states of a video-based multi-view virtual reality viewer according to the present invention.

As shown in FIG. 9 and FIG. 10, the present invention provides an integrated development environment to be executed as an application in a web browser, a standalone executable file operating in a computer operating system such as a PC or a Mac, and each mobile operating system (Android, IOS). It is an integrated development environment such as Unity3D, OpneGL, and Flash IDE, which is developed to correspond to N platforms.

First of all, in the case of Flash IDE, the basic 2D configuration is processed by traditional Flash ActionScript, and the web 3D module necessary for the flat screen configuration can be integrated into Away3D as shown in Table 1 to achieve the result.

The Away3D supports a plane as a basic three-dimensional figure. Create a plane on Away3D and insert the development specification for each operation. Insert a plane creation to create a plane facing the camera and insert a MovieMaterial to apply the image material.

In addition, as shown in Table 2, the user manipulation unit also inserts a MouseEvent, which is a two-dimensional coding according to the user manipulation, to move the plane. The display is processed on the View3D of the screen, and the image is applied to the virtual plane moving in response to the manipulation of the user's virtual camera and the output is integrated.

As described above, the multi-view viewer of the present invention is integrated with the web browser and operates as * .swf in the web browser, * .apk in the Android operating system, and * .ipa in the IOS-based operating system. In addition, the application of mobile devices and smart TV can provide a multi-view video.

The user rotates various viewpoints by moving the video applied to the virtual plane as one frame of the video, so that the contents can be watched while watching the movement of the historical works, viewing multi-views of entertainment works such as sound sources, and all kinds. The product's multi-view appreciation and marketing tools.

On the screen of the video-based multi-view virtual reality viewer shown in FIGS. 9 and 10, arrows are displayed to move up, down, left, and right, or move left and right according to the captured image prepared for the user's content viewing. A user operation unit is provided to display the slide bar to enable this.

As described above, the present invention changes the viewpoint while dynamically watching the motion of the object as it is in the conventional virtual reality viewer, so that it is possible to search for a more accurate shape, motion, sound source, etc. in a realistic quality. For example, if you try to explain the relics by the conventional method, only three-dimensional search of the epidermal shape such as shape, color, and material while turning the relics is possible. Can be configured to view the omni-directional view by the number of viewpoints separated by shooting or computer graphics. That is, the disadvantages of the object viewers of the web browsers that are currently operating are: 1) high quality with the picture quality, 2) continuous connection in response to the user's direction switching button, and 3) downloading multiple files of the example. Considering the mobile and smart TV situation to be easy to manage and apply, complementing these three technical limitations, the present invention can be applied not only image-based but also image method.

In addition, the present invention is a method of realizing high-quality sophisticated image occlusion by converting the thinking of the frame synchronization method of the two-dimensional image file into a three-dimensional method, operating as an application in the operating system leading to the mobile era and early IPTV and smart TV This is made possible by embedding a minimum three-dimensional plane into the model, and can operate as a new way of broadcasting applications.

In addition, although the virtual reality appreciation method of sphere, box, and cylinder type which is mainly used in the conventional virtual reality provider is used, the present invention is to provide a light and intuitive viewpoint change using a basic planar method.

In the present invention, the frame of the image provided mainly by mobile and web coding stays at 12 to 30 frames per second, and the moving speed of the plane supported by the open source 3D module corresponds to at least one frame of the image, thereby ensuring immediate movement and rotation. In addition, if the frame of the multi-view image of the tile type used for texture mapping is exactly matched, the resultant multi-view viewer can be realized with the occlusion that moves immediately in the frame.

Since the present invention uses only the minimum number of image data even in a low specification mobile device, it is possible to design suitable for data loading. In this way, the viewing angle of the multi-view like the real world can be viewed based on video.

It will be apparent to those skilled in the art that various modifications and variations can be made in the present invention without departing from the spirit or scope of the invention as defined in the appended claims. , Alteration, substitution, addition, or the like.

Claims (4)

1) simultaneously photographing an object in the real space using at least one or more cameras in multiple angles to generate an image and an image for each angle of a multi-view pointed at various angles with the object at the center;
2) generating an image and an image for each angle of the panoramic view, which simultaneously photographs the multi-angle direction at one point in the real space using at least one or more cameras and looks around at a point in a 360-degree direction;
3) at least one or more images and images for each angle of the object in the real space at the same time or at least one or more in the digital virtual space to texture-mapped the images and images for each angle by simultaneously photographing the multi-angle direction at a point in the real space; Creating a plane object and editing data coordinates required for a texture mapping process;
4) occluding and editing the angle-specific image or the angle-specific image in a tile format to at least one plane object created in a digital virtual space to be a single image or image file; And
5) generating a multi-view and panoramic virtual reality viewer by texture-mapping an image file or image file edited on the plurality of tiles to at least one plane object; a video using 3D plane tile array texture mapping comprising a How to create a multi-view and panoramic VR viewer. The method of claim 1,
1-1) Create a computer graphics (CG) space using computer graphics software, and use the at least one camera supported by the software to simultaneously shoot objects in the computer graphics space at multiple angles to create images and images for each angle. step; And
2-1) Create computer graphics (CG) space using computer graphics software, and create images and videos for each angle by simultaneously taking multiple angle directions from one point of computer graphics using at least one camera supported by the software. Method of producing a video-based multi-view and panoramic virtual reality viewer using a three-dimensional planar tile array texture mapping further comprising.
3. The method according to claim 1 or 2,
3D planar tile array texture mapping on the screen of the multi-view and panoramic virtual reality viewers, a slide bar for changing the view of the plane facing the virtual reality viewer from side to side or an arrow for changing up, down, left, and right is displayed on the screen Video-based multi-view and panoramic virtual reality viewer production method using.
The method of claim 3, wherein
6) When the user changes the viewpoint of the plane facing the virtual reality viewer up, down, left, and right, it is determined whether the image file or the image file exists at the viewpoint of the changed plane, and if the image file or the image file exists And moving or rotating the plane to a position where the user views the view, and if the image file or the image file does not exist, maintaining the position of the currently output plane. Video-based multi-view and panoramic VR viewer using dimensional plane tile array texture mapping.

Images