KR20200076234A - System for producing 3 dimension virtual reality content - Google Patents

Abstract

The present invention relates to a technology for producing mixed virtual reality (VR) content and a technology for realizing a platform therefor, which can produce VR content immediately operable in an existing commercial hardware device (e.g., PC, smartphone, HMD, etc.) while user interaction is reinforced and merging with a heterogeneous media can be performed by applying a 3D object automatic generating technology, a VR overlay technology, and a responsive 3D sound technology in production of VR content. According to an embodiment of the present invention, VR content can be easily produced, which can be used in various fields such as tour business (e.g., VR promotion, experience, reservation, payment, etc.), shopping (e.g., home shopping, sales and payment of a tour souvenir, etc.), education, real estate, etc. Even small businesses and individuals can produce mixed VR content, which are created by the small businesses and individuals, as an independent application and web to promote and provide the same as a service.

Description

3D VR content production system {SYSTEM FOR PRODUCING 3 DIMENSION VIRTUAL REALITY CONTENT}

In the present invention, user interaction is strengthened by applying 3D object automatic generation technology, VR overlay technology, responsive 3D stereoscopic sound technology, etc. in creating VR (Virtual Reality) content, and merging with heterogeneous media is possible. It relates to a mixed VR content production technology capable of producing VR content that can be operated immediately on a hardware device (for example, a PC, a smartphone, a HMD, etc.) and a platform implementation technology for the same.

Recently, 360-degree VR (Virtual Reality) content has been spotlighted, and individuals are also producing a lot of VR content using action cams, 360-degree cameras, and drones. In addition, companies and countries also produce and provide various types of VR content. Until now, many types of individual VR content were commissioned to production companies by investing a lot of money. For this reason, there is a vicious cycle with few high-quality VR contents and hard to reach the general consumer.

In the case of current VR contents, in order to overcome this in the form of an initial market whose business model has not yet been established and to activate VR contents in real life, more and more individual creators must make various VR contents and actual business using them must be steadily developed. .

In this market situation, a mixed VR platform needs to be provided as an open VR content platform that can be used in various fields, from small businesses to businesses, businesses, and government offices. When this Mixed VR platform is provided, it can be used for various tourism business (e.g., VR promotion, experience, reservation, payment, etc.), shopping (e.g., home shopping, sales of tourist souvenirs, payment, etc.), education, real estate, etc. It can be used as a field.

In addition, according to the Mixed VR platform, a panoramic image (eg, a panoramic image generated by a smartphone or a digital camera), a 360-degree camera (for example, Insta 360, Samsung Gear 360, etc.), an action cam (For example, a 360-degree action cam) can be used to easily operate a VR tourism business, a VR shopping mall, etc. by using images and videos shot using it.

In addition, according to this Mixed VR platform, it is expected that the era of the creation of an open VR content can be opened, as the era of personal video content creator called YouTube is opened due to YouTube.

In the present invention, user interaction is strengthened by applying 3D object automatic generation technology, VR overlay technology, responsive 3D stereoscopic sound technology, etc. in creating VR (Virtual Reality) content, and merging with heterogeneous media is possible. It is intended to provide a mixed VR content production technology and a platform implementation technology for producing VR content that can be operated immediately on hardware devices (eg, PC, smartphone, HMD, etc.).

According to an aspect of the present invention, a 3D VR (3-dimension Virtual Reality) content production system, providing a user with a software tool (software tool) to assist in inserting realistic content into a pre-made 360-degree stereoscopic VR And a GUI (Graphical User Interface) unit for receiving user input; A sensory content application unit that applies the user-designated sensory content to the 360-degree panoramic panorama VR through the GUI unit; A 3D VR content production system is provided that includes an interactive action processing unit that controls execution of sensory content applied in the 360-degree stereoscopic VR in response to a user action.

According to an embodiment of the present invention, in creating VR (Virtual Reality) content, user interaction is enhanced by applying 3D object automatic generation technology, VR overlay technology, responsive 3D stereoscopic sound technology, etc. It is possible and has the effect of providing mixed VR content creation technology and platform implementation technology for creating VR content that can be operated immediately on existing commercial hardware devices (eg, PC, smartphone, HMD, etc.). .

In addition, according to the mixed VR content production and platform implementation technology according to an embodiment of the present invention, tourism business (e.g., VR promotion, experience, reservation, payment, etc.), shopping (e.g., home shopping, sales of tourism souvenirs, etc.) , Payment, etc.), education, real estate can easily produce VR content that can be used in various fields, and small businesses and individuals can create their own Mixed VR content as independent APP and WEB to promote and service.

1 is a view for explaining the overall 3D VR content creation tool and platform according to an embodiment of the present invention.
2 is a block diagram of a 3D VR content production system according to an embodiment of the present invention.
Fig. 3 is an exemplary diagram in which stereoscopic sound is inserted into the 360-degree stereoscopic panorama VR as sensory content.
Fig. 4 is a reference diagram for explaining setting conditions of stereoscopic sound as sensory content.
FIG. 5 is a reference diagram illustrating an operation window for inserting a video of a video as a immersive content within a 360-degree stereoscopic panorama VR.
FIG. 6 is a reference diagram illustrating an example in which gaze intensity of a rainbow color spectrum corresponding to a user's view point is expressed in a coordinate region within a 360-degree stereoscopic panorama VR.
7 is a view for explaining an intermediate filtering process when overlaying a specific target image in a 360-degree stereoscopic panorama VR.
8 is a comparison table for contrasting the performance of an image overlay algorithm and an existing technology according to an embodiment of the present invention when overlaying a specific target image in a 360-degree stereoscopic panorama VR.
9 is a reference diagram for contrasting the image overlay scheme according to an embodiment of the present invention with the image overlay scheme according to the prior art.

The present invention can be applied to a variety of transformations and may have various embodiments, and specific embodiments will be illustrated in the drawings and described in detail in the detailed description. However, this is not intended to limit the present invention to specific embodiments, and should be understood to include all conversions, equivalents, and substitutes included in the spirit and scope of the present invention.

In describing the present invention, when it is determined that detailed descriptions of related known technologies may unnecessarily obscure the subject matter of the present invention, detailed descriptions thereof will be omitted. In addition, the numbers (for example, first, second, etc.) used in the description process of this specification are only identification symbols for distinguishing one component from other components.

Further, in the specification, when one component is referred to as "connected" or "connected" with another component, the one component may be directly connected to the other component, or may be directly connected, but in particular It should be understood that, as long as there is no objection to the contrary, it may or may be connected via another component in the middle. Also, in the specification, when a part “includes” a certain component, it means that the component may further include other components, not to exclude other components, unless otherwise stated. In addition, terms such as “part” and “module” described in the specification mean a unit that processes at least one function or operation, which means that it can be implemented by one or more hardware or software or a combination of hardware and software. .

1 is a view for generally describing a 3D VR content production tool and platform according to an embodiment of the present invention, and FIG. 2 is a block diagram of a 3D VR content production system according to an embodiment of the present invention.

In addition, FIG. 3 is an exemplary diagram in which stereoscopic sound is inserted into the 360-degree stereoscopic VR as sensory content, FIG. 4 is a reference diagram for explaining a setting condition of stereoscopic sound as the sensory content, and FIG. 5 is a 360-degree stereoscopic panorama VR A reference diagram illustrating an operation window for inserting a video of a video as immersive content within, and FIG. 6 shows an example in which gaze intensity of a rainbow color spectrum corresponding to a user's view point is expressed in a coordinate region within a 360-degree stereoscopic VR FIG. 7 is a diagram for explaining an intermediate filtering process when overlaying a specific target image in a 360-degree stereoscopic panorama VR, and FIG. 8 is when overlaying a specific target image in a 360-degree stereoscopic panorama VR. A comparison table for contrasting the performance of an image overlay algorithm and an existing technology according to an embodiment of the present invention, and FIG. 9 is a method for contrasting an image overlay system according to an embodiment of the present invention and It is a reference drawing. Hereinafter, embodiments of the present invention will be described in detail with reference to FIGS. 3 to 9 with reference to FIGS. 1 and 2.

Referring to FIG. 1, the present invention is applied to 3D object automatic generation technology, VR overlay technology, responsive 3D stereoscopic sound technology, etc. in creating VR (Virtual Reality) content, and user interaction is strengthened and merged with heterogeneous media. It provides a mixed VR content creation technology and a platform implementation technology for producing VR content that is possible and can be operated immediately on existing commercial hardware devices (eg, PC, smartphone, HMD, etc.).

To this end, the 3D VR content production system 100 according to an embodiment of the present invention, as shown in FIG. 2, is a software tool that assists in inserting realistic content into a pre-made 360-degree stereoscopic panorama VR (software) tool) to the user, the GUI unit 110 for receiving a user input; A sensory content application unit 120 that applies the user-designated sensory content to the 360-degree panoramic panorama VR through the GUI unit 110; It includes an interactive action processing unit 130 for controlling the execution of the sensory content applied in the 360-degree stereoscopic VR in response to a user action. In addition, the 3D VR content production system 100 according to an embodiment of the present invention may further include a view point analysis unit 140, a stereoscopic sound application unit 122 and an image in the sensory content application unit 120. The overlay processing unit 124 may be further included.

Accordingly, the 3D VR content production system 100 according to an embodiment of the present invention is applied with various functions for the production of the following 3D VR content.

3D stereoscopic sound insertion and playback technology

According to an embodiment of the present invention, the 3D VR content production system 100 plays/stops and volumes sound in a 3D stereoscopic sound set by the content creator according to the angle/distance according to the viewer's gaze in a virtual 3D space Technology that can automatically adjust is applied.

To this end, the GUI unit 110 may provide a user with a tool screen for application of 3D stereoscopic sound, so that a content creator can apply 3D stereoscopic sound within a 360-degree stereoscopic panorama VR. At this time, on the tool screen for application of 3D stereoscopic sound, designation according to setting items as shown in Table 1 below (i.e., designation of horizontal and vertical positions to which the stereoscopic sound will be inserted in the 360-degree stereoscopic panorama VR, the Designation of a sound output direction or angle in which a stereoscopic sound is output, designation of an audio volume of the stereoscopic sound, designation of an audio playback speed of the stereoscopic sound, designation of whether or not the stereoscopic sound is repeatedly played, and a criterion for outputting the stereoscopic sound An operation window that enables the designation of a proximity distance, etc.) may be included.

Accordingly, the content creator can insert 3D stereoscopic sound as sensory content at a specific point in the 3D space through the above-described operation window. At this time, the angle/distance may be subdivided (for example, in step 360) to enable adjustment of whether to play 3D stereoscopic sound/play speed/volume according to the angle/distance (see FIG. 4).

As described above, when the user (that is, the content creator) setting through the operation window is completed, the stereoscopic sound application unit 122 inserts the corresponding 3D stereoscopic sound into the 360-degree stereoscopic panorama VR corresponding to the setting contents (FIG. 3). Accordingly, the audio can be automatically played/paused according to the viewer's gaze.

Image overlay technology

According to the existing technology, 2D image and video playback was performed in a simple pop-up form in 360 VR (refer to FIG. 9(a)), whereas according to an embodiment of the present invention, the 3D VR content production system 100 The overlay technology and the overplay technology of a 2D image and video without a sense of heterogeneity and a background of a 360-degree stereoscopic panoramic VR are applied (see FIG. 9(b) ).

To this end, the operation for designating a position to overlay a specific target image in a 360-degree stereoscopic VR and changing the shape of the target image is first required. Accordingly, in an embodiment of the present invention, the GUI unit 110 provides a user with a tool screen for application of target video content as the sensory content. At this time, on the tool screen for the image overlay, designation according to the setting items as shown in Table 2 below (that is, designation of horizontal and vertical positions to which the video content will be inserted in the 360-degree stereoscopic VR, position of the height axis) An operation window for designating, specifying a size of an image to be inserted, designating a deformation of an outer border shape of an image to be inserted, and the like) may be included. As a manipulation window for image overlay, a controller in the form of a remote controller is illustrated in FIG. 5.

According to this, in setting the overlay position of the target image in the 3D environment, the 3D axes X, Y, and Z coordinates and the correlation axis and angle can be calculated, and the corresponding values can be adjusted, and the user (content creator) can operate according to the operation window. By adjusting the GUI, it is possible to merge desired target images on a 3D screen without complicated calculation. At this time, various setting items such as the position and size of the space to be overlaid can be freely adjusted with an accurate numerical value or a touch and a cursor.

When the user (that is, the content creator) setting through the operation window is completed, the image overlay processing unit 124 inserts the target image content by overlaying it in the 360-degree three-dimensional panoramic VR corresponding to the setting contents (FIG. ) Reference).

In addition, in the embodiment of the present invention, a homography matrix is obtained by using a Speeded Up Robust Features (SURF) algorithm and a RANRAC (RANdomSAmple Consensus) algorithm to merge the target image into one with a 360-degree stereo panorama VR with increased immersion. The corresponding merging point of the target image to be merged is obtained and a sharper and more accurate image merging algorithm is applied. At this time, an intermediate filtering process is performed to remove incorrect correspondence points using the RGB values in the area around the merged coordinates, and a merge method for improving image merge processing speed and improving image merge accuracy (ie, image discrimination and cyan) can be applied. have. This will be described in detail as follows.

That is, the image overlay processor 124 extracts feature information of a target image to be inserted as the sensory content using a feature information extraction algorithm, and uses the RANSAC (Random Sample Consensus) algorithm to perform an outside of the extracted feature information. By overlaying the target image with a specific coordinate region within the 360-degree panoramic panorama VR in a manner to obtain a homography matrix for extracting a corresponding point between images to be overlaid through repetition of regression calculation to remove feature information corresponding to (outlier) Order.

At this time, prior to applying the RANSAC algorithm, it is determined whether or not the difference between the RGB value around the coordinate area to which the target image is to be overlaid in the 360-degree stereoscopic VR and the RGB value of the target image is within a predetermined threshold, By performing an intermediate filtering process to remove the corresponding points where the difference exceeds the threshold, the speed and accuracy of image merging can be further increased (see FIGS. 7 and 8 ).

Interactive Action activation technology

Also, according to the existing technology, an interactive hotspot is defined in a 360-degree three-dimensional panoramic VR, and the interactive action is executed through an action such as a touch. At this time, according to the existing technique, the index definition for satisfying the condition that the interactive action is executed is uniform. For example, when looking at a specific point for 5 seconds or more, or simply touching a specific point.

On the other hand, according to an embodiment of the present invention, the view point of the user (viewer) is analyzed to automatically execute or stop the interactive action when a specific condition is satisfied. The specific condition at this time may be, for example, when the user's view point is overlaid by 60% or more of the time. Accordingly, in the embodiment of the present invention, a pixel of a 360-degree stereoscopic panoramic VR image and a viewpoint viewed by the current user are analyzed (for example, Level 5), and a pixel value that is more than & equal to a specific condition (specific level) is matched. By analyzing, it is possible to determine whether to execute the interactive action. This will be described in more detail as follows.

That is, the view point analysis unit 140 tracks a view point that the user is currently viewing, and matches and analyzes a coordinate pixel in the 360-degree stereoscopic panorama VR and the user's viewpoint. At this time, the interactive action processor 130 may display a color according to a predetermined saturation spectrum according to the length of the gaze time, on the screen position of the coordinate pixel in the 360-degree stereoscopic panorama VR corresponding to the analyzed view point. In the case of FIG. 6, the view point is defined by separating it into a rainbow color. For example, red in FIG. 6 corresponds to a Most-stared-at view point or Longgest-attention holding view point. As described above, in the embodiment of the present invention, a view point is separately defined by a color spectrum, and an interactive action can be determined accordingly.

Accordingly, the interactive action processing unit 130 executes the sensational content applied to the corresponding coordinate area when the gaze retention time at the analyzed view point satisfies a predetermined criterion, and the coordinate area according to the analyzed view point is different from the previous one. If it changes, you can stop running sensory content.

In addition, according to an embodiment of the present invention, as an execution method of interactive action of sensory content applied to a coordinate area corresponding to the analyzed view point, as shown in Table 3 below, a place to a space related to the coordinate area Movement, playback of overlaid video, audio output of stereoscopic sound, connection or link playback to a designated external link, display of augmented information (i.e. pop-up of information box and insertion of information, etc.), dialing to a specified phone number, goods Purchase and payment may be applied.

Although described above with reference to the embodiments of the present invention, those skilled in the art variously modify the present invention without departing from the spirit and scope of the present invention as set forth in the claims below. And it can be easily understood.

Claims (7)

3D VR (3-dimension Virtual Reality) content production system,
A GUI (Graphical User Interface) unit for providing a user with a software tool to assist in inserting sensational content into a pre-made 360-degree panoramic VR, and receiving user input;
A sensory content application unit that applies the user-designated sensory content to the 360-degree panoramic panorama VR through the GUI unit;
Interactive action processing unit that controls the execution of sensational content applied in the 360-degree stereoscopic panorama VR in response to a user action
3D VR content production system comprising a.
According to claim 1,
The GUI unit provides a user with a tool screen for applying stereoscopic sound as the sensory content,
On the tool screen, the horizontal and vertical positions to which the stereoscopic sound is to be inserted in the 360-degree stereoscopic panorama VR, the sound output to which the stereoscopic sound is output, the direction or angle, the audio volume of the stereoscopic sound, An operation window is provided that enables designation of an audio reproduction speed of the stereoscopic sound, designation of whether to reproduce the stereoscopic sound repeatedly, and a reference proximity distance to which the stereoscopic sound is to be output.
The sensory content application unit includes a stereoscopic sound application unit that inserts the stereoscopic sound into the 360-degree panoramic panorama VR in correspondence with a designation specified by a user through the manipulation window.
According to claim 1,
The GUI unit provides a user with a tool screen for applying video content as the sensory content,
On the tool screen, the horizontal and vertical positions to which the video content is to be inserted, the position of the height axis, the size of the image to be inserted, and the modification of the shape of the outer border of the image to be inserted are specified in the 360-degree panoramic VR. Includes an operation window to enable,
The sensory content application unit includes a video overlay processing unit for overlaying and inserting the video content into the 360-degree panoramic panorama VR in correspondence with a designation specified by a user through the manipulation window. VR content production system.
According to claim 3,
The image overlay processing unit,
The feature information extraction algorithm is used to extract the feature information of the target image to be inserted as the sensory content, and the feature information corresponding to the outlier is removed from the extracted feature information using the RANSAC (Random Sample Consensus) algorithm. In a manner of obtaining a homography matrix for extracting a corresponding point between images to be overlaid through repetition of the regression calculation, the target image is overlaid and merged to a specific coordinate region in the 360-dimensional stereoscopic panorama VR,
Before applying the RANSAC algorithm, it is determined whether or not the difference between the RGB value around the coordinate region to which the target image will be overlaid within the 360-degree stereoscopic VR and the RGB value of the target image is within a predetermined threshold, and 3D VR content production system, characterized by performing an intermediate filtering process to remove the corresponding points where the difference exceeds the threshold.
According to claim 4,
Further comprising a view point analysis unit for tracking and matching the viewpoint between the user's point of view and tracking the user's point of view by tracking the view point currently viewed by the user,
The interactive action processor displays a color according to a predetermined saturation spectrum according to the length of the gaze time at a screen position of a coordinate pixel in the 360-degree stereoscopic panorama VR corresponding to the analyzed view point, 3D VR Content production system.
The method of claim 5,
The interactive action processing unit,
If the sensory content is applied to the coordinate area in the 360-degree stereoscopic panorama VR corresponding to the analyzed view point, and the gaze retention time at the analyzed view point satisfies a predetermined criterion, the sensory content applied to the corresponding coordinate area is displayed. 3D VR content production system, characterized in that, when the coordinate region according to the analyzed view point is changed differently from the previous one, the running sensory content is stopped.
The method of claim 6,
The interactive action processing unit,
Execution of sensational content applied to the coordinate area corresponding to the analyzed view point, moving the place to the space related to the coordinate area, playing the overlayed video, outputting stereoscopic sound, connecting or linking to a specified external link, or augmenting A 3D VR content production system that executes at least one interactive action of displaying information, making a call to a designated phone number, purchasing goods, and making payments.

Images