WO2020130275A1

WO2020130275A1 - Three-dimensional vr content production system

Info

Publication number: WO2020130275A1
Application number: PCT/KR2019/010519
Authority: WO
Inventors: 김성국
Original assignee: 주식회사 에이투젯
Priority date: 2018-12-19
Filing date: 2019-08-19
Publication date: 2020-06-25
Also published as: KR102239877B1; KR20200076234A

Abstract

The present invention relates to a mixed virtual reality (VR) content production technology and a platform implementation technology for same, which applies an automatic 3D object generation technology, a VR overlay technology, a responsive 3D stereoscopic sound technology, and the like to production of VR contents so as to enhance user interaction and enable merging with heterogeneous media, and enables production of VR contents which can be immediately operated on an existing commercial hardware device (for example, a PC, a smartphone, an HMD, and the like). According to an embodiment of the present invention relating to a mixed VR content production and platform implementation technology, VR contents which can be used in various fields, such as tourism business (for example, VR promotion, experience, reservation, payment, and the like), shopping (for example, home shopping, sales of tourist souvenirs, etc., payment, and the like), education, and real estate, can be easily produced, and even small business owners and individuals can create independent apps and webs using mixed VR contents that are created thereby, so as to provide promotion and services thereof.

Description

3D VR content production system

In the present invention, user interaction is strengthened by applying 3D object automatic generation technology, VR overlay technology, responsive 3D stereoscopic sound technology, etc. in creating VR (Virtual Reality) content, and merging with heterogeneous media is possible. It relates to a mixed VR content production technology capable of producing VR content that can be operated immediately on a hardware device (for example, a PC, a smartphone, a HMD, etc.) and a platform implementation technology for the same.

Recently, 360-degree VR (Virtual Reality) content has been spotlighted, and individuals are also producing a lot of VR content using action cams, 360-degree cameras, and drones. In addition, companies and countries also produce and provide various types of VR content. Until now, many types of individual VR content were commissioned to production companies by investing a lot of money. For this reason, there is a vicious cycle with few high-quality VR contents and hard to reach the general consumer.

In the case of current VR contents, in order to overcome this in the form of an initial market whose business model has not yet been established and to activate VR contents in real life, more and more individual creators must make various VR contents and actual business using them must be steadily developed. .

In this market situation, a mixed VR platform needs to be provided as an open VR content platform that can be used in various fields, from small businesses to businesses, businesses, and government offices. When this Mixed VR platform is provided, it can be used for various tourism business (e.g., VR promotion, experience, reservation, payment, etc.), shopping (e.g., home shopping, sales of tourist souvenirs, payment, etc.), education, real estate, etc. It can be used as a field.

In addition, according to the Mixed VR platform, a panoramic image (eg, a panoramic image generated by a smartphone or a digital camera), a 360-degree camera (for example, Insta 360, Samsung Gear 360, etc.), an action cam (For example, a 360-degree action cam) can be used to easily operate a VR tourism business, a VR shopping mall, etc. by using images and videos shot using it.

In addition, according to this Mixed VR platform, it is expected that the era of the creation of an open VR content can be opened, as the era of personal video content creator called YouTube is opened due to YouTube.

In the present invention, user interaction is strengthened by applying 3D object automatic generation technology, VR overlay technology, responsive 3D stereoscopic sound technology, etc. in creating VR (Virtual Reality) content, and merging with heterogeneous media is possible. It is intended to provide a mixed VR content production technology capable of producing VR content that can be operated immediately on a hardware device (for example, a PC, smartphone, HMD, etc.) and a platform implementation technology for the same.

According to an aspect of the present invention, a 3D VR (3-dimension Virtual Reality) content production system, providing a user with a software tool (software tool) to assist in inserting realistic content into a pre-made 360-degree stereoscopic VR And a GUI (Graphical User Interface) unit for receiving user input; A sensory content application unit that applies the user-designated sensory content to the 360-degree panoramic panorama VR through the GUI unit; A 3D VR content production system is provided that includes an interactive action processing unit that controls execution of sensory content applied in the 360-degree panoramic VR in response to a user action.

According to an embodiment of the present invention, in creating VR (Virtual Reality) content, user interaction is enhanced by applying 3D object automatic generation technology, VR overlay technology, responsive 3D stereoscopic sound technology, etc. It is possible and has the effect of providing mixed VR content creation technology and platform implementation technology for creating VR content that can be operated immediately on existing commercial hardware devices (eg, PC, smartphone, HMD, etc.). .

In addition, according to the mixed VR content production and platform implementation technology according to an embodiment of the present invention, tourism business (e.g., VR promotion, experience, reservation, payment, etc.), shopping (e.g., home shopping, sales of tourism souvenirs, etc.) , Payment, etc.), education, real estate can easily produce VR content that can be used in various fields, and small businesses and individuals can create their own mixed VR content as independent APP and WEB to promote and service.

1 is a view for explaining the overall 3D VR content creation tool and platform according to an embodiment of the present invention.

2 is a block diagram of a 3D VR content production system according to an embodiment of the present invention.

Fig. 3 is an exemplary diagram in which stereoscopic sound is inserted into the 360-degree stereoscopic panorama VR as sensory content.

Fig. 4 is a reference diagram for explaining setting conditions of stereoscopic sound as sensory content.

FIG. 5 is a reference diagram illustrating an operation window for inserting a video of a video as a immersive content within a 360-degree stereoscopic panorama VR.

FIG. 6 is a reference diagram illustrating an example in which gaze intensity of a rainbow color spectrum corresponding to a user's view point is expressed in a coordinate region within a 360-degree stereoscopic panorama VR.

7 is a diagram for explaining an intermediate filtering process when overlaying a specific target image in a 360-degree stereoscopic panorama VR.

8 is a comparison table for contrasting the performance of an image overlay algorithm and an existing technology according to an embodiment of the present invention when overlaying a specific target image in a 360-degree stereoscopic panorama VR.

The present invention can be applied to a variety of transformations and may have various embodiments, and specific embodiments will be illustrated in the drawings and described in detail in the detailed description. However, this is not intended to limit the present invention to specific embodiments, and should be understood to include all conversions, equivalents, and substitutes included in the spirit and scope of the present invention.

In describing the present invention, when it is determined that detailed descriptions of related known technologies may unnecessarily obscure the subject matter of the present invention, detailed descriptions thereof will be omitted. In addition, the numbers (for example, first, second, etc.) used in the description process of this specification are only identification symbols for distinguishing one component from other components.

Further, in the specification, when one component is referred to as "connected" or "connected" with another component, the one component may be directly connected to the other component, or may be directly connected, but in particular It should be understood that, as long as there is no objection to the contrary, it may or may be connected via another component in the middle. Also, in the specification, when a part “includes” a certain component, it means that the component may further include other components, not to exclude other components, unless otherwise stated. In addition, terms such as “part” and “module” described in the specification mean a unit that processes at least one function or operation, which means that it can be implemented by one or more hardware or software or a combination of hardware and software. .

1 is a view for generally describing a 3D VR content production tool and platform according to an embodiment of the present invention, and FIG. 2 is a block diagram of a 3D VR content production system according to an embodiment of the present invention.

In addition, FIG. 3 is an exemplary diagram in which stereoscopic sound is inserted into the 360-degree stereoscopic VR as sensory content, FIG. 4 is a reference diagram for explaining setting conditions of stereoscopic sound as the sensory content, and FIG. 5 is a 360-degree stereoscopic panorama VR A reference diagram illustrating an operation window for inserting a video of a video as immersive content within, and FIG. 6 shows an example in which gaze intensity of a rainbow color spectrum corresponding to a user's view point is expressed in a coordinate region within a 360-degree stereoscopic VR FIG. 7 is a diagram for explaining an intermediate filtering process when overlaying a specific target image in the 360-degree stereoscopic panorama VR, and FIG. 8 is when overlaying a specific target image in the 360-degree stereoscopic panorama VR. This is a comparison table for contrasting the performance of an image overlay algorithm and an existing technology according to an embodiment of the present invention. Hereinafter, embodiments of the present invention will be described in detail with reference to FIGS. 3 to 8 with reference to FIGS. 1 and 2.

Referring to FIG. 1, the present invention applies a 3D object automatic generation technology, a VR overlay technology, and a responsive 3D stereoscopic sound technology to create VR (Virtual Reality) content, so that user interaction is enhanced and merged with disparate media. It provides a mixed VR content production technology and a platform implementation technology for producing VR content that can be operated and can be operated immediately on existing commercial hardware devices (eg, PC, smartphone, HMD, etc.).

To this end, the 3D VR content production system 100 according to an embodiment of the present invention, as shown in FIG. 2, is a software tool that assists in inserting realistic content into a pre-made 360-degree stereoscopic panorama VR (software) tool) to the user, the GUI unit 110 for receiving a user input; A sensory content application unit 120 that applies the user-designated sensory content to the 360-degree panoramic panorama VR through the GUI unit 110; It includes an interactive action processing unit 130 for controlling the execution of the sensory content applied in the 360-degree stereoscopic VR in response to a user action. In addition, the 3D VR content production system 100 according to an embodiment of the present invention may further include a view point analysis unit 140, a stereoscopic sound application unit 122 and an image in the sensory content application unit 120. The overlay processing unit 124 may be further included.

Accordingly, the 3D VR content production system 100 according to an embodiment of the present invention is applied with various functions for the production of the following 3D VR content.

3D stereoscopic sound insertion and playback technology

According to an embodiment of the present invention, the 3D VR content production system 100 plays/stops and volumes sound in a 3D stereoscopic sound set by the content creator according to the angle/distance according to the viewer's gaze in a virtual 3D space Technology that can automatically adjust is applied.

To this end, the GUI unit 110 may provide a user with a tool screen for application of 3D stereoscopic sound, so that a content creator can apply 3D stereoscopic sound within a 360-degree stereoscopic panorama VR. At this time, on the tool screen for application of 3D stereoscopic sound, designation according to setting items as shown in Table 1 below (i.e., designation of horizontal and vertical positions to which the stereoscopic sound will be inserted in the 360-degree stereoscopic panorama VR, the Designation of a sound output direction or angle in which a stereoscopic sound is output, designation of an audio volume of the stereoscopic sound, designation of an audio playback speed of the stereoscopic sound, designation of whether or not the stereoscopic sound is repeatedly played, and a criterion for outputting the stereoscopic sound An operation window that enables the designation of a proximity distance, etc.) may be included.

Accordingly, the content creator can insert 3D stereoscopic sound as sensory content at a specific point in the 3D space through the above-described operation window. At this time, the angle/distance may be subdivided (for example, in step 360) to enable adjustment of whether to play 3D stereoscopic sound/play speed/volume according to the angle/distance (see FIG. 4).

As described above, when the user (that is, the content creator) setting through the operation window is completed, the stereoscopic sound application unit 122 inserts the corresponding 3D stereoscopic sound into the 360-degree stereoscopic panorama VR corresponding to the setting contents (FIG. 3). Accordingly, the audio can be automatically played/paused according to the viewer's gaze.

Image overlay technology

According to the existing technology, 2D image and video playback was performed in a simple pop-up form in 360 VR, whereas according to an embodiment of the present invention, the 3D VR content production system 100 has a 360 degree stereoscopic VR background and heterogeneity 2D image and video overlay (overlay) technology and overplay (overplay) technology is applied.

To this end, the operation for designating a position to overlay a specific target image in a 360-degree stereoscopic VR and changing the shape of the target image is first required. Accordingly, in an embodiment of the present invention, the GUI unit 110 provides a user with a tool screen for application of target video content as the sensory content. At this time, on the tool screen for the image overlay, designation according to the setting items as shown in Table 2 below (that is, designation of horizontal and vertical positions to which the video content will be inserted in the 360-degree stereoscopic VR, position of the height axis) An operation window for designating, specifying a size of an image to be inserted, designating a deformation of an outer border shape of an image to be inserted, and the like) may be included. As a manipulation window for image overlay, a controller in the form of a remote controller is illustrated in FIG. 5.

According to this, in setting the overlay position of the target image in the 3D environment, the 3D axes X, Y, and Z coordinates and the correlation axis and angle can be calculated, and the corresponding values can be adjusted, and the user (content creator) can operate according to the operation window. By adjusting the GUI, it is possible to merge desired target images on a 3D screen without complicated calculation. At this time, various setting items such as the position and size of the space to be overlaid can be freely adjusted with an accurate numerical value or a touch and a cursor.

When the user (that is, the content creator) setting through the operation window is completed, the image overlay processing unit 124 inserts the target image content overlaid in a 360-degree stereoscopic panorama VR corresponding to the setting contents.

In addition, in the embodiment of the present invention, a homography matrix is obtained by using a Speeded Up Robust Features (SURF) algorithm and a RANRAC (RANdomSAmple Consensus) algorithm to merge the target image into one with a 360-degree stereo panorama VR with increased immersion. The corresponding merging point of the target image to be merged is obtained and a sharper and more accurate image merging algorithm is applied. At this time, an intermediate filtering process is performed to remove incorrect correspondence points using the RGB values in the area around the merged coordinates, and a merge method for improving image merge processing speed and improving image merge accuracy (ie, image discrimination and cyan) can be applied. have. This will be described in detail as follows.

That is, the image overlay processor 124 extracts feature information of a target image to be inserted as the sensory content using a feature information extraction algorithm, and uses the RANSAC (Random Sample Consensus) algorithm to perform an outside of the extracted feature information. By overlaying the target image with a specific coordinate region within the 360-degree panoramic panorama VR in a manner to obtain a homography matrix for extracting a corresponding point between images to be overlaid through repetition of regression calculation to remove feature information corresponding to (outlier) Order.

At this time, prior to applying the RANSAC algorithm, it is determined whether or not the difference between the RGB value around the coordinate area to which the target image is to be overlaid in the 360-degree stereoscopic VR and the RGB value of the target image is within a predetermined threshold, By performing an intermediate filtering process to remove the corresponding points where the difference exceeds the threshold, the speed and accuracy of image merging can be further increased (see FIGS. 7 and 8 ).

Interactive Action activation technology

Also, according to the existing technology, an interactive hotspot is defined in a 360-degree three-dimensional panoramic VR, and the interactive action is executed through an action such as a touch. At this time, according to the existing technique, the index definition for satisfying the condition that the interactive action is executed is uniform. For example, when looking at a specific point for 5 seconds or more, or simply touching a specific point.

On the other hand, according to an embodiment of the present invention, the view point of the user (viewer) is analyzed to automatically execute or stop the interactive action when a specific condition is satisfied. The specific condition at this time may be, for example, when the user's view point is overlaid by 60% or more of the time. Accordingly, in the embodiment of the present invention, a pixel of a 360-degree stereoscopic panoramic VR image and a viewpoint viewed by the current user are analyzed (for example, Level 5), and a pixel value that is more than & equal to a specific condition (specific level) is matched. By analyzing, it is possible to determine whether to execute the interactive action. This will be described in more detail as follows.

That is, the view point analysis unit 140 tracks a view point currently viewed by a user, and matches and analyzes a coordinate pixel in the 360-dimensional stereoscopic panorama VR and the user's viewpoint. At this time, the interactive action processor 130 may display a color according to a predetermined saturation spectrum according to the length of the gaze time, on the screen position of the coordinate pixel in the 360-degree stereoscopic panorama VR corresponding to the analyzed view point. In the case of FIG. 6, the view point is defined by separating it into a rainbow color. For example, red in FIG. 6 corresponds to a Most-stared-at view point or a Longgest-attention holding view point. As described above, in the embodiment of the present invention, a view point is separately defined by a color spectrum, and an interactive action can be determined accordingly.

Accordingly, the interactive action processing unit 130 executes the sensational content applied to the corresponding coordinate area when the gaze retention time at the analyzed view point satisfies a predetermined criterion, and the coordinate area according to the analyzed view point is different from the previous one. If it changes, you can stop running sensory content.

In addition, according to an embodiment of the present invention, as an interactive action execution method of sensory content applied to a coordinate area corresponding to the analyzed view point, as shown in Table 3 below, a place to a space related to the corresponding coordinate area Movement, playback of overlaid video, audio output of stereoscopic sound, connection or link playback to a designated external link, display of augmented information (i.e. pop-up of information box and insertion of information, etc.), dialing to a specified phone number, goods Purchase and payment may be applied.

3D VR content (for example, a 360-degree stereoscopic panoramic image or video, etc.) produced by the 3D VR content production system according to the embodiment of the present invention as described above includes process management in areas such as construction and civil engineering Edo can be effectively used.

Existing process management software has been limited to input costs, resource management, and schedule management in the form of bars and flowcharts. In addition, in the prior art, information that was actually managed at the site was managed and controlled at the discretion of the site personnel, which caused difficulties in effective process management due to difficulties in smooth process information management, verification, and consolidation of information. Therefore, various types of tasks such as process management, site management, and process management reports are heavy on the existing process management system and process management personnel, so it is difficult to efficiently converge the site and process management in the conventional method.

On the other hand, in the case of a system in which 3D VR content produced according to an embodiment of the present invention is introduced into process management (hereinafter simply referred to as a 3D VR process management system), it is intuitive through fusion of field and process management. By providing integrated process information, it is possible to provide effective communication of schedule management and on-site management, and a complex management system based on demonstration of various participants, such as field workers and process managers, is formed to save schedule/quality/efficiency/time saving/ The effect of cost reduction can be obtained.

Such a 3D VR process management system may include a management function/business transfer function/communication tool provision function/data output and storage function.

Here, the management functions include: 1) site manager, site manager, subcontractor, construction personnel, manager, process manager, etc. for various related personnel such as job title and job authority setting function, 2) important part for process management, problems It may include marking functions for parts, 3) fixing a schedule or important information in 3D VR content, and executing a connection program through insertion of an MS project and a pre-medara process management file.

In addition, the business delivery function includes: 1) setting a specific area inside the 3D VR content (e.g., support for shapes such as colored boxes, circles, or line setting), 2) alarm function (e.g., process Set a specific date and time required for management, and when the event arrives, click the hotspot by setting the drawing file (DWG file, 3DS file, DXF file, etc.) as a content hotspot through calendar interlocking, etc. When a drawing list or folder appears, and clicking a drawing icon or file name therein, it may include a function of displaying a drawing file (or showing a drawing PDF file as an alternative or executing an external program viewer).

Also, as a communication tool providing function, 1) the ability to input information such as schedules, work orders, and confirmation items into 3D VR content, and 2) a meeting function where reporters and participants share a 360-degree VR content screen to meet together. This can be included.

In addition, the data output and storage functions are: 1) Information security function (for example, only the general manager can delete business report, delivery and input information, and store the deletion record and the date and time of each content in DB, etc.), 2 ) Process history management function (for example, providing DB for the whole such as business report, delivery and input information, VR meeting date and time, etc.), 3) management report output function (for example, 360-degree content thumbnail image) , Print, PDF, Excel storage, etc. including work contents, input information, etc.) 4) VR minutes function (e.g., a PDF of a 360-degree content thumbnail containing information and meeting topics, meeting date and time, and attendee information through a chat room, Excel storage, etc.), process report storage function (e.g., site information, participant name, work schedule, input information, process history, etc., save and print scenes of the corresponding 3D VR content, etc.) Can be included.

Although described above with reference to the embodiments of the present invention, those skilled in the art variously modify the present invention without departing from the spirit and scope of the present invention as set forth in the claims below. And it can be easily understood.

Claims

3D VR (3-dimension Virtual Reality) content production system,

A GUI (Graphical User Interface) unit for providing a user with a software tool to assist in inserting sensational content into the pre-made 360-degree panoramic VR, and receiving user input;

A sensory content application unit that applies the user-designated sensory content to the 360-degree panoramic panorama VR through the GUI unit;

Interactive action processing unit that controls the execution of sensational content applied in the 360-degree stereoscopic panorama VR in response to a user action

3D VR content production system comprising a.
According to claim 1,

The GUI unit provides a user with a tool screen for applying stereoscopic sound as the sensory content,

On the tool screen, the horizontal and vertical positions to which the stereoscopic sound is to be inserted in the 360-degree stereoscopic panorama VR, the sound output to which the stereoscopic sound is to be output, the direction or angle, the audio volume of the stereoscopic sound, An operation window is provided that enables designation of an audio reproduction speed of the stereoscopic sound, designation of whether to reproduce the stereoscopic sound repeatedly, and a reference proximity distance to which the stereoscopic sound is to be output.

The sensory content application unit includes a stereoscopic sound application unit that inserts the stereoscopic sound into the 360-degree panoramic panorama VR in correspondence with a designation specified by a user through the manipulation window.
According to claim 1,

The GUI unit provides a user with a tool screen for applying video content as the sensory content,

On the tool screen, the horizontal and vertical positions to which the video content is to be inserted, the position of the height axis, the size of the image to be inserted, and the modification of the shape of the outer border of the image to be inserted are specified in the 360-degree panorama VR. Includes an operation window to enable,

The sensory content application unit includes a video overlay processing unit for overlaying and inserting the video content into the 360-degree panoramic panorama VR in correspondence with a designation specified by a user through the manipulation window. VR content production system.
According to claim 3,

The image overlay processing unit,

The feature information extraction algorithm is used to extract the feature information of the target image to be inserted as the sensory content, and the feature information corresponding to the outlier is removed from the extracted feature information using the RANSAC (Random Sample Consensus) algorithm. In a manner of obtaining a homography matrix for extracting a corresponding point between images to be overlaid through repetition of the regression calculation, the target image is overlaid and merged to a specific coordinate region in the 360-dimensional stereoscopic panorama VR,

Before applying the RANSAC algorithm, it is determined whether or not the difference between the RGB value around the coordinate region to which the target image will be overlaid within the 360-degree stereoscopic VR and the RGB value of the target image is within a predetermined threshold, and 3D VR content production system, characterized by performing an intermediate filtering process to remove the corresponding points where the difference exceeds the threshold.
According to claim 4,

Further comprising a view point analysis unit for tracking and matching the viewpoint between the user's point of view and tracking the user's point of view by tracking the view point currently viewed by the user,

The interactive action processor displays a color according to a predetermined saturation spectrum according to the length of the gaze time, on the screen position of the coordinate pixel in the 360-degree stereoscopic panorama VR corresponding to the analyzed view point, 3D VR Content production system.
The method of claim 5,

The interactive action processing unit,

If the sensory content is applied to the coordinate region within the 360-degree stereoscopic panorama VR corresponding to the analyzed view point, and the gaze retention time at the analyzed view point satisfies a predetermined criterion, the sensory content applied to the corresponding coordinate area is displayed. 3D VR content production system, characterized in that, when the coordinate area according to the analyzed view point is changed differently from the previous one, the running sensory content is stopped.
The method of claim 6,

The interactive action processing unit,

Execution of sensational content applied to the coordinate area corresponding to the analyzed view point, moving the place to the space related to the coordinate area, playing the overlayed video, outputting stereoscopic sound, connecting or linking to a specified external link, or augmenting A 3D VR content production system that executes at least one interactive action of displaying information, making a call to a designated phone number, purchasing goods, and making payments.