KR102561903B1 - AI-based XR content service method using cloud server - Google Patents

Abstract

The present invention relates to an XR content service method, which uses an augmented reality (AR) remote rendering method capable of real-time mixed reality service of volumetric 3D video data, to provide a AI-based XR content service using a cloud server that can save a lot of cost and time caused by manual work in the production process, and is very easy to expand (utilize) the service through technical results in various cultural and artistic fields. provides a way

Description

AI-based XR content service method using cloud server

The present invention relates to an XR content service method, and more particularly, to an AI-based XR content service method using a cloud server capable of quickly and efficiently producing 3D content and providing high realism, realism and interaction.

Recently, as the mixed reality industry has grown, related device supply and technology have been proposed. In particular, for real-time mixed reality services such as remote video conferencing, real-time image acquisition-processing-transmission technology research is required.

AR (Augmented Reality) is a technology that leads new services through infinite expansion of the five senses by combining with technologies that lead the 4th industrial revolution, such as AI, Big-data, and 5G Network. As the development of 5G network infrastructure makes it possible to provide high-speed, ultra-low latency, and hyper-connected AR services, large-scale growth is expected centering on personalized real-time services.

In general, mixed reality implementation is a method in which 3D data is stored in a client and loaded. In this case, since the specifications of the client are limited, it is implemented through mesh optimization of 3D data. However, since optimization of the content is required, the quality of the content may be compromised. As another method, a method of storing 3D data in a server and rendering it to a client in a remote location has been proposed. This can greatly increase the rendering quality when a powerful server is used to increase the quality. Generally, it is expressed in terms such as cloud rendering, server rendering, and remote rendering.

Recently, cloud-based AR remote rendering technology is being researched. This is a new mixed reality service technology that can render high-quality, interactive 3D content in the cloud and stream it to devices in real time. In this case, remote rendering methods according to 3D data types may be classified as follows.

Model-based rendering is a method of transmitting 3D data (mesh or point cloud) to a client. In this method, the client needs powerful hardware for 3D graphics rendering, and the server does 3D data processing and generation calculations. Model-based rendering includes an original model, a partial model, a simplified model, and a point cloud method.

Image-based rendering is a method in which all 3D models are rendered on the server and only image information is sent to the client. Therefore, hardware for 3D graphic rendering is not required in the client. Image-based rendering includes multiple depth images, depth images, environment maps, and image impostors.

As described above, model-based rendering is suitable for providing high-quality content, but has a disadvantage in that a client requires hardware for powerful 3D graphic rendering. However, if the content is provided through image-based rendering in consideration of the hardware of the client, there is a problem of deterioration in quality. That is, a method capable of satisfying both the hardware performance of the client and the quality of content is required.

In addition, since the production process of conventional augmented reality (AR) or 3D content is performed manually, there is a problem that a lot of cost, manpower, and time are required, and the work result is less realistic and realistic, and the conventional virtual reality (VR) There is a problem that the technology has limitations in interaction.

The object of the present invention to solve the above problems is as follows.

First, the present invention is to provide an AI-based XR content service method using a cloud server capable of producing 3D content quickly and efficiently and providing high realism, realism, and interaction.

Second, an object of the present invention is to provide an AI-based XR content service method using a cloud server through AR remote rendering capable of real-time mixed reality service of volumetric 3D video data.

Third, by using 360℃ volumetric video capture technology and 3D pose estimation method to produce and service real-life content (AR/VR/MR) based on realism in the field of culture and arts, The task is to provide a service method that provides XR immersive content in the WebRTC mobile web browser environment through the construction of an augmented reality rendering system.

A first feature of the present invention for solving the above problems is an AI-based XR content service method using a cloud server, (a) generating a 3D model template of an object using a 3D volume modeling technology by a modeling unit; (b) After the modeling unit acquires pose estimation data of the object using deep learning technology and estimates the motion of the skeleton of the 3D model template to obtain rigging information, the 3D Generating model information and 3D model operation information and storing them in a 3D model database; (c) rendering 3D animation information by using the 3D model information and 3D model operation information stored in the 3D model database by a cloud server; and (d) providing XR contents by the cloud server by streaming the rendering data to a client terminal.

Here, the step (a) may include: (a1) generating 3D photogrammetry capture data for an object using a plurality of multi-view cameras; and (a2) generating a 3D volumetric model template of the object using the generated 3D photogrammetry capture data.

In addition, the step (b) may include: (b1) a modeling unit calculating pose estimation information of the object using deep learning technology; (b1) obtaining, by a modeling unit, temporal motion vectors between skeletons of the 3D model template and generating rigging information; and (b2) correcting the rigging information using the calculated pose estimation information, generating motion information of the 3D model template using the corrected rigging information, and storing it in the 3D model database. steps may be included.

In addition, the step (c) may include parsing and rendering redundant point data and added point data to reduce the processing amount of the 3D animation information; and rendering, by the cloud server, based on location information received from the client terminal.

In addition, the rendering may be a rendering of a web browser streaming method based on a webRTC API, and in the step (d), the client terminal converts the 3D object model rigged from the 3D template to augmented reality through the streaming data. It may be a step of applying and implementing in an (AR) environment and providing XR contents through a web browser.

And, the second feature of the present invention is installed in a computing terminal or server to enable computing or cloud computing, and is computer-readable for executing the AI-based XR content service method using any one of the cloud servers described above. It may be a computer program stored on a medium.

The effects of the present invention to solve the above problems are as follows.

First, the present invention uses an augmented reality (AR) remote rendering method capable of providing a real-time mixed reality service of volumetric 3D video data, so that manual work in the production process of conventional AR or 3D content can be achieved. It provides an AI-based XR content service method using a cloud server that can save a lot of cost and time, and is very easy to expand (usage) the service through results for technical use in various cultural and artistic fields.

Second, the present invention uses a pose estimation method and photogrammetry 3D scanning to capture 3D model data such as actual K-pop singers through a multi-view camera and extract them in the form of a 3D mesh Finally, it provides an AI-based XR content service method using a cloud server that can implement the target model (content) in a virtual and augmented reality environment that requires a 360-degree experience.

Third, the 3D volumetric technology of the present invention captures human motion using a number of special cinematographic cameras that implement high-resolution image quality and creates a 360-degree 3D stereoscopic image. It provides an AI-based XR content service method using a cloud server that can realize high realism and realism that can overcome the limits of interaction and realism of graphics-based AR technology.

Fourth, the photogrammetry technology applied to the embodiment of the present invention is a surveying technique that obtains reliable information by storing, measuring, and interpreting the shape of a subject using various input media such as a camera. It provides an AI-based XR content service method using a cloud server that can solve problems such as high cost, manpower, and post-processing time by manual design production of a precise design method drawn from modeling production through drawing. .

1 is a diagram showing the flow of an AI-based XR content service method using a cloud server according to an embodiment of the present invention.
2 is a conceptual diagram schematically illustrating a process of an AI-based XR content service method using a cloud server according to an embodiment of the present invention.
3 is a process connection schematic diagram of a block configuration of an AI-based XR content service method using a cloud server according to an embodiment of the present invention.
4 is a conceptual schematic diagram of a 3D photogrammetry capture system in the step of generating a 3D model template in the AI-based XR content service method using a cloud server according to an embodiment of the present invention.
5 is a conceptual schematic diagram of a high-precision 3D object creation algorithm applied to an AI-based XR content service method using a cloud server according to an embodiment of the present invention.
6 is a schematic diagram of a high-precision 3D pose estimation algorithm applied to an AI-based XR content service method using a cloud server according to an embodiment of the present invention.
7 is a conceptual schematic diagram of a joint correction algorithm applied to an AI-based XR content service method using a cloud server according to an embodiment of the present invention.
8 is a conceptual schematic diagram of a rendering process applied to an AI-based XR content service method using a cloud server according to an embodiment of the present invention.
9 is a conceptual schematic diagram of an application service in which an AI-based XR content service method using a cloud server according to an embodiment of the present invention is executed.

Hereinafter, embodiments of the present invention will be described in detail so that those skilled in the art can easily practice the present invention with reference to the accompanying drawings. However, the present invention may be embodied in many different forms and is not limited to the embodiments described herein. And in order to clearly explain the present invention in the drawings, parts irrelevant to the description are omitted, and similar reference numerals are attached to similar parts throughout the specification.

Throughout the specification, when a part is said to be "connected" to another part, this includes not only the case where it is "directly connected" but also the case where it is "electrically connected" with another element interposed therebetween. . In addition, when a part "includes" a certain component, this means that it may further include other components, not excluding other components, unless otherwise stated, and one or more other characteristics. However, it should be understood that it does not preclude the possibility of existence or addition of numbers, steps, operations, components, parts, or combinations thereof.

As used throughout the specification, the terms "about", "substantially", etc., are used at or approximating that value when manufacturing and material tolerances inherent in the stated meaning are given, and do not convey an understanding of the present invention. Accurate or absolute figures are used to help prevent exploitation by unscrupulous infringers of the disclosed disclosure. The term "step of (doing)" or "step of" as used throughout the specification of the present invention does not mean "step for".

In this specification, a "unit" includes a unit realized by hardware, a unit realized by software, and a unit realized using both. Further, one unit may be realized using two or more hardware, and two or more units may be realized by one hardware.

In this specification, some of the operations or functions described as being performed by a terminal, device, or device may be performed instead by a server connected to the terminal, device, or device. Likewise, some of the operations or functions described as being performed by the server may also be performed by a terminal, apparatus, or device connected to the server.

Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the drawings.

1 is a diagram showing the flow of an AI-based XR content service method using a cloud server 200 according to an embodiment of the present invention.

As shown in FIG. 1, in the AI-based XR service method using the cloud server 200 according to an embodiment of the present invention, (a) the modeling unit 100 uses 3D volumetric modeling technology to Generating a 3D model template (S100); (b) The modeling unit 100 obtains pose estimation data of the object using deep learning technology, and obtains rigging information by estimating the movement of a skeleton of the 3D model template Then, generating 3D model information and 3D model operation information and storing them in a 3D model database (S200); (c) rendering 3D animation information by using the 3D model information and 3D model operation information stored in the 3D model database by the cloud server 200 (S300); and (d) the cloud server 200 streaming the rendering data to the client terminal 300 to provide XR content (S400).

Here, looking at the configuration of the system for applying the AI-based XR content service method using the cloud server 200 according to the embodiment of the present invention, the modeling unit 100 for generating 3D modeling and animation data, and the data It may be configured to include a cloud server 200 that renders and streams, and a client terminal 300 that provides XR content, and each configuration may be connected wired or wirelessly through a network including the Internet.

The modeling unit 100 and the cloud server 200 have wired/wireless communication devices and may be computing devices such as computers and laptops that perform computing or arithmetic functions, and the client terminal 300 includes a wireless communication device. It may be a mobile device, such as a smart phone, or a computing device capable of connecting to the Internet through a wired connection.

As such, the present invention uses an augmented reality (AR) remote rendering method capable of providing real-time mixed reality service of volumetric 3D video data, resulting in manual work in the production process of conventional AR or 3D contents Provides an AI-based XR content service method using a cloud server (200) that can save a lot of cost and time to do, and is very easy to expand (utilize) services through results for technical use in various cultural and artistic fields. .

FIG. 2 shows a schematic diagram illustrating a process of an AI-based XR contents service method using a cloud server 200 according to an embodiment of the present invention, and FIG. 3 shows a cloud server 200 according to an embodiment of the present invention. It shows a schematic diagram of the process connection of the block configuration of the AI-based XR content service method using

As shown in FIGS. 2 and 3 , the AI-based XR content service method using the cloud server 200 according to an embodiment of the present invention first captures 3D photogrammetry or captures volumetric 4D Photogrammetry Using the system, objects such as the body, face, and clothing/props of a meta-human are photographed in 360-degree directions using dozens of multi-view cameras to acquire capture image data, and a live-action 3D volume model After creating a live-action 3D metahuman template, the live-action 3D metahuman template is stored in a live-action 3D model database.

Here, the 3D volumetric technology applied to the embodiment of the present invention is a technology that captures human motion using a plurality of special cinematography cameras that implement high-resolution image quality and creates a 360-degree 3D stereoscopic image. It is a volumetric video technology based on live action to go beyond the realism and realism of the movie. In addition, it is a technology capable of realizing high realism and realism that can overcome the limits of interaction of existing VR technology and the limits of realism of graphics-based AR technology.

In addition, the photogrammetry technology applied to the embodiment of the present invention is a surveying technique that obtains reliable information by storing, measuring, and analyzing the shape of a subject using various input media such as a camera. It is a technology that can solve problems such as high cost, manpower, and post-processing time by manual design production in the method of accurate design drawn through drawing in modeling production.

In addition, the modeling unit 100 recognizes and estimates the pose of a metahuman model such as a person using 3D pose estimation technology for the animation sequence of the generated 3D volume model, and rigging the 3D model. ) information and correction information are generated, and 3D spatial operation information of the 3D model is obtained and stored in a real-life 3D model database. Here, rigging means the work of making the skeleton (Skeleton0) of the model as a basic work of modeling.

The cloud server 200 uses the location information received from the client terminal 300 to render the 3D animation information using the 3D model and 3D motion information stored in the real-life 3D model database in the WebRTC method, and streams it to the mobile client to provide XR provide content.

Hereinafter, the AI-based XR content service method using the cloud server 200 according to an embodiment of the present invention will be described in detail with reference to drawings for each step process.

4 is a conceptual schematic diagram of a 3D photogrammetry capture system in the step of generating a 3D model template in the AI-based XR content service method using the cloud server 200 according to an embodiment of the present invention.

As shown in FIG. 4, the 3D photogrammetry capture system installs a plurality of multi-view movie cameras on a chroma key screen background, and captures 3D objects at high precision (average precision of 15 mm or less) at 30 frames per second. and synthesize a 3D model using a volumetric synthesis program. The 3D volumetric method for object models like this can be optimized for video production, suppresses the occurrence of shadows or saturated areas in the chroma key part of the shooting area, and controls the lighting environment to suppress the flicker phenomenon caused by the reflection of the subject. can include

Such a 3D photogrammetry capture system can adjust the angle, aperture, shutter speed, and ISO of the camera according to the content, lighting, and shooting space setting of the content performance, and the brightness, noise, and motion blur of the input image. And it is possible to control the focus according to the depth, and it is possible to optimize the result of the performer through the pre-processing algorithm of the multi-view image.

In addition, a 3D volumetric model can be created using image data of a 3D object captured using the 3D photogrammetry capture system illustrated in FIG. High-precision 3D volume model data at the level of film production can be generated through this process, and SfM (Structure from Motion) method video synthesis technology is optimized for the camera system and lighting environment that are set up in-house, so that high-quality 3D model restoration and refinement technology can be restored. It can be applied, and a parallel processing system can be applied to reduce the 3D object synthesis time.

5 is a conceptual schematic diagram of a high-precision 3D object creation algorithm applied to the AI-based XR content service method using the cloud server 200 according to an embodiment of the present invention.

In the embodiment of the present invention, it is possible to apply an algorithm that precisely makes the 3D volume generation result of volumetric video. As an advanced method of 3D volume synthesis technology, it is possible to use images twice the number of cameras installed in the studio using special spatio-temporal filters. Thus, a technique for obtaining a high-precision 3D volume can be applied.

That is, as shown in FIG. 5, 40 2D images of the current frame and 40 images of the previous frame captured using the above-described 3D photogrammetry capture system are subjected to a temporal-spatial filter. When synthesized using , the result of the synthesized 3D volumetric model can be obtained. In addition, the quality of the result can be improved by analyzing and controlling the 'camera position estimation' algorithm during the processing stage of 3D volume synthesis.

6 shows a schematic diagram of a high-precision 3D pose estimation algorithm applied to an AI-based XR content service method using a cloud server 200 according to an embodiment of the present invention, and FIG. 7 shows a cloud server according to an embodiment of the present invention. It shows a conceptual diagram of the joint correction algorithm applied to the AI-based XR contents service method using (200).

As shown in FIG. 6, the algorithm for obtaining pose estimation data of the human object extracts the multi-view whole body skeleton of the human object from the volumetric model template obtained in step (a), integrates and corrects the skeleton , apply high-precision skeleton capture technology that proceeds with the process of bone analysis and face and hand position discrimination. Then, through multi-view high-definition video, high-resolution images of the face and hands are extracted, hand skeletons are extracted, and skeletons and features are integrated. Pose estimation data of a human object performing an operation may be obtained. Here, motion estimation of the human object can be performed through AI deep learning, and high-precision motion capture results can be obtained through multi-view motion analysis.

In other words, after extracting the multi-view whole body skeleton, several skeletons can be integrated into one skeleton, and correction can be made using the 3D volume model information generated at the same time. After detecting and extracting high-precision images from the same area of the UHD image to extract the geometrical information of the face and hands, integrated motion capture results can be created by integrating them with the full body skeleton.

As shown in FIG. 7, the joint correction algorithm for motion capture correction using 3D volume and human body information is a technology for correcting the position of nodes using point cloud information in a 3D space, considering the 3D model of the extracted skeleton. In point cloud clustering, an algorithm that can correct the position of nodes using point cloud information in a 3D space can be applied. That is, an algorithm capable of correcting an error of the above-described deep learning model for skeleton extraction and signal processing technology using point cloud information of a 3D model can be applied.

In step (b), pose estimation information of the 3D object is calculated, and the modeling unit 100 obtains a temporal motion vector between skeletons of the 3D model template and generates rigging information. and correcting the rigging information using the calculated motion estimation information.

Here, in the step of generating rigging information, a motion vector may be obtained through temporal motion estimation between the extracted bones, and a rigging information analysis technique considering animation based on motion vector analysis may be applied. . In addition, it is possible to apply techniques for compensating and improving error areas through high-precision animation of 3D templates and comparative analysis of 3D volumetric model template sequences.

8 is a conceptual schematic diagram of a rendering process applied to the AI-based XR content service method using the cloud server 200 according to an embodiment of the present invention.

In step (c) of the AI-based XR content service method using the cloud server 200 according to an embodiment of the present invention, the cloud server 200 uses the 3D model information and 3D model operation information stored in the 3D model database. As a step of rendering 3D animation information by using, parsing and rendering redundant point data and added point data to reduce the processing amount of 3D animation information, and the cloud server 200 is the client terminal ( It may be a step of rendering based on the location information received from 300).

That is, as shown in FIG. 8, the modeling unit 100 generates a 3D volume model template through photogrammetry scan as step (a), and pose estimation as step (b) After generating 3D model information and 3D model motion information through , 3D animation information is generated through this. Then, in step (c), the cloud server 200 may render the above-described 3D animation information in a WebRTC method using location information received from the client terminal 300 .

The rendering method of the cloud server 200 applied to the embodiment of the present invention is a WebRTC API-based web browser streaming rendering method, which distinguishes and parses duplicated point data and added point data to reduce data throughput, , The mobile client performs location tracking using a camera, receives the location from the cloud server 200 and renders it, and in step (d), the cloud server 200 transmits the rendered scene information to the client terminal 300. Streaming will provide XR content.

Here, the Web Real-Time Communication (WebRTC) API is an API designed to allow web browsers to communicate with each other without the help of a plug-in, and a peer to peer (P2P) channel capable of sending and receiving arbitrary data can be used. In addition, the cloud server 200 is an AR rendering server, and renders based on browser execution and mobile client location information, and the server canvas is captured as a video through a capture API and streamed to the mobile client. 3D rendered video from the server will be displayed to provide XR content.

9 is a conceptual schematic diagram of an application service in which an AI-based XR content service method using the cloud server 200 according to an embodiment of the present invention is executed.

As shown in FIG. 9, an application executing an AI-based XR content service method using the cloud server 200 according to an embodiment of the present invention applies augmented reality (AR) of a 3D model and is an application in a web browser environment. , A 3D model rigged from a 3D template can be applied as a 3D object in the AR environment, space scanning & anchor point can be specified, and scene information received from the virtual camera of the rendering server can be implemented. You can deliver XR content by streaming.

In addition, web-based AR services can be provided by utilizing the Open XR SDK, which is a standard for access between AR platforms and devices, and cross-platform Open XR standards-based XR content provision services can be implemented. Not only can an augmented reality web (mobile browser)-based service with authority be implemented, but also a service in which contents are provided through a web browser can be implemented.

The AI-based XR content service method using the cloud server 200 according to the above-described embodiment of the present invention is an application basically installed in a terminal (this may include a program included in a platform or operating system basically installed in the terminal). It can be executed by an application (i.e., a program) directly installed in the master terminal through an application providing server such as an application store server, an application, or a web server related to the corresponding service. In this sense, the AI-based XR contents service method using the cloud server 200 according to an embodiment of the present invention described above is implemented as an application (ie, a program) that is basically installed in a terminal or directly installed by a user, and is implemented as a terminal. It can be recorded on a computer-readable recording medium such as E.

The above description of the present invention is for illustrative purposes, and those skilled in the art can understand that it can be easily modified into other specific forms without changing the technical spirit or essential features of the present invention. will be. Therefore, the embodiments described above should be understood as illustrative in all respects and not limiting. For example, each component described as a single type may be implemented in a distributed manner, and similarly, components described as distributed may be implemented in a combined form.

The scope of the present invention is indicated by the following claims rather than the detailed description above, and all changes or modifications derived from the meaning and scope of the claims and equivalent concepts should be construed as being included in the scope of the present invention. do.

100: modeling unit
200: cloud server
300: client terminal

Claims (6)

(a) generating a 3D model template of an object by a modeling unit using 3D volume modeling technology;
(b) After the modeling unit acquires pose estimation data of the object using deep learning technology and estimates the motion of the skeleton of the 3D model template to obtain rigging information, the 3D Generating model information and 3D model operation information and storing them in a 3D model database;
(c) rendering 3D animation information by using the 3D model information and 3D model operation information stored in the 3D model database by a cloud server; and
(d) the cloud server providing XR content by streaming the rendering data to a client terminal,
In step (a),
(a1) generating 3D photogrammetry capture data for an object using a plurality of multi-view cameras; and (a2) generating a 3D volumetric model template of the object using the generated 3D photogrammetry capture data,
In step (c),
dividing, parsing, and rendering redundant point data and added point data to reduce the throughput of the 3D animation information; and rendering, by the cloud server, based on the location information received from the client terminal. delete According to claim 1,
In step (b),
(b1) calculating pose estimation (Pose Estimation) information of the object by a modeling unit using deep learning technology;
(b1) obtaining, by a modeling unit, temporal motion vectors between skeletons of the 3D model template and generating rigging information; and
(b2) correcting the rigging information using the calculated pose estimation information, generating motion information of the 3D model template using the corrected rigging information, and storing it in the 3D model database AI-based XR content service method using a cloud server, comprising the steps of: delete According to claim 1,
The rendering is
AI-based XR content service method using a cloud server, characterized in that the webRTC API-based web browser streaming method rendering. According to claim 1,
In step (d),
Cloud server, characterized in that the step of applying and implementing the 3D object model rigged in the 3D template through the streaming data in the client terminal in an augmented reality (AR) environment and providing XR content through a web browser AI-based XR content service method using