KR102346090B1 - Ar remote randering method for real time mr service with volumetric 3d video data - Google Patents

Abstract

The present specification discloses an AR remote rendering method capable of real-time mixed reality service of volumetric 3D video data. The augmented reality remote rendering method according to the present specification is an augmented reality remote rendering method for a real-time mixed reality service of volumetric 3D video data, comprising the steps of: (a) receiving, by a processor, depth image data from a plurality of depth cameras; (b) generating, by the processor, 3D point cloud data using the received depth image data; (c) the processor modeling 3D data in a virtual space; and (d) transmitting, by the processor, an image of the object viewed from the virtual camera to the client using the client location information.

Description

AR REMOTE RANDERING METHOD FOR REAL TIME MR SERVICE WITH VOLUMETRIC 3D VIDEO DATA}

The present invention relates to an augmented reality remote rendering method, and more particularly, to an augmented reality remote rendering method for real-time mixed reality service of volumetric 3D video data.

Recently, as the mixed reality industry grows, related devices and technologies are being proposed. In particular, for real-time mixed reality services such as remote video conferencing, real-time image acquisition-processing-transmission technology research is required.

AR (Augmented Reality) is a technology that leads new services through infinite expansion of the five senses by combining with technologies that lead the 4th industrial revolution such as AI, Big-data, and 5G Network. With the development of 5G network infrastructure, it is possible to provide high-speed, ultra-low latency, and hyper-connected AR services, and large-scale growth is expected centering on personalized real-time services.

In general, mixed reality implementation is a method in which 3D data is stored in the client and loaded. In this case, since the specification of the client is limited, it is implemented through mesh optimization of 3D data, etc. However, since content optimization is required, the quality of the content may be damaged. As another method, a method in which 3D data is stored in a server and rendered to a client in a remote location has been proposed. If a powerful server is used to increase the quality, the rendering quality can be greatly improved. In general, it is expressed in terms such as cloud rendering, server rendering, and remote rendering.

Recently, cloud-based AR remote rendering technology is being studied. This is a new mixed reality service technology that can render high-quality interactive 3D content in the cloud and stream it to devices in real time. In this case, a remote rendering method according to a 3D data type may be classified as follows.

Model-based rendering is a method of sending 3D data (mesh or point cloud) to the client. In this method, the client needs powerful hardware for 3D graphics rendering, and the server does the 3D data processing and generating calculations. Model-based rendering includes an original model, a partial model, a simplified model, and a point cloud method.

Image-based rendering is a method of rendering all 3D models in the server and sending only image information to the client. Therefore, hardware for rendering 3D graphics is not required on the client. Image-based rendering includes multiple depth images, depth images, environment maps, and image posters.

As mentioned above, model-based rendering is suitable to provide high-quality content, but there is a disadvantage in that hardware for powerful 3D graphics rendering is required for the client. However, there is a problem in that when the content is provided by image-based rendering in consideration of the client's hardware, the quality is deteriorated. That is, there is a need for a method that can satisfy both the hardware performance of the client and the quality of the content.

Shu Shi, Klara Nahrstedt, and Roy Campbell, "A Real-Time Remote Rendering System for Interactive Mobile Graphics," ACM Transactions on Multimedia Computing, Communications and Applications, Vol. 8, No. 3s, Article 46, 2012. DOI: 10.1145/2348816.2348825.

An object of the present specification is to provide an AR remote rendering method capable of real-time mixed reality service of volumetric 3D video data.

The present specification is not limited to the above-mentioned problems, and other problems not mentioned will be clearly understood by those skilled in the art from the following description.

The augmented reality remote rendering method according to the present specification for solving the above-described problems is an augmented reality remote rendering method for a real-time mixed reality service of volumetric 3D video data, wherein (a) a processor receives depth image data from a plurality of depth cameras to do; (b) generating, by the processor, 3D data using the received depth image data; (c) the processor modeling 3D data in a virtual space; and (d) transmitting, by the processor, an image of the object viewed from the virtual camera to the client using the client location information.

According to one embodiment of the present specification, before step (a), the step of the processor generating coordinate information (hereinafter, 'camera coordinate information') for the plurality of depth cameras; may further include.

According to an embodiment of the present specification, the camera coordinate information may be generated by using information about a movement position and a rotation position from a central position of a photographing target.

According to an embodiment of the present specification, the rotation position may be calculated based on one reference depth camera among a plurality of depth cameras.

According to an embodiment of the present specification, the camera coordinate information may be calculated using a charuco board.

According to an embodiment of the present specification, the plurality of depth cameras may be connected to a synchronization signal generator.

According to an embodiment of the present specification, the three-dimensional point cloud is composed of a plurality of point data consisting of x, y, and z coordinate values and R, G, and B color values, and each point data may be binary data. have.

According to an embodiment of the present specification, the step (b) may further include a frame generating step in which the processor generates a frame composed of 3 o data according to a preset size and period.

According to an embodiment of the present specification, in the frame generating step, the processor generates point data and added point data that are duplicated with the previous frame, and separates and parses the duplicated point data and added point data. ) can be a step.

According to an embodiment of the present specification, the client location information includes location information and rotation information of the client terminal, and the location information of the client terminal is a value generated using a SLAM method, The rotation information may be a value generated using an acceleration sensor and a gyroscope sensor.

According to an embodiment of the present specification, the object image viewed from the virtual camera may be a 2D image.

According to an embodiment of the present specification, step (d) may be a step in which the processor transmits the object image to the client using WebRTC.

Other specific details of the invention are included in the detailed description and drawings.

According to the present specification, a real-time mixed reality service made of volumetric 3D video data is possible.

Effects of the present invention are not limited to the effects mentioned above, and other effects not mentioned will be clearly understood by those skilled in the art from the following description.

1 is a schematic flowchart of an augmented reality remote rendering method according to the present specification.
2 is a reference diagram for a process of generating 3D data using a plurality of depth cameras.
3 is a reference diagram of position matching using a charuco board.
4 is an example of a frame generated according to an embodiment of the present specification.
5 is a reference diagram expressed as a block diagram of an augmented reality remote rendering method according to the present specification.
6 is an experimental result according to the present specification.

Advantages and features of the invention disclosed herein, and methods of achieving them will become apparent with reference to the embodiments described below in detail in conjunction with the accompanying drawings. However, the present specification is not limited to the embodiments disclosed below, but may be implemented in various different forms, and only the present embodiments allow the disclosure of the present specification to be complete, and those of ordinary skill in the art to which this specification belongs. It is provided to fully inform those skilled in the art (hereinafter, 'those skilled in the art') the scope of the present specification, and the scope of the present specification is only defined by the scope of the claims.

The terminology used herein is for the purpose of describing the embodiments and is not intended to limit the scope of the present specification. In this specification, the singular also includes the plural unless specifically stated otherwise in the phrase. As used herein, “comprises” and/or “comprising” does not exclude the presence or addition of one or more other components in addition to the stated components.

Unless otherwise defined, all terms (including technical and scientific terms) used herein will have the meaning commonly understood by those of ordinary skill in the art to which this specification belongs. In addition, terms defined in a commonly used dictionary are not to be interpreted ideally or excessively unless specifically defined explicitly. Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings.

1 is a schematic flowchart of an augmented reality remote rendering method according to the present specification.

The augmented reality remote rendering method according to the present specification is an augmented reality remote rendering method for real-time mixed reality service of volumetric 3D video data.

First, in step S10, the processor may receive depth image data from a plurality of depth cameras.

The plurality of depth cameras is a volumetric 3D video camera for capturing a 3D space. The depth camera is a camera capable of further outputting depth data in addition to general RGB data. The depth camera is generally called a 'RBGD camera', and includes a stereo method, a Time-Of-Flight (ToF) method, a structured pattern method, and the like. The plurality of depth cameras according to the present specification may be arranged in a circle around the object so as to photograph the object in three dimensions.

In the next step S20, the processor may generate 3D data using the received depth image data.

2 is a reference diagram for a process of generating 3D data using a plurality of depth cameras.

Referring to FIG. 2 , each depth camera may output image data (including depth information) captured by the depth camera. In this case, since the images captured by each depth camera capture the same object, a portion in which the images overlap each other may occur. Therefore, a process of position matching is required in consideration of a position and a shooting angle of each depth camera.

To this end, the augmented reality remote rendering method according to the present specification may further include the step of generating, by the processor, coordinate information (hereinafter, 'camera coordinate information') for the plurality of depth cameras before step S10.

3 is a reference diagram of position matching using a charuco board.

Referring to FIG. 3 , it can be confirmed that a plurality of RGBD cameras photograph the same charuco board. The processor extracts the corners and codes of the board from the images captured by each RGBD camera. Since the plurality of RGBD cameras photographed the same board at each other's positions in a three-dimensional space, the relative positions of the RGBD cameras can be calculated by comparing the characteristics of the boards in the respective images. For example, the camera coordinate information may be generated by using information about a movement position and a rotation position from a central position of a photographing target. In this case, the rotation position may be calculated as follows based on any one reference depth camera among the plurality of depth cameras.

In the above formula, [u, v, 1]T is a 2D image coordinate value, [x _w , y _w , z _w , 1]T is a 3D real space coordinate value, K is a camera internal parameter, and [R, T] is a camera As an external parameter, R is a value for rotation and T is a value for movement. On the other hand, it is assumed that the plurality of depth cameras are cameras having the same specification, and the internal parameters are the same. In addition, in order to synchronize images captured by the plurality of depth cameras, the plurality of depth cameras may be connected to a synchronization signal generator. In a system using two or more cameras, it synchronizes the Slave Sync Generator with the Master Sync Generator. It can be connected to a signal generator.

Meanwhile, the 3D point cloud data generated by the processor may be composed of a plurality of point data composed of x, y, and z coordinate values and R, G, and B color values. And each point data may be processed as binary data.

The augmented reality remote rendering method according to the present specification may further include, in step S20, a frame generation step in which the processor generates a frame composed of 3D data according to a preset size and period.

4 is an example of a frame generated according to an embodiment of the present specification.

Referring to FIG. 4A , frame 1, frame 25, and frame 55 generated as 3D data can be identified. In the augmented reality remote rendering method according to the present specification, in order to reduce data throughput in the process of continuously generating frames, the processor generates point data duplicated with a previous frame and added point data, and the duplicate point data and The added point data can be divided and parsed.

By comparing the previous frame data and the current frame data continuously generated as 3D data, it can be divided into overlap data and add data.

For example, x, y, and z may each have a vertex value expressed by 8 digits, and R, G, and B may each have a color value expressed by 3 digits. And each point data can be expressed as a single value as shown in the following equation.

P _feature becomes a unique feature value, and a set of P _features means a point cloud constituting one frame. Therefore, it can be divided into overlap data and add data through comparison as shown in the formula below.

_{P f} in Equation 3 is a set of n feature values of the f-th frame. P' _f is any element value of _{P f .} If the operation result (difference value) of P' _{f in P f-1 is 0, it is overlapping data.} Conversely, if the operation result (difference value) is not 0, it is Add data.

Referring back to FIG. 1 , in step S30, the processor may model 3D data in a virtual space. At this time, the virtual space in which the 3D data is modeled is a game engine.

In the next step S40, the processor may transmit the image of the object viewed from the virtual camera to the client using the client location information. In this case, the client location information may include location information and rotation information of the client terminal. The location information of the client terminal may be a value generated using the SLAM method, and the rotation information of the client terminal may be a value generated using an acceleration sensor and a gyroscope sensor.

Also, the object image viewed from the virtual camera is a 2D image. In addition, the processor may transmit the object image to the client using WebRTC. WebRTC is an open framework for the web that supports Real-Time Communications (RTC) functions in browsers. It supports to transmit video, voice and general data from peer to peer. It is also implemented as an open web standard and is available as a JavaScript API in all major browsers. Accordingly, high-end rendering hardware is not required for the client terminal.

5 is a reference diagram expressed as a block diagram of an augmented reality remote rendering method according to the present specification.

Figure 5 (a) is a block diagram expressing the overall flow of the augmented reality remote rendering method, Figure 5 (b) is a divider (Divider) and game engine (Game Engine) block included in the volume capture (Volumetric Capture) block It is a block diagram specifically showing the parsing module included in the block diagram, and (c) of FIG. 5 is the information transmission/reception between the virtual camera included in the game engine block and the client terminal. This is a specific block diagram. The divider module can perform data compression through data comparison operation.

<Experimental example>

The table below shows the experimental environment. The communication environment used Wi-Fi. Server specifications are as follows. The processor used was an Intel Core i7-8700 CPU @ 3.20GHz, and the GPU was NVIDIA GeForce RTX 2060. The memory used was 16 GB RAM. The 3D data used in the experiment is a ply file format with 13 MB - 14 MB. The client used a Samsung Galaxy Note 10+ (SM-N976 N) for the experiment in a mobile environment. The rendered video was implemented in Chrome Mobile Web Browser, and ‘Google DevTools’ was used as the experimental measurement tool in the mobile debugging environment.

<Experiment result>

6 is an experimental result according to the present specification.

Referring to FIG. 6A , it is an implementation result of the divider module shown in FIG. 5 . As a result of the experiment, it can be seen that an average of about 15 MB of original data is compressed by about 1-2 MB for each frame.

Referring to FIG. 6B , it is a streaming result received from a virtual camera using WebRTC. As a result of the experiment, it can be seen that the mobile web browser renders at a speed of about 25 -27 FPS per second.

Through this, it can be confirmed that the augmented reality remote rendering method is possible for real-time mixed reality service of volumetric 3D video data.

The processor may include an application-specific integrated circuit (ASIC), other chipsets, logic circuits, registers, communication modems, data processing devices, etc. known in the art to execute the various calculation and control logics described above. have. In addition, when the above-described control logic is implemented in software, the processor may be implemented as a set of program modules. In this case, the program module may be stored in the memory device and executed by the processor.

The computer program is C/C++, C#, JAVA, Python that a processor (CPU) of the computer can read through a device interface of the computer in order for the computer to read the program and execute the methods implemented as a program , may include code coded in a computer language such as machine language. Such code may include functional code related to a function defining functions necessary for executing the methods, etc., and includes an execution procedure related control code necessary for the processor of the computer to execute the functions according to a predetermined procedure. can do. In addition, this code may further include additional information necessary for the processor of the computer to execute the functions or code related to memory reference for which location (address address) in the internal or external memory of the computer should be referenced. have. In addition, when the processor of the computer needs to communicate with any other computer or server located remotely in order to execute the functions, the code uses the communication module of the computer to determine how to communicate with any other computer or server remotely. It may further include a communication-related code for whether to communicate and what information or media to transmit and receive during communication.

The storage medium is not a medium that stores data for a short moment, such as a register, a cache, a memory, etc., but a medium that stores data semi-permanently and can be read by a device. Specifically, examples of the storage medium include, but are not limited to, ROM, RAM, CD-ROM, magnetic tape, floppy disk, and an optical data storage device. That is, the program may be stored in various recording media on various servers accessible by the computer or in various recording media on the computer of the user. In addition, the medium may be distributed in a computer system connected to a network, and a computer-readable code may be stored in a distributed manner.

As mentioned above, although the embodiments of the present specification have been described with reference to the accompanying drawings, those skilled in the art to which this specification belongs can realize that the present invention may be embodied in other specific forms without changing the technical spirit or essential features thereof. you will be able to understand Therefore, it should be understood that the embodiments described above are illustrative in all respects and not restrictive.

Claims (12)

An augmented reality remote rendering method for real-time mixed reality service of volumetric 3D video data, comprising:
(a) receiving, by a processor, depth image data from a plurality of depth cameras;
(b) the processor using the received depth image data, x, y, and z coordinate values expressed by m digits, and R, G, and B color values expressed by n digits, respectively, are a plurality of binary data. generating point data, and generating a frame composed of 3D data according to a preset size and period;
(c) the processor modeling 3D data in a virtual space; and
(d) transmitting, by the processor, the image of the object viewed from the virtual camera to the client using the client location information;
Generating the frame in step (b) is,
The processor generates a unique feature value (P _feature ) through Equation A below for each point data,

[Formula A]

Through Equation B below, the point data overlapped with the previous frame and the added point data are generated,

[Formula B]

- P _f is the set of n feature values of the f th frame
- P' _f is any element value of P _f

When the difference value of P' _{f in P f-1} is 0, it is divided into the overlapped point data, and when the difference value of P' _{f in P f-1} is not 0, it is divided into added point data and parsed. Augmented reality remote rendering method. The method according to claim 1,
Augmented reality remote rendering method further comprising a; before the step (a), the processor generates coordinate information (hereinafter, 'camera coordinate information') for the plurality of depth cameras. 3. The method according to claim 2,
The camera coordinate information is augmented reality remote rendering method, characterized in that it is generated by using information about the movement position and rotation position from the central position of the object to be photographed. 4. The method according to claim 3,
The rotation position, augmented reality remote rendering method, characterized in that calculated based on any one of the plurality of depth cameras as a reference depth camera. 4. The method according to claim 3,
The camera coordinate information, augmented reality remote rendering method, characterized in that calculated using a charuco board. 3. The method according to claim 2,
The plurality of depth cameras are augmented reality remote rendering method, characterized in that connected to the synchronization signal generator. delete delete delete The method according to claim 1,
The client location information includes location information and rotation information of the client terminal,
The augmented reality remote rendering method, characterized in that the location information of the client terminal is a value generated using a SLAM method, and the location and rotation information of the client terminal is a value generated using an acceleration sensor and a gyroscope sensor. The method according to claim 1,
The augmented reality remote rendering method in which the object image viewed from the virtual camera is a 2D image. The method according to claim 1,
The step (d) is an augmented reality remote rendering method in which the processor transmits the object image to the client using WebRTC.

Images