KR102261739B1 - System and method for adaptive streaming of augmented reality media content - Google Patents

Abstract

The present invention provides an adaptive streaming system and an adaptive streaming method that can solve the limitation of storage space by receiving only minimum information of content, and then receiving and displaying the rendering result provided from the graphic processing server according to the changed viewing information in streaming As relates to, a user terminal that obtains viewing information by driving an augmented reality application, transmits it to the outside, receives a corresponding rendering result to display an augmented reality image, and Internet address (URL) information of a content providing server to the user terminal A service providing server that provides a service, a content providing server that provides minimum information of content to the user terminal, and analyzing the viewing information received from the user terminal to receive the corresponding content from the content providing server, and perform rendering according to the viewing information It includes a graphic processing server that provides the rendering result to the user terminal as 5G streaming, and adaptively streams the rendering result in response to the viewing information changed in real time, so high-capacity augmented reality content is not limited by the storage space of the user terminal. can exhibit the effect of being able to view

Description

The adaptive streaming system and method for adaptive streaming of augmented reality media content {System and method for adaptive streaming of augmented reality media content}

The present invention relates to an adaptive streaming system and an adaptive streaming method of augmented reality media content for improving the quality of an augmented reality streaming service, and more particularly, according to the changed viewing information after receiving only minimal information of the content. It relates to an adaptive streaming system and an adaptive streaming method capable of solving a storage space limitation by receiving and displaying a rendering result provided from a graphic processing server as streaming.

Efforts are being made to develop an improved 5G communication system to meet the increasing demand for wireless data traffic after the commercialization of the 4G communication system. In order to achieve a high data rate, the 5G communication system is being considered for implementation in a very high frequency band (eg, a 60 GHz band).

As operators have recently made large-scale investments in virtual reality services, they are expected to expand into next-generation representative services. In general, virtual reality refers to an environment or situation created by computer graphics having an environment similar to the real one, and refers to an interface that makes people feel through their sensory organs and makes it look like they are actually interacting. The user may interact with virtual reality in real time through manipulation of the device, and may experience a sensory experience similar to the real one.

Augmented reality is a field of virtual reality, and is a computer graphics technique that synthesizes virtual objects or information in a real environment to make them look like objects existing in the original environment. The augmented reality is a technology that superimposes a virtual object on the real world viewed by the user's eyes, and is also referred to as mixed reality (MR) because additional information and the virtual world are combined in real time to show the real world as one image.

In addition, as mobile devices (eg, smart phones, tablet PCs, etc.) become common, augmented reality technology can be easily accessed in various services such as education, games, navigation, advertisements, or blogs. Recently, as a wearable device that can be worn on the body is commercialized, research on augmented reality technology has become more active.

For example, the wearable device is provided in various forms detachable from the human body and clothes, such as a head mounted type, a glasses type, a watch type, a band type, a contact lens type, a ring type, a shoe type, and a clothing type. The wearable device may increase portability and accessibility by allowing the electronic device to be worn on the body, such as clothes or glasses.

On the other hand, the initial augmented reality service started as a non-streaming service that downloads all the dedicated content to the augmented reality device and then plays it. That is, in order to reproduce the existing augmented reality content, it is operated by downloading all elements such as assets and textures, which are constituent elements of the content, into the terminal. Therefore, in order to maintain the quality when approaching the arranged content or expanding the content, all of the high-quality textures provided according to the moving distance and the degree of magnification must be downloaded, so there are many restrictions on the storage space.

An object of the present invention is to provide a system and an adaptive streaming method for adaptively streaming augmented reality content in response to viewing information changed in real time with respect to augmented reality content being reproduced with minimum information.

Another object of the present invention is to provide an adaptive streaming system and an adaptive streaming method capable of viewing high-capacity augmented reality content without limiting a storage space of a user terminal for implementing augmented reality.

In order to achieve this object, the adaptive streaming system of augmented reality media content according to an embodiment of the present invention acquires viewing information for augmented reality content being reproduced with minimum information and transmits it to the outside, and content corresponding to the viewing information a user terminal for receiving the rendered result as full specification information and displaying the augmented reality content; and a graphic processing server that analyzes the viewing information received from the user terminal, renders the content according to the full specification information of the corresponding content, and provides a rendering result to the user terminal.

In the adaptive streaming system of augmented reality media content according to a preferred embodiment of the present invention, the viewing information includes: spatial coordinate information of a state in which the content is placed in the real space through floor recognition; Enlarging/reducing information indicating the degree of enlargement or reduction of content when the content is enlarged or reduced; And it may include at least one of information on the viewpoint viewed by the user.

In the system for adaptive streaming of augmented reality media content according to a preferred embodiment of the present invention, the minimum information of the content includes at least one of a content asset and basic information defined by default.

In the system for adaptive streaming of augmented reality media content according to a preferred embodiment of the present invention, the basic information includes at least one of a content resolution according to a size and an angle defined by default.

In the adaptive streaming system of augmented reality media content according to a preferred embodiment of the present invention, the content information provided from the content providing server to the graphic processing server includes a full spec 3D asset and a high quality texture package.

In the adaptive streaming system of augmented reality media content according to a preferred embodiment of the present invention, the full spec information of the content requested according to the viewing information from the graphic processing server is converted into a full spec 3D asset and a high quality texture package. It may further include a content providing server configured to transmit to the graphic processing server.

When the adaptive streaming system of augmented reality media content according to a preferred embodiment of the present invention receives the ID of the augmented reality content selected to be played by the user terminal, the Internet address (URL) of the content providing server mapped to the content ID Further comprising a service providing server that transmits to the user terminal, wherein the user terminal accesses the content providing server through the Internet address of the content providing server received from the service providing server, the minimum information of the augmented reality content selected to be played It is characterized in that the augmented reality content is played by receiving it.

In the adaptive streaming system of augmented reality media content according to a preferred embodiment of the present invention, the graphic processing server renders the content based on the basic information, and then performs modified rendering according to the viewing information received from the user terminal. characterized.

In the adaptive streaming system of augmented reality media content according to a preferred embodiment of the present invention, the graphic processing server may perform the modified rendering by utilizing the high-quality texture package according to enlarged information among viewing information received from the user terminal. .

A method for adaptive streaming of augmented reality media content according to an embodiment of the present invention includes the steps of: receiving, in a user terminal, minimum information of content from a content providing server, and playing the augmented reality content; generating viewing information in the user terminal and transmitting it to a graphic processing server; receiving, in the user terminal, a result of changing rendering according to full specification information of content corresponding to the viewing information from the graphic processing server; and displaying the augmented reality content using the rendering result received from the user terminal.

The system and adaptive streaming method for streaming augmented reality content according to the present invention adaptively stream a rendering result in response to viewing information that is changed in real time, so that a user can watch high-capacity augmented reality content without restriction of the storage space of the user terminal. possible effects can be shown.

1 is an exemplary diagram schematically illustrating the configuration of an adaptive streaming system for augmented reality media content according to the present invention.
2 is a block diagram illustrating an internal configuration of a user terminal in an adaptive streaming system of augmented reality media content according to the present invention.
3 is an exemplary diagram illustrating an operational relationship between each component in the adaptive streaming system for augmented reality media content according to the present invention.
4 is an exemplary view of a screen for selecting content by driving an augmented reality implementation application.
5 is an exemplary diagram illustrating content provided from a content providing server of an adaptive streaming system for augmented reality media content according to the present invention.
6 is an exemplary diagram illustrating the recognition of the floor in the user terminal of the adaptive streaming system of augmented reality media content according to the present invention.
7 is an exemplary diagram in which content is arranged in the user terminal of the system for adaptive streaming of augmented reality media content according to the present invention.
8 is an exemplary diagram illustrating content viewing information changed in the user terminal of the system for adaptive streaming of augmented reality media content according to the present invention.
9 is an exemplary view of an image according to content information of a full specification provided from a content providing server to a graphic processing server.
10 is an exemplary diagram illustrating content rendered by reflecting viewing information in a graphic processing server of an adaptive streaming system for augmented reality media content according to the present invention.
11 is an exemplary diagram illustrating receiving and displaying a rendered result in a user terminal of the system for adaptive streaming of augmented reality media content according to the present invention.

With respect to the embodiments of the present invention disclosed in the text, specific structural or functional descriptions are only exemplified for the purpose of describing the embodiments of the present invention, the embodiments of the present invention may be implemented in various forms and It should not be construed as being limited to the described embodiments.

Since the present invention can have various changes and can have various forms, specific embodiments are illustrated in the drawings and described in detail in the text. However, this is not intended to limit the present invention to the specific disclosed form, it should be understood to include all modifications, equivalents and substitutes included in the spirit and scope of the present invention.

Terms such as first, second, etc. may be used to describe various elements, but the elements are not limited by the terms. The above terms are used only for the purpose of distinguishing one component from another. For example, without departing from the scope of the present invention, a first component may be referred to as a second component, and similarly, the second component may also be referred to as a first component.

When a component is referred to as being “connected” or “connected” to another component, it is understood that the other component may be directly connected or connected to the other component, but other components may exist in between. it should be On the other hand, when an element is referred to as being “directly connected” or “directly connected” to another element, it should be understood that there is no other element in the middle. Other expressions describing the relationship between elements, such as "between" and "immediately between" or "neighboring to" and "directly adjacent to", etc., should be interpreted similarly.

The terms used in the present application are only used to describe specific embodiments, and are not intended to limit the present invention. The singular expression includes the plural expression unless the context clearly dictates otherwise. In this application, terms such as "comprises" or "having" are intended to designate that the disclosed feature, number, step, action, component, part, or combination thereof exists, but includes one or more other features or numbers, It should be understood that the possibility of the presence or addition of steps, operations, components, parts or combinations thereof is not precluded in advance.

Unless defined otherwise, all terms used herein, including technical or scientific terms, have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs. Terms such as those defined in commonly used dictionaries should be construed as indicating meanings consistent with the meanings in the context of the related art, and should not be construed in an ideal or excessively formal meaning unless explicitly defined in the present application. does not

On the other hand, when an embodiment can be implemented differently, functions or operations specified in a specific block may occur differently from the order specified in the flowchart. For example, two consecutive blocks may be performed substantially simultaneously, or the blocks may be performed in reverse according to a related function or operation.

Hereinafter, a real-time fire detection apparatus and a fire detection method using the same according to the present invention will be described with reference to the accompanying drawings.

The user terminal described in this specification includes a mobile phone, a smart phone, a laptop computer, a digital broadcasting terminal, a personal digital assistant (PDA), a portable multimedia player (PMP), a navigation system, and a slate PC. , tablet PCs, ultrabooks, wearable devices, for example, watch-type terminals (smartwatch), glass-type terminals (smart glass), HMD (head mounted display), etc. may be included. have.

However, it will be readily apparent to those skilled in the art that the configuration according to the embodiment described in this specification may be applied to a fixed terminal such as a digital TV, a desktop computer, a digital signage, etc., except when applicable only to a user terminal. will be.

1 is an exemplary diagram schematically illustrating the configuration of an adaptive streaming system for augmented reality media content according to the present invention. As shown, the configuration includes a user terminal 100 , a service providing server 200 , a content providing server 300 , and a graphic processing server 400 performing mutual data communication through the Internet network.

The user terminal 100 obtains viewing information for the augmented reality content being reproduced by driving an augmented reality application with minimum information and transmits it to the graphic processing server 400 , and the viewing information from the graphic processing server 400 . The augmented reality content is displayed by receiving the rendered result as content pool specification information corresponding to .

When the service providing server 200 receives the ID of the augmented reality content selected to be played by the user terminal, it transmits the Internet address (URL) of the content providing server mapped to the content ID to the user terminal 100 . The user terminal 100 accesses the content providing server 300 through the Internet address of the content providing server transmitted from the service providing server 200 and receives the minimum information of the augmented reality content that has been selected to play the augmented reality. Play real content.

The content providing server 300 provides the minimum information of the content to the user terminal 100, and provides full spec information of the content requested from the graphic processing server 400 according to the viewing information to the full spec (full spec) of 3D assets and a high-quality texture package and transmitted to the graphic processing server 300 .

The graphic processing server 400 analyzes the viewing information received from the user terminal 100 and renders it according to the full specification information of the corresponding content. Methods such as 5G streaming of the rendering result to the user terminal 100 provided as

2 is a block diagram illustrating a user terminal related to the present invention. Referring to FIG. 2 , the user terminal 100 includes a wireless communication unit 110 , an input unit 120 , a sensing unit 140 , an output unit 150 , an interface unit 160 , a memory 170 , and a control unit 180 . and a power supply unit 190 and the like. The components shown in FIG. 2 are not essential for implementing the user terminal, so the user terminal described in this specification may have more or fewer components than those listed above.

More specifically, among the components, the wireless communication unit 110 performs wireless communication between the user terminal 100 and the wireless communication system, between the user terminal 100 and another user terminal, or between the user terminal 100 and an external server. It may include one or more modules that enable In addition, the wireless communication unit 110 may include one or more modules for connecting the user terminal 100 to one or more networks.

The wireless communication unit 110 may include at least one of a broadcast reception module 111 , a mobile communication module 112 , a wireless Internet module 113 , a short-range communication module 114 , and a location information module 115 .

The input unit 120 may include a user input unit 123 such as a camera 121 for inputting an image signal, a microphone 122 for inputting an audio signal, and a touch key for receiving information from a user. The voice data or image data collected by the input unit 120 may be analyzed and processed as a user's control command.

The sensing unit 140 may include one or more sensors for sensing at least one of information in the user terminal, information on a surrounding environment surrounding the user terminal, and user information. For example, the sensing unit 140 may include a proximity sensor 141 , an illuminance sensor 142 , a touch sensor, an acceleration sensor, a magnetic sensor, a gravity sensor, a gyroscope sensor, a motion sensor, an infrared sensor, a fingerprint recognition sensor, and an ultrasonic sensor. , at least one of an optical sensor and an environmental sensor such as a barometer, a hygrometer, a thermometer, a radiation sensor, a thermal sensor, a gas sensor, and a chemical sensor such as an electronic nose, a healthcare sensor, and a biometric sensor may include. Meanwhile, the user terminal disclosed in the present specification may combine and utilize information sensed by at least two or more of these sensors.

The gyroscope sensor is a gyroscope (or tilt and electronic compass) composed of a rotor axis and a horizontal axis and a vertical axis orthogonal to the rotor axis to collect direction information and angle of the terminal. will be. Preferably, the gyroscope sensor may collect data on the direction and tilt angle of the terminal when the user takes an actual image in the photographing unit 110 .

The output unit 150 is for generating an output related to visual, auditory or tactile sense, and includes at least one of a display unit 151 , a sound output unit 152 , a haptic module 153 , and an optical output unit 154 . can do. The display unit 151 may implement a touch screen by forming a layer structure with the touch sensor or being formed integrally with the touch sensor. Such a touch screen may function as the user input unit 123 providing an input interface between the user terminal 100 and the user, and at the same time provide an output interface between the user terminal 100 and the user.

The interface unit 160 serves as a passage with various types of external devices connected to the user terminal 100 . The interface unit 160 includes at least one of a wired/wireless headset port, an external charger port, a wired/wireless data port, a memory card port, a port for connecting a device equipped with an identification module, an audio input/output port, a video input/output port, and an earphone port. may contain one. In response to the connection of the external device to the interface unit 160 , the user terminal 100 may perform appropriate control related to the connected external device.

The memory 170 stores data supporting various functions of the user terminal 100 . The memory 170 may store a plurality of applications or applications driven in the user terminal 100 , data for operation of the user terminal 100 , and commands. At least some of these application programs may be downloaded from an external server through wireless communication or exist on the user terminal 100 from the time of shipment. Meanwhile, the application program may be installed on the user terminal 100 and driven to perform an operation of the user terminal by the controller 180 .

The controller 180 generally controls the overall operation of the user terminal 100 in addition to the operation related to the application program. The controller 180 may provide or process appropriate information or functions to the user by processing signals, data, information, etc. input or output through the above-described components or by driving an application program stored in the memory 170 .

In addition, the controller 180 may control at least some of the components discussed with reference to FIG. 2 in order to drive an application program stored in the memory 170 . Furthermore, in order to drive the application program, the controller 180 may operate at least two or more of the components included in the user terminal 100 in combination with each other.

The power supply unit 190 receives external power and internal power under the control of the control unit 180 to supply power to each component included in the user terminal 100 . The power supply 190 includes a battery, and the battery may be a built-in battery or a replaceable battery.

At least some of the respective components may operate in cooperation with each other to implement the operation, control, or control method of the user terminal according to various embodiments described below. The operation, control, or control method of the user terminal may be implemented on the user terminal by driving at least one application program stored in the memory 170 .

3 is an exemplary diagram illustrating an operational relationship between each component in the adaptive streaming system of augmented reality media content according to the present invention. The user selects content as shown in FIG. 4 by driving the augmented reality implementation application of the user terminal 100 . In the present description, the implementation of the augmented reality application using the user terminal is exemplified, but a method of driving the augmented reality application in a service providing server connected to the user terminal through the Internet is also possible. In addition, the operation processed in the user terminal corresponds to the user's input through the user input unit, is displayed through the display unit by the control operation of the control unit corresponding to the image information obtained by the camera, or is connected to the external server through the wireless communication unit. represents communication.

For example, if the user selects "dice" as the augmented reality content, the user terminal 100 of the corresponding content in order to request an Internet address (URL) that can receive the selected content information from the service providing server 200 ID is provided. The service providing server 200 transmits the Internet address (URL) of the content providing server 300 suitable for the corresponding content ID to the user terminal 100 .

The user terminal 100 accesses the content providing server 300 using the corresponding Internet address (URL). The user terminal 100 requests the content providing server 300 to provide content information, that is, "dice game" content information. The content providing server 300 provides the user terminal 100 with minimum information on content corresponding to the request information received from the user terminal 100 . In this case, the minimum information of the content is, for example, a content asset and basic information defined by default. The basic information may include a content resolution according to a size and an angle defined by default. For example, in the case of a cube dice content as shown in Figure 5, the default size is 3cm X 3cm X 3cm in width, length and height, the number 1 is visible, and the y-axis is rotated by 30 degrees, The quality of the textures can be provided in HD.

The user terminal 100 that has received the minimum information of the content recognizes the floor through the camera as shown in FIG. 6 , and reflects and expresses the received minimum information of the content as shown in FIG. 7 . In this case, the floor recognition function may use Google's AR CORE or Apple's AR KIT. AR CORE is a platform that helps the development of augmented reality contents that can be used on Android smartphones, and AR KIT indicates a platform for augmented reality contents that can be used on Apple's iPhone. In a state in which the augmented reality application is implemented, the user may move to the right from the current position and move forward in order to view the details to enlarge the content. For example, when the user moves 30 degrees to the right and moves close to the content by about 10 cm and enlarges it by about 150%, the viewing information of the content is changed as shown in FIG. 8 . At this time, the viewing information extracted to the controller 180 of the user terminal 100 is spatial coordinate information of a state in which the content is placed in the real space through floor recognition, indicating the degree of enlargement or reduction of the content when expanding or reducing the content It may include at least one of enlargement/reduction information and information on a viewpoint viewed by the user.

Then, the control unit 180 of the user terminal 100 transmits the viewing information to the graphic processing server 400 through the wireless communication unit 110 .

The graphic processing server 400 accesses the Internet address (URL) of the content providing server 300 and requests the content of the full spec of the content. Full-spec content represents high-quality textures beyond content assets and basic information. For example, if the basic texture provided from the content providing server 300 to the user terminal 100 has a quality of HD (High Definition: 1280 X 720) level as shown in FIG. 9A, FIG. 9 As shown in (B) of (B), the high-quality texture provided from the content providing server 300 to the graphic processing server 400 has a resolution of FHD (Full High Definition: 1920 X 1080) or UHD (Ultra High Definition: 3840 X 2160). can have In this way, a collection of textures of various qualities in order to improve rendering speed is called a mipmap.

10 is an exemplary diagram illustrating content rendered by reflecting viewing information in a graphic processing server of an adaptive streaming system for augmented reality media content according to the present invention. That is, the graphic processing server renders based on the “basic information” provided from the content providing server 300 , and then performs modified rendering according to the “viewing information” provided by the user terminal 100 . When rendering change, a higher quality texture is used to show the texture quality corresponding to the basic information according to the degree of magnification, and as shown in FIG. 7 , it is expressed as the texture quality when the content is first placed in the user terminal 100 . When an image is enlarged, the number of pixels constituting it decreases, so it is possible to maintain the image quality by using a mipmap (MIPMAP) when using a high-quality image to enlarge. The graphic processing server 400 transmits the rendered result to the user terminal 100 by a method such as 5G streaming.

The user terminal 100 displays the augmented reality image reflecting the rendering result as shown in FIG. 11 by using the received rendering result.

At this time, as shown in FIG. 3 , the process of obtaining viewing information is repeated until the process of displaying an image according to the rendering result provided from the graphic processing server.

As described above, the adaptive streaming system and streaming method of augmented reality media content configured and operated as described above measures the distance through the gyro sensor and spatial coordinates, and whenever the corresponding distance changes, the graphic processing server renders the optimal resolution. 5G streaming, etc. Accordingly, when viewing high-capacity content as an augmented reality image, it is possible to exhibit the effect of using as much data as necessary without being constrained by a storage space.

Although the above has been described with reference to the preferred embodiment of the present invention, those skilled in the art can variously modify and change the present invention within the scope without departing from the spirit and scope of the present invention described in the claims below. You will understand that it can be done.

100: user terminal 200: service providing server
300: content providing server 400: graphic processing server

Claims (15)

a user terminal that acquires and transmits viewing information on the augmented reality content being reproduced with minimum information to the outside, receives a result rendered as content full specification information corresponding to the viewing information, and displays the augmented reality content;
a graphic processing server that analyzes the viewing information received from the user terminal to obtain full specification information of the corresponding content, performs content rendering, and provides a rendering result to the user terminal;
a content providing server for composing full spec information of content requested according to the viewing information from the graphic processing server into a 3D asset of a full spec and a high quality texture package and transmitting it to the graphic processing server; and
A service providing server receiving the ID of the augmented reality content selected to be played in the user terminal, and transmitting the Internet address (URL) of the content providing server mapped to the augmented reality content ID to the user terminal,
The user terminal accesses the content providing server through the Internet address (URL) of the content providing server received from the service providing server, receives and reproduces only the minimum information of the augmented reality content,
The viewing information is
spatial coordinate information of a state in which content is placed in real space through floor recognition;
Enlarging/reducing information indicating the extent to which the content is enlarged or reduced when the content is enlarged or reduced; and
An adaptive streaming system for augmented reality media content, characterized in that it includes at least one of information on a viewpoint viewed by a user. delete The adaptive streaming system of claim 1, wherein the minimum information of the content includes at least one of a content asset and basic information defined by default. The system of claim 3, wherein the basic information includes at least one of a content resolution according to a size and an angle defined by default. delete delete [Claim 5] The adaptive streaming system of augmented reality media content according to claim 4, wherein the graphic processing server renders the content based on the basic information and then performs modified rendering according to the viewing information received from the user terminal. . The adaptive streaming system of claim 7, wherein the graphic processing server performs the modified rendering by using the high-quality texture package according to enlarged information among viewing information received from the user terminal. selecting augmented reality content in the user terminal;
transmitting, by the service providing server receiving the ID of the augmented reality content from the user terminal, an Internet address (URL) of the content providing server mapped to the ID of the content to the user terminal;
Receiving, by the user terminal, the minimum information of the content from the content providing server, and playing the augmented reality content;
generating viewing information in the user terminal and transmitting it to a graphic processing server;
requesting, by the graphic processing server, 3D content information corresponding to the viewing information to the content providing server;
The graphic processing server, which has received the full spec information of the content composed of a full spec 3D asset and a high quality texture package from the content providing server, performs modified rendering by using the viewing information received from the user terminal, providing the result to the user terminal;
Displaying augmented reality content using the rendering result received from the user terminal,
The viewing information is
spatial coordinate information of a state in which content is placed in real space through floor recognition;
Enlarging/reducing information indicating the extent to which the content is enlarged or reduced when the content is enlarged or reduced; and
An adaptive streaming method of augmented reality media content, characterized in that it includes at least one of information on a viewpoint viewed by a user. delete The method of claim 9, wherein the minimum information of the content includes at least one of a content asset and basic information defined by default. The method of claim 11 , wherein the basic information includes at least one of a content resolution according to a size and an angle defined by default. delete 13. The method of claim 12,
The graphic processing server renders the content based on the basic information, and then performs modified rendering according to the viewing information received from the user terminal. delete

Images