KR20190033022A - Cloud multimedia service providing apparatus and method using computation offloading - Google Patents

Abstract

Provided are an apparatus and method for providing cloud multimedia service using computation offloading. The apparatus of the present disclosure may comprise: a client determining some predetermined operations to perform computation offloading during the entire operation; and a server performing the predetermined operations corresponding to a request from the client. The client may include a network based media processing (NBMP) client. The server may include a NBMP server.

Description

Technical Field [0001] The present invention relates to an apparatus and a method for providing a cloud multimedia service using computation offloading,

The present disclosure relates to an apparatus and method for providing a cloud multimedia service. More particularly, this disclosure relates to an apparatus and method for providing cloud multimedia services using computation offloading.

NBMP (Network Based Media Processing) -Aware client can be implemented by adding a module that performs NBMP function to existing client. NBMP (Network Based Media Processing) -ware client can be called NBMP client.

A client with a module that performs NBMP functions can receive NBMP services, and existing clients can operate as expected by ignoring NBMP data.

An object of the present invention is to provide an apparatus and method for reducing resources and power consumption of a client by providing a cloud multimedia service using computation offloading.

According to the present invention, there is provided a client system comprising: a client for determining a predetermined part of operations to perform calculation offloading during an entire operation; And a server for performing the determined predetermined operation in response to a request from the client, wherein the client includes a Network Based Media Processing (NBMP) client, and the server includes a service providing apparatus including an NBMP server Can be provided.

The features briefly summarized above for this disclosure are only exemplary aspects of the detailed description of the disclosure which follow, and are not intended to limit the scope of the disclosure.

According to the present invention, an apparatus and method for reducing resource and power consumption of a client by providing a cloud multimedia service using computation offloading can be provided.

According to the present invention, according to the present invention, a server performs a task having a large amount of calculation by using computation offloading, thereby greatly reducing resource and power consumption of a client.

1 is a diagram illustrating the relationship between cloud computing and virtualization according to one embodiment of the present disclosure.
2 is a diagram illustrating computed offloading in accordance with one embodiment of the present disclosure.
3 is a diagram illustrating an operation of a service providing apparatus according to an embodiment of the present disclosure.
4 is a diagram illustrating the operation of FRUC (Frame Rate Up-Conversion) using computation off-loading according to an embodiment of the present disclosure.
5 is a diagram illustrating the operation of upsampling using computed offloading in accordance with one embodiment of the present disclosure.
6 is a diagram illustrating the operation of image tagging using computed offloading in accordance with one embodiment of the present disclosure.
7 is a diagram illustrating an augmented reality processing operation using computed offloading in accordance with one embodiment of the present disclosure.
8 is a diagram illustrating the operation of a computer game using computed offloading in accordance with one embodiment of the present disclosure.

Hereinafter, embodiments of the present disclosure will be described in detail with reference to the accompanying drawings, which will be easily understood by those skilled in the art. However, the present disclosure may be embodied in many different forms and is not limited to the embodiments described herein.

In the following description of the embodiments of the present invention, a detailed description of known functions and configurations incorporated herein will be omitted when it may make the subject matter of the present disclosure rather unclear. Parts not related to the description of the present disclosure in the drawings are omitted, and like parts are denoted by similar reference numerals.

In the present disclosure, when an element is referred to as being "connected", "coupled", or "connected" to another element, it is understood that not only a direct connection relationship but also an indirect connection relationship May also be included. Also, when an element is referred to as " comprising "or" having "another element, it is meant to include not only excluding another element but also another element .

In the present disclosure, the terms first, second, etc. are used only for the purpose of distinguishing one element from another, and do not limit the order or importance of elements, etc. unless specifically stated otherwise. Thus, within the scope of this disclosure, a first component in one embodiment may be referred to as a second component in another embodiment, and similarly a second component in one embodiment may be referred to as a first component .

In the present disclosure, the components that are distinguished from each other are intended to clearly illustrate each feature and do not necessarily mean that components are separate. That is, a plurality of components may be integrated into one hardware or software unit, or a single component may be distributed into a plurality of hardware or software units. Thus, unless otherwise noted, such integrated or distributed embodiments are also included within the scope of this disclosure.

In the present disclosure, the components described in the various embodiments are not necessarily essential components, and some may be optional components. Thus, embodiments consisting of a subset of the components described in one embodiment are also included within the scope of the present disclosure. Also, embodiments that include other elements in addition to the elements described in the various embodiments are also included in the scope of the present disclosure.

Hereinafter, embodiments of the present disclosure will be described with reference to the accompanying drawings.

1 is a diagram illustrating the relationship between cloud computing and virtualization according to one embodiment of the present disclosure.

Virtualization is a fundamental component of cloud computing. In FIG. 1, virtualization can be depicted as being involved in cloud computing. For example, virtualization can help realize the value of cloud computing, and virtualization can create other systems that are independent of servers, workstations, storage media, and physical hardware layers.

2 is a diagram illustrating computed offloading in accordance with one embodiment of the present disclosure.

Computation offloading can mean sending specific computing tasks to an external platform, such as the cloud. Computational offloading can be used for intensive and intensive computing tasks. For example, the computing task may include artificial intelligence, artificial vision, object tracking, or computer decision making. Computational offloading can also be used to reduce energy, such as power.

Referring to FIG. 2, a virtualization process and calculation offloading may be used in an NBMP (Network Based Media Processing) process. If the steps to be processed by the client are task 1, task 2 and task 3, and task 2 is computationally heavy, the service providing apparatus of the present disclosure assigns task 2 to the cloud lt; RTI ID = 0.0 > cloud. < / RTI > However, it is assumed that the client has processed the task in step 2 of the task. Accordingly, the client can perform an operation that can not be performed due to a lack of computation resources, and can also reduce the processing energy consumed in the processing. Meanwhile, since the calculation off-loading process requires data to be transmitted between the client and the server, a delay may occur due to an additional data transmission / reception process. Thus, the processing method using computed offloading of the present disclosure may be effective when the gain obtained using computed offloading is greater than the cost caused by the delay. For example, in FIG. 2, when " task 2 > offloading ", the processing method using the calculation offloading of the present disclosure may be effective.

Hereinafter, various embodiments of the present invention will be described in detail.

3 is a diagram illustrating an operation of a service providing apparatus according to an embodiment of the present disclosure. The operations (1) to (9) described in FIG. 3 can represent the operation sequence in the service providing apparatus of the present disclosure.

The service providing apparatus of the present disclosure can add some operation to the operation between the conventional NBMP server and the NBMP client. For example, referring to Fig. 3, operations (1) to (4) and (6) may be added. Specifically, operation (1) indicates the operation in the NBMP server, and the NBMP server can perform the calculation offloaded operation. Operation (2) represents an operation in the NBMP client (NBMP client), and the NBMP client can calculate off-load some operations among the entire operations to be performed to the NBMP server. The entire operation can be completed by adding the operations performed on the NBMP server. In operation (3), the NBMP client may transmit the message and data necessary for requesting computation offload to the NBMP server. In operation (4), the NBMP server may calculate offloaded and transmit the calculated information or data to the NBMP client. In operation 5, the NBMP client adds content (i.e., value added contents) obtained by adding information or data received from an NBMP server to legacy contents (i.e., legacy contents) provided by a conventional legacy decoder to a legacy renderer renderer). In operation (6), a legacy router may transmit the legacy stream to the NBMP server. For example, the legacy stream may be all or part of the stream being sent to the legacy decoder, and may be the data needed to perform the requested NBMP operation for computational offloading. Operation 7 represents the operation in the legacy decoder, and the legacy decoder can perform the legacy decoding. Operation (8) indicates the operation in the legacy renderer, and the legacy renderer can render the raw content. The value added contents can also be entered into the legacy renderer in the same format as the legacy contents. In operation 9, the legacy decoder may send decoded raw contents to the NBMP client before the information received from the NBMP server or data is added (i.e., before value added).

4 is a diagram illustrating the operation of FRUC (Frame Rate Up-Conversion) using computation off-loading according to an embodiment of the present disclosure.

The FRUC processing process can roughly include two tasks. Task 1 (task 1) may be a process of calculating a motion vector at each position of a pixel or a block constituting an image. At this time, the motion vector is for true motion, which can be distinguished from existing motion estimation. Existing motion estimation finds the most similar blocks regardless of the motion of the object. On the other hand, true motion finds the actual motion of the object. Task 2 (task 2) may be a process of restoring a frame using a calculated motion vector. Generally, in the entire operation of the FRUC, the computation amount of the operation 1 may be several times as many as that of the operation 2. Also, Task 1 may have much more computation than basic video decoding.

FRUC processing is generally implemented on TV. According to the present disclosure, since almost all of the FRUC processing is performed in the cloud, the performance of the client's CPU can be relatively low and the power consumption in the client can be greatly reduced. In addition, when the client displays an image at a frame rate of 60 Hz, it increases the frame rate to 60 Hz by transmitting the image at 30 Hz from the server and adding the frame through cloud computing. This is compared with transmitting the image at 60 Hz in the server at 60 Hz Thereby reducing the transmission amount. This is because the frame to be added by temporal interpolation does not have an original, so the generated frame only needs to look like an original frame. Therefore, the sum of the transmission rate R (30 Hz HD) for transmission in 30 Hz HD and the transmission rate R (FRUC side stream) in operation 4 in FIG. 4 may be smaller than the transmission rate R (60 Hz HD) for transmission in 60 Hz HD (60Hz HD)> R (30Hz HD) + R (FRUC side stream)).

4, the operation (1) represents an operation in the NBMP server. The NBMP server decodes 30 Hz HD and performs task 1 (task 1) for calculating a true motion vector necessary for time interpolation Can be performed. Operation (2) indicates the operation in the NBMP client. The NBMP client receives the true motion vector at the NBMP server, receives the decoded 30 Hz HD from the legacy decoder, and performs time interpolation using the received information have. In operation (3), the NBMP client may request the NBMP server to calculate the true motion vector used for FRUC processing (NBMP request). In operation (4), the NBMP client can receive a true motion vector block by block as NBMP data from the NBMP server. In operation (5), the NBMP client can transfer data to the legacy renderer at 60 Hz HD as value added contents. In operation (6), the legacy router may send an encoded data stream of 30 Hz HD as a legacy stream to the NBMP server. Operation 7 represents the operation in the legacy decoder, and the legacy decoder can decode the 30 Hz HD image. Operation (8) indicates the operation in the legacy renderer, and the legacy renderer can render the data in 60Hz HD. In operation (9), the legacy decoder may transmit data of 30 Hz HD as legacy content to the NBMP client.

5 is a diagram illustrating the operation of spatial upsampling using computed offloading in accordance with one embodiment of the present disclosure.

In one embodiment, the process of converting an HD image to a UHD image may include two operations. Task 1 may be a process of calculating position and / or edge information to be edge sharpened from an image being linearly interpolated. In addition, task 2 (task 2) may be a process of calculating the edge sharpened pixel value using the calculated edge data. Generally, the amount of computation of task 1 may be several tens of times larger than that of task 2. Also, Task 1 may have much more computation than basic video decoding.

The process of converting an HD image to a UHD image is necessary in order to view the content currently in HD on a UHD TV. According to the present disclosure, since almost all of the process of converting the HD image into the UHD image is performed in the cloud, the CPU performance of the client can be relatively low and the power consumption in the client can be greatly reduced. In addition, the amount of transmission can be reduced compared with the case where the server is made into UHD and transmitted. This is because the UHD frame generated by converting the HD image into the UHD image has no original, so the generated frame needs only to look like the original. Therefore, the sum of the transmission amount R (HD) transmitted in the HD and the transmission amount R (side stream) in the operation 4 in FIG. 5 may be smaller than the transmission amount R (UHD) R (HD) R (HD) + R ((4) side stream)). Also, the amount of computation at the client may be much less than decoding UHD.

Specifically, referring to FIG. 5, operation (1) represents operation in the NBMP server. The NBMP server decodes the HD and performs operation 1 (task 1) for calculating edge information necessary for edge sharpening ) Can be performed. The operation (2) indicates the operation in the NBMP client. The NBMP client receives the decoded HD from the legacy decoder to perform spatial interpolation, receives the edge information from the NBMP server, and performs edge sharpening have. In operation (3), the NBMP client can request the NBMP server to calculate the edge information for performing the process of converting the HD image into the UHD image (NBMP request). In operation 4, the NBMP client can receive edge information as NBMP data from the NBMP server. In operation (5), the NBMP client can transfer UHD data to the legacy renderer as value added contents. In operation (6), the legacy router may transmit the encoded data stream of HD as a legacy stream to the NBMP server. Operation 7 represents the operation in the legacy decoder, and the legacy decoder can decode the HD image. Operation (8) indicates the operation in the legacy renderer, and the legacy renderer can render the data into the UHD. In operation 9, the legacy decoder can transmit HD data as legacy content to the NBMP client.

6 is a diagram illustrating the operation of image tagging using computed offloading in accordance with one embodiment of the present disclosure.

Image tagging or image annotation can largely involve two tasks. Task 1 (task 1) may be a process of recognizing an object from an input image. Task 2 may be a process of attaching a title or a description to a recognized object. Task 1 typically requires AI (Artificial Intelligence), which can be very computationally expensive.

Specifically, referring to FIG. 6, operation (1) indicates an operation in the NBMP server, wherein the NBMP server decodes video or still images, and can determine an object and annotate the object with each object. Meanwhile, the operation may be performed by decoding only an intra frame. Operation (2) indicates the operation in the NBMP client, where the NBMP client can receive the decoded video from the legacy decoder and receive the annotation from the NBMP server. The NBMP client may then overlay the received annotation on the decoded video. In operation (3), the NBMP client may request the NBMP server to acquire the object recognition information and / or the object-specific annotation information necessary for performing the image annotation (NBMP request). In operation (4), the NBMP client can receive the location of the object and / or the annotation information of each object as NBMP data from the NBMP server. In action (5), the NBMP client may send video data overlaid with annotations as value added contents to the legacy renderer. In operation (6), the legacy router may send the encoded data stream of video as a legacy stream to the NBMP server. Operation 7 represents the operation in the legacy decoder, and the legacy decoder can perform video decoding. Operation (8) indicates the operation in the legacy renderer, and the legacy renderer can render the data as annotated. In operation 9, the legacy decoder may transmit the original video data as legacy content to the NBMP client.

7 is a diagram illustrating an augmented reality processing operation using computed offloading in accordance with one embodiment of the present disclosure.

Augmented Reality (AR) can include two major tasks. Task 1 may be a process of recognizing a context in an image and calculating an overlay graphic. In addition, task 2 may be a process of overlaying the overlay graphic on the original image. Task 1 typically requires a lot of computation because of the AI and image rendering process.

7, the operation (1) indicates the operation in the NBMP server. The NBMP server decodes the video or still image and transmits the additional information (e.g., position information, camera angle, etc.) provided by the NBMP client ) Can be used to recognize the context and calculate overlay graphics. Meanwhile, the operation may be performed by decoding only an intra frame. Operation (2) indicates the operation in the NBMP client, where the NBMP client can receive the decoded video from the legacy decoder and receive the overlay graphics from the NBMP server. The NBMP client may then overlay the received overlay graphics on the decoded video. In operation (3), the NBMP client may request the NBMP server to perform context recognition and / or overlay graphics to perform the AR (NBMP request). If the client is an AR for video taken, it can transmit the video shot by the client and the client's position and / or camera angle information as additional information. In operation 4, the NBMP client may receive overlay graphics and / or alpha data from the NBMP server. At this time, alpha data may mean transparency in the overlay process. In operation (5), the NBMP client may send overlayed video data as value added contents to the legacy renderer. In operation (6), the legacy router may send the encoded data stream of video as a legacy stream to the NBMP server. Operation 7 represents the operation in the legacy decoder, and the legacy decoder can perform video decoding. Operation (8) indicates the operation in the legacy renderer, and the legacy renderer can render the data as overlaid. In operation 9, the legacy decoder may transmit the original video data as legacy content to the NBMP client.

8 is a diagram illustrating the operation of a computer game using computed offloading in accordance with one embodiment of the present disclosure.

On a computer game server, you can render video and compress the rendered video.

Specifically, referring to FIG. 8, operation (1) represents an operation in an NBMP server, wherein the NBMP server receives information received from other users (i.e., information received from a legacy router) and / And can render the video to be transmitted to the NBMP client using the information received from the client. At this time, the NBMP server can compress and transmit the video. Operation (2) indicates the operation in the NBMP client, and the NBMP client can decode the video received from the NBMP server using the information received from the legacy decoder. In addition, the NBMP client can receive user feedback and / or action using a user interface and transmit the received information to the NBMP server. In operation (3), the NBMP client may transmit the received feedback and / or action information to the NBMP server using a user interface (NBMP request). At this time, the information may include a mouse, touch screen information, and / or chat data. In operation 4, the NBMP client may receive the encoded data and / or game information of the game video as NBMP data from the NBMP server. For example, the game information may include information on progress of a computer game, chat data, and the like. In operation (5), the NBMP client can transfer the game video data to the legacy renderer as value added contents. In operation (6), the legacy router transmits the legacy stream to the NBMP server, in which case the data stream may not be present. Operation 7 represents the operation in the legacy decoder, and the legacy decoder can perform video decoding. Operation (8) indicates the operation in the legacy renderer, and the legacy renderer can render the data into the legacy renderer. In operation 9, the legacy decoder may transmit the original video data as legacy content to the NBMP client.

Although the exemplary methods of this disclosure are represented by a series of acts for clarity of explanation, they are not intended to limit the order in which the steps are performed, and if necessary, each step may be performed simultaneously or in a different order. In order to implement the method according to the present disclosure, the illustrative steps may additionally include other steps, include the remaining steps except for some steps, or may include additional steps other than some steps.

The various embodiments of the disclosure are not intended to be all-inclusive and are intended to illustrate representative aspects of the disclosure, and the features described in the various embodiments may be applied independently or in a combination of two or more.

In addition, various embodiments of the present disclosure may be implemented by hardware, firmware, software, or a combination thereof. In the case of hardware implementation, one or more application specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays A general processor, a controller, a microcontroller, a microprocessor, and the like.

The scope of the present disclosure is to be accorded the broadest interpretation as understanding of the principles of the invention, as well as software or machine-executable instructions (e.g., operating system, applications, firmware, Instructions, and the like are stored and are non-transitory computer-readable medium executable on the device or computer.

Claims (7)

A client that determines some of the operations to perform calculation offloading during the entire operation; And
And a server for performing the predetermined predetermined operation in response to a request from the client,
The client includes an NBMP (Network Based Media Processing) client, and the server includes an NBMP server. The method according to claim 1,
A legacy router for transmitting a legacy stream to the server;
A legacy decoder for performing legacy decoding; And
Further comprising a legacy renderer for rendering raw content,
Wherein the legacy stream is data necessary for performing the predetermined operation as all or a part of a stream transmitted to the legacy decoder. 3. The method of claim 2,
The whole operation is a FRUC (Frame Rate Up-Conversion) operation,
Wherein the client requests the server for a calculation of a true motion vector used in the FRUC operation. 3. The method of claim 2,
The entire operation is an image upsampling operation,
Wherein the client requests the server to calculate edge information necessary for converting a low resolution image into a high resolution image. 3. The method of claim 2,
The whole operation is an image annotation operation,
Wherein the client requests the server to acquire at least one of object recognition information necessary for performing image annotation and annotation information for each object. 3. The method of claim 2,
The entire operation is an Augmented Reality (AR) processing operation,
Wherein the client requests the server to acquire at least one of context recognition information and overlay graphic information necessary for performing the AR. 3. The method of claim 2,
The whole operation is a computer game operation,
Wherein the client transmits at least one of feedback and action information of the user received through the user interface to the server and requests the server to acquire at least one of the encoded data and game information of the computer game .

Images