WO2015012644A1

WO2015012644A1 - Method and apparatus for transceiving multimedia content

Info

Publication number: WO2015012644A1
Application number: PCT/KR2014/006828
Authority: WO
Inventors: Kyung-Mo Park; Sung-Ryeul Rhyu; Sung-Oh Hwang; Doug-Young Suh; Kyu-Heon Kim; Gwang-Hoon Park; Jae-Yeon Song; Ji-Hyeok YOON; Yong-Hun Lee
Original assignee: Samsung Electronics Co Ltd; Kyung Hee University
Current assignee: Samsung Electronics Co Ltd; Kyung Hee University
Priority date: 2013-07-26
Filing date: 2014-07-25
Publication date: 2015-01-29
Anticipated expiration: 2016-01-26
Also published as: KR102127800B1; KR20150012724A

Abstract

Provided is a method for transmitting multimedia content. The method includes creating the content, creating at least one of first complexity representative information necessary for consumption of full content of the content and second complexity representative information necessary for consumption of a segment of the content, and transmitting the at least one of the first complexity representative information and the second complexity representative information, and the created content.

Description

METHOD AND APPARATUS FOR TRANSCEIVING MULTIMEDIA CONTENT

The present disclosure relates to a method and apparatus for transceiving multimedia content. More particularly, the present disclosure relates to a method and apparatus for transceiving content, which supports scaling of voltage and frequency resources used in content consumption.

Carbon dioxide (CO2) emissions of the Information and Technology (IT) field, estimated as 2% of worldwide CO2 emissions in 2009 in a similar scale to that of the aerospace industry, are now expected to exceed such an estimate. The Kyoto Protocol mandates reduction of CO2 emissions down to 5.2% of the 1990’s level. In Korea, although reduction of CO2 emissions is not an obligation but a recommendation, the Green Energy Business was introduced in April 2010 to recommend reduction of CO2 emissions, and certified companies have been supported in terms of taxes and finances.

Recently, the use of mobile devices has been explosively increasing. The mobile devices have limited battery power resources. Moreover, as the display size of a mobile device, the complexity of the mobile device for multimedia services, and a transfer rate of the mobile device increase, power consumption also increases drastically. Therefore, a need exists for a control scheme for providing maximum quality while using minimum resources, based on a correlation between multimedia processing and power in the mobile device.

An aspect of the present disclosure is to provide a segment-specific post-decoding scheme as a new way having advantages of Dynamic Voltage and Frequency Scaling (DVFS) schemes.

Another aspect of the present disclosure is to provide a signaling method capable of supporting all of a full post-decoding scheme, a segment-specific post-decoding scheme, and a segment-specific complexity estimation scheme.

Moreover, another aspect of the present disclosure is to provide a method for delivering an absolute complexity representative variable of full content for a full post-decoding scheme.

In addition, another aspect of the present disclosure is to describe a method for delivering a segment-specific complexity of content for a segment-specific post-decoding scheme and to provide a method for using the delivered complexity at a client.

In accordance with an aspect of the present disclosure, a method for transmitting multimedia content is provided. The method includes creating the content, creating at least one of first complexity representative information necessary for consumption of full content of the content and second complexity representative information necessary for consumption of a segment of the content, and transmitting the at least one of the first complexity representative information and the second complexity representative information, and the created content.

In accordance with another aspect of the present disclosure, a method for receiving multimedia content is provided. The method includes receiving the content, receiving at least one of first complexity representative information necessary for consumption of full content of the content and second complexity representative information necessary for consumption of a segment of the content, scaling a voltage and a frequency to consume the content by using the at least one of the first complexity representative information and the second complexity representative information, and consuming the content by using the scaled voltage and frequency.

In accordance with another aspect of the present disclosure, an apparatus for transmitting multimedia content is provided. The apparatus is configured to create the content, to create at least one of first complexity representative information necessary for consumption of full content of the content and second complexity representative information necessary for consumption of a segment of the content, and to transmit the created content and at least one of the first complexity representative information and the second complexity representative information.

In accordance with another aspect of the present disclosure, an apparatus for receiving multimedia content is provided. The apparatus is configured to receive the content, to receive at least one of first complexity representative information necessary for consumption of full content of the content and second complexity representative information necessary for consumption of a segment of the content, to scale a voltage and a frequency to consume the content by using the at least one of the first complexity representative information and the second complexity representative information, and to consume the content by using the scaled voltage and frequency.

According to an embodiment of the present disclosure, by using segment-specific DVFS, improved resource saving effects may be provided and a delay occurring in content playback may be prevented.

Moreover, according to an embodiment of the present disclosure, content does not need to be pre-coded to obtain information needed to reserve resources necessary for content consumption, thereby efficiently using the resources.

Furthermore, according to an embodiment of the present disclosure, it is not necessary to depend on an estimation method, instead of real content information, for information needed to reserve resources necessary for content consumption, thereby further improving decoding energy saving effect.

In addition, according to an embodiment of the present disclosure, a separate module for pre-coding or estimation does not need to be configured, preventing hardware resources from being wasted.

FIG. 1 illustrates reduction of decoding energy over time in a client device;

FIG. 2 is a graph describing a relationship between a request processor speed obtained by a client using full-content complexity representative information and a period-specific complexity according to an embodiment of the present disclosure;

FIG. 3 is a flowchart illustrating a process of performing content Dynamic Voltage and Frequency Scaling (DVFS) by using full-content complexity representative information at a client according to an embodiment of the present disclosure;

FIG. 4 is a graph describing a method for using a time segment-specific relative complexity by using relative segment complexity representative information at a client according to an embodiment of the present disclosure;

FIG. 5 is a flowchart illustrating a process of performing segment DVFS by using relative segment complexity representative information at a client according to an embodiment of the present disclosure;

FIG. 6 is a graph showing a decoding energy saving effect corresponding to complexity estimation of decoding of video data as an example according to an embodiment of the present disclosure;

FIG 7 is a histogram describing the accuracy of decoding estimation corresponding to complexity estimation of decoding of video data as an example according to an embodiment of the present disclosure;

FIG. 8 is a flowchart illustrating a multimedia content transmission method according to an embodiment of the present disclosure;

FIG. 9 is a flowchart illustrating a multimedia content reception method according to an embodiment of the present disclosure;

FIG. 10 is a block diagram of a multimedia content transmission apparatus according to an embodiment of the present disclosure; and

FIG. 11 is a block diagram of a multimedia content reception apparatus according to an embodiment of the present disclosure.

Hereinafter, various embodiments of the present disclosure will be described in relation to the accompanying drawings. In the following description, a detailed description of known functions and configurations incorporated herein will be omitted when it may obscure the subject matter of the present disclosure. Terms used herein are defined based on functions in the present disclosure and may vary according to users, operators’ intention or usual practices. Therefore, the definition of the terms should be made based on contents throughout the specification.

Before a detailed description of the present disclosure is made, examples of interpretable meanings of several terms used in the present disclosure will be suggested. However, it should be noted that the terms are not limited to the examples provided below.

A server is an entity that communicates with a client and provides content data to the client. Thus, when the server creates content, the server may be referred to as a content creation apparatus.

The client is an entity that communicates with the server and consumes content provided from the server. Generally, the client may be referred to as a portable terminal, a User Equipment (UE), a Mobile Station (MS), a Mobile Equipment (ME), a device, or a terminal, but the client is not necessarily limited to this example and a new server may also be a client if it consumes content.

A complexity means a workload required for operating a processor to execute a module needed for decoding/encoding such that the client consumes content. That is, the complexity may be understood as a concept indicating a workload generated due to any work including calculation, an operation, processing, and the like.

Dynamic Voltage and Frequency Scaling (DVFS) is a technique for reducing energy used in calculation by controlling voltage and frequency of a processor that plays multimedia content such as video based on a complexity. Basically, a multimedia content processing apparatus raises voltage to raise the frequency of the processor if a complexity to be processed is high. A correlation between frequency and complexity may be described as below.

If calculation has to be done within a set time (for example, like in processing of a real-time multimedia service), a time t for calculation of a processor may be calculated as follows:

(Equation 1)

A frequency f [Hz] of the processor is reversely proportional to a calculation time t [sec] and is proportional to a complexity c [cycle]. In a Complementary Metal-Oxide Semiconductor (CMOS) circuit, a supply voltage

(V drain) may be expressed as follows:

(Equation 2),

where

(omega),

(phi), and

(theta) are coefficients determined by an underlying platform. For example, an Intel Pentium M1.6GHz processor in a 90 nano-meter process may have

=5.6*10^-10,

=1, and

=0.61.

In the CMOS circuit, a power

(P dynamic) may be expressed as follows and may be calculated in watt:

(Equation 3),

where

indicates an effective circuit capacitance.

By using the foregoing equations, a calculation energy

(E dynamic) may be calculated in joule [J] as follows:

(Equation 4)

FIG. 1 illustrates a measurement result of decoding energy over time when coefficients

(omega),

(phi), and

(theta) determined by an underlying platform are set to 5.6*10^-10, 1, and 0.61.

Referring to FIG. 1, after a predetermined time (about 10^-5 sec) has elapsed, a reduction of decoding energy over time is little. As such, when one frequency is used, an expected energy saving effect is limited. Thus, by dynamically controlling frequency used in content consumption as well as voltage, energy used in calculation needs to be reduced.

DVFS includes a post-decoding scheme for scaling voltage and frequency by using information obtained by calculating in advance a part of content and a complexity estimation scheme for scaling voltage and frequency according to a value modeled or estimated by modeling a complexity with a separate profiler or estimating a complexity with a separate queue.

The post-decoding scheme needs reference complexity information corresponding to a unit period to which DVFS is to be applied. The reference complexity information may be information obtained by parsing and pre-decoding content data corresponding to a unit period.

More specifically, the post-decoding scheme is a scheme for pre-decoding a part of content and scaling voltage and frequency for a next DVFS application unit by using the pre-decoded content part to obtain reference complexity information for use in application of DVFS in a multimedia service. The content part that is to be pre-decoded may be one or more Group of Pictures (GOP) or one or more video frames. The DVFS application unit may be one or more GOP or one or more video frames.

Therefore, the post-decoding scheme is more suitable for voltage and frequency scaling with respect to content that may be pre-decoded, that is, non-real-time content. For the same reason, the post-decoding scheme is not relatively suitable for fine voltage and frequency scaling of real-time content in which complexity information of full content cannot be obtained.

The post-decoding scheme shows a power saving effect for full content, but has a difficulty in fine scaling in real time, thus having a low energy saving efficiency when compared to real-time scaling based on the complexity estimation scheme.

The complexity estimation scheme enables fine voltage and frequency scaling by estimating a segment-specific complexity with a separate profiler or queue and using the estimated complexity for DVFS. Thus, the complexity estimation scheme allows an improved power saving effect for real-time content when compared to the post-decoding scheme.

To be more specific, the complexity estimation scheme estimates a segment-specific complexity to apply voltage and frequency scaling to content every short-term in a multimedia service and performs voltage and frequency scaling by using the estimated complexity. Information used for estimation using the profiler or the queue may include information indicating characteristics of content (for example, a high or low complexity) or indicating a type of the content (for example, a static or dynamic screen), information indicating a scene change, information indicating an increase in a complexity, and the like.

The complexity estimation scheme estimates a complexity every short-term and uses the estimated complexity, such that the complexity estimation scheme may immediately respond to a change in a complexity when compared to the post-decoding scheme, but may have an error because of using the estimated complexity rather than an accurate value of the complexity of the content like in the post-decoding scheme. Moreover, the complexity estimation scheme needs an additional operation for estimation.

The present disclosure proposes a segment-specific post-decoding scheme as a new way having advantages of the two DVFS schemes. The present disclosure also proposes a signaling method capable of supporting all of a full post-decoding scheme, a segment-specific post-decoding scheme, and a segment-specific complexity estimation scheme. In the following description, the full post-decoding scheme may be referred to as content DVFS and the segment-specific post-decoding scheme may be referred to as segment DVFS.

Herein, a segments means a portion divided from content into an arbitrary size according to time. For example, the segment may be one GOP, one frame, or a unit divided smaller than the frame.

For the full post-decoding scheme according to an embodiment of the present disclosure, a description will be made of a method for delivering an absolute complexity representative variable of full content.

For the segment-specific post-decoding scheme according to an embodiment of the present disclosure, a description will be made of a method for delivering a segment-specific complexity of content and a method for the client to use the delivered complexity.

A content creation apparatus (for example, a server), when creating content, also creates a variable indicating a complexity needed for content consumption of a content consumption apparatus (for example, a client) from the content. The content creation apparatus may transmit the created variable together with the content or may store and then transmit the created variable together with the content.

The complexity may be expressed by combining one or more of the number of cycles and the possibility of parallel processing. Thus, a unit of a variable indicating the complexity may be various.

The complexity may be indicated separately for a target service and a global service.

The target service is targeted for a particular service. For example, if a video conference service such as video conference is targeted, an image may be configured with an Intra (I) frame and a unidirectional predicted (P) frame. When an image is configured with an I frame and a P frame, Motion Estimation (ME) and Motion Compensation (MC) modules are not almost used, such that it may be more appropriate to obtain a complexity based on complexities (that is, cycles) of other modules than the ME and MC modules.

Table 1 shows an operation that may be used to express 1(one) cycle for the global service.

Table 1

The global service is intended for a general-purpose service. This service may be used to obtain a complexity of every module used by a general decoder.

For the global service, rather than the target service, one cycle may be expressed as one-time A bytes B point C, and other complexities may be expressed as relative values of a complexity defined as one cycle. For example, one cycle, which is a complexity of the global service, may be expressed as “integer bytes floating point addition”, “integer bytes fixed point subtraction”, or the like

Table 2 shows an operation that may be used to express one cycle for a target service.

Table 2

The target service may be usefully used for a special case, for example, such as a video conference service in which the real-time property is important.

If a particular target service exists, one cycle may be defined as a main operation unit of the target service. In an embodiment, if a video service is a target service, a one-time complexity of several modules (unit operations) executed many times among necessary operations of the service may be defined as a cycle and represented as a relative complexity value to be suitable for the particular target. One cycle may be expressed as a C encoding/decoding module that performs a B unit operation with respect to A macro-blocks (a structure unit of video data).

Discrete Cosine Transformation (DCT), Motion Estimation (ME), Motion Compensation (MC), a loop filter, entropy, quantization, and intra prediction shown in a column C of Table 2 are modules necessary for encoding/decoding.

Table 3 show attributes for expressing a full-content complexity.

Table 3

		Variables	Unit
Request processor speed	r_G	double	[cycles/sec]
Allowed processing time	D	double	Msec
Cycle unit information	CU	unsigned long	MIPS or FLOPS
Peak buffer status	B_max	unsigned long	Bytes	Optional
Relative unit complexity	UR	unsigned long	Cycles	Optional

A request processor speed r_G is a guaranteed calculation speed requested per second, and is an intermediate value in cycle/sec between an average calculation speed requested by content and a maximum calculation speed that is the instantly highest calculation speed. To determine the request processor speed, an allowed processing time needs to be considered. For a large allowed processing time, a request processor speed is low. The request processor speed may be used to reserve resources (that is, voltage and frequency) for consuming some period of content.

An allowed processing time D expresses a maximum allowed time for performing voltage and frequency scaling in second. A processing time may vary with a unit for performing DVFS (for example, a GOP, a frame, or the like), resulting in jitter. Thus, the allowed processing time is a variable for performing DVFS in allowed jitter. The allowed processing time may be determined by information (a frame per second in video) that determines an extra time until execution of an applied task and by the jitter.

Cycle unit information C_U is a variable for delivering a quantitative complexity (that is, a unit complexity) for the aforementioned one cycle. That is, for the general-purpose service, the cycle unit information C_U means a quantitative complexity needed to perform an A bytes B point C operation; for the particular target service (for example, a video service), the cycle unit information C_U means a quantitative complexity needed to perform a C operation with respect to A macro-blocks in units of B pixels (Pel). The unit complexity may be expressed as a Million Instructions Per Second (MIPS) or a Floating Point Operation (FLOP) that is a unit used for quantitatively expressing a task complexity in a computer.

A peak buffer status B_max is expressed in byte. The amount of data used for a buffer may be expressed as a sum of data before being processed and data after being processed. When compressed multimedia service data is decoded, the amount of data greatly increases after the data is processed, such that the amount of data stored after being processed needs to be minimized.

A relative unit complexity U_R is a variable expressing a unit complexity to represent a relative size with respect to a standard complexity. To deliver complexity information of a complex operation such as decoding of used compressed video data as large-unit MIPS or FLOP information, a large amount of data for delivery is needed. Instead, by performing standard calculation that is a basis of a complexity and expressing a complexity as a relative size with respect to the complexity (a standard complexity) used for standard calculation, the complexity may be expressed with a little data. The standard calculation may include decoding one or more GOP or one or more video frames. A complexity needed to calculate video data corresponding to a particular unit (for example, a GOP or a frame) is referred to as the relative unit complexity U_R. Optionally, a complexity of next unit periods or the cycle unit information may be represented as a multiple of the relative unit complexity.

Table 4 describes attributes for expressing a segment-specific complexity of content as an absolute value.

Table 4

		Variables	Unit
Length of time segment	T_i	double	msec
# of cycles (number of cycles)	C_i	float	cycles
Parallel processing flag	P_f	Int	%	Optional
# of parallel processors (number of parallel processors)	n_i	Int	-	Optional
Parallel period	P_p	Int	%	Optional

Segment-specific complexity representative variables are needed to adaptively deliver the amount of resources that temporally change during a service.

The attributes described in Table 4 may be used to deliver complexity representative information for each of multiple segments into which single content is divided. A length of an i^th time segment may be expressed as T_i [sec], and a complexity during T_i may be expressed as C_i [cycle].

Optionally, a parallel period during the segment’s length T_i, P_p%, may indicate that parallel processing by n_i processors is possible. Herein, the parallel period P_p % means a period of the length of the segment during which parallel processing is possible. A video service may be divided into a period in which parallel processing is possible (for example, ME and MC modules) and a period during which parallel processing is impossible (for example, an entropy module), such that information about the period during which parallel processing is possible may be delivered through the parallel period P_p%. In parallel processing, a parallel processing start point in a length of a segment is determined by a parallel processing flag P_f%, and parallel processing may continue during the parallel processing period P_p%.

If the period in which parallel processing is possible is not continuous, start point information P_f and duration period information P_p are delivered in the form of percentage (%), and thus through continuous recording with 2-byte-based extension of a length of a message, the information may be delivered. For example, if execution and stop of parallel processing are repeated, extension may be made by including the next parallel period P_p under a parallel period P_p while adding 2 bytes (for an Int variable of 2 bytes).

Herein, the message is intended to transmit attribute information shown in Table 4, and may also be referred to as meta data.

Table 5 describes attributes for relatively expressing a segment-specific complexity of content.

Table 5

		Variables	Unit
Length of time segment	T_i	double	msec	Optional
Relative complexity	r_i	Int	%
Parallel processing flag	P_f	bool	-	Optional
# of parallel processors	n_i	Int	-	Optional
Parallel period	P_p	Int	%	Optional

The number of cycles representing a complexity is a large number, and the complexity may vary according to a device of a client and an Operating System (OS) environment, such that it is necessary to express the complexity as a relative value for the sake of reducing the amount of information used to express the complexity.

If a complexity of an (i-1)^th time segment is C_i-1 [cycle] and a complexity of an i^th time segment is C_i [cycle], a relative complexity r_i is equal to

.

If there is no previous complexity to be referred to as in random access (direct access to an arbitrary point in time of video to play the video), r_i may be equal to 100 (r_i = 100). Segments following the i^th segment may be expressed with relative complexities with respect to the i^th complexity.

When a relative complexity is used, the relative complexity r_i is determined after one segment period, such that estimation of a complexity of the next segment period is possible. Thus, even when there is no complexity to be referred to like in random access, a complexity of a next segment can be calculated after the first segment by using a relative complexity. For example, when a complexity of the first segment is calculated, the complexity of the first segment is obtained using a maximum frequency and a frequency is properly set using a relative complexity calculated for a next segment to perform scaling with the frequency.

If it is not necessary to separately indicate a start point and it is merely necessary to indicate whether there is parallel processing because the start point of parallel processing is preset, a parallel processing flag may be expressed as a Boolean value of ‘true’ or ‘false’.

In Table 3, attributes for expressing a complexity needed to consume single full content are described, and in Tables 4 and 5, segment-specific complexity variables are described.

A complexity variable for full content is delivered during initialization of content consumption for use in allocation of resources that need to be prepared for client’s content consumption (that is, client’s content playback).

A complexity variable for segment-specific DVFS of the client may be a variable expressing an absolute complexity of Table 4 or a relative complexity of Table 5. A complexity variable for a part of content, that is, each segment is used as information for application of DVFS in shorter-time-units (that is, segment DVFS) and may be periodically or aperiodically delivered. The aperiodic delivery may occur in a particular event situation such as when a complexity needed to consume a segment changes largely when compared to a previous segment due to a scene change or the like.

If a segment-specific complexity of content is larger than a full-content complexity, a delay may occur in a corresponding segment, such that the segment-specific complexity may be smaller than or equal to the full-content complexity.

First, delivery of a full-content complexity variable for content DVFS will be described.

The full-content complexity variable is used in a service where initialization of content consumption exists.

For the service where initialization of content consumption exists, the client receives complexity representative information for full content and determines resources needed for full content consumption during initialization.

Hereinafter, a method for delivering a complexity variable in an exemplary video service will be described. However, the complexity variable delivery method to be described is an example of a technology proposed by the present disclosure and thus the use thereof in other applications is not limited.

1) Delivery of Content Complexity Variable in Broadcast-Type Service

In a video service provided based on broadcast-type transmission, initialization with a client through an outband does not generally exist, unlike in a one-to-one transmission video service (for example, a Video on Demand (VoD) or streaming service).

Thus, a system for transmitting a video service in a broadcast manner uses a method for periodically inserting information for initialization into a channel for media data transmission, that is, inband signaling. The client that initiates a service at an arbitrary point in time performs initialization by using a periodically transmitted inband signal and then receives video data to provide a service to a user.

For example, for a Moving Picture Experts Group (MPEG)-2 system, a Program Map Table (PMT) exists to deliver unique information for each program, such as a Packet Identifier (PID), an elementary stream, and so forth. Initial information for a complexity variable proposed in Table 3 may be transmitted in a head portion of each program through stream reserved bits of an optional descriptor supported in the PMT.

2) Delivery of Content Complexity Variable in Unicast Service

In a video service provided based on unicast-type transmission, unlike in broadcast-type transmission, initialization based on outband signaling is possible. Thus, complexity variable information proposed in the present disclosure may be transmitted to a client terminal through an outband signaling channel. As an example of outband signaling, if a media service uses a Session Initiation Protocol (SIP) to initialize a session, the proposed complexity variable may be delivered through a field of an extension header (for example, a P-Asserted-Service field) supported in the SIP. As another example, when a Session Description Protocol (SDP) is used, complexity information may be delivered in the form of a supplemental descriptor (for example, an Attributes field) that may be situated after information defined in the SDP.

3) Delivery of Content Complexity Variable Using SEI Message

In MEPG-4 Advanced Video Coding (AVC)/H.264 and its subsequent video standards, supplemental information may be provided in a bitstream of encoded video to improve use of video data, and this information is called a Supplemental Enhancement Information (SEI) message. The complexity variable proposed in this disclosure may be periodically or aperiodically delivered to the client by using the SEI message.

4) Delivery of Content Complexity Variable in MPEG Media Transport (MMT)

In MMT, which is a new MPEG transmission system standard, 3 functional layers are provided, which include encapsulation, delivery, and signal layers. The encapsulation layer includes internal sub layers E1, E2, and E3 depending on a unit of media for encapsulation. An MMT asset encapsulated in the E2 sub layer is a unit that is similar with an elementary stream of the MPEG-2 system. Generally, one MMT asset receives one program and Asset Delivery Characteristic (ADC), which is information needed to transmit the MMT asset, exists in a head portion of each MMT asset. The ADC includes information needed to transmit the MMT assets through a network (for example, a bitrate description, a delay, a loss priority, or the like), such that the server and the client may determine a policy during initialization and transmission for media transmission based on the foregoing information.

The complexity information proposed in the present disclosure may also be transmitted similarly with the ADC. That is, information for initialization of the client (that is, full-content complexity variable information) is transmitted through a head portion of each MMT asset, thereby helping DVFS in the client that consumes content.

In the Dynamic Adaptive Streaming over HTTP (DASH) standard known as MPEG-DASH, a type and characteristics of transmission media are included in a Media Presentation Description (MPD). Between representations of the media, a variable for the DVFS function may be delivered through decoding-dependent information. The variable added for the DVFS function may be, for example, an adequate description.

Next, delivery of a segment-specific complexity variable for segment DVFS will be described.

A description will be made of a method for delivering a complexity variable for a part of content, that is, for each segment, to apply DVFS to the segment-specific complexity variable in short-term units. The segment-specific complexity variable may be aperiodically delivered periodically or in a particular event situation (a case where a complexity necessary for segment consumption changes largely when compared to a previous segment, for example, in case of a scene change or the like).

A method for delivering a complexity variable in an exemplary video service will be described below. However, the complexity variable delivery method is an example of a technique proposed by the present disclosure, and thus the use thereof in other applications is not limited.

1) Delivery of Segment-Specific Complexity Variable in Broadcast-Type Service

A segment-specific complexity variable in a broadcast-type service may be transmitted through an optional message at a Program Specific Information (PSI) level.

2) Delivery of Segment-Specific Complexity Variable in Unicast-Type Service

In a unicast-type service, a server may deliver segment-specific complexity information proposed in the present disclosure to a client by using an extension field and a supplemental header field in a protocol such as a Real Time Streaming Protocol (RTSP), a Real Time Protocol (RTP), an RTP Control Protocol (RTCP), or the like.

3) Delivery of Segment-Specific Complexity Variable Using SEI Message

Like in delivery of a full-content complexity variable using an SEI message, a segment-specific complexity variable proposed in the present disclosure may be delivered to a client periodically or aperiodically by using an SEI message.

4) Delivery of Segment-Specific Complexity Variable in MMT

Like in delivery of a full-content complexity variable in MMT, a segment-specific complexity variable proposed in the present disclosure may also be delivered through an MMT asset. In MMT, Transport Characteristics (TC) are delivered by updating an ADC for each MMT asset. Complexity information of a detailed unit (for example, a segment) of content proposed in the present disclosure allows the client to know a segment-specific complexity variable because the segment-specific complexity variable is updated for each segment like TC of the MMT.

In the DASH standard, index information for a segment encapsulated in a file level is provided in a segment index box sidx, through which a variable for a segment-specific DVFS function (that is, a segment-specific complexity variable) may also be delivered.

FIG. 2 is a graph describing a relationship between a request processor speed obtained by a client using full-content complexity representative information and a period-specific complexity according to an embodiment of the present disclosure.

A request processor speed r _G 200 is a value expressing a per-second complexity of content as an absolute value through division of a complexity of a full period of the content by a full time of the content. By using the per-second complexity of the content, a rough complexity of the full content may be recognized, and in this way, a resource for consuming the full content may be reserved. Reservation of the resource for content consumption may include frequency scaling or voltage scaling.

FIG. 3 is a flowchart illustrating a process of performing content DVFS by using full-content complexity representative information at a client according to an embodiment of the present disclosure.

The client receives full-content complexity representative information together with the content. Once the content is input to the client, the client determines whether initialization of content consumption is performed at operation 302. If determining at operation 302 that the initialization has not been performed, the client performs the content initialization at operation 304 and obtains a request processor speed r_G from the complexity representative information at operation 306. If determining at operation 302 that the initialization has been performed, the client obtains the request processor speed from the complexity representative information at operation 306. The client performs content DVS by using the obtained request processor speed at operation 308. For example, the client may perform DVFS with respect to the full content.

FIG. 4 is a graph describing a method for using a time segment-specific relative complexity by using relative segment complexity representative information at a client according to an embodiment of the present disclosure.

A method for controlling DVFS on a segment basis may include using absolute complexity information and using relative complexity information. The use of the absolute complexity information may basically use the same method as DVFS of full content described above, except that a DVFS application unit is a segment.

A description will now be made of the use of the relative complexity information.

Because the first segment has no complexity representative information to be referred to, the client measures a complexity of the segment at a maximum frequency (measure complexity 404). The client obtains a relative complexity r_i from segment complexity representative information and then performs DVFS with respect to a next segment.

By delivering a relative complexity [%] with respect to a complexity of a previous segment for each segment, a complexity to be referred to may be expressed as r_i [%], irrespectively of a device or a type of an OS. Because a complexity for content consumption may vary with a device or an OS environment of the client, the client may use the relative complexity variable r_i.

Referring to FIG. 4, the client may obtain a complexity 402 of an i^th segment from a complexity 400 of an (i-1)^th segment and the relative complexity r_i obtained by the client, and performs segment DVFS by using the complexity 400 of the (i-1)^th segment.

FIG. 5 is a flowchart illustrating a process of performing segment DVFS by using relative segment complexity representative information at a client according to an embodiment of the present disclosure.

The client receives the segment complexity representative information together with content. The client measures a complexity of a first segment at operation 502. Optionally, the client may measure the complexity of the first segment by using a maximum frequency.

Optionally, the client may determine whether an operation with respect to a current segment is random access at operation 504, before obtaining a complexity of the current segment.

If determining that the operation with respect to the current segment is random access, the client sets a relative complexity of the current segment to 100 at operation 506 because there is no complexity information of a previous segment and thus there is no relative complexity of the current segment.

Optionally, the client delivers the relative complexity of the current segment to the next segment at operation 508.

If the process of determining whether the operation with respect to the current segment is random access does not exist or if it is determined at operation 504 that the operation with respect to the current segment is not random access, the client obtains a relative complexity from the complexity representative information at operation 510.

The client performs segment DVFS by using the obtained relative complexity at operation 512.

Optionally, the client may calculate the relative complexity again for DVFS of the next segment at operation 514.

Optionally, the client may deliver the calculated relative complexity to the next segment at operation 508.

According to another embodiment of the present disclosure, content DVFS and segment DVFS may be applied together. That is, content DVFS may be performed using full-content complexity representative information during content initialization, and segment DVFS may be performed using segment-specific complexity representative information after initialization. That is, a combination of FIGS. 3 and 5 may be executed.

FIG. 6 is a graph showing a decoding energy saving effect corresponding to complexity estimation of decoding of video data as an example according to an embodiment of the present disclosure.

For decoding energy saving results corresponding to complexity estimation of decoding of video data, measured decoding energy results of a method (No DVFS) 600 that does not use DVFS, a DVFS method (Content DVFS) 602 using content complexity representative information, and a DVFS method (Segment DVFS) 604 using segment complexity representative information are as shown in FIG. 6.

When compared to the method 600 that does not deliver a complexity variable and does not use DVFS, the content DVFS method 602 using delivered content complexity representative information shows a decoding energy saving efficiency of about 10%. The segment DVFS method 604 may perform fine frequency scaling when compared to the content DVFS method 602, providing an improvement of about 7.8% in energy saving when compared to the energy saving result of the content DVFS method 602.

For a test, in an Android/embedded system board HBE-SM5-S4210, Coretax A9 Dual-core (2Ghz) was used as a processor, Android 2.3.5 (Gingerbread) was used as an OS, and a user-space type that inputs a user input frequency was used as a Central Processing Unit (CPU) governor (a Linux kernel that determines a policy for applying a frequency (a CPU clock) to a CPU). Improvement effects may vary with media content and a test bed used for simulation.

A client device used in the test estimated a complexity by relatively referring to the most adjacent previous complexity for each frame type (for example, an Intra (I) frame, a unidirection Predicted (P) frame, and a Bidirectional (B) predictive frame) without using an estimator for complexity estimation, such as a separate profiler or queue. Because of not using a separate estimator for DVFS application, the client device according to the present disclosure does not need to perform pre-decoding prior to decoding or to store pre-decoding information in a separate memory.

FIG 7 is a histogram describing the accuracy of decoding estimation corresponding to complexity estimation of decoding of video data as an example according to an embodiment of the present disclosure.

The histogram illustrated in FIG. 7 is a histogram of a delay occurring when content is decoded using a frequency obtained by dividing a full-content complexity by a content play time according to a content DVFS method.

In the histogram, a portion where a GOP delay [%] is a negative value is a portion where the frequency (obtained by dividing a full-content complexity by a content play time), which is a fixed value, is larger than a frequency requested by a complexity, such that decoding energy is wasted. In the histogram, a portion where the GOP delay is a positive value is a portion where the frequency (obtained by dividing a full-content complexity by a content play time) is smaller than a complexity to be processed with a frequency, such that a delay occurs.

In a test with respect to 18 GOPs for about 10 seconds, an accumulative delay of about 0.11833 second (118.33 ms) occurs. Such a delay is not a problem in a non-live event service such as a streaming service, but may be an issue in a live event such as video conference. Thus, in the live event service, the segment DVFS method capable of adaptively responding to a segment-specific complexity change needs to be used. That is, segment DVFS performs frequency scaling by dividing an estimated segment-specific complexity by a time during which decoding has to be completed, such that no delay occurs.

FIG. 8 is a flowchart illustrating a multimedia content transmission method according to an embodiment of the present disclosure.

A multimedia content transmission apparatus (for example, a server) creates transmission content at operation 800.

The transmission apparatus creates information needed for a content reception apparatus (for example, a client) to consume content, that is, at least one of complexity representative information for the full content and complexity representative information for a segment, at operation 810. The complexity representative information may be created when the transmission apparatus creates the content. The complexity representative information is information representing a complexity used to perform DVFS, and detailed information included in the complexity representative information may include, for example, the attributes described in Tables 3 through 5.

The transmission apparatus transmits at least one of the complexity representative information for the full content and the complexity representative information for the segment and the created content to the reception apparatus, at operation 820. At least one of the complexity representative information for the full content and the complexity representative information for the segment may be transmitted through an MMT asset for an MMT system, and may be transmitted using the above-described methods for other systems.

FIG. 9 is a flowchart illustrating a multimedia content reception method according to an embodiment of the present disclosure.

A multimedia content reception apparatus (for example, a client) receives content from a content transmission apparatus (for example, a server) at operation 900.

The reception apparatus receives information needed to consume the content, that is, at least one of complexity representative information for the full content and complexity representative information for a segment, at operation 910. The complexity representative information is information representing a complexity used to perform DVFS, and detailed information included in the complexity representative information may include, for example, the attributes described in Tables 3 through 5.

At least one of the complexity representative information for the full content and the complexity representative information for the segment may be received through an MMT asset for the MMT system, and may be received using the above-described methods for other systems.

The reception apparatus performs DVFS by using the received complexity representative information at operation 920.

The reception apparatus consumes (that is, plays, through decoding or the like) the received content by using the scaled resources (that is, voltage and frequency) at operation 930.

FIG. 10 is a block diagram of a multimedia content transmission apparatus according to an embodiment of the present disclosure.

A content transmission apparatus 1000 transmits content under control of a controller 1020, and may include a transceiver 1010 for transmitting complexity representative information and the controller 1020.

The controller 1020 may control the content transmission apparatus (for example, a server) described in the present disclosure to perform a series of methods. For example, the controller 1020 may perform an operation of creating content to be transmitted and an operation of creating complexity representative information for full content and complexity representative information for a segment.

A content reception apparatus 1100 receives content under control of a controller 1120, and may include a transceiver 1110 for receiving complexity representative information and the controller 1120.

The controller 1120 may control a content reception apparatus (for example, a client) described in the present disclosure to perform a series of methods. For example, the controller 1120 may perform an operation of initializing content, an operation of allocating a resource for content consumption, an operation of obtaining complexity representative information to perform DVFS, and an operation of consuming the content by using the resource as a result of the scaling.

It should be noted that the graphs, methods, apparatuses, and test result tables illustrated in FIGS. 2 through 11 are not intended to limit the scope of the present disclosure. That is, all components or operations should not be construed as being essential components for carrying the present disclosure and the present disclosure may be implemented merely with some of them without departing from the gist of the present disclosure.

The above-described operations may be implemented by including a memory device having a corresponding program code stored therein in a component of a server apparatus, a client apparatus, or a terminal apparatus. A controller of the server apparatus, the client apparatus, or the terminal apparatus may execute the above-described operations by a program code stored in a memory device by reading and executing the program code stored in the memory device using a processor or a CPU.

Various components and modules of the server apparatus, the client apparatus, or the terminal apparatus described in the present disclosure may operate by using a hardware circuit, for example, a CMOS-based logic circuit, firmware, software, and a combination of software and/or firmware and firmware and/or software inserted into a machine-readable medium. For example, various electric structures and methods may be implemented using electric circuits such as transistors, logic gates, and on-demand semiconductors.

While the present disclosure has been particularly shown and described with reference to exemplary embodiments thereof, various changes in form and detail may be made therein without departing from the spirit and scope of the present disclosure as defined by the following claims. Accordingly, the scope of the present disclosure will be defined by the appended claims and equivalents thereto.

Claims

A method for transmitting multimedia content, the method comprising:

creating the content;

creating at least one of first complexity representative information necessary for consumption of full content of the content and second complexity representative information necessary for consumption of a segment of the content; and

transmitting the at least one of the first complexity representative information and the second complexity representative information, and the created content.
A method for receiving multimedia content, the method comprising:

receiving the content;

receiving at least one of first complexity representative information necessary for consumption of full content of the content and second complexity representative information necessary for consumption of a segment of the content;

scaling a voltage and a frequency to consume the content by using the at least one of the first complexity representative information and the second complexity representative information; and

consuming the content by using the scaled voltage and frequency.
The method of claim 1 or 2, wherein the first complexity representative information and the second complexity representative information represent a complexity used to perform Dynamic Voltage and Frequency Scaling (DVFS).
The method of claim 1 or 2, wherein the first complexity representative information comprises a request processor speed that is a calculation speed needed to be guaranteed for consumption of the full content.
The method of claim 4, wherein the first complexity representative information further comprises:an allowed processing time representing a maximum allowed time during which DVFS is performed with respect to the full content; andcycle unit information representing a quantitative complexity for one cycle of a processor that is to consume the full content.
The method of claim 1 or 2, wherein the at least one of the first complexity representative information and the second complexity representative information is transmitted through an Moving Picture Expert Group (MPEG) Media Transport (MMT) asset encapsulated in an E2 sub layer that is a sub layer of an encapsulation layer of an MMT system.
The method of claim 1 or 2, wherein second complex representative information comprises:a length of a time segment representing a length of the segment; anda number of cycles representing a complexity during the segment as a number of cycles.
The method of claim 1 or 2, wherein the second complex representative information comprises:a length of a time segment representing a length of the segment; anda relative complexity representing a complexity during the segment as a relative value with respect to a complexity of a previous segment of the segment.
An apparatus for transmitting multimedia content, the apparatus being configured:

to create the content, to create at least one of first complexity representative information necessary for consumption of full content of the content and second complexity representative information necessary for consumption of a segment of the content, and to transmit the created content and at least one of the first complexity representative information and the second complexity representative information.
An apparatus for receiving multimedia content, the apparatus being configured:

to receives the content, to receive at least one of first complexity representative information necessary for consumption of full content of the content and second complexity representative information necessary for consumption of a segment of the content, to scale a voltage and a frequency to consume the content by using the at least one of the first complexity representative information and the second complexity representative information, and to consume the content by using the scaled voltage and frequency.
The apparatus of claim 9 or 10, wherein the first complexity representative information comprises a request processor speed that is a calculation speed needed to be guaranteed for consumption of the full content.
The apparatus of claim 11, wherein the first complexity representative information further comprises:an allowed processing time representing a maximum allowed time during which DVFS is performed with respect to the full content; andcycle unit information representing a quantitative complexity for one cycle of a processor that is to consume the full content.
The apparatus of claim 9 or 10, wherein the at least one of the first complexity representative information and the second complexity representative information is transmitted through an Moving Picture Expert Group (MPEG) Media Transport (MMT) asset encapsulated in an E2 sub layer that is a sub layer of an encapsulation layer of an MMT system.
The apparatus of claim 9 or 10, wherein the second complex representative information comprises:a length of a time segment representing a length of the segment; anda number of cycles representing a complexity during the segment as a number of cycles.
The apparatus of claim 9 or 10, wherein the second complex representative information comprises:a length of a time segment representing a length of the segment; anda relative complexity representing a complexity during the segment as a relative value with respect to a complexity of a previous segment of the segment.