KR102287274B1 - Method and apparatus for encording video - Google Patents

Abstract

The disclosed video encoding apparatus includes a communication unit connected to a network for transmitting video data by streaming, a control unit determining an encoding bit rate for the video data, and an encoding unit encoding video data at the determined encoding bit rate, the control unit comprising: Acknowledgment for signal transmission to the network of the communication unit is received from the communication unit to obtain packet information, to estimate the available network channel capacity of the network, and to video data based on the packet information and available network channel capacity. Determines the encoding bit rate for

Description

Video encoding apparatus and method {METHOD AND APPARATUS FOR ENCORDING VIDEO}

The present invention relates to an apparatus for encoding video data and a method for encoding video data at an encoding bitrate determined by the video encoding apparatus.

In order to raise the sensible performance of real-time streaming services including augmented reality/virtual reality services to the level of telepresence, real-time streaming technology with a round-trip delay of less than 10 ms that can deliver tactile sensations without cognitive dissonance is essential. there is. Round-trip latency of 10ms is physically impossible to achieve in 4G wireless networks, but is physically feasible in wired networks, Wi-Fi, and 5G wireless networks with a minimum round-trip delay of less than 10ms. Regardless of the type of network, the need for a low-latency streaming technology close to the minimum round-trip delay achievable in each network has emerged.

Conventionally, in order to achieve low-delay streaming in a video streaming environment, low-delay performance was achieved by encoding an image at a bitrate that is significantly lower than the available bandwidth of the network, but having a quality problem due to low-quality images and At the same time, video interruption occurs when the available bandwidth is changed due to network congestion or the like.

Korean Patent Publication No. 10-2014-0047575, published on April 22, 2014.

According to an embodiment, a video encoding apparatus and method for determining an encoding bit rate of video data by utilizing packet information of a network through which video data is to be transmitted by streaming are provided.

The problems to be solved of the present invention are not limited to those mentioned above, and other problems to be solved that are not mentioned will be clearly understood by those of ordinary skill in the art to which the present invention belongs from the following description.

A video encoding apparatus according to a first aspect includes a communication unit connected to a network for transmitting video data by streaming, a control unit determining an encoding bit rate for the video data, and encoding for encoding the video data at the determined encoding bit rate. comprising a unit, wherein the control unit receives from the communication unit an acknowledgment of the signal transmission to the network of the communication unit, obtains packet information including delay information, and determines the available network channel capacity of the network. and determines an encoding bit rate for the video data based on the information of the packet including the delay information and the available network channel capacity.

A video encoding method of a video encoding apparatus according to a second aspect includes: obtaining information of a packet including delay information from an acknowledgment for signal transmission of a network through which video data is to be transmitted by streaming; estimating , determining an encoding bitrate for the video data based on information of a packet including the delay information and the available network channel capacity, and encoding the video data at the determined encoding bitrate. includes steps.

The computer-readable recording medium storing the computer program according to the third aspect, when the computer program is executed by a processor, includes instructions for causing the processor to perform the video encoding method.

The computer program stored in the computer-readable recording medium according to the fourth aspect, when the computer program is executed by a processor, includes instructions for causing the processor to perform the video encoding method.

According to the embodiment, by determining the encoding bit rate of the video data by using packet information of the network through which the video data is to be transmitted by streaming, the video data is reduced to a level close to the minimum round-trip delay achievable in the network regardless of the type of network. Data can be streamed. In addition, it can operate adaptively to network congestion without dropping video frames.

1 is a block diagram of a video encoding apparatus according to an embodiment of the present invention.
2 is a flowchart illustrating a video encoding method according to an embodiment of the present invention.

Advantages and features of the present invention and methods of achieving them will become apparent with reference to the embodiments described below in detail in conjunction with the accompanying drawings. However, the present invention is not limited to the embodiments disclosed below, but can be implemented in various forms, and only these embodiments allow the disclosure of the present invention to be complete, and those of ordinary skill in the art to which the present invention pertains. It is provided to fully inform the person of the scope of the invention, and the scope of the invention is only defined by the claims.

In describing the embodiments of the present invention, detailed descriptions of well-known functions or configurations will be omitted except when it is actually necessary to describe the embodiments of the present invention. In addition, the terms to be described later are terms defined in consideration of functions in an embodiment of the present invention, which may vary according to intentions or customs of users and operators. Therefore, the definition should be made based on the content throughout this specification.

In this specification, the singular expression includes the plural expression unless the context clearly dictates otherwise. In the present application, terms such as 'comprise' or 'comprise' are intended to designate that a feature, number, step, operation, component, part, or combination thereof described in the specification is present, and one or more other It should be understood that this does not preclude the possibility of addition or presence of features or numbers, steps, operations, components, parts, or combinations thereof.

In addition, in the embodiment of the present invention, when a part is connected to another part, this includes not only direct connection but also indirect connection through another medium. In addition, the meaning that a part includes a certain component means that other components may be further included, rather than excluding other components, unless otherwise stated.

1 is a block diagram of a video encoding apparatus 100 according to an embodiment of the present invention.

As shown in FIG. 1 , the video encoding apparatus 100 according to the embodiment includes a communication unit 110 , a control unit 120 , and an encoding unit 130 .

The communication unit 110 is connected to a network through which video data is transmitted by streaming. The communication unit 110 may transmit a signal to the network under the control of the control unit 120 , and may receive an acknowledgment for signal transmission and provide it to the control unit 120 . In addition, the communication unit 110 may transmit the encoded video data to the network by streaming under the control of the control unit 120 .

The control unit 120 determines the size of a congestion window (CW) for limiting the size of a transmission unit of a packet transmitted to the network through the communication unit 110 and corresponds to the determined size of the congestion window, the packet transmission unit size can be limited. In addition, the control unit 120 determines an encoding bit rate for video data to be transmitted to the network through the communication unit 110 . For example, the control unit 120 may include a computing means such as a microprocessor (microprocessor). The control unit 120 may receive the acknowledgment for signal transmission to the network from the communication unit 110 to obtain packet information including delay information, estimate the available network channel capacity of the network, and packet An encoding bitrate for video data may be determined based on the information of and the available network channel capacity. For example, the controller 120 may obtain a round trip delay time of a packet and a real-time transmission rate of the network as information of the packet. For example, the controller 120 may estimate, as an available network channel capacity, a transmission rate that achieves a bandwidth-delay product of the network without loss of delay time, based on the round-trip delay time and the real-time transmission rate. Also, the controller 120 may determine the encoding bitrate for the current frame based on an estimate of an additional delay time occurring in a past frame that is encoded and transmitted with an instantaneous bitrate estimate among video data. Here, the control unit 120 compares the total time of the transmission time, the propagation delay time, and the queuing delay time for the past frame with the target delay time of streaming. Based on the additional delay time generated in the past frame can be estimated. For example, if the total time is equal to or less than the target delay time, the controller 120 may determine the instantaneous bitrate estimate for the past frame as the encoding bitrate for the current frame. Alternatively, if the total time is equal to or greater than the target delay time, the controller 120 may calculate a bit rate that satisfies the same condition as the total time and the target delay time, and determine the encoding bit rate for the current frame.

The encoding unit 130 encodes the video data at the encoding bit rate determined by the control unit 120 and provides the encoded video data to the communication unit 110 so that the video data can be transmitted by the communication unit 110 in streaming. can do.

2 is a flowchart illustrating a video encoding method according to an embodiment of the present invention.

Hereinafter, a video encoding method performed by the video encoding apparatus 100 according to an embodiment will be described in detail with reference to FIGS. 1 and 2 .

First, the controller 120 of the video encoding apparatus 100 determines the size of a congestion window (CW) for limiting the size of a transmission unit of a packet transmitted to a network through the communication unit 110, and the congestion window The communication unit 110 is controlled to transmit a predetermined signal to the network while limiting the size of a packet transmission unit corresponding to the size of . Here, the controller 120 may determine and control the initial bit rate based on the size of the initial congestion window and the known level of round trip delay time. Then, the communication unit 110 receives an acknowledgment for signal transmission generated by the receiving end connected to the network and provides it to the control unit 120 .

Subsequently, the control unit 120 obtains information on a packet including delay information from an acknowledgment for signal transmission. For example, the controller 120 may obtain a round trip delay time of a packet and a real-time transmission rate of the network as information of the packet. For example, the controller 120 may estimate the real-time data rate based on a sequence number of the acknowledgment packet ( S210 ).

Here, the control unit 120 may estimate, as an available network channel capacity, a transmission rate that achieves the bandwidth-delay product of the network without loss of delay time, based on the round-trip delay time and the real-time transmission rate. For example, the controller 120 may estimate, as an available network channel capacity, a transmission rate that achieves a delay product of the network bandwidth without loss of delay time, based on the round-trip delay time and the real-time transmission rate (S220).

In addition, the controller 120 may determine the encoding bit rate for the video data based on the packet information extracted from the acknowledgment for signal transmission and the available network channel capacity. Here, the controller 120 may determine the encoding bit rate for the current frame based on an estimate of an additional delay time generated in a past frame that is encoded and transmitted with an instantaneous bitrate estimate among video data. For example, the control unit 120 estimates the additional delay time generated in the past frame based on the result of comparing the total time of the transmission time, the propagation delay time, and the queuing delay time for the past frame with the target delay time of streaming can do. For example, if the total time is equal to or less than the target delay time, the controller 120 may determine the instantaneous bitrate estimate for the past frame as the encoding bitrate for the current frame. This is because the instantaneous bitrate estimate is determined and set to achieve the maximum bitrate of the channel in the past frame. Alternatively, if the total time is equal to or greater than the target delay time, the controller 120 may calculate a bit rate that satisfies the same condition as the total time and the target delay time and determine the encoding bit rate for the current frame ( S230 ). As such, a process in which the controller 120 determines the encoding bit rate will be described in more detail below.

Then, the encoding unit 130 encodes the video data at the encoding bit rate determined by the control unit 120 and provides the encoded video data to the communication unit 110 . Accordingly, the communication unit 110 may transmit the encoded video data to the receiving end of the network by streaming under the control of the control unit 120 ( S240 ).

Meanwhile, a process in which the controller 120 determines the encoding bit rate will be described in more detail.

When determining the instantaneous bitrate estimate that can achieve the maximum data rate, the controller 120 determines the encoding bitrate for the current frame by considering only the congestion window size and round-trip delay time if the delay time increase due to network congestion is not considered. there is. For example, an instantaneous bitrate estimate can be determined using Equation (1).

The denominator of Equation 1 is the smoothed RTT value updated by the ^{i th} acknowledgment, and the numerator is the congestion window value updated by the congestion control equation ^{after the j th RTT flows after the session is established.}

However, if encoding is performed by applying the instantaneous bitrate estimate calculated through Equation 1, it is impossible to solve the problem of increasing delay time that occurs during network congestion. For example, when congestion occurs due to network fluctuations in a situation in which the frame is transmitted by encoding the frame with the instantaneous bitrate estimate calculated through Equation 1, the actual available congestion window value during transmission is the instantaneous bit rate due to congestion. _{It becomes smaller than the CW j} value used for calculating the rate estimate, which inevitably causes an instantaneous queuing delay. Since this situation often occurs in a streaming environment in a moving situation using a mobile device, when determining the encoding bit rate, the queuing delay added when congestion occurs must be reflected in the next frame encoding to prevent the video stream itself due to network congestion. It can reliably transmit real-time frames without experiencing significant performance degradation (eg, video buffering for a long time).

Accordingly, the controller 120 may estimate the additional delay time of the frame to which the instantaneous bitrate estimate calculated through Equation 1 is applied, and determine the encoding bitrate of the current frame based on the estimated result. For example, the controller 120 may estimate the additional delay time for the past frame by using Equation (2).

In Equation 2, b is a tuning parameter for stable operation, t _k is a time stamp of when ^{k th} _{encoding starts, MTE i} is the maximum data rate estimate updated by ^{i th acknowledgment, and} , f(·) is a function designed to estimate the one-way delay, sRTT is the smoothed RTT value updated by the ^{i th} _{acknowledgment, and d v} is the communication medium (e.g., It is a target delay time that can be determined according to the type of application used, cellular, wireless, etc.) or channel conditions.

Using Equation 2, if the total time of the transmission time, propagation delay time, and queuing delay time for the past frame satisfies the condition that is less than or equal to the target delay time, the controller 120 determines the instantaneous bitrate estimate for the past frame may be determined as the encoding bitrate for the current frame. However, if the total time is equal to or greater than the target delay time, the control unit 120 calculates a bit rate that satisfies the same condition as the total time and the target delay time using Equation 3 and converts the calculated bit rate to the encoding bit rate for the current frame. can be decided with

In Equation 3, α _k is a tuning parameter used to more conservatively set _{the bitrate k} by increasing the target delay time d _v by a when the instantaneous bitrate estimate is not continuously satisfied, and a is It is a tuning parameter for stable operation.

As described above, when the encoding bit rate is determined using Equations 1 to 3, the bit rate may be adaptively changed to various network conditions.

If the network condition remains constant or improves, if d _v is tight enough to be close to mRE/2 (minimum one-way delay), then the issue of transmission time is more important to achieving _{d v than the issue of queuing delay.} do. That is, only by keeping the encoding bitrate very small so that the transmission time is very small, d _v can barely be satisfied. In this situation, queuing does not occur because it continues to transmit a much smaller amount of data than the line rate. If d _v is very generous, _{it is possible to increase the encoding bitrate to the line rate estimated by the MTE i} *(1-b) value. In this case, the transmission time becomes the encoding interval. Even in this case, since the network does not deteriorate, queuing does not occur much.

When the network condition continues to deteriorate, an instantaneous bitrate estimate value is calculated with the expectation that the encoding bitrate will be similar to or smaller than the line rate, but the line rate is sharply reduced due to the deterioration of the network condition, and there is a situation in which a queuing delay occurs. do. In this case, the queuing delay may be the cause of not satisfying _{d v} , or the transmission time may be the cause of not satisfying _{d v .} However, regardless of where the delay is a major factor, the queuing that has already occurred is accepted, and _{an encoding bitrate capable of satisfying d v} under the corresponding queuing is calculated from Equation (2). When _{a situation in which d v} is not satisfied continuously occurs (eg, a situation in which the channel continues to deteriorate, etc.), the _{value of α k} may be increased by a in order to more safely estimate the encoding bit rate conservatively.

_{Encoding bitrate is determined by distinguishing between a situation in which d v} is too tight and a situation in which the channel continues to stay because of a constant deterioration, such as when the network condition deteriorates and improves, when the network condition as described above is kept constant or when it improves. There is a need. These conditions when distinguished with the function f (sRTT _i) The value means easily possible the identification, the f (sRTT _i) value, there is no queuing surface close to mRE / 2 values (i.e., dv a tight situation), the In case for α _k Set the value to '0'. In the opposite case, it means that queuing has occurred, and you can prepare for it by increasing the _{α k value.}

As described so far, according to the embodiments of the present invention, the encoding bit rate of the video data is determined using packet information of the network through which the video data is to be transmitted through streaming, so that it can be achieved in the corresponding network regardless of the type of network. Video data can be streamed with a level close to the minimum round-trip delay. In addition, it can operate adaptively to network congestion without dropping video frames.

Combinations of each step in each flowchart attached to the present invention may be performed by computer program instructions. These computer program instructions may be embodied in a processor of a general purpose computer, special purpose computer, or other programmable data processing equipment, such that the instructions performed by the processor of the computer or other programmable data processing equipment provide the functions described in each step of the flowchart. It creates a means to do these things. These computer program instructions may also be stored in a computer-usable or computer-readable medium that may direct a computer or other programmable data processing equipment to implement a function in a particular manner, and thus the computer-usable or computer-readable medium. The instructions stored in the recording medium are also possible to produce an article of manufacture including instruction means for performing the functions described in each step of the flowchart. The computer program instructions may also be mounted on a computer or other programmable data processing equipment, such that a series of operational steps are performed on the computer or other programmable data processing equipment to create a computer-executed process to create a computer or other programmable data processing equipment. It is also possible that instructions for performing the processing equipment provide steps for performing the functions described in each step of the flowchart.

Further, each step may represent a module, segment, or portion of code comprising one or more executable instructions for executing the specified logical function(s). It should also be noted that in some alternative embodiments it is also possible for the functions recited in the steps to occur out of order. For example, it is possible that two steps shown one after another may in fact be performed substantially simultaneously, or that the steps may sometimes be performed in the reverse order depending on the function in question.

The above description is merely illustrative of the technical spirit of the present invention, and various modifications and variations will be possible without departing from the essential characteristics of the present invention by those skilled in the art to which the present invention pertains. Therefore, the embodiments disclosed in the present invention are not intended to limit the technical spirit of the present invention, but to explain, and the scope of the technical spirit of the present invention is not limited by these embodiments. The protection scope of the present invention should be interpreted by the following claims, and all technical ideas within the scope equivalent thereto should be construed as being included in the scope of the present invention.

100: video encoding device
110: communication department
120: control unit
130: encoding unit

Claims (19)

a communication unit connected to a network for transmitting video data by streaming;
a controller for determining an encoding bit rate for the video data;
an encoding unit for encoding the video data at the determined encoding bit rate;
The control unit receives from the communication unit an acknowledgment for signal transmission to the network of the communication unit, obtains information of a packet including delay information, estimates an available network channel capacity of the network, and The encoding bit rate is determined based on the information of the packet including the delay information and the available network channel capacity, and based on an estimate of an additional delay time occurring in a past frame that is encoded and transmitted with an instantaneous bit rate estimate among the video data. to determine the encoding bitrate for the current frame.
video encoding device. The method of claim 1,
The control unit is configured to obtain a round trip delay time of the packet and a real-time transmission rate of the network as the information.
video encoding device. 3. The method of claim 2,
The control unit, based on the round-trip delay time and the real-time transmission rate, estimating, as the available network channel capacity, a transmission rate that achieves a bandwidth-delay product of the network without loss of delay time
video encoding device. delete The method of claim 1,
The control unit, based on a result of comparing the total time of a transmission time, a propagation delay time, and a queuing delay time for the past frame and the target delay time of the streaming to estimate the additional delay time
video encoding device. 6. The method of claim 5,
If the total time is equal to or less than the target delay time, the controller determines the instantaneous bitrate estimate for the past frame as the encoding bitrate for the current frame.
video encoding device. 6. The method of claim 5,
When the total time is equal to or greater than the target delay time, the controller calculates a bit rate that satisfies the same condition as the total time and the target delay time to determine the encoding bit rate for the current frame
video encoding device. 8. The method of claim 7,
If the total time of frames to which the calculated bitrate is applied as the encoding bitrate is equal to or greater than the target delay time, the control unit adjusts a tuning parameter that allows a result value to be calculated to be lower and then a bitrate that satisfies the same condition to determine the encoding bit rate for the next frame by calculating
video encoding device. An augmented reality device comprising the video encoding device of any one of claims 1 to 3 or 5 to 8. A video encoding method of a video encoding device, comprising:
obtaining information of a packet including delay information from an acknowledgment for signal transmission of a network that will transmit video data by streaming;
estimating an available network channel capacity of the network;
An encoding bitrate for the video data is determined based on the information of the packet including the delay information and the available network channel capacity, and an additional delay occurring in a past frame encoded with an instantaneous bitrate estimate among the video data and transmitted determining an encoding bitrate for the current frame based on the estimate of time;
encoding the video data at the determined encoding bitrate;
How to encode video. 11. The method of claim 10,
The step of obtaining the information of the packet may include obtaining a round trip delay time of the packet and a real-time transmission rate of the network as the information.
How to encode video. 12. The method of claim 11,
In the estimating of the available network channel capacity, based on the round-trip delay time and the real-time data rate, a data rate that achieves a bandwidth-delay product of the network without loss of delay time is calculated as the available network channel capacity. estimated as
How to encode video. delete 11. The method of claim 10,
In the determining of the encoding bit rate, a total time of a transmission time, a propagation delay time, and a queuing delay time for the past frame, and a target delay time of the streaming Estimating the additional delay time based on the result of comparing
How to encode video 15. The method of claim 14,
The determining of the encoding bit rate may include determining an instantaneous bit rate estimate for the past frame as an encoding bit rate for the current frame if the total time is equal to or less than the target delay time.
How to encode video. 15. The method of claim 14,
In the determining of the encoding bit rate, if the total time is equal to or greater than the target delay time, calculating a bit rate that satisfies the same condition as the total time and the target delay time is used as the encoding bit rate for the current frame. to decide
How to encode video. 17. The method of claim 16,
In the determining of the encoding bitrate, if the total time of frames to which the calculated bitrate is applied as the encoding bitrate is equal to or greater than the target delay time, after adjusting a tuning parameter that allows a result value to be calculated to be lower, the same Calculating a bitrate that satisfies the condition and determining the encoding bitrate for the next frame
How to encode video. As a computer-readable recording medium storing a computer program,
The computer program, when executed by a processor,
18. A method comprising instructions for causing the processor to perform the method of any one of claims 10 to 12 or 14 to 17.
computer readable recording medium. As a computer program stored in a computer-readable recording medium,
The computer program, when executed by a processor,
18. A method comprising instructions for causing the processor to perform the method of any one of claims 10 to 12 or 14 to 17.
computer program.

Images