KR101668283B1 - Method for displaying video considered latency, apparatus and cloud streaming service system therefor - Google Patents

Abstract

The present invention relates to a video output method considering latency that can solve a latency due to a processing delay in which decoded video data is accumulated by removing a part of video data that has not been rendered and decoded and accumulated over a predetermined frame, And an apparatus including a video output function, wherein the apparatus including a video output function fetches decoded video data stored in an input buffer and sequentially rasterizes the decoded video data stored in the input buffer, And deletes some of them in units of frames.

Description

TECHNICAL FIELD [0001] The present invention relates to a video output method, a device and a cloud streaming service system,

More particularly, the present invention relates to a cloud streaming service, and more particularly, to a cloud streaming service that resolves a latency due to a processing delay in which decoded video data is accumulated, And an apparatus and a cloud streaming service system for performing the method.

With the development of the mobile communication network and the development of the terminal specification, the mobile communication terminal has become a necessity of modern people and has evolved into a total entertainment device beyond the conventional category of simple communication apparatus or information providing apparatus.

In addition, in recent years, it is difficult for a low-performance client terminal to process a large-capacity image, so that the data is processed through a cloud server at a remote location, and only the processed result screen And a cloud computing technology for receiving and displaying data on a client terminal is used.

In the cloud computing environment, it is possible to build and use the information technology (IT) environment in which the service is used without the expertise of individual technologies.

Particularly, a game, a VOD, a real-time broadcast content, and the like are executed on the server side based on the cloud computing technology, and the execution screen is transmitted to the user device in a streaming manner, A cloud streaming service is a service that allows a user to use a desired content regardless of device specifications and performance.

In a case where a request input from a user device is executed in a service device connected to the user device via a communication network and the result is transmitted to a user device and output, such a cloud streaming service, The longer the latency is, the lower the user satisfaction and service quality are.

Accordingly, there is a need to provide a quick response so that the user can feel as if the user is playing or playing the content on the user device.

In the case of the cloud streaming service, video data and audio data of a moving picture corresponding to the execution or playback result are transmitted to the user device through a communication network in a streaming manner in response to a user's request. The network transmission bandwidth, There are many problems to be solved, such as ability, synchronization, etc. These problems can increase the latency and degrade the service quality.

Specifically, a user equipment receiving a bit stream including video data and audio data as a response to a request over a communication network separates video data and audio data from the bit stream through demultiplexing, parses the header, Data and audio data are respectively decoded and synchronized in order to match the reproduction time point between the video data and the audio data and then outputted as video and audio through a rendering process. The time required for processing for demultiplexing, The waiting time of the user whose delay of the video and audio output is delayed is increased.

Therefore, efforts to minimize the time required for demultiplexing, synchronization, and rendering processes are required to improve the service quality by reducing the waiting time of the user.

Korean Registered Patent No. 10-0226528, July 28, 1999 (Name: Multiplexed Compressed Image / Audio Data Decoding Apparatus)

The present invention has been proposed in order to minimize the latency between receiving a user request and a response in providing a cloud streaming service. Particularly, when video data that has not been rendered and decoded is accumulated over a predetermined number of frames, And a latency that can be solved due to a processing delay caused by accumulation of decoded video data by removing the latency of the decoded video data, and an apparatus for performing the method and a cloud streaming service system.

As a means for solving the above-mentioned problems, the present invention provides a video decoding apparatus comprising: a network receiving unit for receiving a bitstream including encoded video data; A decoding unit decoding the bit stream received by the network receiving unit; A rendering unit that stores decoded video data output from the decoding unit in an input buffer, sequentially fetches decoded video data stored in an input buffer on a frame basis, and sequentially composes the decoded video data to form an output image; And a controller for checking the number of decoded video data stored in the input buffer at regular intervals and deleting a part of the decoded video data in a frame unit.

Here, the controller checks whether the video data stored in the input buffer exists within a predetermined time based on the time when the rendering unit fetches the decoded video data of the specific frame from the input buffer and starts rendering, and stores the decoded video data in the input buffer And decrypts the decrypted video data. If there is decrypted video data of two or more frames, it can selectively delete some of the decrypted video data.

In addition, in the user equipment according to the present invention, the network receiving unit receives the encoded video data transmitted in accordance with the reproduction time frame by frame from the cloud streaming service apparatus, and transmits the encoded video data to the decoding unit, Can be decoded as it is received.

In addition, the present invention provides a communication apparatus comprising: a connection interface unit connected to a user apparatus to transmit and receive control signals and video data; A rendering unit that stores decoded video data received from a user apparatus through the connection interface unit in an input buffer, sequentially fetches decoded video data stored in an input buffer on a frame basis, and sequentially composes the decoded video data to form an output image; And a control unit for checking the number of decoded video data stored in the input buffer at regular intervals and deleting a part of the decoded video data in frame units.

In addition, the present invention relates to a user apparatus for receiving coded video data and decoding the coded video data at a predetermined decoding rate; The decoded video data stored in the input buffer is fetched in units of frames and sequentially rendered so as to form an output video, And a video output device for confirming the number of decoded video data stored in the input buffer at predetermined intervals and deleting a part of the decoded video data in a frame unit.

The cloud streaming service system according to the present invention may further comprise a cloud streaming service device for sequentially transmitting coded video data of the content requested by the user device on a frame-by-frame basis in accordance with a playback time of the user device.

In addition, the present invention provides an apparatus including a video output function, wherein the apparatus includes: sequentially fetching and decoding the decoded video data stored in an input buffer on a frame-by-frame basis; And checking the number of decoded video data stored in the input buffer at predetermined intervals and deleting a part of the decoded video data in a frame unit.

The video output method according to claim 1, wherein the deleting step comprises: checking whether video data stored in the input buffer is present within a predetermined time based on a time point at which rendering is started and taken from the input buffer; And deleting the decoded video data present in the input buffer as a result of the checking.

In addition, in the video output method in consideration of the latency, the deleting step may include: checking whether the input buffer has decoded video data of two or more frames; And selectively deleting some of the decoded video data in the case of two or more frames of video data.

In addition, the video output method considering the latency includes: receiving the encoded video data through a communication network in units of frames in accordance with the reproduction time; And decoding the received encoded video data and storing the decoded video data in the input buffer.

In addition, the present invention provides a computer-readable recording medium on which a program for executing the video output method in consideration of the latency described above is recorded.

The present invention decodes encoded video data and constructs an output image through rendering. When unrendered and decoded video data is accumulated over a predetermined number of frames, the decoded video data is partially accumulated, Can be solved.

Particularly, according to the present invention, when the decoded video data is delayed due to the rendering process being slower than the decode process, a part of the video data that can not be selectively processed at a level that does not deteriorate the video quality is removed, And it is possible to minimize the time from the user request to the response in the cloud streaming service.

1 is a block diagram of a cloud streaming service system according to the present invention.
2 is a block diagram illustrating a configuration of a cloud streaming service apparatus in a cloud streaming service system according to the present invention.
3 is a block diagram illustrating a configuration of a user apparatus according to an exemplary embodiment of the present invention.
4 is a block diagram illustrating a video output apparatus according to an embodiment of the present invention.
5 and 6 are views for explaining a video output method according to the present invention.

Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings. In the following description and the accompanying drawings, detailed description of well-known functions or constructions that may obscure the subject matter of the present invention will be omitted. It should be noted that the same constituent elements are denoted by the same reference numerals as possible throughout the drawings.

In addition, terms and words used in the present specification and claims should not be construed to be limited to ordinary or dictionary meanings, and the inventor shall appropriately define it as a concept of terminology for explaining its own invention in the best way. It should be construed as meaning and concept consistent with the technical idea of the present invention. Therefore, the embodiments described in the present specification and the configurations shown in the drawings are merely the most preferred embodiments of the present invention, and not all of the technical ideas of the present invention are described. Therefore, It is to be understood that equivalents and modifications are possible.

In addition, when referring to an element as being "connected" or "connected" to another element, it means that it can be connected or connected logically or physically. In other words, it is to be understood that although an element may be directly connected or connected to another element, there may be other elements in between, or indirectly connected or connected.

Also, the terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention. The singular expressions include plural expressions unless the context clearly dictates otherwise. It is also to be understood that the terms such as " comprising "or" having ", as used herein, are intended to specify the presence of stated features, integers, It should be understood that the foregoing does not preclude the presence or addition of other features, numbers, steps, operations, elements, parts, or combinations thereof.

In the process of decoding encoded video data and outputting it as an image, since the rendering speed is lower than the decoding speed and the rendering process can not be performed, a part of the accumulated decoded video data is deleted so that the decoded video And can be applied to devices and services including a video output function, which is intended to reduce unnecessary waiting time due to accumulation of data.

For example, the present invention can be applied to a cloud streaming service in which a game, a VOD, a real-time broadcast content, and the like are executed on the server side and then the execution screen is transmitted to the user device in a streaming manner, have.

Hereinafter, a video output method considering latency according to the present invention, an apparatus for performing the video output, and a cloud streaming service system will be described as an example of application to a cloud streaming service.

1 is a block diagram of a cloud streaming service system to which the present invention is applied.

Referring to FIG. 1, a cloud streaming service system to which the present invention is applied may include a cloud streaming service apparatus 100 connected through a communication network 10, and one or more user apparatuses 200 and 200 '. At this time, the one or more user devices 200 and 200 'include a user device 200 having video output means (e.g., a touch screen, an LCD display, etc.) and a user device 200' The user device 200 'having no video output means can be connected to the video output device 300 through a predetermined connection interface.

The cloud streaming service system to which the present invention is applied is implemented by providing a streaming method after executing user-requested contents such as game, VOD, real-time broadcasting, etc. using cloud computing technology.

For this purpose, the cloud streaming service apparatus 100 executes or reproduces predetermined content according to a request from the user apparatus 200, encodes video data and audio data corresponding to the result, To the device (200). At this time, as described above, the content may include at least one of a game application, a VOD, a real-time broadcast, and a GUI (Graphic User Interface) for user input.

In particular, in an embodiment of the present invention, in transmitting the encoded video data and audio data to the user device 200, the cloud streaming service device 100 may be configured to transmit the encoded content of the content requested by the user device 200 Video data and audio data in synchronism with each other, and sequentially transmitting the encoded video data and audio data on a frame-by-frame basis in accordance with the reproduction time of the user apparatus 200, It is possible to decode and output the audio signal as it is received without synchronization processing between audio signals. As a result, it is possible to minimize the waiting time of the user by reducing the time required for the demultiplexing processing and the synchronization processing.

That is, in the present invention, the cloud streaming service apparatus 100 encodes the video data and audio data on a frame-by-frame basis, and transmits the encoded video data and audio data to the user device 200 as a bit stream without multiplexing processing .

In addition, the cloud streaming service apparatus 100 sequentially transmits the coded video data and audio data for each frame in accordance with the playback time of the user apparatus 200. For example, when the compression rate of the moving picture is 30 frames per second, the cloud streaming service apparatus 100 may transmit the encoded video data and audio data at a cycle of about 33 msec in accordance with the reproduction time. In addition, it is preferable that the cloud streaming service apparatus 100 transmits the video data and audio data to be transmitted in synchronization with each other.

The one or more user devices 200 and 200 'access the cloud streaming service apparatus 100 through the communication network 10 and receive the encoded video data and audio data transmitted from the cloud streaming service apparatus 100 And the like.

These user devices 200 and 200 'basically receive a bitstream including encoded video data and can decode the video data.

Particularly, the user apparatuses 200 and 200 'access the cloud streaming service apparatus 100 and receive video data and audio data encoded frame by frame from the cloud streaming service apparatus 100 in accordance with the reproduction time, And decode the received video data and audio data as received without demultiplexing and synchronization processing. This is due to transmission of video data and audio data encoded on a frame-by-frame basis in accordance with the playback timing in the cloud streaming service apparatus 100, and it is possible to further reduce the waiting time by reducing the time required for demultiplexing and synchronization.

In addition, the user device 200 including video output means, for example, an LCD display, etc., can render the decoded video data and output an image.

At this time, when performing rendering, the user device 200 confirms whether two or more video data to be rendered, i.e., decoded video data, is waiting and if there are two or more, the user device 200 selectively deletes some of the video data, Thereby reducing the latency occurring during the rendering process.

On the other hand, the user apparatus 200 'not including the video output means performs only the function of receiving and decoding the bit stream including the encoded video data, and transmits the decoded video data to the connected video output apparatus 300 And output it through the video output apparatus 300. [ In this case, the decoding process of the decoded video data may be performed through the video output apparatus 300.

At this time, the video output apparatus 300 confirms whether two or more buffered decoded video data transmitted from the user apparatus 200 'are waiting, and when two or more waiters are waiting, a part of the decoded video data is selectively deleted , Thereby reducing the latency occurring during the rendering process.

In such a cloud streaming service system, the communication network 10 for connecting the cloud streaming service apparatus 100 and the user apparatuses 200 and 200 'to transmit the data may include one or more And a communication method. For example, the communication network 10 may be a radio communication such as a wireless LAN (WLAN), a Wi-Fi, a Wibro, a WiMAX, a High Speed Downlink Packet Access (HSDPA) At least one of wired communication methods such as mobile communication such as Ethernet, Terminology, Term Evolution, Ethernet, xDSL (ADSL, VDSL), Hybrid Fiber Coax (HFC), Fiber to the Curb (FTTC) Or a combination of communication networks according to FIG.

Therefore, the user devices 200 and 200 'for processing the above-described cloud streaming service can be any of various information communication devices used by the user, as long as they can communicate through the communication network 10. For example, the user device 200 may be applied to various terminals such as a multimedia terminal, a wire terminal, a fixed terminal, and an IP (Internet Protocol) terminal.

In particular, the user device 200 having the video output means can be used not only for a mobile communication terminal such as a mobile phone and a smart phone but also for a portable multimedia player (PMP), a mobile internet device (MID), a desktop PC, PCs, notebooks, netbooks, smart TVs that support Internet services through the network 10 in addition to broadcast reception functions, and the like.

On the other hand, the user apparatus 2003 without the video output means can be a set-top box that is connectable to the communication network 10 and receives IPTV broadcast data and reproduces the IPTV broadcast data through the TV.

Next, the configuration and operation of each device of the cloud streaming service system according to the present invention will be described in more detail with reference to FIG. 2 to FIG.

2 is a block diagram illustrating a configuration of a cloud streaming service apparatus according to the present invention.

2, a cloud streaming service apparatus 100 according to the present invention includes a service communication unit 110, a streaming control unit 120, a coding unit 130, a service storage unit 140, (150), and a cloud computing processing unit (160).

The service communication unit 110 is configured to communicate with one or more user devices 200 and 200 'through a communication network 10. The service communication unit 110 may transmit and receive data based on one or more transmission protocols.

Specifically, the service communication unit 110 forms a communication channel with the user device 200 or 200 'requesting the service, identifies the encoded video data and audio data of the requested content through the formed communication channel, (200, 200 '). More specifically, in one embodiment of the present invention, the service communication unit 110 forms two communication channels with the user device 200 and 200 ', and then transmits the encoded video data and audio data to the respective communication Channel. ≪ / RTI > At this time, since the communication channel through which the video data is transmitted and the communication channel through which the audio data is transmitted are distinguished, the user device 200 can distinguish the video data from the audio data based on the communication channel.

In addition, in another embodiment of the present invention, the service communication unit 110 transmits an identifier for distinguishing a data type, for example, video data and audio data, together with the encoded video data and audio data. In this case, the user device 200 or 200 'can distinguish the video data from the audio data using the identifier included in the received data.

Next, the streaming control unit 120 is a configuration for performing transmission processing in the cloud streaming service. Specifically, the streaming control unit 120 transmits the encoded video data of the content requested by the user device 200 or 200 'according to a service request of the user device 200 or 200' connected thereto through the service communication unit 110 And audio data to the user device 200. In particular, the streaming control unit 120 according to the present invention transmits the encoded video data and audio data in synchronization with each other to the user apparatuses 200 and 200 ', and transmits the encoded video data and audio data in advance , And sequentially transmits the encoded video data and audio data on a frame-by-frame basis in accordance with the reproduction time of the user apparatus (200, 200 '). For example, when the moving picture having a compression rate of 30 fps is to be transmitted, the streaming control unit 120 transmits the encoded video data and audio data corresponding to a specific frame, and after 33 msec, Data can be transmitted. In particular, the streaming control unit 120 according to the present invention transmits the encoded bit stream as it is without packetizing and multiplexing the encoded video data and audio data.

In addition, the streaming control unit 120 may include an identifier capable of distinguishing a data type from the encoded video data and audio data so that the data received from the user device 200 or 200 'may be video data or audio data have.

Next, the encoding unit 130 is configured to generate the encoded video data and audio data. That is, the encoding unit 130 encodes the video data and the audio data of the content requested by the user apparatuses 200 and 200 'according to a predetermined compression protocol. At this time, the compression methods of the video data and the audio data may be different from each other. The encoding unit 130 may receive video data and audio data from at least one of the service storage unit 140, the real-time broadcast receiving unit 150, and the cloud computing processing unit 160 to perform encoding .

The service storage unit 140 stores data and programs necessary for the operation of the cloud streaming service apparatus 100 according to the present invention and may store programs and data for service execution. More specifically, the service storage unit 140 may store content data to be served to the user device 200, 200 '. In this case, the content data may include video data and audio data before being encoded, and the content data may include encoded video data and audio data.

In addition, the cloud streaming service apparatus 100 according to the present invention can serve contents broadcasted in real time to the user apparatuses 200 and 200 '.

To this end, the real-time broadcast receiver 150 can receive contents broadcasted in real time from an external device, for example, a broadcast service device such as CATV or IPTV.

In addition, as described above, the cloud streaming service is a service for transmitting or receiving the result (video data and audio data) to the user apparatuses 200 and 200 'after executing or playing back the content requested by the user on the server side .

To this end, the cloud streaming service apparatus 100 according to the present invention may further include a cloud computing unit 160. [

The cloud computing processor 160 executes or plays the content requested by the user device 200 or 200 'on behalf of the user device 200 or 200', and outputs the corresponding video data and / or audio data. At this time, the cloud computing processor 160 may execute or play the content according to an input signal transmitted from the user device 200 or 200 '.

For example, when the content requested by the user device 200 or 200 'includes an application such as a game, the content is executed by the cloud computing processing unit 160, and the video data and audio data output as a result of the execution are encoded And transmits the encoded data to the user apparatuses 200 and 200 'through the streaming control unit 120. [

As another example, when the content requested by the user apparatus 200 or 200 'includes real-time broadcasting, the corresponding content is received by the real-time broadcast receiving unit 150, encoded through the encoding unit 130, 120 to the user device 200, 200 '.

In addition, the contents requested by the user apparatuses 200 and 200 'may include one or more contents of different kinds. For example, a real-time broadcast and chat application. The cloud computing processing unit 160 may mix output of one or more contents to the service storage unit 140 and output the mixed output.

Accordingly, the cloud streaming service apparatus 100 includes a service communication unit 100, a streaming control unit 120, a coding unit 130, a service storage unit 140, a real-time broadcast receiving unit 150, The cloud processing apparatus 100 may provide a cloud streaming service according to the present invention through interworking between two or more configurations of the computing processor 160. [

The user apparatuses 200 and 200 'receiving and outputting the encoded video data and audio data corresponding to the user request from the cloud streaming service apparatus 100 may be configured as shown in FIG. 3 or FIG. 3 and 4 show only the configuration related to the present invention, and the user device 200, 200 'may include other configurations than those shown. 3 and 4 may be implemented by software, hardware, or a combination of software and hardware, and the same configuration that performs the functions described below regardless of its implementation form or name may be understood .

In particular, FIG. 3 shows a configuration of a user device 200 having a video output means among user devices according to the present invention, FIG. 4 shows a configuration of a user device 200 'and a video output device 300 shown in FIG.

3, the user apparatus 200 having the video output unit may include a network receiving unit 210, a decoding unit 220, a rendering unit 230, and a control unit 240).

The network receiving unit 210 is a configuration for accessing the cloud streaming service apparatus 100 via the communication network 10 to receive data.

The network receiving unit 210 receives a bit stream corresponding to a user request from the cloud streaming service apparatus 100, and the bit stream may include encoded video data and audio data.

More specifically, the network receiving unit 210 can receive the video data and the audio data, which are coded on a frame-by-frame basis, in accordance with the reproduction time, and divides the received video data and audio data into the decoded data .

Here, the network receiving unit 210 does not perform demultiplexing on the received data but distinguishes whether the received data is video data or audio data, and transmits the data to the decoding unit 220 as it is. In an embodiment of the present invention, the network receiving unit 210 receives the encoded video data and audio data through different channels, distinguishes the encoded video data and audio data through the received channel, In addition to the encoded video data and audio data, an identifier for distinguishing a data type may be further received, and the encoded video data and audio data may be distinguished from each other using the received identifier.

For this operation, the network receiving unit 210 may include a network buffer for separating and storing the encoded video data and the audio data received through the communication network 10.

The decoding unit 220 is configured to decode the encoded video data and audio data according to respective decoding algorithms. For this, the decoding unit 220 may include an audio decoding module (not shown) and a video decoding module (not shown). In particular, the decoding unit 220 decodes the video data and the audio data transmitted from the network receiving unit 210 as they are received without performing a separate synchronization process. Since the encoded video data and audio data received by the network receiving unit 210 are transmitted in accordance with the reproduction time on a frame-by-frame basis, the video decoding unit 220 and the audio decoding unit 230, Data and audio data are synchronized and can be output at the time of playback.

The video data decoded by the decoding unit 220 is transmitted to the rendering unit 230 on a frame basis and is stored in the input buffer 231 of the rendering unit 230.

Next, the rendering unit 230 is configured to render the decoded video data from the decoding unit 220 and construct an image to be output on the screen. More specifically, the rendering unit 230 sequentially fetches and decodes the decoded video data stored in the input buffer 231 on a frame-by-frame basis.

Conventionally, rendering refers to a process of composing an image having a predetermined resolution from image data through a computer program. The rendering process calculates the digital image data of the primary colors represented by the image output means, And the output means constitutes an output image corresponding to the image data.

The control unit 240 controls the decoding and rendering process described above in the present invention so that the control unit 240 controls the decoding unit 230 to decode the at least one of the two or more decoded video data stored in the input buffer 231 of the rendering unit 230 By selectively removing some of them, the latency due to the delay of the rendering process is reduced. The specific control method by the control unit 240 will be described later in detail with reference to the flowcharts of FIGS. 5 and 6. FIG.

The user device 200 configured as described above can reduce the latency in the rendering step by generating the decoding process faster than the rendering process in the process of decoding and outputting the video data, It is possible to minimize the problem that the video data decoded due to the processing delay of the video data is queued.

Meanwhile, in another embodiment of the present invention, the user device 200 'does not include a video output means and outputs an image through the video output device 300.

4, a user apparatus 200 'having no video output means according to another embodiment of the present invention includes a network receiving unit 210, a decoding unit 220, a connection interface unit 250).

That is, the user apparatus 200 'according to another embodiment of the present invention may be connected to the video output apparatus 300 through the connection interface unit 250 without including the rendering unit 230.

Here, the processing of the network receiving unit 210 and the decoding unit 220 operates in the same manner as in the embodiment of the present invention, and therefore, a description thereof will be omitted.

However, the video data and the audio data decoded by the decoding unit 220 are output to the video output apparatus 300 through the connection interface unit 250 for video output.

The connection interface unit 250 is a structure for connecting data between the user device 200 'and the video output device 300 and transmits an interface of a predetermined standard, for example, a high definition multimedia interface (HDMI) DVI (Digital Visual Interface) or the like.

At this time, the video output apparatus 300 means a device having a video output function, and may be, for example, a monitor, a TV, a projector, or the like.

As shown in FIG. 4, the video output apparatus 300 includes a connection interface unit 310, a rendering unit 320, and a rendering unit 330. The video output unit 300 includes a connection interface 310, 320, and a control unit 330. The control unit 330 may be a microcomputer.

The connection interface unit 310 is connected to the user equipment 200 according to the same interface standard as the connection interface unit 250 included in the user equipment 200 'and is configured to transmit and receive control signals and data . Specifically, the connection interface unit 310 can receive the decoded video data from the user device 200 '.

The decoded video data transmitted through the connection interface unit 310 is sequentially stored in the input buffer 321 of the rendering unit 320 and waits until it is rendered.

The rendering unit 320 reads and decodes the decoded video data stored in the input buffer 321 on a frame-by-frame basis to form an output image.

At this time, the control unit 330 checks the decoded video data stored in the input buffer 321 of the rendering unit 320, and deletes some of the stored video data when two or more are accumulated. More specifically, the control unit 330 manages the input buffer 321 such that only a certain number of frames (for example, one frame) exist in the input buffer 321 of the rendering unit 320.

According to the above description, even when the user apparatus 200 'outputs an image through the other image output apparatus 300, the problem that the video data decoded by the image output apparatus 300 is cumulatively accumulated and the latency increase Can be solved.

Hereinafter, a video output method in consideration of latency according to the present invention will be described in more detail.

5 is a flowchart illustrating an image output method in consideration of latency according to an embodiment of the present invention.

5 shows an image output process performed by an apparatus including a video output function (means), which is performed by the user apparatus 200 shown in FIG. 3 or the video output apparatus 300 shown in FIG. 4 .

5, the rendering unit 230 of the user device 200 including the video output unit decodes the decoded video data, which is decoded by the decoding unit 230 and then stored in the input buffer 231, And performs a rendering process (S105). Since the video data to be rendered in this way is deleted in the input buffer 231, only the decoded video data waiting for the rendering process is stored in the input buffer 231.

The control unit 240 of the user device 200 or the control unit 330 of the video output device 300 may be configured to allow the rendering units 230 and 320 to decode the decoded video data stored in the input buffers 231 and 321 After the data is fetched, it is checked whether the decoded video data waiting in the input buffers 231 and 321 remain (S110).

If the decoding and rendering processes are performed at almost the same rate, while the rendering units 230 and 320 take the video data of the next frame and perform rendering, the decoding unit 230 decodes the video data of the next frame And transmitted to the input buffers 231 and 321. Accordingly, at the time when the rendering units 230 and 320 take decoded video data stored in the input buffers 231 and 321 for rendering for the next frame, the input buffers 231 and 321 are supplied with different decoded video The data does not exist.

In contrast, when the decoding process is performed faster than the rendering process, a plurality of video data is accumulated in the input buffers 231 and 321, and the rendering units 230 and 320 output the input buffer 231 , And 321, the input buffers 231 and 321 may have one or more decoded video data.

Accordingly, the control unit 240 or 330 may transmit the video data of the other frame to the input buffer 231 or 321 within a predetermined time from the time when the rendering unit 230 or 320 starts rendering of the decoded video data of the specific frame, (Step S115). As a result, if there are other frame video data remaining in the input buffers 231 and 321, the frame buffer deletes the frame video data. At this time, the control unit 240 or 330 may delete all of the video data remaining in the input buffers 231 and 321, or may selectively delete some of the video data. In the case of selective deletion, the control unit 240 or 330 may select and delete non-consecutive frames so as not to deteriorate the quality of the image output on the screen. In addition, the control unit 240 and 330 can limit the number of frames to be deleted at the same time, thereby minimizing degradation in image quality.

On the other hand, if there is no video data remaining in the input buffers 231 and 321 as a result of the check in step S110, since no rendering processing delay has occurred, step S115 is not performed.

The above-described processes (steps S105 to S115) are repeatedly executed until the video output ends (S120), and the latency due to the rendering process delay is not generated during the video output.

6 is a flowchart illustrating an image output method in consideration of latency according to another embodiment of the present invention.

The video output process shown in FIG. 6 may also be performed by the user apparatus 200 shown in FIG. 3 or the video output apparatus 300 shown in FIG.

Referring to FIG. 6, the rendering unit 230 of the user apparatus 200 including the video output unit decodes the decoded video data, which is decoded by the decoding unit 230 and stored in the input buffer 231, And performs a rendering process (S205). Since the video data to be rendered in this manner is deleted in the input buffer 231, only the decoded video data waiting for the rendering process is stored in the input buffer 231.

The control unit 240 of the user device 200 or the control unit 330 of the video output device 300 monitors the states of the input buffers 231 and 321 of the rendering units 230 and 320 and controls the input buffer 231 , 321) whether the decoded video data of two or more frames remains (S210).

When the decoding and rendering processes are performed at almost the same speed, while the rendering units 230 and 320 take the next frame of video data and perform rendering, the decoding unit 230 decodes the video data of the next frame Decoding is performed and transferred to the input buffers 231 and 321. Accordingly, the input buffers 231 and 321 have decoded video data of one frame or less.

On the other hand, when the decoding process is performed faster than the rendering process, the input buffers 231 and 321 accumulate a plurality of video data, and there may exist decoded video data of two or more frames.

Accordingly, if it is determined in step S210 that there is more than two frames of video data in the input buffers 231 and 321, the control units 240 and 330 transmit the video data stored in the input buffers 231 and 321 The remaining frames are deleted except for the frame S215). At this time, the control unit 240 and 330 may select and delete non-consecutive frames so as not to deteriorate the quality of the image output on the screen. In addition, the control unit 240 and 330 can limit the number of frames to be deleted at the same time, thereby minimizing degradation in image quality.

On the other hand, if it is determined in step S210 that video data of one frame or less exists in the input buffers 231 and 321, the rendering processing delay does not occur, and therefore, step S215 is not performed.

The above-described processes (steps S205 to S215) are repeatedly executed until the video output ends (S220), and the latency due to the rendering process delay is managed while the video output is performed.

The video output method considering the above latency can be implemented in software form readable by various computer means and recorded in a computer readable recording medium. Here, the recording medium may include program commands, data files, data structures, and the like, alone or in combination. Program instructions to be recorded on a recording medium may be those specially designed and constructed for the present invention or may be available to those skilled in the art of computer software. For example, the recording medium may be an optical recording medium such as a magnetic medium such as a hard disk, a floppy disk and a magnetic tape, a compact disk read only memory (CD-ROM), a digital video disk (DVD) Includes a hardware device that is specially configured to store and execute program instructions such as a magneto-optical medium such as a floppy disk and a ROM, a random access memory (RAM), a flash memory, do. Examples of program instructions may include machine language code such as those generated by a compiler, as well as high-level language code that may be executed by a computer using an interpreter or the like. Such a hardware device may be configured to operate as one or more software modules to perform the operations of the present invention, and vice versa.

In addition, the processor installed in the cloud streaming service apparatus 100, the user apparatus 200, and the video output apparatus 300 according to the present invention can process program instructions for executing the method according to the present invention. In one implementation, the processor may be a single-threaded processor, and in other embodiments, the processor may be a multithreaded processor. Further, the processor is capable of processing instructions stored on a memory or storage device.

The cloud streaming service apparatus 100, the user apparatus 200, and the video output apparatus 300 according to the present invention can be driven by an instruction to cause one or more processors to perform the functions and processes described above. Such instructions may include, for example, interpreted instructions such as script commands, such as JavaScript or ECMAScript commands, or other instructions stored in executable code or computer readable media. Further, the apparatus according to the present invention may be implemented in a distributed manner across a network, such as a server farm, or may be implemented in a single computer device.

Although the present specification and drawings describe exemplary device configurations, the functional operations and subject matter implementations described herein may be embodied in other types of digital electronic circuitry, or alternatively, of the structures disclosed herein and their structural equivalents May be embodied in computer software, firmware, or hardware, including, or in combination with, one or more of the foregoing. Implementations of the subject matter described herein may be embodied in one or more computer program products, i. E. One for computer program instructions encoded on a program storage medium of the type for < RTI ID = 0.0 & And can be implemented as a module as described above. The computer-readable medium can be a machine-readable storage device, a machine-readable storage substrate, a memory device, a composition of matter that affects the machine readable propagation type signal, or a combination of one or more of the foregoing.

A computer program (also known as a program, software, software application, script, or code) that is loaded on an apparatus according to the present invention and which implements the method according to the present invention may be a compiled or interpreted language, a program containing a priori or procedural language Language, and may be deployed in any form including standalone programs or modules, components, subroutines, or other units suitable for use in a computer environment. A computer program does not necessarily correspond to a file in the file system. The program may be stored in a single file provided to the requested program, or in multiple interactive files (e.g., a file storing one or more modules, subprograms, or portions of code) (E.g., one or more scripts stored in a markup language document). A computer program may be deployed to run on multiple computers or on one computer, located on a single site or distributed across multiple sites and interconnected by a communications network.

Implementations of the subject matter described herein may include, for example, a back-end component such as a data server, or may include a middleware component, such as an application server, or may be a web browser or a graphical user, for example a user, who may interact with an implementation of the subject- Front-end components such as client computers with interfaces, or any combination of one or more of such back-end, middleware, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication, such as, for example, a communications network.

While the specification contains a number of specific implementation details, it should be understood that they are not to be construed as limitations on the scope of any invention or claim, but rather on the description of features that may be specific to a particular embodiment of a particular invention Should be understood. Certain features described herein in the context of separate embodiments may be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented in multiple embodiments, either individually or in any suitable subcombination. Further, although the features may operate in a particular combination and may be initially described as so claimed, one or more features from the claimed combination may in some cases be excluded from the combination, Or a variant of a subcombination.

Likewise, although the operations are depicted in the drawings in a particular order, it should be understood that such operations must be performed in that particular order or sequential order shown to achieve the desired result, or that all illustrated operations should be performed. In certain cases, multitasking and parallel processing may be advantageous. Also, the separation of the various system components of the above-described embodiments should not be understood as requiring such separation in all embodiments, and the described program components and systems will generally be integrated together into a single software product or packaged into multiple software products It should be understood.

It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory and are intended to be exemplary and explanatory only and is not restrictive of the invention, It is self-evident to those who have knowledge. Furthermore, although specific terms are used in this specification and the drawings, they are used in a generic sense only to facilitate the description of the invention and to facilitate understanding of the invention, and are not intended to limit the scope of the invention.

Therefore, the scope of the present invention should not be limited by the described embodiments but should be defined by the claims.

The present invention decodes encoded video data and constructs an output image through rendering. When unrendered and decoded video data is accumulated over a predetermined number of frames, the decoded video data is partially accumulated, Can be solved.

Particularly, according to the present invention, when the decoded video data is delayed due to the rendering process being slower than the decode process, a part of the video data that can not be selectively processed at a level that does not deteriorate the video quality is removed, So that the time from the user request to the response in the cloud streaming service can be minimized.

10: communication network 100: cloud streaming service device
110: service communication unit 120: streaming control unit
130: encoding unit 140:
150: real-time broadcast receiver 160: cloud computing processor
200, 200 ': user device 210: network receiver
220: decoding unit 230: rendering unit
240: control unit 250: connection interface unit
300: video output apparatus 310: connection interface unit
320: a rendering unit 330:

Claims (12)

A network receiving unit for receiving a bitstream including encoded video data;
A decoding unit decoding the bit stream received by the network receiving unit;
A rendering unit that stores decoded video data output from the decoding unit in an input buffer, sequentially fetches decoded video data stored in an input buffer on a frame basis, and sequentially composes the decoded video data to form an output image; And
The rendering unit fetches the decoded video data from the input buffer and determines whether other decoded video data stored in the input buffer exists within a predetermined time based on a time point at which the rendering is sequentially started, And a control unit for deleting the video data. delete The apparatus of claim 1, wherein the control unit
And if the decoded video data of two or more frames exists, selectively deletes some of the decoded video data in the input buffer. The method according to claim 1,
The network receiving unit receives coded video data transmitted from the cloud streaming service apparatus on a frame-by-frame basis in accordance with the reproduction time, transfers the coded video data to the decoding unit,
Wherein the decoding unit decodes the encoded video data as it is received. A connection interface unit connected to the user apparatus to transmit and receive control signals and video data;
A rendering unit that stores decoded video data received from the user equipment through the connection interface unit in an input buffer, sequentially fetches decoded video data stored in an input buffer on a frame basis, and sequentially rendering the decoded video data; And
If the decoded video data is stored in the input buffer by checking whether the decoded video data is stored in the input buffer within a predetermined time based on a time point at which the decoded video data is fetched from the input buffer and the rendering is sequentially started, And a control unit for deleting the decoded video data. A user apparatus for receiving coded video data and decoding the coded video data at a predetermined decoding rate;
A decoding unit for decoding the decoded video data stored in the input buffer and decoding the decoded video data stored in the input buffer for each frame and sequentially rendering the decoded video data; And a video output device for deleting the other decoded video data when it is confirmed whether other decoded video data is stored in the input buffer within a predetermined time based on the starting point of time. The method according to claim 6,
Further comprising a cloud streaming service device for sequentially transmitting the encoded video data of the content requested by the user device on a frame-by-frame basis in accordance with a playback time of the user device. An apparatus including a video output function,
Fetching the decoded video data stored in the input buffer in units of frames and sequentially rendering the decoded video data; And
Confirming whether video data stored in the input buffer is present within a predetermined time based on the starting time of the rendering; And
And deleting the decoded video data present in the input buffer as a result of the determination. delete 9. The method of claim 8,
Confirming whether or not two or more frames of decoded video data exist in the input buffer; And
And selectively deleting a part of the decoded video data if there is decoded video data of two or more frames. 9. The method of claim 8,
Receiving encoded video data through a communication network in a frame unit in accordance with a reproduction time;
And decoding the received encoded video data and storing the decoded video data in the input buffer. A program for executing a video output method in consideration of latency as set forth in any one of claims 8 to 10.

Images