JP2022042997A - Method, server, and computer-readable recording medium for minimizing delay in real time live streaming - Google Patents

Abstract

To provide a method, a system and a computer-readable recording medium for minimizing delay in real time live streaming.SOLUTION: With respect to a content received from a streamer in a real time live streaming environment, two or more groups of pictures (GOP) are cached and in response to a request by a client, transmission to the client is started from a GOP completed by caching as a packet starting from an I-frame.SELECTED DRAWING: Figure 5

Description

The following description relates to technologies that improve the quality experienced by users in a real-time livestreaming environment.

Real-time livestreaming service means a technology that processes various transmitted multimedia data like a continuous stream of water.

Real-time livestreaming technology is becoming more important as the Internet grows, partly because most users have fast enough connections to instantly download large volumes of content. Because.

When the streaming technology is used in such a situation, the client player can start displaying the data even before all the files have been transmitted.

For example, Patent Document 1 (publication date: June 21, 2006) discloses a technique for providing a real-time streaming service in consideration of a transmission rate between a streaming server and a client.

Korean Published Patent No. 10-2006-0068547 Gazette

As one of the qualities that users experience in a real-time live streaming platform, we provide a method and system for minimizing the delay time from the time when content on the platform is selected to the time when it is output to the actual screen of the player.

A method implemented by a computer-implemented server, said server comprising at least one processor configured to execute a computer-readable instruction contained in memory, said method is said to be at least one. A stage in which at least two or more GOPs (group of computers) are cached for content received from a streamer in a real-time live streaming environment by a processor, and an I-frame at a request of a client by the at least one processor. Provided is a method including a step of transmitting from a GOP completed by a cache to the client as a packet to be started.

A computer-implemented server that includes at least one processor configured to execute computer-readable instructions contained in memory, said at least one processor receiving from a streamer in a real-time live streaming environment. Provided is a server that processes a process of caching at least two or more GOPs for content, and a process of transmitting a GOP completed by caching as a packet starting from an I-frame to the client in response to a client request.

It is a figure which showed the example of the network environment in one Embodiment of this invention. It is a block diagram for demonstrating the internal structure of the electronic device and the server in one Embodiment of this invention. It is a figure which showed the example of the real-time live streaming environment in one Embodiment of this invention. It is a block diagram which showed the example of the component which the processor of a server can include in one Embodiment of this invention. It is a flowchart which showed the example of the method which a server can execute in one Embodiment of this invention. It is an exemplary figure for demonstrating the GOP transmission process by stream cache in one Embodiment of this invention. It is an exemplary figure for demonstrating the GOP transmission process by stream cache in one Embodiment of this invention. It is an exemplary figure for demonstrating the GOP transmission process by stream cache in one Embodiment of this invention. It is an exemplary figure for demonstrating the GOP transmission process by stream cache in one Embodiment of this invention. It is an exemplary figure for demonstrating the GOP transmission process by stream cache in one Embodiment of this invention. It is an exemplary diagram for demonstrating the process of storing video and audio together in one embodiment of the present invention. It is an exemplary diagram for demonstrating the process of reconstructing and transmitting video and audio in packet units in one embodiment of the present invention.

<Outline of the invention>
A method performed by a computer-implemented server, wherein the server comprises at least one processor configured to execute a computer-readable instruction contained in memory, said method. A stage in which at least two or more GOPs (group of computers) are cached for content received from a streamer in a real-time live streaming environment by a processor, and an I-frame at a request of a client by the at least one processor. Provided is a method including a step of transmitting from a GOP completed by a cache to the client as a packet to be started.

According to one aspect, the size of the GOP cache may be determined based on the settings of the administrator associated with the server or the network state between the server and the client.

According to another aspect, the transmission target GOP may be determined based on the settings of the administrator associated with the server or the network state between the server and the client.

Further, according to another aspect, in the transmission step, when there are a plurality of GOPs completed by the cache, the most recently completed GOP may be transmitted to the client based on the connection time point of the client.

According to another aspect, when the transmission stage is completed by the cache and there are a plurality of GOPs, the GOP of any one of the plurality of GOPs depends on the network state between the server and the client. May be transmitted to the client.

Further, according to another aspect, the transmission step may include a step of erasing or skipping some frames depending on the GOP reception state of the client and then transmitting the remaining frames.

According to another aspect, the caching step includes searching for the audio section corresponding to the GOP by analyzing the video stream and the audio stream of the content, and aligning the video data and the audio data with the timeline. It's fine.

According to another aspect, the caching step analyzes the data type for each frame of the video and combines the video data and the audio data with each I-frame and the time stamp of the audio corresponding to the corresponding I-frame. May include a step of grouping.

Further, according to another aspect, in the transmission step, video data and audio data may be reconstructed and transmitted in a form in which data in packet units alternate.

According to yet another aspect, the alternating number of packets may be determined based on the size of the video data and the audio data.

Provided is a computer-readable recording medium in which a program for causing a computer to execute the above method is recorded.

A computer-implemented server that includes at least one processor configured to execute computer-readable instructions contained in memory, said at least one processor receiving from a streamer in a real-time live streaming environment. Provided is a server that processes a process of caching at least two or more GOPs for content, and a process of transmitting a GOP completed by caching as a packet starting from an I-frame to the client in response to a client request.

<Details of the invention>
Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings.

An embodiment of the present invention relates to a technique for supporting optimum QoE (Quality of Service) and QoS (Quality of Service) in a real-time live streaming environment.

The embodiments including the matters specifically disclosed herein are output to the player's actual screen from the time the content on the platform is selected as one of the important qualities that the user feels on the real-time livestreaming platform. The delay time until the end can be minimized.

FIG. 1 is a diagram showing an example of a network environment according to an embodiment of the present invention. The network environment of FIG. 1 shows an example including a plurality of electronic devices 110, 120, 130, 140, a plurality of servers 150, 160, and a network 170. Such FIG. 1 is merely an example for explaining the invention, and the number of electronic devices and the number of servers are not limited as in FIG.

The plurality of electronic devices 110, 120, 130, 140 may be fixed terminals or mobile terminals realized by a computer system. Examples of a plurality of electronic devices 110, 120, 130, 140 include smartphones, mobile phones, navigation systems, PCs (personal computers), notebook PCs, digital broadcasting terminals, PDAs (Personal Digital Assistants), and PMPs (Tablet Multimedia Players). ), Tablets, game consoles, wearable devices, IoT (Internet of Things) devices, VR (Virtual Reality) devices, AR (Augmented Reality) devices, and the like. As an example, FIG. 1 shows a smartphone as an example of the electronic device 110, but in the embodiment of the present invention, the electronic device 110 substantially utilizes a wireless or wired communication method, and another via the network 170. It may mean one of a variety of physical computer systems capable of communicating with electrical devices 120, 130, 140 and / or servers 150, 160.

The communication method is not limited, and not only a communication method using a communication network that can be included in the network 170 (for example, a mobile communication network, a wired Internet, a wireless Internet, a broadcasting network, a satellite network, etc.), but also a device. Short-range wireless communication between them may be included. For example, the network 170 includes a PAN (personal area network), a LAN (local area network), a CAN (campus area network), a MAN (metropolitan area network), a WAN (wise Internet) network, etc. It may include any one or more of the networks. Further, network 170 may include, but is limited to, any one or more of network topologies, including bus networks, star networks, ring networks, mesh networks, star-bus networks, tree or hierarchical networks, and the like. Will not be done.

Each of the servers 150, 160 is realized by one or more computer devices that communicate with a plurality of electronic devices 110, 120, 130, 140 via a network 170 to provide instructions, codes, files, contents, services, and the like. good. For example, the server 150 may be a system that provides the first service to a plurality of electronic devices 110, 120, 130, 140 connected via the network 170, and the server 160 may also be a system that provides the first service to the plurality of electronic devices 110, 120, 130, 140 connected via the network 170. It may be a system that provides a second service to the devices 110, 120, 130, 140. As a more specific example, the server 150 is a computer program installed and executed in a plurality of electronic devices 110, 120, 130, 140, and the server 150 is a service targeted by the application (for example, real-time live). A streaming service or the like) may be provided to a plurality of electronic devices 110, 120, 130, 140 as a first service. As another example, the server 160 may provide a service for distributing the above-mentioned application installation and execution files to a plurality of electronic devices 110, 120, 130, 140 as a second service.

FIG. 2 is a block diagram for explaining an internal configuration of an electronic device and a server according to an embodiment of the present invention. FIG. 2 describes the internal configuration of the electronic device 110 and the internal configuration of the server 150 as an example for the electronic device. Further, the other electronic devices 120, 130, 140 and the server 160 may have the same or similar internal configuration as the above-mentioned electronic device 110 or the server 150.

The electronic device 110 and the server 150 may include memories 211, 221s, processors 212, 222, communication modules 213, 223, and input / output interfaces 214, 224. The memory 211 and 221 are computer-readable recording media having a large permanent capacity such as a RAM (random access memory), a ROM (read only memory), a disk drive, an SSD (sold state drive), and a flash memory. A recording device may be included. Here, even if a permanent large-capacity recording device such as a ROM, SSD, flash memory, or disk drive is included in the electronic device 110 or the server 150 as another permanent recording device that is separated from the memories 211 and 221. good. Further, the memory 211 and 221 have an operating system and at least one program code (for example, a browser installed and executed in the electronic device 110, or installed in the electronic device 110 to provide a specific service. Code for applications etc.) may be recorded. Such software components may be loaded from a computer-readable recording medium separate from the memories 211 and 221. Such other computer-readable recording media may include computer-readable recording media such as floppy (registered trademark) drives, discs, tapes, DVD / CD-ROM drives, and memory cards. In other embodiments, software components may be loaded into memory 211, 221 through communication modules 213, 223, which are not computer readable recording media. For example, at least one program is a computer program installed by a file provided by a file distribution system (eg, server 160 described above) that distributes a developer or application installation file via a network 170 (eg, described above). It may be loaded into the memory 211 or 221 based on the application.

Processors 212 and 222 may be configured to process instructions in a computer program by performing basic arithmetic, logic, and input / output operations. Instructions may be provided to processor 212, 222 by memory 211, 221 or communication modules 213, 223. For example, the processors 212 and 222 may be configured to execute instructions received according to a program code recorded in a recording device such as memories 211 and 221.

The communication modules 213 and 223 may provide a function for the electronic device 110 and the server 150 to communicate with each other via the network 170, and the electronic device 110 and / or the server 150 may provide another electronic device (as an example). , Electronic device 120) or another server (eg, server 160). As an example, a request generated by a processor 212 of an electronic device 110 according to a program code recorded in a recording device such as a memory 211 may be transmitted to a server 150 via a network 170 under the control of a communication module 213. On the contrary, control signals, instructions, contents, files, etc. provided under the control of the processor 222 of the server 150 are received by the electronic device 110 through the communication module 213 of the electronic device 110 via the communication module 223 and the network 170. May be done. For example, control signals, instructions, contents, files, etc. of the server 150 received through the communication module 213 may be transmitted to the processor 212 and the memory 211, and the contents, files, etc. may be further recorded by the electronic device 110. It may be recorded on a medium (permanent recording device described above).

The input / output interface 214 may be a means for an interface with the input / output device 215. For example, the input device may include a device such as a keyboard, mouse, microphone, camera, and the output device may include a device such as a display, speaker, haptic feedback device, and the like. As another example, the input / output interface 214 may be a means for an interface with a device that integrates functions for input and output, such as a touch screen. The input / output device 215 may be composed of an electronic device 110 and one device. Also, the input / output interface 224 of the server 150 may be a means for connecting to the server 150 or for interfacing with a device (not shown) for input or output that the server 150 can include. As a more specific example, a service screen or content configured by using data provided by the server 150 or the electronic device 120 when the processor 212 of the electronic device 110 processes an instruction of a computer program loaded in the memory 211. May be displayed on the display through the input / output interface 214.

Also, in other embodiments, the electronic device 110 and the server 150 may include more components than the components of FIG. However, most prior art components need not be clearly shown in the figure. For example, the electronic device 110 may be realized to include at least a part of the above-mentioned input / output device 215, such as a transceiver, a GPS (Global Positioning System) module, a camera, various sensors, a database, and the like. Other components may be further included. As a more specific example, when the electronic device 110 is a smartphone, an acceleration sensor or gyro sensor, a camera module, various physical buttons, a button using a touch panel, an input / output port, which are generally included in the smartphone, are used. Various components, such as accelerometers for vibration, may be realized to be further included in the electronic device 110.

In the following, a method for minimizing the delay time experienced by the user in a real-time live streaming environment and a specific embodiment of the system will be described.

FIG. 3 is a diagram showing an example of a real-time livestreaming environment according to an embodiment of the present invention.

FIG. 3 is an example of a real-time live streaming environment, and shows a content delivery network (CDN) service environment including a client 310, a CP (Contents Provider) 320, and a server 350. The server 350 may correspond to the server 150 described with reference to FIGS. 1 and 2, and the client 310 and CP320 may be the electronic devices 110, 120, 130, 140 described with reference to FIGS. 1 and 2. Any one of them may be supported.

The CDN service is a service in a real-time live streaming environment, and the content provided by the CP320 corresponding to the distributor is stored in advance on a large number of servers 350 installed under the network of the ISP (Internet Service Provider). Later, when there is a request from the client 310 corresponding to the viewer (viewer), it is operated by a system of transmitting the corresponding content to the client 310.

The player on the client 310 side may receive the content selected by the user on the platform from the server 350, and then output the received content from an output device (display, speaker, or the like) to play the content.

One of the qualities that the user feels in the real-time live streaming environment as described above is the delay time from the time when the content on the platform is selected to the time when the content is actually output to the player's screen.

Embodiments of the present invention may include techniques for minimizing the delay times described above, and may further include techniques for eliminating side effects due to techniques applied to minimize the delay times.

FIG. 4 is a block diagram showing an example of components that can be included in the processor of the server according to the embodiment of the present invention, and FIG. 5 is a block diagram showing the execution by the server according to the embodiment of the present invention. It is a flowchart which showed the example of the possible method.

The server 350 according to the present embodiment serves as a cache server that transmits the content provided by the CP 320 to the client 310 in a real-time live streaming environment. The server 350 may be configured with a real-time livestreaming system realized by a computer. The server 350 may provide the client 310 with a real-time livestreaming service by connecting to a dedicated application installed on the client 310 or a web / mobile site associated with the server 350.

As shown in FIG. 4, the processor 222 of the server 350 has a stream receiving unit 410, a stream cache unit 420, an audio / video analysis unit 430, and a stream transmission as components for executing the real-time live streaming method according to FIG. Part 440 may be included. Depending on the embodiment, the components of processor 222 may be selectively included or excluded from processor 222. Also, depending on the embodiment, the components of processor 222 may be separated or merged to represent the functionality of processor 222.

Such processors 222 and components of the processor 222 may control the server 350 to perform steps 510 to 540 included in the real-time livestreaming method of FIG. For example, the processor 222 and the components of the processor 222 may be implemented to execute an instruction by the code of the operating system included in the memory 221 and the code of at least one program.

Here, the components of the processor 222 may be representations of different functions executed by the processor 222 according to the instructions provided by the program code recorded in the server 350. For example, the stream receiver 410 may be used as a functional representation of the processor 222 that controls the server 350 according to the instructions described above so that the server 350 receives a stream of content from the CP 320.

The processor 222 may read the necessary instructions from the memory 221 in which the instructions related to the control of the server 350 are loaded. In this case, the read instructions may include instructions for controlling the processor 222 to perform steps 510 to 540 described below.

The steps 510-540 described below may be performed in a different order than shown in FIG. 5, and even if some of the steps 510-540 are omitted, additional steps may be included. May be good.

Referring to FIG. 5, at step 510, the stream receiver 410 may receive a stream of content provided by the streamer CP320 on a real-time livestreaming platform.

In step 520, the stream cache unit 420 may cache the video stream in units of GOP (group of pictures) for the content received in step 510. A video is composed of an I-frame (intra frame), a P-frame (predicted frame), and a B-frame (bidirectional frame), and a minimum of one I-frame is required to start playback. .. The stream cache unit 420 may cache the video stream in GOP units in order to respond to the request of the client 310 and transmit the packet starting from the I-frame.

At step 530, the audio-video analysis unit 430 may search for the audio section corresponding to the GOP unit by analyzing the video stream and the audio stream, and save it together with the corresponding video. In order to solve the side effects that may occur when transmitting a video stream starting from an I-frame to the client 310 in units of GOP, the audio-video analysis unit 430 may use the video in units of GOP and the audio corresponding to the corresponding video. You may search for sections and align and store video and audio together.

At step 540, the stream transmission unit 440 may transmit a video stream in units of GOP to the player of the client 310 together with the audio according to the request of the client 310. The stream transmission unit 440 alternately edits the video and the audio in packet units, which is the minimum data transmission unit, in transmitting the cached video in GOP units and the audio in the section corresponding to the video to the player of the client 310. It may be transmitted in the same form. In order to solve the side effect that may occur when the video stream is transmitted in units of GOP, the video and audio may be reconfigured so as to alternate in units of packets and transmitted to the player of the client 310.

In the following, the real-time livestreaming process will be described with reference to specific examples.

In general, a video has a form in which I-frames, P-frames, and B-frames are repeated in a fixed pattern, and may be encoded in one image frame group, that is, in units of GOP. Referring to FIG. 6, one GOP includes an I-frame, a P-frame, a B-frame, and a B-frame and a P-frame are arranged between the I-frame and the I-frame. In other words, a set of continuous frames starting with an I-frame is called a GOP. In some cases, the GOP may be composed of only the I-frame and the P-frame without the B-frame.

The I-frame means a key frame and corresponds to the core frame that is the standard of GOP. All GOP frames always start with an I-frame. An I-frame is a frame that is preserved in its original form without reference to other frames located in front of it. Further, the P-frame is a forward prediction frame, which is a frame in which partial data having a difference from the I-frame located immediately in front is predicted and stored. Finally, the B-frame is a bidirectional prediction frame, which is located between the I-frame and the P-frame, refers to the frames on both sides, and stores the movements of the two frames as guess data. ..

As shown in FIG. 7, most of the time points (@) when the client 310 connects to the server 350 for viewing the content are located in the middle of the GOP generated when the client 310 is connected, and the corresponding points. It is likely that the stream will be transmitted from. For example, if a connection is made while GOP1 is being generated, the client 310 will receive the stream immediately after the connection point (@), but thereafter, a new GOP (GOP2) video start packet, that is, Until the I-frame (I_2) is received, the viewer cannot see anything on the player screen and feels a delay.

In order to minimize such a delay time, the server 350 executes a cache in units of GOP. In this embodiment, the delay time is minimized by transmitting the most recently cached complete GOP as the GOP starting from the I-frame by the video stream cache instead of transmitting from the stream immediately after the connection point of the client 310. Can be planned. By receiving from the I-frame as a video stream in units of GOP, the client 310 can immediately play the video on the player.

In the present embodiment, the size of the cache for the video stream may be specified in consideration of the settings of the server administrator, the network state between the client 310 and the server 350, and the like.

When updating a new GOP in the video stream cache process, the method of discarding the first completed GOP among the cached GOPs is adopted.

Referring to FIG. 8, in the case of cache 701 of size 1, if the client 310 connects before one GOP is completed, that is, if it connects in the middle of the GOP cache, the stream will be streamed until the GOP is completed. It is not communicated and the viewer feels a delay. This is because if one I-frame is connected before it is completed as a GOP, the delay from the time of connection until the GOP is completed and transmitted to the client 310 via the network is displayed as it is. ..

In order to prevent such a problem, in the present embodiment, a cache 702 having a size of 2 or more may be applied so that at least two or more GOPs can be cached. In other words, the size of the GOP cache may be specified to be at least 2 or more, and it is also possible to set which GOP is transmitted based on the connection time point of the client 310.

For example, the size of the GOP cache may be specified as 2, and it may be specified to approach from the last completed GOP. As shown in FIG. 9, assuming that the GOP1 is completed in the cache 702 and the GOP2 is in the cache, even if the client 310 approaches the point b, it is received from the completed GOP GOP1 and immediately played back. Therefore, it is possible to eliminate the delay time until the GOP2 in progress in the cache is completed and transmitted.

Although it is explained that the most recently completed GOP is transmitted based on the connection time point of the client 310, in some cases, the GOP to be transmitted may be determined to be different. For example, referring to FIG. 10, when the size of the GOP cache is 3, GOP1 and GOP2 are completed first in chronological order, and when the client 310 connects to the point c in the situation where the cache is in progress, the client 310 It may be determined from which GOP of GOP1 and GOP2 is transmitted while considering the network state of. As an example, if the current network bandwidth is equal to or more than the threshold value based on the connection time c of the client 310, it is transmitted from the most recently completed GOP2, but if the current network bandwidth is less than the threshold value, it is completed before GOP2. It may be transmitted from the GOP1.

As another example, ABP (Adaptive Bitrate Public) technology for transmitting data according to the network bandwidth may be applied. In a real-time live protocol (eg, RTMP (real time measuring protocol)) environment, the network bandwidth is measured and the bit rate, fps (frame per second), etc. are changed immediately to adapt to the measured bandwidth. Type data transmission technology may be utilized. Similarly, in a real-time live streaming environment using a stream cache, transmission is performed from the most recently completed GOP based on the connection time of the client 310, but the reception status on the client 310 side is good as a result of monitoring the client 310. If the GOP is delayed, some packets (B-frames and / or P-frames) are deleted (drop) from the GOP, or some GOPs are skipped or jumped according to the delivery time of the content. You may apply the method of doing. The length of the frame to be erased (skip or jump) may be variable depending on the monitoring cycle for the client 310. For example, when the client 310 connects to watch a specific program, the client 310 can normally receive the GOP as a primary monitoring result after transmitting from the most recently completed GOP based on the connection time to the client 310 side. If it gets stuck, the frame of the first length is erased. Similarly, if the GOP continues to be delayed in the secondary monitoring result, the frame having only the second length longer than the first length is deleted. In other words, in order to enable continuous viewing on the client 310 side, some frames are erased based on the reception state of the client 310, and then the remaining frames are transmitted. At this time, depending on the monitoring cycle. You may adjust the length of the frame to be erased.

Although the delay time can be minimized by caching and transmitting the video stream in GOP units starting from the I-frame, it may increase the possibility of malfunction of the player on the client 310 side. Since the GOP contains only video information, the audio-video synchronization mechanism built into the player may malfunction. For example, in the case of a method in which one GOP, which is video data, is transmitted first and then audio is transmitted, depending on the size of the video data, it may be considered that there is no audio data from the reception process and playback may start.

To prevent such problems, the processor 222 analyzes the video information (video stream and audio stream) transmitted from the CP320 and not only the GOP but also the audio to be played when one GOP is played. It may be stored in line with the timeline.

Referring to FIG. 11, the data type (I-frame, B-frame, P-frame) is analyzed for each frame of the video in the stream cache process, and each of the I-frames (I_1, I_2, ...) Corresponds to each. The video and audio may be grouped together with the time stamp of the audio (audio_1, audio_2, ...) Corresponding to the I-frame.

Further, when the video and audio are both cached and transmitted to the player by the above-mentioned grouping, the processor 222 may transmit while considering the audio-video synchronization mechanism of the player.

When transmitting all one GOP corresponding to video data and then transmitting audio, or transmitting audio first and then transmitting one GOP, depending on the size of the transmitted video or audio data, the player There may be problems with the audio-video synchronization mechanism.

In order to solve such a problem, the processor 222 may reconstruct and transmit video and audio in a form edited alternately in packet units. Referring to FIG. 12, when the processor 222 transmits the video data and the audio data (audio_1) corresponding to one GOP (GOP1) to the player of the client 310, the video (V) in packet units is aligned with the timeline. And audio (A) may be reconstructed and transmitted in an alternating manner.

In the packet alternating method, video (V) and audio (A) packets may be arranged alternately on a one-to-one basis. As another example, the size of the video (V) and audio (A) data determines the alternating number of packets, which is one of one-to-one, one-to-many, many-to-one, and many-to-many. It is also possible to arrange them alternately with.

By reconstructing and transmitting the video data and the audio data in a form edited alternately in packet units, the player's audio-video synchronization mechanism works without any problem.

The player can receive the I-frame and play the screen. Since it is unlikely that the point at which the user connects corresponds to the I-frame position, the player will always be able to receive from the I-frame by transmitting the most recently completed GOP based on the point of connection. , It can be guaranteed that the reproduction is as quick as possible.

Real-time live protocols like RTMP run on TCP. In order to sufficiently increase the transmission speed with TCP, it is necessary to secure sufficient bandwidth through a slow start process. By transmitting the GOP by the stream cache from the server 350 to the player, the effect (warm-up) of rapidly executing the slow start process can be obtained.

Not only this, the player generally starts playback after the internal buffer is filled with data of a certain size or more for stable playback. Since the buffer can be quickly filled by transmitting the GOP by the stream cache from the server 350 to the player, not only the first screen by the I-frame can be displayed, but also the actual video reproduction after that can be processed quickly. can.

As described above, according to the embodiment of the present invention, in the real-time live streaming environment, the delay time is minimized and the user experiences it by transmitting the video and audio in a form that can be immediately reproduced by the cache in units of GOP to the player. The quality can be improved. Further, according to the embodiment of the present invention, in the cache process for minimizing the delay time, the video information transmitted from the streamer is analyzed to search for the audio to be played together when the video is played. By transmitting video and audio in a form that is alternately edited in packet units during the transmission process after storing both, it is possible to effectively solve the player synchronization problem that may occur as a side effect of the cache layer.

The devices described above may be implemented by hardware components, software components, and / or combinations of hardware components and software components. For example, the apparatus and components described in the embodiments include a processor, a controller, an ALU (arithmetic logic unit), a digital signal processor, a microcomputer, an FPGA (field programgable gate array), a PLU (programmable log unit), a microprocessor, and the like. Alternatively, it may be implemented using one or more general purpose computers or special purpose computers, such as various devices capable of executing and responding to instructions. The processing device may execute an operating system (OS) and one or more software applications running on the OS. The processing device may also respond to the execution of the software, access the data, and record, manipulate, process, and generate the data. For convenience of understanding, one processing device may be described as being used, but one of ordinary skill in the art may include a plurality of processing elements and / or a plurality of types of processing elements. You can understand. For example, the processing device may include multiple processors or one processor and one controller. Also, other processing configurations such as parallel processors are possible.

The software may include computer programs, codes, instructions, or a combination of one or more of these, configuring the processing equipment to operate at will, or instructing the processing equipment independently or collectively. You may do it. The software and / or data is embodied in any type of machine, component, physical device, computer recording medium or device to be interpreted based on the processing device or to provide instructions or data to the processing device. good. The software is distributed on a computer system connected by a network and may be recorded or executed in a distributed state. The software and data may be recorded on one or more computer-readable recording media.

The method according to the embodiment may be realized in the form of program instructions that can be executed by various computer means and recorded on a computer-readable medium. Here, the medium may be a continuous recording of a computer-executable program or a temporary recording for execution or download. Further, the medium may be various recording means or storage means in the form of a combination of a single piece of hardware or a plurality of pieces of hardware, and is not limited to a medium directly connected to a certain computer system, but is distributed over a network. It may exist. Examples of media include hard disks, floppy (registered trademark) disks, magnetic media such as magnetic tapes, optical media such as CD-ROMs and DVDs, optomagnetic media such as floptic disks, and ROMs, RAMs. , Flash memory, etc., and may be configured to record program instructions. Other examples of media include recording media or storage media managed by application stores that distribute applications, sites that supply or distribute various other software, servers, and the like.

As described above, the embodiments have been described based on the limited embodiments and drawings, but those skilled in the art will be able to make various modifications and modifications from the above description. For example, the techniques described may be performed in a different order than the methods described, and / or components such as the systems, structures, devices, circuits described may be in a different form than the methods described. Appropriate results can be achieved even if they are combined or combined, and confronted or replaced by other components or equivalents.

Therefore, even if it is a different embodiment, if it is equivalent to the claims, it belongs to the attached claims.

The embodiment of the present invention further solves the following problems.

In the cache process to minimize the delay time, by analyzing the video information transmitted from the streamer, the audio that should be played together when the video is played is searched, and the corresponding video and audio are saved together. Provides methods and systems for.

Provides a method and system for delivering ready-to-play video and audio by caching user-selected content and minimizing latency, and processing and delivering to prevent synchronization issues in the player. do.

222: Processor 410: Stream receiving unit 420: Stream cache unit 430: Audio-video analysis unit 440: Stream transmitting unit

Claims (20)

It is a method that a server realized by a computer executes.
The server includes at least one processor configured to execute computer-readable instructions contained in memory.
The method is
A step of caching at least two or more GOPs (group of pictures) for content received from a streamer in a real-time live streaming environment by the at least one processor, and a client request by the at least one processor. A method comprising the step of transmitting from a cache-completed GOP to the client as a packet starting from an I-frame. The method of claim 1, wherein the size of the GOP cache is determined based on the settings of the administrator associated with the server or the network condition between the server and the client. The method according to claim 1 or 2, wherein the transmission target GOP is determined based on the setting of the administrator associated with the server or the network state between the server and the client. The stage of transmission is
The invention according to any one of claims 1 to 3, wherein when there are a plurality of GOPs completed by the cache, the most recently completed GOP is transmitted to the client based on the connection time point of the client. the method of. The stage of transmission is
When there are a plurality of GOPs completed by the cache, the claim is characterized in that the GOP of any one of the plurality of GOPs is transmitted to the client depending on the network state between the server and the client. The method according to any one of Items 1 to 3. The stage of transmission is
The method according to any one of claims 1 to 3, comprising a step of erasing or skipping some frames depending on the GOP reception state of the client and then transmitting the remaining frames. The caching stage is
13. the method of. The caching stage is
Claims 1-6, including the step of analyzing the data type for each frame of the video and grouping the video data and the audio data by combining each I-frame with the audio time stamp corresponding to the corresponding I-frame. The method described in any one of the above. The stage of transmission is
The method according to any one of claims 1 to 3, wherein video data and audio data are reconstructed and transmitted in a form in which data in packet units alternate. 9. The method of claim 9, wherein the alternate number of packets is determined based on the size of the video data and the audio data. A computer-readable recording medium in which a program for causing a computer to execute the method according to any one of claims 1 to 10 is recorded. A server realized by a computer
Contains at least one processor configured to execute computer-readable instructions contained in memory.
The at least one processor
In a real-time live streaming environment, the process of caching at least two GOPs for content received from the streamer, and the process of transmitting to the client from the GOP completed by caching as a packet starting from an I-frame in response to a client request. The server that handles the. 12. The server of claim 12, wherein the size of the GOP cache is determined based on the settings of the administrator associated with the server or the network condition between the server and the client. The server according to claim 12 or 13, wherein the transmission target GOP is determined based on the setting of the administrator associated with the server or the network state between the server and the client. The at least one processor
The invention according to any one of claims 12 to 14, wherein when a plurality of GOPs completed by the cache are transmitted from the most recently completed GOP to the client based on the connection time of the client. Server. The at least one processor
When there are a plurality of GOPs completed by the cache, the GOP is transmitted from any one of the plurality of GOPs to the client depending on the network state between the server and the client. The server according to any one of claims 12 to 14. The at least one processor
The server according to any one of claims 12 to 14, wherein some frames are erased or skipped depending on the GOP reception state of the client, and then the remaining frames are transmitted. The at least one processor
One of claims 12 to 17, wherein the audio section corresponding to the GOP is searched by the analysis of the video stream and the audio stream of the content, and the video data and the audio data are aligned according to the timeline. The server described in. The at least one processor
13. The server according to any one of 17 to 17. The at least one processor
The server according to any one of claims 12 to 14, wherein video data and audio data are reconstructed and transmitted in a form in which data in packet units alternate.

Images