KR102489164B1 - Temporal Placement of Rebuffering Events - Google Patents

Abstract

The method includes receiving, by a computing system, data representative of a video item in a buffer. The method further includes outputting the video item from the buffer to a display system. The method further includes determining that the utilization of the buffer falls below a predetermined threshold. The method further includes determining that a designated rebuffering point exists within a predetermined time frame in response to determining that the utilization of the buffer falls below a predetermined threshold. The method further includes pausing the video item at the designated rebuffering point in response to determining, by the computing system, that the designated rebuffering point exists within the predetermined time frame.

Description

Temporal Placement of Rebuffering Events

CROSS REFERENCES TO RELATED APPLICATIONS

This application claims priority to US Patent Application Serial No. 15/610053, filed on May 31, 2017, the disclosure of which is disclosed in its entirety.

The present disclosure relates generally to video streaming, and more specifically to the timing of rebuffering events within a video stream.

While consumers can access media items such as movies and television shows by receiving them over the air or subscribing to a cable or satellite television provider, consumers are increasingly accessing content through Internet-based systems. Some Internet-based systems allow users to download or stream content to various client devices over the Internet. For example, an Internet-based media system may provide content to users via a personal computer, set top box, or personal mobile device such as a smart phone or tablet computer. In particular, streaming media systems enable users to access media content in streaming video format, whereby users can consume (eg, view and/or view and/or listening) can begin. Such a system allows users to access content while avoiding a potentially lengthy download process before beginning to consume the selected content.

When streaming a media item, the user does not have to download the entire media item before being able to view it. Instead, the user can begin consuming the media item almost immediately as soon as the data representing the media is delivered to the user's device. As data representing the media item is delivered to the user's device, it is temporarily placed in a buffer. Buffers allow for a smoother viewing experience because consumption of media items can continue to use the data in the buffer if the network connection between the server through which the data is being streamed and the user's device is temporarily interrupted or slowed. Ideally, the rate at which data is transferred to the buffer is higher than the rate at which data is read from the buffer for display on the client device. However, in some cases, data is read from the buffer faster than it is passed to the buffer. When the buffer utilization (i.e., the amount of data currently in the buffer) drops to zero, the media stream is typically paused until enough data can be delivered to the buffer. This can degrade the quality of the viewer's experience, since this abrupt stop in playback is perceived by the viewer as a frozen screen (sometimes overlaid with an hourglass or spinning wheel icon) and the absence of corresponding audio, i.e. silence. Because. The user will generally not know the amount of data in the buffer, so the user perceives the stopping of playback as a randomly timed event. When enough data is received back into the buffer, playback of the media item can suddenly resume. As a result, discontinuous playback of media items is not pleasant to the user.

1 is a diagram illustrating an exemplary server computing system and client computing system that may be used to perform optimal placement of rebuffering events, in accordance with some embodiments of the present disclosure.
2 is a diagram illustrating designated rebuffering points within a video item, in accordance with some embodiments of the present disclosure.
3A, 3B, 3C and 3D are diagrams illustrating the use of designated rebuffering points for designated rebuffering events, in accordance with some embodiments of the present disclosure.
4A and 4B are tables illustrating various metadata that may be assigned to portions of a video item to help identify designated rebuffering points, in accordance with some embodiments of the present disclosure.
5 is a diagram illustrating detection of features representative of a close-up shot in a portion of a video item, in accordance with some embodiments of the present disclosure.
6 is a diagram illustrating detection of features representative of a establishing shot in a portion of a video item, in accordance with some embodiments of the present disclosure.
7 is a diagram illustrating detection of features indicative of a zoomed-out shot in a portion of a video item, in accordance with some embodiments of the present disclosure.
8 is a flow diagram illustrating an example method for pausing a video item at a designated rebuffering point, in accordance with some embodiments of the present disclosure.
These drawings will be better understood by those skilled in the art by referring to the detailed description below.

With reference to the figures outlined above, exemplary applications of systems and methods according to the present disclosure are described in this section. These examples are provided to add context and aid understanding of the present invention. Accordingly, it will be appreciated by those skilled in the art that the present invention may be practiced without some or all of these specific details. In other instances, well-known process steps or actions have not been described in detail to avoid unnecessarily obscuring the present disclosure. In addition, other applications of the concepts and principles described herein are possible, and thus the following examples should not be considered limiting. The principles and concepts described herein can be applied to select an optimized time for a rebuffering event that can reduce at least some of the negative aspects of rebuffering of streamed media items to viewers.

In the detailed description that follows, reference is made to the accompanying drawings, which form a part of this description and illustrate by way of example certain embodiments of the present disclosure. Although these embodiments are described in sufficient detail to enable one of ordinary skill in the relevant art to practice the invention, these examples are not limiting, and thus other embodiments may be used, and would not depart from the spirit and scope of the invention. It is understood that changes may be made without departing.

As mentioned above, in conventional streaming implementations, if data is not delivered to the client device fast enough, the video must be paused to wait for more data. After a sufficient amount of data has been received, the video can be resumed. Various factors, such as network conditions, can affect a client device's ability to receive data fast enough. If there are, there is often little control over network conditions, so these pauses can appear randomly to the user, often interrupting important story elements. These interruptions are typically perceived as a still screen with no audio. The user may also be presented with an indication of the delay, such as a "buffering" message, a spinning wheel or an hourglass icon. These pauses allowing the buffer to fill with additional data will be referred to as rebuffering events.

Rebuffering events are frustrating for viewers because they disrupt the viewing experience. Rebuffering can be even more undesirable when a video is paused in the middle of an important scene or important dialogue. Accordingly, the present disclosure is directed to methods and systems for optimizing the placement of pauses for rebuffering events as needed to avoid pauses in a video stream at undesirable times. In other words, if rebuffering events are to occur, the rebuffering points can be set to occur at less inconvenient times for the user instead of randomly.

In one example, locations for rebuffering points may be identified within a specific piece of content, such as a video item. These identified locations will be referred to as designated rebuffering points. Designated rebuffering points may correspond to moments or times within a video item that are less inconvenient for rebuffering. For example, designated rebuffering points may correspond to scene changes, shot changes, or pauses in dialogue. In one example, if the buffer utilization becomes too low, the video item may be paused at one of the designated rebuffering points, even if the buffer utilization is not yet below the point at which the rebuffering event is otherwise triggered. In other words, in conventional implementations of video streaming where rebuffering occurs upon buffer exhaustion, the video stream may be paused for rebuffering before this pause for rebuffering occurs. Also, in some examples, if there is a network connection from the streaming server to the client system such that the rate at which data is delivered to the buffer is less than the rate at which the data is consumed, the video item will be displayed at an apparently random location within the video due to exhaustion of data in the buffer. Instead of in points, it can be paused at specific designated rebuffering points.

1 is a diagram illustrating an exemplary server computing system 102 and client computing system 120 that may be used to perform optimal placement of rebuffering events. Server computing system 102 may be one of many servers owned by, operated by, or otherwise managed by a streaming media service. The server computing system 102 includes a processor 108 , memory 104 , a rebuffering point identification module 110 , and a network interface 112 . The server computing system 102 is a stand-alone and enterprise-class server running a server operating system (OS) such as, for example, MICROSOFT® OS, UNIX® OS, LINUX® OS, or other suitable server-based operating system 102. may additionally include. Server computing system 102 may be a server within a content distribution network. The server computing system 102 illustrated in FIG. 1 can be deployed in different ways, and the operations performed and/or services provided by these servers can be combined or separated for a given implementation and more or more It should be appreciated that it may be performed by fewer individual server devices.

Processor 108 may include one or more separate processors or processing cores. Similarly, memory 104 may include one or more memory devices of different types. Memory 104 may store machine readable instructions for execution on processor 108 . These instructions may be associated with various applications as well as the operating system. Memory 104 may also store video items 106 for streaming. These video items 106 may include, for example, full-length movies, episodes of a series, or portions of a movie or episode. Other types of video items are also contemplated. Video items 106 may be encoded in a machine-readable format suitable for storage and/or streaming. More specifically, video items 106 may include a data set specifying a series of frames, each frame containing an image or information that may be processed to obtain an image. When the frames are presented sequentially, they create a video. Video items 106 may be encoded using various techniques for compressing data. Data comprising video items 106 may also include audio data corresponding to the video data. That is, audio data is temporally aligned with video data.

As noted, memory 104 may include a plurality of memory modules. Memory modules can be of various types. A portion of memory 104 may be volatile memory, such as random access memory (RAM). Some of the memory 104 may be non-volatile memory, such as hard disk drives or solid state drives (SSDs). In addition to storing machine readable instructions forming applications, memory 104 may store video items 106 for analysis and streaming. Memory 104 may also store results of this analysis, such as locations or times within video items 106 corresponding to designated rebuffering points.

The server computing system 102 includes a network interface 112 that allows the server computing system 102 to communicate with other computing systems over a network 114, such as the Internet. Network interface 112 includes hardware, software, or a combination of both to encode data for transmission over network 114 . Network interface 112 may utilize various communication technologies such as Ethernet, fiber optic, and wireless communication technologies.

In some examples, server computing system 102 that stores video items 106 can perform analysis of video items 106 to identify rebuffering points. In these examples, server computing system 102 may include rebuffering point identification module 110 . The rebuffering point identification module 110 includes hardware, software, or a combination of the two that analyzes the data representing the video items 106 and identifies the placement of rebuffering points. Identified rebuffering points can be expressed in a variety of ways. In one example, the identified rebuffering points may be represented as points in time within video items 106 (eg, a timestamp including minutes, seconds, and fractions of a second). In some examples, frames corresponding to rebuffering points may be tagged with metadata indicating that these frames correspond to rebuffering points. The identified placement of rebuffering points may be transmitted to the client computing system 120 at the beginning of streaming or during the streaming process, such as in a manifest file or in a metadata file pointed to within the manifest file. The manifest file may also point to the location of more than one stream available to the user for the same audiovisual content, the streams having different bitrates and/or qualities. A client computing system 120 having a high-quality connection over network 114 to server computing system 102 can select a high-quality stream from among a plurality of streams identified in the manifest file. A client computing system 120 having a low quality (eg, low bandwidth) connection may select a low quality stream. When the quality of the connection over the network 114 changes, the client computing system 120 may begin accessing different streams as appropriate to provide the user with an optimal experience. The placement of the rebuffering points may be indicated in the data contained in the streams, such as in individual frames or in a separate file. For example, rebuffering points may be included in Supplemental Enhancement Information (SEI) messages, such as those included by H.264 codecs or other comparable methods of transmitting supplemental information, such as captioning or descriptive text, into a data stream. can be included

In some examples, rebuffering point identification module 110 identifies rebuffering points within video item 106 before video item 106 is streamed. For example, if video item 106 is just stored and is not currently being streamed, rebuffering point identification module 110 can use that time to perform an analysis of video item 106 to determine rebuffering points. there is. In some embodiments, analysis may be performed when video item 106 is being encoded into a compressed format suitable for streaming video from a format optimized for high-fidelity storage. In some examples, rebuffering points may be identified before video item 106 is uploaded to server computing system 102 . In some examples, the rebuffering points are separate servers located within the network 114 between the server computing system 102 that stores the video item 106 and the client computing system 120 that consumes the video item 106. not shown).

The streaming process involves transmitting data representing a selected one of the video items 106 from a server computing system 102 to a client computing system 120 over a network 114 such as the Internet. In this example, client computing system 120 includes processor 128 , memory 122 , network interface 126 , rebuffering point identification module 130 , and display system 132 . In some examples, client computing system 120 may be, for example, a desktop, laptop, video console, or tablet computing device.

Like memory 104 of server 102 , memory 122 of client computing system 120 may include one or more memory devices of different types. Memory 104 may store machine readable instructions for execution on processor 128 . These instructions may be associated with various applications as well as the operating system. Memory 122 may also include a buffer 124 . Buffer 124 is a portion of memory used to temporarily store data received over network 114 before the data is processed for display via display system 132 to user 116 . The rate at which data is consumed may depend on the resolution and/or bitrate of the current stream. For example, if the resolution or bitrate of the stream is relatively high, more data is used to create the video. Accordingly, more data is consumed within a certain period of time to present the video item to user 116. Conversely, if the resolution or bitrate of the stream is relatively low, less data is used to create the video. Thus, less data is consumed within a certain period of time to present the video to user 116. Accordingly, the size of the buffer 124 may be set or adapted based on the bitrate of the selected video item stream.

In some examples, client computing system 120 may be responsible for identifying rebuffering points within video item 106 . In these cases, the client computing system 120 includes a rebuffering point identification module 130 . In some examples, the rebuffering point identification module 130 can be configured to analyze the content 106 as it is received to identify rebuffering points. In other words, the rebuffering points are identified in-situ on the client computing system 120 . This may eliminate the need for analysis and metadata generation associated with identifying rebuffering points by the server computing system 102 .

Like network interface 112 of server system 102, network interface 126 of client computing system 120 may be hardware, software, or both for encoding and decoding data for communications over network 114. includes a combination of This data is received from server system 102 over network 114 via network interface 126 and the data is placed into buffer 124 .

Display system 132 includes components used to present video items to user 116 . For example, display system 132 may include video and audio processing hardware and software. Display system 132 may also include a video output device such as a monitor or screen. Display system 132 may also include audio output devices, such as speakers or headphones, or connections configured to communicate with audio output devices.

2 is a diagram showing designated rebuffering points 202, 204, 206, 208; According to this example, a video item 200 has rebuffering points 202 corresponding to scene changes, rebuffering points 204 corresponding to shot changes, and rebuffering points corresponding to dialog pauses. 206, and rebuffering points 208 corresponding to images with specific visual characteristics, such as low brightness. Other criteria may be used to predetermine suitable locations for rebuffering events.

In some examples, rebuffering point identification module 110, 130 may analyze the data representing video item 200 and identify each scene change. A scene generally refers to a section of film that involves a specific action in a single location and covers a continuous period of time. The rebuffering point identification module 110, 130 may detect scene changes through a variety of different means. In one example, data representative of colors within an image may be analyzed to determine that the overall color appearance changes from one frame to the next. For example, an outdoor scene may have various colors in the image, while a scene change to an indoor scene may have different and varied colors. Additionally, data representative of the audio portion may be analyzed to determine that the audio ambiance of the video has changed, and to suggest a scene change accordingly. For example, a scene in which a character is driving a car may include sounds of cars and traffic. This scene can then be changed to a scene in a quiet room where the characters are talking to each other. This change in audio ambience can help identify a scene change. Since scene changes typically present a natural movement in the story, it may be less inconvenient for a rebuffering event. Accordingly, frames corresponding to scene changes may be identified as rebuffering points 202 .

In some examples, rebuffering point identification module 110, 130 may analyze data representative of video item 200 and identify each shot change, as described further herein. A shot can represent a portion of a video, typically comprising one continuous camera roll. A shot can also generally refer to a continuous picture between two edits or cuts. The rebuffering point identification module 110, 130 may detect shots in a variety of ways. In one example, features within each frame are analyzed. These characteristics may include the coloring of the frame or the brightness of the frame. Also, image processing algorithms can identify features such as shapes, edges and corners that can track these features across multiple frames. If a particular frame differs substantially from a previous frame, it may be determined that a cut has occurred. That is, the previous shot has ended and a new shot has started. Occasionally, shot changes occur in the middle of dialogue or another continuous sound source, such as a jet passing by. In some cases, shot changes do not occur in the middle of a conversation. Thus, a distinction can be made between shot changes that occur during dialogue or during continuous sound sources and shot changes that occur between dialogues or without continuous sound sources. A rebuffering event in conversations or shot changes that occur between successive sound sources may be perceived as less uncomfortable by user 116 . Accordingly, frames corresponding to these shot changes may be identified as rebuffering points 204 .

In some examples, rebuffering point identification module 110, 130 may analyze audio data within video item 200 and identify pauses in the conversation. Conversation may be detected by detection of audio transients within the appropriate frequency range or by a voice activity detector. Using various functions and audio feature extraction, you can analyze the data and determine when characters speak and when they don't. In some examples, if the length of a pause in the middle of a conversation is higher than a predefined threshold, some point within that pause may be identified as a rebuffering point 206 .

In some examples, rebuffering point identification module 110, 130 may analyze data representing video item 200 for points whose visual properties have specified characteristics, such as low brightness or relatively low brightness. For example, a visually darker portion of a video item is defined as any point within the video item 200 where the overall brightness of the image is below an absolute threshold or the brightness of the image is below a threshold based on the preceding image or frame. It can be. During these portions, there may be fewer distinctive features within the video that are of particular interest to the viewer. Thus, these visually darker portions may be less inconvenient for rebuffering events. Accordingly, these portions may be identified as rebuffering points 208 .

In some examples, instead of a single point in time, single timestamp, or single frame within video item 200 identified as a rebuffering point, a rebuffering time range 210 may be identified. For example, certain time spans 210 are found to correspond to auditory markers such as pauses in conversation, mood changes, silence, or any other sound that facilitates looping, time-stretching, or synthesis of similar sounds. can be determined In some examples, time range 210 may correspond to a visual marker, such as a visually darker portion of video item 200 . In such cases, any period within that time range 210 may be considered less inconvenient for the rebuffering event.

In some instances, certain types of rebuffering points may be preferred over others. For example, where possible, it may be more desirable for the rebuffering event to pause during scene changes rather than during shot changes. Similarly, it may be more desirable for a rebuffering event to pause during a scene change rather than during a pause in dialogue. Accordingly, rebuffering points may be assigned a ranking value. The logic that determines when a rebuffering point should be used for a rebuffering event may take into account the rank value of the immediately adjacent rebuffering point. For example, if the need for a rebuffering event is more immediate, lower order rebuffering points, such as pauses in dialogue or those corresponding to visually darker parts, may be used for rebuffering events. there is. However, if the need for the rebuffering event is less immediate, only higher order rebuffering points such as scene changes may be used for the rebuffering event. In some embodiments, the rebuffering event may occur by default at certain rebuffering points, such as at a dark scene change, ie where the scene goes black or fades to black with silence in the audio data. Rebuffering can be performed as long as the amount of data in the buffer does not exceed a certain threshold, such as 10 seconds, 30 seconds, or 1 minute. In these cases, rebuffering may be performed for a short amount of time. For example, rebuffering can be performed for less than 1 second or less than 1/2 second. Including these pauses when user 116 is unlikely to notice a pause can allow the buffer to expand in anticipation of any drops in network connectivity, such as when switching between cell towers. .

The use of rebuffering points may depend on various conditions associated with the client computing system 120 receiving the video stream. For example, various network conditions may affect the use of rebuffering points. If the network connection between the streaming service provider and the client computing system 120 is slower than the ideal rate for streaming at a particular resolution, the client computing system 120 sets certain rebuffering points to keep the buffer utilization above a desired threshold. It can be configured to pause at. For example, the client computing system 120 may pause for a few seconds or less than a second at each scene change to ensure that the buffer utilization does not drop below a desired threshold. In some examples, if the rate at which content can be delivered over the network to client computing system 120 is lower, then the client computing system also pauses at lower order rebuffering points, such as pauses in a conversation. can do.

In some examples, the length of each pause can depend on the current rate at which data for video item 200 is being communicated to client computing system 120 . In general, lower rates can be handled by pausing for longer periods at rebuffering points. Conversely, higher rates can be handled by pausing for shorter periods at rebuffering points. The length of these pauses can be dynamically adjusted as network conditions change during the streaming process. For example, as network conditions improve, the length of these pauses may be shortened. In some examples, it may be determined that no pauses are needed to keep the buffer utilization above a desired threshold when network conditions substantially improve. Conversely, when network conditions deteriorate, longer pauses at rebuffering points may be applied. Thus, the current rate of rebuffering can be used to determine the types of rebuffering points used for rebuffering as well as the lengths of time used for rebuffering.

The length of pauses for rebuffering events may also depend on the rank or type of particular rebuffering point. For example, higher order rebuffering points, such as those corresponding to scene changes, may have longer pauses because these points are less intrusive to the viewing experience. Conversely, lower order rebuffering points, such as gaps in dialogue, may have shorter pauses, since these points may be slightly more jarring to the viewing experience. For example, given certain conditions, a pause of one second may be used during a scene change, while a pause of less than 500 milliseconds may be used during a gap in dialogue.

3A-3D are diagrams illustrating the use of designated rebuffering points 306 for rebuffering events, in accordance with some embodiments of the present disclosure. In some examples, pausing for rebuffering events may be triggered when buffer utilization 310 falls below a predefined threshold 308 . Buffer utilization 310 can be defined in a variety of ways. In one example, buffer utilization 310 may be defined by the amount of data in the buffer that is not currently consumed. In some examples, buffer utilization 310 may be defined by an amount of play time for a video item in the buffer. For example, this may be the case when the buffer has enough data for a playback time of 1 minute 32 seconds.

3A-3D show a timeline for video item 200 . Line 302 represents a viewpoint within video item 200 that is currently being displayed for viewing. Line 304 represents a point in video item 200 at which the buffer is consumed, assuming no new data is received. In other words, if no more data is passed to the buffer, the video item may continue playing up to the point represented by line 304 . Thus, the time span between lines 302 and 304 represents buffer utilization 310 .

3A shows that the current buffer utilization 310 is above a predefined threshold. Thus, the presentation of the video item will operate as normal. In FIG. 3B , as the video item is played, the buffer utilization 310 drops below a predefined threshold 308 . This may occur because the rate at which data is transferred to the buffer is less than the rate at which data is processed for presentation at display system 132 . Accordingly, it can be determined that the buffer utilization 310 will drop to a point where a rebuffering event is required. Conventional streaming systems can simply pause at any point where it is determined that a rebuffering event is needed or at which point all data in the buffer has been processed. However, according to the principles disclosed herein, a pre-determined or pre-identified rebuffering point may be used as the point at which a rebuffering event occurs. Because the rebuffering points correspond to points in video item 200 that are less intrusive, user 116 may have a better experience.

3C shows that line 302 has reached rebuffering point 306 . At this rebuffering point, the video item 200 may be paused to allow rebuffering. Presentation of video item 200 on display system 132 may be temporarily suspended. 3D illustrates the point in time after the video item 200 is paused and before it is resumed. As illustrated, the additional time provided by the pause allows more data to be added to the buffer, thus extending the buffer utilization 310 to a point well above the predefined threshold 308 . In some examples, the pause for rebuffering at the rebuffering point 306 may be for a period of time necessary to allow the buffer utilization to reach a predetermined size before resuming. In some embodiments, the duration of the pause for rebuffering may be determined by subsequent rebuffering points. For example, the video item 200 may be paused for rebuffering until another rebuffering point or another rebuffering point of a particular type exists in the buffered data. Alternatively, a certain amount of time or data may be added after subsequent rebuffering as a rebuffering pad. For example, a pause for rebuffering may occur until data associated with the second illustrated rebuffering point 306 is added to the buffer or after a threshold amount of data associated with the second illustrated rebuffering point 306. It can persist until additional data is added.

In some examples, the client system displaying the video can evaluate the current utilization of the buffer at each designated rebuffering point. At a specific designated rebuffering point, if the buffer utilization is below a specific threshold, the video item may be paused at that specific designated rebuffering point. Also, the current utilization rate of the buffer can help determine how long the video item should be paused at that particular rebuffering point. For example, if the current buffer utilization is close to a threshold, the pause may be relatively short. However, if the current buffer utilization is well below the threshold, the pause may be relatively long. If the buffer utilization rate is higher than the threshold, it may not be necessary to pause the video item at that rebuffering point.

4A and 4B are tables showing metadata that can be assigned to shots and frames within a piece of content to help identify rebuffering points. In some examples, the rebuffering point identification module 110, 130 includes hardware, software, or a combination of both to analyze the data of the video item 200 and identify different shots and frames within the video item 200. can do. For example, the rebuffering point identification module 110, 130 may, depending on the implementation, determine the start and stop points for each different shot within the video item 200 or each different shot within the current buffered portion of the video item 200. points can be displayed. The rebuffering point identification module 110 , 130 may also identify a selection of frames within the video item 200 . The rebuffering point identification module 110, 130 may also analyze the images associated with each of the frames to detect various features within the frames. These features may include, for example, the faces of the characters appearing in the shot or frame. Based on these characteristics, the rebuffering point identification module 110, 130 may assign metadata to the various shots and images detected within the video item 200. This metadata can then be used to assign rebuffering points within video item 200 as well as ranking values for these rebuffering points. For example, a pause for rebuffering may be desirable during the time the first actor is displayed but not the second actor.

A frame corresponds to a still image that, when sequentially combined with other frames, creates a moving image. The data forming the video item 200 may describe each frame within the video item 200 . The rebuffering point identification module 110, 130 may analyze multiple frames within the video item 200 and assign metadata to each of the frames based on features detected within the frames. A series of frames may form a shot. A shot can represent a portion of a video, typically comprising one continuous camera roll. A shot may also refer to a continuous stream of footage, usually between two edits or cuts. The rebuffering point identification module 110, 130 may detect shots in a variety of ways. In one example, features within each frame are analyzed using image processing algorithms. These features may include the coloring of the frame, the brightness of the frame, or machine identifiable features such as shapes and edges. If a particular frame differs substantially from a previous frame, it may be determined that a cut has occurred. In other words, the previous shot has ended and a new shot has started. After the various shots have been identified, the rebuffering point identification module 110, 130 can identify differences between features of different frames within the shots. These differences can be used to categorize the shot and assign various metadata to the shot.

The rebuffering point identification module 110, 130 also analyzes each shot to assign to that shot metadata including a shot category. This can be done by analyzing machine readable data representing frames within a shot. In one example, the rebuffering point identification module 110, 130 selects at least two frames within a particular shot. The rebuffering point identification module 110, 130 analyzes features found within these frames and identifies differences between features of these frames. For example, a shot may be determined to be a zoomed-out shot if spatial relationships between various features in one frame are greater than spatial relationships between various features in another frame. For example, when it is determined that features of one frame are features of a character's face and features of another frame are also determined to be features of a character's face, the shot is a close-up shot if the face occupies more than a threshold amount of frames. can be determined. For example, if it is determined that features in one frame are moved relative to features in another frame in a specific way, it can be determined that the shot is a setup shot. Other types of shots are also contemplated. In some examples, the type of shot for which a rebuffering point exists may affect the rank value of that rebuffering point. For example, it may be less inconvenient to pause during an establishing shot than a close-up shot.

The rebuffering point identification module 110 , 130 may also analyze a selected number of frames within the video item 200 . Analyzing the frames within the video item 200 may involve examining machine-readable data representing the video item 200 . In some examples, every single frame of a video item may be analyzed. However, in some examples, every X frames may be analyzed. In some examples, X can be in the range of about 5 to 60. Other values for X are also contemplated. The rebuffering point identification module 110, 130 may also assign metadata to each analyzed frame.

4A shows an example table 400 that includes metadata for a particular frame, also referred to as frame metadata. The metadata may indicate the frame's visual properties (401), the frame's structural properties (403), and the frame's temporal properties (405). Visual attributes 401 include brightness 402 , sharpness 404 , and contrast 406 . Other visual attributes may include the color composition or color temperature of the frame. Structural properties 403 include frame faces 408 and frame extrusions 410 . Temporal attributes 405 include frame motion 412 and frame direction 414 .

The rebuffering point identification module 110, 130 may assign a brightness value 402 to the frame based on the average brightness value associated with each pixel in the frame. Specifically, the rebuffering point identification module 110 or 130 may inspect data representing a frame. That data can define color values for each pixel in a frame. For example, if the data for a pixel is represented in the YUV color space, also known as YCbCr, the Y value represents the pixel luminance. For example, if the data for a pixel is expressed in RGB color space, the brightness for a particular pixel can be defined as the average color value for that pixel (e.g., Br = (R+G+B)/3 where Br is the brightness, R is the red color value, G is the green color value, and B is the blue color value). Other ways to determine the brightness value are contemplated.

The rebuffering point identification module 110, 130 may assign a sharpness value 404 and a contrast value 406 to the frame based on an analysis of the data defining the frame. For example, the rebuffering point identification modules 110 and 130 may apply a function for determining a sharpness value and a contrast value of a frame. Sharpness, sometimes referred to as sharpness, is a measure of how strongly the contrast in an image is perceived. Contrast refers to color differences within an image. Various methods may be used to determine the sharpness value and the contrast value based on the data representing the frame.

The rebuffering point identification module 110, 130 may also identify faces 408 appearing within the shot. For example, it may be determined that a frame includes one or more faces based on an analysis of features within the frame. Various face recognition functions can be applied to identify the presence of faces and then to identify actual faces represented in the data. As an example, the Viola-Jones algorithm is one popular choice for detecting faces. The faces of various characters within a frame may also be assigned a popularity value. This popularity value can be derived in various ways. In one example, the popularity value is based on the percentage of time the character appears within the video item. In some examples, external sources may be used to determine a character's popularity. For example, the popularity value may be predefined by human users. In some examples, analysis of publicly available information such as web pages and social media may be applied to assign a popularity value to a particular character. The popularity of characters within a particular shot or scene may also affect the ranking value of rebuffering points within that shot or scene. For example, it may be desirable to pause for rebuffering while showing a more popular character or actor.

The rebuffering point identification module 110, 130 may allocate saliency data to a frame. The saliency data may include, for example, a saliency map. The saliency map identifies the uniqueness of parts of the image. For example, color uniqueness of a portion of an image (eg, a pixel or set of adjacent pixels) may be identified by a value. The saliency map may also include saliency values assigned to people or objects in focus within the image. For example, a saliency value can identify how much the person or object stands out against the background on which the object or person is placed.

The rebuffering point identification module 110, 130 may also determine temporal characteristics of the frame. For example, by analyzing data representative of a frame and adjacent frames, it may be determined that a particular object or person in focus is moving at a particular speed and direction relative to other objects or the background of the image. This information can be determined and assigned frame motion values 412 and frame direction values 414 .

The frame motion value 412 and the frame direction value 414 may be used to determine a designated rebuffering point. For example, if there is a lot of movement within a video item in a particular frame, this may indicate that more action is taking place during that scene. Therefore, it can be more inconvenient to trigger a rebuffering event in that frame. However, if there is less movement, this may be a less inconvenient frame for a rebuffering event. Accordingly, a frame having a frame motion value 412 less than a predetermined motion threshold may be identified as a designated rebuffering point. In some examples, a frame with a low motion value may have a lower rank value than a frame corresponding to a shot change or scene change.

The rebuffering point identification module 110, 130 may also determine a rebuffering point rank value 416 for the frame. The rebuffering point rank value may be based on various factors associated with the frame or the location of the frame within the shot or scene. For example, the ranking value can range from 0-100, where 0 is a bad choice for rebuffering points and 100 is an ideal choice for rebuffering points. Thus, a frame located within dialogue and representing important character speech may have a rebuffering point rank value of zero, and a frame corresponding to a fade out or fade in at the end or beginning of a scene may have a rebuffering point rank closer to 100. can have a value. Also, a frame positioned at the end or beginning of a shot may have a rebuffering point ranking value close to 75, for example. The rebuffering point rank value 416 may also be influenced by other attributes 401, 403, 405 of the frame and other considerations described herein. In some embodiments in which the rebuffering point identification module 110 running on the server computing system 102 determines attributes 401, 403 and 405, the client computing system 120 uses this metadata to: Generates additional metadata including the rebuffering point rank value 416.

4B shows an example table 420 containing metadata for a particular shot, also referred to as shot metadata. According to this example, metadata includes shot category 422 , visual properties 424 , structural properties 426 , and temporal properties 428 .

Shot category 422 identifies the type of shot. For example, shot categories may include, but are not limited to, close-up shots, setup shots, zoom-out shots, or other categories of shots used in the television and film industries. Other types of shot categories may also be defined. As mentioned above, a shot category may be defined based on feature differences between at least two different frames within a shot.

The visual attribute data 424 may include information such as brightness, sharpness, and contrast. These can be expressed as average values over the shot. For example, a sample of frames from a shot can be analyzed and averaged to determine various visual attribute values. Structural attribute data 426 may include structural characteristics of the shot, such as characters appearing in the shot. Temporal attribute data 428, if any, can indicate the direction the shot is moving, or the direction the identified object within the shot is moving. Temporal attribute data 428 may also indicate the direction in which any objects in focus are moving relative to the background. Information of other fragments that may help identify rebuffering points may be included in the shot metadata.

5 is a diagram illustrating detection of features representative of a close-up shot. 5 shows images of two different frames 502 and 504 . In this example, the first frame 502 corresponds to an earlier frame within the shot, and the second frame 504 corresponds to a later frame within the shot. Frames 502 and 504 are analyzed to identify specific features within each frame 502 and 504 . In some examples, these features may be identified as primary features and secondary features. In this example, frames 502 and 504 have a main feature 506a and 506b that is the face of a single character appearing within the shot. Frames 502 and 504 also include auxiliary features 508a and 508b, such as a portion of the character's clothing (in this example, the character's tie).

In some examples, various functions may be applied to identify primary and secondary characteristics. Generally, the characters' faces will be designated as key features. Other objects that stand out against the rest of the background may be designated as auxiliary features. If there are no faces in the shot, other mechanisms may be used to identify key features. For example, the most salient object may be designated as a key feature. Alternatively, no primary feature may be identified and only auxiliary features may be identified. In some instances, no distinction may be made between primary and secondary features.

A shot category can be assigned by comparing features between two frames. In FIG. 5 , a comparison of the main feature 506a from the first frame 502 and the corresponding main feature 506b from the second frame 504 shows that there is little difference in the size or position of the main feature 506. show what Trace lines between features are substantially parallel and horizontal. This indicates that there is little movement between the first frame 502 and the second frame 504 . Further, a comparison between auxiliary feature 508a from the first frame 502 and auxiliary feature 508b from the second frame 504 shows little difference in the location of the auxiliary feature 508 . Also, key features 506 occupy a certain amount of space within the frame. For example, key feature 506 may have overall dimensions including at least one dimension greater than one third of the corresponding dimension of the overall frame. For example, the face identified as key features 506a and 506b has a height greater than one third of the overall height of the frame. The threshold of 1/3 is provided as an example, other values or percentages may be used in other embodiments. Based on this information, it can be determined that the shot is a close-up shot. Accordingly, shots may be categorized accordingly.

6 is a diagram illustrating detection of features indicative of an established shot. 6 shows images of two different frames 602 and 604 . Primary features detected within frames 606a and 606b are people, and secondary features detected within frames 602 and 604 include scenery 608a and 608b. A shot category can be assigned by comparing features between the two frames 602 and 604 . Comparing the relationship between the primary feature 606a and secondary feature 608a in the first frame with the relationship between primary feature 606b and secondary feature 608b in the second frame 604, the distance between the two changes and shows that the relative dimension of one of the identified features (eg, height) changes, while the other dimension (eg, width) does not change or does not change much. This may indicate that the shot includes vertical movement of the camera relative to the identified features. That is, the trace lines between corresponding points within the frames are not completely horizontal, but instead slightly diagonal. The relatively shallow slope of the lines indicates that there is some motion between the two frames 502 and 504, but not sudden or rapid motion. Also, the main features 606a (ie, people) take up a relatively small amount of space compared to the image. Based on this information, it can be determined that the shot is a setup shot. Accordingly, shots may be categorized accordingly.

7 is a diagram illustrating detection of features indicative of a zoomed-out shot. 7 shows images of two different frames 702 and 704 . The features 706a and 706b detected within the frames include the object of focus the character is looking at. A shot category can be assigned by comparing features between the two frames 702 and 704 . Comparison of the relative sizes of features 706a in the first frame and features 706b in the second frame 704 shows that the relative sizes change. Specifically, the features 706b in the second frame 704 are smaller than the corresponding features 706a in the first frame 702 . The convergence characteristics of the trace lines between the corresponding points suggest that the corresponding features are smaller in the second frame 704 than in the first frame 702 . Based on this information, it can be determined that the shot is a zoomed-out shot. Accordingly, shots may be categorized accordingly. If features 706b in the second frame 704 are determined to be larger than corresponding features 706a in the first frame 702, the shot may have been determined to be a zoomed-in shot.

5-7 show several examples of detecting features within shots to assign a shot category to these shots. Other types of shots may also be detected. Additionally, other types of functions for identifying different types of shots may be used in some embodiments of the rebuffering point identification modules described herein.

8 is a flow diagram illustrating an exemplary method for pausing a video item at designated rebuffering points. The method 800 includes several enumerated steps or actions. Embodiments of this method 800 may include additional steps before, after, between, or as part of the listed steps. Some embodiments may omit one or more of the listed actions. Additionally, some embodiments of the method 800 include a non-transitory machine-readable medium storing instructions that cause a processor to perform all or some of the described operations. According to this example, the method 800 includes an operation 802 for receiving a streaming video item in a buffer or extracting or reading data including the streaming video item. A video item may be, for example, an episode of a full-length movie or series. Other examples of video items are also contemplated. A video item can be delivered as a stream. In other words, the video item is displayed by the client device while data is still being transferred from the server device to the client device. Thus, a video item may start presenting before it is completely downloaded to the client device.

The method 800 further includes an act 804 for outputting the video item from the buffer to a display system. A display system may include, for example, a visual output device such as a screen and an audio output device such as a set of speakers or connections thereto. Outputting a video item from the buffer to the display system consumes data in the buffer. When viewing streaming video items in this way, data representing the video item is received from the remote server into a local buffer. Typically, when such data in the buffer is consumed for presentation before additional data can be delivered, the video item must be paused to allow rebuffering. These rebuffering events do not take into account aspects of the content being rendered and may lower the quality of experience by the viewer.

The method 800 further includes an act of determining 806 that a designated rebuffering point exists within a predetermined time frame. A designated rebuffering point may correspond to another point within a video item that is less inconvenient for a scene change, shot change, or pause. Rebuffering points may be identified before the video item is streamed or may be identified by the client computing system as data is put into a buffer. A predetermined time frame may correspond to data currently in the buffer. Specifically, the predetermined time frame may extend between the point at which the video is currently present and the point at which the buffer is completely consumed. In some examples, there may not be a rebuffering point specified in the unconsumed data in the buffer. However, there may be cases where such a rebuffering point appears before the buffer is completely consumed or the rebuffering event is otherwise triggered. Embodiments of this method 800 may include selecting a particular rebuffering point from among a plurality of potential rebuffering points in response to determining that the buffer utilization rate falls below a predetermined threshold. A buffer utilization rate is below a predetermined threshold when the amount of data in the buffer falls below a predetermined data value or when the amount of data in the buffer is decreasing at a rate predicted to fall below the predetermined threshold within a predetermined amount of time. can be determined by falling to Accordingly, this decision may be an actual decision or a predicted decision. Additionally, some embodiments of the method 800 may use a buffer to determine what percentage of the memory allocated to the buffer is actually being occupied, such as to determine if a pause for rebuffering is needed or a lever of buffer fullness. It may include an operation of evaluating . In some embodiments, the evaluation of buffer fullness can be used to determine how long the pause for rebuffering should last or when the pause should be stopped.

The method 800 includes pausing a video item at a specified rebuffering point in response to determining that a specified rebuffering point exists within a predetermined time frame and that rebuffering should be performed to avoid completely emptying the buffer. Operation 808 is further included. By pausing a video item at a designated rebuffering point instead of a random point determined only by the amount of data in the buffer, the inconvenience of rebuffering events is reduced. Thus, viewers are provided with a better experience while watching streaming content. In some cases, rebuffering may be performed without the viewer being aware of the pause for rebuffering. Thus, while conventional implementations of rebuffering events typically involve some change in the scene presented to the user to convey that rebuffering is being performed (eg, a spinning wheel appears over a frame), some implementations of the present disclosure Examples may include no indication to the user that a rebuffering event is occurring. Because rebuffering is performed at identified appropriate times, rebuffering is less noticeable to users. For example, adding one second representing a black frame in an identified fade-out may not be perceived by the user as rebuffering, but simply as a longer transition in the video item.

Additionally, some embodiments of the client computing system 120 may “pause” playback of a video item by stretching the video and/or audio content present in the buffer. Client computing system 120 may interpolate the video and audio information such that a given portion of the video is stretched to increase in time. For example, buffer 124 may contain data that can be processed into 5 seconds of video. Some or all of this data may be “stretched” by adding additional frames or displaying the frames for a longer period of time so that the data is presented over 6 seconds rather than 5 seconds. The audio can be stretched in a way that does not change the pitch correspondingly. In some embodiments, the Fourier spectrum of the relevant portion of the buffered audio may be used to create a spectral profile to synthesize similar audio content. For example, an establishing shot identified in a media item may be stretched from 10 seconds to 12 seconds to allow 2 seconds of rebuffering to occur while the establishing shot is being viewed. The client computing system 120 may identify a setup shot by metadata including a tag indicating a rebuffering point associated with the shot. Accordingly, the client computing system 120 may provide a rebuffering response based on the type of rebuffering point used for rebuffering.

In some embodiments, after the rebuffering point is selected, client computing system 120 may apply a fade out to the data to be displayed by display system 132 . A fade out may be provided for video and/or audio included in the buffered data. In some embodiments, the predetermined audio content may be presented when rebuffering is likely to take a significant amount of time (eg, more than 3, 5 or 10 seconds). Similarly, predetermined video content may be presented, such as a frame showing a title of a video item or a frame associated with a chapter or segment of a video item. When a predetermined audio content is presented, the content can be designed for looping and can be obtained from the video item itself, such as the soundtrack of the video item. Such approaches may provide a better user experience while extending rebuffering periods, and may be included in embodiments of the method 800 .

In some examples, if the rebuffering pause exceeds a certain threshold, the viewer may receive an interim presentation until the buffer receives enough data to proceed with presentation of the original piece of content. An intermediate presentation may be, for example, the last 10-15 seconds of a piece of content before a pause occurs. In some examples, intermediate content may be played after the buffer has received enough data to advance so that the viewer sees 10-15 seconds immediately before video playback resumes. In some examples, the viewer may be notified that such a summary is being presented.

In some examples, ranking values may be determined in part using analysis of data collected using various experiments. For example, subjective visual experiments can be conducted in which subjects are shown simulated interrupted video streams and asked to rank the goodness/quality of each. In this case, subjective scores may be used in conjunction with the principles described above. In some examples, machine learning models such as support vector machines, neural networks, etc. may be used. These models can be used to train systems on how to assign rank values in an optimal way. Then, as additional sets of data are received, the rebuffering points and associated rank values may be recalculated.

Some examples of processing systems described herein may be non-transitory tangible machine readable containing executable code that, when executed by one or more processors, may cause the one or more processors to perform the operations of method 800 described above. Any possible medium may be included. Some common types of machine-readable media that may include the operations of method 800 include, for example, a floppy disk, a flexible disk, a hard disk, magnetic tape, any other magnetic medium, a CD-ROM, any other optical media, punch cards, paper tape, any other physical media with patterns of holes, RAM, PROM, EPROM, FLASH-EPROM, any other memory chip or cartridge, and/or adapted to be read by a processor or computer. any other medium.

Embodiments of the present disclosure may improve server and client systems used in a streaming media environment by optimizing the placement of rebuffering events. An optimized placement can be determined based on the data being streamed, the content of the data being streamed, and the network conditions experienced by the client device. An optimized layout can provide an improved user experience by improving these computing systems.

While exemplary embodiments have been shown and described, a wide range of modifications, changes, and permutations are contemplated in the foregoing disclosure, and in some cases, some features of the embodiments may be utilized without corresponding use of other features. One skilled in the relevant art will recognize many variations, alternatives and modifications. The claims are to be interpreted broadly in a manner consistent with the scope of the embodiments disclosed herein. Accordingly, certain aspects of the present disclosure are presented in the numbered claims below:

Section 1. A method comprising: receiving, by a computing system, data representing a video item in a buffer; outputting, by the computing system, the video item from the buffer to a display system; determining, by the computing system, that the utilization of the buffer falls below a predetermined threshold; in response to determining that the utilization rate of the buffer falls below the predetermined threshold, determining that a designated rebuffering point exists within a predetermined time frame; and pausing, by the computing system, the video item at the designated rebuffering point in response to determining that the designated rebuffering point exists within the predetermined time frame.

Section 2. 2. The method of claim 1, further comprising resuming the video item after a predetermined amount of time, wherein the predetermined amount of time is based on a rate at which the video item is received by the buffer.

Section 3. 3. The method of claim 1 or 2, wherein the designated rebuffering point corresponds to one of a shot change and a scene change.

Section 4. 4. A method according to any one of claims 1 to 3, wherein the designated rebuffering point corresponds to a frame motion value being below a predetermined motion threshold.

Section 5. 5. A method according to any one of claims 1 to 4, wherein the designated rebuffering point corresponds to a point whose values representing visual attributes are below a predefined visual attribute threshold.

Section 6. 6. A method according to any one of claims 1 to 5, wherein the designated rebuffering point is a pause in the conversation of the video item.

Section 7. 7. The method of any one of claims 1 to 6, wherein the designated rebuffering point is selected from among a plurality of potential rebuffering points in the video item, the plurality of potential rebuffering points have different types, and the different types are different How to get assigned rank values.

Clause 8. 8. The method of claim 7, wherein the step of pausing the video item occurs only if the designated rebuffering point is above a certain rank.

Clause 9. 9. A method according to any preceding claim, wherein the designated rebuffering point is identified within the video item before the video item is streamed to the computing system.

Section 10. 10. A method according to any preceding claim, wherein the designated rebuffering point is identified by the computing system while the portion of the video item on which the designated rebuffering point is located is within the buffer.

Section 11. A method comprising: receiving, by a computing system, first data representing a video item in a buffer and receiving second data indicating a set of designated rebuffering points for the video item; outputting, by the computing system, the first data representative of the video item from the buffer to a display system; determining that the rate at which the first data is received is smaller than the rate at which the first data is output from the buffer to the display system by a threshold amount; and in response to determining that the rate at which the first data is received is less than the rate at which the first data is output to the display system by the threshold amount, at one designated rebuffering point of the set of designated rebuffering points. A method comprising pausing a media stream.

Section 12. 12. The method of claim 11, wherein the set of designated rebuffering points includes shot changes, scene changes, scenes with predefined visual characteristics, and pauses in dialogue.

Article 13. 13. The method of claim 12, wherein different types of rebuffering points are assigned different ranks.

Section 14. 14. The method of claim 13, wherein one designated rebuffering point of the set of designated rebuffering points is based on a difference between a rate at which the first data is received and a rate at which the first data is output from the buffer to the display system. How to be chosen.

Article 15. 15. The method of claim 14, wherein the subset of acceptable rebuffering points from which the selected rebuffering point is selected includes additional types of rebuffering points as the difference increases.

Article 16. 16. The method of any one of claims 13 to 15, further comprising pausing the media stream at a plurality of rebuffering points of the set of rebuffering points, wherein pausing at each of the plurality of selected rebuffering points A method for different time periods based on the rank value assigned to that rebuffering point.

Article 17. 17. A method according to any one of claims 11 to 16, wherein the duration for which the video item is paused is based on a prediction when the buffer is below a predetermined value.

Article 18. CLAIMS 1. A method of optimizing placement of rebuffering events in streaming media playback, comprising: receiving, by a server computing system, a streaming video item; performing image processing to identify a plurality of designated rebuffering points in the streaming video item; storing information characterizing the identified plurality of designated rebuffering points in a memory; and transmitting the information characterizing the identified plurality of designated rebuffering points to a client computing system over a network.

Article 19. 19. The method of claim 18, wherein the information characterizing the identified plurality of designated rebuffering points indicates timestamps and frame numbers associated with the designated rebuffering point, and a rebuffering point type associated with each designated rebuffering point. A method for optimizing placement of rebuffering events in streaming media playback comprising at least one of the following tags:

Article 20. 19. The method of claim 18, further comprising transmitting the streaming video item to the client computing system, wherein transmitting the streaming video item comprises transmitting the information characterizing the identified plurality of designated rebuffering points. A method for optimizing placement of rebuffering events in streaming media playback performed in terms of steps.

Claims (20)

As a method,
receiving, by a computing system, data representing a video item in a buffer;
outputting, by the computing system, the video item from the buffer to a display system;
determining, by the computing system, that the utilization of the buffer falls below a predetermined threshold;
determining, by the computing system, that a designated rebuffering point exists within a predetermined time frame in response to determining that the utilization rate of the buffer falls below the predetermined threshold, wherein the designated rebuffering point corresponds to the video is selected from among a plurality of potential rebuffering points in an item, the plurality of potential rebuffering points include rebuffering points of different types assigned different ranking values, the highest ranking rebuffering point being the designated rebuffering point. selected -; and
pausing, by the computing system, the video item at the designated rebuffering point in response to determining that the designated rebuffering point exists within the predetermined time frame;
How to include. According to claim 1,
resuming the video item after a predetermined amount of time, wherein the predetermined amount of time is based on a rate at which the video item is received by the buffer. According to claim 1,
The designated rebuffering point corresponds to one of a shot change and a scene change. According to claim 1,
wherein the designated rebuffering point corresponds to a point within the video item at which a frame motion value is below a predetermined motion threshold. According to claim 1,
wherein the designated rebuffering point corresponds to a point whose values representing visual attributes are below a predefined visual attribute threshold. According to claim 1,
wherein the designated rebuffering point is an auditory marker within the video item. delete According to claim 1,
wherein the step of pausing the video item occurs only if the designated rebuffering point is above a minimum specific rank. According to claim 1,
The designated rebuffering point is identified within the video item before the video item is streamed to the computing system. According to claim 1,
The designated rebuffering point is identified by the computing system while the portion of the video item at which the designated rebuffering point is located is within the buffer. As a method,
receiving, by a computing system, first data representing a video item in a buffer and receiving second data indicating a set of designated rebuffering points for the video item;
outputting, by the computing system, the first data representative of the video item from the buffer to a display system;
determining that the rate at which the first data is received is smaller than the rate at which the first data is output from the buffer to the display system by a threshold amount; and
in response to determining that the rate at which the first data is received is less than the rate at which the first data is output to the display system by the threshold amount, a media stream at a rebuffering point selected from the set of specified rebuffering points. Pausing, wherein the designated set of rebuffering points includes rebuffering points of different types assigned different ranking values, the highest ranking rebuffering point being used as the selected rebuffering point;
How to include. According to claim 11,
wherein the set of designated rebuffering points includes shot changes, scene changes, scenes with predefined visual characteristics, and pauses in dialogue. According to claim 12,
Rebuffering points of different types are assigned different ranks. According to claim 13,
wherein one designated rebuffering point of the set of designated rebuffering points is selected based on a difference between a rate at which the first data is received and a rate at which the first data is output from the buffer to the display system. According to claim 14,
The subset of acceptable rebuffering points from which the selected rebuffering point is selected includes additional types of rebuffering points as the difference increases. According to claim 13,
further comprising pausing the media stream at a plurality of rebuffering points of the set of rebuffering points, wherein pausing at each of a plurality of selected rebuffering points is based on a rank value assigned to that rebuffering point; Methods made during different time periods. According to claim 11,
wherein the duration for which the video item is paused is based on a prediction of when the buffer is below a predetermined value. A method for optimizing the placement of rebuffering events in streaming media playback, comprising:
receiving, by a server computing system, a streaming video item;
performing image processing to identify a plurality of designated rebuffering points in the streaming video item, the plurality of designated rebuffering points comprising different types of rebuffering points assigned different rank values;
storing information characterizing the identified plurality of designated rebuffering points in a memory; and
Transmitting the information characterizing the identified plurality of designated rebuffering points to a client computing system via a network, wherein the streaming video item is rebuffered at the selected highest ranked rebuffering point.
A method for optimizing placement of rebuffering events in streaming media playback, comprising: According to claim 18,
The information characterizing the identified plurality of designated rebuffering points is at least one of timestamps and frame numbers associated with the designated rebuffering point, and a tag indicating a rebuffering point type associated with each designated rebuffering point. A method for optimizing placement of rebuffering events in streaming media playback, comprising: According to claim 18,
further comprising transmitting the streaming video item to the client computing system, wherein transmitting the streaming video item is performed in conjunction with transmitting the information characterizing the identified plurality of designated rebuffering points. How to optimize the placement of rebuffering events in streaming media playback.

Images