TW201517607A - Method and player for rendering condensed streaming content - Google Patents

Abstract

A method for rendering a condensed stream is provided. The method is used in a client player, wherein the client player is coupled to a stream condense unit and a streaming server, comprising: receiving, by a streaming data receive unit, a condensed stream sent by the stream condense unit; extracting, by a condensed stream extract unit, a plurality of source streaming contents from the condensed stream; decoding, by at least one decoder unit instance, the plurality of source streaming contents and generating a plurality of videos; and rendering, by a video rendering unit, the plurality of videos.

Description

Method for playing composite concentrated stream and player

The disclosure relates to a method and a player for a composite condensed stream in a playback network.

Due to the mature development of broadband networks, video streaming has become one of the most popular applications, such as entertainment-based audio-visual sharing, instant Internet TV and other services, users can browse the program table to get information about the channel. And decided to watch the content of streaming programs; on the other hand, in recent years, the monitoring industry has also moved from the closed circuit television (CCTV) system to the IP surveillance system (IP Surveillance system). The video source is like a network camera (IPCAM), a video server (Video Server), a digital video recorder (DVR) and the like. Through the video management system (VMS), the user can monitor the multi-channel surveillance cameras built by the IP network at the same time to achieve comprehensive control. In addition to the end-to-end direct connection, Streaming Server is also often used to increase the service scale of the system.

Figure 1 shows a schematic diagram of a streaming system architecture. Users can access various types of Network-connected Devices 14, such as Desktop PCs, Notebooks, and Tablet PCs. (Tablet PC), Mobile Phone, as a carrier for streaming viewing. When the user views the individual streaming content, the serial communication server 12 first establishes a string with the streaming server 12 via a Connection Control Protocol, such as Real-time Streaming Protocol (RTSP). Streaming, after communication is completed, through the Media Transport Protocol, such as the Real-time Transport Protocol (RTP), receiving the instant streaming content data 16 or the stored streaming content data 18, and In order to avoid the impact of the quality of the network bandwidth changes, there is usually a short time stream buffer on the player side as a precaution. Depending on the degree of bandwidth variation and the size of the buffer space, the entire stream connection setup process can take up to several seconds. When the user wants to watch multiple channels at the same time, multiple sets of serial connections can be established through the player to play independently between channels.

With the advancement of mobile communications, the demand for streaming systems through mobile devices and networks is increasing. However, the mobile network is still limited by the bandwidth and the quality of the reception, and is subject to environmental changes at any time. On the one hand, in the case of unstable or insufficient bandwidth, it is easy to cause a bad visual perception such as a picture failure or frustration. When watching multiple channels (for example, network monitoring applications), each serial connection will compete with each other. To make the situation worse, it is still very difficult for users to watch multiple channels at the same time. On the other hand, a global channel preview is provided, which allows the user to quickly view the pictures currently being broadcast by all the streaming channels and decide what to watch, and assist the user to dynamically and automatically grasp the changes of the channel information. For example, when a channel is added or left or a user is interested in a picture (for example, information such as a license plate, a face or a movement amount is detected), it is an urgent need.

In view of this, a method and system for providing and transmitting multi-source concentrated streaming content is to be developed.

The present disclosure provides a method of playing a composite concentrated stream and a player.

The present disclosure provides a user-side player coupled to a streaming server and a condensing stream unit, including: a stream data receiving unit, configured to receive a composite condensed unit transmitted by the concentrating stream unit a composite condensed stream analyzing unit coupled to the stream receiver for receiving the composite condensed stream transmitted by the stream data receiving unit, and parsing into the composite condensed stream And a plurality of image decoding units, configured to receive the complex source streams parsed by the composite condensed stream parsing unit, and decode the plurality of source streams into a plurality of images; and an image A rendering unit for presenting the plurality of images.

The present disclosure provides a method for playing a composite condensed stream for use in a client player, wherein the client player is coupled to a condensed stream unit and a stream server, including: receiving the condensed And a composite condensed stream transmitted by the streaming unit; parsing the composite condensed stream into the plurality of source streams; decoding the plurality of source streams into a plurality of images; and presenting the plurality of images.

The above and other objects, features, and advantages of the present invention will become more apparent from the description of the appended claims.

10‧‧‧Network

12‧‧‧Streaming server

14‧‧‧Networking device

16‧‧‧Instant streaming content

18‧‧‧Stored streaming content

100‧‧‧Concentrated flow unit

110‧‧‧Streaming server

120‧‧‧Instant source streaming content

130‧‧‧Saved source streaming content

140‧‧‧Streaming content group

150‧‧‧User Player

160‧‧‧Complex Concentration Request

165‧‧‧Composite concentrated stream

170‧‧‧Customer requirements

175‧‧‧Source inquiry

180‧‧‧Response message

190‧‧‧session request

195‧‧‧Source streaming content

210‧‧‧Streaming content analysis unit

220‧‧‧Streaming data input unit

230‧‧‧Screen Time Stamp Adjustment Unit

240‧‧‧Streaming data output unit

250‧‧‧Service Quality Controller

260‧‧‧Customer requires processor

270‧‧‧Stream input table

275‧‧‧Content Importance Score Table

280‧‧‧Screen Time Stamp Adjustment Table

290‧‧‧Key picture sequence generation unit

295‧‧‧ Identification label and content list

301‧‧‧I screen

302, 304, 306‧‧‧ key picture sequence

303, 305‧‧‧P picture

410‧‧‧Source content input unit

420‧‧‧Content grouping unit

430‧‧‧Content group transfer unit

440‧‧‧Direct stream unit

450‧‧‧Code Format Analysis Unit

455‧‧‧Speech group analysis unit

460‧‧‧Static group list

465‧‧‧Group record list

510‧‧‧Streaming data receiving unit

520‧‧‧Composite Concentrated Stream Analysis Unit

530‧‧‧Image decoding unit

540‧‧‧Content Request Processing Unit

550‧‧‧Single area image presentation unit

555‧‧‧Multi-section image presentation unit

Figure 1 is a schematic diagram showing the architecture of a streaming system.

Figure 2 is a diagram showing a system architecture in accordance with an embodiment of the present disclosure.

Figure 3 is a block diagram showing the structure of a concentrated stream unit in accordance with an embodiment of the present disclosure.

Figure 4 is a block diagram showing the structure of a concentrated stream unit in accordance with an embodiment of the present disclosure.

5A-5C are diagrams showing a key picture sequence generating unit generating a key picture according to the embodiment of the present disclosure.

Figure 6 is a diagram showing the output of a stream content analysis unit selection in accordance with an embodiment of the present disclosure.

FIG. 7 is a schematic diagram showing a system architecture for a user-side player to query a streaming server through a condensed stream unit and directly obtain streaming content of a certain source according to an embodiment of the present disclosure.

Figure 8 is a block diagram showing a streaming server in accordance with an embodiment of the present disclosure.

Figure 9 is a block diagram showing the architecture of a client player in accordance with an embodiment of the present disclosure.

Figure 10 is a flow chart showing the connection of a streaming server, a condensing stream unit and a client player in accordance with an embodiment of the present disclosure.

Figure 11 is a flow chart showing the connection of a streaming server, a condensing stream unit and a client player in accordance with an embodiment of the present disclosure.

Figure 12 is a flow chart showing the grouping and transmission of input source of content in a streaming server in accordance with an embodiment of the present disclosure.

13A-13B are flow diagrams showing a method of generating a composite concentrated stream in a concentrated stream unit in accordance with an embodiment of the present disclosure.

14A-14B are flowcharts showing a method for receiving, processing, and playing a composite condensed stream in a client player in accordance with an embodiment of the present disclosure.

Figure 2 is a diagram showing a system architecture in accordance with an embodiment of the present disclosure. The system disclosed herein includes a condensed stream unit 100, a stream server 110, and a client player 150.

In this system, a plurality of instant-released live streaming content 120 and pre-stored streaming content 130 that have been stored on some storage media for streaming will pass via An Internet Protocol (IP) network is published or transmitted to the streaming server 110 in a wired or wireless connection. The term "streaming" as used herein is a multimedia processing mode that allows a user (e.g., client player 150) to simultaneously play a portion of the downloaded multimedia content while downloading the multimedia content. Unlike a traditional "download-and-play" mode, a user is allowed to play its multimedia content only when all multimedia content has been downloaded. This "streaming" mode supports instant multimedia. deal with. A streaming content refers to multimedia content from a single source, such as video, audio, text, image, or other media or any combination thereof. For example, the output of a network surveillance camera (IPCam) is an instant source streaming content (120), and a video content stored as a file (such as MP4 or 3GP) is a stored source stream content (130). . MP4, the full name of MPEG-4 Part 14, is a multimedia computer file format using MPEG-4. The file name is .mp4, which is mainly used for storing digital audio and digital video. The 3GP file is a simplified version of MP4. The image part can be MPEG-4 Part 2, H.263 or MPEG-4. Part 10 (AVC/H.264) and other formats are stored, and the sound part supports AMR-NB, AMR-WB, AMR-WB+, AAC-LC or HE-AAC as the encoding of the sound. The file formats and encoding formats referred to herein are for illustrative purposes only and are not intended to limit the scope of the disclosure.

The streaming server 110 can be a device or a node, such as a workstation, a computer, or other processor, for processing and receiving streaming content information in real time, and the information can be a location and a link. A description of the address and media content, such as the Real Time Streaming Protocol Uniform Resource Identifier (RTSP URI). The streaming server unit may include a plurality of internal modules for performing audio and video content reception, audio and video content transmission, audio and video content recording, and audio and video content classification. Depending on the complexity of the design, the number of concurrent session sessions, and the speed of the operation, the module can be implemented in hardware or software. Each of the stream contents received by the streaming server 110 is transmitted via a separate stream streaming session.

The streaming server 110 analyzes the attributes and settings of the video content and classifies all of the input video content in accordance with the results of the analysis. In an embodiment, the manner of classification may include, but is not limited to, an encoding profile according to the audio and video content, such as an encoder category (H.264, MPEG4, MJPEG, H.263, etc.), a compressed bit rate. (encoding bitrate), group of picture (GOP) size, frame rate, or image format, such as image resolution, but it may be According to geographical attributes, such as Global Positioning System Coordinates (GPS Coordinates), rights organization properties of audio and video content, such as community, company units, etc., content descriptions, such as film abstracts, authors or other categories the way.

The classified audio and video content is transmitted to the condensed stream unit 100 in the same manner as the group 140. In this embodiment, as shown, the streaming server 110 divides all of the input video content into a streaming content group 140 and a streaming content group II 140. In this embodiment, the concentrating stream unit 100 can be present in a separate machine, and in this embodiment, the method can be transmitted in a streaming manner through a network interface (socket). In other embodiments, the concentrating stream unit 100 may also exist in the same machine as the streaming server 110 in the form of a module. In this embodiment, the manner of transmission may be through a local interface. Or other ways to share a piece of physical memory.

The client player 150 can request the streaming content of each source directly from the streaming server 110 through the network, or can request the concentrated streaming unit 100 to provide a composite concentrated stream of a certain content group for playing. The rich stream unit 100 can transmit the content composite stream 165 to the client player 150 via a composite streaming stream request 160.

The client player 150 can be a personal computer (PC), a notebook, a mobile device (such as mobile phones, digital players, personal digital assistants (PDAs), and Mobile terminals, or other types of terminals that can access the network.

It should be noted that the concentrating stream unit 100, the stream server 110 and the client player 150 can be connected to each other by wired and wireless transmission through an Internet Protocol (IP) network. Wired connections can pass through optical networks, asymmetric digital subscriber lines (Asymmetric Digital Subscriber) Transmission media such as Line (ADSL), Cable Network, and Powerline Network. Wireless connectivity can be made up of mobile/wireless/Internet Protocol (IP) network connectivity through general mobile/wireless networking technologies, such as Global Mobile Communications, which is the 2nd generation of mobile technology protocols. System (Global System for Mobile Communications, GSM) and Code Division Multiple Access (CDMA), 2.5th generation of mobile communication technology protocol General Packet Radio Service (GPRS), wireless local area network (wireless local area network, WLAN), and 3rd generation mobile communication technology agreement or related to 3rd generation and Wideband Code Division Multiple Access (WCDMA), Universal Mobile Telecommunications (Universal Mobile Telecommunications) System, UMTS), Enhanced Data Rate for GSM Evolution (EDGE), Wireless Enhanced Data Rate GSM Evolution (Wireless EDGE, WEDGE), Time-Division-Synchronous Code Division Multiple Access (Time) Division-Synchronous Code Division Multiple Access (TDSCDMA), the high-speed downlink of the 3.5th generation mobile communication technology agreement Packet access (High-Speed Downlink Packet Access, HSDPA), WiMax (Worldwide Interoperability for Microwave Access, WiMAX), and 3GPP Long Term Evolution (3GPP Long Term Evolution, 3GPP LTE).

The streaming content source communication between the condensing stream unit 100, the streaming server 110 and the client player 150 can be transmitted and controlled by a Real-Time Streaming Protocol (RTSP). Real-time Transport Protocol (RTP) and Instant Transmission Control Protocol (Real-time Transport Control Protocol, RTCP) can be completed by transmission control protocol (TCP) or User Datagram Protocol (UDP).

Figure 3 is a block diagram showing a condensed stream unit 100 in accordance with an embodiment of the present disclosure. The concentrating stream unit 100 includes a stream data input unit 220, a stream content analysis unit 210, a screen time stamp adjustment unit 230, a stream data output unit 240, a service quality controller 250, and a client request processing. 260.

First, when the video content of the streaming content group 140 is input to the condensing stream unit 100, it is transmitted to the streaming content analyzing unit 210 through the streaming data input unit 220, and the streaming data input unit 220 will output the strings. All of the source stream contents input in the stream content group are recorded in a stream input table 270. For example, the stream input table 270 records a group identification code and a grouping category including each stream content group, such as an encoding attribute (such as an encoding format, a frame rate or an image format), and a semantic attribute ( Such as geographic location, shooting event or photographer organization or community relationship and group), or static attributes (such as preset specific grouping), and individual attributes of each source stream (such as compression settings, geographic coordinates, audio and video content) Authority organization, content description, author or other attributes). Specifically, but not by way of limitation, the stream input table 270 can be transmitted using an RTP Header Extension and a Real-time Transport Control Protocol (RTCP) packet of the instant transfer protocol. Different source streams will have their own independent identification codes as identification tags. For example, MD5 hash method can be used to hash the original resource identifier (URI) of each content, shooting time or photographer. Calculate to produce a unique identification symbol. The stream input table 270 is updated as the source content changes (new or left).

In other embodiments, when a plurality of streaming servers are coupled to the same condensing stream unit, the stream input table 270 also records that all source stream contents input in each stream content group are from Streaming server.

The input stream content group 140 performs a data content analysis and an importance score indicator analysis on the data in the stream in the stream content analysis unit 210, and stores the score index of the content importance obtained by the analysis in the content. The importance score is in table 275. For example, the importance score table 275 records the weights of the stream content of different sources, the detected events and accumulated scores, and the categories, weights, number of key sequence pictures outputted, output score thresholds, and selected outputs of the analyzed events. The amount of information on the screen, etc. In one embodiment, the content analysis may be performed by means of video analysis to determine the importance of the content by detecting certain special events, including but not limited to the degree of change of the image, the person Face detection, remnants, lost objects, or license plate detection, etc., to calculate the importance score indicator to take out the representative content of the screen. More specifically, for example, the importance score indicator can be scored by the strength of the event. In different areas of analysis, it may be the amount of change (such as the degree of image change) and the block size of the detected picture (such as legacy/loss detection). In some embodiments, the scores obtained by the various analyses can be weighted again so that the scores of the different events can be compared for subsequent score calculations.

In another embodiment, if a specific video-related event is not detected, the data amount or the screen interval of the screen input by each source stream content may be checked to determine a certain threshold value. The importance score indicator of the picture. For example, if the content of each source stream is set to be transmitted to the client player 150 at least within a certain time interval T, if after the time interval T, There is no event in a source stream content, and the source stream content also has a high output importance.

Compared with an inter-coded frame, such as a P-picture (Predictive frame, P frame), an Intra-coded frame, such as an Intra frame (I frame), has its own Complete picture information. Therefore, in some embodiments, the in-frame coded picture will also have higher importance and representativeness.

According to the importance score indicator recorded in the content importance score table 275, the stream content analysis unit 210 can determine whether a picture or a clip at a certain point in the source stream content of an input has sufficient importance and is The output is selected as part of the Condensed Stream Content.

Further, from another perspective, the streaming content analysis unit 210 is configured to output a picture or a video segment representing the content when an input stream is considered to have sufficient importance score indicators. Another embodiment is to use a key frame (usually an intra-coded picture) before or after the event as a representative picture, because the in-frame coded picture leaflet The screen itself retains the complete data of the original content and can be transmitted separately and played on the client player 150.

Figure 4 is a block diagram showing a condensed stream unit 100 in accordance with an embodiment of the present disclosure. In this embodiment, the stream content analysis unit 210 further includes a key picture sequence generation unit 290. If the picture selected by the stream content analysis unit 210 as the representative content is not a key picture, the key picture sequence generation unit 290 can be used to encode and output the representative content picture in frame coding. 5A-5C shows a key picture sequence generating unit according to an embodiment of the present disclosure 290 produces a schematic of the key picture. As shown in the figure, I and P in the figure represent I pictures and P pictures, respectively, and the I picture is the key picture of the source stream content. In this embodiment, I picture plus connection is used. The two P-pictures are described as implementations of the key picture sequence, but in some embodiments, whether or not to use the key picture sequence and the total number of pictures selected will vary with event category, importance, and available bandwidth. As shown in FIG. 5A, if the representative content screen selected by the stream content analysis unit 210 is the key screen I screen 301, the stream content analysis unit 210 selects the I picture 301 and the subsequent two P pictures as one. Key frame sequence 302. If the representative content picture selected by the stream content analysis unit 210 is the P picture 303, as shown in FIG. 5B, the stream content analysis unit 210 selects the P picture 303 and the previous I picture and P picture to form a P picture 303. The key picture sequence 304 is output. If the representative content picture selected by the stream content analysis unit 210 is the P picture 305, and the P picture 305 does not have a close I picture, as shown in FIG. 5C, the key picture sequence generation unit 290 will The P picture 305 re-encodes and outputs a key picture I picture in an in-frame coding manner, and composes the I picture and the subsequent two P pictures into a key picture sequence 306. Such an embodiment has a problem that the length of the Group Of Picture (GOP) is particularly long, which avoids the problem that the key picture of the original stream data is too long apart from the output picture time of the representative content picture.

The service quality controller 250 continuously detects the network and buffer status during data transmission with each of the client players 150, and determines whether the amount of data currently transmitted exceeds the network connection, or whether there is still Increase the margin of information. The service quality controller 250 feeds back the result of the judgment to the stream content analysis unit 210 to adjust the degree of the condense ratio and the critical value of the importance score indicator so that the concentrated data amount can be Close but not super The bandwidth that the user can afford on the network. For a detailed technical content of the service quality controller 250, for example, refer to US Patent US Pub. 2010/0161761 Yu et al. (Method for Audio and Video Control Response and Bandwidth Adaptation Based on Network Streaming Applications and Server Using the Same).

A plurality of different input contents in the streaming content group 140, after passing through the streaming content analyzing unit 210, select an image or a video segment to represent each input content at each time point as an output. Depending on the level of concentration ratio, it is possible to select a picture or segment to be a composite concentrated stream from one or both streams at each time point. In a specific example, if the bandwidth estimated by the quality of service controller 250 is not good, there will be a higher concentration level and may only detect the specified event in the source stream content. The stream content analysis unit 210 outputs a picture corresponding to the event source. On the other hand, if the bandwidth is allowed, there may be a lower concentration level or a lower importance selection threshold. In this case, the source stream content of each input may have the picture streamed. The analysis unit 210 selects and outputs. In some embodiments, whether to choose to output a single key picture or a sequence of key pictures, or whether to select multiple input sources at the same time point, may also be related to the limitations and capabilities of the client player. For example, if some client players cannot simultaneously decode multiple channels of video, or have only a single display area, then a higher concentration ratio may be used and multiple streams of content may not be output at the same time.

Further explanation of the adjustment of the concentration ratio, for example, a lower concentration ratio level, may mean that the threshold value for the importance score indicator may be looser (and thus more different events are detected and selected) or Is for Detect The measured event can output more consecutive pictures. Similarly, a higher concentration ratio may mean that each detected event can output fewer pictures or the threshold used to determine the event is more stringent (so the detected events become less). In some embodiments, such a concentration ratio level may be a dynamic adjustment or a plurality of different configuration settings may be selected to accommodate different streaming content, events to be detected, and different network bandwidth conditions.

In another embodiment, when the selected picture content has its importance score indicator reaching a certain degree, and the bandwidth condition reported by the service quality controller 250 allows the lower level of the enrichment ratio, the streaming content analysis unit 210 also The selected image and the image segments of its subsequent time range can be output to provide more condensed content that is easier to understand. In one embodiment, the video segment composed of the selected picture and the subsequent time range may refer to an image segment starting with a key picture before or after the picture, as shown in FIGS. 5A and 5B. In another embodiment, as shown in FIG. 5C, the video frame composed of a key picture and a subsequent stream may be regenerated by the key picture sequence generating unit 290.

Specific examples, but not limited to, Table 1 shows an importance score indicator table, and Table 2 shows a content importance score table. Tables 1 and 2 are an embodiment of a method of selecting a picture by a content concentration score table and an importance score indicator table as a composite condensed stream. The stream content analysis unit 210 analyzes the input source stream content. As shown in Table 1, different event types are given different stream content groups according to importance. The stream content analysis unit 210, after detecting the event type in the different source stream content, gives an importance score according to the current weighting factor. As shown in Table 2, the result update is recorded in the content importance score indicator table.

Figure 6 is a diagram showing the output of a stream content analysis unit 210 in accordance with an embodiment of the present disclosure. In another embodiment, the content of the composite condensed stream output after being selected by the streaming content analysis unit 210 may be as shown in FIG. 6. The image in the figure is divided into an I picture and a P picture. The digital meaning below the picture means, for example, 1-2 represents the video source identification code is 1, and the time stamp of the picture is 2. Since the time points at which the respective image sources start to transmit may not be consistent, images from different sources may have their own timestamp sequences.

The composite condensed stream outputted by the stream content analysis unit 210 is input to the screen time stamp adjusting unit 230. Because of the decoding dependencies between the video frames, the I picture and the P picture of the same image source must be continuous, so In this embodiment, it is necessary to continuously output the same source and the decoding dependent picture by means of adjusting the time stamp. It can be noted from the diagram of Fig. 6 that each picture in the composite condensed stream is accompanied by the processing of time stamps, and the output of the composite condensed stream has an arrangement of pictures that is not aligned with the original stream content in each source. The same (for example, the timestamp order of the frame in the composite condensed stream is 1-1, 1-2, 1-3, 3-82, but among the original source stream content, by time In order of view, the timestamps appearing at timestamps 3-82 are before timestamps 1-3), so the timestamps in the composite condensed stream may not increase in a monotonically increasing manner. As the bandwidth of the user-side player 150 changes in each time unit, the amount of the combined concentrated stream content is also changed (as shown in the lower right of Figure 6).

In some embodiments of the system, limited by the capabilities of the client player 150 (eg, displaying a partition window or a limitation of decoding unit resources), the client player 150 must play with a single decoding unit. And the same display window alternately plays a composite concentrated stream formed by streaming content from different sources. Thus, in these embodiments, the non-sequentially incremented timestamps can cause confusion in the playback time of each of the screens of the client player 150. Therefore, in these embodiments, the time stamp of each picture in the composite condensed stream is adjusted by the picture time stamp adjusting unit 230, so that the user end player 150 in these embodiments can play the composite condensed string in a correct and smooth timing. The various pictures in the stream, here, contain the sequence of pictures that are adjusted for playback so that the same source of content can be played continuously.

In addition, in some embodiments, the screen timestamp adjustment unit 230 may further adjust the interval between the screen and the timestamp between the screens to change the effect of changing the interval of some screens or video clips during playback, so as to change some of the presentations. The effect of important or less important content.

Specifically, but not limited to, for example, by using an Real-time Transport Protocol (RTP) to transmit an H.264 image, the unit of the timestamp field in the header of the instant transmission protocol is 1/90000. That is, every additional 1 second will increase 90,000 units. When a specific event is detected, the streaming content analysis unit 210 outputs a key picture adjacent to the event and subsequent two pictures. If the original stream content is 15 pictures per second (that is, the time stamp interval of each picture is 6000). In some embodiments, the original play time can be extended from 3/15=1/5 seconds to 15000×3/90000=1/2 seconds by adjusting the timestamp interval to 15000, increasing certain important events. Or the proportion at which the content is presented. In some embodiments, according to the capabilities and limitations of the client player, if the client player has the capability of simultaneous multi-channel image decoding and display, the respective stream content can be regarded as separate streams and each The ground is played, and the screen time stamp adjusting unit 230 only adjusts the interval of the time stamp to achieve the presentation effect of the specific event.

The timestamp before and after the composite condensed stream adjustment is recorded in the screen timestamp adjustment table 280 as a basis for subsequent image timestamp adjustment. Specifically, but not by way of limitation, the content recorded by the screen timestamp adjustment table 280 may include a timestamp currently input by each source stream content, a timestamp scale (Scale), and a timestamp used by the composite condensed stream. Table 3 lists the contents of the screen timestamp adjustment table in one embodiment, where the fields record the conversion correspondence of each timestamp and the conversion scale for each source in each concentrated stream. For follow-up Refer to use when converting the stamp. The condensed stream identification code is generated when a user requires a composite condensed stream (that is, every time a single session is established), and different users may generate different contents even if they request the same stream group content. Concentrate the stream identification code. For example, the MD5 hash method can be used to hash the condensed stream resource identifier (URI) (ie, the group of composite condensed streams), the viewing time, or the viewing session to generate a unique identifier and store it. This symbol is for subsequent correspondence and query. The timestamp and timestamp scale of the output may also differ due to the time, bandwidth, device capability of the requesting end of the client, or the type of event and its weight that are of interest in the composite condensed stream.

The composite condensed stream containing each stream content from different sources is finally sent to the stream data output unit 240, mixing all content from different sources and transmitting using the same single session. Therefore, for the client player 150, only one session connection is established to receive from beginning to end. The composite concentrated stream receives the composite concentrated stream. For example, but not limited to, the concentrating stream unit 100 provides the address and connection mode of the user-side player 150 for the composite condensed stream, which may be a complete instant stream protocol resource identifier (Real Time Streaming Protocol). Uniform Resource Identifier (RTSP URI), such as rtsp://11.22.33.44:554/condensed_stream_123, to get a condensed stream belonging to a group named 123, or rtsp://11.22.33.44:554/condensed_stream ? Codec=h264&format=D1&gis=XYZ to obtain a concentrated stream having stream content of codec H.264, image size D1, and shooting geographic location such as XYZ. For some embodiments, when the concentrating stream unit is coupled to the plurality of stream servers, the identification information of the server may be added to specify whether the composite condensed stream to be received originates from a specific server. For example, rtsp://11.22.33.44:554/condensed_stream? Codec=h264&server=ignore can be used to get the composite condensed stream generated by all codec content with h264 attribute and ignore the server from which the content comes from; rtsp://11.22.33.44:554/condensed_stream? Codec=h264&server=server1 is a composite condensed stream that gets the server "server1" and the codec has the h264 attribute.

In some embodiments, the streaming data output unit 240, when transmitting the composite condensed stream, streams packets of video or video segments of different sources at different sources, and carries an identifier for identifying the image or the video segment from the source. The tag 295 is configured to provide the user-side player 150 with the original material content corresponding to a particular content. The identification tag 295 is here referred to as a unique identification symbol for corresponding and querying a stream of content. For example, you can use the MD5 hash method for the original resources of each content. The identifier (URI), the shooting time, or the photographer performs a hash calculation to generate a unique identification symbol and store the symbol for subsequent correspondence and query.

In some other embodiments, the streaming data output unit 240 may also carry a table 295 (Table of Condenseds) for declaring all currently concentrated source stream content when transmitting the composite condensed stream. The client player 150 knows which of the different source streams the currently received composite condensed stream contains. The content list 295 can include a condensed stream identification code, a group identification code, and a grouping category, such as an encoding attribute (such as an encoding format, a frame rate or an image format), a semantic attribute (such as a geographic location, a shooting event, or Photographer organization or community relationship and group), or static attributes (such as preset specific groupings), and individual attributes of each source in the group (such as compression settings, geographic coordinates, rights organization of audio and video content, content) Description, author, server from which the content stream originated, or other attributes). In other embodiments, the content list 295 may further include an event analysis weighting factor used by the stream analysis unit 210 as shown in Table 1, and a result list (such as an event) of the event analysis as shown in Table 2. Type and its weight). Specifically, but not limited to, for example, the identification tag and the content list may be transmitted by the RTP Header Extension and the Real-time Transport Control Protocol (RTCP) packet of the instant transmission protocol, wherein Different source streams will have their own independent identification codes as identification tags. For example, MD5 hashing can be used to generate a unique identification symbol. The frequency of the content list can be transmitted periodically or when the source content changes (new, left, or event updates). Further exemplifying, the foregoing identification tag attached to the extended field may be used as an index when the content list is queried, so that the user-side player may receive different streaming content in the composite concentrated stream. Information can be used The tag identifies the original stream content and queries it in the content list.

The client requires the processor 260 to be a processing module for receiving the request from the client player 150. When the client player 150 receives the composite stream and plays it, it can parse out the identification tag or content list that is captured by the stream data output unit 240 to identify the source of the content. If the user is interested in the picture or video clip of the streaming content of the source, the client player 150 can request the processor 260 to send the request to the source through the identification tag or the content list associated with the source stream. A request to stream content.

When the client requests the processor 260 to receive such a request, the mapping and query may be performed on the identification tag and the content list 295 with the identification tag or content list used by the client player. And knowing which input source the user requested by the client player 150 is (through the query stream input table 270). In some embodiments, the request from the client player 150 can be an importance weight that changes the content of a certain source stream. In this embodiment, the weight of the streaming content analysis unit 210 when calculating the importance score indicator may be changed permanently or temporarily, so that the output composite concentrated stream can be raised (or lowered) from the permanent or temporary The proportion of streaming content from a source.

In still other embodiments, the streaming content analysis unit 210 may also be required to permanently or temporarily raise (or decrease) the weight of the importance score indicator for a particular event analysis.

In other embodiments, the request from the client player 150 may also request an original source stream content from the streaming server 110 directly through the stream data input unit 220. Figure 7 shows a user-side playback in accordance with an embodiment of the present disclosure. A schematic diagram of the system architecture that the concentrator 150 queries the streaming server 110 through the condensing stream unit 100 and directly obtains the streaming content of a certain source. In this embodiment, the client player 150 can directly request to view a source stream content originating from the streaming server 110 and not being subjected to the condensing process through a session request 190 (request session). After receiving the session request 190, the streaming server 110 transmits the requested source stream content 195 to the client player 150 for playback.

As shown in FIG. 7, during the operation of the condensing stream unit 100 in the above manner and providing the composite condensed stream to the user player 150 for receiving, the streaming data input unit 220 passes the check of the input stream content group I 140. And streaming content group II 140, capable of dynamically updating the stream input table 270, the content importance score table 275, and the screen timestamp adjustment table 280 in response to the source stream content added or removed from the streaming server 110. Add new source stream content to the composite condensed stream (or remove the source stream content from the composite condensed stream). The stream data output unit 240 also updates the identification tag and content list 295 entrained in the composite condensed stream in accordance with the newly added or deleted stream input table 270. By continuously updating the identification tag and the content list 295, the client player 150 can understand the source stream content currently concentrated on the composite condensed stream and the content of the source stream by parsing the contents of the identification tag and the content list 295. New changes such as deletions.

In addition, those skilled in the art related to streaming transmission should be able to easily understand that when receiving a composite condensed stream, connections established by different client players 150 will have different quality of service controllers 250. The result is that different client players 150 will also send different composite condensed stream requirements 160 according to different user requirements. Therefore, it should be known from the above architecture that even if it is required to receive the composite concentrated stream 165 generated by the same streaming content group, Because the different quality of service controllers 250 result with different composite condensed stream requirements 160, the composite condensed stream 165 received by each of the client players 150 will have a different ratio of concentrated data volume to content condensed composition.

In addition, the client player 150 can present a client request 170 to certain specific content within the composite condensed stream, indicating that it is desirable to be able to receive the original and unprocessed source stream content corresponding to the source. The concentrating stream unit 100 transmits a source query 175 to the streaming server 110 in accordance with the client request 170 requested by the client player 150 to query the source of the source stream content. After receiving the source query 175 of the condensed stream unit 100, the streaming server 110 returns a response message 180 for replying to the address of the source content requested by the condensing stream unit 100 to establish the stream transmission address and Connection method. Finally, the condensing stream unit 100 transmits the response message 180 replied by the stream server 110 to the client player 150.

Figure 8 is a block diagram showing a streaming server 110 in accordance with an embodiment of the present disclosure. As shown in FIG. 7, the streaming server 110 includes a source content input unit 410, a content grouping unit 420, a content group transmitting unit 430, a direct stream unit 440, an encoding format analyzing unit 450, and a semantic meaning. A group analysis unit 455, a static group list 460, and a group record list 465.

First, a variety of different source streaming content, including the instant release of the instant source streaming content 120 and the stored source stream content 130 that has been stored on some storage media for streaming, are input to the streaming server 110. Source content input unit 410. The direct stream unit 440 can accept the connection request of the client player 150 and transmit the stream content received by the source content input unit 410 to the client player 150.

In addition to the design of the above-described conventional stream server, the present disclosure When the respective source stream content is input to the source content input unit 410, it is transmitted to a content grouping unit 420. The content clustering unit 420 classifies the input source stream content according to the result of the content analysis. In some embodiments, the content grouping unit 420 can use an encoding format analyzing unit 450 to classify according to attributes such as an encoding format, a frame rate, or an image format used by the streaming content. In still other embodiments, it is also possible to use a semantic group analysis unit 455 to classify the attributes of the source stream content in terms of meaning descriptions. Specifically, but not limited to, it can be grouped using geographic locations, shooting events, or high-level attributes such as photographer organization or community relationships and groups. In other embodiments, a static group list 460 may also be used to classify each input source according to a statically defined classification manner, such as presetting certain source identifiers into specific groups or certain network addresses. The source is a specific group.

According to the result of the grouping by the content grouping unit 420, the source streams of the respective inputs are sent to the condensing stream unit 100 in the manner of the streamed content group 140 in the content group transmitting unit 430 as a result of the grouping and for each stream. The streaming content in the content group 140 produces a composite condensed stream. The data of the streaming content group 140 to which each input source stream belongs is currently recorded in the group record list 465. For example, the content of the group record list includes the category group to which each content belongs (for example, content A belongs to group 123); or in other embodiments, it may be the category of each content. Attributes (for example, content B is in the class of the encoder H.264 and the attribute in the geographic location is in the XYZ area). In addition, the group record list 465 also includes an nickname to be used when each content is sent to the condensed stream unit or an identification tag representing a group or category attribute. More specifically, for example, some embodiments may use some specific transport port to convey the content of a certain group. (Example, port 5556~5560 is used to transfer the streaming content belonging to group 123); or some other embodiments may use some unique attribute tags to be appended to the fields in the content delivered to the condensed stream unit (eg, For the content with the encoder H.264 attribute, add the label "h264" when transmitting). In some embodiments, attribute information with the name of the server itself may also be added to enable the concentrating stream unit to recognize the stream server from which each group is derived when coupled to the plurality of stream servers.

In still other embodiments, the group record list 465 can also include attributes of the groups or categories represented or based on the grouping results. For example, such as encoding attributes (such as encoding format, frame rate or image format), semantic attributes (such as geographic location, shooting events or photographer organization or community relations and groups), or static Attributes (such as preset specific groupings), and individual attributes of individual sources within the group (such as compression settings, geographic coordinates, permission organization of audio and video content, content descriptions, authors, or other attributes) and streaming content addresses (such as the Instant Streaming Protocol Resource Identifier (RTSP URI)).

The manner of transmitting the clustered result to the condensed stream unit mentioned here, for example, but not limited to, the method of transmitting the stream content group 140 may be the result of periodically transmitting the record in the group list 465, so that the condensed The stream unit 100 knows information such as groups or categories to which the respective sources belong in the stream content group 140, and may also add common identification tags to the stream contents belonging to the same group, or through different transmission ports ( Port) to deliver streaming content from different groups. Specifically, in the embodiment adopting the Instant Transfer Protocol (RTP), the group list or the group identification information of the group that transmits the streaming content can be extended through the header of the instant transmission protocol (RTP Header). Extension) to add identification tags and Real-time Transport Control Protocol (RTCP) packets to transmit group columns table. The identification tag here can use a representative character with a representative meaning, or some unique special coded characters, such as MD5 hash code.

In the process of grouping, if there is a newly added source stream content in the stream server 110 at this time, the content of the newly added source stream content is determined by the content grouping unit 420 analyzing the content automatically according to the method described above. Group. The content stream transmitting unit 430 is notified through the content group transmitting unit 430 that the stream data input unit 220 in the stream server 110 has the newly added source stream content, and the stream input table 270 is updated and sent to other content in the belonging group. The stream unit 100 is concentrated to produce an updated composite concentrated stream.

In a similar situation, if the source stream content stops transmitting or leaves the original group, the content group transmitting unit 430 notifies the stream data input unit 220 in the condensed stream unit 100. An input source stream content is removed from the stream input table 270. For example, but not limited to, the stream data input unit 220 can be notified of the departure of a source stream by updating the group list or a packet with a special command.

In some embodiments, the streaming content of the same source may also belong to the plurality of classified streaming content groups 140 at the same time. More specifically, the streaming server 110 can transmit only one source stream content to the condensed stream unit 100 in such an embodiment, but the source stream content is included in the plurality of group lists. However, in other embodiments, a source stream content may be transmitted to the concentrating stream unit 100 in each of the associated stream content groups 140, for example, by establishing a connection through a different port. The same stream content is transmitted for different groups; or the tags are identified by different sources and attached to a stream of content as identification intervals.

Figure 9 shows a client player in accordance with an embodiment of the present disclosure Architecture diagram of 150. The client player 150 includes a stream data receiving unit 510, a composite condensed stream analyzing unit 520, at least one image decoding unit 530, and an image presenting unit 540. The client player 150 can be a portable mobile device.

The client player 150 is divided into two parts. One part is to receive the individual source stream content 195 from the direct stream unit 440 of the stream server 110 through the stream data receiving unit 510, and then through an image decoding unit. After decoding 530, a Single Area Video Rendering Unit 550 is used to present the picture in a single window. The function of the conventional client player is the design of this part.

In the disclosure, in addition to the design of the conventional client player, when the stream data receiving unit 510 receives the composite condensed stream 165 sent from the condensing stream unit 100, specifically Obtaining a concentrated stream belonging to a group named 123 by using a Real Time Streaming Protocol Uniform Resource Identifier (RTSP URI), such as rtsp://11.22.33.44:554/condensed_stream_123. Or as rtsp://11.22.33.44:554/condensed_stream? Codec=h264&format=D1&gis=XYZ to obtain a concentrated stream having stream content of codec H.264, image size D1, and shooting geographic location such as XYZ. The composite concentrated stream is sent to a composite concentrated stream analysis unit 520. In the composite condensed stream parsing unit 520, the identification tags or content lists associated with the respective source stream contents originally concentrated in the composite condensed stream 165 are parsed separately. The identification tag allows the client player to distinguish and identify different content in the composite condensed stream. The content list may include a group identification code and a grouping category, such as encoding attributes (such as encoding format, frame rate or image format), semantic attributes (such as geography). Location, shooting event or photographer organization or community relationship and group), or static attributes (such as preset specific groupings), and individual attributes of each source stream (such as compression settings, geographic coordinates, rights organization of audio and video content) , content description, author, server from which the content stream originated, or other attributes). In other embodiments, the content list 295 may further include an event analysis weighting factor used by the stream analysis unit 210 as shown in Table 1, and a result list (such as an event) of the event analysis as shown in Table 2. Type and its weight). For further example, after the user-side player obtains the identification tag, the group, the category attribute, and the event analysis result of the streamed content can be further queried in the content list. More specifically, but not by way of limitation, the identification tag and the content list can be transmitted by the RTP Header Extension and the Real-time Transport Control Protocol (RTCP) packet of the instant transmission protocol.

The parsed source stream content is sent to one or more image decoding units 530 for decoding the source stream content. In some embodiments, the parsed stream content groups are sent to the respective allocated image decoding units 530 for content decoding. The images decoded from the stream content of different sources will each be presented in different split windows in the Multiple-divided Area Video Rendering Unit presentation unit 555. More specifically, but not limitedly, the split window of such an embodiment may be to separate a window of a Graphical User Interface (GUI) to simultaneously present a plurality of different images.

In other embodiments, the parsed source stream content group may be uniformly decoded by the single image decoding unit 530 due to limitation of the device of the client player 150.

In other embodiments, because the capabilities of the single-region image rendering unit 550 and the multi-segment region image rendering unit 555 are limited, the decoded image is in accordance with the time stamp in the composite condensed stream, that is, the concentrated stream. The time stamp adjusted by the screen time stamp adjusting unit 230 in the unit 100 is played in turn in a single area image presenting unit 550 of a single window. In this implementation, each input source stream content is presented in a manner that plays a single video stream, but pictures from different streams of content are presented in a manner that continuously switches in turn.

In some embodiments, when the video is played, the recognized identification tag may be displayed, so that the user can know the source stream to which each currently displayed screen belongs. In still other embodiments, the parsed content list can be presented to the user in a format that is easy for a typical user to read to let the user know the source that is currently present in the composite condensed stream. In still other embodiments, the list of content presented to the user and the identification tag can serve as input for subsequent requests to stream content for the source.

The content request processing unit 540 can obtain the identification tag or content list transmitted by the composite condensed stream parsing unit 520 and included in the composite condensed stream. Through the parsed identification tag or the content list, the user can request the processor 260 to identify the input source stream corresponding to the item in the tag or the content list to perform a composite concentration according to the interest or demand of the user in the streamline unit 100. Streaming request 160. Further, the condensed stream unit 100 generates an identification tag and a content list 295 for each stream content in each group (the identification tag and content list obtained by the client end are transmitted from the condensing stream unit 100) Therefore, a request can be made for a source stream by identifying an item of a tag or a content list. In one embodiment, the corresponding input source stream is required to be analyzed in the stream content. Unit 210 analyzes the weight of the higher or lower score indicator, thereby increasing or decreasing the proportion of the source stream in the composite concentrated stream.

In another embodiment, it may also be required to adjust the weight of certain events in calculating the score indicator during event analysis, thereby changing the proportion of certain events in the composite concentrated stream.

In another embodiment, the streaming server 110 may be required to transmit the original source stream content corresponding to the input identifier to the streaming data receiving unit 510 through the condensing stream unit 100 and play directly to the user end. Receiver and play are performed in the device 150.

When the source stream content is newly added or removed in the condensing stream unit 100, the stream data receiving unit 510 notifies the source stream content included in the composite condensed stream by updating the identification tag or the content list. Changed. When the composite condensed stream parsing unit 520 receives such an update change, the resource allocation of the client player 150 can be dynamically adjusted in accordance with the change in the identification tag or the content list. In an embodiment, but not limited to the limitation, the user-side player 150 may add the image decoding unit 530 to the multi-segment area image presentation unit 555 on the premise of the resource permission when the new input source stream is found. An additional multi-split window to present the newly added source stream content; if the source stream content is found to be removed from the stream content group, the resource originally assigned to decode and render the source stream content may be released.

Figure 10 is a flow chart showing the connection between the streaming server, the concentrating stream unit and the client player according to the embodiment of the present disclosure, and referring to Fig. 2. First, in step S610, the streaming server 110 first transmits the source stream contents belonging to the same group to the condensed stream unit 100 in units of groups. Then, In step S620, the client player 150 can transmit a composite concentrated stream request to the condensed stream unit 100 to request a composite condensed stream of a stream content group.

If the streaming content group required by the client player 150 is present and valid, in step S630, the condensing stream unit 100 transmits the composite condensed stream of the streaming content group to the client player 150. .

While the client player 150 receives and plays the composite condensed stream, in step S640, the client player 150 can request the content of the specific stream stream in the composite condensed stream according to the user's source. Interest in streaming content requires increasing or decreasing the proportion of the source stream content in the composite condensed stream. In still other embodiments, the requirement may also be to adjust the proportion of certain events in the composite condensed stream content.

FIG. 11 is a flow chart showing the connection between the streaming server, the condensing stream unit and the client player according to the embodiment of the present disclosure, and referring to FIG. 2 . First, in step S610, the streaming server 110 transmits the source stream contents belonging to the same group to the condensed stream unit 100 in units of groups. Next, in step S620, the client player 150 can transmit a composite concentrated stream request to the condensed stream unit 100 to request a composite condensed stream of a certain stream content group.

If the streaming content group required by the client player 150 is present and valid, in step S630, the condensing stream unit 100 transmits the composite condensed stream of the streaming content group to the client player 150. .

While the client player 150 receives and plays the composite condensed stream, in step S640, the client player 150 can present a client request 170 to some specific source stream content in the composite condensed stream, indicating Hope to be able Receives source stream content that corresponds to the original stream content and is not condensed.

The concentrating stream unit 100 transmits a source query 175 to the streaming server 110 for inquiring the source stream content in accordance with the source stream content requested by the client player 150 in step S650.

After receiving the source query 175 of the condensed stream unit 100, the streaming server 110 replies back to the condensed stream unit 100 a response message 180 in step S660 to reply to the source stream content of the query. Used to establish the address and connection method of the stream transmission. For example, but not limited to, the address and connection mode may be a complete Real Time Streaming Protocol Uniform Resource Identifier (RTSP URI), such as rtsp://123.4.5.6:554 /live_source.

In step S670, the condensing stream unit 100 transmits the response message 180 replied by the stream server 110 in step S660 to the client player 150.

After receiving the response message 180 replied from the condensing stream unit 100, the client player 150 directly requests the source server string to the streaming server 110 according to the address and connection manner specified in the content of the response message 180 in step S680. A session request 190 of streaming content.

The streaming server 110 checks the connection requirements of the source stream content. If successful, in step S690, a session connection of the stream content is established and the required source stream content 195 is transmitted to the client player 150.

[Method of grouping and transmitting streaming content sources by streaming server]

Figure 12 is a flow chart showing the process of grouping and transmitting input source stream content in the streaming server 110 in accordance with an embodiment of the present disclosure, and referring to the eighth Figure. First, in step S810, the source content input unit 410 in the streaming server 110 receives the stored source stream content 130 and the instant-published source stream content 120. Next, in step S870, the streaming server 110 determines whether the received source stream content has new source stream content.

If the newly added content is found (YES in step S870), in step S820, the content grouping unit 420 analyzes and checks the stream content group to which the newly added source stream content belongs. Here, depending on the embodiment, a method of analyzing the encoded setting or the image format may be used (step S850), an analysis method according to the semantics provided by the metadata (step S855), or a static grouping list (step S860). )based on. In step S880, the result of the analysis is updated to the group record list 465.

If the new source stream content is not found (NO in step 870), in step 875, the content grouping unit 420 determines whether the input source stream content has a source stream leaving or has stopped sending the source string. flow. If it is found that the input source stream content has a source stream leaving or has stopped sending the source stream ("Yes" in step 875), in step 880, the source stream content is also moved from the group record list 465. except. Finally, in step 830, according to the group information of the group record list 465, the content grouping unit 420 transmits the source stream content to the content group transfer unit 430 in a group manner, and then the content group transfer unit. 430 is transferred to the concentration stream unit 100.

If the source stream content of the input is not found to have the source stream content leaving or the source stream content has been stopped (NO in step 875), in step 830, the group information directly in accordance with the group record list 465 is directly followed. The content grouping unit 420 transmits the source stream content to the content group transmission unit in a group manner. 430 is further transmitted by the content group transfer unit 430 to the condensed stream unit 100.

In addition, when the source stream content is received in step S810, in step S840, the stream server 110 may also directly transmit the original source stream content to the client player 150 in a streaming manner. In step S845, the condensing stream unit 100 may transmit a client request 170 to the streaming server 110, query the acquisition manner of the source stream content or obtain the source stream content address used in step S840, and The stream server 110 returns the obtained acquisition mode and address to the condensing stream unit 100.

13A-13B show a flow chart of a method for generating a composite concentrated stream in the concentration stream unit 100 in accordance with an embodiment of the present disclosure, and reference is made to FIG.

First, in step S710, the streaming data input unit 220 in the condensing stream unit 100 receives the streaming content group that is grouped and streamed by the grouping via the streaming server 110. Next, in step S715, the stream material input unit 220 updates the received stream content group to the stream input table 270 (step S760).

In step S720, the streaming content analysis unit 210 performs content analysis on each of the source stream contents in the stream content group. In various embodiments, by way of example and not limitation, the method of analysis may be to detect events such as picture changes, object detection, face detection, or license plate detection. The way of analysis can also be the way of analyzing the accumulated data volume or time interval when there are no special events. According to the analyzed events and contents, the content of each source analyzes an importance score indicator and stores it in the content importance score table 275 (step S765) for continuous tracking statistics.

If the importance score indicator of a source's content meets a certain condition, it is selected as the output to the composite concentrate stream. In some embodiments, the key painting The face sequence generating unit 290 determines in step S755 whether or not the picture selected to output the content needs to be regenerated as a key picture. If the key picture needs to be regenerated (YES in step S755), in step S725, the content picture is again output as a key picture and/or its subsequent segments to the composite condensed stream.

In other embodiments, the key picture sequence generating unit 290 may not be inspected, and the key picture adjacent to the content picture and/or its subsequent segments may be directly output to the composite condensed stream.

In step S730, the screen timestamp adjustment unit 230 adjusts the timestamp of the selected composite condensed stream, the content pictures that may have been adjusted over time or from different sources, in some embodiments. A client player 150 can play in the manner expected by the streamline unit 100. In step S770, the screen time stamp adjusting unit 230 records the time stamp in a screen time stamp adjustment table 280.

In some embodiments, in step S730, the length and proportion of the time that some source stream content is presented at the time of last playback may also be adjusted through the screen timestamp adjustment unit 230.

Next, in step S735, the stream data output unit 240 adds an identification tag or a content list for identifying the content of each source stream in the composite condensed stream whose time stamp is completed, and transmits it to the client player 150. The identification tag or content list used is recorded in the identification tag or content list (step S775).

In addition, in step S740, the service quality controller 250 detects the statistical network status and the buffer overflow level of the client player 150 to control the proportion of the streaming content analysis unit 210 in the data enrichment in step S720. Most control The amount of data for the composite concentrate stream that is ultimately output.

Moreover, in step S745, the client request processor 260 in the condensed stream unit 100 receives the request of the client player 150 for streaming content of a certain source. When the client requires the processor 260 to receive a client request 170, it will query the identification tag or content list with the identification tag or content list indicated in the client request 170 (step S775) and the comparison stream input table. 270 (step S760), to obtain the data of the source stream content requested by the client player 150.

In some embodiments, the streaming content analysis unit 210 directly increases or decreases the weight of the content importance score indicator or satisfies the content analysis of the source stream content in step S720 according to the content requested by the client. A condition to temporarily or permanently change the proportion of the source stream content in the composite condensed stream.

In some other embodiments, the streaming content analysis unit 210 may also increase or decrease the weight of the importance score indicator used in the analysis of the specific event content in step S720 according to the requirements of the client, temporarily or permanently. Change the proportion of certain events in the composite condensed stream.

In some other embodiments, the client requires the processor 260 to identify the tag or content list according to the query (step S775) and the source content data obtained by comparing the stream input table 270 (step S760). In step S750, the stream is streamed. The data input unit 220 transmits a source query to the streaming server 110 to inquire whether the source player 150 can directly obtain the source stream content address and the acquisition mode, and receive the response sent by the stream server 110. message. The returned response message can be further replied to the receiving client player 150 through the process subsequent to step S745.

14A-14B are flowcharts showing a method for receiving, processing, and playing a composite condensed stream in the client player 150 according to an embodiment of the present disclosure, and Please refer to Figure 9.

First, in step S910, the streaming data receiving unit 510 in the client player 150 can directly receive the data of the source streaming content from the streaming server 110, and can also receive the composite concentrated stream from the concentrated streaming unit 100. .

In step S960, after the stream data received by the stream data receiving unit 510, it is first determined whether it is a composite stream. If it is not the composite condensed stream (NO in step S960), the video decoding unit 530 performs stream data decoding in step S915. Finally, in step S920, the image presentation unit 550 presents the material in a single window in an image presentation area.

If the stream data is a composite condensed stream (YES in step S960), in step S930, the composite condensed stream parsing unit 520 first uses the stream content of each source in the composite condensed stream and the accompanying The identification tag or content list for identification is removed and taken out.

After the extracted source stream content is disassembled, in step S965, the composite condensed stream parsing unit 520 determines whether the client player 150 can provide multiple sets of decoding. In some embodiments, if the client player 150 cannot provide multiple sets of decoders due to resource constraints, in step S935, a single image decoding unit 530 is used for data decoding. In other embodiments, if the client player 150 can provide multiple sets of decoders, in step S945, the composite condensed stream parsing unit 520 assigns each source stream content to the respective image decoding unit 530 for data. decoding.

Next, in step S970, the video decoding unit 530 determines whether the video presentation unit supports the video presentation display of the plurality of windows. In some embodiments, if the client player 150 cannot provide multiple split images due to resource limitations. The window is used to present the picture. In step S940, the content of each source stream is displayed in a single window of the image presentation unit in turn according to the time stamp sequence marked by the composite condensed stream. In other embodiments, if the client player 150 can provide a plurality of split windows to present the image frame, in step S950, the different source stream contents are respectively presented in the respective divided windows to display the image.

If the user of the user-side player 150 is interested in the content of a certain source or some sources, the identification tag or the content list existing in the source of interest may be associated (step S980), via step S955, The request for the source stream content is sent to the condensing stream unit 100.

For the requirements of the source content, in some embodiments, it may be desirable to increase or decrease (temporarily or permanently) the proportion of a source stream content in the composite condensed stream.

In other embodiments, it may be desirable to increase or decrease (temporarily or permanently) the proportion of certain events in the composite concentrated stream.

In the manner of these embodiments, the time and resources spent re-establishing a new connection session can be reduced.

In still other embodiments, the client player 150 may directly transmit a source query 175 requesting the condensed stream unit 100 to query whether a source stream content may be obtained with respect to some original source. If the condensed stream unit 100 can obtain content about a certain original source stream, in step S955, the method of obtaining and connecting the original stream content for a certain source may be received (for example, but not limited to For example, a resource identifier (RTSP URI) like an instant stream protocol. In step S910, the connection and data reception of the original stream content are performed in the client player 150. In such an embodiment, the client player 150 can be modified without concentration. The source stream content to maintain all the complete source material.

It should be noted that although the aforementioned streaming server, condensing stream unit and client player are the separate components of the system, these components can be integrated together, thereby reducing the number of components in the system. Similarly, one or more components can be used separately, thereby increasing the number of components within the system.

Therefore, the method and system for generating a composite condensed stream by the present disclosure can stream content from multiple sources, dynamically condense and mix the content data into a single session according to content events or user interests. Transfer. In the process of transmission, the degree and concentration of the concentration are also changed according to the capabilities of the receiving device and the network condition to meet the different needs of each receiving end. After receiving the concentrated stream, the client player can have different presentation modes and streaming content adjustment methods according to its capabilities. In addition, with the system and method of the present disclosure, many advantages can be achieved: (1) shortening the waiting time for the user to simultaneously view multiple streams of content and providing a global channel focus preview. (2) The server automatically and dynamically adjusts the content of the concentrated stream according to the capabilities of the receiving device and the network condition, thereby improving the user's viewing perception. (3) The user can simultaneously watch a plurality of contents with less bandwidth, and provide an appropriate presentation manner according to the received concentrated stream. In combination with the above advantages, the present disclosure can be applied to a network streaming application to provide better viewing performance.

The present disclosure has been disclosed in the above preferred embodiments, and is not intended to limit the disclosure. Any one skilled in the art can make some changes and refinements without departing from the spirit and scope of the disclosure. The scope of protection is subject to the definition of the scope of the patent application.

150‧‧‧User Player

165‧‧‧Composite concentrated stream

195‧‧‧Source streaming content

510‧‧‧Streaming data receiving unit

520‧‧‧Composite Concentrated Stream Analysis Unit

530‧‧‧Image decoding unit

540‧‧‧Content Request Processing Unit

550‧‧‧Single area image presentation unit

555‧‧‧Multi-section image presentation unit

Claims (12)

A player, coupled in a user end, coupled to a streaming server and a concentrating stream unit, comprising: a stream data receiving unit, configured to receive a composite condensed string transmitted by the concentrating stream unit a composite condensed stream analyzing unit coupled to the stream receiver for receiving the composite condensed stream transmitted by the stream data receiving unit, and parsing into the complex condensed stream Source stream content; at least one image decoding unit, configured to receive the complex source streams parsed by the composite condensed stream parsing unit, and decode the plurality of source streams into a plurality of images; and an image rendering A unit for presenting the plurality of images. The player of claim 1, further comprising: a content request processing unit, configured to obtain an identification tag or a content list included in the composite concentrated stream, and receive a user input request by the user And transmitting the client request to the concentrating stream unit. The player of claim 1, wherein the image presentation unit sequentially displays the plurality of images in a single window according to the time stamp adjusted by the composite condensed stream. The player of claim 1, wherein the image presentation unit adjusts the time stamp according to the composite condensed stream to display a plurality of images simultaneously in the plurality of divided windows. The player of claim 1, wherein the stream receiver further receives the source stream transmitted by the stream server in a stream form, the client player further comprising: an image decoding And a unit for decoding the source stream content from the stream server; a single area screen image presenting unit for displaying the plurality of images of the source stream content. The player of claim 1, wherein the client player is a mobile device. A method for playing a composite condensed stream, which is used in a user-side player, wherein the user-side player is coupled to a condensed stream unit and a stream server, comprising: receiving the condensed stream unit and transmitting the same a composite condensed stream; the composite condensed stream is parsed into the plurality of source streams; the plurality of source streams are decoded into a plurality of images; and the plurality of images are rendered. The method for playing a composite condensed stream as described in claim 7 further includes: obtaining an identification tag or a content list included in the composite condensed stream; receiving a content source request input by a user; and transmitting The client requests the centralized stream unit. The method for playing a composite concentrated stream as described in claim 7 of the patent scope, The plurality of images are presented in a single window in sequence according to the time stamp adjusted by the composite condensed stream. The method for playing a composite condensed stream as described in claim 7 , wherein the plurality of images are presented in accordance with the time stamp adjusted by the composite condensed stream, and the plurality of divided windows are simultaneously presented. The method for playing a composite condensed stream as described in claim 7, wherein if the stream server transmits a source stream in a stream form, the method further comprises: decoding from the stream server The source stream content; and the plurality of images that display the source stream content. The method for playing a composite concentrated stream as described in claim 7, wherein the client player is a mobile device.