EP2770743B1 - Methods and systems for processing content - Google Patents
Methods and systems for processing content Download PDFInfo
- Publication number
- EP2770743B1 EP2770743B1 EP14155941.9A EP14155941A EP2770743B1 EP 2770743 B1 EP2770743 B1 EP 2770743B1 EP 14155941 A EP14155941 A EP 14155941A EP 2770743 B1 EP2770743 B1 EP 2770743B1
- Authority
- EP
- European Patent Office
- Prior art keywords
- data
- format
- content
- video
- content data
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
- H04N19/46—Embedding additional information in the video signal during the compression process
- H04N19/463—Embedding additional information in the video signal during the compression process by compressing encoding parameters before transmission
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
- H04N21/43—Processing of content or additional data, e.g. demultiplexing additional data from a digital video stream; Elementary client operations, e.g. monitoring of home network or synchronising decoder's clock; Client middleware
- H04N21/431—Generation of visual interfaces for content selection or interaction; Content or additional data rendering
- H04N21/4312—Generation of visual interfaces for content selection or interaction; Content or additional data rendering involving specific graphical features, e.g. screen layout, special fonts or colors, blinking icons, highlights or animations
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
- H04N21/43—Processing of content or additional data, e.g. demultiplexing additional data from a digital video stream; Elementary client operations, e.g. monitoring of home network or synchronising decoder's clock; Client middleware
- H04N21/434—Disassembling of a multiplex stream, e.g. demultiplexing audio and video streams, extraction of additional data from a video stream; Remultiplexing of multiplex streams; Extraction or processing of SI; Disassembling of packetised elementary stream
- H04N21/4344—Remultiplexing of multiplex streams, e.g. by modifying time stamps or remapping the packet identifiers
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
- H04N21/43—Processing of content or additional data, e.g. demultiplexing additional data from a digital video stream; Elementary client operations, e.g. monitoring of home network or synchronising decoder's clock; Client middleware
- H04N21/434—Disassembling of a multiplex stream, e.g. demultiplexing audio and video streams, extraction of additional data from a video stream; Remultiplexing of multiplex streams; Extraction or processing of SI; Disassembling of packetised elementary stream
- H04N21/4347—Demultiplexing of several video streams
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
- H04N21/43—Processing of content or additional data, e.g. demultiplexing additional data from a digital video stream; Elementary client operations, e.g. monitoring of home network or synchronising decoder's clock; Client middleware
- H04N21/44—Processing of video elementary streams, e.g. splicing a video clip retrieved from local storage with an incoming video stream, rendering scenes according to MPEG-4 scene graphs
- H04N21/4402—Processing of video elementary streams, e.g. splicing a video clip retrieved from local storage with an incoming video stream, rendering scenes according to MPEG-4 scene graphs involving reformatting operations of video signals for household redistribution, storage or real-time display
- H04N21/440263—Processing of video elementary streams, e.g. splicing a video clip retrieved from local storage with an incoming video stream, rendering scenes according to MPEG-4 scene graphs involving reformatting operations of video signals for household redistribution, storage or real-time display by altering the spatial resolution, e.g. for displaying on a connected PDA
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N7/00—Television systems
- H04N7/01—Conversion of standards, e.g. involving analogue television standards or digital television standards processed at pixel level
Definitions
- Data can be communicated in various forms over various communication mediums.
- Data such as video
- Data can often be modified from a first format to include supplemental content such as overlays, icons, menus, and the like.
- supplemental content such as overlays, icons, menus, and the like.
- US 2012/0/82227 discloses a device and method to insert crawling text into a video bit stream.
- US 2009/232221 discloses a method and an apparatus to combine an overlayed icon with a compressed video, in which the macroblocks of the video that will be overlayed, are first decoded, then combined with the overlayed content, and finally re-encoded.
- An encoded video may be completely decoded to a spatial domain; modifications, such as adding supplemental content, are implemented in the spatial domain; and the video is re-encoded for distribution to an end-user.
- the process of completely decoding the video in order to implement changes and/or add content for each end-user or device is both processor and time intensive.
- methods can comprise receiving content data in a first format and receiving context description data, which can be any type of data related to at least a portion of the content data.
- the content data can be converted to an intermediate format.
- the conversion of the content data to the intermediate format can be based at least in part on the context description data.
- the portion of the content data relating to the context description data can be converted to the intermediate format.
- the content data in the intermediate format can be modified, for example, to include supplemental data.
- the portion of the content data in the intermediate format can be converted to a second format.
- the conversion of the content data to the second format can be based at least in part on the context description data.
- the content data in the second format can be rendered via a downstream device such as a user device.
- methods can comprise converting content data having a first format into one or more of content data having an intermediate format and context description data relating to at least a portion of the content data.
- the portion of the content data based upon the context description data can be processed, e.g., encoded.
- the portion of the content data based upon the context description data can be encoded for distribution to one or more downstream devices.
- the encoded portion of the content data can be rendered via a downstream device such as a user device.
- context description data can be determined.
- the context description data can relate to a content data having a first format.
- the content data can be converted from a first format into an intermediate format.
- the context description data and converted content data can be transmitted to an encoder.
- the encoder can process a requisite portion of the converted content data based upon the context description data.
- the methods and systems may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects.
- the methods and systems may take the form of a computer program product on a computer-readable storage medium having computer-readable program instructions (e.g., computer software) embodied in the storage medium.
- the present methods and systems may take the form of web-implemented computer software. Any suitable computer-readable storage medium may be utilized including hard disks, CD-ROMs, optical storage devices, or magnetic storage devices.
- These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including computer-readable instructions for implementing the function specified in the flowchart block or blocks.
- the computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process such that the instructions that execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart block or blocks.
- blocks of the block diagrams and flowchart illustrations support combinations of means for performing the specified functions, combinations of steps for performing the specified functions and program instruction means for performing the specified functions. It will also be understood that each block of the block diagrams and flowchart illustrations, and combinations of blocks in the block diagrams and flowchart illustrations, can be implemented by special purpose hardware-based computer systems that perform the specified functions or steps, or combinations of special purpose hardware and computer instructions.
- a data stream transmitted to the user location can comprise a repeatedly embedded placement signal to indicate that a particular content should be transmitted to the user location.
- FIG. 1 illustrates various aspects of an exemplary network environment in which the present methods and systems can operate. Some aspects of the present disclosure relate to methods and systems for content control. Those skilled in the art will appreciate that present methods may be used in systems that employ both digital and analog equipment. One skilled in the art will appreciate that provided herein is a functional description and that the respective functions can be performed by software, hardware, or a combination of software and hardware.
- the network 100 can comprise a central location 101 (e.g., a control or processing facility in a fiber optic network, wireless network or satellite network, a hybrid-fiber coaxial (HFC) content distribution center, a processing center, headend, etc.), which can receive content (e.g., data, input programming) from multiple sources.
- the central location 101 can combine the content from the various sources and can distribute the content to user locations, such as location 119, via distribution system 116.
- the central location 101 can receive content from a variety of sources 102a, 102b, 102c.
- the content can be transmitted from the source to the central location 101 via a variety of transmission paths, including wireless (e.g. satellite paths 103a, 103b) and terrestrial path 104.
- the central location 101 can also receive content from a direct feed source 106 via a direct line 105.
- Content may also be created at the central location 101.
- Other input sources can comprise capture devices such as a video camera 109 or a server 110.
- the signals provided by the content sources can comprise a single content item or a multiplex that comprises several content items.
- the central location 101 can comprise one or a plurality of receivers 111a, 111b, 111c, 111d that are each associated with an input source.
- MPEG encoders such as encoder 112 are included for encoding/transcoding local content or a video camera 109 feed, terrestrial receiver, or broadcast receiver.
- a data packaging device such as a fragmentor, can be integrated with the encoder 112 (or separate and in communication with the encoder 112) to package the encoded/transcoded content.
- the encoder 112 and/or fragmentor can repeatedly embed markers, flags, and signals into the content data stream for processing by downstream devices.
- a switch 113 or a network can provide access to server 110, which can be, for example, a pay-per-view server, a data server, an internet router, a network system, and/or a phone system. Some signals may require additional processing, such as signal multiplexing or transcoding, prior to being modulated. Such multiplexing can be performed by multiplexer (mux) 114.
- multiplexer multiplexer
- the central location 101 can comprise one or a plurality of modulators, 115a, 115b, 115c, and 115d, for interfacing to the distribution system 116.
- the modulators can convert the received content into a modulated output signal suitable for transmission over the distribution system 116.
- the output signals from the modulators can be combined, using equipment such as a combiner 117, for input into the distribution system 116.
- a control system 118 can permit a system operator to control and monitor the functions and performance of network 100.
- the control system 118 can interface, monitor, and/or control a variety of functions, including, but not limited to, the channel lineup for the television system, billing for each user, and/or conditional access for content distributed to users.
- Control system 118 can provide input to the modulators for setting operating parameters, such as system specific MPEG table packet organization or conditional access information.
- the control system 118 can be located at central location 101 or at a remote location.
- the distribution system 116 can distribute signals from the central location 101 to user locations, such as user location 119.
- the distribution system 116 can be an optical fiber network, a coaxial cable network, a hybrid fiber-coaxial network, a wireless network, a satellite system, a direct broadcast system, or any combination thereof.
- a decoder 120 such as a gateway or communications terminal (CT) can decode, if needed, the signals for display on a display device, such as on a display device 121 such as a television set (TV), a mobile device, or a computer monitor.
- a decoder 120 such as a gateway or communications terminal (CT) can decode, if needed, the signals for display on a display device, such as on a display device 121 such as a television set (TV), a mobile device, or a computer monitor.
- TV television set
- mobile device or a computer monitor.
- the signal can be decoded in a variety of equipment, including an CT, a computer, a TV, a monitor, or satellite dish.
- the methods and systems disclosed can be located within, or performed on, one or more CT's, display devices 121, central locations 101, DVR's, home theater PC's, and the like.
- the decoder 120 can receive and process the embedded markers, flags, and signals in the content data stream for controlling a content received by the decoder 120.
- the decoder 120 or communication terminal can comprise a buffer 122.
- the buffer 122 can comprise a memory or a storage device.
- a number of the content fragments can be loaded into the buffer 122 and stored for playback or processing by the associated decoder 120.
- other data, content, and/or information can be stored in the buffer 122 such as advertisements and alternative content, for example.
- the buffer 122 can be separate from the decoder 120.
- the buffer 122 can be located upstream of the decoder 120 and downstream of the distribution system 116.
- the buffer 122 can store information to be transmitted to any number of decoders 120 or other recipient devices.
- the decoder 120 can retrieve the content fragments from the buffer 122 for playback without having to communicate with the distribution system 116, thereby reducing latency in playback.
- the decoder 120 can process a continuous streaming track, which can comprise audio, video, or other data from the distribution system 116 with a regular cadence (e.g., every two seconds, four seconds, ten seconds). Other data and tracks can be received and processed by the user devices.
- user location 119 is not fixed.
- a user can receive content from the distribution system 116 on a mobile device such as a laptop computer, PDA, smartphone, GPS, vehicle entertainment system, and/or portable media player.
- a mobile device such as a laptop computer, PDA, smartphone, GPS, vehicle entertainment system, and/or portable media player.
- the network of FIG. 1 is illustrative only and other types of networks and configurations may be used to implement the disclosure.
- one or more of a layout element 124 and an encoder 126 can be disposed upstream of the decoder 120.
- one or more of the layout element 124 and the encoder 126 can be configured to receive data, such as video, audio, images, overlays, menus, icons, and the like.
- one or more of the layout element 124 and the encoder 126 can receive data from a data source such as a content provider, the central location 101, and/or the distribution system 116.
- the layout element 124 can be configured to receive data such as content data (e.g., images, audio, video, menus, icons, banners, overlays, etc.).
- the layout element 124 can receive data in a first format, such as hypertext markup language (HTML).
- HTML hypertext markup language
- Other formats can be used such as Scalable Vector Graphics (SVG), Extended Markup Language (XML) with associated XML Style Sheets (XSS), Extended Post Script (EPS), Portable Document Format (PDF), proprietary document formats such as various presentation and document formats that can describe the layout of content, without having to combine the various content into pictures or video.
- the data received by the layout element 124 can represent images, video, and/or frames of video, as illustrated in FIG. 2A .
- the first format of the data can comprise positional and presentation information for rendering an image, such as:
- the layout element 124 can be configured to determine a context description data relating to the data having the first format.
- the context description data can comprise information relating to a layout of data points (e.g., images, video frames, icons, motion, etc.) represented by the data received by the layout element 124.
- FIG. 2B illustrates a layout of a plurality of content components 200a-k, which can be represented by the context description data.
- the context description data can comprise information relating to component list, location information, styling information, movement information, vector operations, vector graphics, excluded regions, relative encoding sensitivity of different regions, an identifier (e.g., file locator, buffer pointer, etc.), a type or classifier of data, a time element, a quality of encoding and/or presentation desired, a size, and/or a position.
- identifier e.g., file locator, buffer pointer, etc.
- one or more of the content components 200a-k can represent a region of rendered content.
- the context description data associated with one or more of the content components 200a-k can represent a type, quality, contextual description, and the like.
- the type can comprise one or more of a static image, text, video, animation, and the like.
- the quality can comprise classifiers such as soft image, crisp image, video, text, and the like.
- the description can comprise an identifier or pointer to a memory or buffer, a rendered position, a rendered size, and a time the encoder can expect the associated content components 200a-k to change.
- the context description data can be processed to determine the level of processing required by the encoder 126.
- the encoder 126 may partially decode the content data, apply changes to the decoded content data, and re-encode the content data (with the changes) for downstream distribution.
- the encoder 126 can be configured to receive one or more of the data (e.g., content data) and the context description data. As an example, the encoder 126 can process only a requisite portion of the content data based upon the context description data. As another example, the encoder 126 can selectively encode one or more content components 200a-k represented by the content data based upon the context description data. As a further example, for one of the content components 200a-k that is an image and does not change between several frames of a video, the encoder 126 can retrieve the content data from a buffer 128 rather than having to re-encode each of the content components 200a-k for every frame.
- the data e.g., content data
- the context description data e.g., the encoder 126 can process only a requisite portion of the content data based upon the context description data.
- the encoder 126 can selectively encode one or more content components 200a-k represented by the content data based upon the context description data.
- the encoder 126
- the context description data can identify which of the content components 200a-k that require further processing by the encoder 126 and which of the content components 200a-k that can be retrieved from the buffer 128.
- the context description data can define processing instructions for the encoder 126 for various formats of data.
- the encoder 126 can process the content data or other data into various formats such as MPEG-2, MPEG-4, H.264, Quicktime®, Adobe Flash®, Motion JPEG, and the like.
- the methods and systems can utilize digital audio/video compression such as MPEG, or any other type of compression.
- MPEG Moving Pictures Experts Group
- ISO International Standards Organization
- the MPEG experts created the MPEG-1 and MPEG-2 standards.
- the combined MPEG-1, MPEG-2, MPEG-4, and subsequent MPEG standards are hereinafter referred to as MPEG.
- MPEG encoded transmission content and other data are transmitted in packets, which collectively make up a transport stream.
- the present methods and systems can employ transmission of MPEG packets.
- the present methods and systems are not so limited, and can be implemented using other types of transmission, and methods of storage.
- the output of a single MPEG audio and/or video coder may be a transport stream, comprised of one or more elementary streams.
- An elementary stream is an endless near real-time signal.
- the elementary stream may be broken into data blocks of manageable size, forming a packetized elementary stream (PES). These data blocks need header information to identify the start of the packets and must include time stamps because packetizing disrupts the time axis.
- PES packetized elementary stream
- a multi-program transport stream has a program clock reference (PCR) mechanism that allows transmission of multiple clocks, one of which is selected and regenerated at the decoder.
- PCR program clock reference
- a multi-program transport stream is more than just a multiplex of data, audio and/or video PESs.
- a transport stream may include metadata describing the bit stream. This includes the program association table (PAT) that lists every program in the multi-program transport stream. Each entry in the PAT points to a program map table (PMT) that lists the elementary streams making up each program.
- PAT program association table
- PMT program map table
- Some programs will be unencrypted, but some programs may be subject to conditional access (encryption), and this information is also carried in the metadata.
- the transport stream can be comprised of fixed-size data packets; for example, each containing 188 bytes. Each packet can carry a program identifier code (PID).
- PID program identifier code
- Packets in the same elementary stream can all have the same PID, so that the decoder (or a demultiplexer) can select the elementary stream(s) it wants and reject the remainder. Packet continuity counts ensure that every packet that is needed to decode a stream is received. A synchronization system can be used so that decoders can correctly identify the beginning of each packet and deserialize the bit stream into words.
- a content item such as a program
- a multi-program transport stream used in digital television might contain three programs, to represent three television channels.
- each channel consists of one video stream, one or two audio streams, and any necessary metadata.
- a receiver wishing to tune to a particular "channel" merely has to decode the payload of the PIDs associated with its program. It can discard the contents of all other PIDs.
- the decoding of the payload can comprise demultiplexing and/or dequanitizing.
- demultiplexing can comprise seperating the input SPTS into packets of video and audio.
- ISO 13818-1 describes a method of multiplexing multiple elementary streams into a program stream or transport stream.
- the intermediate format can remove all packet encapsulation and multiplexing created according to 131818-1.
- the MPEG-2 TS packets corresponding to Packet ID (PID) of video as described in the PMT of the component can be separated.
- each frame of video or field of video e.g., pictures
- the elementary stream packetization can be de-packetized to the extent that each frame of video or field of video is a contiguous stream of bytes.
- audio can be stored as one or more of a packetized stream or as contiguous stream of bytes of audio frames.
- a single content component comprises only audio
- the audio can remain as a stream of MPEG-2 Transport Stream packets. Accordingly, only the Packet Identifier (PID) on the transport stream may be required to be changed during a re-multiplexing process.
- PID Packet Identifier
- FIG. 3 is a block diagram of an exemplary content control system 300.
- the system 300 can comprise one or more data sources 302 for accessing and/or transmitting data, such as a data stream.
- An encoder 304 e.g., encoder 126) can be in signal communication with the data sources 302.
- a fragmentor 306 can be in signal communication with the encoder 304.
- the encoder 304 and the fragmentor 306 can be integrated as part of a unitary device. However, the encoder 304 and the fragmentor 306 can be separate and distinct components.
- the data sources 302 can comprise a content provider for providing one or more of audio content, video content, data, news feeds, sports programming, and advertisements.
- the data sources 302 can be configured to transmit the data (e.g., as a stream, fragments, files, overlays, etc.) to various end-users.
- one or more data source 302 can comprise an advertisement database having a plurality of advertisements stored therein or capable of accessing advertisement stored elsewhere.
- one or more of the data sources 302 can comprise a network data feed transmitting the data stream to users such as subscribers or clients.
- the one or more data sources 302 can be configured to transmit the data stream in a standard video format, such as, but not limited to, any of the Moving Picture Experts Group standards (e.g., MPEG-2, MPEG-4, a single MPEG-4 video encapsulated in an MPEG-2 transport stream over UDP MCAST, etc.), or any of the standards for audio and/or video, such as MP3, Quicktime, and Audio Video Interleave (avi).
- the encoder 304 can be configured to receive the data stream from any source having any format that can be encoded (or transcoded) into a format that is appropriate for streaming or otherwise transmitting.
- the encoder 304 can be any device, system, apparatus, or the like to encode, decode, and/or transcode the data stream.
- the encoder 304 converts a single (high) bit rate to an output of multiple (lower) bitrates.
- the encoder 304 can convert the data stream from the input format received from the data source (such as MPEG-2, QAM, etc.) to an output format for distribution to consumers (such as MPEG-4).
- the encoder 304 can be a device, such as a transcoder, that conditions streaming data and/or changes it from one format to another.
- the encoder 304 can be configured to receive the data stream, or other type of signal, from one or more of the data source 302 and encode/transcode information onto the data stream. In an exemplary embodiment, the encoder 304 adds information to the signal relating to content fragments 308 or data blocks.
- a buffer or storage medium 309 can be in communication with the encoder 304 or integrated therewith. As an example, data received by the encoder 304 can be buffered or stored in the storage medium 309. As a further example, data blocks or content fragments 308 processed by the encoder 304 can be buffered or stored.
- the fragmentor 306 can be in signal communication with the encoder 304 to receive the data stream therefrom.
- the fragmentor 306 and the encoder 304 are configured as a unitary device.
- the fragmentor 306 and the encoder 304 can be integrated or separate.
- fragmentor 306 conditions the data stream (or other data signal) for downstream distribution by a computing device 310, such as a server, through a content distribution network 312 to user devices 314 (e.g. the decoder 120).
- the computing device 310 can be an origin Hypertext Transfer Protocol (HTTP) server.
- HTTP Hypertext Transfer Protocol
- other servers can be used, such as a rate adaptive streaming server.
- the fragmentor 306 can communicate with the computing device 310 using the POST method of the HTTP.
- the computing device 310 can implement the functions described relative to the encoder 304 and/or the fragmentor. However, other protocols, configurations, and communication methods can be used.
- the fragmentor 306 separates or fragments the data stream into each of the content fragments 308 represented by the data stream based upon an information encoded onto the data stream by the encoder 304.
- the fragmentor 306 accesses the information encoded/inserted in the data stream by the encoder 304 to define the content fragments 308 based on, among other things, boundaries, grouping, timing, or duration from the encoded data stream.
- the content fragments 308 can be transmitted to the content distribution network (CDN) 312 for delivery to the user devices 314 for consumption (e.g., playback).
- CDN 312 content distribution network
- the user devices 314 can intercommunicate using the GET method of HTTP.
- other protocols and communication methods can be used.
- one or more of the user devices 314 can comprise a buffer 316.
- the buffer 316 can comprise a memory or a storage device.
- a number of the content fragments 308 can be loaded into the buffer 316 and stored for playback by the associated user device 314. Accordingly, the user device 314 can retrieve the content fragments 308 from the buffer 316 for playback without having to communicate with the CDN 312, thereby reducing latency in playback.
- the user device 314 can process a continuous streaming track, which can comprise audio, video, or other data from the computing device 310 with a regular cadence (e.g., every two seconds, four seconds, ten seconds). Other data, content, and/or information can be stored in the buffer 316 such as advertisements and alternative content, for example.
- a buffer 316' can be separate from the user device 314.
- the buffer 316' can be located upstream of the user device 314 and downstream of the CDN 312.
- the buffer 316' can store information to be transmitted to any number of user devices 314 or other recipient devices. Accordingly, the user device 314 can retrieve the content fragments 308 from the buffer 316' for playback without having to communicate with the CDN 312, thereby reducing latency in playback.
- the layout element 124 can be configured to receive data such as content data (e.g., images, audio, video, menus, icons, banners, overlays, etc.).
- the layout element 124 can receive content data from a plurality of data sources 302.
- the layout element 124 can be configured to process the content data received by one or more data sources 302 to determine context description data associated with the content data.
- the context description data can be processed to determine the level of processing required by the encoder 304.
- the encoder 304 may partially decode the content data, apply changes to the decoded content data, and re-encode the content data (with the changes) for downstream distribution.
- FIG. 4 illustrates an exemplary system and data flow in which the present methods and systems can operate.
- the process of modifying video data can be optimized by observing that the majority of the output is similar to the input. Accordingly, an encoder can decode only a portion of the video, thereby generating an intermediate format or "mezzanine format.” The intermediate format can then be altered and the output format can be generated from the altered intermediate format.
- the intermediate format can be implemented to minimize the amount of encoding required in a given process.
- the intermediate format can be used to overlay an icon or overlap menu bars or tickers on the video rendered from the video data.
- a traditional way of modifying an encoded video is to first decode the video data completely to the spatial domain, then apply the modifications (e.g., adding an icon) in the spatial domain, and then re-encode the complete video data for distribution.
- macro-blocks in an input video data can be selectively modified to render an overlayed icon on a portion of the rendered video.
- the altered video comprising the overlayed icon can be combined with text and images in a subsequent encoding process, as illustrated in FIG. 4 .
- a layout engine 400 (e.g., layout element 124) can be configured to receive data such as content data 402a, 402b (e.g., images, audio, video, menus, icons, banners, overlays, etc.) from one or more sources.
- the layout engine 400 can be configured to process the received data to determine context description data associated with the received data.
- the context description data can be processed to determine the level of processing (e.g., extent of decoding of a compressed video stream 404).
- a decoder 406 can partially decode an input, such as video 404, based on the context description data to generate a partially decoded video 404'.
- the partially decoded video 404' can comprise the intermediate format.
- an encoder 408 can apply changes to the partially decoded video 404' and can re-encode the partially decoded video 404' (with the content data 402b) for downstream distribution.
- content data 402b such as an overlay
- the modified video 404" can be combined with other content data 402a to result in an output format 416 for presentation.
- content data 402a such as images and text, can be encoded separately from the video 404 and combined with the modified video 404".
- one or more frames of content data 402a can be processed (e.g., rendered, encoded, arranged) via a renderer 410, an encoder 412, or a combination thereof to result in an intermediate format of encoded content data 402a' for combination with an intermediate format of the modified video 404".
- a computing device 414 such as a data combiner, encoder, a multiplexer, or the like, can receive the intermediate format of encoded content data 402a' and the intermediate format of the modified video 404" for combination and encoding as the output format 416 for presentation and/or distribution.
- the decoding of the video can comprise demultiplexing and/or dequanitizing.
- demultiplexing can comprise separating the input SPTS into packets of video and audio.
- ISO 13818-1 hereby incorporate herein by reference in its entirety, describes a method of multiplexing multiple elementary streams into a program stream or transport stream.
- the intermediate format can remove all packet encapsulation and multiplexing created according to 131818-1.
- the MPEG-2 TS packets corresponding to Packet ID (PID) of video as described in the PMT of the component can be separated.
- each frame of video or field of video (e.g., pictures) can be stored as a contiguous stream of bytes.
- the elementary stream packetization can be de-packetized to the extent that each frame of video or field of video is a contiguous stream of bytes.
- audio can be stored as one or more of a packetized stream or as contiguous stream of bytes of audio frames.
- a single content component comprises only audio
- the audio can remain as a stream of MPEG-2 Transport Stream packets. Accordingly, only the Packet Identifier (PID) on the transport stream may be required to be changed during a re-multiplexing process.
- PID Packet Identifier
- the decoding of the video data can comprise processing a group of pictures (GOP) structure of the video.
- I-frames can be decoded only to the extent that the macro-blocks covered by a modification to the I-frames can be extracted.
- motion vectors in P-frames and B-frames can be decoded completely, to identify all macro-blocks or regions that will be affected due to addition of icons or partial overlay of elements.
- the motion vectors of a macro-block may be restricted to refer only to other macro-blocks that are within a certain distance from said macro-block.
- the GOP structure can be processed to determine which macro-blocks refer to the portion of the video data that will be modified, for example, by supplemental data such as context description data.
- the DCT coefficients of the "affected" macro-blocks can be be decoded.
- other macro-blocks in the same slice as the affected macro-blocks can be decoded.
- the affected macro-blocks in the P and B pictures can be re-encoded.
- re-encoding can comprise one or more of re-encoding the macro-blocks as intra-only macro-blocks, retaining the motion vectors and re-computing the residual coefficients, and performing a new motion vector search to create both new motion vectors and new DCT coefficients.
- the quantization scale factor used for the first macro-block and the last macro-block of every slice of the video data can be stored.
- the quantization scale factor can be relied upon to re-encode macro-blocks of two adjacent components in separate slices or the same slice.
- static components such as pictures and text that do not overlay other frequently changing components such as videos
- the mezzanine format for static components can be MPEG-2 transport streams.
- the mezzanine format for static components can have a different stream of MPEG-2 packets for each of the I, P and B picture types.
- the mezzanine format can contain the same MPEG-2 transport stream packets for all frames of a certain type.
- the ticker can be treated as an overlay icon.
- macro-blocks in the video can be recomputed.
- the mezzanine format can contain resampled blocks.
- resampling blocks can comprise converting all macro-blocks to the highest sampling. For example, if two videos, one that is 4:2:0 is to be combined with another video that is 4:4:4, then all macro-blocks of the 4:2:0 video can be resampled to 4:4:4 before being stored in the mezzanine format.
- a re-combiner can convert all 4:2:0 blocks to 4:4:4 blocks in real-time.
- videos with different GOP structures can be processed by storing co-efficient data and motion-vector data in their fully decoded (but not inverted) forms in the mezzanine format.
- a mezzanine combiner or encoder can convert I Frames into P or B frames and P frames into B frames.
- frames can be converted by adding null backward and/or forward motion vectors into every macro-block of the frames and partially re-encoding the macro-blocks.
- partial re-encoding can comprise encoding coefficients and motion vectors using variable length coding and creating slice and picture structures from the macro-blocks.
- FIG. 5 illustrates an exemplary method for managing, presenting, and/or controlling content.
- data such as content data can be accessed or otherwise received in a first format.
- the first format can be HTML, JPEG, MPEG, or the like.
- the content data can comprise one or more of audio, text, images, vectors, highlights, animations, video, and overlays.
- context description data e.g., metadata
- the context description data which can be any type of data and can relate to at least a portion of the content data.
- the context description data can comprise one or more of a component list, location information, movement information, vector operations, vector graphics, included/excluded regions, and relative encoding sensitivity of different regions.
- vector graphics can comprise, for example, one or more of scalable vector graphics (SVG) included as part of the HTML, SVG as a separate stream or file in addition to the base HTML, vector graphics supplied through an API, such as OpenGL and the like.
- SVG scalable vector graphics
- the content data or a portion thereof can be converted to an intermediate format, such as a mezzanine format.
- converting to the intermediate format can comprise at least a partial decoding of the first format and/or fragmenting or otherwise partitioning of the content data.
- converting to the intermediate format can comprise modifying the decoded portion of the content data.
- the intermediate format can comprise partially encoded data which may be the results of partially decoding input MPEG-2 content.
- step 508 all or a portion of the content data can be modified.
- the portion of the content data that is converted (e.g., decoded) in the intermediate format can be modified.
- supplemental content can be added to the content data.
- step 510 all or a portion of the content data can be converted from the intermediate format to a second format.
- the second format can be an MPEG format or a format suitable for IP delivery to a user device, etc..
- converting from the intermediate format to a second format can comprise encoding at least a portion of the content data.
- the content data having the second format can be rendered.
- a decoder can receive the content data in the second format and can process the same for presenting content via one or more devices.
- FIG. 6 illustrates one example of a method for converting data to an intermediate format.
- the steps illustrated in FIG. 6 can be performed alone, or in combination with one or more other steps.
- converting data to an intermediate format can comprise partial decoding of the content data.
- partial decoding can comprise removal of any encapsulation such as de-packetizing a transport stream into elementary video and audio streams.
- step 604 converting data to an intermediate format can comprise removing a variable length encoding (e.g., code) of the content data to make the encoding a fixed length.
- a variable length encoding e.g., code
- converting data to an intermediate format can comprise processing transforms.
- converting data to an intermediate format can comprise separating discrete cosine transforms (DCT) into an array and "exploding" the DCT.
- DCT discrete cosine transforms
- converting data to an intermediate format can comprise conversion of macro-blocks from intra, intra P, P, or B type frames to B type frames or blocks.
- converting data to an intermediate format can comprise de-quantizing at least a portion of the content data.
- quantization such as value mapping and/or vector mapping used in content encoding and clustering can be different between multiple screen elements that have to be combined.
- the input content description data can provide information relating to the extent to which each part of the content must be decoded (e.g., dequantized). For example, icons can be encoded heavily (poor definition), while text regions have to be encoded without much loss of definition.
- the context description data can define the expected change of the components over time. For example, "Image @ (100,100, 200, 200) is static till changed" can define a static image with no expectation of change for several frames, GOP or seconds.
- FIG. 7 illustrates an exemplary method for encoding data.
- data having a first format can be converted into content data having an intermediate format and context description data relating to a portion of the content data.
- the data in the first format can be processed to extract the context description data.
- the portion of the content data based upon the context description data can be encoded.
- the context description data can provide information relating to the extent the content data should be encoded.
- the content data in the intermediate format can comprise a portion that is decoded and/or manipulated. As such, the portion of the content data that is decoded and/or manipulated can be re-encoded for distribution.
- encoding the content data can comprise repackaging composited video into a MPEG-2 Transport Stream and multiplexing the video with audio streams.
- the content component can be copied to the output.
- the content component can be copied with one or more of the following transformations:
- regions of the content data can comprise a single content component having portions of the content component covered by other content components (e.g., a cutout background picture with a video overlay).
- encoding content data can comprise changing the quantization of macro-blocks.
- macro-blocks can be re-encoded by one or more of the following processes:
- the DCT can be inverse transformed to the original pixels.
- the original pixels can be combined and transformed back into DCT.
- motion vectors can be determined to reduce the size of the macro-block.
- a macro-block may simply be motion vectors that refer to a macro block from a previous picture.
- a p-only macro-block with motion vectors of magnitude zero can be transmitted.
- the re-encoded content data can be transmitted to a device such as a converter, decoder, terminal, or user device.
- the re-encoded data can be decoded and rendered, as shown in step 708.
- FIG. 8 illustrates an exemplary method for managing, presenting, and/or otherwise controlling content.
- context description data can be determined.
- the context description data can relate to content data having a first format.
- content data having the first format can be converted into an intermediate format.
- the context description data and converted content data can be transmitted to a network device, such as an encoder.
- the encoder can process a requisite portion of the converted content data based upon the context description data.
- FIG. 9 is a block diagram illustrating an exemplary operating environment for performing the disclosed methods.
- One skilled in the art will appreciate that provided herein is a functional description and that the respective functions can be performed by software, hardware, or a combination of software and hardware.
- This exemplary operating environment is only an example of an operating environment and is not intended to suggest any limitation as to the scope of use or functionality of operating environment architecture. Neither should the operating environment be interpreted as having any dependency or requirement relating to any one or combination of components illustrated in the exemplary operating environment.
- the present methods and systems can be operational with numerous other general purpose or special purpose computing system environments or configurations.
- Examples of well known computing systems, environments, and/or configurations that can be suitable for use with the systems and methods comprise, but are not limited to, personal computers, server computers, laptop devices, custom-built hardware and multiprocessor systems. Additional examples comprise set top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, and/or distributed computing environments that comprise any of the above systems or devices.
- the processing of the disclosed methods and systems can be performed by software components.
- the disclosed systems and methods can be described in the general context of computer-executable instructions, such as program modules, being executed by one or more computers or other devices.
- program modules comprise computer code, routines, programs, objects, components, data structures, etc. that perform particular tasks or implement particular abstract data types.
- the disclosed methods can also be practiced in grid-based, cloud computing and distributed computing environments where tasks are performed by remote processing devices that are linked through a communications network.
- program modules can be located in both local and remote computer storage media including memory storage devices.
- the components of the computing system 901 can comprise, but are not limited to, one or more processors or processing units 903, a system memory 912, and a system bus 913 that couples various system components including the processor 903 to the system memory 912.
- processors or processing units 903 can utilize parallel computing.
- Other components such as hardware acceleration cards, pci-e card or external devices can be included.
- the system bus 913 represents one or more of several possible types of bus structures, including a memory bus or memory controller, a peripheral bus, an accelerated graphics port, and a processor or local bus using any of a variety of bus architectures.
- bus architectures can comprise an Industry Standard Architecture (ISA) bus, a Micro Channel Architecture (MCA) bus, an Enhanced ISA (EISA) bus, a Video Electronics Standards Association (VESA) local bus, an Accelerated Graphics Port (AGP) bus, and a Peripheral Component Interconnects (PCI), a PCI-Express bus, a Personal Computer Memory Card Industry Association (PCMCIA), Universal Serial Bus (USB) and the like.
- ISA Industry Standard Architecture
- MCA Micro Channel Architecture
- EISA Enhanced ISA
- VESA Video Electronics Standards Association
- AGP Accelerated Graphics Port
- PCI Peripheral Component Interconnects
- PCI-Express PCI-Express
- PCMCIA Personal Computer Memory Card Industry Association
- USB Universal Serial Bus
- the bus 913, and all buses specified in this description can also be implemented over a wired or wireless network connection and each of the subsystems, including the processor 903, a mass storage device 904, an operating system 905, content control software 906, content control data 907, a network adapter 908, system memory 912, an Input/Output Interface 910, a display adapter 909, a display device 911, and a human machine interface 902, can be contained within one or more remote computing devices 914a,b,c at physically separate locations, connected through buses of this form, in effect implementing a fully distributed system.
- the computing system 901 typically comprises a variety of computer readable media. Exemplary readable media can be any available media that is accessible by the computing system 901 and comprises, for example and not meant to be limiting, both volatile and non-volatile media, removable and non-removable media.
- the system memory 912 comprises computer readable media in the form of volatile memory, such as random access memory (RAM), and/or non-volatile memory, such as read only memory (ROM).
- RAM random access memory
- ROM read only memory
- the system memory 912 typically contains data such as content control data 907 and/or program modules such as operating system 905 and content control software 906 that are immediately accessible to and/or are presently operated on by the processing unit 903.
- the computing system 901 can also comprise other removable/non-removable, volatile/non-volatile computer storage media.
- FIG. 9 illustrates a mass storage device 904 which can provide non-volatile storage of computer code, computer readable instructions, data structures, program modules, and other data for the computing system 901.
- a mass storage device 904 can be a hard disk, a removable magnetic disk, a removable optical disk, magnetic cassettes or other magnetic storage devices, flash memory cards, CD-ROM, digital versatile disks (DVD) or other optical storage, random access memories (RAM), read only memories (ROM), electrically erasable programmable read-only memory (EEPROM), and the like.
- any number of program modules can be stored on the mass storage device 904, including by way of example, an operating system 905 and content control software 906.
- Each of the operating system 905 and content control software 906 (or some combination thereof) can comprise elements of the programming and the content control software 906.
- Content control data 907 can also be stored on the mass storage device 904.
- Content control data 907 can be stored in any of one or more databases known in the art. Examples of such databases comprise, DB2®, Microsoft® Access, Microsoft® SQL Server, Oracle®, mySQL, PostgreSQL, and the like. The databases can be centralized or distributed across multiple systems.
- content control data 907 can comprise information relating to events, event notifiers, placement spots, alternate programming, programming blackout, advertisements, and the like.
- the content control data can comprise information relating to a particular programming being transmitted to a user location.
- the content control data can comprise information and instructions related to processing placement signals and updating content being transmitted to a user location.
- other information can be associated with the content control data, such as information about the subscriber consuming the content including location, device type, and subscription information, and information relating the content for blackouts including blackout locations and alternative content associated with the blackout.
- the user can enter commands and information into the computing system 901 via an input device (not shown).
- input devices comprise, but are not limited to, a keyboard, pointing device (e.g., a "mouse"), a microphone, a joystick, a scanner, tactile input devices such as gloves, and other body coverings, and the like
- a human machine interface 902 that is coupled to the system bus 913, but can be connected by other interface and bus structures, such as a parallel port, game port, an IEEE 1394 Port (also known as a Firewire port), a serial port, or a universal serial bus (USB).
- a display device 911 can also be connected to the system bus 913 via an interface, such as a display adapter 909. It is contemplated that the computing system 901 can have more than one display adapter 909 and the computing system 901 can have more than one display device 911.
- a display device can be a monitor, an LCD (Liquid Crystal Display), or a projector.
- other output peripheral devices can comprise components such as speakers (not shown) and a printer (not shown) which can be connected to the computing system 901 via Input/Output Interface 910. Any step and/or result of the methods can be output in any form to an output device. Such output can be any form of visual representation, including, but not limited to, textual, graphical, animation, audio, tactile, and the like.
- the display 911 and computing system 901 can be part of one device, or separate devices.
- the computing system 901 can operate in a networked environment using logical connections to one or more remote computing devices 914a,b,c.
- a remote computing device can be a personal computer, portable computer, smartphone, a server, a router, a network computer, a peer device or other common network node, and so on.
- Logical connections between the computing system 901 and a remote computing device 914a,b,c can be made via a network 915, such as a local area network (LAN) and/or a general wide area network (WAN).
- LAN local area network
- WAN general wide area network
- a network adapter 908 can be implemented in both wired and wireless environments. Such networking environments are conventional and commonplace in dwellings, offices, enterprise-wide computer networks, intranets, and the Internet.
- Computer readable media can be any available media that can be accessed by a computer.
- Computer readable media can comprise "computer storage media" and communications media.
- Computer storage media comprise volatile and non-volatile, removable and non-removable media implemented in any methods or technology for storage of information such as computer readable instructions, data structures, program modules, or other data.
- Exemplary computer storage media comprises, but is not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disks (DVD) or other optical storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other medium which can be used to store the desired information and which can be accessed by a computer.
- the methods and systems can employ artificial intelligence (AI) techniques such as machine learning and iterative learning.
- AI artificial intelligence
- techniques include, but are not limited to, expert systems, case based reasoning, Bayesian networks, behavior based AI, neural networks, fuzzy systems, evolutionary computation (e.g. genetic algorithms), swarm intelligence (e.g. ant algorithms), and hybrid intelligent systems (e.g. expert inference rules generated through a neural network or production rules from statistical learning).
Description
- Data can be communicated in various forms over various communication mediums. Data, such as video, can often be modified from a first format to include supplemental content such as overlays, icons, menus, and the like. For example,
US 2012/0/82227 discloses a device and method to insert crawling text into a video bit stream.US 2009/232221 discloses a method and an apparatus to combine an overlayed icon with a compressed video, in which the macroblocks of the video that will be overlayed, are first decoded, then combined with the overlayed content, and finally re-encoded. An encoded video may be completely decoded to a spatial domain; modifications, such as adding supplemental content, are implemented in the spatial domain; and the video is re-encoded for distribution to an end-user. The process of completely decoding the video in order to implement changes and/or add content for each end-user or device is both processor and time intensive. These and other shortcomings are identified and addressed by the disclosure. - It is to be understood that both the following summary and the following detailed description are exemplary and explanatory only and are not restrictive, as claimed. Provided are methods and systems for, in one aspect, encoding data. Provided are methods and systems for, in another aspect, selectively processing data based upon contextual data.
- In an aspect, methods can comprise receiving content data in a first format and receiving context description data, which can be any type of data related to at least a portion of the content data. The content data can be converted to an intermediate format. As an example, the conversion of the content data to the intermediate format can be based at least in part on the context description data. As a further example, the portion of the content data relating to the context description data can be converted to the intermediate format. The content data in the intermediate format can be modified, for example, to include supplemental data. The portion of the content data in the intermediate format can be converted to a second format. As an example, the conversion of the content data to the second format can be based at least in part on the context description data. As a further example, the content data in the second format can be rendered via a downstream device such as a user device.
- In an aspect, methods can comprise converting content data having a first format into one or more of content data having an intermediate format and context description data relating to at least a portion of the content data. The portion of the content data based upon the context description data can be processed, e.g., encoded. As an example, the portion of the content data based upon the context description data can be encoded for distribution to one or more downstream devices. As a further example, the encoded portion of the content data can be rendered via a downstream device such as a user device.
- In an aspect, context description data can be determined. As an example, the context description data can relate to a content data having a first format. As a further example, the content data can be converted from a first format into an intermediate format. The context description data and converted content data can be transmitted to an encoder. In an aspect, the encoder can process a requisite portion of the converted content data based upon the context description data.
- Additional advantages will be set forth in part in the description which follows or may be learned by practice. The advantages will be realized and attained by means of the elements and combinations particularly pointed out in the appended claims.
- The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments and together with the description, serve to explain the principles of the methods and systems:
-
Figure 1 is a block diagram of an exemplary network; -
Figure 2A is a representation of an exemplary presentation of content; -
Figure 2B is a representation of an exemplary presentation of data, showing a layout of the content; -
Figure 3 is a block diagram of an exemplary system; -
Figure 4 is a block diagram of an exemplary system and data flow; -
Figure 5 is a flow chart of an exemplary method; -
Figure 6 is a flow chart of an exemplary method -
Figure 7 is a flow chart of an exemplary method; -
Figure 8 is a flow chart of an exemplary method; and -
Figure 9 is a block diagram of an exemplary computing system. - Before the present methods and systems are disclosed and described, it is to be understood that the methods and systems are not limited to specific methods, specific components, or to particular implementations. It is also to be understood that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting.
- As used in the specification and the appended claims, the singular forms "a," "an," and "the" include plural referents unless the context clearly dictates otherwise. Ranges may be expressed herein as from "about" one particular value, and/or to "about" another particular value. When such a range is expressed, another embodiment includes¬ from the one particular value and/or to the other particular value. Similarly, when values are expressed as approximations, by use of the antecedent "about," it will be understood that the particular value forms another embodiment. It will be further understood that the endpoints of each of the ranges are significant both in relation to the other endpoint, and independently of the other endpoint.
- "Optional" or "optionally" means that the subsequently described event or circumstance may or may not occur, and that the description includes instances where said event or circumstance occurs and instances where it does not.
- Throughout the description and claims of this specification, the word "comprise" and variations of the word, such as "comprising" and "comprises," means "including but not limited to," and is not intended to exclude, for example, other components, integers or steps. "Exemplary" means "an example of" and is not intended to convey an indication of a preferred or ideal embodiment. "Such as" is not used in a restrictive sense, but for explanatory purposes.
- Disclosed are components that can be used to perform the disclosed methods and comprise the disclosed systems. These and other components are disclosed herein, and it is understood that when combinations, subsets, interactions, groups, etc. of these components are disclosed that while specific reference of each various individual and collective combinations and permutation of these may not be explicitly disclosed, each is specifically contemplated and described herein, for all methods and systems. This applies to all aspects of this application including, but not limited to, steps in disclosed methods. Thus, if there are a variety of additional steps that can be performed it is understood that each of these additional steps can be performed with any specific embodiment or combination of embodiments of the disclosed methods.
- The present methods and systems may be understood more readily by reference to the following detailed description of preferred embodiments and the examples included therein and to the Figures and their previous and following description.
- As will be appreciated by one skilled in the art, the methods and systems may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the methods and systems may take the form of a computer program product on a computer-readable storage medium having computer-readable program instructions (e.g., computer software) embodied in the storage medium. More particularly, the present methods and systems may take the form of web-implemented computer software. Any suitable computer-readable storage medium may be utilized including hard disks, CD-ROMs, optical storage devices, or magnetic storage devices.
- Embodiments of the methods and systems are described below with reference to block diagrams and flowchart illustrations of methods, systems, apparatuses and computer program products. It will be understood that each block of the block diagrams and flowchart illustrations, and combinations of blocks in the block diagrams and flowchart illustrations, respectively, can be implemented by computer program instructions. These computer program instructions may be loaded onto a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the instructions which execute on the computer or other programmable data processing apparatus create a means for implementing the functions specified in the flowchart block or blocks.
- These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including computer-readable instructions for implementing the function specified in the flowchart block or blocks. The computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process such that the instructions that execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart block or blocks.
- Accordingly, blocks of the block diagrams and flowchart illustrations support combinations of means for performing the specified functions, combinations of steps for performing the specified functions and program instruction means for performing the specified functions. It will also be understood that each block of the block diagrams and flowchart illustrations, and combinations of blocks in the block diagrams and flowchart illustrations, can be implemented by special purpose hardware-based computer systems that perform the specified functions or steps, or combinations of special purpose hardware and computer instructions.
- As described in greater detail below, provided are methods and systems for controlling content transmitted to a user location. In an aspect, a data stream transmitted to the user location can comprise a repeatedly embedded placement signal to indicate that a particular content should be transmitted to the user location.
-
FIG. 1 illustrates various aspects of an exemplary network environment in which the present methods and systems can operate. Some aspects of the present disclosure relate to methods and systems for content control. Those skilled in the art will appreciate that present methods may be used in systems that employ both digital and analog equipment. One skilled in the art will appreciate that provided herein is a functional description and that the respective functions can be performed by software, hardware, or a combination of software and hardware. - The
network 100 can comprise a central location 101 (e.g., a control or processing facility in a fiber optic network, wireless network or satellite network, a hybrid-fiber coaxial (HFC) content distribution center, a processing center, headend, etc.), which can receive content (e.g., data, input programming) from multiple sources. Thecentral location 101 can combine the content from the various sources and can distribute the content to user locations, such aslocation 119, viadistribution system 116. - In an aspect, the
central location 101 can receive content from a variety ofsources central location 101 via a variety of transmission paths, including wireless (e.g. satellite paths terrestrial path 104. Thecentral location 101 can also receive content from adirect feed source 106 via a direct line 105. Content may also be created at thecentral location 101. Other input sources can comprise capture devices such as avideo camera 109 or aserver 110. The signals provided by the content sources can comprise a single content item or a multiplex that comprises several content items. - The
central location 101 can comprise one or a plurality ofreceivers encoder 112, are included for encoding/transcoding local content or avideo camera 109 feed, terrestrial receiver, or broadcast receiver. As a further example, a data packaging device, such as a fragmentor, can be integrated with the encoder 112 (or separate and in communication with the encoder 112) to package the encoded/transcoded content. In an aspect, theencoder 112 and/or fragmentor can repeatedly embed markers, flags, and signals into the content data stream for processing by downstream devices. Aswitch 113 or a network can provide access toserver 110, which can be, for example, a pay-per-view server, a data server, an internet router, a network system, and/or a phone system. Some signals may require additional processing, such as signal multiplexing or transcoding, prior to being modulated. Such multiplexing can be performed by multiplexer (mux) 114. - The
central location 101 can comprise one or a plurality of modulators, 115a, 115b, 115c, and 115d, for interfacing to thedistribution system 116. The modulators can convert the received content into a modulated output signal suitable for transmission over thedistribution system 116. The output signals from the modulators can be combined, using equipment such as acombiner 117, for input into thedistribution system 116. - A
control system 118 can permit a system operator to control and monitor the functions and performance ofnetwork 100. Thecontrol system 118 can interface, monitor, and/or control a variety of functions, including, but not limited to, the channel lineup for the television system, billing for each user, and/or conditional access for content distributed to users.Control system 118 can provide input to the modulators for setting operating parameters, such as system specific MPEG table packet organization or conditional access information. Thecontrol system 118 can be located atcentral location 101 or at a remote location. - The
distribution system 116 can distribute signals from thecentral location 101 to user locations, such asuser location 119. Thedistribution system 116 can be an optical fiber network, a coaxial cable network, a hybrid fiber-coaxial network, a wireless network, a satellite system, a direct broadcast system, or any combination thereof. There can be a multitude of user locations connected todistribution system 116. Atuser location 119, adecoder 120, such as a gateway or communications terminal (CT) can decode, if needed, the signals for display on a display device, such as on adisplay device 121 such as a television set (TV), a mobile device, or a computer monitor. Those skilled in the art will appreciate that the signal can be decoded in a variety of equipment, including an CT, a computer, a TV, a monitor, or satellite dish. In an exemplary aspect, the methods and systems disclosed can be located within, or performed on, one or more CT's,display devices 121,central locations 101, DVR's, home theater PC's, and the like. As an example, thedecoder 120 can receive and process the embedded markers, flags, and signals in the content data stream for controlling a content received by thedecoder 120. - In an aspect, the
decoder 120 or communication terminal can comprise abuffer 122. As an example, thebuffer 122 can comprise a memory or a storage device. As a further example, a number of the content fragments can be loaded into thebuffer 122 and stored for playback or processing by the associateddecoder 120. However, other data, content, and/or information can be stored in thebuffer 122 such as advertisements and alternative content, for example. In an aspect, thebuffer 122 can be separate from thedecoder 120. For example, thebuffer 122 can be located upstream of thedecoder 120 and downstream of thedistribution system 116. As a further example, thebuffer 122 can store information to be transmitted to any number ofdecoders 120 or other recipient devices. Accordingly, thedecoder 120 can retrieve the content fragments from thebuffer 122 for playback without having to communicate with thedistribution system 116, thereby reducing latency in playback. In an aspect, thedecoder 120 can process a continuous streaming track, which can comprise audio, video, or other data from thedistribution system 116 with a regular cadence (e.g., every two seconds, four seconds, ten seconds). Other data and tracks can be received and processed by the user devices. - In an aspect,
user location 119 is not fixed. By way of example, a user can receive content from thedistribution system 116 on a mobile device such as a laptop computer, PDA, smartphone, GPS, vehicle entertainment system, and/or portable media player. The network ofFIG. 1 is illustrative only and other types of networks and configurations may be used to implement the disclosure. - In an aspect, one or more of a
layout element 124 and anencoder 126 can be disposed upstream of thedecoder 120. As an example, one or more of thelayout element 124 and theencoder 126 can be configured to receive data, such as video, audio, images, overlays, menus, icons, and the like. As a further example, one or more of thelayout element 124 and theencoder 126 can receive data from a data source such as a content provider, thecentral location 101, and/or thedistribution system 116. - In an aspect, the
layout element 124 can be configured to receive data such as content data (e.g., images, audio, video, menus, icons, banners, overlays, etc.). As an example, thelayout element 124 can receive data in a first format, such as hypertext markup language (HTML). Other formats can be used such as Scalable Vector Graphics (SVG), Extended Markup Language (XML) with associated XML Style Sheets (XSS), Extended Post Script (EPS), Portable Document Format (PDF), proprietary document formats such as various presentation and document formats that can describe the layout of content, without having to combine the various content into pictures or video. As a further example, the data received by thelayout element 124 can represent images, video, and/or frames of video, as illustrated inFIG. 2A . In an aspect, the first format of the data can comprise positional and presentation information for rendering an image, such as:
<div style="position: absolute; left: 0px; top: 0px;"> <img src="blanker5.png"/> </div>which illustrates style and positioning presentation in HTML for the image source file blanker5.png. Other positioning and presenting information can be represented by the first format. Any number of images and/or data points can be managed by the data in the first format. The
- a. Converting all macro-block types to the destination macro-block type. For example, if video is output using a P frame and the component video has an I frame at that instant, the I frame can be converted to a P frame. As a further example, the type of frame to be used for the output can be a heuristic, which can be an optimization of the processor time it takes to change all the components to the target frame type and the resulting number of bits in that picture; and
- b. Adding slice headers.
- i. completely re-encoding the DCT and adding null motion vectors to reconstruct the macro-block; and
- ii. adjusting and/or adding coefficients to the DCT to compensate for the motion vectors pointing to other content components.
Claims (10)
- A method comprising
receiving content data (402b) and determining context description data associated with the content data (402b), wherein the context description data comprises location information of the content data (402b) in relation to a video frame of a video transmission (404) comprising a plurality of video frames, wherein the context description data is further processed to obtain a level of processing comprising an extent of decoding
receiving the video transmission (404) comprising a plurality of video frames in a first format and to receive context description data associated with the content data (402a), wherein the said means (126, 304, 406) is further configured to convert a portion of the video frame to an intermediate format based on the context description data by decoding the portion of the video frame according to the extent of decoding; and
modifying the portion of the video frame in the intermediate format to incorporate at least a portion of the content data (402b) into the video frame, and to convert at least the portion of the video frame from the intermediate format with the incorporated portion of the content data (402b) to a second format. - The method of claim 1, wherein the first format is hypertext markup language and the decoding comprises removing encapsulation of data.
- The method of claim 1, wherein the second format is the same as the first format and wherein converting the portion of the video frame from the intermediate format with at least the portion of the content data to the second format comprises encoding the portion of the video frame in the intermediate format back to the first format.
- The method of claim 1, wherein the second format is MPEG.
- The method of claim 1, wherein the content data (402b) comprises one or more of text, images, vectors, highlights, animations, videos, and overlays.
- The method of claim 1, wherein the context description data comprises one or more of a component list, movement information, vector operations, and excluded regions.
- The method of claim 1, wherein converting the portion of the video frame to an intermediate format comprises decoding only the portion of the video frame relating to the context description data.
- The method of claim 1, further comprising encoding supplement content data (402a) to be combined with the video frame in the second format.
- The method of claim 1, further comprising rendering the video frame having the second format.
- A system comprising:a layout engine (124, 400) configured to receive content data (402b), and determine context description data associated with the content data (402b), wherein the context description data comprises location information of the content data (402b) in relation to a video frame of a video transmission (404) comprising a plurality of video frames, wherein the context description data is further processed to obtain a level of processing comprising an extent of decoding; a means (126, 304, 406) configured to receive the video transmission (404) comprising a plurality of video frames in a first format and to receive context description data associated with the content data (402a), wherein the said means (126, 304, 406) is further configured to convert a portion of the video frame to an intermediate format based on the context description data by decoding the portion of the video frame according to the extent of decoding; andan encoder (408) configured to modify the portion of the video frame in the intermediate format to incorporate at least a portion of the content data (402b) into the video frame, and to convert at least the portion of the video frame from the intermediate format with the incorporated portion of the content data (402b) to a second format.
Applications Claiming Priority (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US13/774,306 US9648320B2 (en) | 2013-02-22 | 2013-02-22 | Methods and systems for processing content |
Publications (3)
Publication Number | Publication Date |
---|---|
EP2770743A2 EP2770743A2 (en) | 2014-08-27 |
EP2770743A3 EP2770743A3 (en) | 2014-10-29 |
EP2770743B1 true EP2770743B1 (en) | 2018-05-02 |
Family
ID=50193213
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
EP14155941.9A Active EP2770743B1 (en) | 2013-02-22 | 2014-02-20 | Methods and systems for processing content |
Country Status (3)
Country | Link |
---|---|
US (1) | US9648320B2 (en) |
EP (1) | EP2770743B1 (en) |
CA (1) | CA2843718C (en) |
Families Citing this family (5)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
WO2016209125A1 (en) * | 2015-06-23 | 2016-12-29 | Telefonaktiebolaget Lm Ericsson (Publ) | Methods and arrangements for transcoding |
WO2019023488A1 (en) * | 2017-07-28 | 2019-01-31 | Dolby Laboratories Licensing Corporation | Method and system for providing media content to a client |
US10778412B2 (en) | 2017-12-28 | 2020-09-15 | Intel Corporation | Multi-domain convolutional neural network |
US11349753B2 (en) * | 2017-12-28 | 2022-05-31 | Intel Corporation | Converged routing for distributed computing systems |
US20190034716A1 (en) | 2017-12-28 | 2019-01-31 | Intel Corporation | Privacy-preserving demographics identification |
Family Cites Families (45)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
KR100327211B1 (en) * | 1998-05-29 | 2002-05-09 | 윤종용 | Sub-picture encoding method and apparatus |
US6266369B1 (en) | 1998-06-09 | 2001-07-24 | Worldgate Service, Inc. | MPEG encoding technique for encoding web pages |
US6732370B1 (en) | 1998-11-30 | 2004-05-04 | Diva Systems Corporation | Service provider side interactive program guide encoder |
US7444663B2 (en) | 1998-12-31 | 2008-10-28 | Lodgenet Interactive Corporation | Menuing system for controlling content delivery within a video distribution system |
JP2000270322A (en) | 1999-03-17 | 2000-09-29 | Fujitsu Ltd | Device and method for coding moving picture |
JP2000333169A (en) | 1999-05-24 | 2000-11-30 | Mitsubishi Electric Corp | Moving picture coder |
JP4438129B2 (en) * | 1999-07-02 | 2010-03-24 | ソニー株式会社 | Content receiving system and content receiving method |
US6845485B1 (en) * | 1999-07-15 | 2005-01-18 | Hotv, Inc. | Method and apparatus for indicating story-line changes by mining closed-caption-text |
US9094727B1 (en) * | 1999-10-27 | 2015-07-28 | Cox Communications, Inc. | Multi-functional user interface using slice-based encoding |
US6542546B1 (en) * | 2000-02-02 | 2003-04-01 | Mitsubishi Electric Research Laboratories, Inc. | Adaptable compressed bitstream transcoder |
US6507618B1 (en) * | 2000-04-25 | 2003-01-14 | Hewlett-Packard Company | Compressed video signal including independently coded regions |
US6832215B2 (en) | 2000-07-21 | 2004-12-14 | Microsoft Corporation | Method for redirecting the source of a data object displayed in an HTML document |
US7073122B1 (en) | 2000-09-08 | 2006-07-04 | Sedghi Ali R | Method and apparatus for extracting structured data from HTML pages |
US6791548B2 (en) | 2000-12-14 | 2004-09-14 | International Business Machines Corporation | Method and apparatus for automatically displaying dynamic images as a sequence of still frames arranged in a predetermined order |
US7281260B2 (en) | 2001-08-07 | 2007-10-09 | Loral Cyberstar, Inc. | Streaming media publishing system and method |
US7249264B2 (en) * | 2002-04-02 | 2007-07-24 | International Business Machines Corporation | Secure IP based streaming in a format independent manner |
US20040073941A1 (en) | 2002-09-30 | 2004-04-15 | Ludvig Edward A. | Systems and methods for dynamic conversion of web content to an interactive walled garden program |
US8037496B1 (en) * | 2002-12-27 | 2011-10-11 | At&T Intellectual Property Ii, L.P. | System and method for automatically authoring interactive television content |
US7552168B2 (en) | 2002-12-30 | 2009-06-23 | International Business Machines Corporation | Method and apparatus for exchanging data using programmatic conversion to emulated HTML form data |
US7451392B1 (en) | 2003-06-30 | 2008-11-11 | Microsoft Corporation | Rendering an HTML electronic form by applying XSLT to XML using a solution |
US7590997B2 (en) * | 2004-07-30 | 2009-09-15 | Broadband Itv, Inc. | System and method for managing, converting and displaying video content on a video-on-demand platform, including ads used for drill-down navigation and consumer-generated classified ads |
US7762274B2 (en) * | 2006-02-14 | 2010-07-27 | Eaton Corporation | Hydrant valve pilot with detachable actuator |
US8209724B2 (en) * | 2007-04-25 | 2012-06-26 | Samsung Electronics Co., Ltd. | Method and system for providing access to information of potential interest to a user |
US8103125B2 (en) | 2007-03-13 | 2012-01-24 | International Business Machines Corporation | Generating an amalgamated image including a static image and a dynamic image |
US9426479B2 (en) * | 2007-08-09 | 2016-08-23 | Cisco Technology, Inc. | Preserving captioning through video transcoding |
US8681874B2 (en) | 2008-03-13 | 2014-03-25 | Cisco Technology, Inc. | Video insertion information insertion in a compressed bitstream |
US8139872B2 (en) * | 2008-06-27 | 2012-03-20 | Microsoft Corporation | Splitting file types within partitioned images |
US9473812B2 (en) * | 2008-09-10 | 2016-10-18 | Imagine Communications Corp. | System and method for delivering content |
US8051287B2 (en) * | 2008-10-15 | 2011-11-01 | Adobe Systems Incorporated | Imparting real-time priority-based network communications in an encrypted communication session |
US9141859B2 (en) * | 2008-11-17 | 2015-09-22 | Liveclips Llc | Method and system for segmenting and transmitting on-demand live-action video in real-time |
CN101447207B (en) * | 2008-12-30 | 2012-02-15 | 华为终端有限公司 | Media recording method and device thereof |
KR100930303B1 (en) * | 2009-03-19 | 2009-12-08 | 주식회사 파수닷컴 | Digital media contents protection system and method thereof |
CA2759880C (en) * | 2009-03-23 | 2013-09-24 | Azuki Systems, Inc. | Method and system for efficient streaming video dynamic rate adaptation |
US8281343B2 (en) * | 2009-05-19 | 2012-10-02 | Cisco Technology, Inc. | Management and display of video content |
US8457311B1 (en) * | 2010-04-27 | 2013-06-04 | Adobe Systems Incorporated | Protecting video as it is decoded by a codec |
US9014269B2 (en) | 2010-09-30 | 2015-04-21 | General Instrument Corporation | Method and apparatus for managing bit rate |
US20120124172A1 (en) * | 2010-11-15 | 2012-05-17 | Google Inc. | Providing Different Versions of a Media File |
US20120166289A1 (en) * | 2010-12-22 | 2012-06-28 | Ando Media Llc | Real-time media stream insertion method and apparatus |
WO2012094487A2 (en) * | 2011-01-05 | 2012-07-12 | General Instrument Corporation | Secure progressive download for media content playback |
US8782165B2 (en) * | 2011-01-26 | 2014-07-15 | Openwave Mobility, Inc. | Method and transcoding proxy for transcoding a media stream that is delivered to an end-user device over a communications network |
US9661104B2 (en) * | 2011-02-07 | 2017-05-23 | Blackberry Limited | Method and apparatus for receiving presentation metadata |
US9843844B2 (en) * | 2011-10-05 | 2017-12-12 | Qualcomm Incorporated | Network streaming of media data |
US9432704B2 (en) * | 2011-11-06 | 2016-08-30 | Akamai Technologies Inc. | Segmented parallel encoding with frame-aware, variable-size chunking |
US8887193B2 (en) * | 2012-01-27 | 2014-11-11 | Sony Corporation | System, method, and infrastructure for real-time live streaming content |
US8695048B1 (en) * | 2012-10-15 | 2014-04-08 | Wowza Media Systems, LLC | Systems and methods of processing closed captioning for video on demand content |
-
2013
- 2013-02-22 US US13/774,306 patent/US9648320B2/en active Active
-
2014
- 2014-02-20 EP EP14155941.9A patent/EP2770743B1/en active Active
- 2014-02-21 CA CA2843718A patent/CA2843718C/en active Active
Non-Patent Citations (1)
Title |
---|
None * |
Also Published As
Publication number | Publication date |
---|---|
CA2843718A1 (en) | 2014-08-22 |
US20140240591A1 (en) | 2014-08-28 |
US9648320B2 (en) | 2017-05-09 |
EP2770743A3 (en) | 2014-10-29 |
EP2770743A2 (en) | 2014-08-27 |
CA2843718C (en) | 2021-05-25 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
CN108476324B (en) | Method, computer and medium for enhancing regions of interest in video frames of a video stream | |
US9628533B2 (en) | Method and device for generating a description file, and corresponding streaming method | |
US20230232059A1 (en) | Methods and systems for content control | |
US9542953B2 (en) | Intelligent data delivery | |
US11722711B2 (en) | System and method for data stream fragmentation | |
US20140247887A1 (en) | Just-in-time (jit) encoding for streaming media content | |
EP2770743B1 (en) | Methods and systems for processing content | |
KR101482795B1 (en) | METHOD AND APPARATUS FOR TRANSMITING/RECEIVING LASeR CONTENTS | |
US11178467B1 (en) | Use video codecs to deliver images | |
US11451849B2 (en) | Methods and systems for content control | |
US20140098897A1 (en) | Providing a Common Set of Parameters for Sub-Layers of Coded Video | |
US20230247105A1 (en) | Methods and systems for content delivery using server push | |
CA2845986A1 (en) | Methods and systems for dynamic data management | |
US11653054B2 (en) | Method and apparatus for late binding in media content | |
US9060184B2 (en) | Systems and methods for adaptive streaming with augmented video stream transitions using a media server | |
Laghari et al. | The state of art and review on video streaming | |
US20150281740A1 (en) | Transmission device, transmitting method, reception device, and receiving method | |
US20190253738A1 (en) | Method for processing encoded data, method for receiving encoded data, devices, and associated computer programs | |
US10136146B1 (en) | Metadata optimizations for encoding of media content | |
CN116636224A (en) | System and method for replacing networking application program service | |
US11409415B1 (en) | Frame interpolation for media streaming | |
EP1338149B1 (en) | Method and device for video scene composition from varied data | |
Gudumasu et al. | Segment scheduling method for reducing 360° video streaming latency |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
PUAI | Public reference made under article 153(3) epc to a published international application that has entered the european phase |
Free format text: ORIGINAL CODE: 0009012 |
|
17P | Request for examination filed |
Effective date: 20140220 |
|
AK | Designated contracting states |
Kind code of ref document: A2 Designated state(s): AL AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HR HU IE IS IT LI LT LU LV MC MK MT NL NO PL PT RO RS SE SI SK SM TR |
|
AX | Request for extension of the european patent |
Extension state: BA ME |
|
PUAL | Search report despatched |
Free format text: ORIGINAL CODE: 0009013 |
|
AK | Designated contracting states |
Kind code of ref document: A3 Designated state(s): AL AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HR HU IE IS IT LI LT LU LV MC MK MT NL NO PL PT RO RS SE SI SK SM TR |
|
AX | Request for extension of the european patent |
Extension state: BA ME |
|
RIC1 | Information provided on ipc code assigned before grant |
Ipc: H04N 19/463 20140101ALI20140926BHEP Ipc: H04N 7/01 20060101ALI20140926BHEP Ipc: H04N 21/431 20110101AFI20140926BHEP Ipc: H04N 21/4402 20110101ALI20140926BHEP Ipc: H04N 21/434 20110101ALI20140926BHEP |
|
R17P | Request for examination filed (corrected) |
Effective date: 20150429 |
|
RBV | Designated contracting states (corrected) |
Designated state(s): AL AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HR HU IE IS IT LI LT LU LV MC MK MT NL NO PL PT RO RS SE SI SK SM TR |
|
17Q | First examination report despatched |
Effective date: 20160623 |
|
STAA | Information on the status of an ep patent application or granted ep patent |
Free format text: STATUS: EXAMINATION IS IN PROGRESS |
|
GRAJ | Information related to disapproval of communication of intention to grant by the applicant or resumption of examination proceedings by the epo deleted |
Free format text: ORIGINAL CODE: EPIDOSDIGR1 |
|
STAA | Information on the status of an ep patent application or granted ep patent |
Free format text: STATUS: GRANT OF PATENT IS INTENDED |
|
GRAP | Despatch of communication of intention to grant a patent |
Free format text: ORIGINAL CODE: EPIDOSNIGR1 |
|
INTG | Intention to grant announced |
Effective date: 20171120 |
|
GRAS | Grant fee paid |
Free format text: ORIGINAL CODE: EPIDOSNIGR3 |
|
GRAA | (expected) grant |
Free format text: ORIGINAL CODE: 0009210 |
|
STAA | Information on the status of an ep patent application or granted ep patent |
Free format text: STATUS: THE PATENT HAS BEEN GRANTED |
|
AK | Designated contracting states |
Kind code of ref document: B1 Designated state(s): AL AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HR HU IE IS IT LI LT LU LV MC MK MT NL NO PL PT RO RS SE SI SK SM TR |
|
REG | Reference to a national code |
Ref country code: GB Ref legal event code: FG4D |
|
REG | Reference to a national code |
Ref country code: CH Ref legal event code: EP Ref country code: AT Ref legal event code: REF Ref document number: 996477 Country of ref document: AT Kind code of ref document: T Effective date: 20180515 |
|
REG | Reference to a national code |
Ref country code: DE Ref legal event code: R096 Ref document number: 602014024723 Country of ref document: DE Ref country code: IE Ref legal event code: FG4D |
|
REG | Reference to a national code |
Ref country code: NL Ref legal event code: MP Effective date: 20180502 |
|
REG | Reference to a national code |
Ref country code: LT Ref legal event code: MG4D |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: ES Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20180502 Ref country code: LT Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20180502 Ref country code: FI Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20180502 Ref country code: BG Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20180802 Ref country code: SE Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20180502 Ref country code: NO Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20180802 |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: GR Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20180803 Ref country code: HR Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20180502 Ref country code: RS Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20180502 Ref country code: LV Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20180502 Ref country code: NL Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20180502 |
|
REG | Reference to a national code |
Ref country code: AT Ref legal event code: MK05 Ref document number: 996477 Country of ref document: AT Kind code of ref document: T Effective date: 20180502 |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: SK Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20180502 Ref country code: CZ Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20180502 Ref country code: RO Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20180502 Ref country code: PL Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20180502 Ref country code: AT Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20180502 Ref country code: DK Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20180502 Ref country code: EE Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20180502 |
|
REG | Reference to a national code |
Ref country code: DE Ref legal event code: R097 Ref document number: 602014024723 Country of ref document: DE |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: IT Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20180502 Ref country code: SM Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20180502 |
|
PLBE | No opposition filed within time limit |
Free format text: ORIGINAL CODE: 0009261 |
|
STAA | Information on the status of an ep patent application or granted ep patent |
Free format text: STATUS: NO OPPOSITION FILED WITHIN TIME LIMIT |
|
26N | No opposition filed |
Effective date: 20190205 |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: SI Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20180502 |
|
REG | Reference to a national code |
Ref country code: CH Ref legal event code: PL |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: MC Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20180502 Ref country code: LU Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES Effective date: 20190220 |
|
REG | Reference to a national code |
Ref country code: BE Ref legal event code: MM Effective date: 20190228 |
|
REG | Reference to a national code |
Ref country code: IE Ref legal event code: MM4A |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: AL Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20180502 |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: CH Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES Effective date: 20190228 Ref country code: LI Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES Effective date: 20190228 |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: IE Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES Effective date: 20190220 |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: BE Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES Effective date: 20190228 |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: TR Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20180502 |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: PT Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20180903 Ref country code: MT Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES Effective date: 20190220 |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: CY Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20180502 |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: IS Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20180902 |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: HU Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT; INVALID AB INITIO Effective date: 20140220 |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: MK Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20180502 |
|
PGFP | Annual fee paid to national office [announced via postgrant information from national office to epo] |
Ref country code: FR Payment date: 20230223 Year of fee payment: 10 |
|
PGFP | Annual fee paid to national office [announced via postgrant information from national office to epo] |
Ref country code: GB Payment date: 20230227 Year of fee payment: 10 Ref country code: DE Payment date: 20230223 Year of fee payment: 10 |
|
P01 | Opt-out of the competence of the unified patent court (upc) registered |
Effective date: 20230505 |