WO2001086456A1 - Scheduling and delivering low bandwidth media upon detecting high bandwidth media - Google Patents

Scheduling and delivering low bandwidth media upon detecting high bandwidth media Download PDF

Info

Publication number
WO2001086456A1
WO2001086456A1 PCT/US2001/040685 US0140685W WO0186456A1 WO 2001086456 A1 WO2001086456 A1 WO 2001086456A1 US 0140685 W US0140685 W US 0140685W WO 0186456 A1 WO0186456 A1 WO 0186456A1
Authority
WO
WIPO (PCT)
Prior art keywords
media
information
network
data stream
interruption
Prior art date
Application number
PCT/US2001/040685
Other languages
French (fr)
Inventor
Robert C. Plotkin
Jonathan D. Katz
Nathan D. Leight
Original Assignee
Vast Video, Incorporated
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Vast Video, Incorporated filed Critical Vast Video, Incorporated
Publication of WO2001086456A1 publication Critical patent/WO2001086456A1/en

Links

Classifications

    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/20—Servers specifically adapted for the distribution of content, e.g. VOD servers; Operations thereof
    • H04N21/23—Processing of content or additional data; Elementary server operations; Server middleware
    • H04N21/234—Processing of video elementary streams, e.g. splicing of video streams or manipulating encoded video stream scene graphs
    • H04N21/23406—Processing of video elementary streams, e.g. splicing of video streams or manipulating encoded video stream scene graphs involving management of server-side video buffer
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L65/00—Network arrangements, protocols or services for supporting real-time applications in data packet communication
    • H04L65/60—Network streaming of media packets
    • H04L65/70—Media network packetisation
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L65/00—Network arrangements, protocols or services for supporting real-time applications in data packet communication
    • H04L65/60—Network streaming of media packets
    • H04L65/75—Media network packet handling
    • H04L65/752—Media network packet handling adapting media to network capabilities
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L65/00—Network arrangements, protocols or services for supporting real-time applications in data packet communication
    • H04L65/60—Network streaming of media packets
    • H04L65/75—Media network packet handling
    • H04L65/762—Media network packet handling at the source 
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L65/00—Network arrangements, protocols or services for supporting real-time applications in data packet communication
    • H04L65/80—Responding to QoS
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L67/00—Network arrangements or protocols for supporting network services or applications
    • H04L67/14—Session management
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L67/00—Network arrangements or protocols for supporting network services or applications
    • H04L67/50—Network services
    • H04L67/53—Network services using third party service providers
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L67/00—Network arrangements or protocols for supporting network services or applications
    • H04L67/50—Network services
    • H04L67/75—Indicating network or usage conditions on the user display
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L69/00—Network arrangements, protocols or services independent of the application payload and not provided for in the other groups of this subclass
    • H04L69/40—Network arrangements, protocols or services independent of the application payload and not provided for in the other groups of this subclass for recovering from a failure of a protocol instance or entity, e.g. service redundancy protocols, protocol state redundancy or protocol service redirection
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/20—Servers specifically adapted for the distribution of content, e.g. VOD servers; Operations thereof
    • H04N21/23—Processing of content or additional data; Elementary server operations; Server middleware
    • H04N21/234—Processing of video elementary streams, e.g. splicing of video streams or manipulating encoded video stream scene graphs
    • H04N21/2343—Processing of video elementary streams, e.g. splicing of video streams or manipulating encoded video stream scene graphs involving reformatting operations of video signals for distribution or compliance with end-user requests or end-user device requirements
    • H04N21/234327—Processing of video elementary streams, e.g. splicing of video streams or manipulating encoded video stream scene graphs involving reformatting operations of video signals for distribution or compliance with end-user requests or end-user device requirements by decomposing into layers, e.g. base layer and one or more enhancement layers
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/20—Servers specifically adapted for the distribution of content, e.g. VOD servers; Operations thereof
    • H04N21/23—Processing of content or additional data; Elementary server operations; Server middleware
    • H04N21/236—Assembling of a multiplex stream, e.g. transport stream, by combining a video stream with other content or additional data, e.g. inserting a URL [Uniform Resource Locator] into a video stream, multiplexing software data into a video stream; Remultiplexing of multiplex streams; Insertion of stuffing bits into the multiplex stream, e.g. to obtain a constant bit-rate; Assembling of a packetised elementary stream
    • H04N21/23614—Multiplexing of additional data and video streams
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/20—Servers specifically adapted for the distribution of content, e.g. VOD servers; Operations thereof
    • H04N21/23—Processing of content or additional data; Elementary server operations; Server middleware
    • H04N21/24—Monitoring of processes or resources, e.g. monitoring of server load, available bandwidth, upstream requests
    • H04N21/2402—Monitoring of the downstream path of the transmission network, e.g. bandwidth available
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/20—Servers specifically adapted for the distribution of content, e.g. VOD servers; Operations thereof
    • H04N21/25—Management operations performed by the server for facilitating the content distribution or administrating data related to end-users or client devices, e.g. end-user or client device authentication, learning user preferences for recommending movies
    • H04N21/266—Channel or content management, e.g. generation and management of keys and entitlement messages in a conditional access system, merging a VOD unicast channel into a multicast channel
    • H04N21/2662—Controlling the complexity of the video stream, e.g. by scaling the resolution or bitrate of the video stream based on the client capabilities
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
    • H04N21/43—Processing of content or additional data, e.g. demultiplexing additional data from a digital video stream; Elementary client operations, e.g. monitoring of home network or synchronising decoder's clock; Client middleware
    • H04N21/431—Generation of visual interfaces for content selection or interaction; Content or additional data rendering
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
    • H04N21/43—Processing of content or additional data, e.g. demultiplexing additional data from a digital video stream; Elementary client operations, e.g. monitoring of home network or synchronising decoder's clock; Client middleware
    • H04N21/433—Content storage operation, e.g. storage operation in response to a pause request, caching operations
    • H04N21/4331—Caching operations, e.g. of an advertisement for later insertion during playback
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
    • H04N21/43—Processing of content or additional data, e.g. demultiplexing additional data from a digital video stream; Elementary client operations, e.g. monitoring of home network or synchronising decoder's clock; Client middleware
    • H04N21/434—Disassembling of a multiplex stream, e.g. demultiplexing audio and video streams, extraction of additional data from a video stream; Remultiplexing of multiplex streams; Extraction or processing of SI; Disassembling of packetised elementary stream
    • H04N21/4348—Demultiplexing of additional data and video streams
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
    • H04N21/43—Processing of content or additional data, e.g. demultiplexing additional data from a digital video stream; Elementary client operations, e.g. monitoring of home network or synchronising decoder's clock; Client middleware
    • H04N21/44—Processing of video elementary streams, e.g. splicing a video clip retrieved from local storage with an incoming video stream or rendering scenes according to encoded video stream scene graphs
    • H04N21/44004—Processing of video elementary streams, e.g. splicing a video clip retrieved from local storage with an incoming video stream or rendering scenes according to encoded video stream scene graphs involving video buffer management, e.g. video decoder buffer or video display buffer
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
    • H04N21/43—Processing of content or additional data, e.g. demultiplexing additional data from a digital video stream; Elementary client operations, e.g. monitoring of home network or synchronising decoder's clock; Client middleware
    • H04N21/44—Processing of video elementary streams, e.g. splicing a video clip retrieved from local storage with an incoming video stream or rendering scenes according to encoded video stream scene graphs
    • H04N21/4402—Processing of video elementary streams, e.g. splicing a video clip retrieved from local storage with an incoming video stream or rendering scenes according to encoded video stream scene graphs involving reformatting operations of video signals for household redistribution, storage or real-time display
    • H04N21/440227—Processing of video elementary streams, e.g. splicing a video clip retrieved from local storage with an incoming video stream or rendering scenes according to encoded video stream scene graphs involving reformatting operations of video signals for household redistribution, storage or real-time display by decomposing into layers, e.g. base layer and one or more enhancement layers
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
    • H04N21/43—Processing of content or additional data, e.g. demultiplexing additional data from a digital video stream; Elementary client operations, e.g. monitoring of home network or synchronising decoder's clock; Client middleware
    • H04N21/442—Monitoring of processes or resources, e.g. detecting the failure of a recording device, monitoring the downstream bandwidth, the number of times a movie has been viewed, the storage space available from the internal hard disk
    • H04N21/44209—Monitoring of downstream path of the transmission network originating from a server, e.g. bandwidth variations of a wireless network
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
    • H04N21/45—Management operations performed by the client for facilitating the reception of or the interaction with the content or administrating data related to the end-user or to the client device itself, e.g. learning user preferences for recommending movies, resolving scheduling conflicts
    • H04N21/462—Content or additional data management, e.g. creating a master electronic program guide from data received from the Internet and a Head-end, controlling the complexity of a video stream by scaling the resolution or bit-rate based on the client capabilities
    • H04N21/4621—Controlling the complexity of the content stream or additional data, e.g. lowering the resolution or bit-rate of the video stream for a mobile client with a small screen
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/60—Network structure or processes for video distribution between server and client or between remote clients; Control signalling between clients, server and network components; Transmission of management data between server and client, e.g. sending from server to client commands for recording incoming content stream; Communication details between server and client 
    • H04N21/61—Network physical structure; Signal processing
    • H04N21/6106—Network physical structure; Signal processing specially adapted to the downstream path of the transmission network
    • H04N21/6125—Network physical structure; Signal processing specially adapted to the downstream path of the transmission network involving transmission via Internet
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/60—Network structure or processes for video distribution between server and client or between remote clients; Control signalling between clients, server and network components; Transmission of management data between server and client, e.g. sending from server to client commands for recording incoming content stream; Communication details between server and client 
    • H04N21/63—Control signaling related to video distribution between client, server and network components; Network processes for video distribution between server and clients or between remote clients, e.g. transmitting basic layer and enhancement layers over different transmission paths, setting up a peer-to-peer communication via Internet between remote STB's; Communication protocols; Addressing
    • H04N21/643—Communication protocols
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/60—Network structure or processes for video distribution between server and client or between remote clients; Control signalling between clients, server and network components; Transmission of management data between server and client, e.g. sending from server to client commands for recording incoming content stream; Communication details between server and client 
    • H04N21/65—Transmission of management data between client and server
    • H04N21/654—Transmission by server directed to the client
    • H04N21/6543—Transmission by server directed to the client for forcing some client operations, e.g. recording
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/60—Network structure or processes for video distribution between server and client or between remote clients; Control signalling between clients, server and network components; Transmission of management data between server and client, e.g. sending from server to client commands for recording incoming content stream; Communication details between server and client 
    • H04N21/65—Transmission of management data between client and server
    • H04N21/654—Transmission by server directed to the client
    • H04N21/6547—Transmission by server directed to the client comprising parameters, e.g. for client setup
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/60—Network structure or processes for video distribution between server and client or between remote clients; Control signalling between clients, server and network components; Transmission of management data between server and client, e.g. sending from server to client commands for recording incoming content stream; Communication details between server and client 
    • H04N21/65—Transmission of management data between client and server
    • H04N21/658—Transmission by the client directed to the server
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/80—Generation or processing of content or additional data by content creator independently of the distribution process; Content per se
    • H04N21/81—Monomedia components thereof
    • H04N21/812—Monomedia components thereof involving advertisement data
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/80—Generation or processing of content or additional data by content creator independently of the distribution process; Content per se
    • H04N21/81—Monomedia components thereof
    • H04N21/8146—Monomedia components thereof involving graphical data, e.g. 3D object, 2D graphics
    • H04N21/8153—Monomedia components thereof involving graphical data, e.g. 3D object, 2D graphics comprising still images, e.g. texture, background image
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/80—Generation or processing of content or additional data by content creator independently of the distribution process; Content per se
    • H04N21/85—Assembly of content; Generation of multimedia applications
    • H04N21/854—Content authoring
    • H04N21/8549—Creating video summaries, e.g. movie trailer
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L69/00—Network arrangements, protocols or services independent of the application payload and not provided for in the other groups of this subclass
    • H04L69/30—Definitions, standards or architectural aspects of layered protocol stacks
    • H04L69/32—Architecture of open systems interconnection [OSI] 7-layer type protocol stacks, e.g. the interfaces between the data link level and the physical level
    • H04L69/322—Intralayer communication protocols among peer entities or protocol data unit [PDU] definitions
    • H04L69/329—Intralayer communication protocols among peer entities or protocol data unit [PDU] definitions in the application layer [OSI layer 7]

Definitions

  • the present invention generally relates handling network 10 bottlenecks of high bandwidth media such as streaming video, and more particularly to the scheduling and delivering of low bandwidth media to a user upon the detection of an interruption of the high bandwidth media.
  • high bandwidth media such as streaming video over global networks, such as the Internet.
  • Systems such as the one offered by Real Networks or Microsoft streaming media player can deliver and play video at rates in excess of 400 kilobits per second (Kb/sec).
  • high bandwidth systems are defined as those capable of 20 delivering data at a rate above that available with traditional telephony based modem technology, which at the time of this disclosure is 56Kb/sec.
  • one embodiment of a high bandwidth system is a cable modem, which has a data transfer rate capability in excess of 1 megabytes per second (MB/sec.) or greater.
  • a number of transmission technologies also exist that are intended to provide adequate bandwidth so that data streams containing video or audio can be delivered to a client (e.g., a software browser) in a continuous, uninterrupted format.
  • a client e.g., a software browser
  • the Web World Wide Web
  • the network may be a local area network, a wide-area network, or a global network such as the Internet.
  • GIF Graphics Interface Format
  • banners typically used for advertising in such application as "banners”.
  • Static GIF images can be made such that their total size is around 10KB. It can be seen how the download requirements for such a banner are trivial compared to the total bandwidth of a streaming video of 400Kb/sec. In this example, an image could be downloaded in 0.25 second before video (or other media) begins.
  • An embodiment of the invention can load and store this collateral image or images for display upon the interruption of the primary stream.
  • rules can be delivered to determine the location of this collateral information within the user screen. For instance, if a set of user screens included media sections (for video) as well as several ancillary locations for other material such as advertisements, specific static graphical material could be pre-loaded for possible display in the one region of the screen if an interruption to the high bandwidth media is detected.
  • the advertising or related material can be pre-loaded to the client before the streaming media begins, or can be interleaved with the data.
  • the rules as to when this media can be displayed could be delivered at the same time this alternate material is delivered.
  • the rules for display of alternate material can address a number of issues including thresholds for when to display, the length of display, location on the page to display, etc.
  • Figure 1 is a block and flow diagram showing how a server may deliver a combination of the high- and low-bandwidth information and a set of display rules to a client that can detect system bandwidth and display either the high- or low-bandwidth information;
  • FIG. 2 is a block diagram showing an manner in which a data stream may be constructed in accordance with the present invention.
  • Figure 1 shows the manner in which a data stream is constructed in accordance with the present invention.
  • a server 100 combines at least two types of information. into the data stream.
  • First information 101 is high-bandwidth information which may correspond to streaming audio or streaming video which may or may not include accompanying audio.
  • Second information 102 is low- bandwidth information such as a GIF, an animated GIF, or even a comparatively small-sized audiovisual file compressed in accordance with any one of a number of known technologies such as Joint Photographic Experts Group (JPEG) and Moving Picture Experts Group (MPEG) technologies, to name a few.
  • JPEG Joint Photographic Experts Group
  • MPEG Moving Picture Experts Group
  • the low-bandwidth media preferably includes advertising information, other types of information may be included such as a movie trailer, community service announcements, etc.
  • the network may be a global network such as the Internet, or any other type of network including a local-area network (LAN), a wide-area network (WAN), or a virtual private network, or even a combination of these implemented using conventional wire, fiber-optics, or wireless technologies.
  • LAN local-area network
  • WAN wide-area network
  • VPN virtual private network
  • Client computer 300 includes a number of application programs, a detector 301, a cache memory 303, a selector 304, and a display 305.
  • the application programs include a network browser application and streaming video player, all of which may be of conventional types.
  • the detector and selector may be implemented in software or hardware.
  • a user at the client computer issues a request for the high-bandwidth information.
  • the high- bandwidth information may be a news video, a sports video, or an instructional video, to name but a few examples.
  • server 100 constructs data stream 103 which includes the high- bandwidth information and low-bandwidth information which, for example, may be an advertisement that bears some relevance to the content of the high-bandwidth information.
  • Data stream 103 is transmitted through the network to client computer 300.
  • Detector 301 monitors the data rate of the connection from the network to the client and thus detects the rate at which the data stream is received.
  • Detector 301 also detects which portions of the data stream correspond to the high-bandwidth information and the low-bandwidth information, i.e., the streaming video or the advertising information.
  • Detector 301 does this by decoding the header information describing the data.
  • This data which is included with most standard streaming formats such as MPEG-1, MPEG-2, etc., describes details about the bit stream, including length and bit rate. For the case of alternate material, the data can be parsed as a static image.
  • GIF of JPEG images include standard header information, also. This can be used to decode the image when it has been determined by the detector that the data is not streaming video.
  • Software toolkits such as the Microsoft DirectShow Media SDK provide typical media detection algorithms to parse a media file to determine its internal format.
  • detector 301 When detector 301 detects high-bandwidth media in the data stream, the media player program begins to display this information to the user on display 305. When detector 301 detects low-bandwidth media in the data stream, the low-bandwidth information is stored in cache memory
  • a message is sent from detector 301 to selector 304 instructing the selector to display the low- bandwidth information stored in the cache memory.
  • This control program includes, or accesses, a set of rules which define when and the manner in which the low-bandwidth information is displayed when an interruption is detected. These rules, for example, may instruct selector 304 to output the low-bandwidth information from the cache memory after detector 301 detects that an interruption (e.g., an inadequate data rate) has existed for a predetermined period of time, e.g., 3 seconds.
  • the control program may then cause the low-bandwidth information to be displayed to its completion, irrespective of whether the interruption passes (i.e., the data rate returns to normal) during display of the low-bandwidth information.
  • the control program may interrupt the display of the low-bandwidth information as soon as detector 301 detects that the data rate has returned to an acceptable level, i.e., after the interruption has passed. The control program then instructs the medial player to resume display of the streaming high-bandwidth information.
  • the data stream containing the high-bandwidth and low-bandwidth information may be constructed in a variety of ways.
  • the low-bandwidth information may be sent ahead of the high-bandwidth information in the stream.
  • the low- bandwidth information may be interleaved with the high-bandwidth information according to any one of a variety of known techniques.
  • the display rules may also be included within the data stream, either at the front end between the low-bandwidth information and the high-bandwidth information as shown in Figure 2 or interleaved.
  • the display rules may be in the form of an applet.
  • the display rules may be pre-stored in the browser or media player software.
  • the invention may be modified in a number of ways.
  • client 300 can request the low-bandwidth information directly from the web server 100 as the detector monitors more interruptions of the data stream.
  • server 100 may continually integrate display rules 104 and low bandwidth media into the data stream 103.
  • multiple low-bandwidth information may be included in the data stream. If so, the cache memory would separately store each of them and then, in accordance with the control program, switch selector 304 to output a different advertisement for each interruption of the data stream. While the invention has been described in terms of preferred embodiments, those skilled in the art will recognize that the invention can be practiced with modification within the spirit and scope of the appended claims.

Landscapes

  • Engineering & Computer Science (AREA)
  • Multimedia (AREA)
  • Signal Processing (AREA)
  • Computer Networks & Wireless Communication (AREA)
  • Databases & Information Systems (AREA)
  • Computer Security & Cryptography (AREA)
  • Computer Graphics (AREA)
  • Business, Economics & Management (AREA)
  • Marketing (AREA)
  • Information Transfer Between Computers (AREA)

Abstract

Method and apparatus can deliver alternate material related to a primary media (video) stream (101) when the available bandwidth in the primary high bandwidth media streaming system such as a Web based video delivery system is limited. The alternate material, or materials, can relate to the primary stream (101) either by being related in content or being related via an advertising connection. Similarly, this singular or plurality of alternate data may be displayed (305) in either the same or alternate display locations (300) from the primary streaming media. This system is designed for broadband delivery systems, the network overhead to pre-load these low bandwidth collateral pieces becomes insignificant compared to the bandwidth requirements of the full streaming piece.

Description

Title: SCHEDULING AND DELIVERING LOW BANDWIDTH MEDIA UPON DETECTING HIGH BANDWIDTH MEDIA
DESCRIPTION
BACKGROUND OF THE INVENTION
Field of the Invention
The present invention generally relates handling network 10 bottlenecks of high bandwidth media such as streaming video, and more particularly to the scheduling and delivering of low bandwidth media to a user upon the detection of an interruption of the high bandwidth media.
Background Description
A number of technologies in existence today deliver high 15 bandwidth media such as streaming video over global networks, such as the Internet. Systems such as the one offered by Real Networks or Microsoft streaming media player can deliver and play video at rates in excess of 400 kilobits per second (Kb/sec). For the purposes of this disclosure, high bandwidth systems are defined as those capable of 20 delivering data at a rate above that available with traditional telephony based modem technology, which at the time of this disclosure is 56Kb/sec. For instance, one embodiment of a high bandwidth system is a cable modem, which has a data transfer rate capability in excess of 1 megabytes per second (MB/sec.) or greater.
A number of transmission technologies also exist that are intended to provide adequate bandwidth so that data streams containing video or audio can be delivered to a client (e.g., a software browser) in a continuous, uninterrupted format. Unfortunately, because of the complexity of the World Wide Web ("the Web") portion of the Internet and the myriad of systems necessary to deliver data, it is inevitable that an interruption will occur at some point during transmission of data. This can be frustrating to the user and may result in the user going to another Web site.
SUMMARY OF THE INVENTION
It is therefore an object of the present invention to provide a way to deliver alternate lower bandwidth material when an interruption to service of high bandwidth material occurs.
It is another object to provide alternate viewing material to a user who is experiencing delays or interruptions of service to a high bandwidth video presentation over a network as well as taking the display delay as an opportunity to display advertising or other related information to the user which will hold the user's interest. The network may be a local area network, a wide-area network, or a global network such as the Internet.
According to the invention, advantage is taken of the fact that during normal transmission of the high bandwidth material, regions may be reserved for the transmission of alternate material. One example of alternate material is a Graphics Interface Format (GIF) image, typically used for advertising in such application as "banners". Static GIF images can be made such that their total size is around 10KB. It can be seen how the download requirements for such a banner are trivial compared to the total bandwidth of a streaming video of 400Kb/sec. In this example, an image could be downloaded in 0.25 second before video (or other media) begins.
An embodiment of the invention can load and store this collateral image or images for display upon the interruption of the primary stream.
Additionally, rules can be delivered to determine the location of this collateral information within the user screen. For instance, if a set of user screens included media sections (for video) as well as several ancillary locations for other material such as advertisements, specific static graphical material could be pre-loaded for possible display in the one region of the screen if an interruption to the high bandwidth media is detected.
The advertising or related material can be pre-loaded to the client before the streaming media begins, or can be interleaved with the data. The rules as to when this media can be displayed could be delivered at the same time this alternate material is delivered. The rules for display of alternate material can address a number of issues including thresholds for when to display, the length of display, location on the page to display, etc.
BRIEF DESCRIPTION OF THE DRAWINGS
The foregoing and other objects, aspects and advantages will be better understood from the following detailed description of a preferred embodiment of the invention with reference to the drawings, in which: Figure 1 is a block and flow diagram showing how a server may deliver a combination of the high- and low-bandwidth information and a set of display rules to a client that can detect system bandwidth and display either the high- or low-bandwidth information; and
Figure 2 is a block diagram showing an manner in which a data stream may be constructed in accordance with the present invention. DETAILED DESCRIPTION OF A PREFERRED EMBODIMENT OF THE INVENTION
Referring to the drawings, Figure 1 shows the manner in which a data stream is constructed in accordance with the present invention. As shown, a server 100 combines at least two types of information. into the data stream. First information 101 is high-bandwidth information which may correspond to streaming audio or streaming video which may or may not include accompanying audio. Second information 102 is low- bandwidth information such as a GIF, an animated GIF, or even a comparatively small-sized audiovisual file compressed in accordance with any one of a number of known technologies such as Joint Photographic Experts Group (JPEG) and Moving Picture Experts Group (MPEG) technologies, to name a few. While the low-bandwidth media preferably includes advertising information, other types of information may be included such as a movie trailer, community service announcements, etc.
Once constructed, data stream 103 is transmitted over a network 200 to a client computer 300. The network may be a global network such as the Internet, or any other type of network including a local-area network (LAN), a wide-area network (WAN), or a virtual private network, or even a combination of these implemented using conventional wire, fiber-optics, or wireless technologies.
Client computer 300 includes a number of application programs, a detector 301, a cache memory 303, a selector 304, and a display 305. The application programs include a network browser application and streaming video player, all of which may be of conventional types. The detector and selector may be implemented in software or hardware.
In operation, a user at the client computer issues a request for the high-bandwidth information. For illustrative purposes only, the high- bandwidth information may be a news video, a sports video, or an instructional video, to name but a few examples. Once this request is received, server 100 constructs data stream 103 which includes the high- bandwidth information and low-bandwidth information which, for example, may be an advertisement that bears some relevance to the content of the high-bandwidth information.
Data stream 103 is transmitted through the network to client computer 300. Detector 301 monitors the data rate of the connection from the network to the client and thus detects the rate at which the data stream is received. Detector 301 also detects which portions of the data stream correspond to the high-bandwidth information and the low-bandwidth information, i.e., the streaming video or the advertising information. Detector 301 does this by decoding the header information describing the data. This data, which is included with most standard streaming formats such as MPEG-1, MPEG-2, etc., describes details about the bit stream, including length and bit rate. For the case of alternate material, the data can be parsed as a static image. GIF of JPEG images include standard header information, also. This can be used to decode the image when it has been determined by the detector that the data is not streaming video. Software toolkits such as the Microsoft DirectShow Media SDK provide typical media detection algorithms to parse a media file to determine its internal format.
When detector 301 detects high-bandwidth media in the data stream, the media player program begins to display this information to the user on display 305. When detector 301 detects low-bandwidth media in the data stream, the low-bandwidth information is stored in cache memory
303. Then, when an interruption is detected, a message is sent from detector 301 to selector 304 instructing the selector to display the low- bandwidth information stored in the cache memory. Stored within the media player or network browser is a control program for controlling the operations of the client relative to the display of the streaming video and advertising information. This control program includes, or accesses, a set of rules which define when and the manner in which the low-bandwidth information is displayed when an interruption is detected. These rules, for example, may instruct selector 304 to output the low-bandwidth information from the cache memory after detector 301 detects that an interruption (e.g., an inadequate data rate) has existed for a predetermined period of time, e.g., 3 seconds. After this time lapses, the control program may then cause the low-bandwidth information to be displayed to its completion, irrespective of whether the interruption passes (i.e., the data rate returns to normal) during display of the low-bandwidth information. Alternatively, the control program may interrupt the display of the low-bandwidth information as soon as detector 301 detects that the data rate has returned to an acceptable level, i.e., after the interruption has passed. The control program then instructs the medial player to resume display of the streaming high-bandwidth information.
The data stream containing the high-bandwidth and low-bandwidth information may be constructed in a variety of ways. For example, as shown in Figure 2, the low-bandwidth information may be sent ahead of the high-bandwidth information in the stream. Alternatively, the low- bandwidth information may be interleaved with the high-bandwidth information according to any one of a variety of known techniques. If desired, the display rules may also be included within the data stream, either at the front end between the low-bandwidth information and the high-bandwidth information as shown in Figure 2 or interleaved. If desired, the display rules may be in the form of an applet. Optionally, the display rules may be pre-stored in the browser or media player software. The invention may be modified in a number of ways. For example, client 300 can request the low-bandwidth information directly from the web server 100 as the detector monitors more interruptions of the data stream. Also, server 100 may continually integrate display rules 104 and low bandwidth media into the data stream 103.
If desired, multiple low-bandwidth information may be included in the data stream. If so, the cache memory would separately store each of them and then, in accordance with the control program, switch selector 304 to output a different advertisement for each interruption of the data stream. While the invention has been described in terms of preferred embodiments, those skilled in the art will recognize that the invention can be practiced with modification within the spirit and scope of the appended claims.

Claims

Having thus described our invention, what we claim as new and desire to secure by Letters Patent is as follows:
L A method for scheduling and delivering low bandwidth media objects on reduction in available bandwidth for delivering high bandwidth media comprising the steps of: receiving a request from a client through a network for streaming of a selected high bandwidth media object; associating at least one low bandwidth media object and display rules with the selected high bandwidth media object; and streaming the selected high bandwidth media object with the associated low bandwidth media" object and display rules to the requesting client.
2. The method according to claim 1, further comprising the steps of: detecting a low bandwidth media object at the client and upon detecting a low bandwidth media object, storing the low bandwidth media object in a cache memory; and detecting an interruption in delivery of the selected high bandwidth media object at the client and upon detecting an interruption in delivery of the selected high bandwidth media object, displaying the low bandwidth media object stored in the cache memory.
3. A method for aggregating information for display, comprising the steps of: transmitting a data stream over a network to a client computer, said data stream including first media information and second media information; displaying on a screen of said client computer at least a portion of said first media information as said client computer receives said data stream over the network; detecting an interruption in said first media information; and displaying said second media information during the interruption in said first media information.
4. The method according to claim 3, further comprising the steps of: detecting that the interruption has passed; and resuming display of said first media information.
5. The method according to claim 3, wherein said data stream includes rules that instruct a media player application stored in the client computer to display said second media information upon detection of an interruption in said data stream.
6. The method according to claim 3, further comprising the step of storing rules within a media player application program that instruct the media • player to display said second media information upon detection of an interruption in said data stream.
7. The method according to claim 3, wherein said step of displaying said second media information includes: determining whether a first predetermined period of time has passed since detection of an interruption in said first media information; and displaying said second media information after said first predetermined period of time has passed.
8. The method according to claim 7, wherein said second media information is displayed to completion irrespective of whether the interruption in said first media information passes during display of said second media information.
9. The method according to claim 7, further comprising the steps of: terminating display of said second media information immediately upon detection that the interruption in said first media information has passed; and resuming display of said first media information.
10. The method according to claim 3, wherein said first media information includes one of streaming audio, streaming video, and streaming audio/visual information.
11. The method according to claim 10, wherein said second media information includes advertising information.
12. The method according to claim 10, wherein said advertising information is at least one of a Graphic Interface Format (GIF) image, an animated GIF image, a Joint Photographic Experts Group (JPEG) image, audio information, or compressed video.
13. The method according to claim 11 , further comprising the step of transmitting said advertising information ahead of said first media information in said data stream.
14. The method according to claim 11, further comprising the step of interleaving said advertising information with said first media information in said data stream.
15. The method according to claim 3, wherein said network is one of a local network, a wide-area network, and a global network.
16. The method according to claim 3, wherein said step of transmitting the data stream is performed in response to a user request for said first media information.
17. A system for scheduling and delivering low bandwidth media objects on reduction in available bandwidth, for delivering high bandwidth media, said system comprising: a network; a client computer connected to said network; a browser program, in said client computer, which transmits a request through the network for streaming of a selected high bandwidth media object; and a server which receives said request through said network, said server associating at least one low bandwidth media object and display rules with the selected high bandwidth media object, and streaming the selected high bandwidth media object with the associated low bandwidth media object and display rules to the client computer.
18. The system according to claim 17, wherein the display rules are installed on the client computer, and wherein said client computer further includes: a cache memory; means for detecting a low bandwidth media object and upon detecting a low bandwidth media object, storing the low bandwidth media object in the cache memory; and means for detecting an interruption in delivery of the selected high bandwidth media object and upon detecting an interruption in delivery of the selected high bandwidth media object, displaying the low bandwidth media object stored in the cache memory.
19. A system for aggregating information for display, comprising: a network; a client computer connected to the network, said client computer transmitting a request for information; a server which transmits, in response to said request, a data stream over the network to the client computer, said data stream including first media information and second media information, said client computer including: . (a) a media player program for displaying at least a portion of said first media information as said client computer receives said data stream over the network, and (b) means for detecting an interruption in said data stream, said medial player program displaying said second media information during the interruption.
20. The system according to claim 19, wherein said media player program resumes display of said first media i iformation upon detection by said detecting means that the interruption has passed.
21. The method according to claim 19, wherein said server transmits rules in said data stream that instructs said media player program to display said second media information upon detection of an interruption in said data stream by said detecting means.
22. The system according to claim 19, wherein said first media information includes one of streaming audio, streaming video, and streaming audio/visual information.
23. The system according to claim 19, wherein said second media information includes advertising information.
24. The system according to claim 23, wherein said advertising information is transmitted ahead of said first media information in said data stream.
25. The system according to claim 23, wherein said advertising information is interleaved with said first media information in said data stream.
26. The system according to claim 19, wherein said network is one of a local network, a wide-area network, and a global network.
PCT/US2001/040685 2000-05-08 2001-05-08 Scheduling and delivering low bandwidth media upon detecting high bandwidth media WO2001086456A1 (en)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
US56681900A 2000-05-08 2000-05-08
US09/566,819 2000-05-08

Publications (1)

Publication Number Publication Date
WO2001086456A1 true WO2001086456A1 (en) 2001-11-15

Family

ID=24264501

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/US2001/040685 WO2001086456A1 (en) 2000-05-08 2001-05-08 Scheduling and delivering low bandwidth media upon detecting high bandwidth media

Country Status (1)

Country Link
WO (1) WO2001086456A1 (en)

Cited By (8)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
EP1505836A1 (en) * 2003-08-05 2005-02-09 Samsung Electronics Co., Ltd. Apparatus and method for encoding/decoding broadcasted or recorded contents
EP1561171A2 (en) * 2002-11-12 2005-08-10 Claria Corporation System and method for delivery of information based on web page content
EP1777917A1 (en) * 2005-10-21 2007-04-25 LG Electronics Inc. Method and mobile communication terminal for providing contents
WO2009038828A3 (en) * 2007-05-31 2009-10-22 Realnetworks, Inc. Web media asset identification system and method
US20100153858A1 (en) * 2008-12-11 2010-06-17 Paul Gausman Uniform virtual environments
WO2012114140A1 (en) * 2011-02-22 2012-08-30 Telefonaktiebolaget L M Ericsson (Publ) Media switching unit and method
WO2013034944A1 (en) * 2011-09-06 2013-03-14 Telefonaktiebolaget L M Ericsson (Publ) Device and method for progressive media download with multiple layers or streams
US20140289764A1 (en) * 2010-10-21 2014-09-25 Sling Media Pvt Ltd. Handling disruption in content streams received at a player from a content retransmitter

Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US5898833A (en) * 1996-12-31 1999-04-27 Intel Corporation Method and apparatus for increasing the effective bandwidth of video sequences transmitted over a network by using cached data
US5968119A (en) * 1996-12-09 1999-10-19 Wall Data Incorporated Method of accessing information of an SNA host computer from a client computer using a specific terminal emulation
US5978381A (en) * 1997-06-06 1999-11-02 Webtv Networks, Inc. Transmitting high bandwidth network content on a low bandwidth communications channel during off peak hours

Patent Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US5968119A (en) * 1996-12-09 1999-10-19 Wall Data Incorporated Method of accessing information of an SNA host computer from a client computer using a specific terminal emulation
US5898833A (en) * 1996-12-31 1999-04-27 Intel Corporation Method and apparatus for increasing the effective bandwidth of video sequences transmitted over a network by using cached data
US5978381A (en) * 1997-06-06 1999-11-02 Webtv Networks, Inc. Transmitting high bandwidth network content on a low bandwidth communications channel during off peak hours

Cited By (18)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
EP1561171A2 (en) * 2002-11-12 2005-08-10 Claria Corporation System and method for delivery of information based on web page content
EP1561171A4 (en) * 2002-11-12 2006-02-01 Claria Corp System and method for delivery of information based on web page content
CN100379291C (en) * 2003-08-05 2008-04-02 三星电子株式会社 Apparatus and method for encoding/decoding broadcast/spare contents
EP1505836A1 (en) * 2003-08-05 2005-02-09 Samsung Electronics Co., Ltd. Apparatus and method for encoding/decoding broadcasted or recorded contents
US8073437B2 (en) 2005-10-21 2011-12-06 Lg Electronics Inc. Mobile communication terminal for providing contents and method thereof
EP1777917A1 (en) * 2005-10-21 2007-04-25 LG Electronics Inc. Method and mobile communication terminal for providing contents
KR101200475B1 (en) 2007-05-31 2012-11-13 리얼네트웍스 인코포레이티드 Web media asset identification system and method
US7890854B2 (en) 2007-05-31 2011-02-15 Realnetworks, Inc. Web media asset identification system and method
WO2009038828A3 (en) * 2007-05-31 2009-10-22 Realnetworks, Inc. Web media asset identification system and method
US20100153858A1 (en) * 2008-12-11 2010-06-17 Paul Gausman Uniform virtual environments
US20140289764A1 (en) * 2010-10-21 2014-09-25 Sling Media Pvt Ltd. Handling disruption in content streams received at a player from a content retransmitter
US9819998B2 (en) * 2010-10-21 2017-11-14 Sling Media Pvt Ltd Handling disruption in content streams received at a player from a content retransmitter
US10219038B2 (en) 2010-10-21 2019-02-26 Sling Media Pvt Ltd Handling disruption in content streams received at a player from a content retransmitter
CN103380610A (en) * 2011-02-22 2013-10-30 瑞典爱立信有限公司 Media switching unit and method
US20130340012A1 (en) * 2011-02-22 2013-12-19 Andreas Johnsson Media Switching Unit and Method
WO2012114140A1 (en) * 2011-02-22 2012-08-30 Telefonaktiebolaget L M Ericsson (Publ) Media switching unit and method
WO2013034944A1 (en) * 2011-09-06 2013-03-14 Telefonaktiebolaget L M Ericsson (Publ) Device and method for progressive media download with multiple layers or streams
CN103765856B (en) * 2011-09-06 2017-12-15 瑞典爱立信有限公司 The apparatus and method downloaded for the progressive media with multilayer or multiple streams

Similar Documents

Publication Publication Date Title
US20020077900A1 (en) Internet protocol-based interstitial advertising
US6094677A (en) Methods, systems and computer program products for providing insertions during delays in interactive systems
US6338094B1 (en) Method, device and system for playing a video file in response to selecting a web page link
US7203758B2 (en) System and method for selective insertion of content into streaming media
US9477754B2 (en) One click full-screen video system and method
US20040133467A1 (en) Method and apparatus for selecting streaming media in real-time
US20040015986A1 (en) Content/service handling and delivery
EP1667454A1 (en) Content distribution system, content distribution method, content distribution server, content reproduction apparatus, content distribution program, and content reproduction program
CA2364733A1 (en) System and method for interactive distribution of selectable presentations
AU2001251215B2 (en) Insertion of asynchronous data into a synchronous stream
KR20010050678A (en) Method and apparatus for dynamic distribution of controlled and additional selective overlays in a streaming media
EP1166561B1 (en) Selectively caching video to improve on-demand response time
AU2001251215A1 (en) Insertion of asynchronous data into a synchronous stream
JP2005191949A (en) Video image distribution apparatus and video image browsing apparatus
GB2397991A (en) Apparatus, method and a computer readable medium for providing interactive services
WO2001086456A1 (en) Scheduling and delivering low bandwidth media upon detecting high bandwidth media
US7093275B2 (en) Synchronizing services across open cable and packet cable
US20020067907A1 (en) Universal video client/universal video server system
US20090064257A1 (en) Compact graphics for limited resolution display devices
JP2005328269A (en) Client terminal, streaming server, and streaming-switching distribution system
US9166861B2 (en) Method for managing communication channels, corresponding signal and terminal
KR20020000985A (en) Internet service server system, computer system and advertising method thereof
CA2291269A1 (en) Method and apparatus for gap coverage in streaming protocols
KR100434539B1 (en) Interactive moving picture advertisement method using scalability and apparatus thereof
KR20020010285A (en) Advertizing method by Video On Demand Partition

Legal Events

Date Code Title Description
AK Designated states

Kind code of ref document: A1

Designated state(s): CA CN JP KR

AL Designated countries for regional patents

Kind code of ref document: A1

Designated state(s): AT BE CH CY DE DK ES FI FR GB GR IE IT LU MC NL PT SE TR

121 Ep: the epo has been informed by wipo that ep was designated in this application
32PN Ep: public notification in the ep bulletin as address of the adressee cannot be established

Free format text: COMMUNICATION PURSUANT TO RULE 69 EPC (EPO FORM 1205A OF 120303)

122 Ep: pct application non-entry in european phase
NENP Non-entry into the national phase

Ref country code: JP