Method and Apparatus for Integrating Animation in Interactive Video
Field of the Invention
The present invention is in the field of interactive video, and pertains more particularly to apparatus and methods for authoring and enabling animation in interactive video user interfaces.
Background of the Invention
With continuing development of new and better ways of delivering television and other video presentations to end users coupled with parallel development of computerized information systems such as the Internet, there have been concerted efforts to integrate various systems to provide enhanced information delivery and entertainment systems. For example, developers are introducing integrated systems combining TVs with computer subsystems, so a TV may be used as a WEB browser, or a PC may be used for enhanced TV viewing.
In some systems, computer elements such as a CPU, memory, and the like, are built into the familiar chassis of a TV set. In such a system, the TV screen becomes the display monitor in the computer mode. In such a system, conventional TV elements and circuitry are incorporated along with the computer elements, and capability is provided for a user to switch modes, or to view recorded or broadcast video with added computer interaction. One may thus, with a properly equipped system, select to view analog TV programs, digital TV programs, conventional cable TV, satellite TV, pay TV from various sources, video delivered on the Internet, and to browse the WWW as well, displaying WEB pages and interacting with on-screen fields and relational systems for jumping to related information,
databases, and other WEB pages The capabilities are often integrated into a single display, that is, one may view a broadcast presentation and also have a window on the display for WEB interaction
In some other systems, computer elements are provided in an enclosure separate from the TV, often referred to in the art as a set-top box Set-top box systems have an advantage for providers in that they may be connected to conventional television sets, so end users don't have to buy a new TV along with the computer elements
In such integrated systems, whether in a single enclosure or as set-top box systems, user mput is typically through a hand-held device quite similar to a familiar remote controller, usually having infra-red communication with the set-top box or a receiver in the integrated TV For computer modes, such as WEB browsing, a cursor is displayed on the TV screen, and cursor manipulation is provided by buttons or other familiar pointer apparatus on the remote Select buttons are also provided m the remote to perform the familiar function of such buttons on a pointer device, like a mouse or trackball more familiar to computer users
Set-top boxes and computer-integrated TVs adapted as descπbed above typically have inputs for such as a TV antenna (analog), cable TV (analog or digital), more recently direct-satellite TV (digital), to Internet ports, and may also connect to video cassette recorders and to mass storage devices such as hard disk dπves and CD-ROM dπves to provide a capability for uploading video data from such devices and presenting the dynamic result as a display on the TV screen
The present mventors have noted that with the coupling of computer technology with TV, many capabilities familiar to computer users have been made available to TV users For example, ability to provide text annotation for TV presentations is considerably enhanced Computer techniques such a Pix-on-Pix are now available, wherein separate TV presentations may be made in separate windows, or overlaid windows on the display screen Separate windows may also support display from separate sources, such as an analog TV program in one window, a computer game in another, and a video conference in a third
In typical prior art video authoring systems, end users receive a single video stream that contains the video data and any added annotated data such as subtitling, sponsor logos, information blocks, and the like. However, it is desirable to build upon the goal stated in the preceding paragraph above, by having separate streams, one containing video data, and the other containing annotative data, that may arrive at and end user's location via different delivery media and be displayed synchronously on a suitable display screen.
In a system known to the inventors, hyper-video authoring is possible wherein a wide variety of added, often interactive, content is made available to viewers having an active connection to a video feed and an Internet connection for facilitating delivery of interactive results. The system known to the inventor uses separate authoring stations or systems for use in what is termed hyper-video authoring by the inventor. Hyper- video authoring includes creating hot spots (interactive hyperlinks) in a video, providing interactive regions for tracked objects (methods known to the inventor), inserting URL's for user interaction, providing interactive banner ads, providing interactive icons for launching instant replays of short video clips, adding audio/text annotation, and so on.
Annotated content may, in some instances, be delivered via alternate paths (typically the Internet) and synchronized with a main video stream being viewed at the user's end. In other instances, added content is combined at the provider end and delivered as one stream (typically cable or digital feed) to the viewer. As a viewer interacts with provided content, Internet capability is utilized for the purpose of delivering stored information associated with the viewers interaction that may be viewed on the same display containing the offered video such as by a PIX on PIX method and so on. In some cases users, if so equipped, may elect to save and store Internet delivered content to suitable data storage apparatus. In other aspects, such material may be saved and stored on-line at a server adapted for the purpose. In this case, a viewer may at a later time retrieve the material at his or her discretion.
One of the capabilities of the above-described system is the ability to provide tracking coordinates for an object in the video. By tracking the movement of an image in a video presentation, an interactive region or hot spot may be embedded and caused
to follow a target image in the video presentation by virtue of the tracking-coordinate data included in an authored steam or embedded within a main video stream The interactive region may be of the form of a halo surrounding an image, a semi- transparent area covering an Image, a portion of the image that has been altered m some way to indicate that it is interactive, and so on
By clicking on the interactive portion or region associated with an image entity, a viewer may obtain, by virtue of Internet or WAN connection, additional information about the image Such information may be delivered to a viewer in the form of a WEB page, an order form, text information, additional interactive links, or other conventions designed to further engage the viewer regarding the selected image The authoring consists of identifying, creating, and embedding the tracking coordinates (path of the image) into the video data so that a subsequent author may associate a hyper-hnk or interactive region with the moving image
In the present invention a similar capability to the tracking and selection of image entities as hyperlinks is provided, but it not necessary that the hyperlinks follow moving entities in a video presentation This new capability provides new animated entities in the displayed video with new and non-obvious functions
It is known in the art that ammated graphics files are used by artists to create and display with animation on such as Internet Web pages One format used for this purpose is called Graphics Interchange Format (GIF) GIF files are just one of several formats that may be used for animation, and are referred to here as representative and exemplary of all such formats An animated GIF or other animated graphic is typically embedded in a web page An animated graphic may play according to instruction when the web page is loaded, and others may have interactive links or animated graphics on user interaction like mouse over or mouse click Such animations may run in a same physical location, or may be programmed to move across a screen or display either on loading or on user interaction A GIF typically has an 8-bit construction and an excellent compression method termed LZW This makes it easy to transmit GLFs over the bandwidth-limited Internet and to display them on a suitable screen without
using excessive bandwidth or memory Other suitable graphic file formats other than the GIF format may also be used for such interactive content
Providing an executable animated graphic in a manner to be integrated with a primary video stream would allow authors to provide interactive graphic content on top of the video m an entertaining form without being limited to providing interaction capability to an image already in a main video Such interactive overlaid content may be used for interactive advertisements
What is clearly needed is a method and apparatus for embedding and interactively linking animated graphics into a main video or authored control stream such that the graphics would change or move on the screen or appear and disappear within the video automatically and/or by virtue of user participation such as mouse or keyboard interaction Such a method and apparatus would expand prospects for interactive advertising and other information display
Summary of the Invention
In a preferred embodiment of the present invention a video system is provided comprising a content provider streaming a primary video stream and an annotation data stream, and a viewing station having a video display apparatus for displaying both the primary video stream and the annotation data stream The annotation data stream comprises one or more animated graphics in a format such as the Graphics Interchange Format (GIF) which is synchronized and displayed on the video In some embodiments the animated graphic moves in one or both of vertical or horizontal direction, or any other trajectories in the display, directed by data in the annotation data stream, and may be associated with an image entity displayed from the primary video stream, moving with the image entity
The viewing station may further comprise a user input device for cursor control such as a mouse, and the animated graphic may be displayed as a hyperlink User interaction with the animated graphic may either send a back-link signal to the content
provider or may execute some other action as per information in the annotation data stream.
If an action is specified in the annotation data stream in response to user interaction - a back link signal may or may not be used. Such actions on user interaction specified in the primary annotation data stream could be displaying an alternative graphic content, changing the motion of the graphic content, making the graphic content appear or disappear from the video or display other specific information in text or graphic format. Such content may be used for interactive gaming kind of applications to invite more participation from the user while he is watching the video. The advantage of specifying data about the action to be performed on user interaction in the primary annotation data stream is that a back-link signal is not required and the information about response to user interaction can be sent simultaneously to all users without requiring a separate communication channel for each user. If the user interaction is sent to the content provider as a back-link signal, the content provider may, as a response, stream an alternative annotation stream, providing an alternative display on the video display apparatus at the viewing station. In this case the alternative annotation stream which contains the response to the user interaction is sent specifically to users who interact with the content. In some cases the viewing station comprises a first high-bandwidth dedicated connection for receiving the primary video stream, and an Internet connection for receiving the annotation video stream, and the content provider comprises a first facility for streaming the primary video stream via the first connection, and an Internet- connected server for streaming the annotation video stream via the Internet. In these systems the annotation data stream first provides a first displayable indicia as a hyperlink, and in response to a backlink signal to the Internet-connected server initiated by user interaction with the first displayable indicia, the server provides additional information about the action to be performed such as displaying a web page or displaying other graphics on the video.
In some embodiments the first displayable indicia responds to a user interaction such as a mouse-over to display other animated graphics which in turn respond to user interaction to display other graphics or textual content or send a second backlink signal. In some of these embodiments the server responds to the second backlink signal by sending a third displayable indicia in the annotation data stream.
In another aspect of the invention an interactive video system is provided, comprising a computerized display system comprising a video display apparatus, receiving apparatus for receiving and displaying both a primary video stream and an annotation stream on the display apparatus, a user input apparatus for a user to interact with display entities, and a mechanism for sending backlink signals from the computerized display system; and an Internet-connected server for providing at least the annotation data stream and connected to the computerized display system in a manner to receive the backlink signals. The annotation data stream provides a first displayable indicia as a hyperlink, and the server responds to a backlink signal resulting from user interaction with the first displayable indicia by providing in the annotation data stream a second displayable indicia.
In embodiments from this the second displayable indicia may also be a hyperlink. Also in these embodiments the first displayable hyperlink may send a first backlink signal as a result of a mouse-over, and the second display able indicia may then send a second backlink signal as a result of a mouse click. The second displayable indicia may be a company logo, which may display for a pre-determined time period and then disappear, and the annotation data stream from the server then provides a third displayable indicia. One of the first and second displayable indicia may comprise an animated Graphics Image Format (GLF) display. In this case the second GIF may persist in the display for a pre-determined time period, then disappear, and the annotation data stream from the server then provides a third displayable indicia.
In these embodiments the first and second displayable indicia may move in one or both of vertical and horizontal directions in the display, directed by data in the annotation data stream, and may also be associated with an image entity in the primary video stream, and move with the image entity.
In addition to the apparatus of the invention in multiple embodiments, as described generally above, methods for practicing the invention using the apparatus are taught as well, all in enabling detail below The functionality taught for video display systems according to embodiments of the invention provides new ways of advertising, and new ways of presenting information to viewers with viewer participation in interactive video systems, and also provides new ways of doing an e-commerce transaction by introducing interactive advertising
Brief Description of the Drawing Figures
Fig 1 is an overview diagram of a video authoring and delivery system in an embodiment of the present invention
Fig 2 is a block diagram illustrating components of the authoπng station of
Fig 3 A is a plan view of an interactive PC/TV illustrating display and viewer interaction with a first animated graphic according to an embodiment of the present invention
Fig 3B is a plan view of the PC/TV of Fig 3 A illustrating a second animated graphic invoked in Fig 4A
Fig 3C is a plan view of the PC/TV of Fig 3 A illustrating viewer interaction with the second animated graphic of Fig 3B according to an embodiment of the present invention
Fig 3D is a plan view of the PC/TV of Fig 3 A illustrating a third animated graphic invoked in Fig 3C
Description of the Preferred Embodiments
It is an object of the present invention to provide a method and apparatus that enables an author annotating a primary video stream, which may be either digital or analog, to create and provide interactive animated graphics in video presentations. In embodiments of the invention animated graphics, for example graphics in a GLF format, may be used in concert with hyperlinks of various sorts to provide activity sequences in an interactive video displays. The method and apparatus is described in enabling detail below. Fig. 1 is a block diagram illustrating an exemplary overview of a video authoring and delivery system 9 as known to the Inventor and suitable for practicing the present invention. System 9 is exemplary of a system known to the inventor for receiving, authoring and sending video content to viewers who subscribe to a service defined by this capability. Exemplary components of system 9 include a broadcast system 11, a video head end 13, an authoring station 15, a wide-area-network (WAN) 19 having an Internet Service Provide (ISP) including modems 24 and a server 26, and a plurality of subscribed viewers 17a through 17n.
Video head end 13 is defined as a source for live or pre-stored video feeds for the purpose of video authoring before delivery to viewers such as viewers 17a-n. Video head-end 13 delivers video feed 21 , which, in this embodiment, is diverted to authoring station 15 for the purpose of providing annotative and interactive content that will ultimately be displayed on equipment maintained by viewers 17a-n. Such content will either be combined or muxed into feed 21 and be displayed from one stream delivered to viewers, or may be delivered as a separate stream to be combined or caused to display in synchronous mode on a viewer's system.
Feed 21 may be either an analog feed or a digital feed. If feed 21 is analog, it is converted to digital format for the purpose of authoring. Feed 21 may continue on to viewers as an analog feed. In another embodiment, feed 21 may be converted to digital format and remain as such for delivery to viewers 17a-n. Such variations in video format, delivery methods of feed 21, and delivery methods of authored content
will depend, in part, on system architecture and customer equipment capability, and in part on the nature of added content and the intent of providers There are many possible variations
In this example, broadcast system 11 is capable of video delivery in digital or analog format over a cable connection, a satellite connection, or by any other high- bandwidth dedicated medium The connections illustrated between viewers 17a-n and broadcast station 11 illustrate logical connections only As previously described, video feed 21 is diverted in this instance, to station 15 for authoring
Authoring station 15 is enhanced with a software application 22 which enables an author to provide video annotation to the primary video stream The video annotation provided may be of a variety of sorts as described m the related prior application under the heading above "Cross-reference to related Documents" Such annotation includes, for example, associating a moving image entity (for example, a specific player in a sports event) in a primary video with a hyperlink by tracking the image and including coordinates of the image entity m the annotation data
Displayable graphics, such as icons, may be annotated to primary video streams and be displayed either as fixed icons or moving with tracked entities, and such icons may be rendered as hyperlinks, such that selecting an icon initiates a backlink signal to a server, just as a hyperlink works in a WEB page In some cases in the present invention animated GIFs are provided by an author through software 22 to be integrated with a primary video stream and used in ways more fully described below Such animated GLFs may be created at the authoring station or retrieved from another source, inserted as metadata in a pπmary stream or in a separate stream to be later coordinated with the primary stream, and displayed as hyperlinks at a user's facility
In the case of a pre-stored analog video, authoring may, of course, be performed off-line at the authors convenience In this case, created content including animated GIFs may be timed to stream at the same time of the main video, unless, of course, the content is made part of the mam video by virtue of VBI or HBI inserted metadata In some cases authored content may be delivered ahead of a pπmary video
and later coordinated (synchronized) with the video All analog video is converted to digital format at station 15 before authoring Separate control streams remain digital and must be synchronized either at station 15, or at the viewer's end before successful display of an annotated presentation Because VBIs and HBIs in analog video streams are limited in capacity for inserted data, and may be overwritten by downstream processes, it is preferred that metadata including any animated GIFs be delivered in a separate digital control stream to be synchronized to the associated primary video at authoring station 15, at station 11 or at each viewer's location A feed 21a is illustrated as output from authoring station 15 Feed 21a represents the analog or digital video feed 21 after leaving station 15 A control stream 23, illustrated as output from station 15, represents a combined control stream containing any GIF metadata and, perhaps other forms of annotation metadata inserted by one or a plurality of authors A control stream 23 a represents stream 23 being integrated with main feed 21 for combining or synchromzation at station 11 This operation may alternatively be performed within station 15, in which case, only one feed would be output to station 11 Control stream 23 may also (and preferably) be delivered through WAN 19 to viewers by virtue of separate WAN connections maintained by viewers 17a -17n WAN 19 may, in preferred embodiments, be the Internet
WAN 19 is shown, as briefly described above, having a representative server 26 and modem bank 24 constituting an Internet Service Provider (ISP) accessed by users 17a-n through a public-switched telephone network represented by line 23 In practice separate users may have separate ISPs which may each access server 26 In embodiments of the invention software 28 at server 26 in conjunction with stored or otherwise accessible content provides displayable entities, including ammated GLF entities, to users 17 a-n in response to backlink signals as a result of users interacting with hyperlinks provided by metadata integrated with or coordinated with a primary video stream displayed at a user's station
System 9 as illustrated herein may deliver analog or digital video content to viewer's 17a-n along with separate digital control streams that may arrive via alternate paths System 9 may be altered with respect to architecture to facilitate many alternative embodiments as long as the main or basic components exist without departing from the spirit and scope of the present invention
Fig 2 is a block diagram illustrating components of authoring station 15 of Fig 1 according to an embodiment of the present invention Station 15 may have a plurality of authors 1 -4 and an authoring server 27 Authors 1 -4 are preferably operating separate computerized control stations There may be more or fewer authors operating at station 15 without departing from the spirit and scope of the present invention The inventor deems these four as adequate for the purpose of illustrating the present invention Some of authors 1-4 may be automated stations and unmanned by live persons whereas persons assigned to specific authoring assignments may man some other stations This example supports real-time authoring as demonstrated by the architecture of authors 1-4 Video stream 21 is fed simultaneously in to each authoring station 1-4 If feed 21 is analog, it is converted to digital format either before arriving to, or at the location of each station 1-4 Each author 1-4 outputs a control stream 25a- 25d respectively Each control stream 25a-d contains metadata particular to a particular authoring assignment For example, stream 25a contains animated GIF metadata
Streams 25b-d may contain other types of authored metadata In this example, all the control streams are combined in an authoring server 27 and are output as one control stream 23, which is analogous to stream 23 of Fig 1
Stream 21 re-merges after exiting control stations 1-4 (authors) In some embodiments, stream 21 is passed through server 27 for the purposed of synchromzing it with the combined control stream 23, however, this is not required to practice the present invention, as other methods for stream synchronization are known to the inventor and may be practiced at the provider end or at the viewers end
In this case, main video 21 and control stream 23 are separate but synchronized streams as they exit authoring station 15 In alternative embodiments, as previously
descπbed, there may be only one combined stream leaving station 15 The ammated GIF metadata is included in stream 23 with other annotation
In this embodiment, authoring station 1 creates and prepares annotation that will be associated with animated GIFs If the authoring at station 1 includes ammated GIFs then the author retrieves such GIFs from a library or other content provider, and inserts these as metadata into control stream 25a in real time In many cases this first annotation will be a transparent hyperlink to a moving image entity in the primary video, as described above, or a static, displayable graphic, or icon The initial authoring also includes adding interactive metadata, which provides hyperlink capability according to viewer initiation such as a mouse-over or a click or doubleclick in a viewer's interface, described in further detail below In many cases of the present invention, viewer interaction with a first hyperlink provided by the authoring capability described provides a second display, which may be an ammated GLF, connection to additional information, or even further changes in display and action In the case of such further display resulting from an initial interaction, the further metadata annotation may provided by server 26 (Fig 1) for example, in response to a backlink signal resulting from a user interacting with a hyperlink
User interaction with hyperlinks whether or not the hyperlinks are associated with animated graphics is provided by conventional methods such as through pointer operation, keyboard input or other known input techniques
Fig 's 3A-3D are exemplary displays on an interactive PC/TV at a viewer's premise illustrating various stages of viewer interaction with ammated GLFs during a presentation of a video
In Fig 3 A, an exemplary video presentation 45 is being displayed on a viewer's PC/TV 43 For the purpose of discussion and example, the primary video is a short video clip sponsored by a travel agency Presentation 45 illustrates a sequence from a vacation location being offered by the travel agency for the purpose of attracting vacation prospects to the area
As presentation 45 displays on PC/TV 43, a first animated GLF 49 appears as a dorsal fin of a fish swimming up and down a stream in the video The path instruction
for GIF 49 has been programmed by an author at an authoring station into an annotation stream as described above, the annotation stream being sent to the user's station via server 26, such that GIF 49 appears to swim within the stream boundaries in the video sequence contaimng the stream Gif 49 is provided as a hyperlink To progress to a next level, a viewer moves a cursor to the GIF
There are several ways one may interact with a hyperlink Activation may occur by a mouse-over, which means moving the screen cursor over the hyperlink area, by a click, by a double-click, or by combinations
In Fig 3B the dorsal GIF 49 is terminated as a result of the mouse-over, which sends a backlink signal to server 26 (Fig 1), which responds by sending a new annotation data stream to the user's station The new annotation stream provides a second GIF 51 in the form of a full-bodied fish Alternatively the GIF 51 may be sent along with GIF 49 in the primary annotation stream along with information about displaying GIF 51 as a result of user interaction to GIF 49 GIF 51 is programmed to jump in and out of the stream staying within the stream's boundaries GLF 51 may continue to display throughout the video waiting on a viewer's response, or may be timed to terminate automatically if the viewer does not respond In one embodiment, when a scene change occurs from, for example, a sequence showing the stream to, perhaps a sequence showing a cabin offered for rent, GIF 51 will terminate and a new interactive and animated GIF associated with the next sequence (cabin) may automatically appear If a viewer chooses, he or she may obtain additional information by clicking on the second GLF (51) before the scene changes
In Fig 3C, the viewer has elected to click on GIF 51 in order to proceed to a next level Clicking on GIF 51, in this example terminates the second GLF and executes yet a third GLF In Fig 3D, a third G F 53 in the form of a full bodied fish executes as described above, and immediately jumps to a sidebar area or to another specified location in the video where upon reaching it's destination, an information block 55 automatically pops up Information block 55 contains text information associated with the vacation spot such as accommodation availability and pricing,
travel directions and optimum season information, available guide information for fishing in the stream, or any other imaginable information that may be provided
In addition to simple text information, block 55 may contain interactive links in the form of text or graphics leading to additional sources of information such as to a WEB page hosted by the travel agency containing booking arrangements and the like There are many possible arrangements Block 55 may, in some embodiments, be an animated GIF with animation in the form of moving text The use of a vacation video and travel agency as a host is an isolated example used simply to describe one possible application that may be provided in the practice of the present invention Such an application may, perhaps, be provided in conjunction with a special interactive channel such as a travel channel where such interactive vacation videos regularly air
In a preferred embodiment of the invention authoring at authoring stations of authoring system 15 of Fig 1 includes associating complete packages of icons, GLFs and the like with a primary video stream For example, the sequential displays described with reference to Figs 3A-3C above are pre-authored as a related package and associated with the primary video at the same time The alternative forms are all present in the annotation stream, and server 26 is programmed to provide the first form until a hyperlink is activated by a first method (mouse-over), then the second form in response to a second signal, and so forth It will be apparent to one with skill in the art that a very wide vaπety of desirable effects may be provided by the techniques of the present invention in different embodiments For example, a first interactive hyperlink may be provided as an icon static anywhere in a displayed video upon a mouse-over, the icon may show a text annotation, such as "click for local weather" Upon clicking, the icon may be replaced by an annotated GIF according to the weather, a blue sky with moving clouds for clear weather, a storm cloud with rain falling for stormy weather, and so on
As another example, a first annotated display may provide an icon with a text script "weather update" A mouseover causes the icon to change temporarily to a Starbucks™ logo After a short time, the logo disappears and a weather update appears in a sidebar or other window, or by an ammated GIF showing the weather, as
descπbed above In this manner, short advertisements may be provided in the process of performing other functions
In another example, a GIF may be provided in a primary video of such as a basketball game informing viewers that there are tickets available for an upcoming game Clicks can be processed in a manner that the first 100 respondents, or even numbered respondents of the first 200 for example, will be eligible to buy the 100 available tickets The respondents identities are known, as the service is a subscription service, and the responses are by backlink from selecting a hyperlink Applications to purchase may then be mailed to the lucky respondents, or there may be provision made for online purchase, or purchase may be automatic and billed on the subscriber's account
In yet another embodiment animated graphics may take the form of animated company logos or brands, and user interaction may result in any number of responses, including such as downloading coupons for discounts and the like Applications may be conceived for virtually any type of video presentation wherein an advertiser or sponsor may gain benefit by providing graphics such as GLFs and information associated with them In other embodiments, ammated GLFs may be prepared and used purely for entertainment or educational value instead of for selling products or for other advertisement purposes Therefore, the method and apparatus of the present invention should be afforded the broadest scope The spirit and scope of the present invention is limited only by the claims that follow