GB2350742A

GB2350742A - Interactive video system

Info

Publication number: GB2350742A
Application number: GB0005436A
Authority: GB
Inventors: Andrew Bateman
Original assignee: University of Hertfordshire
Current assignee: University of Hertfordshire
Priority date: 1999-03-12
Filing date: 2000-03-13
Publication date: 2000-12-06
Also published as: GB9905785D0; GB0005436D0

Abstract

An interactive video system comprising: means for accessing a plurality of video sequences; a memory for storing, for each accessed video sequence, one or more temporal anchors which comprise information about temporal locations in that video sequence, and one or more spatial anchors which comprise information about spatial locations in that video sequence, and wherein said memory is also arranged to store, for each spatial anchor, a link to another spatial or temporal anchor or to itself; a user input for receiving information about a spatial location in one of the video sequences; a video display arranged to display a first one of the video sequences; a processor arranged to control the video display such that if a user input is received comprising information about a spatial anchor stored for the first video sequence then a second video sequence, that is determined by the link attached to the spatial anchor, is displayed instead of the first video sequence.

Description

2350742 INTERACTIVE VIDEO SYSTEM

Field of the Invention

This invention relates to an interactive video system and to a method of controlling an interactive video system as well as a computer program for controlling an interactive video system. The invention also relates to a system and computer program for creating an interactive video for display on the interactive video system.

Description of the prior art

An interactive video system is a system for displaying video images where the sequence in which the video images are presented is not fixed and can be influenced by the user. Known interactive video systems suffer from several problems. For example, in many interactive video systems there is significant disruption to the flow of the video picture and the soundtrack when a user interacts with the video display. This disrupts the viewers attention and detracts from the effect of the video presentation itself. Another problem is that most known interactive video systems present discontinuous video and audio material embedded in some sort of visible interface which detracts from the video display itself.

As well as this a problem has arisen in that the underlying system used by known interactive video systems, for presenting different video sequences as a result of user inputs, is complex. Creators or authors of interactive videos have had to be experts at programming and software development as well as being able to create artistic videos. Also, known interactive video systems are often suitable tor only one type of interactive video presentation and are not versatile.

The advent of the laser disk player and the development of the personal computer brought about the introduction of viable desktop interactive video, where

1 the personal computer was used to control the random access to the analogue video material that was stored on the laser disk.

Before this there had been systems that used videotape, but the endless spooling required to play different sections of tape meant that the access to non sequential video sequences was far from instantaneous. Initially, either the computer's output and the video picture were displayed on different screens or, with more sophisticated systems, the video and computer images were "genlocked" together and displayed on a single video monitor. With the production of video overlay cards for the computer the video picture was then displayed within a box on the computer screen. The laser disk based interactive video installation tended to be used only by large companies for interactive training and kiosk applications because others could not afford the large cost of equipment and the complexity of software development.

Over the past 10 years as digital video has developed it has become possible to dispense with the laser disk player and play back, under software control, video from either a hard disk, a CD-ROM drive, or latterly a DVD (digital versatile disk) drive resident within the %PU" (central processing unit) case of the personal computer. This has reduced the cost of producing interactive video, but the complexity of creating interactive video remains.

General purpose multimedia authoriing packages do exist, for example, MacroMedia Director (trade mark). However, using these tools the author of interactive video is forced to program the vast majofity of the interaction themselves or use "behaviours", which if not used carefully can have unpredictable effects. With more restricted tools, such as MacroMedia Authorware (which is particularly suited to producing "courseware"), the author can make use of ready built constructs but these are not tailored to the production of interactive video.

2 Research into interactive video narratives has been done by Davenport et al in the Interactive Cinema Group at MIT Media Lab. They have produced a number of interesting experimental pieces, latterly Agent Stories (Brooks) and Fluid Transition (Tiongson) but all using different hardware and software. Because of this a single versatile software architecture that can be used to build many different types of interactive narrative has not been provided.

The Phillips CD-1 player (trade mark) is an example of a system that promised an interactive video system. However a major problem with this system was that creation of suitable interactive videos for display using the system was extremely complex and time consuming and involved hard coding much of the required software.

It is accordingly an object of the present invention to provide an interactive video system which overcomes or at least mitigates one or more of the problems noted above. Another object of the present invention is to provide a method of controlling an interactive video system which overcomes or at least mitigates one or more of the problems noted above.

Summary of the Invention

A software system is described herein for creating and viewing interactive digital video artefacts and is referred to as "InterMovie". The creator (or creators) of such an artefact is (collectively) the author and the user (only one person can interact with the artefact at any one time, though several may watch it) is the viewer. InterMovie is designed primadly, but not exclusively, for the production of drama and documentary programmes. The screen interface for the viewer is designed to detract as little as possible from the content of the presented video and audio. InterMovie is intended for filmmakers to tell stories in a new way and for their audience to have control of how they view and review these stories.

3 According to a first aspect of the present invention there is provided an interactive video system comprising:

(i) means for accessing a plurality of video sequences; (ii) a memory for storing, for each accessed video sequence, one or more temporal anchors which comprise information about temporal locations in that video sequence, and one or more spatial anchors which comprise information about spatial locations in that video sequence, and wherein said memory is also arranged to store, for each spatial anchor, a link to another spatial or temporal anchor or to itself; (iii) a user input for receiving information about a spatial location in one of the video sequences; (iv) a video display arranged to display a first one of the video sequences; (v) a processor arranged to control the video display such that if a user input is received comprising information about a spatial anchor stored for the first video sequence then a second video sequence, that is determined by the link attached to the spatial anchor, is displayed instead of the first video sequence.

A corresponding method of controlling an interactive video system is also provided, comprising the steps of:

(i) accessing a plurality of video sequences and for each video sequence storing one or more temporal anchors which comprise information about temporal locations in that video sequence, and one or more spatial anchors which comprise information about spatial locations in that video sequence; (ii) for each spatial anchor storing a link to another spatial or temporal anchor or to itself; (iii) providing a user input for receiving information about a spatial location in one of the video sequences; 4 (iv) displaying a first one of the video sequences on a video display; (v) if a user input is received comprising information about a spatial anchor stored for the first video sequence then a second video sequence, that is determined by the link attached to the spatial anchor, is displayed instead of 5 the first video sequence. This provides the advantage there is no significant disruption to the flow of the video picture and the soundtrack when a user interacts with the video display. This enables the viewers attention to be maintained and for the full effect of the video presentation to be appreciated. Also it is not necessary to use any visible 10 interface which detracts from the video display itself. As well as this the underlying system used by the interactive video system is simple and this enables the system to be used easily by novice as well as expert users. According to another aspect of the present invention there is provided a method of creating an interactive video suitable for display on an interactive video system comprising the steps of:

(i) accessing a plurality of video sequences and for each video sequence storing one or more temporal anchors which comprise information about temporal locations in that video sequence, and one or more spatial anchors which comprise information about spatial locations in that video sequence; (ii) for each spatial anchor storing a link to another spatial or temporal anchor or to itself; (iii) providing a user input for receiving information about a spatial location in one of the video sequences; (vi) configuring a processor such that if a user input is received comprising information about a spatial anchor stored for the first video sequence then a second video sequence, that is determined by the link attached to the spatial anchor, is displayed instead of the first video sequence.

This provides the advantage that creators or authors of interactive videos do not need to be experts at programming and software development as well as being able to create artistic videos. Because the underlying architecture of the system is simple the system is versatile and suitable for use to create many different types of 5 interactive video presentation.

The present invention also encompasses a computer program stored on a computer readable medium and arranged to control an interactive video system such that:

(i) a plurality of video sequences are accessed and for each video sequence 10 one or more temporal anchors which comprise information about temporal locations in that video sequence are stored, and one or more spatial anchors are stored, each spatial anchor comprising information about spatial locations in that video sequence; (ii) for each spatial anchor, a link is stored to another spatial or temporal anchor 15 or to itself (iii) a user input is provided for receiving information about a spatial location in one of the video sequences; (iv) a first one of the video sequences is displayed on a video display; (V) if a user input is received comprising information about a spatial anchor 20 stored for the first video sequence then a second video sequence, that is determined by the link attached to the spatial anchor, is displayed instead of the first video sequence, Brief description gf the drawings Figure 1 is a schematic diagram showing the relationship between movies, anchors and links.

Figure 2 is a flow diagram illustrating a sequence of events when a user selects a spatial anchor.

6 Figure 3 is a flow diagram illustrating a sequence of events when a temporal anchor is activated.

Figure 4 illustrates a user interface display for use by an editor or author.

Figure 5 is an exploded view of an editing user interface showing editing modes.

Figure 6 illustrates an alternative version of a user interface display for use by an editor or author.

Figure 7 shows a review compass.

Figure 8 shows an example of a view list.

Figure 9 shows an example of a history schematic.

Detailed description of the Inmention

Embodiments of the present invention are described below by way of example only. These examples represent the best ways of putting the invention into practice that are currently known to the Applicant although they are not the only ways in which this could be achieved.

The term "script" is used to refer to a collection of one or more conditions such that when certain of the conditions are met particular outcomes of the script result. For example, a script can be an executable section of programmed logic.

The term "video sequence" is used to refer to a series of images arranged in a particular chronological order.

The term "audio sequence" is used to refer to a continuous stream of sound possibly including periods of silence.

The term Intefflovie artefact" is used to refer to one or more linked segments, where a segment comprises one or more linked movies, entrypoints, transitions and cursors and none or more feedback sounds, soundtracks, scripts and variables.

Unlike linear video artefacts, such as film and television programmes, InterMovie artefacts have multiple views. A view consists of sections of video that 7 have been watched in a particular order. For a linear video artefact there is only one view. This is determined by the director and editor during the postproduction process and consists of all the scenes contained in the artefact in the order they were assembled in the edit. Every piece of video that is contained in the final artefact can be seen by the viewer simply by them watching the artefact from beginning to end. In an InterMovie artefact the view experienced by the viewer can be shaped by his/her interaction with the artefact or other programmable factors, such as the time of day that the artefact is being viewed. Thus, every different view of the artefact may show the viewer a different selection of the artefact's content - possibly giving the viewer a different comprehension of the artefact's narrative. It is possible for the author to provide a default view within an InterMovie artefact. This is comparable to the single view of a linear video artefact and is what the viewer sees if s/he chooses not to interact.

An interactive video system comprises a video display, a memory, a user input and a processor. The system preferably also comprises means for displaying audio sequences, for example a loudspeaker system.

Any suitable type of video display can be used, for example a computer screen or a television. The user input device can be a computer mouse, keyboard, a joystick, a remote control device or any other suitable user input device. The memory can be the hard disk of a computer, a CD-ROM, a Digital Video Disk (DVD), a computer network, or any other medium that is suitable for storing digital video and audio information. In the case that a WD is used it may be necessary to use a data buffering system in order to produce apparently continuous digital video streams. The processor can be provided by a personal computer or any other suitable processing device.

An interactive video system according to the present invention provides three modes of operation, a review mode, an interaction mode and an edit mode. In the 8 review mode the viewer is effectively able to "retrace" his/her steps; that is the user is able to replay the interactive video display as seen so far in a particular interactive video session. The user is able to "forward wind" or "rewind" within the review mode as for a conventional video tape player. in the interaction mode the viewer is able to make user inputs that affect the sequence of presentation of video sequences on the display. In the edit mode an author is able to create an interactive video for presentation using the interactive video system described herein. All three modes require a user input device such as a pointer device or mouse for their operation. In addition the edit mode requires the use of a user input device such as a computer keyboard for data entry.

The interactive video system is provided with a control architecture that comprises movies, links and temporal and spatial anchors. These are discussed in turn below and illustrated in Figure 1. As well as this the interactive video system comprises segments, entry-points, transitions, variables, cursors, feedback sounds, sound-only movies, scripts and histories. These are also discussed below.

Movies The basic unit of the architecture of the present invention is the movie 101 to 105. This is a reference to a continuous sequence of digital video andlor audio data. That is a movie can be a video sequence or an audio sequence. This sequence may be a few seconds or many minutes in duration and may have none or more synchronous soundtracks. Each movie has two or more temporal anchors and none or more spatial anchors. All digital data belonging to a movie is stored externally to a segment. A segment consists of a collection of resources including movies, cursors, sounds, entry points, transitions, variables, sound-only movies and scripts. The architecture does not alter this data, it simply plays it on to the computer screen and out of the speakers. All movies within the same segment have 9 the same pixel dimensions but the colour depth and frame rate of individual movies may be different. The movie is essentially the surface on which the viewer interacts.

Each movie can itself contain video edits (cuts and transitions that were included during construction of the movie). In the present invention movies are connected via links 120, 121. By linking movies together a segment is produced. An InterMovie artefact comprises one or more linked segments.

UDEZ A link comprises a source and an optional target and transition. The source is either a temporal or a spatial anchor. Temporal and spatial anchors are described below. The target is a temporal anchor in the same movie or another movie in the same segment. The target may also specify a temporal anchor in another segment via reference to an entry-point in that segment. A transition occurs when the link is triggered. The source, a particular frame of a particular movie, is defined by how the link is attached to the movie. In the case of a temporal anchor the frame of the movie that the anchor is attached to defines the movie frame of the source. In the case of a spatial anchor the frame of the movie is fixed at the point the link is triggered. The target and transition may be fixed (a fixed link), but can also be derived. Derivation is by the execution of a script (a conditional link). A null link has neither a target nor transition.

Explicitly, there are three types of link - fixed, conditional and null:

Fixed links have a single pre-set target - another movie within the segment. For example, Figure 1 shows fixed link 120.

Conditional links have programmed (scripted) logic assigned to them. This logic may include several possible targets, the actual target of the link being resolved by the execution of this script when the link is activated. For example, Figure 1 shows conditional link 121.

Null (or empty) links do nothing, but can be used as place-holders. A null link can later be replaced by either a fixed or conditional link. No null links are illustrated in Figure 1.

It is possible for a conditional link to be "non- branching", i.e. have no target.

For example, a conditional link may increment a counter. This action has no immediate perceivable outcome to the viewer, but the value of the counter may be used by the logic of a later conditional link. Figure 1 illustrates non-branching conditional link 202.

The links are attached to their source movie by anchors. Anchors are comparable to those found in HTML (Hyper text mark up language), where the anchor attaches a URL to a word or phrase within the HTML document. There are two types of anchors - spatial and temporal.

Temporal anchQrs are associated with particular frames of the movie. For example, in Figure 1, movie 101 has three temporal anchors 112, 113 and 114. A link assigned to one of these anchors is executed when the frame to which it is attached is displayed. For example, in the event that movie 101 is being played, when the frame corresponding to temporal anchor 113 is reached, link 122 is activated. This is a conditional link and according to the particular situation and the conditions that are met for this conditional link, one of movie 102, 103 and 104 is selected for display next.

Temporal anchors can also be used as alternative entry points to a movie. As well as specifying a particular movie the target of a link can include a temporal anchor within that movie as the point (frame) at which the target movie should begin its playback. All movies are automatically assigned "startn and "end" temporal anchors and the interaction author can add as many others to a movie as desired with the proviso that no two temporal anchors can be attached to the same frame. For example, movie 101 has start anchor 112, end anchor 114 and also anchor 113.

11 A temporal anchor that is used as an in-point to a movie does not trigger any associated link. This is to avoid the possible immediate exit from a movie that has just started playing.

In summary, a temporal anchor is a particular frame of a movie that has associated with it a link or may be used as an in-point to the movie or both. The link is triggered when playback of the movie reaches the particular frame. By default when a temporal anchor is created a null link is assigned to it.

Spatial anchors 107, 108 are the active areas of the video picture 106 that have been described above. These active areas may remain static during the playback of the movie or may change shape, appear or disappear to follow the motion of the object in the picture to which they belong. A movie containing active areas is comparable to an HTML image map. The difference is that, as stated, the position, size and presence of the active areas may change as the movie plays, whereas in an image map the active areas are fixed. A link assigned to a spatial anchor is activated by viewer interaction - i.e. the clicking of the pointer device when the cursor is within the active area of the anchor. For example, in Figure 1, if the user clicks on an area of the screen within spatial anchor 108 then link 120 is activated. This is a fixed link that causes movie 105 to be displayed next. Similarly, if spatial anchor 107 is selected by the user then conditional link 121 is activated.

Then, depending on the particular conditions that are met at the time for that link, one of movies 103 or 104 will be displayed.

In summary a spatial anchor is a particular area of the video image of a movie. The size and location of this may change over the duration of the movie.

Associated with the spatial anchor is a link. The link is triggered by viewer interaction. For example, the pressing of a button on a pointer device when the cursor is within the defined area. The spatial anchor may also have associated with 12 it a cursor image and a feedback sound. The cursor image is adopted by the cursor whilst it is within the spatial anchor. The feedback sound is played when the link is triggered. By default when a spatial anchor is created a null link is assigned to it.

Cursors The present invention does not automatically highlight the active areas that are present within a movie's video image. The viewer must actively interrogate the video image by moving the cursor around the screen and noting where the shape of the cursor changes. The author can assign each active area its own cursor shape.

Active areas that have no cursor assigned to them remain hidden. The different cursor shapes available are stored within the segment. Of course, the video image of the movie can be pre-constructed so as to make the position of the active areas obvious. For example, when a sequence of digital video is originally generated a glowing halo effect could be placed around a particular object in the video image.

Indeed, in the near future it will be possible to generate this type of effect "on-the fly", but it is questionable how useful this would be, as it goes against the stated aim of the addition of interactivity with the minimal of disruption to the video picture.

With the presence of the halo the viewer has to resolve the question: is the halo intrinsic to narrative or only present to show that an object can be "clicked" on? If the latter is the case this visual disturbance may in fact detract from the dramatic impact of the scene being viewed.

The cursor object stored within a segment consists of the data required to generate a cursor shape.

Feedback Sounds To provide positive feedback for the viewer to indicate that a link attached to a spatial anchor has been triggered an author-selected sound can be played. These sounds, which like the cursor shapes are stored within the segment, can be of any 13 length, but a definite, short sound, akin to a beep, is usually the most appropriate causing only minimal disruption of the movie soundtrack. indeed it is possible for the interaction author to tailor the type of feedback sound that is played to fit within the context of the movie - e. g. in a street scene when the viewer clicks on a 5 speeding car the feedback sound might be the screech of brakes.

The feedback sound object within a segment consists of the data required for the production of a feedback sound.

Sound-only movies The present invention, as it has been described up to now, is very much a movie-centred structure. In interactive mode the narrative unfolds by a steady progression through a linked series of movies with all the synchronisation of sound and picture internal to each movie. However, it is possible that the interaction author would want to synchronise a selection of movies to a continuous soundtrack. For an interactive "pop promo" where, no matter where the viewer had navigated to in the course of the verse of the song, come the chorus the viewer is returned to a set point within a set movie. In order to accommodate this type of soundtrack-driven interaction, the present invention includes the facility to play a sound-only movie concurrently with video-based movies that contain video andlor sound data. The sound-only movie can have temporal anchors with links (both fixed and conditional) that can trigger either a video-based movie or a replacement sound-only movie. With the exception of the audio cross-fade between two sound-only movies, only one sound-only movie can be played at a time. As with video- based movies, a subsequent sound-only movie replaces a previous one. Another use of sound-only movies is to disguise the transition between two video-based movies by providing a continuous soundtrack that spans the transition.

14 5-Qn12t$ A script consists of a section of program logic that can invoke a transition to a target movie andlor can access and manipulate variables.

Thus scripts are executable sections of programmed logic, stored as entitles within the segment, which can be assigned to links. The assignment of this logic to a link causes the link to become conditional. The invention also allows scripts to be designated as global scripts. One of these scripts can contain an "idle" routine which is executed at regular intervals whilst in interactive mode. Global scripts can also contain (global) routines that can be called from any of the non-global scripts.

The inclusion of these two features makes it possible for the architecture to support the creation of orchestrated interactive movies as well as those steered primarily by the interruption of the viewer.

Ent!t point This consists of a movie identifier (identifying a particular movie in a segment), a frame identifier (temporal anchor) for that movie and none or more histories. Transition This consists of a specification of a video effect (e.g. type of dissolve, wipe, etc.), the duration of the effect and an audio cross fade duration.

Variable A variable consists of a variable identifier with an associated list of variable states. As with any programming language, a variable is used to store values during the execution of program logic. Uniquely, however, when a value is assigned to a variable InterMovie does not simply update the variable to this new value. Instead it stores the new value as a new state of the variable. All states of the variable are retained. This enables the state of the entire InterMovie system to be reconstituted completely for any point in a saved history. In turn, this means that the viewer can rewind through a history to any particular point and then re-enter run mode with the system in the same state as if the viewer had never seen beyond this point. If this were not the case the system would not remain consistent, making it extremely difficult for the author toensure that his/her intentions are accurately articulated by the InterMovie artefact.

Video sequence A video sequence is a contiguous digital video sequence with none or more synchronous digital audio sound tracks - all videolaudio data is stored separately (possibly remotely) to the segment data.

Audio sequence An audio sequence is a piece of contiguous digital audio data stored with the segment data.

In-point/put-point In points and out points specify particular frames within a movie. If they are in the same movie then the out-point follows the in-point in normal (forward) playback.

Hiahliah A highlight is a sub-sequence of a movie as defined by an in-pointlout- point pair. History Each time a segment is entered via a particular entry-point a history is generated and associated with that entry-point. A history is a record of the viewer's interaction with the content of the segment stored as a list (or tree) of events within the segment. Figure 9 is a simplified diagram of a run time history. The most common event is an edit. An edit consists of an out- point in a source movie, an in-point in a target, usually different, movie and a record of the transition employed to move from the source to target movies. An edit is generated each time a link is triggered.

Another event is a vahabie-update. This is generated when a new value is assigned to a variable during the execution of a script. The variable-update event consists of a run-point (the frame of the current movie where the script was executed) a 16 reference to the state of the variable before it was updated and a reference to the state of variable after the update occurred.

Histories are used by the review mode to allow the viewer to retrace his/her progression through the InterMovie artefact. Each history, which belongs to a particular segment, references its source and target histories in other segments. This allows the viewer to seamlessiy move from one history (and its segment) to another when in review mode. In this mode the viewer is, in effect, reviewing the edit s/he has created by his/her own interaction with the InterMovie artefact.

Figure 9 is a simplified diagram of a run time history showing only entry (A, C, G, 1, J, L and M) and branch (13, D, E, F, H and K) event nodes. If shown, other types of event node, such as edit and variable-update event nodes would appear between entry-to-branch or branch-to-branch sections of the tree. The arrows signify progression through the history during forward playback. Figure 9 also shows the display on the review compass when video playback is at each of the nodes shown in the run time history. This assumes that node M is the current latest entry node. For example, when the video playback is at node A, then the review compass shows a node indicator with a symbol indicating that we are at the start of a segment and that the entry node M is in the direction of the red east arrow. When the video playback is at node D then the review compass shows a node indicator with a symbol indicating a branch node. The north arrow is coloured red. The East arrow is not coloured. The west and south arrows are blue. This indicates that if the user clicks on the red arrow the display will move towards entry node M and any other arrow selection would move the display away from node M.

Figure 2 illustrates the sequence of events in interactive mode when the user selects a spatial anchor by clicking on an active area or spatial anchor within the display. Initially a first video sequence belonging to a first movie is displayed 21 to the user on the video display. The user is then able to provide inputs in order to 17 select spatial anchors. For example, this can be done using a computer mouse and clicking on a region of the display which is within a spatial anchor 107, 108. Once such a user input is received 22 a link that is connected to the spatial anchor 107, 108 is activated. A video sequence belonging to a second movie, that is determined by the attached link, is then retrieved from memory 23 (e.g. a DVIE1 disk or computer network server) and displayed in place of the first video sequence 24.

Figure 3 illustrates the sequence of events in interactive mode when a temporal anchor is reached in a movie. In this case, a first video sequence belonging to a first movie is being displayed 31. The computer or other processor monitors an elapsed display time for the first video sequence 32 and checks to see whether this elapsed display time corresponds to a temporal anchor for the current displayed movie. If it does, then the link for that temporal anchor is activated and the video sequence belonging to the movie that is determined by the attached link is retrieved from memory 33. This retrieved video sequence is then displayed in place of the currently displayed sequence 34.

A system for creating interactive videos for presentation using the interactive video system described herein is now explained. This system is referred to as 'InterEdit".

InterEdit is a software tool to create InterMovie artefact segments. It is written for the Apple Power Macintosh platform in Incwell SuperCard, with extensions in C, and makes extensive use of Apple QuickTime digital media architecture.

As is defined by the architecture, a segment created by InterEdit contains the temporal and spatial anchors assigned to a movie and the available cursor shapes, feedback sounds, entry points, transitions, variables, sound-only movies and any defined scripts, but not the movie's video or audio data. All that is stored in the file 18 is a reference to this data. This data is not changed in any way by the InterEdit application.

Intergctive mode In this mode the viewer is presented with a single window that displays the video content of the currently playing move. The viewer interacts with the movie by moving a context-sensitive cursor around the screen. When inside an active area (spatial anchor) the cursor changes shape. if the viewer depresses the mouse button at this point the link associated with the spatial anchor is activated. The usual consequence of this is for a new movie, one that replaces the original, to begin playing in the window. To maintain continuous video playback the author assigns a link to at least one temporal anchor (if none other, this is usually the last one). When playback reaches this anchor the associated link is triggered and a new movie begins playing without any direct viewer interaction.

In summary the interactive mode uses a single window, in which a segment's video sequences are displayed, with a context-sensitive cursor that (if programmed to do so by the author) changes to an alternate form whilst within a spatial anchor. No other controls or interface devices are visible on the screen. In this mode all spatial anchors are 'live' pressing the button on the pointer device causes any link attached to such an anchor to be executed. All video sequences play forward at normal speed unless programmed differently by the author.

Review mode The review mode uses the same window as in the run mode but with the spatial and temporal anchors no longer active. Additionally, the review compass window is displayed. Figure 7 shows an example of a review compass as displayed in a review compass window. This has standard video transport controls 70 and allows the viewer to replay the view of the InterMovie artefact s/he has generated. The 19 generated view ends at the point where the review mode was entered. In this mode the viewer can spool backward and forward through the current view of the InterMovie artefact as though it was a videotape recording. Additionally, the viewer can re-enter interactive mode at any point within this view. By doing this the viewer creates a branch in the current history. Next time s/he returns to the review mode and rewinds past this point the review compass indicates the presence of a branch. The viewer can choose to ignore this or switch the video to forward playback. In this case the video stops playing when the branch point is encountered and the viewer is offered the choice of following the branch to watch the view just created or continuing to watch the original view. At all times the review compass shows the user the direction in which the video must be wound in order to return to the last point s/he entered the review mode. If the viewer continually enters and exits the review mode at points that were not previously entrance points an underlying multiple branching history tree is created. This tree encapsulates multiple views - all views sharing a common start point (the entry-point of the segment to which the history belongs), but branching to different end points.

In order to return to run mode using the review compass the user clicks panel 71 on the review compass. This panel displays the current time.

In order to open a view list using the review compass the user clicks panel 72 on the 20 review compass.

In order to return the playback to the entry node the user clicks panel 73 on the review compass.

In order to initiate automatic recapitulation, the user clicks button 74 on the review compass and in order to resume the run mode from the most recent entry node button 75 is selected.

The review compass also has a node indicator 76 and four arrows referred to as a north arrow 77, a south arrow 78, an east arrow 79 and a west arrow 80. Each of 20 these arrows is a display panel that changes colour from red to blue. If an arrow is red it indicates the direction towards the most current entry node. If an arrow is blue it indicated the direction away from the most current entry node. The node indicator itself is a display panel that displays a symbol indicating the type of node that the current node is. For example, the symbol shown in the node indicator of Figure 7 indicates a non-terminal node. In summary, the colour and symbol displayed by the node indicator shows the viewer, if it is red, that the current node of the history is part of the current view or, if it is blue, that is not the case.

View lis For more advanced viewers a visual representation of the history is also available. This is the view list window that is accessed via the review compass using panel 72. An example of a view list window is given in Figure 8. The view list allows the viewer random access to events in the history. Selecting an event from the listing 81 causes the display of a still 82 of the video image that was being shown when this event was generated. This allows the viewer to quickly browse through the entire interMovie artefact that has been viewed thus far. The viewer has the choice to return to review compass with the video playback at the same point as when s/he switched to the view list (using arrow 83) or for the video playback to be set to match the image of the currently selected event (using button 84). In the latter case the display on the review compass is adjusted so that this point becomes the current entrance point in the current history.

When the viewer is using either the review compass or view list s/he is seamlessly able to move from one segment to another. In the review mode this is achieved by simply winding through the video. In the case of the view list selecting a down arrow icon 85, from the last line of the listing, moves to the previous segment and pressing an up arrow icon 86, situated above an end point icon in the listing, moves to a 21 subsequent segment. If any of these icons are not present it indicates that either no previous or subsequent segments have been viewed.

AutomatiQ recapitulation Another facility of the review mode is that of automatic recapitulation. When this is invoked InterMovie moves forward through the history from the current point to the point at which the history was entered. Unlike normally in review mode, where all the video content that was seen in the particular view that is being reviewed is replayed, only the highlights are shown. If a movie has no highlights it is not shown.

In-points and out-points within highlights are maintained. If there are overlapping highlights preference is given to those which can be shown in their entirety or, failing that, those which are encountered first. During automatic recapitulation the review compass is hidden, the cursor takes a particular form and depressing the button on the pointer device returns the viewer to review mode.

Edit mode This mode has the same movie window present in the interactive mode. However, in edit mode the spatial anchors (called hotspots) and the temporal anchors (called markers) are not active. Clicking on a hotspot or moving the play back of the current movie past the frame to which marker is attached, given the Editor window is in the correct mode, causes the link assigned to that anchor to be displayed in this window. The Editor window (see Figure 4), which is always visible when in edit mode, contains the majority of the editing functions available in interEdit - the other windows accessible in edit mode being the Transport, Viewer and Soundtrack windows. Figure 4 illustrates an example of a user interface display for InterEdit. This shows the visible current hotspot in the Movie window. The Movie window - top left of the screen - bears the name of the interaction file - "Test 22 Interaction". The name of the current hotspot - "Earth" is shown in the third field from the left in the Editor window - bottom of the screen. Also shown is the Viewer window - top right of the screen which is "locked" to the Editor window. This causes the content of the link displayed in the Editor window to appear in the Viewer 5 window.

In summary, the edit mode provides a multiple window editing environment which is used by the author to create the InterMovie artefact's linked segments. Figure 6 shows an example of the windows that are available in the edit mode. By building the artefact in self-contained segments it is possible for a team of authors to each work on their own segment - coming together towards the end of the production process to link these segments together.

In order to make the user interface as compact, yet readable, as possible all InterEdit's modes and functions are accessed via iconic buttons. Very little text is used - indeed in the Editor window the only text that is displayed (the name of the movie, marker or hotspot) is almost entirely chosen by the author. When the author adds a digital video file to a segment to create a movie the reference to the movie within InterEdit is given the name of the file and all hotspots, scripts and markers (except the "start" and "end" ones) are named by the author when they are created. (The author can at any time re- name any object if she so desires).

The brevity made possible by the use of icons does add to the complexity of the interface for the novice user. An initial learning period is required by a new author to familiarise him/herself with the meaning of the icons. To keep this time as short as possible the position of a button, as well as its icon, is used to provide information about its functionality. This is best illustrated by the hierarchical modality of the layout and operation of the Editor window. Figure 5 shows an exploded view of InterEdit's edit window showing editing modes.

23 In the Editor window the buttons in the leftmost field give access to global properties which apply to the entire segment file. The next field to the right has two modes - one for video-based movies and one for soundonly movies (but only the former is currently implemented). In videobased movie mode the next field to the right shows the anchors that belong to the current movie. This section has two modes - one for markers and one for hotspots. In marker mode the name of the current marker is displayed. In InterEdit's edit mode a marker remains current from the frame to which it is attached through to the frame before the location of the subsequent marker. In hotspot mode the name of the current hotspot is displayed.

If there are no hotspots then all buttons in the field, except the add button, are disabled. In both marker and hotspot modes the next field to the right displays the link attached to the current anchor. This is predominantly blank for a null or empty link, shows the name of the target movie and marker for a fixed link or shows the name of a script for a conditional link.

Scripts, which are written in an augmented subset of SuperTalk (the xTa(k dialect of SuperCard) are created in the Viewer window. This window has four modes: null, movie, script and variable. The former three match the three possible link types. The viewer window can be "locked" to automatically display the content of the link shown in the Editor window. In hotspot mode this changes when the author selects a different hotspot, but in marker mode playing or jumping to a new frame in the movie can cause the current marker to change and thus the link field to be updated. The Viewer window can also be used to browse through all the videobased movies that stored in the segment file.

The playback of a movie in edit mode is controlled from the transport window 25 which, in addition to the usual VCR-type functions, has a scroll bar which the author can use to instantly access any frame of the current movie.

24 The high degree of modularity present within the InterEdit interface means that generally only one instance of an object type (movie, marker, hotspot or script) can be viewed at once. An exception to this is the Soundtrack window. This window can only be accessed if the current videobased movie has at least one soundtrack. The Soundtrack window displays a graphical representation of one of the available soundtracks - the author can select which one. In addition it also displays icons above the length of the soundtrack's waveform which indicate the positions of the movie's markers.

The advantage of the modular approach is that the screen area needed to display different object types is kept to a minimum. If the author wants to view a different object of a particular type s/he simply scrolls through those available with the previous/next buttons or directly selects an object by name from a pop-up menu. However, only being able to view one instance of an object type at a time does make comparison between instances of the same object type difficult. To alleviate this problem to some extent InterEdit allows the copying and pasting of the content of objects. For example, if an author wishes to the link attached to one anchor identical to that attached to another, rather than viewing both anchors together and setting the content of the second to be the same as the first (which is not possible in InterEdit), the author simply selects the first anchor, copies its contents and then selects the second anchor and pastes the copied contents into it.

InterEdit may also contain a waveform window, similar to the Soundtrack vAndow, for sound-only movies. It is also possible to access Transition and Resource Manager windows. The Transition window allows the specification of video effects for transitions between video sequences and in the Resource Manager window the author is able to add external resources to the segments. In this window, as well as adding resources such as cursors and feedback sounds, the author is able to add multiple video-based movies to the segment in a single operation.

With the current state of technology the prime medium for the delivery of InterMovie artefacts is either DVD or video-on-demand asynchronous digital subscriber line (ASDL) telephony networks. However, with the implementation of video transitions a problem arises. While the video transition is active the image displayed on the screen is a composite of two different movies which are stored in two different places on the same disk or are being accessed simultaneously by the same network connection. To circumvent this problem and make DVD or ADSL networks the ideal medium for the delivery of interactive narratives to the consumer, a data buffering means can be incorporated into the system. This overcomes the time delay problem in an elegant and novel manner.

It should be noted that using ADSL, or similar, networks, an architecture such as InterMovie could provide true interactive television, where you can explore rather than follow a, "television programme" without the loss of any dramatic conceit.

The present invention provides the advantage that the interactive video appears to be a natural extension to watching traditional linear television. When the viewer chooses not to interact s/he is presented with what appears to be a linear piece of video with no visible user interface encroaching on the picture. However, every person who views the same interactive video according to the present invention, may not necessarily be shown the same content. This is because the author of the interactive video may specify other variables not under the control of the viewer, such as time of day, that affect which video sequences are delivered.

When the viewer does choose to interact then minimal perceived disruption of the flow of the video picture and sound track occurs. The viewer uses a mouse, or some other hand held pointing device, to move a position indicator that appears on 26 a video display screen. Any suitable type of pointing device can be used, for example a remote control device. The form of the position indicator (cursor) changes when its position is within an active area of the display. An active area is a region of the video image which usually corresponds to an object in the video picture, such as the face of a man or the image of a door. When the viewer "clicks" the pointing device whilst the cursor is inside one of these active areas an action associated with that area is invoked. Most often this action will result in the displayed video picture changing. The change may be an abrupt cut or a gradual transition to a different video shot. At any point the viewer can stop the InterMovie artefact and enter a review mode. In this mode the viewer can rewind and fast forward through the video s/he has seen, although she cannot view beyond the point at which she entered the review mode. The review mode can only replay video that has already been displayed as a result of the viewer's interaction; it cannot predict what the viewer may do in the future. The viewer can stop the playback of the reviewed video wherever s/he wishes and re-enter the interactive mode. This time s/he may interact in a different way which will cause him/her to see video images that s/he has not seen before. Also, at any point the viewer can stop the interaction with the ability to return at some later time to continue from where s/he left off. The viewer of this type of interactive video becomes much more like the reader of a book who can re-read previolus sections and put down and pickup the book at will. The interactive video system enables pre-prepared video sequences to be assembled into what appears to the viewer to be a seamless interactive video and audio presentation. The interactive video architecture, InterMovie, described herein is flexible enough to be used to produce kiosk applications and standard computer based training materials and is particularly useful for creating interactive narratives. An interactive narrative is a narrative in which every viewer is not necessarily always presented with the elements of the narrative in the same order. Also, if developed in 27 conjunction with video-on-dernand networks and servers, InterMovie could bring about a new generation and paradigm of television true television on demand.

28

Claims

1. An interactive video system comprising:

(i) means for accessing a plurality of video sequences; (ii) a memory for storing, for each accessed video sequence, one or more temporal anchors which comprise information about temporal locations in that video sequence, and one or more spatial anchors which comprise information about spatial locations in that video sequence, and wherein said memory is also arranged to store, for each spatial anchor, a link to another spatial or temporal anchor or to itself; (iii) a user input for receiving information about a spatial location in one of the video sequences; (iv) a video display arranged to display a first one of the video sequences, (v) a processor arranged to control the video display such that if a user input is received comprising information about a spatial anchor stored for the first video sequence then a second video sequence, that is determined by the link attached to the spatial anchor is displayed instead of the first video sequence.

2. An interactive video system as claimed in claim I wherein each temporal anchor is either an in-point or has a link to another anchor or to itself.

3. An interactive video system as claimed in claim 2 which further comprises a frame counter arranged to count the number of frames that have been displayed in use; and wherein the processor is further arranged such that if the counted number of frames corresponds to a temporal anchor, stored for the first video sequence, then the link or in-point associated with that temporal anchor is activated.

4. An interactive video system as claimed in any preceding claim wherein the memory is further arranged to store one or more scripts, each script 29 containing information about a set of conditions and each script being associated with a link that is linked to a plurality of said video sequences; and wherein said processor is arranged to determine which of these video sequences to display on the basis of which conditions are met in the associated script.

5. An interactive video system as claimed in any preceding claim which further comprises means for providing sounds.

6. An interactive video system as claimed in claim 5 which further comprises means for accessing a pfuraiity of audio sequences and wherein said memory is arranged to store, for each audio sequence, one or more temporal anchors which comprise information about temporal locations in that audio sequence.

7. An interactive video system as claimed any preceding claim wherein said processor is further arranged to store a record of a user's interaction with said interactive video system, said record comprising information about events such as the display of the second video sequence instead of the first video sequence.

8. An interactive video system as claimed in claim 7 wherein said record is stored in the form of a tree of events.

9. An interactive video system as claimed in claim 7 or claim 8 which is further arranged to enter a review mode in which the processor is arranged to display a plurality of said video sequences on the video display on the basis of the stored record of a user's interaction.

10. An interactive video system as claimed in any of claims 7, 8 or 9 which further comprises a user interface arranged to allow a user to control display of said video sequences during the review mode.

11. An interactive video system as claimed in claim 10 wherein said user interface comprises a display arranged to indicate the direction of a specified location in the record of a user's interaction.

12. An interactive video system as claimed in any of claims 7, 8, 9 or 10 wherein said processor is arranged to display a visual representation of said record of the user's interaction.

13. An interactive video system as claimed in any preceding claim wherein said processor is further arranged to store information about one or more highlights each highlight comprising a sub-sequence of one of the video sequences.

14. An interactive video system as claimed in claim 14 wherein said system is arranged to enter an automatic recapitulation mode in which the processor is arranged to display said highlights on the basis of the stored record of a user's interaction.

15. A method of controlling an interactive video system comprising the steps of:

(i) accessing a plurality of video sequences and for each video sequence storing one or more temporal anchors which comprise information about temporal locations in that video sequence, and one or more spatial anchors which comprise information about spatial locations in that video sequence; (5) for each spatial anchor storing a link to another spatial or temporal anchor or to itself; (iii) providing a user input for receiving information about a spatial location in one of the video sequences, (iv) displaying a first one of the video sequences on a video display; (v) if a user input is received comprising information about a spatial anchor stored for the first video sequence then a second video sequence, that is 31 determined by the link attached to the spatial anchor is displayed instead of the first video sequence.

16. A method as claimed in claim 15 which further comprises the steps of.

(i) counting the number of frames of the first video sequence that have been displayed; (ii) if the counted number of frames is associated with a temporal anchor stored for the first video sequence then activating the link for that temporal anchor.

17. A method as claimed in claim 15 or claim 16 which further comprises storing one or more scripts, each script containing information about a set of conditions and each script being associated with a link that is linked to a plurality of said video sequences and wherein the method further comprises determining which of these video sequences to display on the basis of which conditions are met in the associated script.

18. A method of creating an interactive video suitable for display on an interactive video system comprising the steps of:

(j) accessing a plurality of video sequences and for each video sequence storing one or more temporal anchors which comprise information about temporal locations in that video sequence, and one or more spatial anchors which comprise information about spatial locations in that video sequence; (iii) for each spatial anchor storing a link to another spatial or temporal anchor or to itself., (iv) providing a user input for receiving information about a spatial location in one of the video sequences; (vi) configuhng a processor such that if a user input is received comprising information about a spatial anchor stored for the first video sequence then a second video sequence that is determined by the link attached to the spatial anchor is displayed instead of the first video sequence.

32 19. A computer program stored on a computer readable medium and arranged to control an interactive video system such that:

(i) a plurality of video sequences are accessed and for each video sequence one or more temporal anchors which comprise information about temporal locations in that video sequence are stored, and one or more spatial anchors are stored, each spatial anchor comprising information about spatial locations in that video sequence; (ii) for each spatial anchor, a link is stored to another spatial or temporal anchor or to itself (iii) a user input is provided for receiving information about a spatial location in one of the video sequences; (iv) a first one of the video sequences is displayed on a video display; (v) if a user input is received comprising information about a spatial anchor stored for the first video sequence then a second video sequence, that is determined by the link attached to the spatial anchor is displayed instead of the first video sequence.

21. An interactive video system substantially as described herein with reference to and as illustrated in any combination of the accompanying drawings.

22. A method of controlling an interactive video system substantially as described herein with reference to and as illustrated in any combination of the accompanying drawings.

23. A method of creating an interactive video suitable for display on an interactive video system substantially as described herein with reference to and as illustrated in any combination of the accompanying drawings.

33