AU672650B2 - Render system for the rendering of storyboard structures on a real time animation system - Google Patents

Description

-1- Render System for the Rendering of Storyboard Structures on a Real Time Animation System.

Field of the Invention The present invention relates to the display of video animation and in particular, the rendering of video segments to produce a multilevel simultaneous display of multiple video clips.

Background Art Traditional film production techniques usually involve the shooting of multiple video or film 'takes' or 'clips'. Normally much more film is shot than is required for the final S. 10 result. The resultant film is normally edited by a film editor who uses scissor like splicers and adhesive tape to cut and paste the desired shots together. Artistic decisions about the selection of shots to enhance the performance are made and a final result produced.

If dissolves or wipes are required between two sources, then the processing becomes much more complicated, often involving the use of multiple video recorders and vision switchers.

Although editing and mixing can be implemented on a frame by frame basis, it is desirable to be able to format the editing/production process with what is known in the industry as a "storyboard". The storyboard represents a plan of the film video or animation that is to be created and assists in the visualisation of the production. Storyboards are traditionally created by hand, going through many interactions and revisions during production and is basically a chart of various sources of audio/video and how they are to be combined over time to produce the final film. Furthermore, as a director, or the like may wish to alter the production once an initial storyboard has been created, it is highly desirable that the storyboard itself be readily editable.

Recently, computer systems have been introduced which go part of the way to automate this process. Typically, these systems require animations to be created on a frame by frame basis, saving each frame to disk, with the image being played back at a later date by reading each frame off the disk and displaying it on a computer screen or the like.

Recently, in Australian Patent Application No. 38238/93 (Attorney Ref: (RTO1)(224857)), entitled 'An Audio and Video Processor System", claiming priority from Australian Provisional Patent No. PL2141 filed 29th April, 1992, the contents of which are hereby incorporated by cross-reference, a video animation system was proposed that was capable of producing real-time video animation through the use of real-time object (RTO) technology. That document, and other co-related documents describe apparatus that can perform audio and video mixing, editing, sampling, and the like at a substantially (RT024)(259858) -2reduced cost than that ot prior art arrangements. For a discussion of RTO technology, the reader is referred to Australian Patent Application No. 38244/93 (Attorney Ref: (RT07)(202788)), entitled "A Real-Time Object Based Graphics System" claiming priority from Australian Provisional Patent No. PL2147, filed 29th April, 1992, the contents of which are hereby disclosed by cross-reference.

Australian Patent Application No. 53\4/ q- (A ttnnip, Fe Pr- M117 it entitled "Storyboard System for the Simultaneous Timing of Multiple Independent Video Animation Clips" claiming priority from Australian Provisional Patent No. PL6835, filed 29th April, 1992 and filed simultaneously herewith, the contents of which are hereby 10 incorporated by cross-reference discloses an automated storyboard system capable of editing, ordering and timing multiple simultaneous portions of animation or video source .information.

Hence there exists a general need for a system capable of taking audio and/or video source information and rendering it in real time on an audio and video processing system (AVAPS) such as that disclosed in the abovementioned patent application.

It is an object of the present invention to provide an automated storyboard mechanism capable of the automated rendering of multiple video clips on a audio and video animation processor system to produce real time video animation.

Summary of the Invention 20 In accordance with the present invention, there is provided an automated storyboard rendering system for the reproduction of a storyboard on a real-time processor, said storyboard comprising a number of columns each containing references to at least one information clip intended for reproduction, said system comprising: current column determination means adapted to determine those information clips that are active in a current column, and column rendering means connected to said column determination means and adapted to receive said currently active column and simultaneously render those information clips in said currently active column.

Preferably said column rendering means renders the animation clips in real time, Brief Description of the Drawings A preferred embodiment of the present invention will now be described with reference to the accompanying drawings in which: Fig. 1 illustrates the AVAPS system and its operational environment.

Fig. 2 illustrates the AVAPS system in more detail. and its operational environment.

-3- Fig. 3 illustrates a simple storyboard containing 4 columns, two rows, and therefore 8 cells.

Fig. 4 illustrates two clips in a sequence.

Fig. 5 illustrates two clips in a parallel.

Fig. 6 illustrates a storyboard example that could be used for a presentation describing the uses of a particular product in two countries.

Fig. 7 is a schematic block diagram representation of an AVAPS system such as that shown in Fig. 2.

Fig. 8 illustrates the data and process flow in the host processor of the AVAPS when 10 rendering astoryboard.

Fig. 9 illustrates an element of the render list.

Fig. 10 illustrates the sound information structure.

Fig. 11 illustrates the camera information structure.

Fig. 12 illustrates a timeline reference structure.

Fig. 13 illustrates the structure of the render column.

Fig. 14 illustrates the structure of a render clip r' erence.

Fig. 15 illustrates the structure of a render clip.

Fig. 16 illustrates the structure of the base render handle.

Fig. 17 illustrates the structure of the sound render handle.

Fig. 18 illustrates the structure of the camera render handle.

Fig. 19 illustrates the structure of the graphic render handle.

Fig. 20 illustrates the structure of the handle transform.

Fig. 21 illustrates the structure of the shape information.

Fig. 22 illustrates the structure of the colour information Fig. 23 illustrates the structure of an animated blend.

Fig. 24 illustrates the structure of the Start/Extent reference.

Fig. 25 illustrates the structure of the render shape.

Fig. 26 illustrates the structure of a graphic transform.

Description of the Preferred Embodiment Referring now to Fig. 1 there is shown the AVAPS 1 and its operational environment. Users purchase ROMCards 2, and operate the AVAPS 1 to sequence clips from the ROMCards 2 into storyboards. Storyboards are played on one or many AVAPSs.

Users can edit and save their sequenced clips (storyboards) onto RAMCards 3 for future use.

(RT024)(259858) Referring now to Fig. 2 there is shown the AVAPS system in more detail.. When playing a storyboard, the AVAPS 1 generates graphics and selects shapes for miming based on audio input. These generated images are merged in a controlled manner with an incoming video signal 4, producing an output video signal 5. Sound effects are also generated, merged with the input audio signal and leave the AVAPS 1 in the form of audio out 6. Limitu,, run-time animation control can be provided by the user, and a limited amount of synchronisation can occur with other the AVAPSs.

The overriding emphasis in the AVAPS i is on giving the user easy access to sophisticated but canned animations, which the user can modify in simple ways and combine to achieve the desired effect. It will be appreciated by those skilled in the art that whilst the AVAPS 1 operates on both audio and video information, either input or canned, corresponding arrangements can be configured for operation on either audio or video information only.

A clip is a predesigned 'canned' animation component that contains graphics and sound. A clip is capable of manipulating shapes, position, size, and colour and synchronising with the user by means of a keypress).

Clips can contain user-editable components (such as editable text) and fixed (noneditable) components. The author can specify defaults for all of the user-editable components, so that a clip can be used without any customisation.

Each clip provided by the author has a duration type. The type is one of: fixed, where the clip can determine its duration without outside constraints by examining its own components). For example, a clip may show scrolling credits at a constant speed. The number of lines of text to scroll will determine the duration of the clip.

unknown, where the clip has no idea of its duration, nor can it easily work it out.

For example, if a clip waits for the user to press a key, the clip will not know ahead of play-time how long it will be before the user presses the key; and infinite, where the clip has no duration limit, and will display for as long as the user wishes it to. An example of this is a background image behind some sequencedanimation. The background image should be displayed as long as necessary in the animation. Typically, a clip with infinite duration will need to be tied to the duration of another clip. For instance, a background image should be tied to last as long as the scrolling credit clip lasts: The preferred embodiment of a storyboard system is configured to allow a user to: select a clip from ROMCards 2; (RT024)(259858) edit those components of a clip that are editable; play a clip; and sequence, layer and synchronize multiple clips, The user can decided to combine clips into a storyboard. The storyboard represents the display order and sequence of the clips. It indicates which clips are run together, what dependencies they have on their starting and finishing times, and what display order in terms of relative visual priority each clip has.

Consequently a storyboard represents: time order. This includes representing clips that run in parallel (clip A run at the 10 same time as clip and sequence (clip A follows clip B); display order. This includes representing the relative display order priorities of clips that are active at the same time (clip A should be displayed above clip and synchronization. This includes starting and stopping clips together (clip A should terminate when clip B terminates) as well as clip markers for random access during playback.

Users can perform the following functions with a storyboard in an easy to use manner: create; save to a RAMcard; ooe• load from a RAM/ROMcard; edit (both the storyboard and the clips in the storyboard); and play.

S"Referring now to Fig. 3 there is shown a simple storyboard containing 4 columns, two rows, and therefore 8 cells. A storyboard is conceptually like a spreadsheet, with rows, columns, and cells. Consecutive columns represent consecutive time segments, and rows represent display order within a time segment. Both dimensions are relative rather than absolute. Thus each column may represent a different length of time.

A storyboard is read from left to right, one column at a time. Each column represents one time segment, and the cells in each column represent relative display levels duringthat time segment.

Each cell within a column may contain one of the following items: nothing. If a cell contains nothing, it means that there is nothing of consequence happening during that time segment on that relative display level.

a clip. If a cell contains a clip, it means that the clip starts at the beginning of the cell's time segment at that cell's relative display level.

(RT024)(259858) -r 3 -6a continuation marker. If a cell contains a continuation marker, it means that the clip that began in an earlier time segment on this same row is continuing through this cell's time segment.

Each cell also has a stopped flag, which indicates if the clip is to be forced to stop at the end of that cell's time segment. Additionally, each column can have a clip marker associated with it as long as there is a clip in that column.

The length of time represented by a particular column is defined to be the minimum time of all cells in that column. All clips that are in a particular column are defined to start together. Their finishing times will vary depending on the length of each clip. The length of S 10 the column will therefore be the length of the shortest clip. All other clips will then ;continue into the next column (or time segment) via a continuation marker. The length of the next column will be the shortest of all remaining time for each clip and any new clips '.that have started in that column. For the purposes of calculating a minimum, an unknown duration is treated as zero time, and an infinite duration is treated as infinite.

From this it can be seen that continuation markers are automatically added and removed by the AVAPS 1, and therefore the number of columns that exist in a storyboard changes dynamically based on the number of time segments that are required to represent the clips.

•A clip can be terminated prematurely by setting the stopped flag in a cell. This can be S 20 accomplished directly by changing the stopped flag explicitly), or indirectly by stauting another clip where a continuation marker existed previously). In either case, the clip will then terminate at the end of the column, regardless of how much time was remaining on that clip.

Storyboards also contain user-defined clip markers. Each column can have a clip marker associated with it provided there is at least one clip in that colunm The clip marker refers to all clips that begin in the column.

Given the above representation of a storyboard, it is possible to interpret the meaning of a given storyboard.

Referring now to Fig. 4 there is shown a storyboard having two clips in a sequeace, clip A and clip B. The first column is of length clip A, and the length of the second column is of length clip B. Consequently clip A is run first, then clip B is run when clip A has finished. This is true regardless of the length of A or B. If clip A had an unknown duration, then clip B would be run only when clip A finished (however long that was). If clip A had an infinite duration, then clip B would never be run, as clip A would never terminate.

(RT024)(259858) I~ II Referring now to Fig. 5 there is shown a storyboard having the same two clips in parallel. The first column has the two clips, so its length is of the shortest of the two clips clip As clip A has finished by the end of the first time-segment, it does not continue to the next time-segment. Consequently, the second column has a continuation marker in the cell of the same row that clip B was in, to indicate that it continues to the next time-segment. The minimum time of the clips active in this second time-segment is the time (clip B clip A).

The relative display order of the two clips is apparent from the storyboard. Clip A is S. displayed on a higher priority to clip B, therefore when rendering the storyboard, if an object from clip A is drawn overlapping an object from clip B, the viewer will see the object from clip A on-top-of the object from clip B.

The first column represents the starting time of the animation, the second column *represents the time when clip A has finished (and B is continuing), and the third column represents the time when clip B has finished.

Referring now to Fig. 6 there is shown a storyboard example that could be used for a presentation describing the uses of a particular product in two countries. Clips A, B, and C (the first column) may represent the animated company logo and title. Clip D may represent the text describing the product's use in the first country, while clip G may represent the text describing the product's use in the second country. Clip E represents the graphic of the product itself, while clips F and H represent the graphical maps of the two countries.

*9 9 The first column contains clips A, B, and C. If clips A and B have their stopped flags set, they will both terminate when clip C terminates. i.e. the animated title and logo will be on the screen for the duration of the delay clip C.

The second column shows that clips D, E, and F are started simultaneously when clips A, B, and C terminate. Clip F is terminated by the shortest clip in the column (clip D), and clip E continues after clips D and F are terminated. The third column shows that clips G and H start when D and F terminate, and finally all clips terminate when clip G finishes.

The storyboard is changed by performing actions on cells and columns in the storyboard. Each action affects the cell, and then the storyboard is re-evaluated.

Effectively, the minimum time for the cell's column is recalculated, and this calculation is propagated as far as required.

A full description of one possible storyboard structure suitable for use with the preferred embodiment is given in Australian Patent Application No. S ti. L /<MA4 52 entitled "Story Board System for the Simultaneous Timing of Multiple I I -8- Independent Video Animation Clips", claiming priority from Australian Provisional Patent No. PL 6835, filed 15 January 1993, the contents of which are hereby incorporated by cross-reference.

Referring now to Fig. 7 there is shown the example of an AVAPS system such as that shown in Fig. 2 in still more detail. In order to display the clips corresponding to those set out in the story board, the corresponding clips must be rendered by the AVAPS 1. The AVAPS 1 has a host processor 11 that is responsible for the formation of storyboard structures in its system RAM and the subsequent rendering thereof by an RTO processor 17 such as that disclosed in Australian Patent Application No. 38244/93, entitled "A Real- Time Object Based Graphics System", claiming priority from Australian Provisional Patent No. PL 2147, filed 29 April, 1992, and the contents of which are hereby incorporated by cross-reference.

The following high level functions are required to be performed by the rendering software system of the host processor 11 which acts as a "Renderer": play a storyboard from a column; pause; continue; stop; notify; and get an error message.

These different functions are described in the following paragraphs: Play Storyboard This causes the renderer to stop preparing the column that is due to start next, and instead, to prepare the specified column from the specified storyboard. When the preparation of the column is complete, the currently animating column from a storyboard will stop animating, and the new column (and storyboard) will animate in its place. When the end of the storyboard is reached no more clips to play in any subsequent columns), the animation will loop to the first column in the storyboard.

The following parameters are required: the address of the storyboard to play; the column number to start playing from; a flag indicating whether a notify event is required; and a column to be used for the notify event (only valid if a notify event is required).

The following errors are possible during execution of this command: external variables unresolved; (RT024)(259858)

I

-9- ROMcards not present; column out of range; and column is empty.

Pause This causes the renderer to freeze the current display so that no further animation takes place until a Continue call occurs.

No parameters are required.

The following errors are possible: S1 storyboard is not playing; and storyboard already paused.

Continue This causes the renderer to continue an animation that was paused by the Pause command, or by a Pause Handle. Continuing a Pause command takes precedence over continuing an internal Pause Handle.

No parameters are required. The following errors are possible: storyboard is not playing; and storyboard is not paused.

Stop This causes the renderer to stop playing any animation, and to set the screen to black.

20 No parameters are required. The following error is possible: storyboard is not playing.

Notify This causes the renderer to post a PlayingColumn Event to a Storyboard application 34 (seen in Fig. 8 and described further hereinafter) when the specified column is reached.

The following parameters are required: the column number to cause the PlayingColumn Event; and a flag to determine whether the Event should be posted if the current column is past the specified column (this can be used to post an event next time through the looping of the storyboard).

The following errors are possible: storyboard is not playing; and column out of range.

Get Error Message The parameter for this operation is: the error code to get the message for.

(RT024)(259858) 9 1-' This cannot fail, and always returns an error message. If the error code passed to the function is out of range, an out-of-range error message will be passed back.

Valid errors can include: no error; external variables unresolved; ROMcards not present; column out of range; storyboard is not playing; storyboard is not paused; column already paused; column is empty; and illegal reference number.

SOverview of Data and Processes in Storyboard Renderer Referring now to Fig. 7 and Fig. 8, the storyboard system involves the use of several programs or processes and data structures which reside in system RAM 12, system ROM 13 in addition to ROMCards and RAMCards being memory card 14 and memory card These programs and data structures are manipulated by the host processor 11.

Fig. 8 shows the main data and process flow 16 in the host processor 11 of the AVAPS 1 when rendering a storyboard. Data structures are indicated by squares and S: 20 processes are indicated by circles. The various processes are implemented by means of a multi-tasking operating system and process stack. Each shaded band indicates a runtime priority level of the enclosed processes. Hence, the storyboard application 34 runs at the lowest priority level it obtains access to the processor 11 when none of the processes above it are using the processor 11). Each data store and process of Fig. 8 is described S 25 below.

Render List A render list 23 is provided as a list of pointers to the RTO processor 17 object lists to be prepared and subsequently rendered by the RTO processor 17, The list is terminated by a NULL pointer. The rendering of the render list 23 is performed as previously described in the above mentioned Australian Patent Application No. 38244/93, previously incorporated by cross-reference.

The RTO processor driver An RTO processor driver 20 is responsible for displaying the render list 23 via the RTO processor 17. There may be no entries in the render list 23 in which case the image (RT024)(259858) -11displayed by the RTO processor 17 should be a standard background colour emitted by the RTO processor 17 defined to be at level 0, effects 0.

The RTO processor driver 20 is invoked by various interrupts from the RTO processor 11, including an Image Render Complete and an Image Prepare Complete interrupt, as well as directly by a Render Manager 25 when starting or stopping the rendering process.

After the RTO processor 17 has finished rendering the current field, the RTO processor driver 20 calls a field sync process 21 routine so that any inter-field processing can be accomplished.

The RTO processor driver 20 software runs on the host processor 11 and uses the RTO processor 17 for rendering to the screen, as outlined in the aforementioned patent **application.

A display list comprising the objects pointed to by the render list is formed and passed to the RTO processor driver 20 which start the hardware to display the display list to 15 the screen. As well as displaying the display list, the RTO processor driver 20 calls the field sync process 21 on the occurrence of a field blanking sync which allows the application 34 to be called via the event list 33 at every field blanking time. The application 34 and event list 33 are user definable to achieve a particular visual effect. This can be used to update the display list to allow animation to oc,,r. This function can also use the functions of the 20 RTO processor driver 20 to swap the display lists at this time if desired.

If an RTO processor error occurs at any time the RTO processor driver 20 calls a application installed error function. This function accommodates and informs the higher level software that an error has occurred.

Whilst generally the RTO processor driver 20 will be in use at all times, it is not actually required to run continuously.

Once the AVAPS 1 is started, the RTO processor driver 20 calls an initialization routine which ensures all the required hardware is setup correctly.

The function used to maintain the display list is then installed. This function is called every frame blanking period, by the RTO processor driver 20, and is responsible to ensure that all required modifications to the display list are carried out. This function is passed a single parameter being a pointer to a structure or a class.

The error handling function is also installed at this stage. This function is called when an error in the RTO processor 17 occurs. This function is expected to tidy up and signal the application 34 software that an error has been detected.

(RT024)(259858) -12- Once ihe initial display list is ready, a call to a renderer 24 is made by the field sync process 21, and the renderer 24 is started. This attaches a function to a RTO processor int r-upt.

Once started, the display list is rendered by the RTO processor driver 20. The display list is not, however, static as the renderer 24 is called each frame blanking period and the renderer 24 can modify or replace the display list as required.

If an RTO Processor error occurs, the RTO processor driver 20 will set an internal display list pointer to NULL, causing the screen to go blank on the next field and a error function can be called.

The renderer 24 can then queue an event for the application 34 which then can determine what should be displayed. The RTO processor driver 20 will continue operation uncer the assumption that the error function has established the real display list to something appropriate, for example, a NULL, and that the update fiunction will operate I correctly on the new display list.

15 Thus, in an error condition there will be a blank frame displayed and then the RTO processor driver 20 will display whatever the error function has placed in the display list.

This could be further blank frames or possibly a pre canned-error message.

In order to display an image to the screen there are a number of functions that must be performed. Firstly, the RTO processor 17 must prepare the image and then it must be 20 rendered to the display, generally a video display unit (VDU). For an image to be displayed continuously it must be prepared and rendered for each field in which it will be displayed.

This means that the RTO processor 17 must be preparing the next field as it renders the current one. It also means that there is some delay after a renderer 24 has been called to when the first field can be displayed.

S Within the RTO processor driver 20 there are two RTO processor interrupt handlers.

One to actually start up the display and one to handle the normal running of the RTO processor 17. The startup harndler is responsible for getting the RTO processor 17 going in the render/prepare cycle as well as synchronising the RTO processor 17 to the appropriate odd/even field. Once configured, the startup handler attaches the normal RTO Processdr interrupt handler to allow it to carry on.

The startup handler only displays a dummy image made up of Quadratic Polynomial Fragments (QPF's) upon which the preferred embodiments of RTO technology is based.

Thus the screen will be blank until after the RTO processor 17 is operating and the real interrapt handler is used. The time taken to set up the RTO processor 17 can be 5 or 6 fields.

(RT024)(259858) -13- Interrupts from the RTO processor 17 are generated in resp, to either a render completing, a prepare completing or an error occurring. If the RTO Processor interrupt was caused by a prepare completing, then the display list is checked to see if there are more elements to t. repare. If there are more elements to prepare then the next element will be sent to be prepared. Otherwise, a flag is set indicating that the current preparation is complete.

If the RTO Processor interrupt was caused by a render completing then the preparation flag is checked to see if the prepare for the frame about to be rendered has completed. If the prepare is completed then the render for the next field is setup to be started automatically on the next active edge of a filed sync. If the prepare has not completed an error message can be printed to the terminal, if one is connected, and a halt of all processing called.

If the interrupt was caused by an error the handler will check that error against a current set of flags to see whether the error is to be reported. If the error is to be acted upon i then the current prepare will be aborted and a prepare of the dummy QPF initiated. The 15 application installed error function will be called. The RTO processor driver 20 assumes that the application installed error function will fix all the problems and so the handler sets itself up to carry on as normal, after displaying the dummy QPF.

The field sync process 21 The field sync process 21 is responsible for synchronising the renderer 24 with the 20 display device and for ensuring that any sound is adequately sent to the audio channel 18. It firther ensures that the renderer 24 fields are updated to the screen in one field blocks.

The first task of the field sync 9 is to synchronise the colours in a colour and mixing unit 19 (CMU), seen in Fig. 7, with the objects being drawn by the RTO processor 17. The RTO processor 17 takes a field to prepare, and then renders in the subsequent field. If it updated the render list 23 being drawn in the previous field, then it must update a colour and mixing unit 19 (Fig. 7) during this field sync so that when the field is rendered, the correct colours will be in the colour and mixing unit 19. The colours cannot be written to the colour and mixing unit 19 when the field is being prepared as a different image may be being rendered which requires different colours. It therefore calls a CMU Driver: Unit 10 to install the appropria-e colours.

The next task is to call a Sound Driver 37 (Fig. 8) to send the next batch of sound data to audio channel processor 18, seen in Fig. 7.

The next task is to update the render list 23 for the RTO processor driver 20. The field sync process 21 routine checks to see if the renderer 24 has finished calculating the 'next-field' (it may take the renderer 24 many fields to calculate a complicated image). If (RT024)(259858) -14the 'next-field' is ready, the current column being rendered will be used as a basis for writing a new render list 23, and for submitting a new sound to the Sound Driver 37.

Now that the field sync process 21 process has synchronised with the renderer 24, it clears the 'next-field ready' flag (which the renderer 24 had previously set) and causes a software interrupt which causes the renderer 24 to be invoked once the RTO processor driver and Field-Sync interrupt routines have terminated.

As long as a global 'DisplayFrozen' flag is TRUE, the current field will continue to be displayed the 'next-field ready' flag is ignored. This mechanism allows the application 34 to freeze the animation at any field, although any sound will continue to play.

The colour and mixing unit driver 29 The colour and mixing unit driver 22 is responsible for updating the colour and mixing unit 19 using colour entries. It is called by the Field-Sync process.

The sound driver 37 15 The sound driver 37 is resnonsible for updating the Audio Unit 18 a little at a time in each field that is rendered for display. It keeps track of how much it has written against how much the audio unit 18 has played, and attempts to update as much as possible in a given field. It has the ability to play a particular sound N times, where N can be a very large number. Only one sound can be active at a time.

20 Columns The data and process flow of Fig. 8 also contains information about the current state of a set of clips that are animating in a particular column of a storyboard. There are several types of columns maintained in the system including: 25- current column 27; 25 first column 26; next column 28; and previous (prev.) column 29.

The clip information is comprises a pointer to the clip data, a flag indicating if the clip terminates at the end of the column, a flag indicating if the clip is continuing fror'the previous column, and a flag to indicate that this is a new column (that hasn't been rendered yet). The different types of columns are described more fully as follows: The current column 27 This block contains the state information about the various clips that are currently animating in a particular column of a storyboard. It contains the current set of colours, (RT024)(259858) object lists, graphics, handles and sounds as well as information about the attributes that are animating.

The first column 26 This is used to begin an animation starting at a particular column. When this column is ready, the renderer 24 knows to use it to replace current column 27.

The nextcolumn 2 This is used to hold the initial state information for the column following current column 27 in the storyboard and the components of current column 27 that no longer continue into the next column. Under normal circumstances, next column 28 will be used to make a new current column 27 and prev column 29 when current column 27 has terminated. If a first column 26 is available instead, then it is used in preference, and the next column 28 is discarded to prev column 29.

The prey column 29 This is used to hold the state information about those previous columns that are no 15 longer being calculated, but are still being displayed by the RTO processor 17 for the next field only.

The renderer 24 The first task for renderer 24 is to cause all the colours that changed last field to be updated, ensuring that by the end of this field the colours will be ready for installation into 20 the colour and mixing unit 19 by the field sync process 21 routine.

The next task for renderer 24 is to take prev column 29 (lists those clips that terminated last field) and queue a Cleanup Event in the event list 33. These can now be safely destroyed since it is at least one field sync since they were marked for termination, and therefore the object lists and colours have all been completely read from memory.

25 The renderer 24 must also determine if a Notify event should be sent to the application 34 (due to a column being rendered).

The renderer 24 is also responsible for determining that a particular column has finished animating, and replacing the currently animating column by a new column. This can be the next column 28 in the same storyboard, or the First Column of a completely new animation (from the same or a different storyboard). Consequently, the next task that the renderer 24 must perform is to replace the active column by a new one if the current column (current column 27) has expired. Expiration can occur by the column terminating naturally or by a new render instruction via first column 26. The terminating column information is placed into a new prev column 29.

(RT024)(259858) -16- If current column 27 has terminated, the current animation continues while waiting for the next column 28 to be ready. If a first colunm 26 is available (which also causes current column 27 to terminate), the renderer 24 adds all of the next column 28 to the prev column 29 (as the data must be de-allocated), and makes first colunm 26 become current column 27. Otherwise the components that are flagged to terminate in current column 27 are placed in prev column 29, and next column 28 is made current column 27. It must also update all the base colours of the clips that are continuing on.

Once the renderer 24 has created a new current column 27, it clears the next-columnready flag and sends a software interrupt to prepare next column 30 to allow the next column to be calculated.

The final task for renderer 24 is to calculate the next-field to be displayed for the continuing elements of current column 27. As such it evaluates the timelines for the various attributes (including colour, translation, and camera) that describe a particular image (it doesn't need to evaluate the timelines for the first field since the initial values have been 15 evaluated already). Whnen it has calculated the next field's values, it sets a 'next-field ready' flag to inform the Field-Sync routine that the entire column is ready for visual synchronisation.

The prepare first column 31 This module takes a specified column from a storyboard, and generates a first 20 column 26 for renderer 24 to use.

When the first column has been prepared, prepare next column 30 informs renderer 24 via a 'first-field ready' flag.

The prepare next column This process creates the next column from the storyboard after the current column 27 and generates a next column 28 which holds the next column to render, and a prev column 29 which contains the clips to be de-allocated as they will no longer be displayed.

When the next column has been prepared, prepare next column 30 informs renderer 24 via the 'next-field ready' flag.

The storyboard 36 This contains the cell structure of clips and clip markers, together with the various flags that are set by the renderer 24 such as current_column and #active_clips. The application 34 builds and edits storyboards, while the renderer 24 is responsible for rendering them.

(RTO24)(259858) -17- The rende The render manager 25 of Fig. 8 is responsible for all of the high level rendering commands such as! Play, Stop, Pause, Continue, and GetErrorMsg. It is called by the application 34.

The application 34 This is the AVAPS application and user-interface. It edits storyboards and plays them via the render manager 25. It reads the event list 33 for actions to perform, some of which are user-driven and some are events from the rendering system. It is also responsible for deallocating the render columns that had been allocated by prepare first column 31 and prepare next column The cleanupcolumnl This is a list of events in priority order that is maintained by the event manager 32.

The renderer 24 post events to the event list 33 as required. Events include: Cleanup events, which indicate a call to the Cleanup Prev Column; 15 Notify Events, which indicate that the renderer 24 is rendering a particular column; and Error Events which indicate that an internal error has occurred during the render.

Structure of Data Stores The various data structures manipulated by the various process used in Fig. 8 will 20 now be described in further detail The render ist23 Referring now to Fig. 9 there is shown an element of the render list. The render list 23 (Fig. 8) is a array of pointers to the RTO processor Object Lists. It indicates which of the RTO processor object lists need to be prepared during the following field. The structure of a single entry is in the following figure. The list is terminated by a NULL pointer. A list containing only contain one pointer (the NULL pointer) is valid.

Sound Information Referring now to Fig. 10 there is shown the sound information structure. The sound information structure is used by the sound driver 37 (Fig. 8) to produce sound effects during AVAPS animation. Only one sound effect can be produced at any time. When there is sound data available, it is submitted to the sound manager and then cleared out. The sound information structure is as follows: Sound data is a pointer to the ROMcard sound data that represents one play through of the sound; (RTO24)(259858) 18- Sound duration indicates how many time units the sound runs for when played once; and Iterations indicates how many times the sound should be played. 0 indicates an infinite number.

Camera Information Referring now to Fig. 11 there is shown the camera information structure. A single pointer (CurrCamera) points to the current Camera Info structure that is used by the renderer 24 to transform the final view of a scene to screen coordinates. Consequently, only one camera can be active at any one time, and the data used has the following structure: X Translation is the camera's current X coordinate value; Y Translation is the camera's current Y coordinate value; Z Translation is the camera's current Z coordinate value; Zoom CHTransform is the camera's current zoom coordinate value; TransChanged is a boolean that indicates if any camera translation value has 15 changed this field; ZoomChanged is a boolean that indicates if the camera zoom value has changed this field; and TimelineRef points to an ordered set of timeline references to be evaluated, each of which refers to one of the camera's values. This value may be NULL, indicating that 20 there are no timelines to be evaluated.

The Timeline Reference Referring now to Fig. 12 there is shown a timeline reference structure. The timeline reference structure is pointed to by the camera structure of Fig. 11. Each component of the timeline reference handle that is being animated has a number of parts to the transformation. The structure is as follows: -Type determines the type of the data that the timeline refers to. It can be used for bounds checking during timeline evaluation. Valid values are: XTranslation, YTranslation, ZTranslation, Zoom, XScale, YScale, Existence, Object, Luminance, Chrominance, Mix, MinTend, and RndMask; Next is a pointer to the next timeline reference in the list. A NULL pointer indicates the end of the list; Base Value is a pointer to a 32 bit value defining the current value of the attribute; Changed Flag is a pointer to a boolean that is set if the value pointed to by Base Value is updated during the evaluation of the timeline reference; (RT024)(259858) -19- Time Remaining indicates the number of ticks that remain in the current keyframe before the next keyframe is used; TL Keyframe points to the keyframe of a timeline (likely in ROM) for the attribute; Reference Value points to the value being referenced by a keyframe. When the keyframe is first encountered, the address of the value being referenced is stored here for use in calculating based on a reference. When the reference is an external, it points to the location of the value to be used; AValue contains the current value for the Avalue (velocity where simulated springs are designed to be used) for the keyframe being evaluated; AAValue contains the current value for the AAvalue (acceleration in the case of simulating springs) for the keyframe being evaluated; Reference Base is a pointer to the first element to be used as a reference if a timeline's keyframe is a reference. It is used in conjunction with the Data Size to 15 calculate the address of the value being referenced (stored in Reference Value); Data Size indicates how large the base record is for referencing purposes. The reference address is obtained by adding (Index# Data Size) to Reference Base; and Factor is a pointer to a factor to use if the handle should be applied to the base value of the attribute. This is only necessary when the attribute is being referred to by a 20 simulated spring. If a timeline times out, and this value is non-NULL, it is attached to the current handle's ApplyHandle list. Non-animating attributes that are referred to by springs are in the handle's initial ApplyHandle list.

The Render Column Referring now to Fig. 13 there is shown the structure of the render column. The S 25 render column defines a column with regards to a storyboard. It has the following structure: Storyboard is a pointer to the storyboard structure being rendered; Column# indicates which column of the storyboard is being defined in this Render Column; #Clips indicates how many clips are being animated this column (it also indicates how many Render Clip References will be in the array); RenderClipRefs is a pointer to an array of Render Clip References, each of which holds the current status of a clip being rendered from the column; and NewColumn indicates if the column is new or not. A new column is one that has only been prepared, not rendered.

(RT024)(259858) The Render Clip Reference Referring now to Fig. 14 there is shown the structure of a render clip reference. Each render clip reference has the following structure: RenderClip is a pointer to a Render Clip record that holds the current status of the clip being rendered.

TerminateFlag indicates whether the clip will terminate at the end of this column.

RenderClip Referring now to Fig. 15 there is shown the structure of a render clip. The render clip record holds the current status of a clip being rendered. It has the following structure: Clip Ref points to the RAM storyboard clip reference structure that the clip is representative of; NonGraphic RHandles points to the non-graphic render handles; Graphic RHandles points to the graphic render handles that are currently animating they have completed their delay period). These handles have their 15 timelines evaluated if they are active. Anything delayed will not be visible (so the objects in RTO processorObjListl and RTO processorObjList2 would both be set to

NULL);

RTO processor ObjListl points to a RTO processor object list that represents a field of an animation; 20 RTO processor ObjList2 points to a RTO processor object list that represents a second field of an animation; Selectl/2 indicates which of the two RTO processor object lists are being used by which process. RTO processorlRender2 indicates that the RTO processor 17 is reading RTO processorObjListl, and render process is calculating RTO processorObjList2.

RTO processor2Renderl indicates that the RTO processor 17 is reading RTO S" processorObjList2 and the render process is calculating RTO processorObjListl; LevelsChanged indicates whether the clip's colour levels have changed, implying an update of all other priority levels in the RTO processor object lists and in the colour and mixing unit 19; and TimeRemaining indicates how many fields remain for the clip (0 infinite) before it terminates.

The Base Render Harl Referring now to Fig. 16 there is shown the structure of the base render handle. The render handle record holds the current status of a handle within a clip. It has the following structure: (RT024)(259858) -21- Next is a pointer to the next handle in the lisc. A NULL pointer indicates there are no more handles in the list; Delay is the number of ticks before the clip becomes active. When this count is 0, the handle is made active; Type is one of Sound, Delay, Pause, Camera and Graphic. It indicates the remaining structure of the handle.

The Sound Render Handle Referring now to Fig. 17 there is shown the structure of the sound render handle.

When it becomes active, it is submitted to the Sound Manager and then ignored. A sound handle has the following structure Base Render Handle as described previously in the paragraph entitled base render handler; and Sound Info as described previously in the paragraph entitled sound information.

The Delay Render Handle 999909 A delay render handle has no extra data. The delay in the base record would include the delay of the handle. When the delay render handle is made active, it is ignored.

Thie Pause Render Handle A pause handle has no extra data. When the pause render handle becomes active, it marks the column as being paused (which means that the column will continue to animate 20 until the pause is satisfied. A pause can be satisfied by either a Continue call to the Render Manager, or if the column terminates and the clip that contains the pause is terminating at the end of the column.

S The Camera Render Handle Referring now to Fig. 18 there is shown the structure of the camera render handle.

25 The camera render handle contains information necessary for the manipulation of the camera. Its structure is: Base Render Handle is as described in the previous paragraph entitled 'Base Render Handle'; and Camera Info is described in the previous paragraph entitled Camera Informatipn.

The Graphic Render Handle Referring now to Fig. 19 there is shown the structure of the graphic render handle.

The graphic render handle contains information necessary to animate a graphic handle and its graphic (and colours if not sharing colours). Its structure is as follows: Base Render Handle is as described previously; (RT024)(259858) -22- Arena is one of Foreground, Midground and Background, indicating the animation arena of the graphic; X HTransform is a Handle Transform structure describing the X transform on the graphic; XScale HTransform is a Handle Transform structure describing the XScale transform of the graphic. This must follow the X HTransform structure; Y HTransform is a Handle Transform structure describing the Y transform on the graphic; YScale HTransform is a Handle Transform structure describing the YScale transform on the graphic. This must follow the Y HTransform structure; Z HTransform is a Handle Transform structure describing the Z transform on the graphic (valid only if Arena Midground); Existence HTransform is a Handle Transform structure defining the visual existence of the graphic; 15 TimelineRef points to an ordered set of timeline references to be evaluated, each of which refers to one of the handle's attribute values. This pointer may be NULL, indicating that there are no timelines to be evaluated; X HFactor holds the value (XTranslation XScale) for use in shape and colour transformation calculations involving simulated springs. It is recalculated whenever an X handle value changes; Y HFactor holds the value (YTranslation YScale) for use in shape and colour transformation calculations involving simulated springs. It is recalculated whenever a Y handle value changes; X Changed is a boolean that is set if either the XTranslation or XScale changed 25 during the field; Y Changed is a boolean that is set if either the YTranslation or YScale changed S during the field; Z Changed is a boolean that is set if the ZTranslation value changed during the field; Exist Changed is a boolean that is set if the Exist value changed during the field; APixelScale is the value to use as the scaling information to place in the RTO processor 17 for each of the shapes in the graphic; AAPixelScale is the value to use as the scaling information to place in the RTO processor 17 for each of the shapes in the graphic; (RT024)(259858) 23 Apply Handle points to a set of timeline references, each one referring to a value that is itself not currently animating, but is referred to by an attribute via a simulated spring, and as such, needs to have the handle's transform specifically applied to its Base Value as the handle animates. If the handle is not animating TimelineRef is NULL), then Apply Handle is not required to be called; Shape Info is described below; and ColourInfo is described below.

The Handle Transform Referring now to Fig. 20 there is shown the structure of the handle transform. Each component of the handle that is being animated has a number of parts to the transformation.

The structure is as follows: Base Value is a 32 bit value defining the local current value; and Camera Value is a 32 bit value defining the attribute's value after the camera has been applied to it.

The Shape Information Referring now to Fig. 21 there is shown the structure of the shape information. The Shape Information is a structure that contains information about the various shapes that are animating in the graphic. The structure is as follows: Shapes indicates how many shapes represent the graphic; Shapes points to an array of #Shapes RenderShapes structures. Each one contains the current state of the shape in the graphic; TimelineRef points to an ordered set of timeline references to be evaluated, each of which refers to one of the shape's attribute values. This pointer may be NULL, i indicating that there are no timelines to be evaluated; RTO processor ObjectListl is a pointer to the RTO processor objects within the clip's first RTO processor object list; and RTO processor ObjectList2 is a pointer to the RTO processor objects within the clip's second RTO processor object list.

The Colour Info Referring now to Fig. 22 there is shown the structure of the colour information.

Colour Info is a structure that contains information about the colours and effects that are used by the graphic in the handle. Its structure is as follows: Colours indicates how many colours are stored in the graphic; Base Colour indicates the base colour number of the graphic's colours. This number is added to the offset colour to obtain the RTO processor level; (RT024)(259858) -24- The CMU Colour Group is a pointer to the Colour and Mixing Unit record that defines all the colours for the graphic; TimelineRef is a pointer to a list of Timeline References that update the values of the colours and effects in the Colour and Mixing Unit Colour Group. If this is NULL, then none of the colours are explicitly animated (blends may be implicitly animated as their handles move); Animated X Blends is a pointer to a list of animated blend records for blends in X; Animated Y Blends is a pointer to a list of animated blend records for blends in Y; Changed Colour is a flag that determines whether any of the colours in the Colour and Mixing Unit Colour Group was updated during the calculation of the next field; #OpaqueEffects indicates how many opaque effects (other than the default one) are stored in the graphic; Base Effect indicates how many transparent effects are stored in the graphic.

Base Effect indicates the base effect number of the graphic's effects. This number is added to the offset effect to obtain the RTO processor effect mode; Changed Effect is a flag that determines whether any of the effects in the CMU Effect Group was updated during the calculation of the next field; and CMU Effect Group is a pointer to the Colour and Mixing Unit record that defines all the effects for a graphic.

The AnimatedBlend Referring now to Fig. 23 there is shown the structure of an anit: iteL Jlend. Blends make use of a start and extent value in the direction of the blend. A blend may be defined as static (in a graphic), but the graphic may be placed in a graphic handle that animates.

I' urther, a graphic handle may be in an arena that causes further computation to be performed based on the camera value. The animated blend entry allows the rendering software access to only those blends which may change.

S' Each entry has the following structure: Next points to the next Animated Blend entry in the list; Start Ref is a structure that defines the high precision values that are needed to define the start position of the blend. Its structure is described below and shown in Fig.

24; Extent Ref is a structure that defines the high precision values that are needed to define the extent of the blend. Its structure is described below and shown in Fig. 24; and (RT024)(259858)

I

Changed is a flag that determines if the particular value has been changed via a timeline during the field. If it has changed (or another special circumstance has occurred), then the value must be written to the colour and mixing unit 19 via the CMU Value pointer, Fig. 24 shows the Start Ref and Extent Ref structures which include: Base Value contains the high precision value of the start or extent amount before any transformation has been applied to the graphic that it is in; Handle Value contains the high precision value of the start or extent amount after the handle transformation has been applied to the Base Value. It is only calculated if the colour's start or extent is referred to by another colour via a spring; and Colour av'i Mixing Unit Value is a pointer to the low precision start or extent values actually stored in the Colour and Mixing Unit record.

The Render Shape Referring now to Fig. 25 there is shown the structure of the render shape. A render shape represents the current settings of a shape within a particular graphic. It has the following structure: #Objects defines the maximum number of RTO processor objects that comprise the shape; X GTransform is a Graphic Transform structure defining the current X coordinate of the shape within the graphic: Y GTransform is a Graphic Transform structure defining the current Y coordinate S: of the shape within the graphic; v,,I'ject GTransform is a Graphic Transform structure defining the current objects used to define the shape within the graphic; and Written Last Field is a set of flags that indicate what new parts of the shape were written to the RTO processor object list last field. The actual information written last field is a combination of what was newly written the field before last, together with the new information that was written last field. This causes propagation of a value from one object list to the next. Each of the bits has a meaning: as indicated in the table below: Bit# Meaning 0 Updated Pixel and Pixel_scale 1 Updated Line and Line_scale 2 Updated DelaPixelScale and DeltaDeltaPixelScale (RT024)(259858) s -26- Bit Meaning 3 Updated Colour Levels 4 Updated Object-list (and therefore everything) Reserved (must be 0)

L

The Graphic Transform Referring now to Fig. 26 there is shown the structure of a graphic transform. Each attribute that is being animated within a graphic has several components to its transformation. The structure is as follows: Base Value is a 32 bit value defining the local current value of the attribute; Handle Value is a 32 bit value defining the value of the component after the handle has been applied to the base value. This is only computed if the shape is referred to by another via a simulated spring; and Changed is a boolean that defines whether the Base Value has changed since the 15 last field.

The event list 33 The event list 33 structure of Fig. 8 is private to the event manager 32, but the renderer 24 piocess are able to post events to the event manager 32 (which are placed in the event list 33). The types of events that are possible are: Cleanup Event, containing a pointer to the prev column 29 to be cleaned up; Notify Event, containing the column number that is being rendered; and RenderError Event, containing the error number that caused the renderer 24 to stop.

Th Structure of Processes Referring again to Fig. 8, the main data and process flow 16 is made up of a number of process as previously described. Each of these process is made up of a number of subprograms which implement the overall main data and process flow 16.

The following paragraphs setout, in more detailed pseudo code, the overall structure of the computer processes running on the host processor 11, and their interaction withthe data structures as previously described.

1. The Public Interface of the Render Manager StarttUp *RenderList NULL ColourMixingUnitDriver->StartUpO SoundDriver->StartUp() ThisColumn.rc NULL (RT024)(259858) 27 ThisColumn.NextFieldReady TRUE NextColumn.rc NULL NextColumn.NextFieldReady TRUE FirstColun.preparing FALSE FirstColumn.rc NULL FirstColumn.NextFieldReady FALSE ColumnTerminated FALSE ErrorEncountered FALSE DisplayFrozen =FALSE ColumnPaused =FALSE RenderActive FALSE StopRender FALSE RTOProcessorDriver->Startup(& RTOProcessorEror, &FieldSync) iPlay Storybhoard Do the old error checking routine...

if (!sb) return InvalidStoryboard if (sb->#UnresolvedExternals) return UnresolvedReferences if (sb->#UnresolvedClips) return ROMCardsNotAvailable if (notify AND (column sb->#columns)) return ColumnOutOfRange 25 Theoretically all is only remains to attempt to play it...

c-rr =PrepareFirstColumn(sb, column) *if (err OK) if (notify) ColurrnnToNotify WhenReached =notify column else ColumnToNotify W'henReached -1.

RenderActive TRUE -etumn err Pause if (RenderActive) (RT024)(259858) -28if (DisplayFrozen) return StoryboardAlreadyPaused else DisplayFrozen TRUE return OK else return StoryboardNotPlaying Continue if (RenderActive) Could be stopped from several sources: Frozen by Pause, or paused by a 'pause' handle. Unfreeze first the entire animation, and then the column animation.

if (DisplayFrozen) i DisplayFrozen FALSE 15 else if (Columrnaused) ColumnPaused FALSE ColumnTerminated TRUE else return StoryboardNotPaused else Sreturn StoryboardNotPlaying Stop Set the flag that will cause Render to halt at the next logical field 25 if (RenderActive) StopRender TRUE else return StoryboardNotPlaying Notify if (RenderActive AND (rc ThisColumn.rc) if (col rc->sb.#columns) return ColumnOutOfRange else if ((col rc->Column#) AND (NotifyIfPastAlready)) Queue Notify Event for col (RT024)(259858) 29 else ColuinnToNotify WhenReached col else return StoryboardNotPlaying 2. The Public Interface of the Render Manager 25 to RTO Processor Driver ?0 TTO Processor Error The animation has stopped, so kill the render list RenderList NULL ErrorEncountered TRUE ErrorType =RTOProcessorError ErrorSubType IntStatus RenderActive FALSE Field Sync Process~l if (RenderActive) update -ColourMixingUnito update soundo *0 if ((This Column.NextFieldReady) AND (IDisplayFrozen)) if (RenderActive) Update RenderList if (CurrSound) *submit -sound(CurrS ound) CurtSound NUJLL ThisColumn.NextFieldReady FALSE set software interrupt to enable Render process ipdate Renderdds For each RenderClip in ThisColumn if (GraphicRHandles) if (Selectl/2 RTOProcessorlRender2) install RTOProcessorObjList2 in RenderList Selectl/2 RTOProcessor2Renderl else install RTOProcessorObjListl in RenderList Selectl/2 RTOProcessorlRender2 advance RenderList (RT024)(259858) write NULL entry in RenderList 3. The Render Process Do this first so that at the next field sync Colour Mixing Unit will update correctly if (ErrorEncountered) Queue ErrorEvent for error that caused 'Stop' StopRender TRUE ErrorEncountered FALSE Get rid of any old column data if (PrevColumn.rc) Queue Cleanup Event for PrevColumn.rc PrevColumn.rc NULL See if its supposed to stop. If so, it is required to must wait until the next column has been fully prepared so that we can delete it if (StopRender) RenderActive FALSE if (ThisColumn.rc) if (NextColumn.NextFieldReady) Queue Notify Event for NextColumn.rc NextColumn.rc NULL Prev.Column.rc ThisColumn.rc ThisColumn.rc NULL StopRender FALSE else StopRender FALSE ThisColumn.NextFieldReady TRUE return Register this column as being rendered, not just prepared tc ThisColumn.rc if (tc) Update Colours tc->NewColumn FALSE (RT024)(259858) 31 Notify application of any specific events relating to column if (N'otifyApplication) Queue Notify Event for Colun# NotifyApplication FALSE A new column may be starting is it the next of same storyboard, or the first of a new (or same) one? if (FirstColumn.preparing AND FirstColumn.NextFieldReady AND NextColumn.NextFieldReady) Queue Cleanup Event for NextColumn FirstColium-n.preparing FALSE FirstColimn.NextFieldReady FALSE NextColtimn.NextFieldReady =FALSE PrevColumn.rc ThisColumn.rc ThisCohlrmn.rc FirstColumn.rc FirstColuirnn.rc NextColumn.rc NULL if ((ThisColumn.rc)-> Column# ColurnnToNotify-WhenReached) NotifyApplication TRUE ColuinnToNotify WhenReached -1 Adjust Levels ColurninTei-ninated FALSE set software interrupt to enable NextColumn process else if (Co lurunTerminated AND (NextColurnn.NextFieldReady) NextColunui.NextFieldReady FALSE PrevColumn.rc ThisColumn.rc ThisColumn.rc NextColurnn.rc NextColumn NULL Adjust Levels, ColumnTerminated FALSE set software interrupt to enable NextColumn process Finally prepare the next field to be rendered if there is one if (ThisColumn.rc) CalcNextField (RT024)(259858) I- I I -32 ThisColumn.NextFieldReady TRUE UpdateColours make sure that the ColourMixingUnitDriver will have the proper colours at the next vblank so that the colours will match the objects being drawn by the RTO Processor. It must be the first thing done by Render.

For each RenderClipRef in ThisColumn For each GraphicRHandle in the RenderClip if (LevelsChanged) ColourMixingUnitColourGroup->updateall colourso ColourMixingUnitEffectGroup->update_all_effects() ColourInfo.ChangedColour FALSE EffectInfo.ChangedEffect FALSE else if (ColourInfo.ChangedColour) ColourMixingUnitColourGroup->update_animated colourso ColourInfo.ChangedColour FALSE if (EffectInfo.ChangedEffect) ColourMixingUnitEffectGroup->update_alleffects) EffectInfo.ChangedEffect FALSE LevelsChanged FALSE Adjust Levels For each of the clips: Calculate the level number for the clip. If it has changed, update the base level numbers for each of the graphics within the clip.

Remember that the RTO Processor colour level 0 is reserved for black. A handle V with a 0 colour usage will reuse the levels from the previous handle.

This assumes that there is sufficient colours and effects to allocate.

Any higher routine would need to check colour effect usage pre-render ColourBase 1 lastcolourusage 0 EffectBase 0 TransparentEffectsRemaining 2 For each RenderClip in ThisColumn Ic FALSE For each GraphicRHandle in the RenderClip (RT024)(259858) fnn Update the colours, flagging the colours that changed if (#Colours 0) ColourBase last-colour-usage last-colour usage #Colours if (ColourBase !=BaseColour-4) lc TRUE BaseColour# =ColourBase ColourMixingUnitColourGrouip->set -colour -level(ColourBase) Update the effects, flagging the levels that changed if (#OpaqueEffects) EffectBase 1 else if (#TransparentEffects) TransparentEffectsRemaining #TransparentEffects EffectsBase 2 TransparentEffectsRemaining 15 if (EffectBase !=BaseEffect#) o Ic TRUE BaseEffect# =EffectBase ColourMixingUnitEffectGroup->set effect level(EffectBase) clip->LevelsChanged lc CalcNextFiedd For each of the clips: First calculate the non-graphic handles Then calculate the camera Finally go through each graphic handle and calculate them 25 For each RenderClip in ThisColumn CalcNonGraphicRHandles CalcCarnera For each RenderClip in ThisColumn CalcGraphic~andles if (TimeRemaining) TimeRemainingif (TimeRemaining 0) ColumnTerminated =TRUE CurrCamera->ZoomChanged CurrCamnera->TransChanged =FALSE (,RT024)(259858) -34- CzkNon ~rap~hic RH4andles These handles are: Camera, Pause, Delay, and Sound.

It doesn't matter that the memory pointed to by the handle is left floating when it becomes active since the memory was allocated in one chunk only hptr &NonGraphicRHandle while (handle *hptr) if (handle->Delay 0) handle->Delay- else *hptr handle->next disconnect this handle switch (handle->type) case Sound: C00rou1 &(handle->Soundlnfo) case Delay: 0, Do nothing case Pause: ColumnPaused =TRUE case Camera: CurrCamera &(handle->Cameralnfo) advance to the next handle to evaluate hptr =&(handle->next) Calculate the new camera attributes based on CurrCamera if (CurrCamera) EvaluateHandleTimelines(&CurrCamera->TimelineRef) Cabc Graphic Handles handle GraphicRHandle while (handle) if (handle->Delay 0) handle->Delay-- (RT024) (2598 58) 35 else EvaluateHandleTimelines(&harndle->TimelineRef) ApplyCameraToHandle ApplyHandleToSpringReferences(handle) CaicShapes CaicColoursAndEffects XChanged YChanged =ZWhanged ExistChanged FALSE advance to the next handle to evaluate handle handle->next Apply Camera To Handle Calculate the transform of the camera applied to the handle in the appropriate arena so that when it is required calculate the RTO Processor object transform later we can simply have: RTOProcessorScale Scale.CameraValue RTOProcessorTransformn (ShapeTransform RTOProcessorScale)+ Transform. CameraValue recalcX =recalcY FALSE switch (arena) case Foreground: existence. CameraValue =existence.BaseValue if (XChanged) XScale.CameraValue =XScale.BaseValue XTrans.CameraValue =XTrans.BaseValue recalcX TRUE XHFactor XTranslation.BaseValue XScale.BaseValue if (YChanged) YScale.CameraValue YScale.BaseValue YTrans.CameraValue YTrans.BaseValue recalcY =TRUE YHFactor YTranslation.BaseValue YScale.BaseValue case Midground: ZTrans.CameraValue ZTrans.BaseValue CurrCame. a->ZTrans if (ZTrans.CameraValue 1) existence.changed OR existence. CameraValue (RT024)(259858) -36existence. CameraValue FALSE else existence. CameraValue existence.BaseVallie sf =CurrCamera->Zoom ZTrans.CameraValue calc CurrCamera->ZoomChanged OR CurrCamera->TransChanged OR ZWhanged if (caic OR XChanged) recalcX =TRUE XScale.CameraValue XScale.BaseValue sf XTrans.CameraValue (XTrans.BaseValue CurrCamera->XTrans) *sf xviewport -offset XHFactor XTranslation.BaseValue XScale.BaseValue if (caic OR YChanged) recalcY TRUE 15 YScale.CameraValue YScale.BaseValue s YTrans.CameraValue (YTrans.BaseValue CurrCamera->YTrans) *sf+ *yviewport-offset YHFactor YTranslation.BaseValue YScale.BaseValue case Background: existence. CameraValue existence.BaseValue if (XChanged OR CurrCamera->ZoomChanged) recalcX TRUE XScale.CamneraValue XScale.BaseValuLe CurrCamera->Zooni XTrans.CameraValue (XTrans.BaseValue CurrCai-nera->Zoom) x_viewport -offset XHFactor XTranslation.BaseValue XScale.BaseValue if (YChanged OR CurrCamera->ZoomChanged) recalcY =TRUE YScale.CameraValue YScale.BaseValue CurrCamera->Zoorn YTrans.CameraValue (YTrans.BaseValue CurrCamera->Zoorn) yviewport -offset YH~actor YTranslation.BaseValue YScale.BaseValue 3 5 Now calculate the DPixelScale and DDPixelScale for the RTO Processor (RT024)(259858) -37transforms if (existence. CameraValue AND (recalcX OR recalcY) if (,XScale.CameraValue YScale.CameraValue) DPixelScale 1 DDPixel~cale 1 XScale.CameraValue else DPixel.Scale YScale.CameraValue XScale.CameraValue DDPixelScale DPixelScale XScale.CameraValue Apply Handle To Spring R~eferences Go through all the components that require updating as they are not animated, but are referred to by springs if (handle->XChanged OR handle->YChanged) tlr =handle->ApplyHandle while (tIr) (tlr->BaseValue)->HandleValue= *(tlr >BaseValue)+ *(tlr->Factor) tir tlr->next CacColours And E~ffects Go through the colour timeline reference list if there is one and update the colours that huve changed if (Colourlnfo.TimelineRefs) EvalGraphiicTimelines(&Colourlnfo .TimelineRefs) Only write to ColourMixingUnitGroup if necessary, but always update high p pos 25 if (handle->Existence.CameraVal) Flush any starts or extents in X to ColourMixingUnitColourGroup.

They could have been changed via the timelines bref AnimatedXB lends while (bref) if ((recalcX) OR (bref->Changed)) (bref->Start.ColourMixingUnitValue) lowprecision( (bref->Start.BaseVai handle->XScale.CameraVal) handle->XTrans.CanieraVal) *(bref->Extent. ColourMixingUnitValue) lowprecision( bref->Extent.BaseVal handle->XScale.CameraVal) (RT024)(259858) 38 bref->Changed =FALSE bref =bref->next Flush any starts or extents in Y to Colour Mixing Unit ColourGroup.

They could have beer, changed via the timelines bref =AnimatedYB lends while (bref) if ((recalcY) OR (bref->Changed)) *(bref->Start.ColourMixingUnitValue) lowprecision( (bref->Start.BaseVal handle->YScale. CameraVal) handle->YTrans.CameraVal) *(bref.>.Extent.ColourMixingUnitValue) lowprecision( bref->Extent.BaseVal handle->YScale. CameraVal) 15 bref->Changed =FALSE F~valua ef bref->next This evaluate timeline references for a handle (be it graphic, camera, or colour component) and therefore is absolute rather than relative in value.Currently there is no bounds checking on the value.

If there's no time remaining then read new keyframes (we're already pointing at the first) until a non-loop keyframe is encountered while (tlr *ptlr) kf tlr->Keyframe 25 if (tlr->time_remnaining 0) while ((type kf->Type) Loop) kf tlr->TLKeyframe PATCH(tlr->TLKeyframe->NextKeyfrarne) Now at a non-loop keyframe, and ready to go. Type discloses what sort of keyframe we're at kf =tlr->Keyfrarne tlr->time-remaining kf->Time ChangedFlag TRUE switch (type) case External: (RT024)(259858) 39 tlr->ReferenceValue =CurrClip->ClipRef->Externals~kf->Index] *BaseValuO *(tlr->ReferenceValue) case Independent: *BaseValue kf->Value tlr->DVaI tlr->DDVaI 0 case Dependent: if (kf->Index ATTACHEDTOHANDLE) tlr->ReferenceValue &ZeroFactor else tlr->ReferenceValue tlr->ReferenceBase (kf->Index tlr->DataSize) switch (kf->type) case Derived: *BaseValue =*(tlr->ReferenceValue) caeod 15 ase od: (tlr->RcferenceValue) kf->Offset case Spring: *Bas~alu (tlr->ReferenceValue) k-Ofe kf->InitialTranslation tlr->DVaI tlr->Initial Velocity else There is still time remaining in the keyframe, so lets do the calcs switch (kf->type) .~case External: val *(+pi>ReferenceValue) if (val BaseValue) *Base'vfalue val *ChanpedFlag TRUE case Independent: switch (kf->SubType) case Step: (RT024)(259858) 40 Do Nothing case Linear: *BaseValue kf->DVal ChangedFlag TRUE case Parabolic: BaseValu !->DVal tlr->DVa. DD'VaI ChangedFlag TRUE case Cubic: *BaseValue 4- tlr->DVaI tlr->DVaI tlr->DDVal tlr->DDVaI kif->DDDVal ChangedFlag TRUE 15 case Dependent: switch (kf->type) case Derived: vatl *(tlr..>ReferenceValue) :if (val *BaseValue) *BaseValue =val *ChangedFlag TRUE case Rod: val (tlr->ReferenceValue) kf->Offset if (val *BaseValue) *BaseValue val *ChangedFlag TRUE ~:case Spring: extension *(tlr..>ReferenceValue) kf->Offset *BaseValue acceleration kf->Stiffness extension tlr->DVaI acceleration (tlr->Dval kf->Damping) if (tlr->Dval 1= 0) *BaseValue tlr->Dval Now that the field has been evaluated, we can decrement the time -remaining for this timeframe and see if it is required to advance to the next keyframe (RT024)(259858) -41or simply terminate right here tlr->time-remainingif (tlr->time-remaining 0) if (kf->terminate) *ptlr =tlr->next else ptlr &(tlr->next) Evaluate Graphic Timelines It is required to take the graphic handle into account in our calculations If there's no time remaining then read new keyframes (we're already pointing at the first) until a non-loop keyframe is encountered.,m while (fir *ptlr) kf tlr->Keyframe if (tlr->time -remaining 0) 15 while ((type kf->Type) Loop) kf =tlr->TLKeyframe =PATCH(tlr->TLKeyframe->NextKeyframe) Now we are at a non-loop keyframne, and ready to go. Type tells us 'what sort of keyframe we're at kf tlr->Keyframe tlr->time-remaining kf->Time switch (type) case External: tLlr->ReferenceVaiue CurrClip->ClipRef->Ecternals [kf->Index] 25 val *BaseValue *(tlr..>ReferenceValue) case Independent: v. val *BaseValue kf->Value tlr->DVaI tlr->DDVaI 0 case Dependent: if (kf->Index ATTACKED_-TO_-HANDLE) tlr->ReferenceValue &ZeroFactor else tlr->ReferenceValue tlr->ReferenceBase (kf->Index tlr->DataSize) switch (kf->type) (RT024)(259858) 42 case Derived: val *Baseyal (tlr->ReferenceValue)->BaseVal case Rod: val *BaseValue (tlr->ReferenceValue)->BaseVal kf->Offset case Spring: val *BaseVal (tlr->ReferenceValue)->BaseVal kf->Offset kf->InitialTranslation tlr->DVal =tlr->Initial Velocity *ChangedFlag =TRUE *HandleValue =val Factor else There is still time remaining in the keyframe, so lets do the caics switch (kf->type) case External: val (tlr->ReferenceValue) if (val BaseVal) *BaseVal val *ChangedFlag TRUE *HandleValue val *Factor case Independent: switch (kf->SubType) case Step: Do Nothing case Linear: vat *BaseVal kf->DVal *ChangedFlag TRUE *HandleValue val *Factor case Parabolic: val *BaseVal tlr->DVaI tlr->DVal kf->DD Vat *ChangedFlag TRUE *HandleValue val *Factor (RT024)(259858) 43 case Cubic: val *Baseyal tlr->DVal tlr->DVal tlr->DDVal tlr->DDVa1 kf->DDDVaI ChangedFlag TRUE *Handleyalue val *Factor case Dependent: switch (kf->type) case Derived.

val (tlr->ReferenceValue)->BaseVal if (val *BaseVal) *BaseVal val *ChangedFlag TRUE adleValue =vt+*Facntr case Rod: vat (tlr->ReferenceValue)->BaseVal kf->Offset *if (val BaseVal) *Baseyal =val ChangedFlag TRUE *HandleValue val *Factor case Spring: extension (tlr->ReferenceValue)->HandleValue kf->Offset (tlr->BaseValue)->HandleValue 25 acceleration kf->Stiffness extension tlr->DVal =acceleration (tlr->Dval kf->Damping) V. if (tlr->Dval 0) val *HandleVal tlr->Dval *BaseVal val *Factor Now that we've evaluated the field, we can decrement the time remaining for this timeframe and see if we need to advance to the next keyframe or simply terminate right here.

Note that if a timeline ref terminates but is referred to by a spring then the handles must continue to be applied to the base value for the (RT024)(259858) 44 referencing timelines to make use of (as the handle moves) tlr->time-remainingif (tlr->time-remaining 0) if (kf->terminate) *ptlr tlr->next if (ApplyHandle) tlr->next handle->ApplyHandles handle->ApplyHandles tir else ptlr &(tlr->next) Calculate Shapes EvaluateGraphicTimelines(handle->Shapelnfo .TimelineRef) olist (handle->S electl1/2 RTOProcessorlRender2) handle-> RTOProcessorObJList2: handle-> RTOProcessorObjListl ~sh =handle->RenderShapes while (count) Update Shape(shi) sh++ count-- Finally set the termination bit on the last of the objects in the list SetTerm-inationBit(olist- 1) Update Shape (inline) 25 First determine if we have to write to everything The 'flags' variable should be in a register, so that the writing can be as fast as possible if ((handle->ChangedExist) OR (sh->Obj ect. Changed)) flags OR= OxiF sh->X.Changed FALSE sh->Y.Changed FALSE sh->Obj ect. Changed FALSE else flags OxOO if (haridle->XChanged OR sh->X.Changed) (RT024)(259858) 45 sh->X.Changed FAL-SE flags OR= Ox0l if (handle->YChanged OR sh->Y.Changed) sh->Y.Changed =FALSE flags OR= 0x02 if (flags) flags OR= 0x04 if (handle->LevelsChanged) flags OR= 0x08 Save what was new this field, and merge with what was new last field to determine what we need to write this field temp flags flags OR= sh->WrittenLastField sh->V/'rittenLastField temp 0: Now can update based on the flags ocount sh->#Objects if (flags Ox0O) olist (ocount sizeof(RTOProces,3orObject)) else if (handle->Exists. CameraValue) if (flags AND OxO 1) xval (sh->X.value handle->XScale.CameraValue)+ 25 handle->XTrans.CameraValue xscale handle->XScale.CamneraValue if (flags AND OxOl) yval (sh->Y.value handle->YScale.CameraValtie) handle->YTrans.CameraValue yscale handle->YScale.CameraValue o sh->Object.Value do if (flags AND OxOlI) olist->Pixel x-val.

olist->PixelScale x-scale (RT024) (25 98 58)

M

-46if (flags AND 0x02) olist->Line y- val olist->LineScale =y_scale if (flags AND 0x04) olist->DeltaPixelScale handle->DeltaPixelScale olist->DeltaDeltaPixelScale handle->DeltaDeltaPixelSca'Lie if (flags AND 0x08) e o->effect C o->colour handle->Colourlnfo.BaseColour# if -l) c OR (e handle->Colourlnfo BaseEffectll) «<6 olist->Colour c if (flags AND OxI 0) olist->QPFadr PATCH(o->QPFadr) olist++ .4*4*while (--ocount AND (o++)->Continues)) rest of (or all of) object is invisible, so must be NULLed out **:while (ocount) olist->QPFadr &NullQPF olist+ ocount-- 4. Prepare First Column First inform the renderer that we are preparing something to take the place of the currently rendered storyboard/column 25 FirstColunin.preparing TRUE col endcol column Keep attempting to prepare the first column while there is nothing in the column, but if we've made the rounds, forget it.

do col+± err Build~irstColuinn(sb, col) while ((err NoClipsFound) AND (col! endcol) (RT024) (259858) 47 Now we've either made the rounds, succeeded, or failed.

if (err OK) FirstColumn.FirstFieldReady TRUE else FirstColumn.preparing FALSE return err Build First Column If an error occurs during the construction of the first column, we can still leave the existing system running, as that will still be ok.

Try to build the Render Column for the first column to be rendered FirstColumn.rc BuildRenderColumn(sb, column) if (FirstColumn.rc NULL) return NotEnoughiMemory Now go through the actual column, and build each clip for rendering Even if there are cont-markers, we use them as clips **cell sb->cells (column) count sb->#rows rcr (RenderColunmnEntry *)((void *)rc sizeof(RenderColumn)) while (count) if (cell.type ContinuationMarker OR Clip) rclip ProcessClip(&(rcr->RenderClip)) if (rclip NULL) 25 CleanupColumn(rc) return NotEnoughMemory cell->clipref->RenderClip rcr->RenderClip =rclip rcr->TerminateFlag (cel ->TimeRemaining 0) rcr++ rc->#Clips++ cell++ countreturn OK Prepare Next Column col endcol ThisColunin->rc->columng (RT024)(259858) -48sb =ThisColumn->rc->sb Keep attempting to prepare the next column while there is nothing in the column, but if we've made the rounds, forget it.

do col++ err BuildNextColumn(sb, col) while ((err NoClipsFound) AND (col !=endcol) Now we've either made the rounds, succeeded, or failed.

setjriority v.high if (err! OK) ErrType err ErrSubType 0 ErrorEncountered TRUE 6 5 NextColumn.NextFieldReadly

=TRUE

setpriority back low again Bu~ild Next Column Try to build the Render Column for the next column to be rendered We need to work out which is the next column to be rendered column ThisColumn->rc-.>colun# 1 sb ThisColumn->rc->sb if (column =sb->#colurnns) column 0 NextColunin.rc BuildRenderColumn(sb, column) e 25 if (NextColumn.rc NULL) return NotEnougliMemory Now go through the autual column, and build each clip for rendering cell =sb->cells(column) count sb->#rows rcr (RenderColumEntry *)((void *)NextColumn.rc sizeof(RenderColumn)) while (count) switch (cell.type) case NOTHING: case CLIPMARKER: (RT024)(259858) 49 rolip NULL case CLIP: rolip ProcessClip(&(rcr->RenderClip)) if (rclip NULL) return NotEnougliMemory else cell->clipref->RenderClip rcr->RenderClip =rclip case CONTINUATIONMARKER rclip cell->clipraf->RenderClip if (rclip NULL) return StoryboardInvalid else rcr->RenderC lip rclip if (rclip) rcr->TerminateFlag (cell->TimeRemaining 0) rcr-H- 20 rc->#Clips++ cell++count-- Final check to make sure that we have actually got something if (rc->#Clips =0) return NoClipsFouncl return OK Build Render Column We don't know how many clips there are in the column, so allocate enough space in case there is one per cell. The extra space is only two 32 bit words per cell, so it doesn't really matter about the waste.

rc alloc(sb->#rows sizeof (RenderClipRef) sizeof(RenderColumn)) We can allocate the RenderClipReferences. Build the main record first if (rc) rc->Storyboard sb (RT024)(259858) rc->Column# columan rc->#Clips 0 rc->NewColumn TRUE rc->RClipRef (RenderClipRef ((void *)rc sizeof(RenderColunm)) return rc Process-Clip size clipref->SizeRequiredForRenderClip miminem alloc(size) maxmem =minm size if (minm= NULL) return NotEnoughMemory Havre allocated enough space for the clip... .now can fill it in BuildRenderClip sb->#ClipsPlaying±+ return OK .Bui dRendet 'ICip First allocate the RenderClip itself rc (RenderClip minem mimnem sizeof (RenderClip) Setup the basic render clip record rc->ClipRef clipref o FObjSize clipref->#ObjectsRequired *sizeof (RTOProcessorObject) fobj 1 rc-> RTOProcessorObj Listi (RTOProcessorObject *)mninem minm FObj Size rc-> RTOProcessorObjList2 (RTOProcessorObject *)nirnem inn FObj Size, rc->Selectl/2 RTOProcessorlRender2 rc->LevelsChanged =TRUE rc->TimeRernaining clip->Duration grhandle =&(rc->GraphicRllandles) ngrjhandle &(rc->NonGraphicRHandles) Now we go through the clip itself, processing each of the handles (RT024)(259858) 51 clip PATCH(clipref->ClipDirEntry->data) BuidLYBTable count clip->#Handles h PATCH(clip->CHB) convertedHandles clipref->convertedHandles while (count) handle PATCH(*h) ProcessHandle(handle->type, handle) count-- Close off the end of the various handles now that we've finished *nor handle NULL *ghandle NULL 15 Build T.VB Table Builds a table for the clip that details the default and user LVBs for each LVB type. The table is permanently allocated.

First add the default entries count clip->#ComponentTypes lvb, PATCH(clip->lvb) while (count) type lvb->ComponentType lvbrefltype] .#editable lvbref~type] .default-ivbs PATCH(lvb->entries)l lvb++ count-- Now add the user entries count clipref->#ComponentTypes lvb PATCH(clip->lvb) while (count) type lvb->ComponentType lvbref~type] .user-lvbs lvb->entries lvb+± (RT024)(259858) 52 count-- Process Delay Handle Potentially builds a delay handle at the end of the available data space lvb =ResolveLVBAddress(Delay, handle->lvbi#, handle 1) if (lvb) Not empty, so we add the delay to the current free data area and add the data to the record maxrnem sizeof(RDelayHandle) *ng_handle rhandle (RDelayI-andle *)(maxmem.) rhandle->type Delay rhandle->delay handle->delay lvb->delay ng_handle =&(rhandle->next) Process Pause Handle Potentially builds a pause handle at the end of the available data space 15 lvb ResolveLVBAddress(Pause, handle->lvb#, handle 1) *.if (lyb) Not empty, so we add the pause to the current free data area and add the data to the record maxnaem sizeof(RPauseHandle) *ng_handle rhandle (RPauseHandle *)(maxmem) rhandle->type Pause rhandle->delay handle->delay ngr _handle &(rhandle->next) 0 Process xuxnd Handle Potentially builds a sound handle at the end of the available data space lvb ResolveLVBAddress(Sound, hanidle->lvbH, handle 1) if (lyb) Not empty, so wNe add the sound record, and add the data to it maxmem sizeof(RSoundHandle) rhandle (RSoundHandle *)(maxmem) data Re.-'lveDataAddress(lvb) rhandle->type Sound rhandle->delay handle->delay rhandle->SoundData data->SoundData rhandle->SoundDuration data->Duration (RT024)(259858) 5 3 rhandle->#Iterations handle->#TimesToPlay ngr-handle &(rhandle->next) Process Camera Handle Potentially builds a sound handle at the end of the available data space lvb ResolveLVBAddress(Camera, handle->lvbH, handle 1) if (!lvb) return Not empty, so we add the camera and add the data to the record maxmem sizeof(RCameraHandle) rhandle (RCameraHandle *)(maxmem) rhandle->type Camera rhandle->delay handle->delay rhandle->ZoomChanged =rhandle->TransChanged TRUE tir =&(rhandle->tlr) BuildAbsTimeLineRef(XTranslation, &(handle->Xtrans), &(rhandle->Xtrans), &(rhandle->TransChanged), NULL, 0) BuildAbsTimeLineRef(YTranslation, &(handle->Ytrans), &(rhandle->Ytrans), &(rhandle->TransChanged), NULL, 0) BuildAbsTimeLineRef(ZTranslation, &(handle->Ztrans), &(rhandle->Ztrans), &(rhandle->TransChanged), NULL, 0) IBuildAbsTimeLineRef(ZTranslation, &(handle->Zoom), &(rhandle->Zoom), &(rhandle->ZoomChanged), NULL, 0) ngr__handle =&(rhandle->next) Process Text Haadlt Potentially builds known render handle from the converted text handle handle *convertedHandle convertedHandle++ if (handle 1=NULL) ProcessHandle(handle->type, handle) Process Graphic Handle Potentially builds a graphic handle at the start of the dafta space. The graphic's data is stored at the end of the data space. This ensures that the graphic render handles are contiguous so that referencing works.

lvb ResolveLVBAddress(Graphic, handle->lvb#, handle 1) 3 5 if (!lvb) (RT024) (259858) 54 return Not empty, so we add the graphic. First setup the base info *grhandle rhandle (GraphicRi-andle minmern minmem sizeof(GraphicRHandle) grhandle &(rhandle->next) rhandle->type Graphic rhanA"' delay =handle->delay rhandle->arena handle->arena if (FirstGHandle =NULL) FirstGHandle rhandle Now we can copy the handle's timelines as necessary tir &(rhandle->tlr) BuildAbsTimeLineRef(XTranslation, &(handle->Xtrans), &(rhandle->Xtrans), &(rhandle.>XChanged), &(FirstGHandle->Xtrans), BuildAbsTimeLineRef(YTranslation, &(handle->Ytrans), &(rhandle->Ytrans), &(rhandle->YChanged), &(FirstGHandle->Ytrans), sizeof(GraphicRHandle, NULL) if (rhandle->arena MidGround) BuildAbsTimeLineRef(ZTranslation,&(handle->Ztrans),& (rhlardle->Ztrans), &(rhandle->ZChanged), &(FirstGHandle->Ztrans), sizeof(GraphicRHandle, NULL) else rhandle->ZTrans 1 BuildAbsTimeLineRef(XScale, &(handle->XScale), &(rhandle->XScale), &(rhandle->XChanged), &(FirstGHandle->XScale), sizeof(GraphicRHandle, NULL) BuildAbsTimeLineRef(YScale, &(handle->YScale), &(rhandle->YScale), &(rhandle->YChanged), &(FirstGHandle->YScale), sizeof(GraphicRHandle, NULL) BuildAbsTimeLineRef(Existence, &(handle->Existence),&(rhandle->Existence), &(rhandle->ExistChanged), &(FirstGHandle->Existence), sizeof(GraphicR~andle, NULL) (RT024)(259858) 55 *tlr NULL Now copy the colour and effect information across data =ResolveDataAddress(lvb) SetupObjects(data, rhandle) SetupColour(data, rhandle) Se-Lp Ojects allocate the storage for the object pointers rhandle-> RTOProcessorObjectListl fobj 1 rhandle-> RTOProcessorObjectList2 fobj 1 RTOProcessorObjectslnClip shcount rhandle->#shapes =data->#shapes rhandle->WrittenLastField OxOO maxmeni (data->#shapes sizeof(Shape)) *,first-sh. rsh rhandle->RenderShapes maxmem tlr &(rhandle->tlr) sh PATCH(data->Shapes) while (shcount) rsh->#Objects =sh->MaxObjectslnShape if (sh->SpringConnected) BuildRelTimelineRef(XTrans, &(rsh->X.BaseVal), Changed), &(first -sh->X.BaseVal), sizeof(RShape),&(rhandle->X.HFactor)) BuildRelTimelineRef(YTrans, &(rsh->Y.BaseVal), Change d), &(first_sh->Y.BaseVal), sizeof(RShape), &(rhandle->YHFactor)) else BuildAbsTimelineRef(XTrans, &(rsh->X.BaseVal), &(rsh->X.Changed), &(first_sh->X.BaseVal), sizeof(RShape)) BuildAbsTimelineRef(YTrans, &(rsh->Y.BaseVal), Changed), &(first -sh->Y.BaseVal), sizeof(RShape)) BuildAbsTimelineRef(Object, &(rsh->Obj .BaseVal), &(rsh->Obj. Changed), &(first sh->obj), sizeof(RShape)) fobj 1 rsh->#Objects (RT024)(259858) 56 advance to next shape sh++ rsh-Hshcount-- *tlr =NULL Setup Color Try to re-use the colours if possible. Note that a 'shared' colours with blends is not permitted (it is expanded out) for now.

if (data->colour NULL) OR ((lvb->flags.SharedColours AND (grprev-lvb lvb) AND (!data->hasB lends))) ReUseLevels else *grprev -lvb lvb tir =&(rhandle->Colourlnfo.tlr) SetupEffectsTLs(data, rhandle) SetupColourTLs(data, rhandle) ReusaeLevels Reuse the colour and effect levels from the previous graphic 0 We can't assign real levels yet. Assigning levels happens at render time.

rhandle->Colourlnfo.#Colours 0 rhandle->Colourlnfo.BaseColour# 0 rhandle->Colourlnfo.ColourMixingUnit~olourGroup =NUL rhandle->Colourlnfo.TLRefList NULL 25 rhandle->Colourlnfo.#AnimatedB lends =0 rhandle->Colourlnfo.AnimatedXBlends =NULL rhandle->Colourlnfo.AnimatedYBlends =NULL rhandle->Colourlnfo.ChangedColour FALSE rhiandle->Colourlnfo.#OpaqueEffects 0 rhandle->Colourlnfo.#TranspEffects 0 rhandle->Colourln-fo .BaseEffectit 0 rhandle->Colourlnfo.ChangedEffect FALSE rhandle->Colourlnfo.ColourMixingUnitEffectGroup =NULL &SflpffectI,,count data->#effects (RT024)(259858) -57rhandle->Colourlnfo.flEffects grprev_#effects count if (count) rhandle->Colourlnfo.ColourMixingUnitEffectGroup new ColourMixingUnitEffect(O, count) rhandle->Colourlnfo. ChangedEffect TRUE me &(rhandle->Colourlnfo .ColourMixingUnitEfectGroup->EffectEntry[0]) first-me me e data->effects else rhandle->Colourlnfo.ColourMixingUnitEffectGroup

NULL

rhandle->Colourlnfo.ChangedEffect FALSE Now do the installing of the effects and possible timelines while (count) 15 Setup the colour tend BuildAbsTimeLineRef(Luma, &(e->tend.y), &(me->tend.y), &(rhandle->Colourlnfo.EffectsChanged), &(first -me->tend.y), sizeof(ColourMixingUnitEffectEntry)) BuildAbsTimeLineRef(Chroma, &(e->tend.Cr), 20 &(me->tend.Cr), &(rhandle->Colourlnfo .EffectsChanged), &(first-me->tend.Cr), sizeof(ColourMixingUnitEffectEntry)) BuildAbsTimeLineRef(Chroma, &(e->tend.Cb), &(me->tend.Cb), &(rhandle->Colourlnfo.EffectsChanged), &(first -me->tend.Cb), sizeof(ColourMixingUnitEffectEntry)) BuildAbsTimeLineRef(Mix, &(e->tend.mix), 0 &(me->tend.mix), &(rhandle->Colourlnfo.EffectsChanged), &(first-me->tend.mix), sizeof(ColourMixingUnitEffectEntry)) Set up the min tend BuildAbsTimeLineRef(MinTend, &(e->mintend.y), &(me->mintend.y), &(rhandle->Colourlnfo.EffectsChanged), &(first -me->mmntend.y), sizeof(ColourMixingUnitEffectEntry)) BuildAbsTimeLineRef(MinTend, &(e->mintend.Cr), &(me->mintend.Cr), &(rhandle->Colourlnfo.EffectsChanged), &(first-me->mintend.Cr), sizeof(ColouirMixingUnitEffectEntry)) (RT024)(259858) 58 BuildAbsTimeLineRef(MinTend, &(e->mintend. Cb), &(me->mintend.Cb), &(rhandle->Colourlnfo.EffectsChangcd), &(first-me->mintend.Cb), sizeof(ColourMixingUnitEffectEn .ry)) BuildAbsTimeLineRef(Min.Tend, &(e->mintend.mix), &(me->mintend.mix), &(rhandle->Colourlnfo.EffectsChanged), &(first-me->mintend.mix), sizeof(ColourMixingUnitEffectEntry)) Set up the md mask BuildAbsTimeLineRef(RndMask, &(c->rndmask.y), &(me->rndrnask.y), &(rhandle->Colourlnfo.EffectsChanged), &(first -me->rndmask.y), sizeof(ColourMixingUnitEffectEntry)) BuildAbsTimeLineRef(RndMask, &(e->rndmask. Cr), &(me->rndmask.Cr), &(rhandle->Colourlnfo .EffectsChangyed), &(first-me->rndmask.Cr), sizeof(ColourMixingUnitEffectEntry)) 15 BuildAbsTimeLineRef(RndMask, &(e->rndmask.Cb), &(me->rndmask.Cb), &(rhandle->Colourlnfo .EffectsChanged), &(first me->rndmask. Cb), sizeof(ColourMixingUnitEffectEntry)) BuildAbsTimeLineRef(RndMask, &(e->rndmask.mix), &(me->rndmask.mix), &(rhandle->Colourlnfo.EffectsChanged), 20 &(first-me->rndmask.mix), sizeof(ColourMixingUnitEffectEntry)) advance to next effect a me+Hcount-- V. 25 e-H- 4 SetupCoouruLILs count rhandle->Colourlnfo.#Colours =data->#colours if (count 0) rhandle->Colournfo.ColourMixingUnitColourGroup new ColourMixingUnitColour(0, count) mc &(rhandle->ColourInfo.ColourMixingUnitColourGroup>ColourEntry[V)) first-mc =me c data->colours We have to allocate a lot of space for blends for fast lookup even (RT024)(259858) -59though it won't be used very often... such is life... At least we only allocate the space if we have blends at all. Not much (,Ise we %can do at least the render will jump to exact members rather than process sequentially if (data->ContainsB lends) maxmem (#Colours sizeof(AnimatedBlend)) first-ab blend (AnimatedBlend maxmem maxmemn (#Colours sizeof(AnimatedBlend)) first-amnix blend2 (AnimatedBlend maxmemn else rhandle->Colourlnfo.ColourMixingUnitColourGroup =NULL *%Keep an eye open for xand yblends xbl (rhiandle->Colourlnfo.AnimatedXB lends) 15 ybl &(rhandle->Colourlnfo.AnimatedYB lends) ~:while (count) setup the basic colour mc->cmode c->BlendedColour.BlendType mc->mmode c->BlendedMix.BlendType BuildAbsTimeLineRef(Luma, &(c->StartColour.y), &(mc->start.y), &(rhandle.Colourlnfo.ColourChanged), &(first -mc->start.y), sizeof(ColourMixingUnitColourEntry)) IBuildAbsTimeLineRef(Chroma, &(c->StartColour.Cr), &(mc->start.Cr), &(rhandle.Colourlnfo .ColourChanged), &(first -mc->start. Cr), sizeof(ColourMixing~nitColourEntry)) BuildAbsTimeLineRe.f(Chroma, &c->StartColour.Cb), &(mc->star-t. Cb), &(rhandle.Colourlnifo.ColourChanged), &(first -mc->star.t.Cb), sizeof(ColourMixingUnitColourEntry)) BuildAbsTimeLineRef(Mix, &(c->StartCoiour.mix), &(mc->start.mix), &(rhandle.Colourlnfo.ColourChanged), &(first-nmc->start.mix), sizeof(ColourMixingUnitColourEnty)) Setup the end colour if there's a blend if (mc->mode FlatColour) BuildAbsTimeLineRef(Luma, &(c->EndColour.y), (RT024)(259858) 60 &(rhlandle.Colourlnfo. ColourChianged), &(first_mc->end.y), sizeof(ColourMixingUnitColourEntry)) BuildAbsTimeLineRef(Chroma, &(c->BndColour. Cr), &(rnc->end. Cr), &(rhandle.Colourlnfo.ColourChanged), &(first-mc->end.Cr), sizeof(ColourMixingUnitColourEntry)) BuildAbsTimeLineRef(Chroma, &(c->EndColour.Cb), &(mc->end.Cb), &(rhandle.Colourlnfo.ColourChanged), &(first-mc->end.Cb), sizeof(Colouri\ixingUnitColourEntry)) Setup the blend info if (mc->crnode XBlend) *xbl blend xbl &(blend->next) else *ybI blend ybl &(blend->next) blend->Start.ColourMixingUnitValue &(mc->colour.startXY) blend->Extent.ColourMixingcUnitValue &(mc->colour.extent) BuildAbsTimeLineRef(XTranslation, &(c->colour. StartXY), &(blend->Start.BaseVal), &(blend-> Changed) &(first_ab->Start.baseVal), sizeof(Anirnatedfllend)) BuildAbsTimeLineRef(XTranislation, &(c->colour.Extent), &(blend->Extent.BaseVal), &(blend->Changed) &(first-ab->Extent.BaseVal), sizeof(AnimatedB lend)) S 25 if (mc->mmode !=FlatColour) BuildCETineLipeRef(Mix, &(c->EndColour.mix), &(mc->end.niix), &(rhandle.Colourlnfo.ColourChanged), &(first_mc->end.mix), sizeof(Colo QrMixingUnitColourEntry)) Setup the blend info if (mc->emode X-Blend) *xbl blend2 xbl &(blend2->next) else *ybl blend2 ybl &(blend2->next) (RT024)(259858) 61 blend2->Start.ColourMixingUnitValue &(mc->mix ~startXY) blend2->Extent. ColourMixingUnitValue &(mc->mix.extent) BuildAbsTimeLineRef(XTranslation, &(c->StartXY), &(blend->Start.BaseVal), &(blend-> Changed) &(first -amix->Start.baseVal), sizeof(AnimatedB lend)) BuildAbsTimeLineRef(XTranslation, &(c->Extent), &(blend->Extent.BaseVal), &(blend-> Changed) &(first -amix->Extent.BaseVal), sizeof(AnimatedB lend)) advance to next colour mc++ countblend++ blend2++ c++ 15 terminate the blend lists *xbl =NULL *ybl NULL *Cmmn ij Resolve Data Address (inline) Given an lvb address that points to a possible directory entry structure, calculate the address of the data if (lvb.DirEntry.LowerBit 1) data Ivb.DirEntry OxFFFFFFFFE else S 25 data PATCH(lvb.DirEntry->data) Resolve TVB Address (inline) Determine the adress of the LVB Parms: lvb# the index into the LVB table type the type of LVB to find adr the adress following the current handle (in case lvb# if (lvb# 0) if (ivb# lvbref~type].#editable) lvb &(lvbref~type] .user -lvbs [lvb#]) if lvb->flags.OverrideDefault) (RT024)(259858) L_ -62- Ivb &(lvbref[type].default_lvbs[lvb#]) else Ivb &(lvbref[type].default_lvbs[lvb#]) There may be no LVB at all else if (lvb# -1) Ivb NULL Final case is when the LVB follows the reference else if (lvb# -2) Ivb adr If the empty bit is set, there is effectively nothing there if (lvb AND (lvb->flags.empty)) S 15 Ivb NULL return LVB Build Absolute TimeLine Ref (inline) Builds a Timline Reference based on data (timelines build up from bottom) while the data builds down from top. The 'abs' stands for absolute, indicating that there is no need for a special factor to be taken into account for spring calculations.

Type of data Source address of source data S 25 Dest address of dest value CFlag address of changed flag to set when the value changes BaseVal address of first structure for reference purposes DataSize: size of each similar structure (to be added for references) if (*source 0) *dest *source *changed TRUE else maxmer sizeof(TimelineReference) tlr *tlr maxmem tlr->Type type (RT024)(259858) -63tlr->TimeRemaining 0 tlr->BaseValue dest tlr->ChangedFlag cflag tlr->TLKeyframe PATCH(*source) AND FFFFFFFFE tlr->BaseVal BaseVal tlr->DataSize DataSize tlr &(tlr->next) tlr->Factor NULL The following are not required tlr->ReferenceBase NULL tlr->DValue 0 tlr->DDValue 0 Build Relative TimeLine Ref (inline) S Builds a Timline Reference based on data (timelines build up from bottom) while the data builds down from top. The 'abs' stands for relative, indicating that there is a need for a special factor to be taken into account for spring calculations.

Type :of data Source address of source data Dest address of dest value CFlag :address of changed flag to set when the value changes BaseVal address of first structure for reference purposes DataSize: size of each similar structure (to be added for references) Factor address of scaling factor to use for springs S 25 if (*source 0) *dest *source *changed TRUE if (Factor) maxmem sizeof(TimelineReference) *ftlr maxmem ftlr->BaseVal BaseVal ftlr->Factor Factor ftlr &(ftlr->next) else maxmem sizeof(TimelineReference) (RT024)(259858) 64 tir *tlr maxmem tlr->Type type tlr->TimeRemaining 0 tlr->BaseValue dest tlr->ChangedFlag, cflag tlr->TLKeyframe PATCH(* source) AND FFFFFFFFE tlr->BaseVal =BaseVal tlr->DataSize =DataSize tir &(tlr->next) tlr->Factor Factor The following are not required tlr->ReferenceBase =NULL tlr->D Value 0 YO tlr->DDValue 0 15 C1eanA~p&~iumn We go through each of the entries and delete the ones that we have finished with. Anything that is terminating can be cleaned up, and anything in a new column can be cleaned up.

For each entry in Column if (TerminateFlag OR NewColumn) CleanupClip Finally we can deallocate the column itself delete column 25 CIleanup.Clip Storyboard->ClipsPlaying-- For each Graphic RHandle in the RenderClip delete Colourlnfo.ColourMixingUnitColourGroup delete Colourlnfo.ColourMixingUnitEffectGroup Finally cleanup the clip memory itself delete RenderClip 7. Calculate Render Clip Requirements Go through the clip and calculate the requirements for the RenderClip (RT024)(259858) 65 version of it. Specifically: how many colours are required how many opaque effects are required how many transparent effects are required how many RTOProcessor objects are required how much space for the RenderClip is required SetupCounts if (clipref->Resolved) CaicRequirements UpdateRequirements .Seftp Counts Assume that there are no requirements as yet ColourU sage 0 OpaqueEffectsUsage 0 15 TransparentEffectsUs age 0 RTOProcessorObjectCount 0 mem =0 UpdateReqirements Write the RenderClip requirements back to the clip reference clipref->ColourUsage ColourUsage clipref->#OpaqueEffects =OpaqueEffectsUsage clipref->#TranspEffects TransparentEffectsUsage clipref-># RTOProcessorObJects RTOProcessorObjectCount *clipref->MemnForRenderClip =memn S 25 Calculate Requirernents clip PATCH(clipref->ClipDirEntry->data) lvbref BuildLVBTable count =clip->#Handles convertedHandles clipref->convertedHandles h PATCH(clip->CHB) while (count) handle PATCH(*h) CalcHRequirements(handle->type, handle) count-- (RT024.)(25985 8) 66 If eveything is empty, there may not even be a clip! if (mem) mem sizeof(RenderClip) Calculate Delay Handle Requirement.if (ResolveLVBAddress(Delay, handle->1vb#, handle 1)) mem. sizeof(RDelayI-andle) akiatei-juise Handle Requirements, if (ResolveLVBAddress(Pause, handle->lvbH, handle 1)) mem. sizeof(RPauseHandle) Calculate Sound Handl Reurents if (ResolveLVBAddress(Sound, handle->lvbH, handh- 1)) 2 £~iciiatemem sizeof(RSoundHandle) CluaeCamera Handle Requirements 15 if (ResolveLVBAddress(Camera, handle->lvb#, handle 1) mem sizeof(RCameraHandle) Calculate Graphic Handle Requirements lvb ResolveLVBAddress(Graphic, handle->lvb#, handle 1) if (!lvb) return memn sizeof(RGraphicHandle) Now add space for timelines AddAbsTLRequirements(&XTranslation) AddAbsTLRequirements(&YTranslation) if (handle->arena Midground) AddAbsTLRequirements(&ZTranslation) AddAbsTLRequirements(&XScale) AddAbsTLRequirements(&YScale) AddAbsTLRequirements(&Existence) We gather the 'data' pointer (to the graphic) data ResolveDataAddress(lvb) CalcObj ects (data) (RT024)(259858) -67- Now calculate requirements for colours and effects too, only cori,:ered with non-sharing colours. Having no colours is automatically sharing with previously defined colours if (data->colours AND ((!lvb->flags.SharedColours) OR (gr~jnrev lvb lvb))) grprev-lvb lvb CalcEffects(data) CalcColours(data) Calclte Objects Now allocate the space for the effects count data->#Shapes s data->Shapes while (count) ruem sizeof(RenderShape) BuildAbsTimeLine(&s->Object) BuildAbsTimeLine(&s->XTranslation) BuildAbsTimeLine(&s->YTranslationi) o s->MaxObjectslnShape RTOProcessorObjectCount o mem o sizeof(RTOProcessorObject) *2 count-- Cal culiate Effects S Now allocate the space for the effects 25 e data->#OpaqueEffects OpaqueEffectsUsage e e data->#TranspEffects TransparentEffectsUsage data->#TranspEffects effect data->Effects while (e) BuildAbsTimeLine(&effects->tend.y) BuildAbsTimeLine(&effects->tend.Cr) BuildAbsTimeLine(&effects->tend.Cb) BuildAbsTimeLine(&effects->tend.mix) (RT024)(259858) 68 BuildAbsTirneLine(&effects->mintend.y) BuildAbsTimeLine(&effects->mintend.Cr) BuildAbsTimeLine(&effects->mintend.Cb) BuildAbsTimeLine(&effects->mintend.mix) BuildAbsTimeLine(&effects->mdmask.y) BuildAbsTimeLine(&effects->mdmask.Cr) BuildAbsTimeLine(&effects->rndmask.Cb) BuildAbsTimeLine(&effects->mdmask.mix) effect++- Calculate Colour Allocate enough space for the colours c data->#Colours) ColourUsage c if (data->ContainsB lends) mem (c sizeof(AnimatedB lend) 2) col data->Colours while (c) BuildAbsTimeLine(&col->StartCoiour.y) *.*BuildAbsTimeLine(&col->StartColour.Cr) BuildAbsTimeLine(&col->StartColour.Cb) BuildAbsTimeLine(&col->StartColour.rnix) if (col->BlendedColour.BlendType 1= FlatColour) BuildAbsTimeLine(&col->EndColour.y) BuildAbsTimeLine(&col->EndColour.Cr) BuildAbsTimeLine(&col->EndColour.Cb) BuildAbsTimeLine(&coL->EndColour.mix) BuildAbsTimeLine(&co->colour.Extent) mem sizeof(TimeLineReference) for startXY (always TL) if (col->BlendedMix.BlendType !=FlatColour) BuildAbsTimeLine(&col->mix.E~xtent) mem sizeof(TimeLineReference) for startXY (always TL) (R-1 024)(259858) L -69ccol++ Add Absolute TL Requirements if (*source AND Ox01) NULL) mem sizeof(TimeLineReference) Calculate Text Handle Requirements We convert the text handle to be a graphic handle, and then calculate the requirements of the graphic handle. We increment the current position in the convertedHandles list so that the next handle that requires converting will be ok S* ConvertTextToGraphic(convertedHandles, textHandle) handle *convertedHandles convertedHandles++ if (handle NULL) 15 CalcHRequirements(handle->type, handle) convertedHandles++ The forgoing describes only one embodiment of the present invention and modifications, obvious to those skilled in the art can be made thereto without departing from the scope of the present invention.

*2 2 i (RT024)(259858)

Claims (22)

1. An automated storyboard rendering system for the reproduction of a storyboard on a real-time processor, said storyboard comprising a number of columns each containing references to at least one information clip intended for reproduction, said system comprising: current column determination means adapted to determine those information clips that are active in a current column, and column rendering means connected to said column determination means and adapted to receive said currently active column and simultaneously render those information clips in said currently active column.

2. A system as claimed in claim 1 wherein at least one of said columns has a multiplicity of simultaneously active clips, said column rendering means renders said simultaneously active clips in a predetermined reproduction order.

3. A. system as claimed in claim 2 wherein said clips are stored within a sequencing order in said columns and said reproduction order has a predetermined correspondence to said sequencing order.

4. A system as claimed in any one of the preceding claims wherein said information clips extend across multiple columns so as to be active in said multiple columns. S: 20 5. A system as claimed in any one of the preceding claims wherein said system further comprises means for simultaneously terminating different information clips which are active in the same column. S6. A system as claimed in any one of the preceding claims wherein said current column determination means includes means for initiating rendering of said storyboard at 25 an arbitrary initial column.

7. A system as claimed in any one of the preceding claims further comprising means for pausing and continuing said rendering at arbitrary points. S. A system as claimed in any one of the preceding claims wherein said information clips comprise audio information.

9. A system as claimed in any one of the preceding claims wherein said information clips comprise displayable image information. A system as claimed in any one of the preceding claims wherein said information clips are animation clips configured for reproduction by display on a display device. eV, ~-(RgV 24)(259858)(CFP0200AU) TT> -71

11. A system as claimed in claim 10 wherein said column rendering means renders said active information clips in real time.

12. A system as claimed in claim 10 or 11 wherein said display occurs in a frame by frame order and said column rendering means begins rendering subsequent frames before the current frame has completed display.

13. A system as claimed in claim 10, 11 or 12 wherein said real-time processor is cable of rendering object based graphics in real time.

14. A system as claimed in claim 10, 11, 12 or 13 wherein each animation clip includes audio information and said column rendering means includes audio output and audio timing information. A system as claimed in claim 14 wherein said column rendering means outputs said audio output and audio timing information on a frame by frame basis.

16. A system as claimed in any one of claims 10 to 15 wherein said column rendering means includes a colour table update means adapted to update a colour table so that it corresponds to the colours used in the currently active column.

17. A system as claimed in any one of claims 10 to 16 wherein said animation clips are made up of objects and said objects are stored in an object based format.

18. An automated storyboard rendering system for the reproduction of a storyboard substantially as described herein with reference to the drawings. 20 19. An automated storyboard rendering system for the reproduction of a storyboard using a real-time processor on a reproduction device, said storyboard comprising a number of columns each containing references to at least one information clip intended for reproduction, each said information clip extending over a plurality of image frames displayed by said reproduction device, said system comprising: 25 current column determination means adapted to determine those information clips that are intended for reproduction, and thereby active, in a current column, and if said current column includes at least one active information clip, identifying said current column as an active column, and column rendering means connected to said column determination means and adapted to receive a current active column and to then simultaneously render in real-time those active information clips in said current active column on said reproduction device. A system as claimed in claim 19 wherein at least one of said columns has a multiplicity of simultaneously active clips, said column rendering means renders said simultaneously active clips in a predetermined reproduction order. R y' TO 4)(259858)(CFP0200AU) T" o 72

21. A system as claimed in claim 20 wherein said clips ar-, stored within a sequencing order in said columns and said reproduction order h, a predetermined correspondence to said sequencing order.

22. A system as claimed in claim 19, 20 or 21 wherein said information clips extend across multiple columns so as to be active in multiple ones of said columns.

23. A system as claimed in any one of claims 19 to xx wherein said system further comprises means for simultaneously terminating different information clips which are active in the same columin.

24. A system as claimed in any one of claims 19 to 23 wherein said current column determination means includes means for initiating rendering of said storyboard at an arbitralv initial column. A system as claimed in any one of claims 19 to 24 further comprising means for pausing and continuing said rendering at arbitrary points in time.

26. A system as claimed any one of claims 19 to 25 wherein said information clips comprise audio information.

27. A system as claimed in any one of claims 19 to 26 wherein said information clips comprise displayable image information.

28. A system as claimed in any one of claims 19 to 27 wherein said information clips are animation clips configured for reproduction by display on a display device. 20 29. A system as claimed in claim 28 wherein said column rendering means includes said real-time processor which renders said active information clips on said display device in real time.

30. A system as claimed in claim 28 wherein said display occurs in a frame by frame order and said column rendering means begins rendering subsequent frames before 25 the current frame has completed display. :331. A system as claimed in claim 29 wherein said real-time processor is capable of rendering object-based graphic image data in real-time. 0 0032. A system as claimed in claim 28 wherein each animation clip includes audio and said column rendering means includes audio output and audio timing information. 3.A system as claimed in claim 32 wherein said column rendering means outputs said audio output and audio timing information on a frame by frame basis.

34. A system as claimed in claim 28 wherein said column rendering means includes a colour table update means adapted to update a colour table so that it corresponds to the colours used in the currently active columrn. [)(259858)(CFPO200AU) -73- A system as claimed in claim 28 wherein said animation clips are made up of graphic objects and said objects are stored in an object-based data format. Dated this Seventh day of August 1996 CANON INFORMATION SYSTEMS RESEARCH AUSTRALIA PTY LTD CANON KABUSHIKI KAISHA Patent Attorneys for the Applicants Spruson Ferguson 0* s so Tr Abstract Render System for the Rendering of Storyboard Structures on a Real Time Animation System. Video and other forms of animation are traditionally manually laid out on storyboards which set cut the nature and timings of the events for which animation is desired. The storyboard then forms a plan for implementation in the creation of the visual performance. A automated storyboard system (Fig. 6) is disclosed which allows the user to flexibly created different sequences of animation clips and to be able to easily alter the ordering and nature of tb;h clips, with the storyboard being automatically rendered on an audio and video processing system (Fig. 7). o* too S Fig. 8 t (RT024)(259858)