AU9734901A - Image compositing with efficient memory usage - Google Patents

Description

S&FRef: 580811

AUSTRALIA

PATENTS ACT 1990 COMPLETE SPECIFICATION FOR A STANDARD PATENT

ORIGINAL

Name and Address of Applicant: Actual Inventor(s): Address for Service: Canon Kabushiki Kaisha 30-2, Shimomaruko 3-chome, Ohta-ku Tokyo 146 Japan George Politis Spruson Ferguson St Martins Tower,Level 31 Market Street Sydney NSW 2000 (CCN 3710000177) Invention Title: Image Compositing with Efficient Memory Usage ASSOCIATED PROVISIONAL APPLICATION DETAILS [33] Country [31] Applic. No(s) AU PR2314 [32] Application Date 22 Dec 2000 The following statement is a full description of this invention, including the best method of performing it known to me/us:i- .:uV 1

II

5815c -1- IMAGE COMPOSITING WITH EFFICIENT MEMORY USAGE Technical Field of the Invention The present invention relates generally to interactive and animated computer graphics and, in particular, to a method and apparatus for the update of computergenerated images in response to changes in a data structure. The invention also relates to a computer program product including a computer readable medium having recorded thereon a computer program for the update of computer-generated images in response to changes in a data structure.

Background Art With improvements in computer technology over the past few decades has come a proliferation of methods and systems for image synthesis and animation. Computer- •"•generated images have been extensively used in human-computer interaction, in which a computer is programmed to modify an image displayed to a user in response to the latter's *ee 15 input. This field of development, the set of techniques it employs, and the images so generated are collectively known as computer graphics.

S" A common software architecture for computer graphics image-synthesis and animation systems is where one party provides an API application programming interface) for a general-purpose software library used for generating and modifying computer graphics, while other parties develop various application programs for a variety of tasks that use the API for graphical output and interaction. Two common examples of prior art graphics APIs are GDI for the Windows T M operating system and QuickDraw T M for Macintosh T M computers.

Some APIs work in what is known as "immediate mode" where an application issues a command to draw a primitive), which the API software performs immediately. Others work in what is known as retained mode where the application 580811.doc -2constructs a representation of the image to be created in terms of the constituent primitives and the operations that combine the primitives for the image. The API software then uses the representation to synthesize the image. Some APIs may work in either mode, such as OpenGL T M a well known 3D graphics API.

When an application using an immediate mode API changes a displayed image, the application typically establishes which parts of the output have to be redrawn, and issues the appropriate commands to the API to do so. If the image has many complex components, the process of changing a displayed image can involve substantial analysis and/or many commands issued to the API. In contrast, for retained mode, an application typically makes changes to an image representation and calls the API to update the displayed image. The retained mode method is usually less onerous to the application than the immediate mode method, the burden of analysis and image regeneration being transferred to the API software. With either of the above mentioned approaches, it is advantageous for some bounds to be placed on the part of the output that needs to be regenerated. Prior art systems typically determine these bounds as the union of the output S. regions occupied by the primitives that have been added, deleted or changed since the last update. By limiting image regeneration to such an update region, both valuable computing time and video memory access time are minimized thus helping output animation flow more smoothly.

Another development in computer graphics has been the use of transparency.

For example, a semi-transparent primitive laid over an opaque primitive may yield a result in which the colours of the two primitives are mixed in proportion to the degree of transparency of the overlying primitive. This technique is commonly called "alphachannel compositing".

Expression trees have been used to represent the composition of images using transparency. Leaf nodes in the tree represent graphical primitives such as scanned 580811.doc -3images, geometric shapes and text, while internal nodes represent compositing operators that combine corresponding operands to synthesize a new image. Operators may, for example, mix transparent colours as described above, or may use the transparency of one operand to limit the visibility of the other, or perform other predefined operations. The root node of the tree represents the final output displayed.

One known prior art retained-mode computer graphics API system uses such an expression tree, hereinafter called a compositing tree, as the image representation. In accordance with this known API system, the application software constructs a compositing tree and then asks for the tree to be rendered to the output device via API calls. The application can make modifications to the tree, for example, in response to user input, and call the API to update the output. This known API system performs *ooo display updates very efficiently. At every node in the compositing tree, the system eo..

maintains a partitioning of the image represented by the node, with a cached image and an identifier (referred to as a contents label) of the contents in each partition. By storing 15 dependencies between content labels, the API system can determine which parts of the image need to be regenerated in response to changes in the tree since the system caches .images at each node and, parts of the image composition that have not changed do not have to be rendered.

S•One disadvantage of the above-described API system is that caching images at every node consumes a considerable amount of memory. Even though some lossless image compression techniques can be used, such as partitions that have the same contents reusing the same image data, it would be better if no images were cached at all. The problem with not caching images is that each time the display needs to be updated in response to changes in the compositing tree, the image data that is presently cached needs to be regenerated, used and discarded. Such a process would ordinarily be too slow for a typical computer even by today's standards. However, in recent times there have been 58081 1.doc -4advances in computer graphics hardware technology that enable alpha-channel compositing of graphics primitives to be performed very quickly and efficiently. Such hardware is now becoming commonplace even in personal computers.

Hardware for graphics acceleration is typically stored on a separate printed circuit board from the CPU and has its own separate video memory. One disadvantage of such hardware is that there is only a limited amount of video memory. Any utilization of such hardware must take into account the limited nature of this resource.

Summary of the Invention It is an object of the present invention to substantially overcome, or at least go: •ameliorate, one or more disadvantages of existing arrangements.

According to one aspect of the present invention there is provided a method of modifying a compositing expression representing an image to optimise memory usage when rendering said image, said image being formed by rendering at least a plurality of graphical objects to be composited according to a hierarchical data structure representing S: said compositing expression, said hierarchical data structure including a plurality of nodes each representing an operand of said compositing expression or an operation for combining said operands, wherein each of said graphical objects is represented by at least one of said operands, said method being characterised in that at least one node representing an operation for combining said operands is modified by interchanging operands associated with said node and altering said operation to form a modified compositing expression for said node, if memory required for rendering and combining said operands associated with said node is minimised by said modification.

According to another aspect of the present invention there is provided a method of modifying a compositing expression representing an image to optimise memory usage when rendering said image, said image being formed by rendering at least a plurality of 58081 1.doc graphical objects to be composited according to a hierarchical structure representing said compositing expression, said hierarchical structure including a plurality of nodes each representing an operand of said compositing expression or an operation for combining said operands, wherein each of said graphical objects is represented by at least one of said operands, said method comprising the steps of: for at least one node representing an operation for combining said operands, determining a first maximum memory value required for rendering and combining operands associated with said node; determining a second maximum memory value required for rendering and combining operands associated with said node, if said operands were interchanged and said operation represented by said node was suitably altered so as to render an image identical to said image; and if said second maximum memory value is less than said first maximum memory value, interchanging said operands associated with said node and altering said %oo .15 operation to form a modified compositing expression for said node.

According to still another aspect of the present invention there is provided a method of modifying a compositing expression representing an image to optimise memory usage when rendering said image, said image being formed by rendering at least 0 a plurality of graphical objects to be composited according to a hierarchical data structure representing said compositing expression, said hierarchical data structure including a plurality of nodes each representing an operand of said compositing expression or an operation for combining said operands, wherein each of said graphical objects is represented by at least one of said operands, said method comprising the steps of: for at least one node representing an operation for combining said components, if at least one operand associated with said node can be rendered and composited without an intermediate buffer, 58081 1.doc -6- (al) determining a first maximum memory value required for rendering said at least one operand and combining said at least one operand with any other operand associated with said node; otherwise, (a2) determining said first maximum memory value on the basis of rendering said at least one operand into an intermediate buffer and compositing said at least one operand with the other operand associated with said node; determining a first provisional maximum memory value required for rendering and compositing operands associated with said node, using said first maximum memory value; S• if said other operand associated with said node can be rendered and oooo ""composited without an intermediate buffer, (cl) determining a second maximum memory value required 15 for rendering said other operand and combining said other operand with said at 9 least one operand associated with said node; otherwise, (c2) determining said second maximum memory value on the S" basis of rendering said other operand into an intermediate buffer and combining said other operand with said at least one operand associated with said node; determining a second provisional maximum memory value required for rendering and combining said operands associated with said node using said second maximum memory value, if a position of said operands were interchanged and said associated operation suitably altered so as to render an image identical to said image; S58081 1doc -7if said second maximum memory value is less than said first maximum memory value, interchanging said operands associated with said node and altering said operation to form a modified compositing expression for said node.

According to still another aspect of the present invention there is provided a method of modifying a compositing expression representing an image to optimise memory usage when rendering said image, said image being formed by rendering at least a plurality of graphical objects to be composited according to a hierarchical data structure representing said compositing expression, said hierarchical data structure including a plurality of nodes each representing an operand of said compositing expression or an operation for combining said operands, wherein each of said graphical objects is represented by at least one of said operands, said method being characterised in that for at least one node representing an operation for combining said operands, if said operation is associative as herein defined, said structure is modified such that the number of intermediate buffers required to render said node is minimised, said modified structure 15 forming a modified compositing expression for said node.

S: According to still another aspect of the present invention there is provided a •method of modifying a compositing expression representing an image to optimise memory usage when rendering said image, said image being formed by rendering at least oeoo• S•a plurality of graphical objects to be composited according to a hierarchical structure representing said compositing expression, said hierarchical structure including a plurality of nodes each representing an operand of said compositing expression or an operation for combining said operands, wherein each of said graphical objects is represented by at least one of said operands, said method comprising at least the step of: for at least one node representing an operation for combining said operands, determining a maximum memory value for rendering and combining operands associated with said node; and 58081 1.doc -8if said maximum memory value is greater than a predetermined memory value, (bl) dividing a display region associated with said node into two subregions; (b2) performing region minimisation as herein defined on each of said subregions; and (b3) performing the method according to claim 1, on each of said subregions and if said second maximum memory value is less than said first maximum memory value for either of said regions, forming a modified compositing expression for said node.

l.:o According to still another aspect of the present invention there is provided a S•method of creating a series of images, said images being formed by rendering and compositing a plurality of graphical objects to be composited according to a hierarchical data structure representing a compositing expression, each of said objects having a 15 predetermined boundary outline, said data hierarchical structure including a plurality of nodes each representing an operand of said compositing expression or an operation for o combining said operands, wherein each of said graphical objects is represented by at least one of said operands, said method comprising the steps of: o• determining a visible region for each node, wherein said visible region is defined by a region outline substantially following at least one of said predetermined outlines or parts thereof, depending on those said objects which contribute to said region, said contribution being determined on the basis of occlusion of said graphical object by any other graphical object, rendering a first image of said series by compositing said graphical objects according to said hierarchical data structure utilising said visible regions; for at least one node representing an operation for combining said operands, 580811.doc

I'

-9determining a first maximum memory value required for rendering and combining operands associated with said node; (ii)determining a second maximum memory value required for rendering and combining operands associated with said node, if said operands were interchanged and said operation represented by said node was suitably altered so as to render said image; and (iii)if said second maximum memory value is less than said first maximum memory value, interchanging said operands associated with said node and altering said associated operation to form a modified compositing expression for said node; 0 in response to at least one change to at least one of said nodes; 0* S determining a further visible region for each node, wherein said *000* further visible region is defined by a region outline substantially following at least one of said predetermined outlines or parts thereof, depending on those said objects S 15 which contribute to said further visible regions; (ii) forming an update region as the union of said visible regions and .said further visible regions for those nodes affected by said at least one change; (iii) creating a next image of said series by compositing said graphical •o•objects according to said hierarchical data structure using said modified compositing expressions formed in step by rendering an image space in which said graphical objects are defined (only) in an area substantially equal to said update region; repeating steps to for further changes to at least one of said nodes.

According to still another aspect of the present invention there is provided an apparatus for modifying a compositing expression representing an image to optimise memory usage when rendering said image, said image being formed by rendering at least a plurality of graphical objects to be composited according to a hierarchical data structure 580811.doc representing said compositing expression, said hierarchical data structure including a plurality of nodes each representing an operand of said compositing expression or an operation for combining said operands, wherein each of said graphical objects is represented by at least one of said operands, said apparatus being characterised by a processor means whereby at least one node representing an operation for combining said operands is modified by said processor means interchanging operands associated with said node and altering said operation to form a modified compositing expression for said node, if memory required for rendering and combining said operands associated with said node is minimised by said modification.

According to still another aspect of the present invention there is provided an apparatus for modifying a compositing expression representing an image to optimise memory usage when rendering said image, said image being formed by rendering at least 1- a plurality of graphical objects to be composited according to a hierarchical structure representing said compositing expression, said hierarchical structure including a plurality ooooo 15 of nodes each representing an operand of said compositing expression or an operation for combining said operands, wherein each of said graphical objects is represented by at least S"one of said operands, said apparatus comprising first determination means for determining a first maximum memory value *required for rendering and combining operands associated with at least one node representing an operation for combining said operands; second determination means for determining a second maximum memory value required for rendering and combining operands associated with said node, if said operands were interchanged and said operation represented by said node was suitably altered so as to render an image identical to said image; and 58081 1.doc -11processor means for interchanging said operands associated with said node and altering said operation to form a modified compositing expression for said node, if said second maximum memory value is less than said first maximum memory value.

According to still another aspect of the present invention there is provided an apparatus for modifying a compositing expression representing an image to optimise memory usage when rendering said image, said image being formed by rendering at least a plurality of graphical objects to be composited according to a hierarchical data structure representing said compositing expression, said hierarchical data structure including a plurality of nodes each representing an operand of said compositing expression or an operation for combining said operands, wherein each of said graphical objects is g* Sg* represented by at least one of said operands, characterised in that for at least one node oooS S. representing an operation for combining said operands, if said operation is associative as :oooo• herein defined, said structure is modified by a processor means such that the number of intermediate buffers required to render said node is minimised, said modified structure 15 forming a modified compositing expression for said node.

According to still another aspect of the present invention there is provided an •11 apparatus for modifying a compositing expression representing an image to optimise :.memory usage when rendering said image, said image being formed by rendering at least *a plurality of graphical objects to be composited according to a hierarchical structure representing said compositing expression, said hierarchical structure including a plurality of nodes each representing an operand of said compositing expression or an operation for combining said operands, wherein each of said graphical objects is represented by at least one of said operands, said apparatus comprising: determination means for determining a maximum memory value for rendering and combining operands associated with at least one node representing an operation for combining said operands; 580811.doc -12dividing means for dividing a display region associated with said node into two subregions if said maximum memory value is greater than a predetermined memory value; region minimisation means for performing region minimisation as herein defined on each of said subregions; and processor means for performing the method according to claim 1, on each of said subregions and if said second maximum memory value is less than said first maximum memory value for either of said regions, and for forming a modified compositing expression for said node.

According to still another aspect of the present invention there is provided a e:o. program for modifying a compositing expression representing an image to optimise e memory usage when rendering said image, said image being formed by rendering at least 0 a plurality of graphical objects to be composited according to a hierarchical data structure representing said compositing expression, said hierarchical data structure including a 15 plurality of nodes each representing an operand of said compositing expression or an operation for combining said operands, wherein each of said graphical objects is represented by at least one of said operands, said program being characterised in that at least one node representing an operation for combining said operands is modified by interchanging operands associated with said node and altering said operation to form a modified compositing expression for said node, if memory required for rendering and combining said operands associated with said node is minimised by said modification.

According to still another aspect of the present invention there is provided a program for modifying a compositing expression representing an image to optimise memory usage when rendering said image, said image being formed by rendering at least a plurality of graphical objects to be composited according to a hierarchical structure representing said compositing expression, said hierarchical structure including a plurality 580811.doc -13of nodes each representing an operand of said compositing expression or an operation for combining said operands, wherein each of said graphical objects is represented by at least one of said operands, said program comprising: for at least one node representing an operation for combining said operands, code for determining a first maximum memory value required for rendering and combining operands associated with said node; code for determining a second maximum memory value required for rendering and combining operands associated with said node, if said operands were interchanged and said operation represented by said node was suitably altered so as to 10 render an image identical to said image; and .0o if said second maximum memory value is less than said first maximum 0 memory value, code for interchanging said operands associated with said node and 0 altering said operation to form a modified compositing expression for said node.

According to still another aspect of the present invention there is provided a o .o program for modifying a compositing expression representing an image to optimise S0: memory usage when rendering said image, said image being formed by rendering at least *•oo.i a plurality of graphical objects to be composited according to a hierarchical data structure 00 representing said compositing expression, said hierarchical data structure including a plurality of nodes each representing an operand of said compositing expression or an operation for combining said operands, wherein each of said graphical objects is represented by at least one of said operands, said program comprising: for at least one node representing an operation for combining said components, if at least one operand associated with said node can be rendered and composited without an intermediate buffer, 58081 1.doc -14code for determining a first maximum memory value required for rendering said at least one operand and combining said at least one operand with any other operand associated with said node; otherwise, code for determining said first maximum memory value on the basis of rendering said at least one operand into an intermediate buffer and compositing said at least one operand with the other operand associated with said node; code for determining a first provisional maximum memory value required for rendering and compositing operands associated with said node, using said first maximum memory value; if said other operand associated with said node can be rendered and composited without an intermediate buffer, code for determining a second maximum memory value o eo required for rendering said other operand and combining said other operand with S•said at least one operand associated with said node; o oo i otherwise, o code for determining said second maximum memory value on the basis of rendering said other operand into an intermediate buffer and combining said other operand with said at least one operand associated with said node; code for determining a second provisional maximum memory value required for rendering and combining said operands associated with said node using said second maximum memory value, if a position of said operands were interchanged and said associated operation suitably altered so as to render an image identical to said image; 58081 1.doc if said second maximum memory value is less than said first maximum memory value, interchanging said operands associated with said node and altering said operation to form a modified compositing expression for said node.

According to still another aspect of the present invention there is provided a program for modifying a compositing expression representing an image to optimise memory usage when rendering said image, said image being formed by rendering at least a plurality of graphical objects to be composited according to a hierarchical data structure representing said compositing expression, said hierarchical data structure including a plurality of nodes each representing an operand of said compositing expression or an operation for combining said operands, wherein each of said graphical objects is represented by at least one of said operands, characterised in that for at least one node •representing an operation for combining said operands, if said operation is associative as herein defined, said structure is modified such that the number of intermediate buffers "'!required to render said node is minimised, said modified structure forming a modified oo compositing expression for said node.

According to still another aspect of the present invention there is provided a .oo.°i program for modifying a compositing expression representing an image to optimise memory usage when rendering said image, said image being formed by rendering at least a plurality of graphical objects to be composited according to a hierarchical structure representing said compositing expression, said hierarchical structure including a plurality of nodes each representing an operand of said compositing expression or an operation for combining said operands, wherein each of said graphical objects is represented by at least one of said operands, said program comprising: for at least one node representing an operation for combining said operands, code for determining a maximum memory value for rendering and combining operands associated with said node; and 580811.doc -16if said maximum memory value is greater than a predetermined memory value, code for dividing a display region associated with said node into two subregions; code for performing region minimisation as herein defined on each of said subregions; and code for performing the method according to claim 1, on each of said subregions and if said second maximum memory value is less than said first maximum memory value for either of said regions, 10 code for forming a modified compositing expression for said node.

°o 00According to still another aspect of the present invention there is provided a program for creating a series of images, said images being formed by rendering and *9° compositing a plurality of graphical objects to be composited according to a hierarchical data structure representing a compositing expression, each of said objects having a predetermined boundary outline, said data hierarchical structure including a plurality of 9.

S. nodes each representing an operand of said compositing expression or an operation for 999999 9 combining said operands, wherein each of said graphical objects is represented by at least one of said operands, said program comprising the steps of: 9 code for determining a visible region for each node, wherein said visible region is defined by a region outline substantially following at least one of said predetermined outlines or parts thereof, depending on those said objects which contribute to said region, said contribution being determined on the basis of occlusion of said graphical object by any other graphical object, code for rendering a first image of said series by compositing said graphical objects according to said hierarchical data structure utilising said visible regions; for at least one node representing an operation for combining said operands, 580811.doc -17code determining a first maximum memory value required for rendering and combining operands associated with said node; code for determining a second maximum memory value required for rendering and combining operands associated with said node, if said operands were interchanged and said operation represented by said node was suitably altered so as to render said image; and if said second maximum memory value is less than said first maximum memory value, code for interchanging said operands associated with said node and altering said associated operation to form a modified compositing expression for 10 said node; in response to at least one change to at least one of said nodes; S code for determining a further visible region for each node, wherein said further visible region is defined by a region outline substantially following at least •one of said predetermined outlines or parts thereof, depending on those said objects •o which contribute to said further visible regions; S•code for forming an update region as the union of said visible regions oeooo and said further visible regions for those nodes affected by said at least one change; and code for creating a next image of said series by compositing said graphical objects according to said hierarchical data structure using said modified compositing expressions formed, by rendering an image space in which said graphical objects are defined (only) in an area substantially equal to said update region.

Brief Description of the Drawings A number of preferred embodiments of the present invention will now be described with reference to the drawings, in which: 580811.doc -18- Fig. 1 is a flowchart showing a method for displaying an image using retainedmode graphics APIs; Fig. 2 is a state transition diagram for nodes in a binary compositing tree; Fig. 3 is a data structure for nodes in a binary compositing tree; Fig. 4 is a flowchart showing a method of rendering a binary-compositing tree; Fig. 5 shows an example compositing tree and the image represented by the tree; Fig. 6 shows have and opaque regions for the nodes in the tree of Fig. Fig. 7 shows the display regions of the nodes in the tree of Fig. Fig. 8(a) shows an image comprising four objects; S' 10 Fig. 8(b) shows a tree that can be used to represent the image of Fig. 8(a); Fig. 8(c) shows a tree that can be used to represent the image of Fig. 8(a); Fig. 9 is a flowchart showing a method for performing an incremental update of an image in response to changes in a binary tree; •Fig. 10 shows a modified version of the tree of Fig. 5 and the correspondingly o4 modified image; "Fig. 11 shows the Display Regions of the nodes in the tree of Fig. .oe.°i Fig. 12 shows the update region for updating the image shown in Fig. 5 to the ott.

image shown in Fig. Fig. 13 shows the compositing tree of Fig. 10 optimized to only render the update region of Fig. 12; Fig. 14 shows a state transition diagram for nodes in an n-ary tree that represent graphical primitives; Fig. 15 shows a state transition diagram for nodes in an n-ary tree that represent compositing operators without the operand independence property; Fig. 16 shows a state transition diagram for operand slots of nodes that represent compositing operators without the operand independence property; 58081 1.doc -19- Fig. 17 shows a state transition diagram for nodes in an n-ary tree that represent compositing operators with the operand independence property; Fig. 18 shows a state transition diagram for operand slots of nodes that represent compositing operators with the operand independence property; Fig. 19 shows a data structure for nodes in an n-ary compositing tree; Fig. 20 shows a data structure for binary nodes derived from n-ary nodes; Fig. 21 shows a flowchart of a method for rendering an n-ary tree; Fig. 22 shows an n-ary tree and an equivalent binary tree; Fig. 23 illustrates the application of the associative law; 10 Fig. 24 shows a flowchart of a method for performing incremental update of an image in response to changes in an n-ary tree; ~Fig. 25 shows a flowchart of a method for rendering a tree subject to memory constraints; and S• Fig. 26 shows a schematic block diagram of a general purpose computer upon which the embodiments of the present invention can be practiced.

Detailed Description including Best Mode Where reference is made in any one or more of the accompanying drawings to steps and/or features, which have the same reference numerals, those steps and/or features 9..

have for the purposes of this description the same function(s) or operation(s), unless the contrary intention appears.

Fig. 1 is a flowchart showing a method for displaying an image. The method of the flow chart of Fig. 1 is preferably implemented as one or more modules of an application program being resident on the hard disk drive 2610 of a computer system 2600 and being read and controlled in its execution by the processor 2605 of the computer system 2600, as described below with reference to Fig. 26. The method preferably utilises a retained-mode graphics API being executed by the procesor 2605. As an 58081 1.doc example, assume that an application program is using a retained-mode computer graphics API to control a graphical user interface. The method begins at step 110, where the processor 2605 constructs a data structure representing the initial appearance of the display. The data structure is preferably in the form of a compositing tree. At the next step 120, the image represented by the data structure is rendered by the API. In accordance with the example above, rendering a compositing tree involves rendering the primitives that are represented by the leaf nodes of the tree and compositing them together according to the operations represented by the internal nodes of the tree. Also at step 120, information to facilitate efficient incremental update of the display is recorded 10 by the processor 2605, as will be described in more detail below. At the next step 170, processor 2605 determines whether to make changes to the tree by following branch 150, or to create a fresh data structure by following branch 160 as the needs of the user interface demand. At the next step 130 changes are made to the data structure. For .ooeo: example, nodes are added, removed or modified. At the next step 140, the API examines the data structure, bounds the portion or portions of the display that need to be rendered to conform to the new form of the data structure and renders said portion or portions.

Preferably, the API records any information needed to facilitate further incremental updates. Following step 140, the processor 2605 can continue to make changes to the tree by returning to step 170 and following branch 150, or the processor 2605 can create a fresh data structure by following branch 160 as the needs of the user interface demand.

A RETAINED-MODE SYSTEM EMPLOYING A BINARY TREE The steps 110 through 170 of the above method will now be explained in further detail. For ease of explanation, the steps shall firstly be described in terms of a binary compositing tree, and later extended to cover n-ary trees. In a binary tree, internal nodes have two descendants, customarily referred to as left and right operands. In an n-ary tree, internal nodes can have any number of descendants. Initially the situation where 580811.doc -21hardware memory limits are not a concern will be considered below. In practice, however, memory limitations exist, especially when using hardware acceleration, and methods to deal with said limitations are described later.

2.1 CONSTRUCTION OF A BINARY COMPOSITING TREE At step 110, in accordance with the example, the compositing tree constructed comprises leaf nodes, representing graphical primitives, and internal nodes, representing operations that composite primitives together to produce more complex graphics. A state transition diagram for binary nodes is shown in Fig. 2, and a data structure for binary nodes in shown in Fig. 3.

10 When a compositing tree is first constructed each node is in state new/modified o 210. This state is recorded in the State field 310 of the node data structure 300. The Display Region 320 is initialized to an empty region, the Dirty Counter 360 to zero, and the Last Count as Root 370 to zero. The purpose of each of these fields shall be described •in more detail below.

2.2 INITIAL RENDERING OF A BINARY COMPOSITING TREE At step 120, the compositing tree is rendered and the resulting output is typically displayed to the user. There may be other reasons for rendering the tree, for example, creating an image that will be saved to file on disk, or that will be saved in memory and subjected to further processing before being displayed to the user. The method described herein works in all such cases. The state of all nodes is moved to updated 220, which is reflected in field State 310. In addition, for each node, a representation of the region of output that is directly affected by the node is saved in field Display Region 320. This representation is not necessarily the same as the entirety of the graphics represented by the node, as for example, the representation can be subjected to clipping or occlusion by other nodes in the compositing tree.

58081 1.doc -22- Fig. 4 is a flowchart showing the method of rendering a binary tree step 120) in more detail and is subdivided into substeps. Each substep involves a depth-first traversal of the binary tree being processed. The method of the flow chart of Fig. 4 is preferably implemented as one or more modules of the application program being resident on the hard disk drive 2610 and being read and controlled in its execution by the processor 2605.

2.2.1 DETERMINE HAVE AND OPAQUE REGIONS In the first traversal of the tree, at step 410, a Have Region 340 and an Opaque Region 350 for each node are determined by the processor 2605. The Have Region 340 is 10 a representation of the geometric area within which the node can supply image data. The o Have Region 340 is preferably approximated, for example, to a bounding box, or to an :'"""irregular region bounded by horizontal and vertical lines, or to a quadtree. The exact representation is not critical, so long as set operations can be efficiently performed. The Opaque Region 350 is a representation of the area within which the node is guaranteed to supply fully opaque data. Again, the Opaque Region 350 may be approximate, so long as no portions that are not opaque are included. The Opaque Region 350 can be empty, for example, for semi-transparent primitives and for detailed primitives such as text for which it is too difficult or costly to determine an opaque region.

The method of determining Have and Opaque Regions will now be described.

For primitives the Have and Opaque Regions are determined from the specification of the primitive itself. Consider, for example, the compositing tree 510, which represents an image 500 as seen in Fig 5. Primitives 512, 516, 517 and 515 are composited using operators 511, 513 and 514. Primitive 512 is an opaque image, primitive 516 is a semitransparent image, primitive 517 is an opaque triangle and primitive 515 is an opaque background covering the entire output area. The have and opaque regions of these 580811.doc -23primitives can be seen in Fig. 6. Note that regions in the example have been approximated to regions bounded by horizontal and vertical lines.

Determination of have and opaque regions of internal nodes depends upon the nature of the operation represented by each node. For alpha-channel compositing operators, Table 1 lists the resultant have and opaque regions of the composition of operands A and B.

a a a a Operator Have Region Opaque Region over A.Have u B.Have A.Opaque u B.Opaque in A.Have n B.Have A.Opaque n B.Opaque out A.Have A.Opaque B.Opaque atop B.Have B.Opaque xor A.Have u B.Have (A.Opaque B.Have) u (B.Opaque A.Have) plus A.Have u B.Have A.Opaque u B.Opaque and A.Have n B.Have A.Opaque n B.Opaque upon B.Have B.Opaque Table 1 Additional operators "and" and "upon" are defined by: A and B (A in B) plus (B in A) and A upon B (A in B) plus B.

It is also useful to define reverse operators for non-commutative compositing operators, as shown in Table 2.

580811.doc -24- Operator Definition rover A rover B B over A rin A rin B B in A rout A rout B B out A ratop A ratop B B atop A rupon A rupon B =B upon A Table 2 During the first pass of the tree a copy of the links to the parent and left and 999* 5 right operands of the node is made in the Working Tree Parent and Left and Right field 9.

380 in each internal node. The subsequent steps will often modify the tree, removing and rearranging nodes. This is best done using the working tree references 380 so as to leave 9.99.. .e 9o "the original tree intact from the point of view of the user.

2.2.2 REGION MINIMIZATION AND OCCLUSION 9 10 At the next step 420, the tree is again traversed to determine the Display Region 320 for each node. A display region for a node is the geometric area of the output that is affected by the graphics represented by the node. The display region for a node is in many cases smaller than the have region for the node since the display region excludes the areas hidden by clipping (such as with "in" or "out" operations), occluded by overlying opaque objects, or outside the bounds of the required output.

The have and opaque regions of the root node are intersected by the region representing the required output area. The process is then continued down the tree from the root node, intersecting both operands' have and opaque regions with the new have region of the parent the root node). For "over" nodes, the opaque region for the 580811.doc operands of the over node is subtracted from the have region for the right operand of the node. The Old Display Region 330 for every node is set to a copy of the Display Region 320 for the node, and the Display Region 320 is set to the resultant have region.

At any point, if the operand of a node has its have region reduced to an empty region, that node and the affected operand are replaced in the working tree by the other operand of the node. The removed node and subtree will no longer participate in further rendering steps 430 and 440. The Display Region 320 of the removed node is set to the have region of the node and the display regions of all nodes in the removed subtree are set to empty regions.

10 Returning to the example compositing tree 510, with have and opaque regions as illustrated in Fig. 6, the display regions 711 to 717 that result from the above process are

S

shown in Fig. 7. The have region 611a for node 511 is the same as the output region, so the have region and opaque region for node 511 remain unchanged. The display region 711 for node 511 is set to the have region 611a for the node 511. The have region 612a

S

for node 512 and opaque region 612b are intersected by have region 611a, again S. S S"unchanged by the operation. The display region 712 for node 512 is set to the have

S

region 612a. The have and opaque regions for node 513 likewise are not affected by the

SS•

parent of node 511 have region 611 a, but since the parent 511 is an "over", the opaque region 612b is subtracted from the have region of node 513. The display region 712 of node 512 is set to its new have region, as shown in Fig. 7. As can be seen in Fig. 7 the new have region excludes an occluded portion 721 that corresponds to the display region 712 of node 512.

At this point, for example, if a change were to be made to node 513, then the change would affect the portion of the output represented by the display region 713 of node 513, which is not the entirety of the output. To update the display only the contents 58081 1.doc -26of region 713 need to be regenerated, saving the time to regenerate the occluded area represented by region 712.

Continuing with the example, the have region 614a of node 514 and opaque region 614b are intersected by the new have region (which is the same as region 713) for node 513, yielding new have and opaque regions. The resultant region 714 of node 514 is set to the new have region, and, as shown in Fig. 7, excludes the portion 719 of the node 514 graphics that are obscured by the opaque image at node 512. The opaque region of node 514 remains empty.

Continuing in this way, nodes 515, 516 and 517 end up with display regions 715, 10 716 and 717 respectively as shown in Fig. 7.

2.2.3 MEMORY OPTIMIZATION The compositing tree produced by step 420 is further processed at step 430 with the aim of minimizing the peak video memory usage during the rendering phase 440.

Given a tree as input to step 430, depending on what operators are used in the internal nodes of the tree, it may be possible to rearrange the tree without changing the output that S° the tree represents. For example, the expressions: A over (B over C);

C*~

(A over B) over C; and A over (C rover B), among other possibilities all represent the same output (apart from rounding errors), achieved by a different sequence of operations. Therefore, different arrangements of a tree would not necessarily consume equal amounts of memory resources during rendering step 440.

Consider the example of Figs. and The same output image 800 as seen in Fig. can be produced by either tree 810 or tree 820 as seen in Figs 8(b) and respectively. However, the alternative trees require different amounts of memory to 58081 1.doc -27store intermediate results. With tree 810, primitive 801 can be rendered into a frame buffer. Then an intermediate buffer is needed for the "plus" operation, in which primitive 802 is first rendered and then composited with primitive 803. The intermediate buffer is then composited into the frame buffer. Finally, primitive 804 is composited into the frame buffer.

In contrast, with tree 820, no intermediate buffer is necessary to render the image 800. Primitive 802 can be rendered into the frame buffer, then primitives 803, 801 and 804 composited in turn.

If memory a memory 2606 of the computer system 2600) usage is a 10 concern, a method for rearranging a given tree is required so as to minimize the peak memory usage for the tree. If rendering and compositing are to be hardware accelerated !'"'"using commodity graphics boards, video memory on these boards is a critical resource whose usage is important to minimize. Even if rendering and compositing were *performed in software using host memory, it is often advantageous to minimize memory usage so as to reduce the risk of virtual memory paging.

Being able to find the optimal tree is dependent on having a means of determining the memory usage of the various alternatives. Some primitives can need to be rendered into intermediate buffers and then composited, while other primitives can be rendered and composited in one step without the need for an intermediate buffer. This may depend on the compositing operator, the type of primitive and the type of hardware available.

When the right-hand operand of a compositing operator is not a primitive, it requires a buffer with size equal to the bounding box of the resultant region.

Buffers exist while a primitive or right-hand subtree is being rendered, then cease to exist, so that the buffer memory can be freely reused in further rendering.

58081 1.doc -28- Therefore the maximum memory use of alternative trees is determined, and the alternative with lowest maximum memory use can be chosen.

Assuming an initial frame buffer and a tree to render into the frame buffer. How much additional memory is required to cope with peak demand? If the root node is a primitive that can be rendered without any additional memory it is a simple primitive), then the answer is zero. If the root node is a primitive that requires an intermediate buffer it is a complex primitive), then the answer is the size of the intermediate buffer. If the root node is a binary node, then the left operand is firstly rendered, during which time some memory can be freed and used, and secondly the right 10 operand is rendered, during which time some memory can be freed and used. The memory use for the root node is the maximum of the left and right node usages. If the 9 right node was itself a binary node, then part of the memory usage is a buffer bounding the result region.

Mathematically, memory usage m(A) of a node A can be defined as follows: m(A) m load(A), if A is a primitive =[max(m(A.left), r A.op(A.right)), if A in binary where m_load(A) is the memory required to render A into a buffer without compositing.

A.left and A.right are the operands of A, and r_op(A) is the memory required to render and composite A as the right operand of operator Function rop is defined as follows: r op(A)=im op(A), if A is a primitive if A is binary where m_op(A) is the memory required to render and composite A as the right operand of operator "op" and bounds(A) is the size of a temporary buffer required for rendering A before composting.

580811.doc -29- In practice, it would be necessary to calculate the memory usage of each leaf node, then each binary node up the tree. This information can be used in choosing between alternatives.

Some compositing operators "over", have the associativity property, for all operands A, B and C, A op (B op C) (A op B) op These operators can be termed "associative". The associativity property can be taken advantage of wherever possible to skew the compositing tree so as to minimize the number of occasions in which intermediate buffers need to be allocated. When this is performed, the region of the intermediate operator node must be recalculated to allow for the fact that the operands •60:0" 10 have changed. The have region of the root node of the affected subtree does not need to oo o be changed since the have region still represents the same image.

S. ~With every binary node in a tree, there is a choice of either leaving the node °O 0° alone or swapping the operands and reversing the operator. The costs of both alternatives oot.. can be determined and the cheaper one chosen. This can be done once for each node as a 6@ee ooo• post-order traversal of the tree is performed. The have regions of the node are not affected by swapping the operands and reversing the operator of the node.

0 2.2.4 RENDERING THE TREE Returning now to Fig. 4, having completed step 430, the processor 2605 0 proceeds to step 440. Rendering the optimized tree involves traversing the tree, first rendering the left operand of a node, then compositing the right operand using the operator of the node. If the left operand is a subtree, the process is applied recursively. If the left operand is a leaf node, the frame buffer is cleared and the primitive represented by the leaf node is rendered into the frame buffer in the bounds of the optimized have region for the node. If the right operand is a leaf node, the right operand is rendered and composited into the frame buffer within the bounds of the optimized have region for the right operand. If the right operand is a subtree, an intermediate buffer is created large 58081 1.doc enough to hold the optimized have region for the right operand, and the process is applied recursively to render the subtree into the intermediate buffer and finally the intermediate buffer is composited into the frame buffer using the operator of the node.

Consider, for example, the tree 510 in Fig. 7. The display regions 711 to 717 shown are in this instance identical to the optimized have regions. Initially, the traversal comes to root node 511. As root node 511 is a binary node, the left operand 512 is rendered. Node 512 is a leaf node, so the frame buffer is cleared and the primitive represented by node 512 is rendered into region 712 of the frame buffer. The traversal then returns to node 512 and proceeds to render the right operand 513. Since operand 513 is a binary node, it cannot be composited directly into the frame buffer, so an intermediate S• buffer large enough to contain region 713 is created. Continuing in this way, the subtree whose root is node 513 is rendered into the intermediate buffer, and returning to node 511, the frame buffer is composited together with the intermediate buffer using the "over" operation of node 511 storing the result in the frame buffer. As the intermediate buffer is 15 no longer required, it can be freed at this point.

oooo Note that it is possible when rendering a subtree into a first intermediate buffer to encounter the need of a further intermediate buffer, and so on recursively. In such a case the above method is followed, treating the first intermediate buffer as the frame buffer and the further intermediate buffer as the intermediate buffer.

2.2.5 SET NODE STATES TO UPDATED The final step in Fig. 4, step 450, involves the processor 2605 traversing the original tree 510 constructed at step 110, prior to the optimizations of steps 420 and 430.

In other words, the original tree linkage is traversed and the Working Tree Parent, Left and Right pointers 380 are ignored. At step 450, the State field 310 of each node is set from new/modified 210 to updated 220.

2.3 MODIFIYING THE BINARY TREE 58081 1.doc -31- After step 450 has been performed, step 120 of Fig. 1 has been completed, a compositing tree has been created, rendered, and the following information has been set: State 310, set to updated; Display Region 320; Dirty Counter 360; and Last Count as root 370.

Before moving on to step 130, it is necessary to explain the role of the Dirty Counter 360 and Last Count as Root 370 fields. The Last Count as Root field 370 is used to determine whether the tree to be rendered is a new tree or a possible modified version of the previous tree. A global counter is initialized to 1 when the system is started. Each time the display is updated, the counter is incremented and the new value is assigned to the Last Count as Root field 370 of the root of the tree used to update the display. The next time the screen is to be updated, the current value of the global counter is compared with the Last Count Root field 370 of the given root node. If they are equal, then it is o.

known that branch 150 has been taken, so the tree can be examined for differences and the display updated incrementally. If the current value and the Last Count as Root differ, then it is known that branch 160 has been followed, so the tree can be rendered to the display afresh in step 120. Since newly created nodes have Last Count as Root field 370 initialized to 0, a newly created root will never match the global counter so the new node will always be treated correctly.

The purpose of the Dirty Counter 360 in each node is to help narrow down the search during step 140 of the parts of the tree that have been modified during step 130.

The current value of the global counter is assigned to the Dirty Counter 360 of every node that is modified itself or that is the parent in the tree of a node that is modified. During 58081 .doc -32update step 140, nodes in any subtree whose associated root Dirty Counter 360 differs from the current value of the global counter are known to be unchanged. Conversely, in any subtree whose associated root Dirty Counter 360 equals the current value of the global counter there will be at least one modified node. Once the update occurs, the global counter is incremented thus efficiently clearing all information about modifications during the previous update cycle, leaving the compositing tree with a clean slate, ready for the next cycle.

Newly created nodes have their Dirty Counter field 360 set to 0, so they can never be considered modified. However, a new node is either used as a new root, thus forcing a full update (rather than an incremental one), or it has to be linked into the tree, thus modifying the new parent node. The entirety of the subtree of the new parent is then treated as the modified node.

A node in the tree can be modified by, for example: unlinking an existing operand; ooeoo linking a new operand; changing an attribute such as shape, colour or degree of transparency; and/or applying a geometric transformation.

The above list is not exhaustive and other modifications are possible without falling o* outside the scope of the invention.

When a node is modified during step 130) the state of the node moves from updated 220 to new/modified 210. The new state is recorded in field State 310 of the data structure 300 associated with the node. In addition, all nodes in the sequence of nodes from the modified node up to the root of the tree need to have their Dirty Counter 360 set to the current value of the global counter. To avoid repeating the same work when several successive changes are made to the same branch of a tree, setting the sequence of 580811.doc -33dirty counter fields can stop when a node has been reached that has already had its dirty counter set to the same value.

2.4 INCREMENTAL UPDATE Continuing on to step 140 of Fig. 1, when modifications to the compositing tree have been made, a request is made to update the displayed output so that the output agrees with the new version of the tree. Preferably, information stored in the nodes in previous steps is used in order to determine what portion or set of portions of the display need to be re-rendered. Step 140 also saves the possibly different display region of each node in readiness for future incremental updates. The global counter is incremented and a new value is copied to the Last Count as Root field 370 of the root node of the compositing S"tree. Step 140 is subdivided into substeps as shown in Fig. 9. Each substep of Fig. 9 involves a depth-first traversal of the binary tree. The method of the flow chart of Fig. 9 is o .e preferably implemented as one or more modules of the application program being resident on the hard disk drive 2610 and being read and controlled in its execution by the .:oo.i processor 2605.

In the first traversal, a step 910, the Have Region 340 and Opaque Region 350 fields for each node are determined by the processor 2605 and a copy of the parent, left and right links is stored in field 370. Step 910 is the same as step 410 of Fig. 4, and can be implemented as a common subroutine.

In the second traversal, at step 920, an optimized region is generated by the processor 2605 for each node and the resulting region is stored in both the Have Region 340 and Display Region 320 fields of each node. Before setting a Display Region 320, the Display Region 320 as set by the previous update is copied to the Old Display Region 330, so that both old and new display regions are available. Regions becoming empty are handled as described above for step 420. Step 920 differs from step 420 only in that the previous value of display region 320 is saved in Old Display Region 330. If the previous 580811.doc -34value of the display region 320 was saved at step 420, it would do no harm, so steps 420 and 920 can be implemented as a common subroutine.

The third traversal, at step 930, determines the update region. The update region represents the geometric portion or set of portions of a display that need to be rendered in order to make the display consistent with the changes in the compositing tree. Starting off with an empty update region, the tree representing the image is traversed and the following actions are performed based on the type and state of each node encountered: Compositing operator in updated state 220: If the left operand is not marked dirty the Dirty Counter 360 of the left operand differs from the current value of the global counter), then the left branch of the tree is not traversed down. Likewise, if the right •operand is not marked dirty, then the right branch of the tree is not traversed down.

•oo• Compositing operator in new/modified state 210: Both the Old Display Region 330 and Display Region 340 for the node are incorporated into the update region. That is, a set union of the Old Display Region 330, the Display Region 340 and the current update region is performed and the result stored as the new value of the update region. The left or right branches are not traversed.

Leaf node in update state 220: Nothing is done.

Leaf node in new/modified state 210: The Old Display Region 330 of the node and Display Region 340 of the node are incorporated into the update region.

At the end of the third traversal, the update region has collected a representation of all the areas of the display that need to be re-rendered. Significantly this method works on a compositing tree rather than, say, a display list, and the method avoids rendering areas where changes are not visible due to occlusion or clipping. Consider, for example, the tree 510 shown in Fig. 5, with display regions assigned to each node during step 420 as shown in Fig. 7. Suppose the tree 510 is modified by moving the position of the triangle represented by node 517 slightly to the left, resulting in the. image 1000 580811.doc represented by the tree 1010 as shown in Fig. 10. This is the only difference between tree 1010 and tree 510. In the process of changing node 517, the state of that node was moved from updated 220 to new/modified 210. All nodes from there to the root, viz., nodes 517, 514, 513 and 511, are marked as dirty by assigning the' current value of the global counter to the Dirty Counter field 350 for each node.

To update the display, steps 910 through 970 as shown in Fig. 9 can be performed by the processor 2605. Step 920 will result in display regions assigned to each node in the Display Region field 320 as shown in Fig. 11. The display regions previously assigned in step 420 and shown in Fig. 7 are saved in the Old Display Region 330 of each node.

Step 930 results in an optimal update region by traversing the tree 1010 in ooo Fig. 10. Firstly, the root node 511 is examined and is a binary node in updated state. The I Dirty Counter 360 of the left child 512 is checked and it is found that the node 511 is not marked dirty. Hence the left branch of the tree 1010 does not need to be checked. The right child 513 however, is marked dirty so the traversal down the right branch is *r continued. Examining node 513, a binary node in updated state is found. The left child 514 is marked dirty so the traversal is continued. Note that the right child 515 is not marked dirty so that the traversal will not extend into that node 515. Continuing in this oo% way, node 517 is arrived at and is in new/modified state. Both the Old Display Region 330, shown at 717 in Fig. 7, and the New Display Region 320, shown at 1117 in Fig. 11, are included in the update region. The combined, and in this case, final update region 1200 is shown in Fig 12. Note that the display region 712 of node 512, which occludes part of modified node 527, is excluded from the update region 1200.

Continuing the process of Fig. 9, the fourth traversal by the processor 2605, at step 940, repeats the region minimisation process of step 920, but instead of using the entire output area of the display as the region with which is intersected with the have 58081 1.doc -36region of the root node, the update region that was calculated in the previous step 930 is used. It is important to note that since the update region is likely to be significantly smaller than the entire output area, large portions of the tree 1010 are likely to be pruned due to the associated have regions being reduced to empty regions. Another difference between steps 940 and 920 is that in step 940 the Display Area 320 or Old Display Area 330 fields of the compositing tree nodes are not set. These must only be set in step 920 (and of course in step 420).

The next traversal, at step 950 is identical to step 430 and optimises the tree to minimise video memory usage.

Step 960 is identical to step 440 and renders the optimized tree to the output ,00:0, device.

Step 970 is identical to step 450 and is performed by the processor 2605 on the original pre-optimized) tree and changes the state (stored in field State 310) of all nodes to be updated 220.

Returning to the example of Fig. 11, the update region 1200 can be used as the output region for a region minimisation process on tree 1010. The update region 1200 is intersected with the have region of root node 511. The resulting have region is intersected with the have regions of the operands, nodes 512 and 513, and so on down the tree. Note that since the have region for node 512 prior to intersection (identical to region 1112 in Fig. 11) and the update region 1200 do not intersect, the have region for node 512 ends up empty, so node 512 and parent node 511 are replaced by the other operand 513. This simplification of the tree will result in more efficient rendering. The resulting tree 1300 with the final have regions is shown in Fig. 13.

The above description describes incremental update being performed on a binary tree. While this may be useful of itself for some applications, a method for incremental 58081 1.doc -37update of output based on n-ary trees will now be described. N-ary trees offer much more flexibility to a user.

A RETAINED-MODE SYSTEM EMPLOYING AN N-ARY TREE A retained mode graphics API using an n-ary compositing tree will typically follow, as in the case of a binary tree already described, the steps shown in Fig. 1. The method constructs a data structure at step 110. This data structure represents the image to be rendered and displayed in the next step 120 and takes the more general form of an nary tree. The method can then continue to modify the n-ary tree at step 130 and invoke the API to correspondingly update the display at step 140, or may at any time start again at step 110 and create a fresh data structure.

3.1 DEFINITION AND PROPERTIES OF N-ARY TREES The preferred form and semantics of the n-ary tree will now be described. Note that some degree of variation is possible without departing from the scope and spirit of the invention. As with a binary tree, an n-ary tree contains nodes that represent graphical primitives, such as images, geometric shapes and text, which are combined using nodes that represent compositing operators. The difference is that a node that represents a compositing operator may have zero, one, two or more operands. The preferred semantics chosen for an n-ary node are as follows. A node utilising an operator f(A, B) with n operands A 1

A

2 An represents: a fully transparent image ifn 0; the operand Al unchanged if n 1; f(Ai, A 2 ifn 2; f(f(Al, A 2

A

3 ifn 3; and

A

2

A

3 An) ifn 3.

Any operand can be independently enabled or disabled. A disabled operand is ignored when determining the image represented by an operator node. For example, an n- 580811.doc -38ary node with five operands, two of which are disabled, is equivalent to a node with the same operator but only the three enabled operands.

Operands can be unlinked from and linked into n-ary nodes. A slot in an n-ary node with no actual operand linked is treated as if that slot was disabled. Empty slots can be inserted into or deleted from the set of operands for an n-ary node.

Given an n-ary node that represents the composition of a group of operands, most compositing operators have the property that if operands are added to or removed from the composition, the portion of the display that needs to be re-rendered in order to bring the output up to date is the union of two regions, the region of the output which the new composition occupies and the region of the output which the previous composition '06:0, occupies. The reason for including the new region is straightforward: it is the region occupied by the graphics that must be rendered and displayed. The reason for including the old region is that the new graphics can occupy a different area that does not fully include the area rendered previously. In that case, the old area must be re-rendered in ooooo order to erase the graphics that used to be displayed and now should no longer be displayed.

A few compositing operators have an advantageous property referred to as "operand independence". If operands are added to or removed from a composition, only the region of the output occupied by the added or removed operands themselves needs to be re-rendered. With such operators, it is not necessary to re-render the entire old and new regions for the composition. Amongst the operators given in Table 1, operators "over", "plus" and "rover" have this property.

Adding an operand to a composition can include either linking an operand into an empty, enabled slot in an n-ary node, or enabling a disabled slot in which an operand is already linked. Similarly, removing an operand from a composition means either unlinking or disabling an enabled operand.

58081 1.doc -39- 3.2 STATE TRANSITIONS Consider now the state transition diagrams in Figs. 14 to 18. Nodes that represent graphical primitives follow the state transition diagram shown in Fig. 14 and will be explained in detail in the following paragraphs. N-ary nodes that represent compositing operators without the operand independence property follow state transition diagrams in Figs. 15 and 16, which again will be explained in detail in the following paragraphs. Nodes that represent operators with the operand independence property follow the diagrams in Figs. 17 and 18.

Before examining Figs. 14 to 18, it is necessary to understand the various operations that can be applied to n-ary nodes and to each of the variable number of 6 00 •operand slots each n-ary node has. A created operand slot can undergo the following operations: enabled; disabled; 15 an operand can be linked in to an empty slot; e an existing operand can be unlinked; or an enabled slot with a linked operand can be involved in a display update (this is only relevant to operators with the operand independence property).

A created node can undergo the following operations: new empty operand slots can be inserted and are marked "insert" in the state diagrams; empty slots can be removed and marked "remove" in the state diagrams; the node can be modified in some way geometrically transformed); an operand can be added to or removed from the composition (as defined above) and are marked "operand" in the state diagrams; the node can be involved in a display update; 580811.doc pending damage can need to be cached and is marked "pending" in the diagrams.

Pending damage will be explained in more detail later in this document and is only relevant to operators with the operand independence property.

A data structure 1900 for nodes in an n-ary tree is shown in Fig. 19. Field State 1910 stores the current state of a node.

An overview of the state transition diagrams in Figs. 14 to 18 will now be explained. A more detailed explanation will follow.

Turning to Fig. 14, each node representing a primitive is initially in state new/modified 1401. When a node is created, action 1411 takes place, which initialises the display region for the node to an empty region. The display region is stored in field Display Region 1915 and is abbreviated to R in the state transition diagrams of Figs. 14 to oF•• o 18. When a node is involved in a screen update, action 1412 takes place and the state moves to the updated state 1402. If the graphical primitive represented by the node is modified in some way, action 1413 takes place and the state of the node returns to new/modified 1401. Finally, if an updated node is involved in a further display update without any intervening modifications being made, action 1414 takes place and the state a.

is left unchanged.

Turning to Fig. 15, each node representing a compositing operator for which operand independence does not hold is initially in state new/modified 1501. The display region is initialized to an empty region by action 1511. When in state new/modified 1501, any changes made to the node do not affect the state of the node and do not result in any actions. When the node is involved in a display update, action 1512 takes place and the state moves to the updated state 1502. When in state updated 1502, the node responds differently to each operation that can happen. If empty operand slots are inserted or removed, no change in state occurs and no action takes place since empty operand slots are ignored in the composition and hence do not affect the image to be displayed in any 580811.doc -41way. If the updated node is modified in some way, for example, by a geometric transformation, or if an operand is added to or removed from the composition, action 1513 takes place and the state of the node returns to new/modified 1501. Finally, if an updated node is involved in a further display update, action 1514 takes place and the state remains unchanged.

Fig. 16 shows a state transition diagram for an operand slot of a node for an operator without the operand independence property. A newly inserted operand slot is in state 1601, in which the operand is both disabled and null absent). A state 1602 is also a suitable initial state. If an operand is linked in, the slot moves to state 1603. Then if the operand slot is enabled, the slot moves to state 1604. Similarly, if a slot in state 1601 is enabled, the slot moves to state 1602. If an operand is then linked in, the slot moves to 600e state 1604. Note that all these transitions are reversible. Transitions between states 1604 00 0*P0 *r sand 1602, and between states 1604 and 1603 invoke action 1611. Action 1611 indicates to the node that an operand has been added to or removed from the composition, which can @0S055 15 affect the state of the node and invoke further action.

S*06 Turning to Fig. 17, each node representing a compositing operator for which 0S 00 operand independence holds is initially in state new/modified 1701. The display region is initialized to an empty region by action 1711. When in state new/modified 1701, any changes made to the node do not affect the state of the node and do not result in any actions. When the node is involved in a display update, action 1712 takes place and the state moves to updated 1702. When in the updated state 1702, the node responds differently to each operation that can happen. If empty operand slots are inserted or removed, no change in state occurs and no action takes place, for the reasons discussed above. If an operand is added to or removed from the composition to or from state 1804 in Fig. 18, action 1713 takes place. However, if an operand is removed from the composition such that pending damage must be saved from state 1505 in Fig. 18), 580811.doc -42action 1714 takes place. If the updated node is modified in any other way, for example, by a geometric transformation, action 1715 takes place and the state of the node returns to new/modified 1701. Finally, if an updated node is involved in a further display update, action 1714 takes place and the state remains unchanged.

Fig. 18 shows a state transition diagram for an operand slot of a node for an operator with the operand independence property. A newly inserted operand slot is in state 1801, in which the operand is both disabled and null absent). Note again that state 1802 is also a suitable initial state. If an operand is linked in, the slot moves to state 1803. Then if the slot is enabled, the slot moves to state 1804. Similarly, if a slot in state 1801 is enabled, the slot moves to state 1802. If an operand is then linked in, the slot moves to state 1804. Note that all these transitions are reversible. Transitions between e states 1804 and 1802, and between states 1804 and 1803 invoke action 1811 in an analogous manner to nodes for compositing operator without operand independence, as discussed for Fig. 16.

°°eee Slots in nodes for operators with operand independence also undergo state S. transitions when the node is involved in a screen update. Continuing with Fig. 18, when *an operand slot is in one of states 1801, 1802 or 1803, the slot is not involved in the composition, so a display update has no effect on the state. If the node is in state new °o S° 1804, a display update invokes action 1812 and the state moves to the updated state 1805.

Action 1812 is part of action 1716 shown in Fig. 17 if the node is in state updated 1702. A further display update when the operand slot is already in state updated 1805 leaves the state unchanged and invokes no action. Finally, if an operand whose slot is in state updated 1805 is removed from the composition, whether by disabling or unlinking, action 1813 takes place. Action 1813 indicates to the node to save pending damage for use during the next update, and is part of action 1713 if the node is in state updated 1702.

580811.doc -43- For n-ary nodes with operand independence, a set of boolean flags are stored in field Link Flags 1945 in n-ary node data structure 1900 shown in Fig. 19. Each flag represents whether the corresponding operand slot is in updated state 1805 or in one of the other states shown in Fig. 18.

3.3 CONSTRUCTION OF AN N-ARY COMPOSITING TREE The steps of Fig. 1 with respect to n-ary trees as the image representation data structure will now be explained in more detail. At step 110, a compositing tree is constructed by the processor 2605. All nodes are initially in state new/modified states 1401, 1501, 1701), represented by a value in field State 1910 of n-ary node data structure 1900. The display region of each new node, stored in field Display Region 1915, is initialized to an empty region. The Dirty Counter 1925 and Last Count as Root 1930 fields are initialized to zero, as in the case of a binary tree. For n-ary nodes that represent compositing operators with the operand independence property, a set of flags in field Link Flags 1945 are set to indicate that operand slots are not updated.

eeooe 3.4 INTITIAL RENDERING OF AN N-ARY COMPOSITNG TREE ~At step 120, the n-ary compositing tree being processed by the processor 2605 is .rendered and the resulting output is displayed. Also at step 120, various actions as indicated in the state transition diagrams in Figs. 14, 15, 16, 17 and 18 take place and the e• state of all nodes is moved to updated 1402, 1502, 1702), this being reflected in field State 1910. For each node, the display region is stored in field Display Region 1915, as is the case of a binary tree.

For nodes representing an operator with operand independence, the pending damage region stored in field Damage Region 1920 is initialized to an empty region. This region is abbreviated to P in the Figs. 14 to 18. Operand slots are moved to state updated 1805 where appropriate, this being indicated in field Link Flags 1945. Note that the field Damage Region 1920 is used by nodes with operator independence for storing pending 58081 1.doc -44damage, as will be explained later. In addition, the field Damage Region 1920 is used by all nodes to store the previous display region, as was field Old Display Region 330 for a binary compositing tree.

For n-ary nodes, step 120 is subdivided into substeps as shown in the flowchart of Fig. 21. Each substep 2110 to 2150 involves a depth-first tree traversal being performed by the processor 2605.

3.4.1 CREATE BINARY TREE WITH HAVE AND OPAQUE REGIONS During a first traversal of the n-ary tree being processed, at step 2110, the method creates a binary tree that is equivalent to the given n-ary tree. In accordance with the above mentioned definition ofn-ary nodes: an n-ary node with no enabled, non-null operands is represented in the equivalent i" binary tree by a node that represents a fully transparent image; an n-ary node with a single enabled, non-null operand is replaced by the operand the node is effectively ignored); ooo.oi an n-ary node with exactly two enabled, non-null operands is replaced by a single binary node with the same two operands; and an n-ary node with three or more enabled, non-null operands is replaced by a sequence of two or more binary nodes with operands chosen in accordance with the definition.

As an example, Fig. 22 shows an n-ary node 2210 and a binary tree 2220 that is equivalent to the n-ary node 2210. The n-ary node 2210 with operator "over" has six operands. In accordance with the example, operand 2212 B is disabled and Operand 2215 is null. The four remaining operands 2211 A, 2213 C, 2214 D and 2216 E are themselves replaced by equivalent binary trees 2221 2223 2224 D' and 2226 These equivalent binary trees 2221 2223 2224 D' and 2226 E' are combined to create a binary tree 2220 that is equivalent to the n-ary node 2210. The binary nodes 2221 A' and 580811.doc 2223 C' corresponding to the first two enabled n-ary operands 2211 A and 2213 C are first composited. Then each remaining enabled, non-null operand is composited in turn to the result of the previous composition.

A preferred data structure 2000 for each binary node is shown in Fig. 20. Each node has a reference to its parent node in the binary tree in field Parent 2010. Each node also has a reference to the equivalent n-ary node in field Equivalent N-ary 2025. If several binary nodes are created for the one n-ary node, then the top-most most ancestral) binary node stores the reference to the corresponding n-ary node. When an n-ary node has exactly one enabled, non-null operand, both the n-ary node and its operand correspond to the same binary node. In this instance, field Equivalent N-ary 2025 acts like the first element of a list of equivalent n-ary nodes. Field Next Equivalent N-ary 1935 in the n-ary node data structure 1900 continues this list.

S Binary nodes also store a node type in field Node Type 2035. Preferably, there are three types, "void", "operator" and "primitive". Void binary nodes represent fully 15 transparent images and are derived from n-ary nodes that represent compositing operators but which have no enabled, non-null operands. Void binary nodes are preferably S•optimized out before rendering takes place. Operator nodes represent compositing operations, in which case field operator 2040 stores the compositing operator and fields °e S•Left 2045 and Right 2050 store the left and right operands of the operator. Primitive nodes represent graphical primitives, in which case field Primitive 2055 stores a reference to the specification of the primitive, for example, a pointer to the n-ary tree node for the primitive.

As the binary tree is being created at step 2110, the processor 2605 also constructs the have and opaque regions of the nodes in the binarytree in the same manner as described for binary compositing tees. The binary tree can be created and the have and 580811.doc -46opaque regions constructed in two steps. However, the binary tree and the have and opaque regions can easily be constructed in one step, and thus execution time is saved.

Pseudocode for step 2110 is as follows: procedure nary_tobinary(n-ary tree t) begin procedure new_primitive_node(n-ary node p) begin b allocate binary node.

b.Parent null.

b.Equivalent N-ary p, p.Next Equivalent N-ary null. Create list of one equivalent n-ary node.

b.Node Type primitive.

15 b.Primitive p.

Sreturn b.

end procedure new_operatornode(op, binary nodes left, right) 20 begin b allocate binary node.

b.Parent null.

b.Equivalent N-ary null. Create empty list of equivalent n-ary nodes.

b.Node Type operator.

25 b.Operator op.

b.Left left.

b.Right right.

left.Parent b.

right.Parent b.

return b.

end procedure newvoid_node(p) begin b allocate binary node.

b.Parent null.

b.Equivalent N-ary p, p.Next Equivalent N-ary null. Create list of one equivalent n-ary node.

b.Node Type void.

return b.

end 58081 1.doc -47nary_to_binary...

Perform depth-first traversal ofn-ary tree t, visiting each node p: When arriving at a primitive: b new_primitive_node(p).

Determine the primitive's opaque and have regions and store in b.Opaque Region and b.Have Region.

When first arriving at an operator node: p.Binary Tree Temp null.

Continue traversal down to each enabled, non-null operand.

After each operand of operator node p is visited: At each stage, p.Binary Tree Temp stores the binary tree corresponding to the composition of all operands up to but not including the current one. b will store the binary tree corresponding to the current operand.

if p.Binary Tree Temp null then p.Binary Tree Temp b thefirst enabled, non-null operand.

else begin x p.Binary Tree Temp.

if p's compositing operator is a reverse operator ("rover", "rin etc.) then p.Binary Tree Temp new_operatornode(reverse of p's compositing S 20 operator, b, The initial binary tree has no reverse operators in order to simplify later processing.

else p.Binary Tree Temp newoperatornode(p's compositing operator, x, b).

Determine p.Binary Tree Temp.Opaque Region, Have Region from x.Opaque S• 25 Region, Have Region and b.Opaque Region, Have Region.

end Before leaving an operator node: o* if p.Binary Tree Temp null then b newvoidnode(p). There were no enabled, non-null operands.

else begin Finished all operands, so p.Binary Tree Temp stores the binary tree equivalent to n-ary tree p.

b p.Binary Tree Temp. p's parent node is expecting b to be set.

Insert p into b's list of equivalent n-ary nodes.

end End of traversal.

return b.

end nary_to binary 3.4.2 REGION MINIMIZATION AND OCCLUSION The next step shown in Fig. 21, step 2120, is performed on the newly created binary tree, in a similar manner to that described earlier for a binary compositing tree.

580811.doc -48- The difference is that while the have and opaque regions are stored in the binary nodes, the display region and the old display region are stored in the n-ary nodes. Thus, when the final value of Have Region 2020 is determined, each of the n-ary nodes in the list of equivalent n-aries is examined (starting from field Equivalent N-ary 2025) and for each n-ary node, Damage Region 1920 is set to Display Region 1915 and Display Region 1915 is set to the Have Region field 2020 of the binary node. Step 2120 is equivalent to the first two steps 1421 and 1422 of action 1412 in Fig. 14, the first two steps 1521 and 1522 of action 1512 in Fig. 15 and the first two steps 1721 and 1722 of action 1712 in Fig. 17.

The third step 1423, 1523, 1723) does not need to be performed as part of step 120 because the entire output area is being updated and so an update area is not needed.

At step 2120, the binary tree constructed in step 2110 by the processor 2605 can be changed to prune out clipped and occluded nodes, as described for step 420. Void binary nodes are also optimized out in step 2120. In step 420, working tree pointers were used when modifying the tree so as to leave the data structure unchanged. This is 15 unnecessary in step 2102 since the original data structure is the n-ary tree.

Pseudocode for step 2120 is as follows: procedure region_minimization(binary tree t, want_region,flag updatenary_regions) begin procedure clobber(binary tree t) begin Set to empty the display regions of all n-ary nodes referred to from the binary nodes in a given subtree.

Perform depth-first traversal of binary tree t, visiting each node b: for all nodes p in b's list of equivalent n-ary nodes do begin if p.State new/modified then Transfer p.Display Region to p.Damage Region.

p.Display Region 0.

end End of traversal.

580811.doc -49end region_minimization...

Restrict root node's region to want region.

t.Have Region t.Have Region n wantregion.

if t.Have Region 0 then begin The case of the vanishing root.

if update_nary_rgns then clobber(t).

t null. There is nothing to render.

end else begin Perform depth-first traversal of binary tree t, visiting each node b: At every node do the following as soon as we come to it: if updatenary_rgns then for all nodes p in b's list of equivalent n-ary nodes do begin 20 if p.State new/modified then Transfer p.Display Region to p.Damage Region.

Set p.Display Region to a copy ofb.Have Region.

end If the node is an internal node (not a leaf), also do the following when we first come to it: Restrict both operands' result regions to the parent's result region.

b.Left.Have Region b.Left.Have Region n b.Have Region.

b.Right.Have Region b.Right.Have Region n b.Have Region.

Make left operand of "over" occlude the right.

if b.Operator "over" then b.Right.Have Region b.Right.Have Region b.Left.Opaque Region.

.Handle the cases where one of the operands has vanished.

if b.Left.Have Region 0 then begin if update_nary_rgns then clobber(b.Left).

Replace b with its right subtree.

Continue traversal by visiting (former) right node.

end else if b.Right.Have Region 0 then begin if updatenary_rgns then clobber(b.left).

Replace b with its left subtree.

580811.doc Continue traversal by visiting (former) left node.

end End of traversal.

end end region_minimization This procedure is called with parameter update_nary_regions passed a boolean true value.

3.4.3 MEMORY OPTIMIZATION The binary tree resulting from step 2120 is further processed by the processor 2605 at step 2130 with the view of minimizing the peak memory the memory 2606) usage during the rendering phase 2140. The method used at step 2130 is the same as that described above for binary composting trees.

Pseudocode for step 2130 is as follows: o 15 procedure memory_optimization(binary tree t) begin Perform depth-first traversal of binary tree t, visiting each node b: When arriving at a leaf node, do the following: b.Cost memory needed to load a primitive of type of b.Primitive in area of b.Have 20 Region.

When first arriving at an operator node, do the following: if b.Operator is associate then begin Apply associativity property.

25 while b.Right is not a leaf and b.Right.Operator b.Operator do begin Rearrange b.Left, b.Right, b.Right.Left and b.Right.Right according to associative law, keeping b at the root. As shown in Figure 23, this requires rearranging nodes in tree 2310 to form tree 2320.

if b.Operator is "over" or "plus" then begin b.Left.Opaque Region b.Left.Left.Opaque Region u b.Left.Right.Opaque Region.

b.Left.Have Region b.Left.Left.Have Region u b.Left.Right.Have Region.

end end end After visiting an operator node, do the following before returning to its parent: 580811.doc -51 Firstly determine cost ofpresent arrangement.

if b.Right is an operator node then cost b.Right.Cost memory cost of compositing a subtree of area of b.Right.Have Region with operator b.Operator.

else cost memory needed to composite a primitive of type of b.Right.Primitive, of area of b.Right.Have Region, with operator b.Operator.

cost 4- max(cost, b.Left.Cost).

Secondly determine cost of alternative arrangement.

if b.Left is an operator node then alt_cost b.Left.Cost memory cost of compositing a subtree of area of b.Left.Have Region with reverse of operator b.Operator.

else alt_cost memory needed to composite a primitive of type of b.Left.Primitive, of area of b.Left.Have Region, with reverse of operator b.Operator.

alt_cost max(alt_cost, b.Right.Cost).

If the alternative is better then swap the operands.

if alt cost cost then begin 20 Interchange b's left and right operands.

Reverse b's operator.

b.Cost altcost.

end else b.Cost cost.

End of traversal.

end memory_optimization 3.4.4 RENDERING THE TREE 30 Continuing with Fig. 21, having completed step 2130, the processor 2605 proceeds to step 2140 and the image represented by the tree is rendered. As the tree is now an optimized binary tree, the same method as described for step 220 above is followed.

Pseudocode for step 2140 is as follows: procedure composite_tree(binary tree t) begin Set up the initial frame buffer.

Perform depth-first traversal of binary tree t, visiting each node b: 580811.doc -52- When arriving at a leaf node, do the following: Render b.Primitive into the current frame buffer.

When arriving at an internal node after having visited the left child, do the following: if b.Right is a leaf node then begin Render and composite b.Right.Primitive, with b.Operator as the compositing operator and b.Right.Have Region as the region to composite.

Do not continue traversal down to the right child; go back to b's parent node.

end else Create a new frame buffer for the right subtree and push it on the stack.

When arriving at an internal node after having visited the right child, do the following: Composite the two top buffers on the stack, which represent b's left and right operands, storing the result in the lower buffer.

Discard the top buffer from the stack.

End of traversal.

end composite_tree 20 3.4.5 SET STATES TO UPDATED The final step in Fig. 21, step 2150, includes the processor 2605 traversing the original n-ary tree constructed in step 110. At step 2150, any node involved the 0 node is not a disabled operand or part of a disabled operand) in the update is moved to state updated (1402, 1502, 1702) by setting Field State 1910 to the appropriate value. All operand slots of n-ary nodes involved in the update that are enabled and have operands presented are moved to state updated 1805 by setting the appropriate flag in field Link Flags 1945. Field Damage Region 1920 is set to an empty region in each updated node, finishing the last step 1724 of action 1712.

Pseudocode for step 2150 is as follows: procedure set_updatedflags(n-ary tree t) begin Perform depth-first traversal of n-ary tree t, visiting each node p: When first arriving at each node, do the following: p.State updated.

p.Damage Region 0.

580811.doc -53- Before descending down to each operand i of compositing operator node p: if operand i is null or disabled then Do not continue traversal down to child i.

if The operator represented by node p has operand independence property ("over", "rover" or "plus") then Set flag for operand i in p.Link Flags to indicate that the operand slot is updated.

End of traversal.

end 3.5 MODIFYING THE N-ARY TREE As described earlier for binary trees, a global counter can be used along with fields Dirty Counter 1925 and Last Count as Root 1930 in each node to help find changes to a tree between updates and to detect if the root node of the tree has changed since the previous update.

o• Moving to step 130 of Fig. 1, the processor 2605 can make modifications to the 9 n-ary compositing tree in accordance with general graphical output requirements. Some changes need to be made to the n-ary nodes' state and data structure in response to the various modifications that may be made to the n-ary compositing tree. A typical API will have procedures for enabling an application to make changes to the image representation 20 data structure. These procedures will also preferably perform the steps described below.

*9*999 *••oo If a node that represents a graphical primitive is modified, and that node is in the updated state 1402, then the state of the node is changed to new/modified 1401 and action 999999 1413 shown in Fig. 14 is performed. This action involves setting the Dirty Counter field 1925 to the current value of the global counter in the sequence of nodes to form the modified node at the root of the tree. If a node in this sequence is encountered that has already had the associated dirty counter set to the same value, then it is not necessary to proceed further as this indicates that another modification has set the dirty counters from that point on. Finally, if a node is encountered that is a disabled operand, then again it is 580811.doc -54not necessary to proceed further, as the modified node is not part of the displayed image and hence modification of the node does not effect any change.

Pseudocode for action 1413 is as follows: procedure modify_node(node) begin if node.State updated then begin node.State new/modified.

set_dirty(node).

end Else we do not need to mark it as dirty because it has either already been marked dirty or it is a new node. A new node will either be used as a new root, or it will be linked somewhere and its parent will be marked as dirty.

15 end procedure set_dirty(node) begin while node.Dirty Counter global counter do S* 20 begin node.Dirty Counter global counter.

if node has no parent or node is disabled in its parent then Return.

node node.Parent.

.i 25 end end If a node that corresponds to a compositing operation has a new operand linked into an enabled slot or an existing disabled operand to such a node is enabled, then there are a number cases to consider. If the node is in state new/modified 1501 or 1701, no action need happen. If the node represents an operator without the operand independence property and is in state updated 1502, then action 1513 takes place and the state of the node is changed to new/modified 1501. Action 1513 is identical to action 1413 with pseudocode given above. If the node represents an operator with operand independence, and is in state updated 1702, then action 1714 takes place and the state is left unchanged.

Pseudocode for action 1714 is as follows: 580811.doc Handle an operand enabled/linked into a node whose operator has operand independence property.

if node.State updated then set_dirty(node).

If a node that corresponds to a compositing operation has an operand unlinked from an enabled slot or an enabled operand of such a node is disabled, then again there are a number of cases. Firstly, if the node is in state new/modified 1501 or 1701, no action is taken. Secondly, if the node represents an operator without the operand independence property and is in state updated 1502, then action 1513 takes place and the state of the node is changed to new/modified 1501. Thirdly, if the node represents an operator with operand independence and is in state updated 1702, then two further cases exist. Either the operand slot involved is in state new 1804 or in state updated 1805. If 15 the operand is in state new 1804, then action 1714 as described above takes place. If the operand is in state updated 1805, then action 1713 takes place. Action 1713 sets the dirty counters as described earlier, but also saves the unlinked or disabled operands display region as pending damage for the next update.

Pseudocode for action 1713 is as follows: *fee.: 4o6:00 .Handle an operand disabled/unlinked from a node whose operator has operand independence property.

if node.State updated then begin if node.Link Flag for operand to be disabled/unlinked is updated then begin node.Damage Region node.Damage Region u operand node.Display Region.

Clear node.Link Flag for operand.

end set_dirty(node).

end 580811.doc -56- Finally, if a change is made to a node that represents a compositing operation which changes the image in some way, then, if the node is in state updated 1502 or 1702, action 1513 or 1715 takes place and the state of the node is changed to new/updated 1501 or 1701. Actions 1513 and 1715 are identical to action 1413 described above.

The method described above covers modification of existing nodes. It is also possible to create new nodes and delete nodes that are no longer required. Any new node is initially in state new/modified (1401, 1501 or 1701). When a node is added to the compositing tree the node is linked in, thereby modifying the new parent node of the particular node. Similarly, when a node is to be deleted, the node is first unlinked from the compositing tree. In this way, creation and deletion of nodes is handled by treating the node as a modification of the parent node involved.

3.6 INCREMENTAL UPDATE .Continuing on to step 140 of Fig. 1, when modifications to the n-ary compositing tree have been made, a request is made to update the displayed output so that the output agrees with the new version of the tree. For n-ary trees, step 140 is subdivided into substeps as shown in Fig. 24. Each step involves a depth-first tree traversal. The method o•.

of the flow chart of Fig. 24 is preferably implemented as one or more modules of the application program being resident on the hard disk drive2610 and being read and S"controlled in its execution by the processor 2605.

The first traversal, at step 2410, is the same as at step 2110. Procedure naryto_binary can be used.

The second traversal, at step 2420, is on the newly created binary tree. Step 2420 is the same as step 2120. Procedure region_minimiztion can be used, again with a boolean value of true passed for parameter updatenaryregions. Recall that procedure regionminimiztion performs steps 1421 and 1422 of action 1412, steps 1521 and 1522 of action 1512, and steps 1721 and 1722 of action 1712 for new and modified nodes. For 580811.doc -57nodes that have not been modified, this procedure performs actions 1414, 1514 and step 1732 of action 1716.

The third traversal by the processor 2605, at step 2430, is on the n-ary tree, and the processor 2605 determines the update region in a similar way as described for binary trees. As the n-ary tree is traversed, only branches that are enabled and also whose Dirty Counter field 1925 equals the current value of the global counter are followed. When any node in new/modified state is reached, the new and old display regions for the node, stored in fields Display Region 1915 and Damage Region 1920 are included in the update region. Step 2430 is equivalent to step 1423 of action 1412, step 1523 of action 1512 and step 1723 of action 1712. When an updated node that represents an operator with operand independence is reached, the effects of any removed or added operands are included in the update region. The Damage Region field 1920 contains the union of the display regions of all operands that were removed since the last update, and is included in the update area, as indicated in step 1734 of action 1716. Each enabled non-null operand whose operand slot is in state new 1804 represents an added operand. Each such root node operand display region is included in the update area, as indicated by action 1812, 0o.

Sthus performing step 1734 of action 1716.

Pseudocode for step 2430 is as follows: o. procedure obtain_updateregion(n-ary tree t) begin S 0. Start with an empty update region and add to it as we traverse the tree and discover nodes that have changes since the last update.

Perform depth-first traversal of n-ary tree t, visiting each node p: When first arriving at each node, do the following: if p.State new/modified then begin S S u p.Damage Region u p.Display Region.

p.Damage Region O.

Do not traverse this node's children; go back to its parent.

580811.doc -58end When first arriving at an operator node, do the following: if p.Operator has operand independence property ("over", "rover" or "plus") then begin S S u p.Damage Region.

p.Damage Region 0.

end Before descending down to each operand i of compositing operator node p: if operand i is null or disabled then Skip this operand; continue with next operand, if any.

else if p.Operator has operand independence property ("over", "rover" or "plus") and flag for operand i in p.Link Flags indicates that it is not updated then begin S S u operand i's Damage Region.

Do not traverse this down to this operand; go to next operand.

end else if operand i's Dirty Counter the global counter then Skip this operand; continue with next operand, if any.

End of traversal.

20 return S.

S* end obtain_updateregion Continuing with Fig. 24, step 2440 is a repetition of the region minimisation process of step 2420 but with two important differences. The first difference is that the 25 region to which all output is restricted is the update region determined in step 2430, instead of the entire output display area used in step 2420. This has the effect of producing a minimal tree in terms of the output that needs to be generated to update the display. Portions of the tree that do not affect the part of the output that has changed are pruned from the tree, as a result of which the subsequent rendering step is made more efficient. The second difference between step 2440 and step 2420 is that a boolean value of false is passed for parameter update_naryregions, which stops the n-ary tree nodes' display regions being set. The display regions are already set by step 2420 as setting them at step 2420 would result in invalid regions being stored.

580811.doc -59- The remaining steps in Fig. 24, namely 2450, 2460 and 2470 are the same as steps 2130, 2140 and 2150 described earlier. Pseudocode has been presented above for steps 2450, 2460 and 2470.

HANDLING MEMORY LIMITATIONS The methods described above perform various optimizations prior to rendering.

In particular, the binary form of the compositing tree representing an image is rearranged as far as possible to minimize the peak video memory usage during the rendering phase.

However, it may happen that a compositing tree is sufficiently complex that even the optimal form requires more memory than is available. It would be desirable to have a means of producing the required output image, even if perhaps at the expense of increased e:i •execution time.

Preferably, the region to be rendered, whether the entire output area (in step 120) *or the updated area (in step 140), is subdivided. For each subdivision, a copy of the corresponding tree can be made, region minimisation and memory optimisation applied with respect to the smaller region of output, and the re-optimized tree rendered. For each subdivision, the re-optimized tree will be simpler, and the amount of video memory required less, so that it is more likely to be able to be rendered within the available memory constraints. If, however, there is still insufficient memory, the process can be S"repeated recursively by the processor 2605. That is, the subdivided area may itself be further subdivided, and so on, until memory constraints are satisfied.

Preferably the subdivision is performed using a quadtree. The bounds of the output region are halved both vertically and horizontally, creating four quadrants. Any quadrant that requires further subdivision can then be similarly subdivided into four.

However, other methods of subdivision are possible and fall within the scope of the invention. Fig. 25 shows a flowchart for performing step 960 (which is the same as step 2460). It is important the procedure regionminimization is called with a boolean value of 580811.doc false passed for parameter updatenary rgns. The process of the flow chart of Fig. 25 is preferably implemented as one or more modules of the application program being resident on the hard disk drive 2610 and being read and controlled in its execution by the processor 2605.

The process of Fig. 25 begins at step 2510 where a check is performed by the processor 2605 to determine if there is sufficient memory the memory 2606) to render and composite the image represented by a compositing tree being processed. If the result of step 2510 is true then the process proceeds to step 2520 where the image represented by the tree is rendered and composited. If the result of step 2510 is false then the process proceeds to step 2530 where the output area is subdivided into four quadrants by the processor 2605. At the next step 2540, a copy of the binary tree representing the o first quadrant is made. The process continues at the next step 2541, where the regionminimization procedure is performed by the processor 2605 on the first quadrant.

At the next step 2542, memory optimisation as described above is performed. The process continues at the next step 2543, where step 960 is performed by the processor 2605 on the copy of the tree made at step 2540. At the next step 2550, a copy of the o•.

*•"binary tree representing the second quadrant is made. The process continues at the next step 2551, where the regionminimization procedure is performed on the second quadrant o by the processor 2605. At the next step 2552, memory optimisation as described above is performed. The process continues at the next step 2553, where step 960 is performed on the copy of the tree made at step 2550. At the next step 2560, a copy of the binary tree representing the third quadrant is made. The process continues at the next step 2561, where the regionminimization procedure is performed on the third quadrant. At the next step 2562, memory optimisation as described above is performed. The process continues at the next step 2563, where step 960 is performed on the copy of the tree made at step 2560. At the next step 2570, a copy of the binary tree representing the fourth quadrant is 580811.doc -61made by the processor 2605. The process continues at the next step 2571, where the regionminimization procedure is performed on the fourth quadrant. At the next step 2572, memory optimisation as described above is performed. The process continues at the next step 2573, where step 960 is performed on the copy of the tree made at step 2570 and the process concludes.

The aforementioned preferred methods comprise a particular control flow. There are many other variants of the preferred methods which use different control flows without departing the spirit or scope of the invention. Furthermore one or more of the steps of the preferred methods can be performed in parallel rather then sequentially.

The preferred methods are preferably practiced using a conventional generalpurpose computer system 2600, such as that shown in Fig. 26 wherein the processes of S Figs. 1 to 25 can be implemented as software, such as an application program executing within the computer system 2600. In particular, the steps of the preferred methods are effected by instructions in the software that are carried out by the computer. The software can be divided into two separate parts; one part for carrying out either of the preferred methods; and another part to manage the user interface between the latter and the user.

o•.

S°The software can be stored in a computer readable medium, including the storage devices described below, for example. The software is loaded into the computer from the S"computer readable medium, and then executed by the computer. A computer readable medium having such software or computer program recorded on it is a computer program product. The use of the computer program product in the computer preferably effects an advantageous apparatus for preforming the methods in accordance with the invention.

The computer system 2600 comprises a computer module 2601, input devices such as a keyboard 2602 and mouse 2603, output devices including a printer 2615 and a display device 2614. A Modulator-Demodulator (Modem) transceiver device 2616 is used by the computer module 2601 for communicating to and from a communications 580811.doc -62network 2620, for example connectable via a telephone line 2621 or other functional medium. The modem 2616 can be used to obtain access to the Intemrnet, and other network systems, such as a Local Area Network (LAN) or a Wide Area Network (WAN).

The computer module 2601 typically includes at least one processor unit 2605, a memory unit 2606, for example formed from semiconductor random access memory (RAM) and read only memory (ROM), input/output interfaces including a video interface 2607, and an 110 interface 2613 for the keyboard 2602 and mouse 2603 and optionally a joystick (not illustrated), and an interface 2608 for the modem 2616. A storage device 2609 is provided and typically includes a hard disk drive 2610 and a floppy disk drive 2611. A magnetic tape drive (not illustrated) may also be used. A CD- ROM drive2612 is typically provided as a non-volatile source of data. The

V.

components 2605 to 2613 of the computer module 2601, typically communicate via an interconnected bus 2604 and in a manner which results in a conventional mode of operation of the computer system 2600 known to those in the relevant art. Examples of computers on which the embodiments can be practised include IBM-PC's and compatibles, Sun Sparcstations or alike computer systems evolved therefrom.

•ooTypically, the application program of the preferred embodiment is resident on the hard disk drive 2610 and read and controlled in its execution by the processor 2605.

S° Intermediate storage of the program and any data fetched from the network 2620 may be accomplished using the semiconductor memory 2606, possibly in concert with the hard disk drive 2610. In some instances, the application program may be supplied to the user encoded on a CD-ROM or floppy disk and read via the corresponding drive 2612 or 2611, or alternatively may be read by the user from the network 2620 via the modem device 2616. Still further, the software can also be loaded into the computer system 2600 from other computer readable medium including magnetic tape, a ROM or integrated circuit, a magneto-optical disk, a radio or infra-red transmission channel between the 580811.doc -63computer module 2601 and another device, a computer readable card such as a PCMCIA card, and the Intemrnet and Intranets including email transmissions and information recorded on websites and the like. The foregoing is merely exemplary of relevant computer readable mediums. Other computer readable media may be practiced without departing from the scope and spirit of the invention.

The preferred methods can alternatively be implemented in dedicated hardware such as one or more integrated circuits performing the functions or sub functions of Figs.

1 to 25. Such dedicated hardware can include graphic processors, digital signal processors, or one or more microprocessors and associated memories.

The foregoing describes only some embodiments of the present invention, and .9 :oo modifications and/or changes can be made thereto without departing from the scope and °o ,6 Sspirit of the invention, the embodiments being illustrative and not restrictive.

o.9° 0 In the context of this specification, the word "comprising" means "including principally but not necessarily solely" or "having" or "including" and not "consisting only 15 of'. Variations of the word comprising, such as "comprise" and "comprises" have corresponding meanings.

6 0 0 °OOOO9 S. 9 *99000 58081 1.doc

Claims (9)

1. A method of modifying a compositing expression representing an image to optimise memory usage when rendering said image, said image being formed by rendering at least a plurality of graphical objects to be composited according to a hierarchical data structure representing said compositing expression, said hierarchical data structure including a plurality of nodes each representing an operand of said compositing expression or an operation for combining said operands, wherein each of said graphical objects is represented by at least one of said operands, said method being characterised in that at least one node representing an operation for combining said operands is modified by interchanging operands associated with said node and altering said operation to form a modified compositing expression for said node, if memory required for rendering and combining said operands associated with said node is So minimised by said modification. 15

2. The method according to claim 1, said method comprising the steps of: for at least one node representing an operation for combining said operands, 0: determining a first maximum memory value required for rendering and combining operands associated with said node; 000005 determining a second maximum memory value required for rendering and combining operands associated with said node, if said operands were interchanged and said operation represented by said node was suitably altered so as to render an image identical to said image; and if said second maximum memory value is less than said first maximum memory value, interchanging said operands associated with said node and altering said operation to form said modified compositing expression for said node. 58081 1.doc

3. The method according to claim 1, said method comprising the steps of: for at least one node representing an operation for combining said components, if at least one operand associated with said node can be rendered and composited without an intermediate buffer, (al) determining a first maximum memory value required for rendering said at least one operand and combining said at least one operand with any other operand associated with said node; otherwise, (a2) determining said first maximum memory value on the basis of rendering said at least one operand into an intermediate buffer and S°compositing said at least one operand with the other operand associated with said oooo S"node; determining a first provisional maximum memory value required for rendering and compositing operands associated with said node, using said first maximum memory value; if said other operand associated with said node can be rendered and ".composited without an intermediate buffer, o• (cl) determining a second maximum memory value required for rendering said other operand and combining said other operand with said at least one operand associated with said node; otherwise, (c2) determining said second maximum memory value on the basis of rendering said other operand into an intermediate buffer and combining said other operand with said at least one operand associated with said node; 5808 11.doc -66- determining a second provisional maximum memory value required for rendering and combining said operands associated with said node using said second maximum memory value, if a position of said operands were interchanged and said associated operation suitably altered so as to render an image identical to said image; if said second maximum memory value is less than said first maximum memory value, interchanging said operands associated with said node and altering said operation to form a modified compositing expression for said node.

4. The method according to claim 3, wherein step includes the further sub-steps of: (bl) determining a third maximum memory value required for rendering one operand associated with said node; S (b2) determining a fourth maximum memory value required for rendering one operand associated with said node and combining said one operand with the other 15 operand associated with said node; and (b3) comparing said third and fourth operands. o#

5. The method according to claim 1, comprising at least the step of recomputing at ooooo least one region associated with said node.

6. The method according to any one of claims 1 to 5, said method further including the step of recording which nodes of said hierarchical data structure are affected by said modification.

7. A method of modifying a compositing expression representing an image to optimise memory usage when rendering said image, said image being formed by

580811.doc -67- rendering at least a plurality of graphical objects to be composited according to a hierarchical structure representing said compositing expression, said hierarchical structure including a plurality of nodes each representing an operand of said compositing expression or an operation for combining said operands, wherein each of said graphical objects is represented by at least one of said operands, said method comprising the steps of: for at least one node representing an operation for combining said operands, determining a first maximum memory value required for rendering and combining operands associated with said node; determining a second maximum memory value required for rendering and combining operands associated with said node, if said operands were interchanged and said operation represented by said node was suitably altered so as to render an image identical to said image; and if said second maximum memory value is less than said first maximum memory value, interchanging said operands associated with said node and altering said S•-o operation to form a modified compositing expression for said node. e•

8. A method of modifying a compositing expression representing an image to .ee.ei S"optimise memory usage when rendering said image, said image being formed by rendering at least a plurality of graphical objects to be composited according to a hierarchical data structure representing said compositing expression, said hierarchical data structure including a plurality of nodes each representing an operand of said compositing expression or an operation for combining said operands, wherein each of said graphical objects is represented by at least one of said operands, said method comprising the steps of: for at least one node representing an operation for combining said components, 580811.doc -68- if at least one operand associated with said node can be rendered and composited without an intermediate buffer, (al) determining a first maximum memory value required for rendering said at least one operand and combining said at least one operand with any other operand associated with said node; otherwise, (a2) determining said first maximum memory value on the basis of rendering said at least one operand into an intermediate buffer and compositing said at least one operand with the other operand associated with said node; determining a first provisional maximum memory value required for rendering and compositing operands associated with said node, using said first maximum memory value; if said other operand associated with said node can be rendered and 15 composited without an intermediate buffer, (cl) determining a second maximum memory value required •for rendering said other operand and combining said other operand with said at ".least one operand associated with said node; otherwise, (c2) determining said second maximum memory value on the basis of rendering said other operand into an intermediate buffer and combining said other operand with said at least one operand associated with said node; determining a second provisional maximum memory value required for rendering and combining said operands associated with said node using said second maximum memory value, if a position of said operands were interchanged and said associated operation suitably altered so as to render an image identical to said image; 580811.doc -69- if said second maximum memory value is less than said first maximum memory value, interchanging said operands associated with said node and altering said operation to form a modified compositing expression for said node.

9. The method according to claim 8, wherein step includes the further sub-steps of: (bl) determining a third maximum memory value required for rendering one operand associated with said node; (b2) determining a fourth maximum memory value required for rendering one operand associated with said node and combining said one operand with the other •.operand associated with said node; and (b3) comparing said third and fourth operands. A method of modifying a compositing expression representing an image to optimise memory usage when rendering said image, said image being formed by rendering at least a plurality of graphical objects to be composited according to a hierarchical data structure representing said compositing expression, said hierarchical data structure including a plurality of nodes each representing an operand of said oe.oei S"compositing expression or an operation for combining said operands, wherein each of said graphical objects is represented by at least one of said operands, characterised in that for at least one node representing an operation for combining said operands, if said operation is associative as herein defined, said structure is modified such that the number of intermediate buffers required to render said node is minimised, said modified structure forming a modified compositing expression for said node. 580811.doc 11. The method according to claim 10, further comprising the step of recomputing at least one region associated with said node. 12. A method of modifying a compositing expression representing an image to optimise memory usage when rendering said image, said image being formed by rendering at least a plurality of graphical objects to be composited according to a hierarchical structure representing said compositing expression, said hierarchical structure including a plurality of nodes each representing an operand of said compositing expression or an operation for combining said operands, wherein each of said graphical objects is represented by at least one of said operands, said method comprising at least the step of: for at least one node representing an operation for combining said operands, determining a maximum memory value for rendering and combining operands associated with said node; and if said maximum memory value is greater than a predetermined memory value, (bl) dividing a display region associated with said node into two ~subregions; o• (b2) performing region minimisation as herein defined on each of said subregions; and (b3) performing the method according to claim 1, on each of said subregions and if said second maximum memory value is less than said first maximum memory value for either of said regions, forming a modified compositing expression for said node. 580811.doc -71- 13. A method of creating a series of images, said images being formed by rendering and compositing a plurality of graphical objects to be composited according to a hierarchical data structure representing a compositing expression, each of said objects having a predetermined boundary outline, said data hierarchical structure including a plurality of nodes each representing an operand of said compositing expression or an operation for combining said operands, wherein each of said graphical objects is represented by at least one of said operands, said method comprising the steps of: determining a visible region for each node, wherein said visible region is defined by a region outline substantially following at least one of said predetermined outlines or parts thereof, depending on those said objects which contribute to said region, said contribution being determined on the basis of occlusion of said graphical object by 9. 9. any other graphical object, rendering a first image of said series by compositing said graphical objects according to said hierarchical data structure utilising said visible regions; for at least one node representing an operation for combining said operands, determining a first maximum memory value required for rendering 9999.9 and combining operands associated with said node; (ii)determining a second maximum memory value required for 9999•9 rendering and combining operands associated with said node, if said operands were interchanged and said operation represented by said node was suitably altered so as to render said image; and (iii)if said second maximum memory value is less than said first maximum memory value, interchanging said operands associated with said node and altering said associated operation to form a modified compositing expression for said node; in response to at least one change to at least one of said nodes; 58081 I.doc -72- determining a further visible region for each node, wherein said further visible region is defined by a region outline substantially following at least one of said predetermined outlines or parts thereof, depending on those said objects which contribute to said further visible regions; (ii) forming an update region as the union of said visible regions and said further visible regions for those nodes affected by said at least one change; (iii) creating a next image of said series by compositing said graphical objects according to said hierarchical data structure using said modified compositing expressions formed in step by rendering an image space in which said graphical objects are defined (only) in an area substantially equal to said update region; repeating steps to for further changes to at least one of said nodes. 14. An apparatus for modifying a compositing expression representing an image to optimise memory usage when rendering said image, said image being formed by rendering at least a plurality of graphical objects to be composited according to a hierarchical data structure representing said compositing expression, said hierarchical data structure including a plurality of nodes each representing an operand of said compositing expression or an operation for combining said operands, wherein each of said graphical objects is represented by at least one of said operands, said apparatus being characterised by a processor means whereby at least one node representing an operation for combining said operands is modified by said processor means interchanging operands associated with said node and altering said operation to form a modified compositing expression for said node, if memory required for rendering and combining said operands associated with said node is minimised by said modification. The apparatus according to claim 14, said apparatus comprising: 580811.doc -73- first determination means for determining a first maximum memory value required for rendering and combining operands associated with' at least one node representing an operation for combining said operands; second determination means for determining a second maximum memory value required for rendering and combining operands associated with said node, if said operands were interchanged and said operation represented by said node was suitably altered so as to render an image identical to said image, and wherein said processor means interchanges said operands associated with said node and alters said operation to form said modified compositing expression for said node, if said second maximum memory 10 value is less than said first maximum memory value. 0 16. An apparatus for modifying a compositing expression representing an image to optimise memory usage when rendering said image, said image being formed by rendering at least a plurality of graphical objects to be composited according to a hierarchical structure representing said compositing expression, said hierarchical structure including a plurality of nodes each representing an operand of said compositing S° expression or an operation for combining said operands, wherein each of said graphical objects is represented by at least one of said operands, said apparatus comprising S" first determination means for determining a first maximum memory value required for rendering and combining operands associated with at least one node representing an operation for combining said operands; second determination means for determining a second maximum memory value required for rendering and combining operands associated with said node, if said operands were interchanged and said operation represented by said node was suitably altered so as to render an image identical to said image; and 58081 1.doc -74- processor means for interchanging said operands associated with said node and altering said operation to form a modified compositing expression for said node, if said second maximum memory value is less than said first maximum memory value. 17. An apparatus for modifying a compositing expression representing an image to optimise memory usage when rendering said image, said image being formed by rendering at least a plurality of graphical objects to be composited according to a hierarchical data structure representing said compositing expression, said hierarchical data structure including a plurality of nodes each representing an operand of said compositing expression or an operation for combining said operands, wherein each of said graphical objects is represented by at least one of said operands, characterised in that for at least one node representing an operation for combining said operands, if said S.operation is associative as herein defined, said structure is modified by a processor means such that the number of intermediate buffers required to render said node is minimised, said modified structure forming a modified compositing expression for said node. ooo 18. The apparatus according to claim 17, wherein said processor means recomputes at least one region associated with said node. ooooo 19. An apparatus for modifying a compositing expression representing an image to optimise memory usage when rendering said image, said image being formed by rendering at least a plurality of graphical objects to be composited according to a hierarchical structure representing said compositing expression, said hierarchical structure including a plurality of nodes each representing an operand of said compositing expression or an operation for combining said operands, wherein each of said graphical objects is represented by at least one of said operands, said apparatus comprising: 580811.doc determination means for determining a maximum memory value for rendering and combining operands associated with at least one node representing an operation for combining said operands; dividing means for dividing a display region associated with said node into two subregions if said maximum memory value is greater than a predetermined memory value; region minimisation means for performing region minimisation as herein defined on each of said subregions; processor means for performing the method according to claim 1, on each of said subregions and if said second maximum memory value is less than said first maximum memory value for either of said regions, and for forming a modified compositing expression for said node. :oo A program for modifying a compositing expression representing an image to optimise memory usage when rendering said image, said image being formed by S°.i rendering at least a plurality of graphical objects to be composited according to a hierarchical data structure representing said compositing expression, said hierarchical data structure including a plurality of nodes each representing an operand of said oooo• compositing expression or an operation for combining said operands, wherein each of said graphical objects is represented by at least one of said operands, said program being characterised in that at least one node representing an operation for combining said operands is modified by interchanging operands associated with said node and altering said operation to form a modified compositing expression for said node, if memory required for rendering and combining said operands associated with said node is minimised by said modification. 58081 1.doc -76- 21. The program according to claim 20, said program comprising: for at least one node representing an operation for combining said operands, code for determining a first maximum memory value required for rendering and combining operands associated with said node; code for determining a second maximum memory value required for rendering and combining operands associated with said node, if said operands were interchanged and said operation represented by said node was suitably altered so as to render an image identical to said image; and if said second maximum memory value is less than said first maximum memory 10 value, code for interchanging said operands associated with said node and altering said o• operation to form said modified compositing expression for said node. 22. The program according to claim 20, said program comprising: for at least one node representing an operation for combining said components, if at least one operand associated with said node can be rendered and :composited without an intermediate buffer, 9**o* S"code for determining a first maximum memory value required for rendering said at least one operand and combining said at least one operand with any other operand associated with said node; otherwise, code for determining said first maximum memory value on the basis of rendering said at least one operand into an intermediate buffer and compositing said at least one operand with the other operand associated with said node; 580811.doc -77- code for determining a first provisional maximum memory value required for rendering and compositing operands associated with said node, using said first maximum memory value; if said other operand associated with said node can be rendered and composited without an intermediate buffer, code for determining a second maximum memory value required for rendering said other operand and combining said other operand with said at least one operand associated with said node; otherwise, code for determining said second maximum memory value on o. ~the basis of rendering said other operand into an intermediate buffer and S•combining said other operand with said at least one operand associated with said node; code for determining a second provisional maximum memory value required for rendering and combining said operands associated with said node using said second maximum memory value, if a position of said operands were interchanged and e• said associated operation suitably altered so as to render an image identical to said image; if said second maximum memory value is less than said first maximum memory value, code for interchanging said operands associated with said node and altering said operation to form a modified compositing expression for said node. 23. The program according to claim 22, wherein said program further comprises: code for determining a third maximum memory value required for rendering one operand associated with said node; 580811.doc -78- code for determining a fourth maximum memory value required for rendering one operand associated with said node and combining said one operand with the other operand associated with said node; and code for comparing said third and fourth operands. 24. The program according to claim 20, further comprising code for recomputing at least one region associated with said node. The program according to any one of claims 20 to 24, said program further comprising code for recording which nodes of said hierarchical data structure are affected OS. O ooby said modification. #oo *o 4* S. 26. A program of modifying a compositing expression representing an image to optimise memory usage when rendering said image, said image being formed by 15 rendering at least a plurality of graphical objects to be composited according to a hierarchical structure representing said compositing expression, said hierarchical structure S. including a plurality of nodes each representing an operand of said compositing expression or an operation for combining said operands, wherein each of said graphical objects is represented by at least one of said operands, said program comprising: for at least one node representing an operation for combining said operands, code for determining a first maximum memory value required for rendering and combining operands associated with said node; code for determining a second maximum memory value required for rendering and combining operands associated with said node, if said operands were interchanged and said operation represented by said node was suitably altered so as to render an image identical to said image; and 58081 1.doc -79- if said second maximum memory value is less than said first maximum memory value, code for interchanging said operands associated with said node and altering said operation to form a modified compositing expression for said node. 27. A program for modifying a compositing expression representing an image to optimise memory usage when rendering said image, said image being formed by rendering at least a plurality of graphical objects to be composited according to a hierarchical data structure representing said compositing expression, said hierarchical data structure including a plurality of nodes each representing an operand of said compositing expression or an operation for combining said operands, wherein each of said graphical objects is represented by at least one of said operands, said program comprising: for at least one node representing an operation for combining said components, o if at least one operand associated with said node can be rendered and S 15 composited without an intermediate buffer, code for determining a first maximum memory value required for rendering said at least one operand and combining said at least one operand with any other operand associated with said node; otherwise, oeooo code for determining said first maximum memory value on the basis of rendering said at least one operand into an intermediate buffer and compositing said at least one operand with the other operand associated with said node; code for determining a first provisional maximum memory value required for rendering and compositing operands associated with said node, using said first maximum memory value; 58081 1.doc if said other operand associated with said node can be rendered and composited without an intermediate buffer, code for determining a second maximum memory value required for rendering said other operand and combining said other operand with said at least one operand associated with said node; otherwise, code for determining said second maximum memory value on the basis of rendering said other operand into an intermediate buffer and combining said other operand with said at least one operand associated with said node; code for determining a second provisional maximum memory value required for rendering and combining said operands associated with said node using said second maximum memory value, if a position of said operands were interchanged and said S•associated operation suitably altered so as to render an image identical to said image; 15 if said second maximum memory value is less than said first maximum memory value, interchanging said operands associated with said node and altering said operation to form a modified compositing expression for said node. ooooo 28. The program according to claim 27, further including: oo oi code for determining a third maximum memory value required for rendering one operand associated with said node; code for determining a fourth maximum memory value required for rendering one operand associated with said node and combining said one operand with the other operand associated with said node; and code for comparing said third and fourth operands. 580811.doc -81- 29. A program for modifying a compositing expression representing an image to optimise memory usage when rendering said image, said image being formed by rendering at least a plurality of graphical objects to be composited according to a hierarchical data structure representing said compositing expression, said hierarchical data structure including a plurality of nodes each representing an operand of said compositing expression or an operation for combining said operands, wherein each of said graphical objects is represented by at least one of said operands, characterised in that for at least one node representing an operation for combining said operands, if said operation is associative as herein defined, said structure is modified such that the number of intermediate buffers required to render said node is minimised, said modified structure forming a modified compositing expression for said node. o• oooo The program according to claim 29, further comprising code for recomputing at least one region associated with said node. A program for modifying a compositing expression representing an image to :optimise memory usage when rendering said image, said image being formed by ooooo rendering at least a plurality of graphical objects to be composited according to a hierarchical structure representing said compositing expression, said hierarchical structure including a plurality of nodes each representing an operand of said compositing expression or an operation for combining said operands, wherein each of said graphical objects is represented by at least one of said operands, said program comprising: for at least one node representing an operation for combining said operands, code for determining a maximum memory value for rendering and combining operands associated with said node; and 580811.doc -82- if said maximum memory value is greater than a predetermined memory value, code for dividing a display region associated with said node into two subregions; code for performing region minimisation as herein defined on each of said subregions; and code for performing the method according to claim 1, on each of said subregions and if said second maximum memory value is less than said first maximum memory value for either of said regions, code for forming a modified compositing expression for said node. 32. A program for creating a series of images, said images being formed by rendering and compositing a plurality of graphical objects to be composited according to -a hierarchical data structure representing a compositing expression, each of said objects 15 having a predetermined boundary outline, said data hierarchical structure including a plurality of nodes each representing an operand of said compositing expression or an .i operation for combining said operands, wherein each of said graphical objects is *ooo° represented by at least one of said operands, said program comprising the steps of: code for determining a visible region for each node, wherein said visible region is defined by a region outline substantially following at least one of said predetermined outlines or parts thereof, depending on those said objects which contribute to said region, said contribution being determined on the basis of occlusion of said graphical object by any other graphical object, code for rendering a first image of said series by compositing said graphical objects according to said hierarchical data structure utilising said visible regions; for at least one node representing an operation for combining said operands, 58081 1.doc -83- code determining a first maximum memory value required for rendering and combining operands associated with said node; code for determining a second maximum memory value required for rendering and combining operands associated with said node, if said operands were interchanged and said operation represented by said node was suitably altered so as to render said image; and if said second maximum memory value is less than said first maximum memory value, code for interchanging said operands associated with said node and altering said associated operation to form a modified compositing expression for said node; in response to at least one change to at least one of said nodes; code for determining a further visible region for each node, wherein said further visible region is defined by a region outline substantially following at least one of said predetermined outlines or parts thereof, depending on those said objects S 15 which contribute to said further visible regions; code for forming an update region as the union of said visible regions and said further visible regions for those nodes affected by said at least one change; and code for creating a next image of said series by compositing said graphical objects according to said hierarchical data structure using said modified compositing expressions formed, by rendering an image space in which said graphical objects are defined (only) in an area substantially equal to said update region. 33. A program according to any one of claims 20 to 32, wherein said program is stored in a memory configured within an apparatus and is executed by a processor means in said apparatus. 58081 1.doc -84- 34. A computer readable medium, having a program recorded thereon, where the program is configured to make a computer execute a procedure to perform the method of any one of claims 1 to 8. A method of modifying a compositing expression representing an image to optimise memory usage when rendering said image, substantially as herein described with reference to any one of the embodiments with reference to Figs. 1 to 26. DATED this Eighteenth Day of December 2001 Canon Kabushiki Kaisha S Patent Attorneys for the Applicant S SPRUSON FERGUSON SS S S S S 58081 .doc