AU772753B2 - Incremental update of computer graphics - Google Patents

Description

S&FRef: 580809

AUSTRALIA

PATENTS ACT 1990 COMPLETE SPECIFICATION FOR A STANDARD PATENT

ORIGINAL

Name and Address of Applicant Actual Inventor(s): Address for Service: Invention Title: Canon Kabushiki Kaisha 30-2, Shimomaruko 3-chome, Ohta-ku Tokyo 146 Japan George Politis Spruson Ferguson St Martins Tower,Level 31 Market Street Sydney NSW 2000 (CCN 3710000177) Incremental Update of Computer Graphics ASSOCIATED PROVISIONAL APPLICATION DETAILS [33] Country [31] Applic. No(s) AU PR2310 [32] Application Date 22 Dec 2000 The following statement is a full description of this invention, including the best method of performing it known to me/us:- 5815c -1- INCREMENTAL UPDATE OF COMPUTER GRAPHICS Technical Field of the Invention The present invention relates generally to interactive and animated computer graphics and, in particular, to a method and apparatus for the update of computergenerated images in response to changes in a data structure. The invention also relates to a computer program product including a computer readable medium having recorded thereon a computer program for the update of computer-generated images in response to changes in a data structure.

Background Art With improvements in computer technology over the past few decades has come a proliferation of methods and systems for image synthesis and animation. Computer- V. generated images have been extensively used in human-computer interaction, in which a computer is programmed to modify an image displayed to a user in response to the latter's 15 input. This field of development, the set of techniques it employs, and the images so .o.oo: S. generated are collectively known as computer graphics.

A common software architecture for computer graphics image synthesis and 0V0 animation systems is where one party provides an Application Programming Interface S::1 (API) for a general-purpose software library used for generating and modifying computer ooo 20 graphics, while other parties develop various application programs for a variety of tasks that use the API for graphical output and interaction. Two common examples of prior art graphics APIs are GDI for the WindowsTM operating system and QuickDraw T M for MacintoshTM computers.

Some APIs work in what is known as "immediate mode" where an application issues a command to draw a primitive), which the API software performs immediately. Others work in what is known as retained mode where the application 580809.doc -2constructs a representation of the image to be created in terms of the constituent primitives and the operations that combine the primitives for the image. The API software then uses the representation to synthesize the image. Some APIs may work in either mode, such as OpenGL T M a well known 3D graphics API.

When an application using an immediate mode API changes a displayed image, the application typically establishes which parts of the output have to be redrawn, and issues the appropriate commands to the API to do so. If the image has many complex components, the process of changing a displayed image can involve substantial analysis and/or many commands issued to the API. In contrast, for retained mode, an application typically makes changes to an image representation and calls the API to update the S.displayed image. The retained mode method is usually less onerous to the application than the immediate mode method, the burden of analysis and image regeneration being transferred to the API software. With either of the above mentioned approaches, it is advantageous for some bounds to be placed on the part of the output that needs to be regenerated. Prior art systems typically determine these bounds as the union of the output regions occupied by the primitives that have been added, deleted or changed since the last S• update. By limiting image regeneration to such an update region, both valuable computing time and video memory access time are minimized thus helping output animation flow more smoothly.

Another development in computer graphics has been the use of transparency.

For example, a semi-transparent primitive laid over an opaque primitive may yield a result in which the colours of the two primitives are mixed in proportion to the degree of transparency of the overlying primitive. This technique is commonly called "alphachannel compositing".

Expression trees have been used to represent the composition of images using transparency. Leaf nodes in the tree represent graphical primitives such as scanned 580809.doc -3images, geometric shapes and text, while internal nodes represent compositing operators that combine corresponding operands to synthesize a new image. Operators may, for example, mix transparent colours as described above, or may use the transparency of one operand to limit the visibility of the other, or perform other predefined operations. The root node of the tree represents the final output displayed.

One known prior art retained-mode computer graphics API system uses such an expression tree, hereinafter called a compositing tree, as the image representation. In accordance with this known API system, the application software constructs a compositing tree and then asks for the tree to be rendered to the output device via API calls. The application can make modifications to the tree, for example, in response to user input, and call the API to update the output. This known API system performs oooo display updates very efficiently. At every node in the compositing tree, the system maintains a partitioning of the image represented by the node, with a cached image and an identifier (referred to as a contents label) of the contents in each partition. By storing S 15 dependencies between content labels, the API system can determine which parts of the image need to be regenerated in response to changes in the tree since the system caches images at each node and, parts of the image composition that have not changed do not have to be rendered.

One disadvantage of the above-described API system is that caching images at every node consumes a considerable amount of memory. Even though some lossless image compression techniques can be used, such as partitions that have the same contents reusing the same image data, it would be better if no images were cached at all. The problem with not caching images is that each time the display needs to be updated in response to changes in the compositing tree, the image data that is presently cached needs to be regenerated, used and discarded. Such a process would ordinarily be too slow for a typical computer even by todays standards. However, in recent times there have been 580809.doc -4advances in computer graphics hardware technology that enable alpha-channel compositing of graphics primitives to be performed very quickly and efficiently. Such hardware is now becoming commonplace even in personal computers.

Hardware for graphics acceleration is typically stored on a separate printed circuit board from the CPU and has its own separate video memory. One disadvantage of such hardware is that there is only a limited amount of video memory. Any utilization of such hardware must take into account the limited nature of this resource.

Summary of the Invention It is an object of the present invention to substantially overcome, or at least ameliorate, one or more disadvantages of existing arrangements.

ooo.

According to one aspect of the present invention there is provided a method of creating a series of images, said images being formed by rendering and compositing a plurality of graphical objects, each of said objects having a predetermined boundary outline, said method comprising the steps of: determining a visible region for each graphical object, wherein said visible region is defined by a region outline substantially following at least one of said predetermined outlines or parts thereof, depending on those said objects which contribute 0* S"to said region, said contribution being determined on the basis of clipping and/or occlusion of said graphical object by any other graphical object, and/or on the basis of the compositing of said graphical object with any other graphical object; storing said visible regions; rendering a first image of said series by compositing said graphical objects; in response to at least one change to at least one of said objects; determining a further visible region for each graphical object, wherein said further visible region is defined by a region outline substantially 580809.doc following at least one of said predetermined outlines or parts thereof, depending on those said objects which contribute to said further visible regions; (ii) forming an update region as the union of said stored visible regions and said further visible regions for those objects affected by said at least one change; (iii) creating a further image of said series by rendering a portion of said further image, said portion being substantially equal to said update region; and repeating step for further changes to at least one of said objects.

According to another aspect of the present invention there is provided a method of creating a series of images, said images being formed by rendering and compositing a plurality of graphical objects to be composited according to a hierarchical data structure representing a compositing expression, each of said objects having a predetermined boundary outline, said hierarchical data structure including a plurality of nodes each representing an operand of said compositing expression or an operator for combining said 15 operands, wherein each of said graphical objects is represented by at least one of said S.i operands, said method comprising the steps of: determining a visible region for each node, wherein said visible region is defined by a region outline substantially following at least one of said predetermined outlines or parts thereof, depending on those said objects which contribute to said region, 20 said contribution being determined on the basis of the compositing of said graphical object with any other graphical object, storing said visible regions; rendering a first image of said series by compositing said graphical objects according to said hierarchical structure; in response to at least one change to at least one of said nodes; 580809.doc -6determining a further visible region for each node, wherein said further visible region is defined by a region outline substantially following at least one of said predetermined outlines or parts thereof, depending on those said objects which contribute to said further visible regions; (ii) forming an update region as the union of said stored visible regions and said further visible regions for those nodes affected by said at least one change; (iii) creating a further image of said series by rendering a portion of said further image, said portion being substantially equal to said update region; and repeating step for further changes to at least one of said nodes.

According to still another aspect of the present invention there is provided a method of creating a series of images, said images being formed by rendering and compositing a plurality of graphical objects to be composited according to a hierarchical data structure representing a compositing expression, each of said objects having a predetermined boundary outline, said hierarchical data structure including a plurality of oooo 15 nodes each representing an operand of said compositing expression or an operator for combining said operands, wherein each of said graphical objects is represented by at least •one of said operands, said method comprising the steps of: for each node: examining said node to determine those of said objects which 20 contribute to said node, said contribution being determined on the basis of occlusion of any graphical object associated with said node by any other graphical object associated with said node; (ii) determining a visible region for said node depending on said examination, wherein said visible region is defined by a region outline substantially following at least one of said predetermined outlines or parts thereof; 580809.doc -7rendering a first image of said series by compositing said graphical objects according to said hierarchical structure; determining memory required for rendering and combining said operands associated with each node; modifying at least one node representing an operator for combining said operands, by interchanging operands associated with said node and reversing said operator to render a visible region associated with said node, if memory required for rendering and combining operands associated with said node is minimised by said modification; in response to at least one update to at least one of said nodes; determining a further visible region for each node using any modified compositing expressions determined at step depending on those said objects which contribute to said further visible regions; (ii) forming an update region as the union of said visible regions and 15 said further visible regions for those nodes affected by said at least one update; (iii) creating a next image of said series by rendering an image space in which said graphical objects are defined (only) in an area substantially equal to said update region; repeating steps to for further changes to at least one of said nodes.

20 According to still another aspect of the present invention there is provided an apparatus for creating a series of images, said images being formed by rendering and

S

compositing a plurality of graphical objects, each of said objects having a predetermined boundary outline, said apparatus comprising: means for determining a visible region for each graphical object, wherein said ;vbl region is oefined by a region outline substantially following at lt one nf said predetermined outlines or parts thereof, depending on those said objects which contribute 580809.doc -8to said region, said contribution being determined on the basis of clipping and/or occlusion of said graphical object by any other graphical object, and/or on the basis of the compositing of said graphical object with any other graphical object; means for storing said visible regions; and means for rendering a first image of said series by compositing said graphical objects, wherein in response to at least one change to at least one of said objects, said means for determining a visible region: determines a further visible region for each graphical object, wherein said further visible region is defined by a region outline substantially following at least one of said predetermined outlines or parts thereof, depending on those said objects which contribute to said further visible regions, forms an update region as the union of said stored visible regions and said further visible regions for those objects affected by said at least one change; and 15 creates a further image of said series by rendering a portion of said

S.

S.further image, said portion being substantially equal to said update region; and According to still another aspect of the present invention there is provided an apparatus for creating a series of images, said images being formed by rendering and compositing a plurality of graphical objects to be composited according to S 20 a hierarchical data structure representing a compositing expression, each of said objects .i having a predetermined boundary outline, said hierarchical data structure including a ooo o plurality of nodes each representing an operand of said compositing expression or an .:operator for combining said operands, wherein each of said graphical objects is represented by at least one of said operands, said apparatus comprising: means for determining a visible region for each node, wherein said visible region is defined by a region outline substantially following at least one of said predetermined 580809.doc -9outlines or parts thereof, depending on those said objects which contribute to said region, said contribution being determined on the basis of the compositing of said graphical object with any other graphical object, means for storing said visible regions; means for rendering a first image of said series by compositing said graphical objects according to said hierarchical structure, wherein in response to at least one change to at least one of said nodes, said determining means; determines a further visible region for each node, wherein said further visible region is defined by a region outline substantially following at least one of said predetermined outlines or parts thereof, depending on those said objects which contribute to said further visible regions; forms an update region as the union of said stored visible regions and said further visible regions for those nodes affected by said at least one change; creates a further image of said series by rendering a portion of said S 15 further image, said portion being substantially equal to said update region; and According to still another aspect of the present invention there is provided an apparatus for creating a series of images, said images being formed by rendering and compositing a plurality of graphical objects to be composited according to a hierarchical data structure representing a compositing expression, each of said objects having a S 20 predetermined boundary outline, said hierarchical data structure including a plurality of nodes each representing an operand of said compositing expression or an operator for combining said operands, wherein each of said graphical objects is represented by at least one of said operands, said apparatus comprising: means for examining each said node to determine those of said objects which contribute to said node, said contribution being determined on the basis of occlusion of 580809.doc any graphical object associated with said node by any other graphical object associated with said node; means for determining a visible region for said node depending on said examination, wherein said visible region is defined by a region outline substantially following at least one of said predetermined outlines or parts thereof; means for rendering a first image of said series by compositing said graphical objects according to said hierarchical structure; means for determining memory required for rendering and combining said operands associated with each node; means for modifying at least one node representing an operator for combining said operands, by interchanging operands associated with said node and reversing said operator to render a visible region associated with said node, if memory required for rendering and combining operands associated with said node is minimised by said modification, wherein in response to at least one update to at least one of said nodes: 15 a further visible region is determined for each node using any modified compositing expressions, depending on those said objects which contribute to •said further visible regions; an update region is formed as the union of said visible regions and said further visible regions for those nodes affected by said at least one update; 20 a next image of said series is created by rendering an image space in S.•i which said graphical objects are defined (only) in an area substantially equal to said update region; According to still another aspect of the present invention there is provided a program for creating a series of images, said images being formed by rendering and compositing a plurality of graphical objects, each of said objects having a predetermined boundary outline, said program comprising code for performing the following steps 580809.doc -11determining a visible region for each graphical object, wherein said visible region is defined by a region outline substantially following at least one of said predetermined outlines or parts thereof, depending on those said objects which contribute to said region, said contribution being determined on the basis of clipping and/or occlusion of said graphical object by any other graphical object, and/or on the basis of the compositing of said graphical object with any other graphical object; storing said visible regions; rendering a first image of said series by compositing said graphical objects; in response to at least one change to at least one of said objects; determining a further visible region for each graphical object, wherein said further visible region is defined by a region outline substantially following at least one of said predetermined outlines or parts thereof, depending on those said objects which contribute to said further visible regions; (ii) forming an update region as the union of said stored visible regions and said further visible regions for those objects affected by said at least one change; 0 (iii) creating a further image of said series by rendering a portion of said further image, said portion being substantially equal to said update region; and repeating step for further changes to at least one of said objects.

20 According to still another aspect of the present invention there is provided a 500* .i program for creating a series of images, said images being formed by rendering and %0o0 compositing a plurality of graphical objects to be composited according to a hierarchical Sodata structure representing a compositing expression, each of said objects having a predetermined boundary outline, said hierarchical data structure including a plurality of nodes each representing an operand of said compositing expression or an operator for 580809.doc hacombining said operands, wherein each of said graphical objects is represented by at least one of said operands, said program comprising code for performing the following steps: 580809.doc -12determining a visible region for each node, wherein said visible region is defined by a region outline substantially following at least one of said predetermined outlines or parts there6f, depending on those said objects which contribute to said region, said contribution being determined on the basis of the compositing of said graphical object with any other graphical object, storing said visible regions; rendering a first image of said series by compositing said graphical objects according to said hierarchical structure; in response to at least one change to at least one of said nodes; determining a further visible region for each node, wherein said further visible region is defined by a region outline substantially following at least one of said predetermined outlines or parts thereof, depending on those said objects which contribute to said further visible regions; oo* (ii) forming an update region as the union of said stored visible regions and said further visible regions for those nodes affected by said at least one change; (iii) creating a further image of said series by rendering a portion of said 0 further image, said portion being substantially equal to said update region; and repeating step for further changes to at least one of said nodes.

Brief Description of the Drawings A number of preferred embodiments of the present invention will now be described with reference to the drawings, in which: Fig. 1 is a flow chart showing a method for displaying an image using retained-mode graphics APIs; Fig. 2 is a state transition diagram for nodes in a binary compositing tree; Fig 3 is a data structure for nodes in a binary compositing tree; Fig2. 4 is a flow chart showing a method of rendering a binary-compositing tree; 580809.doc 13- Fig. 5 shows an example compositing tree and the image represented by the tree; Fig. 6 shows have and opaque regions for the nodes in the tree of Fig. Fig. 7 shows the display regions of the nodes in the tree of Fig. Fig. 8(a) shows an image comprising four objects; Fig. 8(b) shows a tree that can be used to represent the image of Fig. 8(a); Fig. 8(c) shows a tree that can be used to represent the image of Fig. 8(a); Fig. 9 is a flow chart showing a method for performing an incremental update of an image in response to changes in a binary tree; Fig. 10 shows a modified version of the tree of Fig. 5 and the correspondingly modified image; Fig. 11 shows the Display Regions of the nodes in the tree of Fig. Fig. 12 shows the update region for updating the image shown in Fig. 5 to the image shown in Fig. Fig. 13 shows the compositing tree of Fig. 10 optimized to only render the 15 update region of Fig. 12; Fig. 14 shows a state transition diagram for nodes in an n-ary tree that represent Sgraphical primitives; Fig. 15 shows a state transition diagram for nodes in an n-ary tree that represent compositing operators without the operand independence property; Fig. 16 shows a state transition diagram for operand slots of nodes that represent compositing operators without the operand independence property; Fig. 17 shows a state transition diagram for nodes in an n-ary tree that represent compositing operators with the operand independence property; Fig. 18 shows a state transition diagram for operand slots of nodes that represent compositing operators with the operand independence property; Fig. 19 shows a data structure for nodes in an n-ary compositing tree; 580809.doc -14- Fig. 20 shows a data structure for binary nodes derived from n-ary nodes; Fig. 21 shows a flow chart of a method for rendering an n-ary tree; Fig. 22 shows an n-ary tree and an equivalent binary tree; Fig. 23 illustrates the application of the associative law; Fig. 24 shows a flow chart of a method for performing incremental update of an image in response to changes in an n-ary tree; Fig. 25 shows a flow chart of a method for rendering a tree subject to memory constraints; and Fig. 26 shows a schematic block diagram of a general purpose computer upon which the embodiments of the present invention can be practiced.

Detailed Description including Best Mode Where reference is made in any one or more of the accompanying drawings to see steps and/or features,'which have the same reference numerals, those steps and/or features have for the purposes of this description the same function(s) or operation(s), unless the 15 contrary intention appears.

Fig. 1 is a flow chart showing a method for displaying an image. The method preferably utilises a retained-mode graphics API. As an example, assume that an •o o

S

application program is using a retained-mode computer graphics API to control a graphical user interface. The method of the flow chart of Fig. 1 is preferably

C

implemented as one or more modules of an application program being resident on the hard disk drive 2610 of a computer system 2600 and being read and controlled in its execution by the processor 2605 of the computer system 2600, as described below with reference to Fig. 26. The method begins at step 110, where the processor 2605 constructs a data structure representing the initial appearance of the display. The data structure is preferably in the form ofa compositing tree. At the next step 120, the image represented by the data structure is rendered by the API. In accordance with the example above, 580809.doc rendering a compositing tree involves rendering the primitives that are represented by the leaf nodes of the tree and compositing them together according to the operations represented by the internal nodes of the tree. Also at step 120, information to facilitate efficient incremental update of the display is recorded, as will be described in more detail below. At the next step 170, the processor 2605 determines whether to make changes to the tree by following branch 150, or to create a fresh data structure by following branch 160 as the needs of the user interface demand. At the next step 130 changes are made to the data structure. For example, nodes are added, removed or modified. At the next step 140, the API examines the data structure, bounds the portion or portions of the display that need to be rendered to conform to the new form of the data structure and renders said portion or portions. Preferably, the API records any information needed to facilitate further incremental updates. Following step 140, the processor 2605 can continue to make S"changes to the tree by returning to step 170 and following branch 150, or the processor 2605 can create a fresh data structure by following branch 160 as the needs of the user interface demand.

2.0 A RETAINED-MODE SYSTEM EMPLOYING A BINARY TREE .oo.o: The steps 110 through 170 of the above method will now be explained in further S. •detail. For ease of explanation, the steps shall firstly be described in terms of a binary 0oooo compositing tree, and later extended to cover n-ary trees. In a binary tree, internal nodes have two descendants, customarily referred to as left and right operands. In an n-ary tree, internal nodes can have any number of descendants. Initially the situation where hardware memory limits are not a concern will be considered below. In practice, however, memory limitations exist, especially when using hardware acceleration, and methods to deal with said limitations are described later.

2.1 CONSTRUCTION OF A BINARY COMPOSITING TREE 580809.doc -16- At step 110, in accordance with the example, the compositing tree constructed comprises leaf nodes, representing graphical primitives, and internal nodes, representing operations that composite primitives together to produce more complex graphics. A state transition diagram for binary nodes is shown in Fig. 2, and a data structure for binary nodes in shown in Fig. 3.

When a compositing tree is first constructed each node is in state new/modified 210. This state is recorded in the State field 310 of the node data structure 300. The Display Region 320 is initialized to an empty region, the Dirty Counter 360 to zero, and the Last Count as Root 370 to zero. The purpose of each of these fields shall be described 10 in more detail below.

:2.2 INITIAL RENDERING OF A BINARY COMPOSITING TREE V At step 120, the compositing tree is rendered and the resulting output is typically displayed to the user. There may be other reasons for rendering the tree, for example: creating an image that will be saved to file on disk, or that will be saved in memory and subjected to further processing before being displayed to the user. The method described .i herein works in all such cases. The state of all nodes is moved to updated 220, which is reflected in field State 310. In addition, for each node, a representation of the region of output that is directly affected by the node is saved in field Display Region 320. This ooooo representation is not necessarily the same as the entirety of the graphics represented by the node, as for example, the representation can be subjected to clipping or occlusion by other nodes in the compositing tree.

Fig. 4 is a flow chart showing the method of rendering a binary tree step 120) in more detail and is subdivided into substeps. Each substep involves a depth-first traversal of the binary tree being processed. The method of the flow chart of Fig. 4 is preferably implemented an application program being resident on the hard disk drive 2610 and being read and controlled in its execution by the processor 2605.

580809.doc -17- 2.2.1 DETERMINE HAVE AND OPAQUE REGIONS In the first traversal of the tree, at step 410, a Have Region 340 and an Opaque Region 350 for each node are determined by the processor 2605. The Have Region 340 is a representation of the geometric area within which the node can supply image data. The Have Region 340 is preferably approximated, for example, to a bounding box, or to an irregular region bounded by horizontal and vertical lines, or to a quadtree. The exact representation is not critical, so long as set operations can be efficiently performed. The Opaque Region 350 is a representation of the area within which the node is guaranteed to supply fully opaque data. Again, the Opaque Region 350 may be approximate, so long as 10 no portions that are not opaque are included. The Opaque Region 350 can be empty, for *•go example, for semi-transparent primitives and for detailed primitives such as text for which .o •it is too difficult or costly to determine an opaque region.

•The method o determining Have and Opaque Regions will now be described.

For primitives the Have and Opaque Regions are determined from the specification of the primitive itself. Consider, for example, the compositing tree 510, which represents an image 500 as seen in Fig 5. Primitives 512, 516, 517 and 515 are composited using o:oo• operators 511, 513 and 514. Primitive 512 is an opaque image, primitive 516 is a semitransparent image, primitive 517 is an opaque triangle and primitive 515 is an opaque background covering the entire output area. The have and opaque regions of these primitives can be seen in Fig. 6. Note that regions in the example have been approximated to regions bounded by horizontal and vertical lines.

Determination of have and opaque regions of internal nodes depends upon the nature of the operation represented by each node. For alpha-channel compositing operators, Table 1 lists the resultant have and opaque regions of the composition of operands A and B.

580809.doc 18 Operator IHave Region Opaque Region over A.Have u B.Have [A.Opaque u B.Opaque in A.Have ni B.Have A.Opaque r) B.Opaque out A.Have A.Opaque B.Opaque atop B.Have B.Opaque xor A.Have u B.Hav'e (A.Opaque B.Have) u (B.Opaque A.Have) plus A.Have u B.Have A.Opaque u B.Opaque and A.Have B.Have A.Opaque n B.Opaque upon B.Have B.Opaque Table 1 Additional operators "and" and "upon" are defined by: A and B (A in B) plus (B in A) A upon B (A in B) plus B.

It is also useful to define reverse operators for non-commutative compositing operators, as shown in Table 2.

Operator Definition rover A rover B B over A rin Afn B =B in A rout A rout B =Bout A ratop A ratop B Batop A rupon A rupon B Bupon A 580809.doc -19- Table 2 During the first pass of the tree a copy of the links to the parent and left and right operands of the node is made in the Working Tree Parent and Left and Right field 380 in each internal node. The subsequent steps will often modify the tree, removing and rearranging nodes. This is best done using the working tree references 380 so as to leave the original tree intact from the point of view of the user.

2.2.2 REGION MINIMIZATION AND OCCLUSION At the next step 420, the tree is again traversed by the processor 2605 to S 10 determine the Display Region 320 for each node. A display region for a node is the geometric area of the output that is affected by the graphics represented by the node. The display region for a node is in many cases smaller than the have region for the node since the display region excludes the areas hidden by clipping (such as with "in" or "out" "operations), occluded by overlying opaque objects, or outside the bounds of the required output.

The have and opaque regions of the root node are intersected by the region representing the required output area. The process is then continued down the tree from the root node, intersecting both operands' have and opaque regions with the new have region of the parent the root node). For "over" nodes, the opaque region for the operands of the over node is subtracted from the have region for the right operand of the node. The Old Display Region 330 for every node is set to a copy of the Display Region 320 for the node, and the Display Region 320 is set to the resultant have region.

At any point, if the operand of a node has its have region reduced to an empty region, that node and the affected operand are replaced in the working tree by the other operand of the node. The removed node and subtree will no longer participate in further rendering steps 430 and 440. The Display Region 320 of the removed node is set to the 580809.doc have region of the node and the display regions of all nodes in the removed subtree are set to empty regions.

Returning to the example compositing tree 510, with have and opaque regions as illustrated in Fig. 6, the display regions 711 to 717 that result from the above process are shown in Fig. 7. The have region 611 a for node 511 is the same as the output region, so the have region and opaque region for node 511 remain unchanged. The display region 711 for node 511 is set to the have region 61la for the node 511. The have region 612a for node 512 and opaque region 612b are intersected by have region 611a, again unchanged by the operation. The display region 712 for node 512 is set to the have region 612a. The have and opaque regions for node 513 likewise are not affected by the parent of node 511 have region 61 la, but since the parent 511 is an "over", the opaque region 612b is subtracted from the have region of node 513. The display region 712 of node 512 is set to its new have region, as shown in Fig. 7. As can be seen in Fig. 7 the new have region excludes an occluded portion 721 that corresponds to the display region 712 of node 512.

"i At this point, for example, if a change were to be made to node 513, then the change would affect the portion of the output represented by the display region 713 of node 513, which is not the entirety of the output. To update the display only the contents ooeoe of region 713 need to be regenerated, saving the time to regenerate the occluded area represented by region 712.

Continuing with the example, the have region 614a of node 514 and opaque region 614b are intersected by the new have region (which is the same as region 713) for node 513, yielding new have and opaque regions. The resultant region 714 of node 514 is set to the new have region, and, as shown in Fig. 7, excludes the portion 719 of the node 514 graphics that are obscured by the opaque image at node 512. The opaque region of node 514 remains empty.

580809.doc -21 Continuing in this way, nodes 515, 516 and 517 end up with display regions 715, 716 and 717 respectively as shown in Fig. 7.

2.2.3 MEMORY OPTIMIZATION The compositing tree produced by step 420 is further processed by the processor 2605 at step 430 with the aim of minimizing the peak video memory usage during the rendering phase 440. Given a tree as input to step 430, depending on what operators are used in the internal nodes of the tree, it may be possible to rearrange the tree without changing the output that the tree represents. For example, the expressions: A over (B over C); 10 (A over B) over C; and A over (C rover B), among other possibilities all represent the same output (apart from rounding errors), achieved by a different sequence of operations. Therefore, different arrangements of a tree would not necessarily consume equal amounts of memory resources during rendering step 440.

Consider the example of Figs. and The same output image 800 as seen in Fig. can be produced by either tree 810 or tree 820 as seen in Figs 8(b) and respectively. However, the alternative trees require different amounts of memory to store intermediate results. With tree 810, primitive 801 can be rendered into a frame buffer. Then an intermediate buffer is needed for the "plus" operation, in which primitive 802 is first rendered and then composited with primitive 803. The intermediate buffer is then composited into the frame buffer. Finally, primitive 804 is composited into the frame buffer.

In contrast, with tree 820, no intermediate buffer is necessary to render the image 800. Primitive 802 can be rendered into the frame buffer, then primitives 803, 801 and 804 composited in turn.

580809.doc -22- If memory usage is a concern, a method for rearranging a given tree is required so as to minimize the peak memory usage for the tree. If rendering and compositing are to be hardware accelerated using commodity graphics boards, video memory on these boards is a critical resource whose usage is important to minimize. Even if rendering and compositing were performed in software using host memory, it is often advantageous to minimize memory usage so as to reduce the risk of virtual memory paging.

Being able to find the optimal tree is dependent on having a means of determining the memory usage of the various alternatives. Some primitives can need to be rendered into intermediate buffers and then composited, while other primitives can be rendered and composited in one step without the need for an intermediate buffer. This may depend on the compositing operator, the type of primitive and the type of hardware available.

When the right-hand operand of a compositing operator is not a primitive, it requires a buffer with size equal to the bounding box of the resultant region.

Buffers exist while a primitive or right-hand subtree is being rendered, then cease to exist, so that the buffer memory can be freely reused in further rendering.

ooooo Therefore the maximum memory use of alternative trees is determined, and the alternative with lowest maximum memory use can be chosen.

*oooo Assuming an initial frame buffer and a tree to render into the frame buffer. How much additional memory is required to cope with peak demand? If the root node is a primitive that can be rendered without any additional memory it is a simple primitive), then the answer is zero. If the root node is a primitive that requires an intermediate buffer it is a complex primitive), then the answer is the size of the intermediate buffer. If the root node is a binary node, then the left operand is firstly rendered, during which time some memory can be freed and used, and secondly the right operand is rendered, during which time some memory can be freed and used. The 580809.doc -23memory use for the root node is the maximum of the left and right node usages. If the right node was itself a binary node, then part of the memory usage is a buffer bounding the result region.

Mathematically, memory usage m(A) of a node A can be defined as follows: m(A) m load(A), if A is a primitive max(m(A.left), r_A.op(A.right)), if A in binary where m_load(A) is the memory required to render A into a buffer without compositing.

A.left and A.right are the operands of A, and r_op(A) is the memory required to render and composite A as the right operand of operator Function rop is defined as follows: r op(A) m_op(A), if A is a primitive 0 r bounds(A) if A is binary where m_op(A) is the memory required to render and composite A as the right operand of operator "op" and bounds(A) is the size of a temporary buffer required for rendering A before composting.

In practice, it would be necessary to calculate the memory usage of each leaf 15 node, then each binary node up the tree. This information can be used in choosing a between alternatives.

Some compositing operators "over", have the associativity property, for all operands A, B and C, A op (B op C) (A op B) op These operators can be termed "associative". The associativity property can be taken advantage of wherever possible to skew the compositing tree so as to minimize the number of occasions in which intermediate buffers need to be allocated. When this is performed, the region of the intermediate operator node must be recalculated to allow for the fact that the operands have changed. The have region of the root node of the affected subtree does not need to be chn.ge r rino ti he hl- regionn st* ll represents th-o scom im- ae.

s±±UISU flt. U V11 II.~ A 5 LV J11Sll tLJL t.lL3 Slf' fLUI llltJ.5%.

580809.doc -24- With every binary node in a tree, there is a choice of either leaving the node alone or swapping the operands and reversing the operator. The costs of both alternatives can be determined and the cheaper one chosen. This can be done once for each node as a post-order traversal of the tree is performed. The have regions of the node are not affected by swapping the operands and reversing the operator of the node.

2.2.4 RENDERING THE TREE Returning now to Fig. 4, having completed step 430, the processor 2605 proceeds to step 440. Rendering the optimized tree involves traversing the tree, first rendering the left operand of a node, then compositing the right operand using the 10 operator of the node. If the left operand is a subtree, the process is applied recursively. If •ooo the left operand is a leaf node, the frame buffer is cleared and the primitive represented by o V the leaf node is rendered into the fi-ame buffer in the bounds of the optimized have region for the node. If the right operand is a leaf node, the right operand is rendered and composited into the frame buffer within the bounds of the optimized have region for the right operand. If the right operand is a subtree, an intermediate buffer is created large enough to hold the optimized have region for the right operand, and the process is applied oooo• recursively to render the subtree into the intermediate buffer and finally the intermediate buffer is composited into the frame buffer using the operator of the node.

Consider, for example, the tree 510 in Fig. 7. The display regions 711 to 717 shown are in this instance identical to the optimized have regions. Initially, the traversal comes to root node 511. As root node 511 is a binary node, the left operand 512 is rendered. Node 512 is a leaf node, so the frame buffer is cleared and the primitive represented by node 512 is rendered into region 712 of the frame buffer. The traversal then returns to node 512 and proceeds to render the right operand 513. Since operand 513 is a binary node, it cannot be composited directly into the frame buffer, so an intermediate buffer large enough to contain region 713 is created. Continuing in this way, the subtree 580809.doc whose root is node 513 is rendered into the intermediate buffer, and returning to node 511, the frame buffer is composited together with the intermediate buffer using the "over" operation of node 5 11 storing the result in the frame buffer. As the intermediate buffer is no longer required, it can be freed at this point.

Note that it is possible when rendering a subtree into a first intermediate buffer to encounter the need of a further intermediate buffer, and so on recursively. In such a case the above method is followed, treating the first intermediate buffer as the frame buffer and the further intermediate buffer as the intermediate buffer.

2.2.5 SET NODE STATES TO UPDATED 10 The final step in Fig. 4, step 450, involves the processor 2605 traversing the .o original tree 510 constructed at step 110, prior to the optimizations of steps 420 and 430.

In other words, the original tree linkage is traversed and the Working Tree Parent, Left and Right pointers 380 are ignored. At step 450, the State field 310 of each node is set from new/modified 210 to updated 220.

2.3 MODIFIYING THE BINARY TREE After step 450 has been performed, step 120 of Fig. 1 has been completed, a compositing tree has been created, rendered, and the following information has been set:

P

State 310, set to updated; Display Region 320; Dirty Counter 360; and Last Count as root 370.

Before moving on to step 130, it is necessary to explain the role of the Dirty Counter 360 and Last Count as Root 370 fields. The Last Count as Root field 370 is used to determine whether the tree to be rendered is a new tree or a possible modified version 580809.doc -26of the previous tree. A global counter is initialized to 1 when the system is started. Each time the display is updated, the counter is incremented and the new value is assigned to the Last Count as Root field 370 of the root of the tree used to update the display. The next time the screen is to be updated, the current value of the global counter is compared with the Last Count Root field 370 of the given root node. If they are equal, then it is known that branch 150 has been taken, so the tree can be examined for differences and the display updated incrementally. If the current value and the Last Count as Root differ, then it is known that branch 160 has been followed, so the tree can be rendered to the display afresh in step 120. Since newly created nodes have Last Count as Root field 370 initialized to 0, a newly created root will never match the global counter so the new node will always be treated correctly.

purpose of the Dirty Counter 360 in each node is to help narrow down the search during step 140 of the paris of the tree that have been modified during step 130.

The current value of the global counter is assigned to the Dirty Counter 360 of every node 15 that is modified itself or that is the parent in the tree of a node that is modified. During update step 140, nodes in any subtree whose associated root Dirty Counter 360 differs from the current value of the global counter are known to be unchanged. Conversely, in any subtree whose associated root Dirty Counter 360 equals the current value of the global counter there will be at least one modified node. Once the update occurs, the global counter is incremented thus efficiently clearing all information about modifications during the previous update cycle, leaving the compositing tree with a clean slate, ready for the next cycle.

Newly created nodes have their Dirty Counter field 360 set to 0, so they can never be considered modified. However, a new node is either used as a new root, thus forcing a full update (rather than an incremental one), or it has to be linked into the tree, 580809.doc -27thus modifying the new parent node. The entirety of the subtree of the new parent is then treated as the modified node.

A node in the tree can be moiified by, for example: unlinking an existing operand; linking a new operand; changing an attribute such as shape, colour or degree of transparency; and/or applying a geometric transformation.

The above list is not exhaustive and other modifications are possible without falling outside the scope of the invention.

When a node is modified during step 130) the state of the node moves from updated 220 to new/modified 210. The new state is recorded in field State 310 of the data structure 300 associated with the node. In addition, all nodes in the sequence of nodes from the modified node up to the root of the tree need to have their Dirty Counter 360 set to the current value of the global counter. To avoid repeating the same work when o:oo0 several successive changes are made to the same branch of a tree, setting the sequence of dirty counter fields can stop when a node has been reached that has already had its dirty counter set to the same value.

2.4 INCREMENTAL UPDATE Continuing on to step 140 of Fig. 1, when modifications to the compositing tree have been made, a request is made to update the displayed output so that the output agrees with the new version of the tree. Preferably, information stored in the nodes in previous steps is used in order to determine what portion or set of portions of the display need to be re-rendered. Step 140 also saves the possibly different display region of each node in readiness for future incremental updates. The global counter is incremented and a new value is copied to the Last Count as Root field 370 of the root node of the compositing 580809.doc -28 tree. Step 140 is subdivided into substeps as shown in Fig. 9. Each substep of Fig. 9 involves a depth-first traversal of the binary tree.

In the first traversal, at step 910, the Have Region 340 and Opaque Region 350 fields for each node are determined by the processor 2605 and a copy of the parent, left and right links is stored in field 370. Step 910 is the same as step 410 of Fig. 4, and can be implemented as a common subroutine being executed by the processor 2605.

In the second traversal, at step 920, an optimized region is generated for each node and the resulting region is stored in both the Have Region 340 and Display Region 320 fields of each node. Before setting a Display Region 320, the Display Region 320 as set by the previous update is copied to the Old Display Region 330, so that both old and S"new display regions are available. Regions becoming empty are handled as described above for step 420. Step 920 differs from step 420 only in that the previous value of display region 320 is saved in Old Display Region 330. If the previous value of the display region 320 was saved at step 420, it would do no harm, so steps 420 and 920 can be implemented as a common subroutine.

The third traversal, at step 930, determines the update region. The update region represents the geometric portion or set of portions of a display that need to be rendered in ooooo order to make the diliy r_,.icii with the changes in the compositing tree. Starting off with an empty update region, the tree representing the image is traversed and the following actions are performed based on the type and state of each node encountered: Compositing operator in updated state 220: If the left operand is not marked dirty the Dirty Counter 360 of the left operand differs from the current value of the global counter), then the left branch of the tree is not traversed down. Likewise, if the right operand is not marked dirty, then the right branch of the tree is not traversed down.

Compositing operator in new/modified state 210: Both the Old Display Region 330 and Display Region 340 for the node are incorporated into the update region. That is, 580809.doc -29a set union of the Old Display Region 330, the Display Region 340 and the current update region is performed and the result stored as the new value of the update region. The left or right branches are not traversed.

Leaf node in update state 220: Nothing is done.

Leaf node in new/modified state 210: The Old Display Region 330 of the node and Display Region 340 of the node are incorporated into the update region.

At the end of the third traversal, the update region has collected a representation of all the areas of the display that need to be re-rendered. Significantly this method works on a compositing tree rather than, say, a display list, and the method avoids rendering areas where changes are not visible due to occlusion or clipping. Consider, for example, the tree 510 shown in Fig. 5, with display regions assigned to each node during step 420 shown in Fig. 7. Suppose the tree 510 is modified by moving the position of the triangle represented by node 517 slightly to the left, resulting in the image 1000 represented by the tree 1010 as shown in Fig. 10. This is the only difference between tree 1010 and tree 510. In the process of changing node 517, the state of that node was moved from updated 220 to new/modified 210. All nodes from there to the root, viz., nodes 517, 514, 513 and 511, are marked as dirty by assigning the current value of the global counter to the Dirty Counter field 350 for each node.

To update the display, steps 910 through 970 as shown in Fig. 9 can be performed. Step 920 will result in display regions assigned to each node in the Display Region field 320 as shown in Fig. 11. The display regions previously assigned in step 420 and shown in Fig. 7 are saved in the Old Display Region 330 of each node.

Step 930 results in an optimal update region by traversing the tree 1010 in Fig. 10. Firstly, the root node 511 is examined and is a binary node in updated state. The Dirty Counter 360 of the left child 512 is checked and it is found that the node 511 is not marked dirty. Hence the left branch of the tree 1010 does not need to be checked. The 580809.doc right child 513 however, is marked dirty so the traversal down the right branch is continued. Examining node 513, a binary node in updated state is found. The left child 514 is marked dirty so the traversal is cdntinued. Note that the right child 515 is not marked dirty so that the traversal will not extend into that node 515. Continuing in this way, node 517 is arrived at and is in new/modified state. Both the Old Display Region 330, shown at 717 in Fig. 7, and the New Display Region 320, shown at 1117 in Fig. 11, are included in the update region. The combined, and in this case, final update region 1200 is shown in Fig 12. Note that the display region 712 of node 512, which occludes part of modified node 527, is excluded from the update region 1200.

Continuing the process of Fig. 9, the fourth traversal, at step 940, repeats the region minimization process of step 920, but instead of using the entire output area of the display as the region with which is intersected with the have region of the root node, the update region that was calculated in the previous step 930 is used. It is important to note that since the update region is likely to be significantly smaller than the entire output area, 15 large portions of the tree 1010 are likely to be pruned due to the associated have regions being reduced to empty regions. Another difference between steps 940 and 920 is that in step 940 the Display Area 320 or Old Display Area 330 fields of the compositing tree nodes are not set. These must only be set in step 920 (and of course in step 420).

The next traversal, at step 950 is identical to step 430 and the processor 2605 optimises the tree to minimise video memory usage.

Step 960 is identical to step 440 and renders the optimized tree to the output device.

Step 970 is identical to step 450 and is performed on the original preoptimized) tree and changes the state (stored in field State 310) of all nodes to be updated 220.

580809.doc -31 Returning to the example of Fig. 11, the update region 1200 can be used as the output region for a region minimization process on tree 1010. The update region 1200 is intersected with the have region of root node 511. The resulting have region is intersected with the have regions of the operands, nodes 512 and 513, and so on down the tree. Note that since the have region for node 512 prior to intersection (identical to region 1112 in Fig. 11) and the update region 1200 do not intersect, the have region for node 512 ends up empty, so node 512 and parent node 511 are replaced by the other operand 513. This simplification of the tree will result in more efficient rendering. The resulting tree 1300 with the final have regions is shown in Fig. 13.

The above description describes incremental update being performed on a binary tree. While this may be useful of itself for some applications, a method for incremental °update of output based on n-ary trees will now be described. N-ary trees offer much more flexibility to a user.

A RETAINED-MODE SYSTEM EMPLOYING AN N-ARY TREE 15 A retained mode graphics API using an n-ary compositing tree will typically follow, as in the case of a binary tree already described, the steps shown in Fig. 1. The method constructs a data structure at step 110. This data structure represents the image to be rendered and displayed in the next step 120 and takes the more general form of an nary tree. The method can then continue to modify the n-ary tree at step 130 and invoke the API to correspondingly update the display at step 140, or may at any time start again at step 110 where the processor 2605 creates a fresh data structure.

3.1 DEFINITION AND PROPERTIES OF N-ARY TREES The preferred form and semantics of the n-ary tree will now be described. Note that some degree of variation is possible without departing from the scope and spirit of the invention. As with a binary tree, an n-ary tree contains nodes that represent graphical primitives, such as images, geometric shapes and text, which are combined using nodes 580809.doc -32that represent compositing operators. The difference is that a node that represents a compositing operator may have zero, one, two or more operands. The preferred semantics chosen for an n-ary node are as follows. A node utilizing an operator f(A, B) with n operands A 2 An represents: a fully transparent image if n 0; the operand A, unchanged ifn 1; f(Ai, A 2 ifn 2; f(f(Ai, A 2

A

3 ifn 3; and

A

2

A

3 An) ifn 3.

Any operand can be independently enabled or disabled. A disabled operand is *ignored when determining the image represented by an operator node. For example, an nary node with five operands, two of which are disabled, is equivalent to a node with the same operator but only the three enabled operands.

Operands can be unlinked from and linked into n-ary nodes. A slot in an n-ary 15 node with no actual operand linked is treated as if that slot was disabled. Empty slots can be inserted into or deleted from the set of operands for an n-ary node.

S"Given an n-ary node that represents the composition of a group of operands, most compositing operators have the property that if operands are added to or removed from the composition, the portion of the display that needs to be re-rendered in order to bring the output up to date is the union of two regions, the region of the output which the new composition occupies and the region of the output which the previous composition occupies. The reason for including the new region is straightforward: it is the region occupied by the graphics that must be rendered and displayed. The reason for including the old region is that the new graphics can occupy a different area that does not fully include the area rendered previously. In that case, the old area must be re-rendered in 580809.doc -33order to erase the graphics that used to be displayed and now should no longer be displayed.

A few compositing operators have an advantageous property referred to as "operand independence". If operands are added to or removed from a composition, only the region of the output occupied by the added or removed operands themselves needs to be re-rendered. With such operators, it is not necessary to re-render the entire old and new regions for the composition. Amongst the operators given in Table 1, operators "over", "plus" and "rover" have this property.

Adding an operand to a composition can include either linking an operand into an empty, enabled slot in an n-ary node, or enabling a disabled slot in which an operand is already linked. Similarly, removing an operand from a composition means either unlinking or disabling an enabled operand.

3.2 STATE TRANSITIONS *o Consider now the state transition diagrams in Figs. 14 to 18. Nodes that 15 represent graphical primitives follow the state transition diagram shown in Fig. 14 and will be explained in detail in the following paragraphs. N-ary nodes that represent compositing operators without the operand independence property follow state transition diagrams in Figs. 15 and 16, which again will be explained in detail in the following paragraphs. Nodes that represent operators with the operand independence property a follow the diagrams in Figs. 17 and 18.

Before examining Figs. 14 to 18, it is necessary to understand the various operations that can be applied to n-ary nodes and to each of the variable number of operand slots each n-ary node has. A created operand slot can undergo the following operations: enabled; disabled; 580809.doc -34an operand can be linked in to an empty slot; an existing operand can be unlinked; or an enabled slot with a linked operand can be involved in a display update (this is only relevant to operators with the operand independence property).

A created node can undergo the following operations: new empty operand slots can be inserted and are marked "insert" in the state diagrams; empty slots can be removed and marked "remove" in the state diagrams; the node can be modified in some way geometrically transformed); an operand can be added to or removed from the composition (as defined above) and are marked "operand" in the state diagrams; the node can be involved in a display update; to pending damage can need to be cached and is marked "pending" in the diagrams.

Pending damage will be explained in more detail later in this document and is only 99° relevant to operators with the operand independence property.

999999 S• 15 A data structure 1900 for nodes in an n-ary tree is shown in Fig. 19. Field State 1910 stores the current state of a node.

to An overview of the state transition diagrams in Figs. 14 to 18 will now be too.*: 9..

explained. A more detailed explanation will follow.

*999 Turning to Fig. 14, each node representing a primitive is initially in state new/modified 1401. When a node is created, action 1411 takes place, which initializes the display region for the node to an empty region. The display region is stored in field Display Region 1915 and is abbreviated to R in the state transition diagrams of Figs. 14 to 18. \\hen a node is involved in a screen update, action 1412 takes place and the state moves to the updated state 1402. If the graphical primitive represented by the node is modified in some way, action 1413 takes place and the state of the node returns to new/modified 1401. Finally, if an updated node is involved in a further display update 580809.doc without any intervening modifications being made, action 1414 takes place and the state is left unchanged.

Turning to Fig. 15, each node representing a compositing operator for which operand independence does not hold is initially in state new/modified 1501. The display region is initialized to an empty region by action 1511. When in state new/modified 1501, any changes made to the node do not affect the state of the node and do not result in any actions. When the node is involved in a display update, action 1512 takes place and the state moves to the updated state 1502. When in state updated 1502, the node responds differently to each operation that can happen. If empty operand slots are inserted or removed, no change in state occurs and no action takes place since empty operand slots .00 are ignored in the composition and hence do not affect the image to be displayed in any a* way. If the updated node is modified in some way, for example, by a geometric 66 transformation, or if an operand is added to or removed from the composition, action 0*6* 6Se 1513 takes place and the state of the node returns to new/modified 1501. Finally, if an 15 updated node is involved in a further display update, action 1514 takes place and the state remains unchanged.

Fig. 16 shows a state transition diagram for an operand slot of a node for an *I 6 operator without the operand independence property. A newly inserted operand slot is in 6*66 state 1601, in which the operand is both disabled and null absent). A state 1602 is *I also a suitable initial state. If an operand is linked in, the slot moves to state 1603. Then if the operand slot is enabled, the slot moves to state 1604. Similarly, if a slot in state 1601 is enabled, the slot moves to state 1602. If an operand is then linked in, the slot moves to state 1604. Note that all these transitions are reversible. Transitions between states 1604 and 1602, and between states 1604 and 1603 invoke action 1611. Action 1611 indicates to the node that an operand has been added to or removed from the composition, which can affect the state of the node and invoke further action.

580809.doc -36- Turning to Fig. 17, each node representing a compositing operator for which operand independence holds is initially in state new/modified 1701. The display region is initialized to an empty region by action 1711. When in state new/modified 1701, any changes made to the node do not affect the state of the node and do not result in any actions. When the node is involved in a display update, action 1712 takes place and the state moves to updated 1702. When in the updated state 1702, the node responds differently to each operation that can happen. If empty operand slots are inserted or removed, no change in state occurs and no action takes place, for the reasons discussed above. If an operand is added to or removed from the composition to or from state 1804 in Fig. 18, action 1713 takes place. However, if an operand is removed from the ~composition such that pending damage must be saved from state 1505 in Fig. 18), e action 1714 takes place. If the updated node is modified in any other way, for example, by o a geometric transformation, action 1715 takes place and the state of the node returns to new/modified 1701. Finally, if an updated node is involved in a further display update, action 1714 takes place and the state remains unchanged.

Fig. 18 shows a state transition diagram for an operand slot of a node for an operator with the operand independence property. A newly inserted operand slot is in state 1801, in which the operand is both disabled and null absent). Note again that state 1802 is also a suitable initial state. If an operand is linked in, the slot moves to state 1803. Then if the slot is enabled, the slot moves to state 1804. Similarly, if a slot in state 1801 is enabled, the slot moves to state 1802. If an operand is then linked in, the slot moves to state 1804. Note that all these transitions are reversible. Transitions between states 1804 and 1802, and between states 1804 and 1803 invoke action 1811 in an analogous manner to nodes for compositing operator without operand independence, as discussed for Fig. 16.

580809.doc -37- Slots in nodes for operators with operand independence also undergo state transitions when the node is involved in a screen update. Continuing with Fig. 18, when an operand slot is in one of states 1801, 1802 or 1803, the slot is not involved in the composition, so a display update has no effect on the state. If the node is in state new 1804, a display update invokes action 1812 and the state moves to the updated state 1805.

Action 1812 is part of action 1716 shown in Fig. 17 if the node is in state updated 1702. A further display update when the operand slot is already in state updated 1805 leaves the state unchanged and invokes no action. Finally, if an operand whose slot is in state updated 1805 is removed from the composition, whether by disabling or unlinking, action 1813 takes place. Action 1813 indicates to the node to save pending damage for use .during the next update, and is part of action 1713 if the node is in state updated 1702.

For n-ary nodes with operand independence, a set of boolean flags are stored in 000o field Link Flags 1945 in n-ary node data structure 1900 shown in Fig. 19. Each flag represents whether the corresponding operand slot is in updated state 1805 or in one of 15 the other states shown in Fig. 18.

3.3 CONSTRUCTION OF AN N-ARY COMPOSITING TREE i The steps of Fig. 1 with respect to n-ary trees as the image representation data structure will now be explained in more detail. At step 110, a compositing tree is o• S•constructed. All nodes are initially in state new/modified states 1401, 1501, 1701), represented by a value in field State 1910 of n-ary node data structure 1900. The display region of each new node, stored in field Display Region 1915, is initialized to an empty region. The Dirty Counter 1925 and Last Count as Root 1930 fields are initialized to zero, as in the case of a binary tree. For n-ary nodes that represent compositing operators with the operand independence property, a set of flags in field Link Flags 1945 are set to indicate that operand slots are not updated.

3.4 INTITIAL RENDERING OF AN N-ARY COMPOSITNG TREE 580809.doc -38- At step 120, the n-ary compositing tree being processed by the processor 2605 is rendered and the resulting output is displayed. Also at step 120, various actions as indicated in the state transition diagrams in Figs. 14, 15, 16, 17 and 18 take place and the state of all nodes is moved to updated 1402, 1502, 1702), this being reflected in field State 1910. For each node, the display region is stored in field Display Region 1915, as is the case of a binary tree.

For nodes representing an operator with operand independence, the pending damage region stored in field Damage Region 1920 is initialized to an empty region. This region is abbreviated to P in the Figs. 14 to 18. Operand slots are moved to state updated 1805 where appropriate, this being indicated in field Link Flags 1945. Note that the field Damage Region 1920 is used by nodes with operator independence for storing pending damage, as will be explained later. In addition, the field Damage Region 1920 is used by eeee all nodes to store the previous display region, as was field Old Display Region 330 for a binary compositing tree.

°eeee For n-ary nodes, step 120 is subdivided into substeps as shown in the flow chart of Fig. 21. Each substep 2110 to 2150 involves a depth-first tree traversal. The method of the flow chart of Fig. 21 is preferably implemented as an application program being resident on the hard disk drive 2610 and being read and controlled in its execution by the processor 2605.

3.4.1 CREATE BINARY TREE WITH HAVE AND OPAQUE REGIONS During a first tra\versal of the n-ary tree being processed, at step 2110, the method creates a binary tree that is equivalent to the given n-ary tree. In accordance with the above mentioned definition of n-ary nodes: an n-ary node with no enabled, non-null operands is represented in the equivalent binary tree by a node that represents a fully transparent image; 580809.doc -39an n-ary node with a single enabled, non-null operand is replaced by the operand the node is effectively ignored); an n-ary ndde with exactly two enabled, non-null operands is replaced by a single binary node with the same two operands; and an n-ary node with three or more enabled, non-null operands is replaced by a sequence of two or more binary nodes with operands chosen in accordance with the definition.

As an example, Fig. 22 shows an n-ary node 2210 and a binary tree 2220 that is equivalent to the n-ary node 2210. The n-ary node 2210 with operator "over" has six operands. In accordance with the example, operand 2212 B is disabled and Operand 2215 is null. The four remaining operands 2211 A, 2213 C, 2214 D and 2216 E are themselves o replaced by equivalent binary trees 2221 2223 2224 D' and 2226 These equivalent binary trees 2221 2223 2224 D' and 2226 E' are combined to create a binary tree 2220 that is equivalent to the n-ary node 2210. The binary nodes 2221 A' and 2223 C' corresponding to the first two enabled n-ary operands 2211 A and 2213 C are first composited. Then each remaining enabled, non-null operand is composited in turn to the result of the previous composition.

A preferred data structure 2000 for each binary node is shown in Fig. 20. Each *node has a reference to its parent node in the binary tree in field Parent 2010. Each node also has a reference to the equivalent n-ary node in field Equivalent N-ary 2025. If several binary nodes are created for the one n-ary node, then the top-most most ancestral) binary node stores the reference to the corresponding n-ary node. When an n-ary node has exactly one enabled, non-null operand, both the n-ary node and its operand correspond to the same binary node. In this instance, field Equivalent N-ary 2025 acts like the first element of a list of equivalent n-ary nodes. Field Next Equivalent N-ary 1935 in the n-ary node data structure 1900 continues this list.

580809.doc Binary nodes also store a node type in field Node Type 2035. Preferably, there are three types, "void", "operator" and "primitive". Void binary nodes represent fully transparent images and are derived from n-ary nodes that represent compositing operators but which have no enabled, non-null operands. Void binary nodes are preferably optimized out before rendering takes place. Operator nodes represent compositing operations, in which case field operator 2040 stores the compositing operator and fields Left 2045 and Right 2050 store the left and right operands of the operator. Primitive nodes represent graphical primitives, in which case field Primitive 2055 stores a reference to the specification of the primitive, for example, a pointer to the n-ary tree node for the primitive.

As the binary tree is being created by the processor 2605 at step 2110, the .processor 2605 also constructs the have and opaque regions of the nodes in the binary tree in the same manner as described for binary compositing tees. The binary tree can be created and the have and opaque regions constructed in two steps. However, the binary S 15 tree and the have and opaque regions can easily be constructed in one step, and thus execution time is saved.

SPseudocode for step 2110 is as follows: procedure nary to binary(n-ary tree t) 20 begin procedure new_primitive_node(n-ary node p) begin b allocate binary node.

b.Parent null.

b.Equivalent N-ary p, p.Next Equivalent N-ary null. Create list of one equivalent n-aiy node.

b.Node Type primitive.

b.Primitive p.

return b.

end 580809.doc procedure newoperator_node(op, binary nodes left, right) begin b allocate binary node.

b.Parent null.

b.Equivalent N-ary null. Create empty list of equivalent n-ary nodes.

b.Node Type operator.

b.Operator op.

b.Left left.

b.Right right.

left.Parent b.

right.Parent b.

return b.

end procedure new_void_node(p) begin b allocate binary node.

b.Parent null.

b.Equivalent N-ary p, p.Next Equivalent N-ary null. Create list of one 20 equivalent n-ary node.

b.Node Type ,-,void.

return b.

end 25 nary_to binary...

Perform depth-first traversal of n-ary tree t, visiting each node p: When arriving at a primitive: b new_primitive_node(p).

Determine the primitive's opaque and have regions and store in b.Opaque Region and b.Have Region.

When first arriving at an operator node: p.Binary Tree Temp null.

Continue traversal down to each enabled, non-null operand.

After each operand of operator node p is visited: At each stage, p.Binary Tree Temp stores the binary tree corresponding to the composition of all operands up to but not including the current one. b will store the binary tree corresponding to the current operand.

if p.Binary Tree Temp null then p.Binary Tree Temp b thefirst enabled, non-null operand.

else begin x p.Binary Tree Temp.

if p's compositing operator is a reverse operator ("rover", "rin etc.) then 580809.doc 42 p.Binary Tree Temp new_operator node(reverse of p's compositing operator, b, The initial binary tree has no reverse operators in order to simplify later processing.

else p.Binary Tree Temp new_operatornode(p's compositing operator, x, b).

Determine p.Binary Tree Temp.Opaque Region, Have Region from x.Opaque Region, Have Region and b.Opaque Region, Have Region.

end Before leaving an operator node: if p.Binary Tree Temp null then b new_void_node(p). There were no enabled, non-null operands.

else begin Finished all operands, sop.Binary Tree Temp stores the binary tree equivalent to n-ary tree p.

b p.Binary Tree Temp. p's parent node is expecting b to be set.

Insert p into b's list of equivalent n-ary nodes.

end S End of traversal.

return b.

20 end nary_tobinary 3.4.2 REGION MINIMIZATION AND OCCLUSION The next step shown in Fig. 21, step 2120, is performed by the processor 2605 25 on the newly created binary tree, in a similar manner to that described earlier for a binary compositing tree. The difference is that while the have and opaque regions are stored in the binary nodes, the display region and the old display region are stored in the n-ary i nodes. Thus, when the final value of Have Region 2020 is determined, each of the n-ary nodes in the list of equivalent n-aries is examined (starting from field Equivalent N-ary 2025) and for each n-ary node, Damage Region 1920 is set to Display Region 1915 and Display Region 1915 is set to the Have Region field 2020 of the binary node. Step 2120 is equivalent to the first two steps 1421 and 1422 of action 1412 in Fig. 14, the first two steps 1521 and 1522 of action 1512 in Fig. 15 and the first two steps 1721 and 1722 of action 1712 in Fig. 17. The third step 1423, 1523, 1723) does not need to be 580809.doc -43performed as part of step 120 because the entire output area is being updated and so an update area is not needed.

At step 2120, the binary tree constructed in step 2110 can be changed by the processor 2605 to prune out clipped and occluded nodes, as described for step 420. Void binary nodes are also optimized out in step 2120. In step 420, working tree pointers were used when modifying the tree so as to leave the data structure unchanged. This is unnecessary in step 2102 since the original data structure is the n-ary tree.

Pseudocode for step 2120 is as follows: procedure region_minimization(binary tree t, want_region,flag update_nary_regions) begin procedure clobber(binary tree t) begin 15 Set to empot the display regions of all n-ary nodes referred to from the binary nodes in a given subtree.

Perform depth-first traversal of binary tree t, visiting each node b: i for all nodes p in b's list of equivalent n-ary nodes do begin if p.State new/modified then :Transfer p.Display Region to p.Damage Region.

p.Display Region 0.

end End of traversal.

25 end region_minimization...

Restrict root node's region to want region.

t.Have Region t.Have Region n want_region.

if t.Have Region 0 then begin The case of the vanishing root.

if update_nary_rgns then clobber(t).

t null. There is nothing to render.

end else begin 580809.doc -44- Perform depth-first traversal of binary tree t, visiting each node b: At every node do the following as soon as we come to it: if update_nary_rgns then for all nodes p in b's list of equivalent n-ary nodes do begin if p.State new/modified then Transfer p.Display Region to p.Damage Region.

Set p.Display Region to a copy of b.Have Region.

end If the node is an internal node (not a leaf), also do the following when we first come to it: Restrict both operands' result regions to the parqnt's result region.

b.Left.Have Region b.Left.Have Region n b.Have Region.

b.Right.Have Region b.Right.Have Region r b.Have Region.

Make left operand of "over occlude the right.

if b.Operator "over" then b.Right.Have Region b.Right.Have Region b.Left.Opaque Region.

0 Handle the cases where one of the operands has vanished.

if b.Left.Have Region 0 then 20 begin .if updatenary_rgns then clobber(b.Left).

Replace b with its right subtree.

Continue traversal by visiting (former) right node.

25 end else if b.Right.Have Region 0 then begin if update_nary_rgns then clobber(b.left).

Replace b with its left subtree.

Continue traversal by visiting (former) left node.

end End of traversal.

end end regionminimization This procedure is called with parameter update_nary_regions passed a boolean true value.

3.4.3 MEMORY OPTIMIZATION The binary tree resulting from step 2120 is further processed by the processor 2605 at step 2130 with the view of minimizing the peak memory usage during the 580809.doc rendering phase 2140. The method used at step 2130 is the same as that described above for binary composting trees.

Pseudocode for step 2130 is as follows: procedure memory_optimization(binary tree t) begin Perform depth-first traversal of binary tree t, visiting each node b: When arriving at a leaf node, do the following: b.Cost memory needed to load a primitive of type of b.Primitive in area ofb.Have Region.

When first arriving at an operator node, do the following: if b.Operator is associate then begin Apply associativity property.

15 while b.Right is not a leaf and b.Right.Operator b.Operator do begin Rearrange b.Left, b.Right, b.Right.Left and b.Right.Right according to associative law, keeping b at the root. As shown in Figure 23, this requires rearranging nodes in tree 2310 to form tree 2320.

20 if b.Operator is "over" or "plus" then begin b.Left.Opaque Region b.Left.Left.Opaque Region u b.Left.Right.Opaque Region.

b.Left.Have Region b.Left.Left.Have Region u b.Left.Right.Have Region.

end end end After visiting an operator node, do the following before returning to its parent: 30 Firstly determine cost ofpresent arrangement.

if b.Right is an operator node then cost b.Right.Cost memory cost of compositing a subtree of area of b.Right.Have Region with operator b.Operator.

else cost memory needed to composite a primitive of type of b.Right.Primitive, of area of b.Right.Have Region, with operator b.Operator.

cost max(cost, b.Left.Cost).

Secondly determine cost of alternative arrangement.

if b.Left is an operator node then alt_cost b.Left.Cost memory cost of compositing a subtree of area of b.Left.Have Region with reverse of operator b.Operator.

else 580809.doc -46alt_cost memory needed to composite a primitive of type of b.Left.Primitive, of area of b.Left.Have Region, with reverse of operator b.Operator.

alt_cost max(alt_cost, b.Right.Cost).

If the alternative is better then swap the operands.

if alt cost cost then begin Interchange b's left and right operands.

Reverse b's operator.

b.Cost alt cost.

end else b.Cost cost.

End of traversal.

end memory_optimization 3.4.4 RENDERING THE TREE Continuing with Fig. 21, having completed step 2130, the method proceeds to step 2140 and the image represented by the tree is rendered by the processor 2605. As the tree is now an optimized binary tree, the same method as described for step 220 above is followed.

Pseudocode for step 2140 is as follows: procedure composite_tree(binary tree t) .begin Set up the initial frame buffer.

Perform depth-first traversal of binary tree t, visiting each node b: :When arriving at a leaf node, do the following: Render b.Primitive into the current frame buffer.

When arriving at an internal node after having visited the left child, do the following: if b.Right is a leaf node then begin Render and composite b.Right.Primitive, with b.Operator as the compositing operator and b.Right.Have Region as the region to composite.

Do not continue traversal down to the right child; go back to b's parent node.

end else Create a new frame buffer for the right subtree and push it on the stack.

When arriving at an internal node after having visited the right child, do the following: 580809.doc -47- Composite the two top buffers on the stack, which represent b's left and right operands, storing the result in the lower buffer.

Discard the top buffer from the stack.

End of traversal.

end compositetree 3.4.5 SET STATES TO UPDATED The final step in Fig. 21, step 2150, includes traversing the original n-ary tree constructed in step 110. At step 2150, any node involved the node is not a disabled operand or part of a disabled operand) in the update is moved to state updated (1402, 1502, 1702) by setting Field State 1910 to the appropriate value. All operand slots of nary nodes involved in the update that are enabled and have operands presented are moved to state updated 1805 by setting the appropriate flag in field Link Flags 1945. Field Damage Region 1920 is set to an empty region in each updated node, finishing the last step 1724 of action 1712.

Pseudocode for step 2150 is as follows: procedure setupdated_flags(n-ary tree t) begin Perform depth-first traversal of n-ary tree t, visiting each node p: When first arriving at each node, do the following: p.State updated.

•p.Damage Region 0.

Before descending down to each operand i of compositing operator node p: if operand i is null or disabled then Do not continue traversal down to child i.

if The operator represented by node p has operand independence property ("over", "rover" or "plus") then Set flag for operand i in p.Link Flags to indicate that the operand slot is updated.

End of traversal.

end MODIFYING THE N-ARY TREE 580809.doc -48 As described earlier for binary trees, a global counter can be used along with fields Dirty Counter 1925 and Last Count as Root 1930 in each node to help find changes to a tree between updates tnd to detect if the root node of the tree has changed since the previous update.

Moving to step 130 of Fig. 1, the method can make modifications to the n-ary compositing tree in accordance with general graphical output requirements. Some changes need to be made to the n-ary nodes' state and data structure in response to the various modifications that may be made to the n-ary compositing tree. A typical API will have procedures for enabling an application to make changes to the image representation data structure. These procedures will also preferably perform the steps described below.

If a node that represents a graphical primitive is modified, and that node is in the updated state 1402, then the state of the node is changed to new/modified 1401 and action 1413 shown in Fig. 14 is performed by the processor 2605. This action involves setting the Dirty Counter field 1925 to the current value of the global counter in the sequence of nodes to form the modified node at the root of the tree. If a node in this sequence is encountered that has already had the associated dirty counter set to the same value, then it is not necessary to proceed further as this indicates that another modification has set the dirty counters from that point on. Finally, if a node is encountered that is a disabled operand, then again it is not necessary to proceed further, as the modified node is not part of the displayed image and hence modification of the node does not effect any change.

Pseudocode for action 1413 is as follows: procedure modi fy_node(node) begin if node.State updated then begin node.State new/modified.

set_dirty(node).

580809.doc end Else we do not need to mark it as dirty because it has either already been marked dirty or it is a new node. A new node will either be used as a new root, or it will be linked somewhere and its parent will be marked as dirty.

end procedure set_dirty(node) begin while node.Dirty Counter global counter do begin node.Dirty Counter global counter.

if node has no parent or node is disabled in its parent then Return.

node node.Parent.

end end If a node that corresponds to a compositing operation has a new operand linked into an enabled slot or an existing disabled operand to such a node is enabled, then there 20 are a number cases to consider. If the node is in state new/modified 1501 or 1701, no action need happen. If the node represents an operator without the operand independence property and is in state updated 1502, then action 1513 takes place and the state of the node is changed to new/modified 1501. Action 1513 is identical to action 1413 with pseudocode given above. If the node represents an operator with operand independence, and is in state updated 1702, then action 1714 takes place and the state is left unchanged.

Pseudocode for action 1714 is as follows: Handle an operand enabled/linked into a node whose operator has operand independence property.

if node.State updated then set_dirty(node).

If a node that corresponds to a compositing operation has an operand unlinked from an enabled slot or an enabled operand of such a node is disabled, then again there are a number of cases. Firstly, if the node is in state new/modified 1501 or 1701, no 580809.doc action is taken. Secondly, if the node represents an operator without the operand independence property and is in state updated 1502, then action 1513 takes place and the state of the node is changed to new/modified 1501. Thirdly, if the node represents an operator with operand independence and is in state updated 1702, then two further cases exist. Either the operand slot involved is in state new 1804 or in state updated 1805. If the operand is in state new 1804, then action 1714 as described above takes place. If the operand is in state updated 1805, then action 1713 takes place. Action 1713 sets the dirty counters as described earlier, but also saves the unlinked or disabled operands display region as pending damage for the next update.

Pseudocode for action 1713 is as follows: Handle an operand disabled/unlinked from a node whose operator has operand independence property.

if node.State updated then 15 begin if node.Link Flag for operand to be disabled/unlinked is updated then begin node.Damage Region node.Damage Region u operand node.Display Region.

Clear node.Link Flag for operand.

20 end setdirty(node).

end Finally, if a change is made to a node that represents a compositing operation which changes the image in some way, then, if the node is in state updated 1502 or 1702, action 1513 or 1715 takes place and the state of the node is changed to new/updated 1501 or 1701. Actions 1513 and 1715 are identical to action 1413 described above.

The method described above covers modification of existing nodes. It is also possible to create new nodes and delete nodes that are no longer required. Any new node is initially in state new/modified (1401, 1501 or 1701). When a node is added to the compositing tree the node is linked in, thereby modifying the new parent node of the 580809.doc -51 particular node. Similarly, when a node is to be deleted, the node is first unlinked from the compositing tree. In this way, creation and deletion of nodes is handled by treating the node as a modification of the parent node involved.

3.6 INCREMENTAL UPDATE Continuing on to step 140 of Fig. 1, when modifications to the n-ary compositing tree have been made, a request is made to update the displayed output so that the output agrees with the new version of the tree. For n-ary trees, step 140 is subdivided into substeps as shown in Fig. 24. Each step involves a depth-first tree traversal. The process of the flow chart of Fig. 24 is preferably implemented as an application program being resident on the hard disk drive 2610 and being read and controlled in its execution by the processor 2605.

The first traversal performed by the processor 2605, at step 2410, is the same as at step 2110. Procedure nary_to_binary can be used.

The second traversal, at step 2420, is on the newly created binary tree. Step 2420 15 is the same as step 2120. Procedure regionminimiztion can be used, again with a boolean value of true passed for parameter update_nary_regions. Recall that procedure region_minimiztion performs steps 1421 and 1422 of action 1412, steps 1521 and 1522 of action 1512, and steps 1721 and 1722 of action 1712 for new and modified nodes. For nodes that have not been modified, this procedure performs actions 1414, 1514 and step 1732 of action 1716.

The third traversal, at step 2430, is on the n-ary tree, and the processor 2605 determines the update region in a similar way as described for binary trees. As the n-ary tree is traversed, only branches that are enabled and also whose Dirty Counter field 1925 equals the current value of the global counter are followed. When any node in new/modified state is reached, the new and old display regions for the node, stored in fields Display Region 1915 and Damage Region 1920 are included in the update region.

580809.doc 52- Step 2430 is equivalent to step 1423 of action 1412, step 1523 of action 1512 and step 1723 of action 1712. When an updated node that represents an operator with operand independence is reached, the effects of any removed or added operands are included in the update region. The Damage Region field 1920 contains the union of the display regions of all operands that were removed since the last update, and is included in the update area, as indicated in step 1734 of action 1716. Each enabled non-null operand whose operand slot is in state new 1804 represents an added operand. Each such root node operand display region is included in the update area, as indicated by action 1812, thus performing step 1734 of action 1716.

Pseudocode for step 2430 is as follows: procedure obtain_update_region(n-ary tree t) begin S 0. Start with an empty update region and add to it as we traverse the tree and 15 discover nodes that have changes since the last update.

Perform depth-first traversal of n-ary tree t, visiting each node p: •When first arriving at each node, do the following: if p.State new/modified then begin 20 S S u p.Damage Region u p.Display Region.

p.Damage Region 0.

Do not traverse this node's children; go back to its parent.

end When first arriving at an operator node, do the following: if p.Operator has operand independence property ("over", "rover" or "plus") then begin S S p.Damage Region.

p.Damage Region 0.

end Before descending down to each operand i of compositing operator node p: if operand i is null or disabled then Skip this operand; continue with next operand, if any.

else if p.Operator has operand independence property ("over", "rover" or "plus") and flag for operand i in p.Link Flags indicates that it is not updated then begin S S u operand i's Damage Region.

580809.doc -53 Do not traverse this down to this operand; go to next operand.

end else if operand i's Dirty Counter the global counter then Skip this operand; continue with next operand, if any.

End of traversal.

return S.

end obtain_update_region Continuing with Fig. 24, step 2440 is a repetition of the region minimization process of step 2420 but with two important differences. The first difference is that the region to which all output is restricted is the update region determined in step 2430, instead of the entire output display area used in step 2420. This has the effect of producing a minimal tree in terms of the output that needs to be generated to update the display. Portions of the tree that do not affect the part of the output that has changed are 15 pruned from the tree, as a result of which the subsequent rendering step is made more efficient. The second difference between step 2440 and step 2420 is that a boolean value of false is passed for parameter update_nary_regions, which stops the n-ary tree nodes' o: display regions being set. The display regions are already set by step 2420 as setting them at step 2420 would result in invalid regions being stored.

.i 20 The remaining steps in Fig. 24, namely 2450, 2460 and 2470 are the same as steps 2130, 2140 and 2150 described earlier. Pseudocode has been presented above for steps 2450, 2460 and 2470.

HANDLING MEMORY LIMITATIONS The methods described above perform various optimizations prior to rendering.

In particular, the binary form of the compositing tree representing an image is rearranged as far as possible to minimize the peak video memory usage during the rendering phase.

However, it may happen that a compositing tree is sufficiently complex that even the optimal form requires more memory than is available. It would be desirable to have a 580809.doc -54means of producing the required output image, even if perhaps at the expense of increased execution time.

Preferably, the region-to be rendered, whether the entire output area (in step 120) or the updated area (in step 140), is subdivided. For each subdivision, a copy of the corresponding tree can be made, region minimization and memory optimization applied with respect to the smaller region of output, and the re-optimized tree rendered. For each subdivision, the re-optimized tree will be simpler, and the amount of video memory required less, so that it is more likely to be able to be rendered within the available memory constraints. If, however, there is still insufficient memory, the process can be repeated recursively. That is, the subdivided area may itself be further subdivided, and so *.99 on, until memory constraints are satisfied.

Preferably the subdivision is performed using a quadtree. The bounds of the output region are halved both vertically and horizontally, creating four quadrants. Any quadrant that requires further subdivision can then be similarly subdivided into four.

99*e99 *o 9e S. 15 However, other methods of subdivision are possible and fall within the scope of the invention. Fig. 25 shows a flow chart for performing step 960 (which is the same as step 9. 9 2460). It is important the procedure region_minimization is called with a boolean value of 0. .false passed for parameter update_nary_rgns. The process of the flow chart of Fig. 25 is preferably implemented as an application program being resident on the hard disk drive 2610 and being read and controlled in its execution by the processor 2605.

The process of Fig. 25 begins at step 2510 where a check is performed by the processor 2605 to determine if there is sufficient memory memory 2606) to render and composite the image represented by a compositing tree being processed. If the result of step 2510 is true then the process proceeds to step 2520 where the image represented by the tree is rendered and composited. If the result of step 2510 is false then the process proceeds to step 2530 where the output area is subdivided by the processor 2605 into four 580809.doc quadrants. At the next step 2540, a copy of the binary tree representing the first quadrant is made. The process continues at the next step 2541, where the region_minimization procedure is performed by the processor 2605 on the first quadrant. At the next step 2542, memory optimisation as described above is performed. The process continues at the next step 2543, where step 960 is performed on the copy of the tree made at step 2540 and stored on the hard disk drive 2610 or memory 2606. At the next step 2550, a copy of the binary tree representing the second quadrant is made by the processor 2605. The process continues at the next step 2551, where the regionminimization procedure is performed on the second quadrant. At the next step 2552, memory optimisation as described above is performed. The process continues at the next step 2553, where step 960 is performed on the copy of the tree made at step 2550. At the next step 2560, a copy of the binary tree representing the third quadrant is made. The process continues at the

OS

next step 2561, where the region_minimization procedure is performed by the processor 2605 on the third quadrant. At the next step 2562, memory optimisation as described above is performed. The process continues at the next step 2563, where step 960 is &too performed on the copy of the tree made at step 2560. At the next step 2570, a copy of the binary tree representing the fourth quadrant is made. The process continues at the next step 2571, where the region_minimization procedure is performed by the processor 2605 Son the fourth quadrant. At the next step 2572, memory optimisation as described above is performed. The process continues at the next step 2573, where step 960 is performed on the copy of the tree made at step 2570 and the process concludes.

The aforementioned preferred methods comprise a particular control flow. There are many other variants of the preferred methods which use different control flows without departing the spirit or scope of the invention. Furthermore one or more of the steps of the preferred methods can be performed in parallel rather then sequentially.

580809.doc -56- The preferred methods are preferably practiced using a conventional generalpurpose computer system 2600, such as that shown in Fig. 26 wherein the processes of As described above, te methods of Figs. 1 to 25 can be implemented as software, such as an application program executing within the computer system 2600. In particular, the steps of the preferred methods are effected by instructions in the software that are carried out by the computer. The software can be divided into two separate parts; one part for carrying out either of the preferred methods; and another part to manage the user interface between the latter and the user. The software can be stored in a computer readable medium, including the storage devices described below, for example. The software is loaded into the computer from the computer readable medium, and then executed by the computer. A computer readable medium having such software or computer program recorded on it is a computer program product. The use of the computer program product in the computer preferably effects an advantageous apparatus for preforming the methods in accordance with the invention.

15 The computer system 2600 comprises a computer module 2601, input devices such as a keyboard 2602 and mouse 2603, output devices including a printer 2615 and a display device 2614. A Modulator-Demodulator (Modem) transceiver device 2616 is °oooo used by the computer module 2601 for communicating to and from a communications S.network 2620, for example connectable via a telephone line 2621 or other functional oooo° medium. The modem 2616 can be used to obtain access to the Internet, and other network systems, such as a Local Area Network (LAN) or a Wide Area Network (WAN).

The computer module 2601 typically includes at least one processor unit 2605, a memory unit 2606, for example formed from semiconductor random access memory (RAM) and read only memory (ROM), input/output interfaces including a video interface 2607, and an 1/O interface 2613 for the keyboard 2602 and mouse 2603 and optionally a joystick (not illustrated), and an interface 2608 for the modem 2616. A 580809.doc -57storage device 2609 is provided and typically includes a hard disk drive2610 and a floppy disk drive 2611. A magnetic tape drive (not illustrated) may also be used. A CD- ROM drive2612 is typically7 provided as a non-volatile source of data. The components 2605 to 2613 of the computer module 2601, typically communicate via an interconnected bus 2604 and in a manner which results in a conventional mode of operation of the computer system 2600 known to those in the relevant art. Examples of computers on which the embodiments can be practised include IBM-PC's and compatibles, Sun Sparcstations or alike computer systems evolved therefrom.

Typically, the application program of the preferred embodiment is resident on the hard disk drive 2610 and read and controlled in its execution by the processor 2605.

Intermediate storage of the program and any data fetched from the network 2620 may be ooe.

accomplished using the semiconductor memory 2606, possibly in concert with the hard eeoc.

disk drive 2610. In some instances, the application program may be supplied to the user eee e encoded on a CD-ROM or floppy disk and read via the corresponding drive 2612 or 2611, 15 or alternatively may be read by the user from the network 2620 via the modem device 2616. Still further, the software can also be loaded into the computer system 2600 from other computer readable medium including magnetic tape, a ROM or integrated ee.o•i circuit, a magneto-optical disk, a radio or infra-red transmission channel between the r••e -computer module 2601 and another device, a computer readable card such as a PCMCIA card, and the Internet and Intranets including email transmissions and information recorded on websites and the like. The foregoing is merely exemplary of relevant computer readable mediums. Other computer readable media may be practiced without departing from the scope and spirit of the invention.

The preferred methods can alternatively be implemented in dedicated hardware such as one or more integrated circuits performing the functions or sub functions of Figs.

580809.doc 58- 1 to 25. Such dedicated hardware can include graphic processors, digital signal processors, or one or more microprocessors and associated memories.

The foregoing describes only some embodiments of the present invention, and modifications and/or changes can be made thereto without departing from the scope and spirit of the invention, the embodiments being illustrative and not restrictive.

In the context of this specification, the word "comprising" means "including principally but not necessarily solely" or "having" or "including" and not "consisting only of'. Variations of the word comprising, such as "comprise" and "comprises" have corresponding meanings.

e S* 580809.doc

Claims (4)

1. 580809.doc
2. 2. The method according to claim 1, wherein said step d(iii) includes the sub-step of compositing those said objects which contribute to said further image within said update region.
3. 3. The method according to any one of claims 1 or 2, wherein said union comprises previously stored visible regions for any objects removed from said image.
4. 4. The method according to any one of claims 1 or 2, wherein said union comprises visible regions for any objects that are added to said image. The method according to any one of the preceding claims, wherein said graphical objects are composited according to a hierarchical structure representing a compositing expression, said hierarchical structure including a plurality of nodes each representing an operand of said compositing expression or an operator for combining said operands, 15 wherein each of said graphical objects is represented by at least one of said operands. oooo 6. A method of creating a series of images, said images being formed by rendering and compositing a plurality of graphical objects to be composited according to a hierarchical data structure representing a compositing expression, each of said objects having a predetermined boundary outline, said hierarchical data structure including a plurality of nodes each representing an operand of said compositing expression or an o operation for combining said operands, wherein each of said graphical objects is •represented by at least one of said operands, said method comprising the steps of: he determining a visible region for each node, wherein said visible region is defined by a region outline substantially following at least one of said predetermined outlines or parts thereof, depending on those said objects which contribute to said region, 580809.doc -61 said contribution being determined on the basis of the compositing of said graphical object with any other graphical object, storing said visible regions; rendering a first image of said series by compositing said graphical objects according to said hierarchical structure; in response to at least one change to at least one of said nodes; determining a further visible region for each node, wherein said further visible region is defined by a region outline substantially following at least one of said predetermined outlines or parts thereof, depending on those said objects which contribute to said further visible regions; (ii) forming an update region as the union of said stored visible regions and said further visible regions for those nodes affected by said at least one change; (iii) creating a further image of said series by rendering a portion of said further image, said portion being substantially equal to said update region; and repeating step for further changes to at least one of said nodes. 7. The method according to claim 6, wherein said contribution is determined on the 4 basis of clipping of said graphical object by any other graphical object and/or on the basis of the compositing of said graphical object with any other graphical object; 8. The method according to any one of claims 6 and 7, wherein said step d(iii) includes the sub-step of compositing those said objects which contribute to said update region. 9. The method according to any one of claims 6, 7 and 8, wherein said union also includes previously stored visible regions for any nodes removed from said image. 580809.doc The method according to any one of claims 6, 7 and 8, wherein said union also includes visible regions for any nodes that are added to said image. 11. The method according to claim 10, wherein said union also includes new and previously stored visible regions for nodes affected by said at least one change and for which operand independence does not apply. 12. The method according to claim 10, wherein said union also includes visible regions for any nodes representing operands which were added to said image and which are directly associated with an operation for which operand independence applies. 13. The method according to any one of claims 11 and 12, wherein said previously stored visible regions are each associated with a corresponding parent node. 14. The method according to any one of claims 6 to 13, wherein step includes the further sub-step of recording which nodes of said hierarchical data structure are affected by said at least one change. 15. The method according to claim 14, wherein step d(ii) includes the sub-step of examining (only) those said nodes affected by said at least one change. 16. The method according to any one of claims 6 to 15, wherein step d(iii) includes the sub-step of modifying said hierarchical data structure based on said visible and further visible regions, wherein said further image is rendered according to a compositing expression represented by said modified hierarchical data structure. 580809.doc -63- 17. A method of creating a series of images, said images being formed by rendering and compositing a plurality of graphical objects to be composited according to a hierarchical data structure representing a compositing expression, each of said objects having a predetermined boundary outline, said hierarchical data structure including a plurality of nodes each representing an operand of said compositing expression or an operator for combining said operands, wherein each of said graphical objects is represented by at least one of said operands, said method comprising the steps of: for each node: examining said node to determine those of said objects which contribute to said node, said contribution being determined on the basis of occlusion of any graphical object associated with said node by any other graphical object associated with said node; (ii) determining a visible region for said node depending on said 15 examination, wherein said visible region is defined by a region outline substantially following at least one of said predetermined outlines or parts thereof; rendering a first image of said series by compositing said graphical objects according to said hierarchical structure; determining memory required for rendering and combining said operands associated with each node; modifying at least one node representing an operator for combining said S•operands, by interchanging operands associated with said node and reversing said ••go operator to render a visible region associated with said node, if memory required for rendering and combining operands associated with said node is minimised by said modification; in response to at least one update to at least one of said nodes; 580809.doc -64- determining a further visible region for each node using any modified compositing expressions determined at step depending on those said objects which contribute to said further visible regions; (ii) forming an update region as the union of said visible regions and said further visible regions for those nodes affected by said at least one update; (iii) creating a next image of said series by rendering an image space in which said graphical objects are defined (only) in an area substantially equal to said update region; repeating steps to for further changes to at least one of said nodes. 18. The method according to claim 17, wherein said step e(iii) includes the sub-step of compositing those said objects which contribute to said update region. 19. The method according to claim 17, wherein said union comprises previously stored visible regions for any objects removed fr-om said image. 20. The method according to claim 17, wherein said union comprises visible regions for any objects that are added to said image. 21. An apparatus for creating a series of images, said images being formed by rendering and compositing a plurality of graphical objects, each of said objects having a predetermined boundary outline, said apparatus comprising: means for determining a visible region for each graphical object, wherein said visible region is defined by a region outline substantially following at least one of said predetermined outlines or parts thereof, depending on those said objects which contribute to said region, said contribution being determined on the basis of clipping and/or 580809.doc 65 occlusion of said graphical object by any other graphical object, and/or on the basis of the compositing of said graphical object with any' other graphical object; means for storing said visible regions; and means for rendering a first image of said series by compositing said graphical objects, wherein in response to at least one change to at least one of said objects, said means for determining a visible region: determines a further visible region for each graphical object, wherein said further visible region is defined by a region outline substantially following at least one of said predetermined outlines or parts thereof, depending on those said objects which contribute to said further visible regions, forms an update region as the union of said stored visible regions and said further visible regions for those objects affected by said at least one change; and creates a further image of said series by rendering a portion of said -15 further image, said portion being substantially equal to said update region; and 22. The apparatus according to claim 21, further comprising means for compositing i those said objects which contribute to said further image within said update region. 23. The apparatus according to any one of claims 21 or 22, wherein said union comprises previously stored visible regions for any objects removed from said image. 24. The apparatus according to any one of claims 21 or 22, wherein said union comprises visible regions for any objects that are added to said image. 580809.doc -66- The apparatus according to any one of claims 21 to 24, wherein said graphical objects are composited according to a hierarchical structure representing a compositing expression, said hierarchical structure including a plurality of nodes each representing an operand of said compositing expression or an operator for combining said operands, wherein each of said graphical objects is represented by at least one of said operands. 26. An apparatus for creating a series of images, said images being formed by rendering and compositing a plurality of graphical objects to be composited according to a hierarchical data structure representing a compositing expression, each of said objects having a predetermined boundary outline, said hierarchical data structure including a plurality of nodes each representing an operand of said compositing expression or an operator for combining said operands, wherein each of said graphical objects is represented by at least one of said operands, said apparatus comprising: means for determining a visible region for each node, wherein said visible region 15 is defined by a region outline substantially following at least one of said predetermined outlines or parts thereof, depending on those said objects which contribute to said region, said contribution being determined on the basis of the compositing of said graphical object with any other graphical object, means for storing said visible regions; S 20 means for rendering a first image of said series by compositing said graphical objects according to said hierarchical structure, wherein in response to at least one change to at least one of said nodes, said determining means; determines a further visible region for each node, wherein said further visible region is defined by a region outline substantially following at least one of said predetermined outlines or parts thereof, depending on those said objects which contribute to said further visible regions; 580809.doc -67- forms an update region as the union of said stored visible regions and said further visible regions for those nodes affected by said at least one change; creates a further image of said series by rendering a portion of said further image, said portion being substantially equal to said update region; and 27. The apparatus according to claim 26, wherein said contribution is determined on the basis of clipping of said graphical object by any other graphical object and/or on the basis of the compositing of said graphical object with any other graphical object; 28. The apparatus according to any one of claims 26 and 27, further includes the sub-step of compositing those said objects which contribute to said update region. 29. The apparatus according to any one of claims 26, 27 and 28, wherein said union S"also includes previously stored visible regions for any nodes removed from said image. S The apparatus according to any one of claims 26, 27 and 28, wherein said union also includes visible regions for any nodes that are added to said image. 31. The apparatus according to claim 30, wherein said union also includes new and previously stored visible regions for nodes affected by said at least one change and for which operand independence does not apply. 32. The apparatus according to claim 30, wherein said union also includes visible regions for any nodes representing operands which were added to said image and which are directly associated with an operation for which operand independence applies. 580809.doc -68- 33. The apparatus according to any one of claims 31 and 32, wherein said previously stored visible regions are each associated with a corresponding parent node. 34. The apparatus according to any one of claims 26 to 33, further including means for recording which nodes of said hierarchical data structure are affected by said at least one change. The apparatus according to claim 34, further including means for examining (only) those said nodes affected by said at least one change. 36. The apparatus according to any one of claims 26 to 35, further including means for modifying said hierarchical data structure based on said visible and further visible regions, wherein said further image is rendered according to a compositing expression represented by said modified hierarchical data structure. 37. An apparatus for creating a series of images, said images being formed by rendering and compositing a plurality of graphical objects to be composited according to a hierarchical data structure representing a compositing expression, each of said objects having a predetermined boundary outline, said hierarchical data structure including a plurality of nodes each representing an operand of said compositing expression or an operator for combining said operands, wherein each of said graphical objects is represented by at least one of said operands, said apparatus comprising: means for examining each said node to determine those of said objects which S"contribute to said node, said contribution being determined on the basis of occlusion of any graphical object associated with said node by any other graphical object associated with said node; 580809.doc -69- means for determining a visible region for said node depending on said examination, wherein said visible region is defined by a region outline substantially following at least one of said predetermined outlines or parts thereof; means for rendering a first image of said series by compositing said graphical objects according to said hierarchical structure; means for determining memory required for rendering and combining said operands associated with each node; means for modifying at least one node representing an operator for combining said operands, by interchanging operands associated with said node and reversing said operator to render a visible region associated with said node, if memory required for rendering and combining operands associated with said node is minimised by said modification, wherein in response to at least one update to at least one of said nodes: a further visible region is determined for each node using any modified compositing expressions, depending on those said objects which contribute to said further visible regions; an update region is formed as the union of said visible regions and said further visible regions for those nodes affected by said at least one update; a next image of said series is created by rendering an image space in which said graphical objects are defined (only) in an area substantially equal to said update region; o 38. The apparatus according to claim 37, further including means for compositing those said objects which contribute to said update region. 39. The apparatus according to claim 37, wherein said union comprises previously stored visible regions for any objects removed from said image. 580809.doc The apparatus according to claim 37, wherein said union comprises visible regions for any objects that are added to said image. 41. A program for creating a series of images, said images being formed by rendering and compositing a plurality of graphical objects, each of said objects having a predetermined boundary outline, said program comprising code for performing the following steps determining a visible region for each graphical object, wherein said visible region is defined by a region outline substantially following at least one of said predetermined outlines or parts thereof, depending on those said objects which contribute to said region, said contribution being determined on the basis of clipping and/or occlusion of said graphical object by any other graphical object, and/or on the basis of the compositing of said graphical object with any other graphical object; storing said visible regions; rendering a first image of said series by compositing said graphical objects; o in response to at least one change to at least one of said objects; determining a further visible region for each graphical object, wherein said further visible region is defined by a region outline substantially following at least one of said predetermined outlines or parts thereof, depending on those said objects which contribute to said further visible regions; (ii) forming an update region as the union of said stored visible regions and said further visible regions for those objects affected by said at least one change; (iii) creating a further image of said series by rendering a portion of said further image, said portion being substantially equal to said update region; and 580809.doc -71 repeating step for further changes to at least one of said objects. 42. The program according to claim 41, further comprising code for compositing those said objects which contribute to said further image within said update region. 43. The program according to any one of claims 41 or 42, wherein said union comprises previously stored visible regions for any objects removed from said image. 44. The program according to any one of claims 41 or 42, wherein said union comprises visible regions for any objects that are added to said image. The program according to any one of claims 41 to 44, wherein said graphical objects are composited according to a hierarchical structure representing a compositing expression, said hierarchical structure including a plurality of nodes each representing an operand of said compositing expression or an operator for combining said operands, wherein each of said graphical objects is represented by at least one of said operands. 0 0* 46. A program for creating a series of images, said images being formed by rendering and compositing a plurality of graphical objects to be composited according to a hierarchical data structure representing a compositing expression, each of said objects having a predetermined boundary outline, said hierarchical data structure including a plurality of nodes each representing an operand of said compositing expression or an operator for combining said operands, wherein each of said graphical objects is represented by at least one of said operands, said program comprising code for performing the following steps: 580809.doc -72- determining a visible region for each node, wherein said visible region is defined by a region outline substantially following at least one of said predetermined outlines or parts thereof, depending oh those said objects which contribute to said region, said contribution being determined on the basis of the compositing of said graphical object with any other graphical object, storing said visible regions; rendering a first image of said series by compositing said graphical objects according to said hierarchical structure; in response to at least one change to at least one of said nodes; determining a further visible region for each node, wherein said further visible region is defined by a region outline substantially following at least *o one of said predetermined outlines or parts thereof, depending on those said objects which contribute to said further visible regions; o (ii) forming an update region as the union of said stored visible regions i 15 and said further visible regions for those nodes affected by said at least one change; (iii) creating a further image of said series by rendering a portion of said i further image, said portion being substantially equal to said update region; and 9 repeating step for further changes to at least one of said nodes. 47. The program according to claim 46, wherein said contribution is determined on the basis of clipping of said graphical object by any other graphical object and/or on the basis of the compositing of said graphical object with any other graphical object; 48. The program according to any one of claims 46 and 47, further comprising code for compositing those said objects which contribute to said update region. 580809.doc -73- 49. The program according to any one of claims 46, 47 and 48, wherein said union also includes previously stored visible regions for any nodes removed from said image. The program according to any one of claims 46, 47 and 48, wherein said union also includes visible regions for any nodes that are added to said image. 51. The program according to claim 50, wherein said union also includes new and previously stored visible regions for nodes affected by said at least one change and for which operand independence does not apply. .00• 52. The program according to claim 50, wherein said union also includes visible :regions for any nodes representing operands which were added to said image and which are directly associated with an operation for which operand independence applies. t OV, o.. to 15 53. The program according to any one of claims 51 and 52, wherein said previously stored visible regions are each associated with a corresponding parent node. 6, 54. The program according to any one of claims 41 to 53, wherein said program is stored in a memory medium of an apparatus and is executed by a processor of said 666666 apparatus. A method of creating a series of images, said images being formed by rendering and compositing a plurality of graphical objects, each of said objects having a predetermined boundary outline, said method being substantially as herein before described with reference to anyone of the embodiments as illustrated with reference to Figs. 1 to 26. 580809.doc -74- 56. An apparatus for creating a series of images, said images being formed by rendering and compositing a plurality of graphical objects, each of said objects having a predetermined boundary outline, said apparatus being substantially as herein before described with reference to anyone of the embodiments as illustrated with reference to Figs. 1 to 26. 57. A program for creating a series of images, said images being formed by rendering and compositing a plurality of graphical objects, each of said objects having a predetermined boundary outline, said program being substantially as herein before described with reference to anyone of the embodiments as illustrated with reference to Figs. 1 to 26. 0 *00 DATED this Nineteenth Day of December 2001 15 Canon Kabushiki Kaisha Patent Attorneys for the Applicant SPRUSON FERGUSON 00: 580809.doc