AU767407B2 - Remote graphics display rendering - Google Patents

Description

S&F Ref: 546904

AUSTRALIA

PATENTS ACT 1990 COMPLETE SPECIFICATION FOR A STANDARD PATENT

ORIGINAL

Name and Address of Applicant: Actual Inventor(s): Address for Service: Canon Kabushiki Kaisha 30-2, Shimomaruko 3-chome, Ohta-ku Tokyo 146 Japan Anthony Ronald Allen Spruson Ferguson St Martins Tower,Level 31 Market Street Sydney NSW 2000 Remote Graphics Display Rendering Invention Title: ASSOCIATED PROVISIONAL APPLICATION DETAILS [33] Country [31] Applic. No(s) AU PQ7361 AU PQ7291 [32] Application Date 08 May 2000 04 May 2000 The following statement is a full description of this invention, including the best method of performing it known to me/us:- 5815c REMOTE GRAPHICS DISPLAY RENDERING Field of the Invention The present invention relates to the creation of computer-generated images in a remote location with respect to the host computer that generates the images, and, in particular, relates to efficient process, apparatus, and system for creating an image made up by compositing multiple components and displaying the image on a remote display.

The present invention also relates to the capture and display of real-time video (i.e.

dynamic images) on a remote terminal.

Background Many situations occur where a computer display terminal is separated from the computer that generates the images. This can occur for reasons of portability, convenience, space, environmental concerns or aesthetic concerns. As a result, if the r display is required to be of high-resolution colour then a huge amount of data must be ooo o transmitted from the computer to the display screen to display an image. Low-cost wire connected technologies have a typical maximum data rate of under 10M bits per second and wireless links have an even smaller data rate. However, to transmit an image of 640 x 480 pixels with 16-bit colour and a frame rate of 20 frames per second requires almost 100M bits per second if every pixel is transmitted for each frame.

There are a number of conventional methods for reducing the amount of information transmitted to render an image remotely. For example, the Intemrnet uses a mixture of coded commands in the form of HTML (Hyper Text Markup Language) and compressed images (eg. JPEG or GIF) to convey complex screens. However, this method is not suitable for full screen high-quality imaging to a low-cost remote display since the image compression compromises the quality of the images. Further, substantial processing power is required within the remote display.

Conventional methods for rendering images at a remote graphics terminal are particularly inefficient in situations where the final image is composed of numerous, possibly overlapping and semi-transparent graphical primitive components such as text, static images, geometric shapes and lines, and video images. Computer generated images are typically made up of many differing graphical primitive components or graphical elements which are rendered and composited together to construct a final image. In recent times, an "opacity channel" (also known as a "matte", an "alpha channel", or simply "opacity") has been commonly used. The opacity channel contains information regarding the transparent nature of each element. The opacity channel is stored alongside 546904.doc -2each instance of a colour, so that, for example, a pixel-based image with opacity stores an opacity value as part of the representation of each pixel. An element without explicit opacity channel information is typically understood to be fully opaque within some defined bounds of the element, and assumed to be completely transparent outside those bounds.

An expression tree offers a systematic means for representating an image in terms of the constituent elements of the image and which facilitates later rendering.

Expression trees typically comprise a plurality of nodes including leaf nodes, unary nodes and binary nodes. Nodes of higher degree, or of alternative definition may also be used.

A leaf node, being the outer most node of an expression tree, has no descendent nodes and represents a primitive constituent of an image. Unary nodes represent an operation which modifies the pixel data coming out of the part of the tree below the unary operator.

Unary nodes include such operations as colour conversions, convolutions (blurring etc) and operations such as red-eye removal. A binary node typically branches to left and right subtrees, wherein each subtree is itself an expression tree comprising at least one leaf node. Binary nodes represent an operation which combines the pixel data of the two children of the node to form a single result. For example, a binary node may be one of the standard "compositing operators" such as OVER, IN, OUT, ATOP and alpha-XOR, examples of which and other are seen in Fig. 20 Several of the above types of nodes may be combined to form a compositing tree. An example of this is shown in Fig. 1. The result of the left-hand side of the compositing tree may be interpreted as a colour converted image being clipped to spline boundaries. This construct is composited with a second image.

Although the non-transparent area of a graphical element may of itself be of a certain size, it need not be entirely visible in a final image, or only a portion of the element may have an effect on the final image. For example, assume an image of a certain size is to be displayed on a display. If the image is positioned so that only the top left corner of the image is displayed by the display device, the remainder of the image is not displayed. The final image as displayed on the display device thus comprises the visible portion of the image, and the invisible portion in such a case need not be rendered.

Another way in which only a portion of an element may have an effect is when the portion is obscured by another element. For example, a final image to be displayed (or rendered) may comprise one or more opaque graphical elements, some of which obscure other graphical elements. Hence, the obscured elements have no effect on the final image.

546904.doc A conventional compositing model considers each node to be conceptually infinite in extent. Therefore, to create the final image, a conventional system would apply a compositing equation at every pixel of the output image. Interactive frame rates of the order greater than 15 frames per second can be achieved by relatively brute-force approaches in most current systems, because the actual pixel operations are quite simple and can be highly optimised. This highly optimised code is fast enough to produce acceptable frame rates without requiring complex code. However, this is certainly not true in a compositing environment.

The per-pixel cost of compositing is quite high. This is because typically an image is rendered in 24-bit colour in addition to an 8-bit alpha channel, thus giving 32 bits per pixel. Each compositing operator has to deal with each of the four channels.

Therefore, the approach of completely generating every pixel of every required frame when needed is inefficient, because the per-pixel cost is too high.

i Problems arise with prior art methods of rendering an image on a remote graphical display when rendering graphical elements which include transparent and partially-transparent areas. Such methods typically do not handle the full range of compositing operators.

Many conventional methods for rendering images at a remote graphics terminal have excessive bandwidth requirements, poor image resolution or quality, require complex processing at the receiving end and do not retain the integrity of the components S of an image allowing the components to be added, removed or altered. Further, conventional methods for rendering images at a remote graphics terminal do not minimise the data required to update the remote image when one or more components are altered at 0 the host computer.

Summary of the Invention It is an element of the present invention to substantially overcome, or ameliorate, one or more of the deficiencies of the above mentioned methods by the provision of a method for creating an image made up by compositing multiple components.

According to one aspect of the present invention there is provided a method of rendering at least a first and second image on a graphical display that is remotely spaced from a computer, said computer being connected to said display via a communications link, said second image being formed by rendering and compositing at least a plurality of graphical primitives according to a compositing expression, said method comprising the steps of: capturing said first image on a camera connected to said display; 546904.doc determining a bounding area within which said first image is to be rendered; transmitting a description of said bounding area to said computer; determining at least one region for one or more of said graphical primitives, said region representing at least a portion of a corresponding primitive outside of which said corresponding primitive is transparent; manipulating said regions to determine a plurality of further regions, said further regions being dependent on said description of said bounding area, wherein each said further region comprises at least one contributing primitive; encoding said further regions into at least one rendering instruction substantially representing said contributing primitives; rendering said first image within said bounding area utilising said graphical display; and transmitting said rendering instructions via said communications link to said graphical display in order to render said second image.

According to another aspect of the present invention there is provided a method 0 of creating an image on a graphical display that is remotely spaced from a computer, said computer being connected to said display via a communications link, said image being .formed by rendering and compositing at least a plurality of graphical primitives according to a compositing expression, said method comprising the steps of: 0: 0. 20 determining at least one region for one or more of said graphical primitives, said region representing at least a portion of a corresponding primitive outside of which said corresponding primitive is transparent; manipulating said regions to determine a plurality of further regions, wherein each said further region comprises at least one contributing primitive; encoding said further regions into at least one rendering instruction substantially representing said contributing primitives; and transmitting said rendering instructions via said communications link to said graphical display in order to create said image.

According to still another aspect of the present invention there is provided a method of creating an image on a graphical display that is remotely spaced from a computer associated with said display, said computer being connected to said display via a communications link, said image being formed by rendering and compositing at least a plurality of graphical primitives, each said primitive having a predetermined outline, said method comprising the steps of: dividing a space in which said outlines are defined into a plurality regions, each 546904.doc said region being defined by at least one region outline substantially following at least one of said predetermined outlines or parts thereof; manipulating said regions to determine a plurality of further regions, wherein each said further region has a corresponding compositing expression; classifying said further regions according to at least one attribute of said graphical primitives within said further regions; modifying each said corresponding compositing expression according to a classification of each said further region to form an augmented compositing expression for each said further region; and encoding said further regions, according to said augmented compositing expressions, into a plurality of rendering instructions; and •.transmitting said plurality of rendering instructions via said communications link to said graphical display in order to create said image.

::According to still another aspect of the present invention there is provided a method of creating a series of images on a graphical display that is remotely spaced from a computer associated with said display, said computer being connected to said display via a communications link, each member of said series being related to a preceding member, said images being formed by rendering a plurality of graphical primitives to be composited according to a hierarchical structure representing a compositing expression, 20 said hierarchical structure including a plurality of nodes each representing a component 2 of at least one of said images, each of said primitives having a predetermined outline, said method comprising the steps of: for each said node: dividing a component image space in which said outlines are defined into at least one mutually exclusive region, each said region being related to at least one graphical primitive; (ii) examining each said region to determine those primitives that contribute to the region; creating intemrnodal dependency information identifying those said regions that will be affected by a change in any one of said regions; encoding all of those primitives that contribute to said regions, substantially according to said hierarchical structure, into a plurality of rendering instructions representing said first image; transmitting said plurality of rendering instructions via said communications link to said graphical display in order to create said first image; 546904.doc -6in response to at least one change to at least one of said nodes; examining said intemodal dependency information to identify those of said regions affected by said at least one change; (ii) for each node with affected regions, updating the corresponding identified regions and incorporating into said node those (any) new regions arising from the change and/or removing any of said regions that are no longer relevant; (iii) updating said intemrnodal dependency information to reflect changes to said hierarchical structure; (iv) examining each said identified region to determine those primitives that contribute to the identified region; encoding all of those primitives that contribute to said identified regions, substantially according to said hierarchical structure, into a plurality of rendering instructions representing said identified regions; (vi) transmitting said rendering instructions representing said identified regions, via said communications link, to said graphical display in order to render a further image of said series; and repeating step for further changes to at least one of said nodes.

According to still another aspect of the present invention there is provided a method of creating a series of images on a graphical display that is remotely spaced from 20 a computer associated with said display, said computer being connected to said display via a communications link, each member of said series being related to a preceding member, said images being formed by rendering a plurality of graphical primitives to be •composited according to a compositing expression, said method comprising the steps of: dividing a component image space in which said outlines are defined into at least one mutually exclusive region, each said region being related to at least one graphical primitive; examining each said region to determine those primitives that contribute to the region; encoding all of those primitives that contribute to said regions into a plurality of rendering instructions representing a first image; transmitting said plurality of rendering instructions, via said communications link, to said graphical display in order to create said first image; in response to at least one change to at least one of said nodes; identifying those of said regions affected by said at least one change; 546904.doc -7- (ii) for each node with affected regions, updating the corresponding identified regions and incorporating into said node those (any) new regions arising from the change and/or removing any of said regions that are no longer relevant; (iii) examining each said identified region to determine those primitives that contribute to the identified region; (iv) encoding all of those primitives that contribute to said identified regions into a plurality of rendering instructions representing said identified regions; (vi) transmitting said rendering instructions representing said identified regions, via said communications link, to said graphical display in order to render a further image of said series; and repeating step for further changes to at least one of said nodes.

According to still another aspect of the present invention there is provided an @ooo apparatus for creating an image on a graphical display that is remotely spaced from a computer, said computer being connected to said display via a communications link, said image being formed by rendering and compositing at least a plurality of graphical primitives according to a compositing expression, said apparatus comprising: means for determining at least one region for one or more of said graphical 0 primitives, said region representing at least a portion of a corresponding primitive outside of which said corresponding primitive is transparent; 20 means for manipulating said regions to determine a plurality of further regions, wherein each said further region comprises at least one contributing primitive; means for encoding said further regions into at least one rendering instruction substantially representing said contributing primitives; means for transmitting said rendering instructions via said communications link to said graphical display in order to create said image; means for capturing a further image on a camera connected to said display; means for displaying said further image and a still further image on said display.

According to still another aspect of the present invention there is provided anapparatus for creating an image on a graphical display that is remotely spaced from a computer associated with said display, said computer being connected to said display via a communications link, said image being formed by rendering and compositing at least a plurality of graphical primitives, each said primitive having a predetermined outline, said apparatus comprising: means for dividing a space in which said outlines are defined into a plurality regions, each said region being defined by at least one region outline substantially 546904.doc following at least one of said predetermined outlines or parts thereof; means for manipulating said regions to determine a plurality of further regions, wherein each said further region has a corresponding compositing expression; means for classifying said further regions according to at least one attribute of said graphical primitives within said further regions; means for modifying each said corresponding compositing expression according to a classification of each said further region to form an augmented compositing expression for each said further region; and means for encoding said further regions, according to said augmented compositing expressions, into a plurality of rendering instructions; and means for transmitting said plurality of rendering instructions via said communications link to said graphical display in order to create said image.

According to still another aspect of the present invention there is provided an apparatus for creating an image on a graphical display that is remotely spaced from a 15 computer associated with said display, said computer being connected to said display via a communications link, said image being formed by rendering and compositing at least a •plurality of graphical primitives, each said primitive having a predetermined outline, said apparatus comprising: means for dividing a space in which said outlines are defined into a plurality regions, each said region being defined by at least one region outline substantially following at least one of said predetermined outlines or parts thereof; •means for manipulating said regions to determine a plurality of further regions, .i wherein each said further region has a corresponding compositing expression; means for classifying said further regions according to at least one attribute of said graphical primitives within said further regions; means for modifying each said corresponding compositing expression according to a classification of each said further region to form an augmented compositing expression for each said further region; and means for encoding said further regions, according to said augmented compositing expressions, into a plurality of rendering instructions; and means for transmitting said plurality of rendering instructions via said communications link to said graphical display in order to create said image.

According to still another aspect of the present invention there is provided an apparatus for creating a series of images on a graphical display that is remotely spaced from a computer associated with said display, said computer being connected to said 546904.doc -9display via a communications link, each member of said series being related to a preceding member, said images being formed by rendering a plurality of graphical primitives to be composited according to a hierarchical structure representing a compositing expression, said hierarchical structure including a plurality of nodes each representing a component of at least one of said images, each of said primitives having a predetermined outline, said apparatus comprising means for performing the following steps: for each said node: dividing a component image space in which said outlines are defined into at least one mutually exclusive region, each said region being related to at least one graphical primitive; (ii) examining each said region to determine those primitives that 0*contribute to the region; creating internodal dependency information identifying those said regions that will be affected by a change in any one of said regions; encoding all of those primitives that contribute to said regions, substantially according to said hierarchical structure, into a plurality of rendering instructions representing said first image; transmitting said plurality of rendering instructions via said communications link to said graphical display in order to create said first image; in response to at least one change to at least one of said nodes; examining said intemodal dependency information to identify i those of said regions affected by said at least one change; (ii) for each node with affected regions, updating the corresponding identified regions and incorporating into said node those (any) new regions arising from the change and/or removing any of said regions that are no longer relevant; (iii) updating said internodal dependency information to reflect changes to said hierarchical structure; (iv) examining each said identified region to determine those primitives that contribute to the identified region; encoding all of those primitives that contribute to said identified regions, substantially according to said hierarchical structure, into a plurality of rendering instructions representing said identified regions; 546904.doc (vi) transmitting said rendering instructions representing said identified regions, via said communications link, to said graphical display in order to render a further image of said series; and repeating step for further changes to at least one of said nodes.

According to still another aspect of the present invention there is provided a method of rendering at least a first and second image on a graphical display that is remotely spaced from a computer, said computer being connected to said display via a communications link, said second image being formed by rendering and compositing at least a plurality of graphical primitives according to a compositing expression, said method comprising the steps of: capturing said first image on a camera connected to said display; determining a bounding area within which said first image is to be rendered; transmitting a description of said bounding area to said computer; dividing a space in which the second image is to be rendered into at least one mutually exclusive region depending on said bounding area, and for each of the mutually exclusive regions performing the following steps to determining at least one further region for one or more of said graphical primitives falling within said further region representing at least a portion of a corresponding primitive outside of which said corresponding primitive is transparent; manipulating said further regions to determine a plurality of still further regions, wherein each said still further region comprises at least one contributing primitive; (d)(iii) encoding said still further regions into at least one rendering oooo.

546904.doc -11instruction substantially representing said contributing primitives; transmitting said rendering instructions via said communications link to said graphical display in order to render a portion of said second image within said mutually exclusive region; and rendering said first image within said bounding area utilising said graphical display.

Brief Description of the Drawings A number of arrangements of the present invention will now be described with reference to the following drawings: Fig. 1 is an example ofa compositing tree; Fig. 2 illustrates an image containing a number of overlapping elements and the corresponding compositing tree; Fig. 3 shows the image of Fig. 2 illustrating the different regions which exist in the image and listing the compositing expression which would be used to generate the pixel data for each region; Fig. 4 is the image of Fig. 3, illustrating the compositing operations after being olo optimised; Fig. 5 illustrates the result of combining two region descriptions using a Union operation; Fig. 6 illustrates the result of combining two region descriptions using an Intersection operation; Fig. 7 illustrates the result of combining two region descriptions using a Difference operation; Figs. 8A to 8D illustrate the steps involved in combining two region groups using an Over operation; Fig. 9 illustrates an image and compositing tree; Fig. 10 illustrates another image and compositing tree; Fig. 11 illustrates the effect on the image of Fig. 10 of moving region A; Fig. 12 illustrates a still further image and compositing tree; Fig. 13 illustrates the effect on the image of Fig. 12 of moving region A; Fig. 14 illustrates the effect on the image of Fig. 12 of moving region B; Fig. 15 illustrates those nodes in a compositing tree which need to have their region groups updated if leaf nodes B and H change; Fig. 16 illustrates a region and the x and y co-ordinates of the region; Fig. 17 illustrates two regions and their x and y co-ordinates; 546904.doc 12- Fig. 18 illustrates a still further image and compositing tree; Fig. 19 illustrates an apparatus upon which the methods described herein can be implemented; Fig. 20 depicts the result of a variety of compositing operators; Fig. 21 illustrates regions formed by combining two circles with non-gridaligned regions; Fig. 22 illustrates improved regions formed by combining two circles with gridaligned regions; Fig. 23 is a flowchart showing a method of rendering a digital image on a graphical display that is remotely positioned with respect to a host computer associated with the display; Fig. 24 is a flowchart showing a method of combining regions groups; Fig. 25 is a flowchart showing a method of rendering a series of images on a graphical display that is remotely positioned with respect to a host computer associated with the display; Fig. 26 is a flowchart showing a method of updating an image rendered using the method ofFig. 23; Fig. 27 shows an overview of an information system configured in accordance with the methods described herein; Fig. 28 is a schematic block diagram showing the overall structure of the remote multimedia terminal, of Fig. 27, comprising a digital video camera; Fig. 29 illustrates how a camera image in the Local Frame Buffer of the remote *multimedia terminal of Fig. 27 is transferred to a smaller rectangular region on the local display of the multimedia terminal; Fig. 30 is a flow chart showing a method of rendering a digital image captured by a video camera on the multimedia terminal of Fig. 28; and Fig. 31 shows the local display of the multimedia terminal of Fig. 27 with a rectangular region within which a captured video image can be displayed.

Detailed Description 1.1 General Overview Where reference is made in any one or more of the accompanying drawings to steps and/or features, which have the same reference numerals, those steps and/or features have for the purposes of this description the same function(s) or operation(s), unless the contrary intention appears.

Some portions of the detailed description which follows are explicitly or 546904.doc -13implicitly presented in terms of algorithms and symbolic representations of operations on data within a computer memory (ie. computer code). These algorithmic descriptions and representations are the means used by those skilled in the data processing arts to most effectively convey the substance of their work to others skilled in the art. An algorithm is here, and generally, conceived to be a self-consistent sequence of steps leading to a desired result. The steps are those requiring physical manipulations of physical quantities. Usually, though not necessarily, these quantities take the form of electrical or magnetic signals capable of being stored, transferred, combined, compared, and otherwise manipulated. It has proven convenient at times, principally for reasons of common usage, to refer to these signals as bits, values, elements, symbols, characters, terms, numbers, or the like.

9*99 It should be borne in mind, however, that all of these and similar terms are to be associated with the appropriate physical quantities. Unless specifically stated otherwise as apparent from the following discussions, it is appreciated that throughout the present description, discussions utilising terms such as "processing", "computing", "generating", "creating", "operating", "communicating", "rendering", "providing", and "linking" or the i like, refer to the action and processes of a computer system, or similar electronic device, that manipulates and transforms data represented as physical (electronic) quantities within the computer system's registers and memories into other data similarly represented as physical quantities within the computer system memories or registers or other such information storage, transmission or display devices.

A number of methods of rendering a digital image on a graphical display that is 9e9o remotely positioned with respect to a host computer associated with the display are described below. In accordance with the methods, proxy information is determined, which substantially represents non-overlapping (ie. mutually exclusive) regions of an image that need to be rendered to construct an image (ie. in the case of static rendering) or re-rendered to re-construct an image (ie. in the case of dynamic rendering) if the image is updated (ie. changes) from one frame to the next. The methods described herein ensure that the image rendered on a remote graphical display corresponds with an update to the image by only conveying the information represented by the proxies to the display.

Fig. 27 shows an overview of an information system 2700 configured in accordance with the methods described herein. The information system 2700 comprises a remote multimedia terminal 2701 connected to a host computer 2702, preferably via a Universal Serial Bus (USB) data connection 2703. The remote multimedia terminal includes a graphical display. Alternatively, other connection mechanisms can be used to 546904.doc -14connect the remote multimedia terminal 2701 with the host computer 2702. For example, an Ethernet connection or a wireless communications link radio, infra-red, etc) can be used to connect the remote multimedia terminal 2701 with the host computer 2702.

However, a characteristic of all of these communications links is that they are band-width limited. As a result, the speed of data communications and display screen updates, can be greatly affected if the band-width limit of the communication link is reached or exceeded during communications between the remote multimedia terminal 2701 and the host computer 2702.

The remote multimedia terminal 2701 can include human interface devices 2704 and drivers 2705. Examples of the human interface devices 2704 include a speaker, microphone, touch panel and buttons. The drivers 2705 interface the devices 2704 to a data stream transported over the USB data connection 2703. The host computer 2702 can run application programs 2707 that provide a wide range of home convenience functions.

The host computer 2702 can include drivers 2706 to interface the data stream with the application programs 2707 household calendars, reminders, household messaging, rosters, shopping lists and world-wide-web queries) running on the host computer 2702.

~The remote multimedia terminal 2701 is a peripheral device which is preferably configured to be located where the typical household calender, message pad and/or shopping list would normally be located. The information system 2700 can preferably provide family messaging, calenders, shopping lists, radio, phone, clock, and other features.

The remote multimedia terminal 2701 communicates with the host computer 2702 via the USB data connection 2703. The system 2700 is preferably configured so that the user can input data or a command, using the remote multimedia terminal 2701.

For example, the user can calculate the total sum of outgoing bills for the month. The user selects a calculator function button which is pre-configured on the remote multimedia terminal 2701 and a request is sent to the host computer 2702. The host computer 2702 receives the request and passes it to software. The host computer 2702 activates a calculator application and draws a screen update. Subsequently, the graphical display screen on the remote multimedia terminal 2701 is updated. The user then uses the calculator application which is substantially executing in the computer 2702 while being at the location of the remote multimedia terminal 2701.

It is also advantageous to include a miniature video camera on the remote multimedia terminal 2701 to extend the range of applications the terminal is capable of performing. For example, the use of miniature video cameras using CCD technology is 546904.doc becoming increasingly common for applications such as video phones and web cameras.

In such applications it is usual to provide a local image representing the image being transmitted by the camera so that the local user knows what the transmitted image looks like. The image may be displayed on the remote multimedia terminal 2701 either by filling the full screen of the display or in a rectangular window occupying only part of the screen. For maximum efficiency and minimal latency this should be performed by the remote multimedia terminal 2701 itself without intervention from the host Personal Computer 2702 and therefore should function even if the host Personal Computer is disconnected.

Fig. 28 is a schematic block diagram showing the remote multimedia terminal 2701 in more detail. As shown in Fig. 28, the remote multimedia terminal 2701 comprises a digital video camera 2803. The remote multimedia terminal 2701 also comprises a module 2805 and a local display 2807. The module 2805 can include at least one central processor unit (CPU) 2809, read only memory (ROM) 2811, random access memory (RAM) 2813, user controls 2815 connected to buttons 2817 or touch screen circuitry (not illustrated), a communication interface 2819, a video controller 2821 and a display controller 2823. The communication interface 2819 is used by the module 2701 for communicating to and from the host personal computer 2702 via the USB data connection 2703. The components 2803 to 2823 of the module 2701 typically 0 20 communicate via an interconnected bus 2831, in a manner known to those in the relevant art.

The RAM 2813 includes a general purpose RAM 2825, a local frame buffer 2827, and a local display buffer 2829. The local frame buffer 2827 receives an input .o S•from the video controller 2821, which is connected to the video camera 2803.

Fig. 23 is a flowchart showing a method of rendering a digital image on a graphical display 2807) that is remotely positioned with respect to a host computer 2702) associated with the display. The host computer is connected to the graphical display via a bandwidth limited communications link as discussed above. The image is preferably rendered by compositing a plurality of graphical primitive components (hereinafter "primitive components") according to a hierarchical expression tree representing a compositing expression for the image. The hierarchical expression tree includes a plurality of nodes each representing at least a primitive component of the image or an operation for combining sub-expressions of the compositing expression.

Further, each primitive component has a predetermined boundary outline within the space in which the image is defined and can have one or more attributes line style, colour, 546904.doc -16opacity, surface texture etc) associated with a particular component.

The process of Fig. 23 begins at step 2301, where a space in which the predetermined outlines are defined is divided into a number of mutually exclusive regions. Each of the mutually exclusive regions is defined by at least one of the predetermined boundary outlines or parts thereof. As will be described below, the regions are substantially formed by segments of a grid which encompasses the space in which the predetermined outlines are defined. At the next step 2303, for each compositing operation which combines one or more primitive components, the regions are manipulated to determine a number of further regions. Each of the further regions has a corresponding compositing expression. The process of dividing the space into a number of mutually exclusive regions and manipulating those regions is described in detail particularly with reference to section 2.3 below. Section 2.3 includes two pseudocode listings which describe steps 2301 and 2303 for the "OVER" and "IN" compositing :operations.

The process continues at step 2305, where the further regions are classified according to attributes of the primitives that fall within the further regions. As will be described in more detail later in this document the further regions are classified according to whether the primitives are wholly opaque, wholly transparent or otherwise (herewith deemed "ordinary"). At the next step 2307, each of the corresponding compositing 20 expressions are modified according to a classification of each of the further regions. The modifications form an augmented compositing expression for each of the further regions.

The process of classifying the further regions and modifying each of the corresponding compositing expressions is described in detail particularly with reference to section 2.4 below. Section 2.4 includes two pseudocode listings which describe steps 2305 and 2207 for the "OVER" and "IN" compositing operations. The further regions generated at step 2303, substantially represent the visible portions of the image.

At the next step 2309, for one or more of the further regions, the region outline and any attributes associated with the further region are encoded into one or more rendering instructions that completely define the operations needed to render the image defined by the further regions. The rendering instructions are determined in accordance with the augmented compositing expression for each of the further regions. As will be described below, particularly with reference to section 5.0, the rendering instructions preferably comprise proxy information (ie. some means of caching the pixel data resulting from applying the operations specified by the augmented compositing expression at every pixel inside the further region). The proxy information preferably 546904.doc -17represents the next line to be updated, the first column to be updated and the number of pixels to follow, followed by the colour of each pixel. If groups of pixels are all of the one colour, then each individual pixel does not need to be transmitted, simply a proxy representing a particular mutually exclusive region and the colour to paint the region need to be transmitted. Further, where the attributes of two or more further regions are the same, then preferably one set of attributes referencing the two or more further regions need to be transmitted. In accordance with one example of the methods, the proxy information that represents the regions of an image to be rendered can be converted into a series of rectangles representing the further regions, as will be explained below particularly with reference to section 5.0. The process concludes at the next step 2311, where the rendering instructions are transmitted to the remote graphical display, via the bandwidth limited communications link, so that the image can be constructed. The process concludes when the entire image has been Fig. 29, this causes the image captured by the video camera 2821 to appear in a rectangle area 2904 of the local display 2807.

Fig. 26 is a flowchart showing a method of updating the image rendered at step 2311 in response to at least one change to the image. As will be explained below, the change to the image can include, but is not limited to, adding new primitive components, deleting new primitive components, changing primitive components or changing the operations for combining the components. The process begins at step 2601, where any of 20 the mutually exclusive regions, created at step 2301, that have been affected by the change to the image are identified. The method of identifying the mutually exclusive regions which have been affected by the changes will be explained in more detail later in this document. At the next step 2603, the affected regions are manipulated to determine a S•number of still further regions. Each of the still further regions have a region outline and have a corresponding compositing expression. The process continues at the next step 2605, where the still further regions are classified according to attributes of the primitives that fall within the still further regions. As above, the still further regions are classified according to whether the primitives are wholly opaque, wholly transparent or ordinary.

At the next step 2607, each of the compositing expressions that correspond to the still further regions are modified according to the classifications determined at step 2605 to form an augmented compositing expression for each of the further regions. At the next step 2609, for one or more of the still further regions, the outline defining the still further regions and any attributes associated with the primitives that fall within the still further regions are encoded into rendering instructions in accordance with the augmented compositing expression for each of the further regions. At the next step 2611, the 546904.doc -18rendering instructions representing operations needed to render the still further regions are transmitted to the remote graphical display, via the bandwidth limited communications link, in order to reconstruct the first image to create a further image which has resulted from the changes to the image constructed at step 2311. Therefore, only regions of the image that have been changed are transmitted to the remote graphical display at step 2611.

A method of rendering a series of images on a graphical display that is remotely positioned with respect to a host computer associated with the display is also described below with particular reference to Fig. 1.1 Underlying Principles of the Methods described herein The basic shape of operands to compositing operators in most current rendering systems is the rectangle, regardless of the actual shape of the element being composited.

It is extremely easy to write an operator which composites within the intersection area of Itwo bounding boxes. However, as a bounding box typically does not accurately represent the actual bounds of a graphical element, this method results in a lot of unnecessary compositing of completely transparent pixels over completely transparent pixels.

Furthermore, when the typical make-up of a composition is examined, it can be noticed that areas of many of the elements are completely opaque. This opaqueness can be exploited during the compositing operation. However, these areas of complete 20 opaqueness are usually non-rectangular and so are difficult to exploit using compositing arguments described by bounding boxes. If irregular regions are used for exploiting opaque elements when compositing, then these regions can be combined in to determine where compositing should occur. Furthermore, if any such region is known to be fully transparent or fully opaque, further optimisations are possible.

Most current systems fail to exploit similarities in composition between one frame and the next. It is rare for everything to change from frame to frame and therefore large areas of a compositing tree will remain unchanged. An example of this is where a cartoon type character comprising multiple graphical elements is rendered on a display.

If, for example, the character spilt some paint on its shirt in the next frame, then it is not necessary to render the entire image again. For example, the head and legs of the character may remain the same. It is only necessary to render those components of the image that have been altered by the action. In this instance, the part of the shirt on which the paint has been spilt may be re-rendered to be the same colour as the paint, whilst the remainder of the character stays the same. Exploiting this principle can provide large efficiency improvements. If incremental changes are made to the compositing tree, then 546904.doc -19only a reduced amount of updating is necessary to affect the change.

Many current graphical systems use what is known as an immediate mode application program interface (API). This means that for each frame to be rendered, the complete set of rendering commands is sent to the API. However, sending the complete set of rendering commands is somewhat inefficient in a compositing environment, as typically, large sections of the compositing tree will be unchanged from one frame to the next, but would be completely re-rendered anyway in immediate mode. On the other hand, the methods described herein are considered by the present inventors to be best described as a retained mode API. Retained mode means that instead of providing the complete compositing tree on a per-frame basis, the user provides an initial compositing tree, and then modifies it on a per-frame basis to effect change. Changes which can be made to the tree include geometrically transforming part or all of the tree, modifying the tree structure (unlinking and linking subtrees), and modifying attributes (eg: color) of individual nodes. Note that such modifications may not necessarily mean that the tree structure, for example as seen in Fig. 1, will change where only the attributes of an individual node have been modified.

Some of the principles which contribute to the method of Fig. 23 are: The use of irregular regions to minimise per-pixel compositing. For example, if one graphical element is on top of another, then pixel compositing is only 20 needed inside the area where the two elements intersect. Having the ability to use irregular regions gives the ability to narrow down areas of interest much more accurately.

(ii) An assumption is made that in the transition from one frame to the next, only part of the tree will change. This can be exploited by caching away expensive-togenerate information regarding the composition so that it can be re-used from one frame to the next. Examples of expensive-to-generate information are regions of interest (boundaries of areas of intersection between elements etc); pixel data (representing expensive composites etc); and topological relationships between elements.

(iii) If an opaque element is composited with another element using the OVER operator, then the opaque element completely obscures what it is composited onto (ie. inside the opaque elements area). This is a very useful property since it means that no expensive pixel compositing is required to achieve the output pixel within the area of overlap. (The pixel value is the same as that at the equivalent spot on the opaque element). Opaque elements induce similar behaviour in most of the compositing operators. Therefore, the methods described herein attempts to exploit opaque areas as much as possible.

546904.doc Basic Static Rendering Static Rendering refers to the generation of a single image from a compositing tree. Some of the pixel compositing methods described herein will be explained using a static rendering example.

An example of a simple compositing tree which consists of leaf node elements and only using the "OVER" operator is shown in Fig. 2. Conventionally, each node is considered to be conceptually infinite in extent. One method to construct the final image is to apply the compositing equation OVER B) OVER C) OVER (A OVER at every pixel of the output image. However, this is quite an inefficient method.

A composition can generally be subdivided into a number of mutually exclusive irregular regions. The above compositing expression can be simplified independently within each region. In the example of Fig. 2, A, C and E represent opaque elements. B and D, on the other hand are partially transparent. Fig. 3 shows the different regions X (1-10) produced using the five elements which exist in the example, and the compositing expression which can be used to generate the pixel data for each specific region.

The compositing expressions provided in Fig. 3 make no attempt to exploit the properties of the opacity of the element. If these properties are used to simplify the compositing expressions for each region, the expressions of Fig. 4 are obtained resulting in a simplification of the rendering of regions 2, 3, 5, 6, 7, 8 and 9 compared with Fig. 3.

20 These simplified compositing expressions result in far fewer pixel compositing operations being performed to produce the final picture.

Fig. 4 represents the region subdivision for the root of the compositing tree.

However, every node in the compositing tree can itself be considered the root of a complete compositing tree. Therefore, every node in the compositing tree can have associated with it a group of regions which together represent the region subdivision of the subtree of which the node is the root. Region subdivision provides a convenient means of managing the complexity of a compositing tree and an efficient framework for caching expensive data.

Using the principles noted above, a compositing expression can be simplified dependent upon whether the graphical elements being composited are wholly opaque, wholly transparent or otherwise (ie. ordinary), as discussed above.

Table 1 shows how the compositing operations of Fig. 20 can be simplified when one or both operands are opaque or transparent.

546904.doc -21- TABLE 1 Expression As opacity Bs opacity Optimised AoverB Transparent Transparent Neither Transparent Ordinary B Transparent Opaque B Ordinary Transparent A Ordinary Ordinary AoverB Ordinary Opaque AoverB Opaque Transparent A Opaque Ordinary A Opaque Opaque A AroverB Transparent Transparent Neither Transparent Ordinary B Transparent Opaque B Ordinary Transparent A Ordinary Ordinary BoverA Ordinary Opaque B Opaque Transparent A Opaque Ordinary BoverA Opaque Opaque B AinB Transparent Transparent Neither Transparent Ordinary Neither Transparent Opaque Neither Ordinary Transparent Neither Ordinary Ordinary AinB Ordinary Opaque A Opaque Transparent Neither Opaque Ordinary AinB Opaque Opaque A ArinB Transparent Transparent Neither Transparent Ordinary Neither Transparent Opaque Neither Ordinary Transparent Neither 546904.doe 22 Ordinary Ordinary BinA Ordinary Opaque BinA Opaque Transparent Neither Opaque Ordinary B Opaque 0paque B AoutB Transparent Transparent Neither Transparent Ordinary Neither Transparent Opaque Neither Ordinary Transparent A Ordinary Ordinary AoutB Ordinary Opaque Neither Opaque Transparent A Opaque Ordinary AoutB Opaque Opaque Neither AroutB Transparent Transparent Neither Transparent Ordinary B Transparent Opaque B Ordinary Transparent Neither Ordinary Ordinary BoutA Ordinary Opaque BoutA Opaque Transparent Neither Opaque Ordinary Neither Opaque .Oaque Neither AatopB Transparent Transparent Neither Transparent Ordinary B Transparent Opaque B Ordinary Transparent Neither Ordinary Ordinary AatopB Ordinary Opaque AatopB Opaque Transparent Neither Opaque Ordinary AatopB Opaque Opaque A AratopB Transparent Transparent Neite 546904.doc -23r r I r r r r Transparent Ordinary Neither Transparent Opaque Neither Ordinary Transparent A Ordinary Ordinary BatopA Ordinary Opaque BatopA Opaque Transparent A Opaque Ordinary BatopA Opaque Opaque B AxorB Transparent Transparent Neither Transparent Ordinary B Transparent Opaque B Ordinary Transparent A Ordinary Ordinary AxorB Ordinary Opaque AxorB Opaque Transparent A Opaque Ordinary AxorB Opaque Opaque Neither 2.1 Basic Data Model Associated with every node in a compositing tree is a group of mutually exclusive regions which together represent the non-transparent area of the node. It should 5 be noted that the region descriptions that the methods described herein use are generally not pixel accurate. A region can in fact contain some transparent pixels. However, any point lying outside of all the regions at a node is certain to be transparent. The set of the mutually exclusive regions at a node is known as a region group. A leaf node region group can contain only one or two regions. The region group at the root of the tree may contain hundreds of regions. Each region in a region group contains the following basic data: A Region Description is a low-level representation of the boundaries of the region. The region descriptions of all the regions in a region group are mutually exclusive (ie. non-intersecting). However, the methods described herein are not limited to using axis-parallel (ie. every side parallel or perpendicular to a scan line of an output device) region descriptions. The methods allow region descriptions which more closely represent arbitrary shaped regions.

r 546904.doc 24 (ii) A Proxy is some means of caching the pixel data resulting from applying the operations specified by the compositing expression at every pixel inside the region description. A proxy can be as simple as a 24-bit colour bitmap, or something much more complicated (eg. a run-length encoded description). Fundamentally, a proxy represents pixel data in some way which makes it efficient to retrieve and use.

Every region group preferably contains a region description which is the union of all the region descriptions of the regions in the region group. The region description essentially represents the entire coverage of the region group.

2.2 Region Descriptions and Region Arithmetic The region arithmetic and data structure of the methods described herein has the following properties: The region arithmetic and data structure allow the representation of complex regions, including convex regions, concave regions and regions with holes. This is necessary so that a region will be reasonably able to follow the geometry of the graphic 15 element it represents; (ii) The region arithmetic and data structure is space efficient. In a complicated composition there will be many regions. For memory efficiency, it is therefore preferable that the cost of storing these regions is reasonably small; (iii) The region arithmetic can support basic set operations (eg. Union, Intersection and Difference); (iv) The above-noted basic set operations are efficient in terms of speed. In a complex compositing tree, it is possible that a large amount of region arithmetic will be undertaken. Poor region arithmetic can lead to the time taken by region arithmetic being greater than the time saved from the reduction in per-pixel compositing; The region description can be geometrically translated efficiently. In cases where a graphic element is translated, the graphics elements associated regions can then be translated quickly; and (vi) The region description and data structure allow for the quick comparison of two regions to determine if the regions are the same. It is not necessary to obtain any other statistics on their similarity, simple equality is all that is required.

Two conventional region description methods were considered and rejected for described methods. Firstly, polygons can be used to represent almost any element.

However, the disadvantage of using a polygon, is that the generality of a ploygon makes implementing the set operations slow and inefficient. Secondly, using quadtrees, set operations can be easy to implement and are quite efficient. In addition, they can 546904.doc 25 represent a wide variety of regions given sufficient granularity since all edges in a quadtree have to be axis-parallel. However, the disadvantage of quadtrees is that quadtrees must be aligned on the same grid. This means that it is impossible to simply translate a quadtree by an arbitrary amount. Unless that amount is a multiple of the underlying grid size, the quadtree will need to be recalculated from the element it describes (otherwise it will keep growing). Therefore, quadtrees are not suitable in application domains where geometric translation is a frequent operation.

The region description data structure of the described methods can be understood by assuming that along a vertical line every coordinate has a state which is one of either inside or outside the region. The data structure stores those y co-ordinates at which some change of state between inside and outside occurs. For each such y co-ordinate, the data contains spans of coordinates each of which toggles the state of every vertical line running through the data. Each span of x co-ordinates is called a run. The sequence of runs associated with a y co-ordinate is called a row. For example, the region of Fig. 16 S 15 can be described by the following: row y= 10: x= 10, x 100 row y= 100:x 10, x =100 Similarly, the regions of Fig. 17 could be described by the following: row y= 10 x 10, x =100 20 row y= 30 x 30, x row y 70 x 30, x row y 100 x +10, x 100 The data representing a region is preferably represented by an array of integer values.

S"There are two "special" values R_NEXT_I SY A beginning-of-row marker. Indicates that the next integer in the sequence will represent a y coordinate.

R_EOR Stands for End-of-Region. Indicates that the region description has finished.

All other values represent x or y coordinates. The x coordinates in a row represent runs. The first two co-ordinates represent a run, then the next two represent the next run and so on. Therefore, the x coordinates in a row should always be increasing.

Also, there should always be an even number of x-coordinates in a row. The region data 546904.doc -26stream for Fig. 17 is shown below.

R NEXT IS Y 10 10 100 R NEXT IS Y 30 30 R NEXT IS Y 70 30 R NEXT IS Y 100 10 100 R EOR The described methods can also contain the bounding box of the region, as this is useful in certain set operations.

As seen in Fig. 5, if two region descriptions are combined using a Union operation, then the resultant region description will describe an area in which either region description is active.

As seen in Fig. 6, if two region descriptions are combined using the Intersection operation, then the resultant region description will describe an area in which both the region descriptions are active.

S "If two region descriptions are combined using the Difference operation, then the resultant region will describe an area in which only the first region is active, as seen in t Fig. 7.

20 2.3 Constructing Region Groups: 2.3.1 Constructing Leaf Node Region Groups A region group for a leaf node typically contains one or more regions, which together fully contain the non-transparent area of a graphical element represented by a S"leaf node. Typically, the non-transparent area can be divided into regions where each region has some property that facilitates optimisation. For example, the non-transparent area of some graphical element can be divided into two regions, one fully opaque and the other with ordinary opacity. The above mentioned compositing optimisations can apply where the opaque region is composited.

Alternatively, a leaf node can be subdivided based on some other attribute. For example, a leaf node can be divided into two regions, one representing an area of constant colour, the other representing blended colour. Areas of constant colour can be composited more efficiently than areas with more general colour description.

2.3.1.1 Region Formation and Phasing When creating regions, it is not always beneficial that region boundaries follow graphical element boundaries precisely. What is important is that any property that 546904.doc -27facilitates optimisation is valid at all points within a region said to have that property. For example, an opaque circle can be covered exactly by one circular region which is classified as opaque, or by two approximate regions, one fully opaque octagonal region inscribed in the circle, and one annular octagonal region of ordinary opacity that includes the remainder of the circle plus some area exterior to the circle.

There is typically a trade-off between how closely region boundaries follow graphical element boundaries and the benefits obtained. If region boundaries follow element boundaries very closely, a lot of work is usually involved in creating the region boundaries and in performing intersections and differences of regions (the reasons for needing to perform such operations are explained in later sections). However, if region boundaries are too approximate, they can either include large areas that are outside the elements' boundaries, resulting in too much unnecessary compositing, or they can fail to include large areas where known properties lead to optimisation.

Region boundaries can be limited to sequences of horizontal and vertical 15 segments. In this instance, the typical segment size is chosen so that there is neither too much detail so that the region operations are overburdened, nor too much approximation to result in wasted compositing or insufficient optimisation.

One method to improve the efficiency of region operations is to select as many of the horizontal and vertical segments of substantially all region boundaries to be in 20 phase. In other words, the horizontal and vertical segments can be chosen from the horizontal and vertical lines of the same grid. The grid need not be regularly spaced, nor have the same spacing horizontally and vertically, although typically it will.

Choosing the horizontal and vertical segments from the horizontal and vertical lines of the same grid improves the efficiency of region operations by seeking to keep all region boundary detail to the level of detail contained in the underlying grid. Without constraining the majority of region boundary segments to a grid, region operators such as difference and intersection tend to produce a lot more fine detail. For example, in Fig.

21, two circles 901 and 902 are shown with respective regions 903 and 904 that are not grid-aligned. These circles are overlapped yielding difference regions 905 and 907, and intersection region 906. In Fig. 22, the same circles 901 and 902 have regions 913 and 914 that are aligned to grid 910. These circles are overlapped yielding difference regions 915 and 917 and intersection region 916. It can be seen in this example that the gridaligned regions yield less detailed results at the expense of slightly less efficient region coverage. Regions 905, 906 and 907 together contain a total of sixty segments, while regions 915, 916 and 917 together contain only fifty-two.

546904.doc .I -28- 2.3.2 Creating Binary Region Groups The region groups of binary nodes in a compositing tree on the other hand are the result of combining the region groups of their child nodes. It will now be explained how region groups can be combined to form new region groups. In this section, for simplicity only "OVER" and "IN" binary nodes will be dealt with. The operations required for binary nodes representing other compositing operators can easily be inferred from combining the "OVER" and "IN" cases in various ways.

For the sake of clarity, the method of combining region groups to form new region groups is initially described without reference to optimisation based properties such as opacity.

The following notation will be beneficial when considering binary region group creation: Notation

S

S

S

S.

55.5 S. S S S RG1 RG2

RG

RGl-+urgn RG2->urgn RG->urgn rgli rg2j rgli--rgn rg2j-+rgn rgli->proxy rg2j -proxy The region group of the binary node's left child The region group of the binary node's right child The region group of the binary node. It is this region group that is being initialised The region description representing the union of all RGl's region descriptions (RGl's coverage region).

The region description representing the union of all RG2's region descriptions (RG2's coverage region).

The union of all RG's region descriptions (to be initialised) (RG's coverage region) The current region in RG1 The current region in RG2 rgli's region description rg2j's region description rgli's proxy rg2j's proxy 546904.doc -29- 2.3.2.1 Constructing "OVER" Region Groups When constructing "OVER" region groups, only areas where the contributing region groups intersect need to be composited. Areas where one operand does not overlap the other involve no compositing. Fig. 24 shows a method of combining region groups. The process begins at step 2401, where the coverage region of the region group of the binary node that is being initialised (RG->urgn) is made equal to the union of the coverage regions of the binary nodes left child (RG1->urgn) and the right child (RG2->urgn) of the binary node. At the next step 2403, for each region rgi in RG1, the difference (diffrgn) between that region and the coverage region (RG2->urgn) of RG2 is calculated. If the difference (diff rgn) is non-empty then a new region with diff rgn as the region description of the new region is added to RG. The proxy of this new difference region can be the same as the proxy rgli. No compositing is required to generate it. The difference regions between regions of RG2 and coverage region of RG1 are similarly constructed and added to RG. The process continues at the next step 2405, 15 where the intersection (inter rgn) between each region rgli in RG1 and each region rg 2 j in RG2 is calculated. At the next step 2407, if the result of the intersection is non-empty, 0 then a new proxy (new_p) is constructd by compositing the proxy of rgli with the proxy ofrg 2 j using the over operation with the interrgn, at step 2408. The process continues at the next step 2409, where a new region is then added to RG with interrgn as the region 20 description of the new region and new_p as the proxy of the new region. If the result of the intersection at step 2407 is empty, the process concludes. The method of constructing "OVER" groups is described below using pseudo-code.

RG->urgn RG1-+urgn union RG2->urgn FOR i 0 TO number of regions in RG1 DO diff_rgn rgli->rgn difference RG2->urgn IF diff_rgn is non-empty THEN ADD to RG a new region with diff_rgn as its region description and rgli--proxy as its proxy. END IF FOR j 0 TO number of regions in RG2 DO interrgn rgli->rgn intersection rg2j->rgn IF inter_rgn is non-empty THEN create new proxy new_p initialised to OVER of rgli->proxy and rg2j->proxy inside inter rgn.

546904.doc ADD to RG a new region with inter_rgn as its region description and new_p as its proxy. END IF END DO END DO FOR j 0 TO number of regions in RG2 DO diffrgn rg2j-rgn difference RG1-urgn IF diffrgn is non-empty THEN ADD to RG a new region with diff_rgn as its region description and rg2j-proxy as its proxy. END IF END DO The regions added by the ADD operations marked with asterisks above are termed difference regions since their shape is the result of a difference operation. Such S regions are very cheap computationally since their proxies require no compositing. The only work involved is the administrative overhead of adding a new region to the region group and the cost of the difference operation itself. Preferably, a proxy is inherited from the region (in one of the child region groups) on which it is based. It can be seen that 20 proxies which originate low in the compositing tree can be propagated upwards towards the root with minimal overhead (both in terms of speed and memory) by the use of difference regions.

The regions added by the ADD operation marked with the plus are termed intersection regions since their shape is the result of an intersection operation. The proxies of such regions are more expensive to generate than difference regions since they involve per-pixel compositing operations to be done within the area defined by the intersection. The more fidelity granted the region descriptions, the greater the saving in pixel processing costs, at the cost of a greater administrative overhead (more complex regions require longer to intersect etc).

Figs. 8A to 8D show an example of combining "OVER" region groups using the above described method of combining region groups. The region group resulting from the combination contains 5 regions, 3 difference regions and 2 intersection regions.

Fig. 8A represents two region groups RG1 and RG2 which can be combined. RG1 contains two regions 81 and 82, whereas RG2 only contains a single region 83. As seen in Fig 8B, for each region in RG1, the region coverage of RG2 is subtracted from it. If 546904.doc -31 the resultant region is non-empty, the resultant region becomes a region in the new region group. In this example both regions 81 and 83 produce non-empty difference regions 84 and 85 respectively. For each region in RG2, the region coverage of RG1 is subtracted from it, as seen in Fig 8C. In this example difference region 86 is produced. Finally, every region in RG1 is intersected with every region in RG2, as seen in Fig 8D. Any non-empty region becomes a region in the new region group. In this example, regions 81 and 83 produce 87. Further, regions 82 and 83 produce 88.

2.3.2.2 Constructing "IN" Region Groups The properties of the "IN" operator lead to the fact that an "IN" binary region group only produces pixel data in the region of intersection between the two contributing region groups. Essentially, when compared to the method used for "OVER" region groups, only intersection regions are generated. Therefore, for each region rgli of RG1, and for each region rg2j of RG2 the intersection (interrgnij) between rgli and rg 2 j is calculated. If the intersection is non-empty then a new proxy (new_p) is created by 15 compositing the proxy of rgli with the proxy of rg2j using the "in" operation within interrgnij. A new region is then added to RG with inter_rgn as the region description and new_p as the proxy for the new region. The pseudocode describing the method of constructing "IN" region groups is provided below: RG-urgn RG1-urgn intersection RG2-urgn FOR i 0 TO number of regions in RG1 DO FOR j 0 TO number of regions in RG2 DO interrgn rg1->rgn intersection rg2j->rgn IF inter_rgn is non-empty THEN create new proxy new_p initialised to IN of rgli-proxy and rg2j-proxy inside inter rgn.

ADD to RG a new region with interrgn as its region description and new_p as its proxy. END IF END DO END DO The major difference between the "IN" and the "OVER" cases is that the "OVER" case generates difference regions while "IN" does not. In the example demonstrated by Figs. 8A to 8D, only new regions 97 and 98 would be generated, as 546904.doc -32a.

these are intersection regions. Difference regions 94, 95 and 96 would not be generated using "IN".

Using Table 2 below and the pseudocode examples of "OVER" and the relevant code for other compositing operators can be derived.

2.3.2.3 Constructing Region Groups of Other Compositing Operators Other compositing operators typically generate the same intersection regions as the "OVER" and "IN" cases do. However, they typically differ from one another (as indeed from "OVER" and in what difference regions they generate. This is dependent on the particular properties of each compositing operator. Table 2 summarises which difference regions are generated for some commonly used compositing operators.

TABLE 2 Compositing Operator Generate Diff Rgns from Generate Diff Rgns from RG1? RG2? Over Yes Yes In No No Out Yes No Atop No Yes Xor Yes Yes Plus Yes Yes 2.4 Optimising using Opaque Areas Preferably, within each region a flag is stored indicating whether the pixel data in the region proxy is completely opaque. It is therefore possible to reduce the number of per-pixel compositing operations by exploiting the effect opaque operands have on the compositing operators.

2.4.1 Opaque Area Optimisation for "Over" Region Groups If an opaque region is "OVER" another region, then there is no need to compute the result of the composite, as no part of the proxy of the right operand region is visible through the proxy of the left operand. Preferably, the resultant region is made to reference the proxy of the right operand, which has the same effect as actually doing the composite.

for optimising opaque areas for "OVER" regions is illustrated using the following pseudocode: 546904.doc -33- RG-urgn RGl->urgn union RG2->urgn FOR i 0 TO number of regions in RG1 DO diff_rgn rgli->rgn difference RG2-*urgn IF diffrgn is non-empty THEN ADD to RG a new region with diff_rgn as its region description and rgli-proxy as its proxy. END IF FOR j 0 TO number of regions in RG2 DO inter_rgn rgli->rgn intersection rg2j->rgn IF inter_rgn is non-empty THEN IF rgl, is OPAQUE THEN new_p rgli-proxy

ELSE

create new proxy new_p initialised to OVER of rgli->proxy 15 and rg2j->proxy inside interrgn.

**END

IF

ADD to RG a new region with inter_rgn as its region description and new_p as its proxy. END IF END DO END DO FOR j 0 TO number of regions in RG2 DO diff_rgn rg2j->rgn difference RG1-+urgn IF diff_rgn is non-empty THEN ADD to RG a new region with diff_rgn as its region description and rg2j->proxy as its proxy. END IF END DO 2.4.2 Opaque Area Optimisation for "IN" Region Groups If a region is "IN" an opaque region, then according to the properties of the "IN" operator, the resultant pixel data is the same as that of the left operand. This can be achieved by having the resultant region simply reference the proxy of the left operand.

The method of optimising opaque areas for "IN" region groups is a slightly modified version of the "IN" region group construction method provided previously. The 546904.doc -34difference is that when calculating the intersection region of the current region in RG1 and each region of RG2, a check is performed to determine whether the current region in RG2 is opaque. If this is the case, then the proxy of the newly calculated region (new_p) will be the proxy of the current region in RG1.

The technique is illustrated using the following pseudocode: e 546904.doc .1 RG-urgn RG1-urgn intersection RG2->urgn FOR i 0 TO number of regions in RG1 DO FOR j 0 TO number of regions in RG2 DO inter_rgn rgli-+rgn intersection rg2j->rgn IF inter_rgn is non-empty THEN IF rg2j is OPAQUE THEN new_p rgl->proxy

ELSE

create new proxy new_p initialised to IN of rgli-proxy and rg2j-proxy inside inter rgn.

END IF ADD to RG a new region with interrgn as its region description and new_p as its proxy. END IF END DO END DO Initialising the Entire Tree The entire compositing tree can be initialised by using the above-described methods for constructing region groups on every binary region group in the tree. A node cannot be initialised until the children of the node have been initialised. Therefore, the process preferably starts at the bottom of the tree and works its way up towards the root.

The process first checks to see if the current node is a leaf node. If this is the case, then a leaf node region group is constructed. However, in the case that the current node is a binary node then a binary node region group is constructed using the methods outlined in sections 2.4.1 and 2.4.2. The following pseudocode outlines a method for initialising all the region groups of the tree. The method utilises a recursive function, which is called passing the root of the tree as an argument.

tree_init(node tree ptr)

BEGIN

IFnode is a leaf node THEN CONSTRUCT leaf node region group

ELSE

tree_init(node-left) 546904.doc Li C -36tree_init(node->right) CONSTRUCT binary node region group by combining region groups of the left and right children END IF END treeinit 2.6 Constructing the Resultant Image Once the compositing tree has been initialised, the region group at the root of the tree contains a group of zero or more regions which together represent the partitioning of the resultant image into areas which differ in the way the image data was generated.

Some of the regions' proxies can refer to image data directly from leaf nodes of the tree, having not required any compositing. Other regions, on the other hand, can have proxies which are the result of compositing operations. If a single resultant image is required, *2 such as an image stored in a pixel buffer, this can be achieved by copying the image data from the proxy of each region to the pixel buffer within the area corresponding to the region. The process is demonstrated in the pseudocode provided below, which is generalised and able to restrict the construction of the final image to any nominated update region.

construct_image output_image pixel data ptr, urgn region description

BEGIN

FOR i 0 TO number of region in RG DO int_rgn rgi--rgn intersection urgn IF int_rgn is non-empty THEN COPY image data from rg,->proxy to output_image inside int_rgn END IF END DO END construct_image Dynamic Rendering Dynamic Rendering refers to the generation of multiple successive images.

546904.doc -37- Given a compositing tree, it is possible to generate the region groups of the tree (containing regions and proxies) using the method described above. A method of rendering a digital image on a graphical display that is remotely positioned with respect to a host computer associated with the display, which supports dynamic rendering, is described below. The compositing tree represents an image. Changes to the tree can be made to make the tree represent a new image. The region groups of the tree (and tree region description and proxies) are updated to reflect this modified tree. Performance is improved by exploiting commonality between the two images. An example will illustrate the techniques and terminology of the method which supports dynamic rendering.

Fig. 3 shows the region subdivision and the respective compositing expressions advantage is not taken of opacity) for the simple compositing tree of Fig. 2.

Consider an example in which element A moves by a small amount relative to the other :i •elements. Some regions in the region group at the root of the tree will be affected by A moving.

If opaque case optimisations are ignored, the regions with compositing expressions which include A will be significantly affected by A moving. The region numbers which are so affected are 2, 3, 5 and 6. When updating the region group at the root of the tree, those regions will need both their region descriptions and their proxies completely recalculated. This situation is herein defined as primary damage. Any region whose compositing equation includes an element which has changed in some way, may be said to suffer primary damage.

Regions that abut regions which have A in their compositing expression are also effected by A moving, though not as severely as those regions with primary damage. In the example, these other affected regions are 1, 4, 7 and 8. When updating the region group at the root of the tree, these regions will need their region descriptions recalculated.

However, the proxies of the affected regions only need to be recalculated in areas of the new region which were not included in the corresponding earlier region. This situation is defined herein as secondary damage. Generally, secondary damage is incurred if an element upon which the boundary of a region (but not content) depends, changes in some way.

In order to reduce the per-frame update cost of transmitting an image to a remote graphical display, it is important to reduce, as far as is practicable, the amount of work necessary, both in terms of per-pixel operations, but also in terms of region group operations. The concepts of primary and secondary damage are a way of facilitating this.

If a minimum set of regions throughout all the compositing tree which have some kind of 546904.doc -38damage can be accurately determined, then the amount of work being done to render the image at a remote display is reduced.

Fig. 25 shows a method of rendering a series of images on a graphical display that is remotely positioned with respect to a host computer associated with the display.

The host computer is preferably connected to the graphical display via a bandwidth limited communications link as discussed above. Each member of the series of images is related to a preceding member. Again, the image is preferably rendered by compositing a plurality of primitive components according to a hierarchical expression tree representing a compositing expression for the image. The hierarchical expression tree includes a plurality of nodes each representing a primitive component of the image or an operation for combining sub-expressions of the compositing expression. Further, each primitive component has a predetermined outline within the space in which the image is defined and can have one or more attributes line style, colour, opacity, surface texture etc) associated with a particular component. The process begins at step 2501, where a hierarchical compositing tree representing a first image is accessed. At the next step 2503, for each node of the hierarchical compositing tree associated with a primitive component, the component image space is subdivided into a number of mutually exclusive regions.

Again, the mutually exclusive regions are substantially formed by segments of a grid which encompasses the space in which the predetermined outlines are defined. The process continues at the next step 2505, where each of the mutually exclusive regions is examined and internodal dependency information is created. The data structure of the internodal dependency information and the process for creating it are described below, particularly with reference to sections 3.1 to 3.9. Each of the mutually exclusive regions is defined by a region outline substantially following at least one of the predetermined outlines or parts thereof.

At the next step 2507, for one or more of the mutually exclusive regions, the region outline and any attributes associated with the mutually exclusive region are encoded into one or more rendering instructions that completely define the operations needed to render the image defined by the mutually exclusive regions. As described above, the rendering instructions preferably comprise proxy information representing the next line of pixels to be rendered, the first column to be rendered and the number of pixels to follow, followed by the colour of each pixel. If groups of pixels representing a particular mutually exclusive region are all of the one colour, then each individual pixel does not need to be transmitted, simply a proxy representing the particular mutually exclusive region and the colour to paint the region need to be transmitted. Further, as 546904.doc -39described above, where the attributes of two or more regions are the same, then only one set of attributes referencing the two or more regions need to be transmitted. Still further, the proxy information that represents the regions of an image to be rendered can be converted into a series of rectangles, as will be explained below particularly with reference to section 5.0. The process continues at the next step 2509, where the rendering instructions are transmitted to the remote graphical display, via the communications link, so that the first image can be created.

The process continues at step 2511, where the image changes resulting in changes to the hierarchical expression tree. The changes can include, but are not limited to, adding new primitive components, deleting components, changing components or changing the operations for combining the components. At the next step 2513, the dependency information created at step 2505 is examined to determine which of the :o mutually exclusive regions have been affected by the changes to the hierarchical compositing tree. The process continues at the next step 2515, where the mutually 15 exclusive regions affected by the changes are updated. For each of the mutually exclusive regions that depend on components which have been modified, the associated region boundary outlines are updated to produce one or more further regions. Preferably, where the attributes associated with a particular further region is the same as that of another region that already exists, a pointer can be associated with the particular further region referencing the attributes of the other region. The process of examining and updating the regions is described in sections 3.6 to 3.9, below. In particular, section 3.9 includes a pseudocoding listing which illustrates a method for updating a binary "OVER" region. At the next step 2517, the internodal dependency information is updated to reflect the changes to the hierarchical compositing tree. At the next step 2519, for each of the

S.

further regions, the region outline and any attributes associated with the further region are encoded into one or more rendering instructions that define the operations needed to render the further regions, according to the changed hierarchical expression tree. As described above, if groups of pixels representing a particular further region are all of the one colour, then each individual pixel does not need to be transmitted, simply a proxy representing the particular further region and the colour to paint the region need to be transmitted. Again, where the attributes of two or more further regions are the same, then only one set of attributes referencing the two or more further regions need to be transmitted. Further, the proxy information that represents the regions of an image to be updated can be converted into a series of rectangles, as will be explained below particularly with reference to section 5.0. The process continues at the next step 2521, 546904.doc where the rendering instructions for the new mutually exclusive regions are transmitted, via the bandwidth limited communication link, to the remote graphical display so that a second image can be created. Steps 2511 to 2521 can repeat for any number of successive changes to the image being displayed on the remote graphical display.

3.1 Basic Data Model The data model used for static rendering, consisting as it does of a region description and a proxy, is insufficient for use in dynamic rendering. This is because, for primary and secondary damage to be determined, it must be possible to associate regions of the same content between frames. To support the association of regions of the same content, some extra information is required in each region in a region group. Therefore, each region in a region group preferably contains the following data: A Region Description: A low-level representation of the boundaries of region. The region descriptions of all the regions in a region group must be mutually exclusive (non-intersecting, non-overlapping).

15 (ii) A Proxy: Some means of caching the pixel data resulting from applying the operation specified by the compositing expression at every pixel inside the region description. A proxy can be as simple as a 24-bit colour bit-map, or something much more complicated (such as a run-length encoded description). Fundamentally, a proxy simply has to represent pixel data in some way which makes it efficient to retrieve and 20 use.

(iii) A Contents Label: A contents label represents a unique symbolic expression that describes the method of construction of image data. The terms in the symbolic expression distinguish between different categorisations of a source of image data. Therefore, the region groups of two distinct leaf nodes in the compositing tree preferably contain regions which are labelled with distinct contents labels even if their actual image data is equivalent. The methods described herein use a system of unique integers to represent contents labels. For example "23" can represent over B) over

C".

(iv) A Flag Register: A general-purpose flag register can be used to store state during the region group update process. The exact flags stored here will be outlined in a later section.

3.2 Contents Labels Leaf node region groups can contain multiple regions, with each region naturally having a unique contents label. For example, the region group of a leaf node in a compositing tree can contain a single region (tagged with a single contents label) 546904.doc -41representing the non-transparent area of the leaf node. Alternatively, the leaf node region group can contain two regions each tagged with a different contents label. One region representing the area of the leaf node which is completely opaque, the other representing the remaining non-transparent area. A leaf node can also be categorised even further, into an arbitrary number of regions (and associated contents labels).

One way a contents label can be created is by assigning a new label to a region of a leaf node region group. Another way is taking other contents labels and combining them to create a new contents label that represents the symbolic expression that represents the combination of the contributing expressions. For example, if the contents label representing comp B) comp C) is combined with the contents label representing (D comp E) then a new contents label will be created which represents comp B) comp C) comp (D comp *.As well as contents labels, dependency information is also required.

Dependency information indicates how a given contents label is related to other contents labels, both in terms of how the contents of one region contribute to contents of other regions, and how changes to a region boundary affect the boundary of other regions. The •"method ofFig. 25 associate the following data with each contents label.

Primary Dependency List: Each primary dependency is a contents label L' to which a contents label L directly contributes. In other words, a "primary dependency" is a contents label L' representing an expression which has been constructed by combining L and some other contents label. Each time contents labels are combined, the contents label for the combination is added to the primary dependencies of all contributors.

(ii) Secondary Dependency List: Each secondary dependency is a contents label L" which can be indirectly affected if the image represented by the contents label L has changed in some way that affects the boundary of L. Whenever contents labels are combined, a contributing contents label can be added to the secondary steps of the continuation if and only if the compositing operator yields a difference region with the contributing contents label. Table 2 shows which of some commonly used operators yield difference regions for their left and right operands. In addition, for a combination of (A comp the secondary dependencies of the combination contents labels for all (A comp bi) and all (aj comp B) are added, where aj are the secondary dependencies of A and bi are the secondary dependencies of B.

(iii) Property Information: Each contents label can represent contents which have properties which the compositing engine may be able to exploit. An example is that 546904.doc 42 of opaqueness. If a contents label represents opaque content, then compositing that content can be much faster, as for certain operators, no per-pixel compositing operations need be required.

3.3 Contents Label Implementation The described methods preferably use unique integers as contents labels, and store a number representing the number of contents labels which currently exist. When a new contents label is created, the number is incremented and becomes the unique integer representing the contents label. This technique of assigning a contents label by monotonically incrementing an integer means that the contents labels' associated data structures can be stored in a one dimensional array which grows as more contents labels are added. The data structure of a contents label can be referenced by using the contents label as an index. When a leaf node contents label is created, the contents label can be initialised to have no primary or secondary dependencies. If the current leaf node contents label is opaque, then a flag is set in content label the properties if i.

S 15 The pseudocode below illustrates a method used to create a new contents label which is not dependent on other contents labels (ie: a leaf node region group contents •label): Notation opaque A flag passed to the function which indicates whether or not the contents label represents opaque content or not.

curclab A global integer which stores the last contents label created.

clabs A global array which stores the associated data structures of the contents label.

clabs[i]->pri_deps A pointer to the head of content label i's primary dependency list.

clabs[i]->sec_deps A pointer to the head of content label i's secondary dependency list.

clabs[i]->properties A flag register representing contents label i's properties.

create new contents label 546904.doc -43opaque: boolean RETURNS unsigned int

BEGIN

INCREMENT curclab.

clabs[cur_clab]-pri_deps NULL.

clabs[cur_clab]-sec_deps NULL.

IF opaque THEN clabs[cur_clab]->properties OPAQUE.

ELSE

clabs[cur_clab]->properties 0.

END IF RETURN curclab.

S.6 END create new contents label.

15 Contents labels can also be created to represent the combination of existing contents labels as seen in the following pseudocode. This can be achieved by a hash table o: which maps an operation and the contents labels of the operands of the operation (hashed together to create a key) to a single contents label representing the result.

When a region is created which represents an intersection between two other S 20 regions each with its own contents label), a new contents label is generated which is used to tag the new region. When this new contents label is generated, it must be added to the primary dependency lists of both the contributing operands of the region. A secondary dependency list which depends on the secondary dependencies of the two contributing contents labels as well as the properties of the compositing operator must also be generated.

The process is recursive and begins by adding the newly created contents label (newcl) to the primary dependency lists of the contributing contents labels. Then, depending on the properties of the compositing operator, none, either or both of the contributing contents labels can be added to the secondary dependency list. Then every contents label representing (clabl op sd2i) and (sdli op tab2) can be added to the secondary dependency list.

Notation clab The first contributing contents label.

546904.doc -44clab2 The second contributing contents label.

sdli The i'th element of clabl's secondary dependency list.

sd2i The i'th element of clab2's secondary dependency list.

create_binary_contents_label clabl contents label, clab2 :contents label, op: compositing operator

BEGIN

IF the hash table already contains an entry representing clabl op clab2 10 THEN RETURN the existing contents label representing the combination.

END IF Generate a new entry in the hash table representing clabl op clab2, mapping to new_cl.

(Add the new contents label to the primary dependency lists of the contributing contents labels if the compositing op requires it) add_to_primary_dep_list(clabl, new_cl) addtoprimarydep list(clab2, new cl) (Generate the secondary dependencies) IF op generates left diff rgns THEN add clabl to secondary deps END IF IF op generates right diff rgns THEN add clab2 to secondary deps END IF FOR i 0 TO number of elements in sdl DO addtosecondary_dep_list newcl, 546904.doc create_binary_contents_label(sd 1 clab2) END DO FOR i 0 TO number of elements in sd2 DO addtosecondary_dep_list new_cl, create_binary_contents_label(clabl, sd2j) END DO END constuct_binary_contents_label 3.4 Combining Region Groups for Dynamic Rendering 15 Before any incremental updates can be made to a compositing tree, the compositing tree must be constructed to be in a consistent initial state. The basic method for achieving this is the same as that used for static rendering, except that support for contents labels is included.

Leaf node region groups are initialised essentially as with the static rendering 20 case, except that each region in each leaf node region group is tagged with a unique contents label. Each contents label can in turn be tagged with various categorisation properties which may help a renderer to be more efficient. For example, a contents label can be tagged as being completely opaque.

The initialisation of binary nodes is also similar to the static rendering case. By way of example, the method in which the region group for an "OVER" binary node is constructed will now be explained. The methods for constructing the region groups of the other compositing operators can easily be inferred from the "OVER" case.

When a difference region between rgi of one operand and the coverage region of the other operand is calculated, the difference region inherits the contents label rgi. When an intersection region is created, on the other hand, a new contents label is created by combining the contents labels of the two contributing regions since the two contributing regions had their proxies composited into a new proxy which means new content. The pseudocode for constructing an "OVER" region group which includes contents label management is provided below: 546904.doc -46- RG1 RG2

RG

RG1->urgn RG1-+urgn RG->urgn rgli rg2j rgli-*rgn rg2j-+rgn rg 1 i-proxy rg2j -proxy Notation The region group of the binary node's left child The region group of the binary node's right child The region group of the binary node. It is this region group that we are initialising The region description representing the union of all RGl's region descriptions (RGl's coverage region).

The region description representing the union of all RG2's region descriptions (RG2's coverage region).

The union of all RG's region descriptions.

The current region in RG1 The current region in RG2 rgli's region description rg2j's region description rgli's proxy rg2j's proxy r *r rr r .1 RG-urgn RG1->urgn union RG2-urgn FOR i 0 TO number of regions in RG1 DO diff_rgn rgli-rgn difference RG2-urgn IF diff_rgn is non-empty THEN ADD to RG a new region with diff_rgn as its region description, rgli-proxy as its proxy and rgli->clab as its contents label.

END IF FOR j 0 TO number of regions in RG2 DO inter_rgn rgli->rgn intersection rg2j->rgn IF inter_rgn is non-empty THEN new_clab GENERATE a new unique contents label as a result of combining rgli->clab and rg2j->clab.

IF rgli-clab is OPAQUE THEN 546904.doc -47new_p rgli-proxy

ELSE

create new proxy new_p initialised to OVER of rgli->proxy and rg2j->proxy inside interrgn.

END IF ADD to RG a new region with inter_rgn as its region description, new_p as its proxy and new_clab as its contents label.

END IF END DO END DO FOR j 0 TO number of regions in RG2 DO diff_rgn rg2j->rgn difference RG1->urgn IF diff_rgn is non-empty THEN ADD to RG a new region with diff_rgn as its region description, 15 rg2j->proxy as its proxy and rg2j->clab as its contents label.

END IF END DO 3.5 Secondary Dependencies and Over 20 Secondary dependencies are only generated when a new contents label is created by combining two other contents labels. As can be seen in the above pseudocode, this only occurs when an intersection region is generated. Essentially, contents labels generated for intersection regions can be used as triggers. The regions tagged with two contents labels cannot indirectly affect one another unless they intersect. The secondary dependency list for a particular contents label is dependent on the compositing operator the composite contents label represents, the two contributing contents labels and their secondary dependency lists.

A method of generating a secondary dependency list for a new contents label (C) which represents one contents label composited over another contents label using the "OVER" operator will now be explained. Elements of the secondary dependency lists of A and B are referred to as Ai and Bi respectively. Firstly, both A and B are added to the secondary dependency list for C. This is because if the region tagged with C changes its boundary, then it is likely that any regions tagged with A and B will need to be recalculated (because their regions are likely to abut the region of Secondly, for each element of the secondary dependency list for B, each contents labels representing (A 546904.doc -48- OVER Bi) is added. A mapping representing A OVER Bi may not currently exist in the system and needs to be created. A secondary dependency list can contain contents labels which are not represented by any region in a region group. Secondary dependencies can come into existence by changes in region boundaries. The rationale is that A intersects B, and therefore it is likely that A also intersects regions tagged with contents labels which exist in the secondary dependency list for B. Similarly, for each element of the secondary dependency list of A, each contents label representing (Ai OVER B) is added.

3.6 Contents Labels and Damage The concepts of primary and secondary damage were introduced with reference to Fig. 3 to demonstrate that it is not always necessary to regenerate an entire image as a result of a change to the compositing tree. By keeping track of dependencies between regions of different content, it only becomes necessary to regenerate image data in regions whose contents have become damaged. The following explanation outlines the dependencies and damage for compositing tree changes where only leaf nodes are modified. More complex change scenarios such as tree structure changes etc will be outlined in later sections.

""If a leaf node is modified, the contents labels of the affected regions for the leaf node are said to be "primary damaged". Primary-damaging a contents label involves recursively primary-damaging all the primary dependencies of the contents label.

Whenever a contents label is primary-damaged, all the secondary dependencies of the contents label are non-recursively marked with secondary damage. The process begins by flagging the contents label to be damaged. The following pseudocode demonstrates how contents labels can be damaged: Notation clab The contents label to be damaged pdi The i'th element of clab's primary dependency list.

sdi The i'th element of clab's secondary dependency list.

damage_contents_label clab contents label, clab contents label, 546904.doc -49-

BEGIN

FLAG clab with PRIMARY damage FOR i 0 TO number of elements in sd DO FLAG sdi with SECONDARY damage END DO FOR i 0 TO number of elements in pd DO damage_contents_label(pdi) END DO END damage_contents_label SWhen a tree update occurs, any region having a contents label marked as having 15 primary damage will need to recalculate both the region boundaries and the proxy for the region. Any region with having a contents label marked as having secondary damage will need to recalculate a region description but will only need to recalculate a proxy in areas of the new region that were not included in the earlier region.

3.7 Examples of Contents Labels and Dependencies In order to clarify the concepts of contents labels and damage, some examples of varying complexity will be presented.

3.7.1 Example 1 The configuration of elements A and B shown in Fig. 9 will result in the following contents labels, as seen in Table 3, after the compositing tree is initially constructed (Note: in the following table contents labels are represented as unique strings not as integers where "over" has been abbreviated to This is simply for readability.): Table 3 Contents Label Primary Deps. Secondary Deps.

A AoB B AoB AoB B If A moves, then AoB will have primary damage, resulting in B having secondary 546904.doc damage.

3.7.2 Example 2 The configuration of elements A, B and C shown in Fig. 10 will result in the following contents labels, shown in Table 4, after the compositing tree is initially constructed: Table 4 Contents Label Primary Deps. Secondary Deps.

A AoB, AoC B AoB, BoC AoB AoBoC A, B C AoC, BoC, (AoB)oC AoC A, C BoC B, C (AoB)oC AoB, C, AoC, BoC In this example, every element intersects every other element, so if something changes, everything will be damaged in some way everything which is a primary dependency of the changed element will have primary damage, whereas everything else will have secondary damage.

Fig. 11 illustrates the effect of A moving in a subsequent frame. As can be seen, if A is damaged, the regions defined by A, AoB, AoC and (AoB)oC will each have primary damage. The regions defined by B, C and BoC will each have secondary damage.

3.7.3 Example 3 The configuration of elements A to E as shown in Fig. 12 will result in the following contents labels, as shown in Table 5, after the compositing tree is initially constructed: Table Contents Label Primary Deps. Secondary Deps.

A AoB, AoC, AoE, Ao(DoE), AoD B A__oB, BoC, BoE 546904.doc 51 eq's St 9@ S

S.

CC

S C 0 AoB AoBoE A, B DoE, AoD, CoD, (AoC~oD E DoE, AoE, (AoB)oE, BoE, (BoC~oE, (AoC~oE DoE Ao(DoE), (AoC)o(DoE), D, E C AoC, BoC, Co(DoE), CoE, ____CoD AoC AoCoE, (AoC)o(DoE), A, C BoC (BoC~oE BC AoE A, E (AoB)oE AoB, E AoE, BoE BoE BE CoE C, E (BoC)oE E,BoE, CoE AoD A,D CoD C,D (AoC~oE AoC, E, AoE, CoE Ao(DoE) A, DoE, AoD, AoE Co(DoE) C, DoE, CoD, CoE (AoC~o(DoE) AoC, DoE, Ao(DoE), Co(DoE), (AoC~oD, ___(AoC)oE I(AoC)oD koC, D,AoD, CoD Since A intersects every other element, if A moves, a large amount of the compositing tree will need to be recomputed. Fig. 13 shows that the only part of the tree that is left alone is the area corresponding to BoC and the dependent BoCoE of BoC. The regions that are primary and secondary damaged if A moves can be summarised as follows: Primary Damage A, AoB, AoC, AoE, Ao(DoE), (AoB)oE, (AoC~oE, (AoC)o(DoE), AoD, (AoC~oD egos .0 C 546904.doc -52- (ii) Secondary Damage B, C, E, DoE, BoE, CoE, DoE, CoDoE On the other hand, if B moves, the amount of damage is less than if A moved, since B doesn't intersect D. The regions DoE, Ao(DoE), (AoC)o(DoE), Co(DoE) and (AoC)oE and their ancestors are not damaged when B moves. This is shown in Fig. 14.

The regions that are primary and secondary damaged if B moves can be summarised as follows: Primary Damage B, AoB, BoC, BoE, (AoB)oE, (BoC)oE (ii)Secondary Damage A, E, C, AoE, CoE The examples presented above are simple, but they are sufficient to demonstrate that the dependencies methods described above will damage those contents labels which are affected when a particular contents label/s is/are damaged. In a typical complex composite, it is rare for large numbers of elements to intersect a large number of other elements, meaning that large areas of the compositing tree should be untouched during updates using the above methods.

15 3.8 Example of Secondary Dependencies and Compositing Operators Consider a modified version of Example 3 above. The configuration of elements shown in Fig. 18 will result in the following contents labels, as shown in Table 6, after oo the compositing tree is initially constructed. Note that AaB represents A ATOP B and AiB represents A IN B etc: Table 6 5

SW..

55

OS

*5 Contents Label Primary Deps Secondary Deps A AaB B AaB, BoC AaB B C BoC, Co(DiE) BoC B, C D DiE E DiE DiE Co(DiE) Co(DiE) C, DiE As seen in Fig. 18, the ATOP operator clips A to the bounds of B, meaning that intersections between A and any of C, D or E never occur. Similar things occur with the 546904.doc 53 IN operator. This means that the elements in this scene are less tightly coupled. For example, if A is changed, then only B and AaB are immediately damaged. Similarly, if E is damaged, it is only possible for DiE to be damaged.

3.9 Updating Region Groups The methods described herein use the contents label and damage framework to reduce the amount of work that has to be done to make a binary region group consistent with operands updated during an update. The method of Fig. 25 only updates those regions in a region group whose contents labels have primary or secondary damage, adding any new region which comes into existence as a result of the changes made to the compositing tree, and deleting any region in the right group whose contact no longer exists.

Each different binary operator has a different updating function which deals with the specific requirement of that operator. The process of updating region groups is a two- ~pass process. The first pass updates any intersection regions that have been primary S 15 damaged and adds any new intersection regions generated due to the damage. Each region of the region group for one operand is intersected with each region of the region group for the other operand whenever one or both of their corresponding contents labels are primary damaged. If the intersection is non-empty, then preferably it can be determined if a contents label representing the combination exists. If the contents label 20 doesn't exist, one is created and primary damaged. Note that primary damaging a contents label will mark all the secondary dependencies of the contents label with secondary damage.

If a region in the region group is currently tagged with the primary damage contents label, the regions boundary and proxy are updated. If no such region exists in this region group, then a new region keyed by this contents label is added to the region group. A new proxy is generated and assigned to this region along with the right description relating from the intersection operation.

A difference between each region group of one operand and the coverage region of the other operand is calculated whenever the contents label of the region has primary or secondary damage. If the difference is non-empty and a region tagged with the contents label exists in the region group, then the region description and proxy reference are updated. If such a region doesn't exist then a region keyed by the contents label is added to the region group. The added region is assigned as a coverage region of the difference result and references the proxy of current region.

Each region of the region group for one operand is interacted with each region of 546904.doc -54the region group for the other operand whenever the contents label representing their combination has secondary damage and no primary damage. If the intersection is nonempty, the region group is searched looking for a region keyed by the contents label. If such a region exists, the description of the region is updated and the associated proxy is updated as the difference between the new and old regions. If such a region doesn't exist, then a region keyed by the contents label is created. The created region description is assigned the result of the interaction operation and the proxy for the created region is generated.

Pseudocode which illustrates a method for updating a binary "OVER" region group is provided below: Notation RG1 RG2

RG

RGl-urgn RGl->urgn RG->urgn rgli rg2j Rgli-+rgn Rg2j -rgn rgll->proxy rg2j->proxy Rgli-+clab Rg2j->clab The region group of the binary node's left child The region group of the binary node's right child The region group of the binary node. It is this region group that is being initialised.

The region description representing the union of all RGl's region descriptions (RGl's coverage region).

The region description representing the union of all RG2's region descriptions (RG2's coverage region).

The union of all RG's region descriptions.

The current region in RG1 The current region in RG2 rgli's region description rg2j's region description rgli's proxy rg2j's proxy rgli's contents label rg2j's contents label 546904.doc RG->urgn RG1-urgn union RG2-*urgn (First Pass this pass is used to deal with primary damage of intersection regions and any new intersection regions generated) FOR i 0 TO number of regions in RG1 DO FOR j 0 TO number of regions in RG2 DO IF rgli-clab has PRIMARY damage OR rg2j-+clab has PRIMARY DAMAGE THEN interrgn rgli->rgn intersection rg2j-+rgn IF inter_rgn is non-empty THEN comp_clab SEARCH for an existing contents label which represents (rgli->clab comp rg2j->clab).

IF a region tagged with comp_clab already exists in RG S 15 THEN IF rgli-clab is OPAQUE THEN new_p rgli-+proxy

ELSE

create new proxy new_p initialised to OVER of rgli->proxy and rg2j-proxy inside inter_rgn.

END IF MODIFY the existing region to have inter_rgn as its region description and new_p as its proxy.

ELSE

new_clab create_binary_contents_label(rgli->clab, rg2j->clab).

IF rgli-clab is OPAQUE THEN new_p rgli-+proxy

ELSE

create new proxy new_p initialised to OVER of rgli-+proxy and rg2j->proxy inside inter_rgn.

END IF damage_contents_label(new_clab) ADD to RG a new region with inter_rgn as its region description, newp as its proxy and new_clab as its contents label. 546904.doc -56- END IF FLAG the region as being 'RETAIN AFTER UPDATE' END IF END IF END DO END DO (Second Pass this pass is used to deal with primary and secondary damage of difference regions and secondary damage of intersection regions) FOR i 0 TO number of regions in RG1 DO IF rgli->clab has PRIMARY or SECONDARY damage THEN diff_rgn rgli-rgn difference RG2->urgn IF diff_rgn is non-empty THEN IF a region tagged with rgli->clab already exists in RG THEN MODIFY it to have diff_rgn as its region description and rg1i->proxy as its proxy.

ELSE

•ADD to RG a new region with diff_rgn as its region description, rgli--proxy as its proxy and rg1->clab as its contents label. END IF FLAG the region as being 'RETAIN AFTER UPDATE' END IF END IF FOR j 0 TO number of regions in RG2 DO comp_clab SEARCH for an existing contents label which represents (rgli->clab comp rg2j->clab).

IF comp_clab exists AND comp_clab has SECONDARY damage but NO PRIMARY damage THEN interrgn rgli->rgn intersection rg2j->rgn IF inter rgn is non-empty THEN GET a reference to the existing region tagged in this region group with comp_clab which MUST exist in this region group IF rgli->clab is OPAQUE THEN existing regions proxy =rgl i ->proxy

ELSE

546904.doc -57update_rgn inter_rgn difference the region's previous region description.

update existing regions proxy to include OVER of rgli->proxy and rg 2 j proxy inside update region.

END IF MODIFY the existing region to have inter_rgn as its region description and new_p as its proxy.

FLAG the region as being 'RETAIN AFTER UPDATE' END IF END IF END DO END DO FOR j= 0 TO number of regions in RG2 DO IF rg2j-+clab has PRIMARY or SECONDARY damage THEN diff_rgn rg2j->rgn difference RG1-urgn IF diff_rgn is non-empty THEN IF a region tagged with rg2j-clab already exists in RG THEN MODIFY it to have diff_rgn as its region description and rg2j->proxy as its proxy.

ELSE

ADD to RG a new region with diff_rgn as its region description, rg2j->proxy as its proxy and rg2j-clab as its contents label. END IF FLAG the region as being 'RETAIN AFTER UPDATE' END IF END IF END DO DELETE all regions of RG which are not marked RETAIN AFTER UPDATE but whose contents labels have damage, and CLEAR flag in retained regions.

Tree Modifications (Linking and Unlinking) More complex changes to a compositing tree can be achieved by changing the 546904.doc -58structure of the tree. Most typical tree structure changes can be made by using two low level operations, link and unlink.

The unlink operation can be used to separate a child node from the parent of the child. After the operation is completed, the child node has no parent the child node can be linked in somewhere else), and the parent has a link available some other node can be linked there instead). Nodes in the compositing tree above the unlinked child contain content which is dependent on the unlinked child. Therefore, at the time of the next update, the contents label present in the unlinked child at the time of unlinking must be damaged to ensure that the dependent region groups higher in the tree are appropriately updated. The updating is achieved by the parent node caching away those contents labels existing in the unlinked child of the parent node. If another subtree is linked in place of the unlinked child and subsequently unlinked without the region group of the parent being updated, it is not necessary to cache the contents labels of this new subtree. Pseudocode for the unlink operation is provided below. Note that the UNLINKEDLEFT or UNLINKEDRIGHT flag is set so that the contents labels of a newly linked subtree can be damaged when region groups (including their proxies) higher in the tree must then be updated.

unlink S node compositing tree node

BEGIN

.t parent node ->parent.

node ->parent NULL.

IF node is parent's left child THEN parent ->left NULL.

IF parent doesn't have UNLINKED_LEFT set THEN SET the UNLINKED_LEFT flag in parent.

ELSE.

RETURN.

END IF ELSE IF node is parent's right child THEN parent ->right NULL.

IF parent doesn't have UNLINKED_RIGHT set THEN 546904.doc -59- SET the UNLINKED_RIGHT flat in parent.

ELSE

RETURN

END IF END IF COPY all the contents labels in node's region group into an array stored in parent -+unlinked_clabs.

END unlink The link operation involves linking a node with no parent to a free link of a parent node. Pseudocode for the preferred updating method operation is provided below: link 15 child compositing tree node, parent compositing tree node, which link either LEFT or RIGHT

BEGIN

child -+parent parent IF which link is LEFT THEN parent -left child.

ELSE

parent right child.

END IF END LINK 4.1 Updating the Entire Compositing Tree I If a leaf node in the compositing tree changes, the region group of every node in a direct line from the leaf node to the root of tree must be updated using the methods described above. Fig. 15 shows circled those nodes which need to have their region groups updated if leaf nodes B and H change in some way.

Pseudocode for the tree updating method is provided below: update_tree 546904.doc node compositing tree node

BEGIN

IF node is leaf node THEN Rerender the leaf node and update its region group.

ELSE

IF unlinking occurred in left subtree or left subtree contains dirty leaf nodes THEN update_tree(node ->left).

END IF.

IF unlinking occurred in right subtree or right subtree contains dirty leaf nodes THEN update_tree(node ->right).

15 END IF.

IF node has UNLINKED_LEFT or UNLINKED_RIGHT flags set THEN CALL damage_contents_label on every element of •node->unlinked clabs.

IF node has UNLINKED LEFT set THEN CALL damage_contents_label on every contents label existing in node>left's region group.

CLEAR the UNLINKED_LEFT flag in node.

END IF IF node has UNLINKED RIGHT set THEN CALL damage_contents_label on every contents label existing in node-+right's region group.

CLEAR the UNLINKED_RIGHT flag in node.

END IF END IF CALL the region group update routine appropriate for node's compositing operator.

END IF END update_tree 5.0 Encoding Image Information into Region Render 546904.doc -61 The proxy information generated by the methods detailed above substantially represent non-overlapping (ie. mutually exclusive) regions of an image that need to be rendered to construct an image (ie. in the case of static rendering) or re-rendered to reconstruct an image (ie. in the case of dynamic rendering) if the image is updated (ie.

changes) from one frame to the next. The proxy information can be as small as a single pixel or as large as an entire screen to be rendered. In order to ensure that the image on a remote graphical display corresponds with an update to the image at the host computer, only the information in the proxies needs to be conveyed to the remote display.

There are a number of ways that the proxy information can be transmitted to the remote display, as at steps 2311, 2509 and 2521. For example, the coordinates of each pixel and the attributes (eg. colour) of the pixel in the proxy can be transmitted.

However, a more efficient method is to notify the remote display of the next line to be o:oo updated, the first column to be updated, and the number of pixels to follow, followed by the colour of each pixel. There can be several lengths of pixels on each line to be rendered. If groups of pixels are all of the one colour, then each individual pixel does not need to be transmitted, simply a description of a particular mutually exclusive region and the colour to paint the region need to be transmitted. Preferably, each particular proxy 999999 9 can be broken down into rectangles with similar characteristics and a description of the rectangle and the contents of the rectangle sent to the remote display in order to construct or reconstruct an image.

A data compression module can be inserted between the proxy, the composition operation and the data transmission. In this instance, a matching data de-compression module needs to be inserted between the remote graphics display receiver and a renderer associated with the display. Any suitable streaming data compression method can be used. For example, the most common data compression method used in modems, V.42bis, would be suitable since V.42bis offers a good compression ratio with modest computing requirements.

The proxy data that represents the regions of an image to be rendered or updated can be converted into a series of rectangles. The rectangles are preferably classified as one of three types: Colour fill: whereby each pixel in the rectangle is the same colour; (ii) Bitmap information: whereby each pixel is defined; and (iii) Run encoded information in one of five formats, as described below.

Run encoding breaks a rectangle into groups of continuous pixels on the same line. A run consists of one or more pixels that can be described as belonging to the same 546904.doc -62run type. In accordance with the described methods, there are five different run types as follows: Transparent pixels: whereby no pixels are plotted for the length of the run; (ii) Colour fill: whereby all pixels in the run are the same colour; (iii) Bitmap: whereby each pixel in the run has the same colour; (iv) Linear colour: whereby the colour changes linearly from a given start colour to a given end colour; and Quadratic colour: whereby the colour changes according to the parameters of a quadratic expression from a given start colour to a given end colour.

Each rectangle preferably consists of a header section and a body section. The header section for each rectangle specifies the destination coordinate of the rectangle on the display screen, the width and height of the rectangle and the rectangle type. The body :i section of the rectangle preferably contains the data needed to fully described the rectangle. In the case of a fill colour, the body section simply specifies the colour. In the case of a bitmap, the body section of the rectangle contains the colour of each pixel. For run encoded, the body section specifies, for each run, the type of the run and the length of the run and the data needed to describe the run.

ooooo 6.0 Local Video Processing on the Remote Multimedia Terminal The digital video camera 2803 of the remote multimedia terminal 2701 can be used to capture frames of a visual scene (not illustrated). A frame consists of a rectangular array of pixels which, when viewed on a screen, appears as an image. To maintain the illusion of smooth motion, at least 20 frames per second must be captured by the video camera 2821 and/or displayed on the local display 2870. After capturing a frame, the video camera 2803 generates a stream of data, representing all of the individual pixels of the frames. The data stream is received by the video controller device 2821, which extracts synchronising signals and determines correct pixel coordinates (row and column) for each pixel. The pixel data are stored in the local video frame buffer 2827.

The local video frame buffer 2827 preferably stores the pixel data in an array.

Each pixel may be stored using a number of bits that will give the required colour accuracy but preferably rounded up to be the nearest multiple of 8 bits (ie 8, 16, 24, or 32 bits). However, any number of bits, including 1 (black/white monochrome) may be used.

The multimedia terminal 2701 displays an image stored in the Local Display Buffer 2829 on the local display 2807 under the control of the Display Controller 2823.

546904.doc 63 The image in the local display buffer could be derived from many sources. For example, it may be transferred from the host computer 2702, via the communication interface 2703 into the local display buffer.

The CPU 2809, under control of a computer program stored in ROM 2811 or running out of general purpose RAM 2813, transfers image data from the local frame buffer 2827 to memory positions in the local display buffer 2829, such that the memory positions correspond to a rectangle on the local display 2807. Referring to Fig. 29, this causes the image captured by the video camera 2821 to appear in a rectangle area 2904 of the local display 2807.

The rectangle area 2904 may be smaller than the total display area 2902 of the local display. The remainder of the display area 2902 may be filled with another image generated by the CPU 2869 or alternatively received from the host computer 2702 via the communication interface 2703. In the instances where the remainder of the display area *2902 is filled with an image received from the host computer 2701, the remote multimedia terminal 2701 preferably communicates the dimensions of the rectangle area 0.0. 2904 to the host computer 2701. This allows the host computer 2702 to avoid communicating data that is going to be overwritten by the camera image to the remote multimedia terminal 2701, thereby saving communication link bandwidth. A method of *00. rendering a digital image including an image captured by the video camera 2803, on a graphical display that is remotely positioned with respect to a host computer associated with the display, will be explained in more detail later in this document.

If the size of the memory in the local display buffer 2829 that is reserved for the video image is smaller than the size of the local frame buffer 2827, then the video image memory values in the local display buffer 2829 will have to be derived from the memory values in the local frame buffer 2827 by an algorithm that calculates the correct colour of each pixel in the local display buffer 2829 that corresponds to the rectangle area 2904.

This algorithm could be as simple as only copying selected pixels or it could calculate the colour of each pixel by averaging the colours of pixels that are being replaced. For example, if an image is being reduced by a factor of 2 then the algorithm may only copy every second pixel from every second line. Alternatively, each square of 4 pixels may be replaced by a single pixel. The colour of the single pixel should preferably be the average of the four pixels that are being replaced. The computation is more difficult when non whole multiples are used for the reduction factor, however the same principle can still be applied by weighting the contribution of the replaced pixels depending on whether they are wholly replaced or partially replaced. The selection of the image reduction algorithm 546904.doc 64 depends on the quality required and the amount of CPU processing power available.

The user may wish to freeze the image displayed to examine it more closely or perhaps save the image. The user may indicate this by activating the user control 2815, for example by pressing buttons 2817, to notify the application program that is controlling the transfer of data from the local frame buffer 2827 to the local display buffer 2829. If the copying process is stopped, then the image displayed will be frozen on the last frame transferred. The user may unfreeze the image by activating another user control 2815 to notify the program to resume copying the local frame buffer 2827 to the local display buffer 2829.

The user may also request processing of the image before it is displayed.

Possibilities of processing include: Mirror image the pixels by transferring the pixels from the local frame buffer 2827 to the local display buffer 2829 in a reverse (left to right) byte order.

Remove the static background image and replace it with another image leaving only those parts of the picture that are moving.

Decoding bar-codes from products in the view of the camera 2803.

Add a delay to the update to simulate the view that a user on a remote communication link might see. The user may also view the effects of image compression to determine the optimum trade-off between speed and image quality.

Advantageously, as the transfer of data from the local frame buffer 2827 to the local display buffer 2829 does not involve the host computer 2701, this transfer can be performed even when the host computer 2702 is not connected.

As discussed above, in another example, an image captured by the digital video S. camera 2803 of the remote multimedia terminal 2701 can be superimposed on top of an image being transmitted from the host computer 2702. Fig. 30 is a flow chart 3000 showing a method of rendering a digital image captured by a video camera 2803) of a multimedia terminal that is remotely positioned with respect to a host computer associated with the multimedia terminal, wherein a further image provided by the host computer is to be displayed on the multimedia terminal. The process of flow chart 3000 begins at step 3001, where an image is captured using the video camera 2803) and pixel data corresponding to the video image is stored in a local video frame buffer, as discussed above. At the next step 3003, the remote multimedia terminal 2701 sends a signal to the host computer 2702 flagging an image captured by the video camera 2803, and informing the computer 2702 that the video camera image is to be superimposed on top of any image being transmitted from the computer 2702. At the next step 3005, the 546904.doc 65 CPU 2809 of the remote multimedia terminal 2701 transmits the dimensions of a rectangular bounding box 2904) within which the captured video image is to be displayed, to the host computer 2702.

The process of flow chart 3000 continues at the next step 3009, where regions of a further image provided by the host computer 2702 and to be displayed on the remote multimedia terminal at step 3001, are manipulated by the host computer 2702, in a manner similar to that discussed above with reference to steps 2301 and 2303 of Fig. 23.

For each compositing operation combining one or more primitives of the further image to be displayed, the regions of the further image are manipulated to determine a number of further regions. Each of the further regions has a corresponding compositing expression.

The further regions are determined based on the rectangle bounding box determined at step 3005), which is classified as a fully opaque primitive. Therefore, the above ""mentioned compositing optimisations can applied where the rectangle bounding box is to be composited. During any subsequent rendering of the further image any graphical primitive which is dependent on the region defined by the rectangle bounding box can effectively be ignored where the graphical primitive falls within the region defined by the rectangle bounding box, since the region defined by the rectangle bounding box is classified as a fully opaque primitive.

The process of flow chart 3000 continues at the next step 3010, where the further regions are classified according to attributes of the primitives that fall within the further regions. The further regions are classified according to whether the primitives are wholly opaque, wholly transparent or otherwise. At the next step 3011, the compositing expressions corresponding to each of the further regions, are modified according to a classification of each of the further regions. The modifications form an augmented compositing expression for each of the further regions in a manner similar to the method of Fig. 23 described above.

At the next step 3013, for one or more of the further regions, the region outline and any attributes associated with the further regions are encoded into one or more rendering instructions that completely define the operations needed to render the image defined by the further regions. The rendering instructions are determined in accordance with the augmented compositing expression for each of the further regions. As described above, the rendering instructions preferably comprise proxy information. The proxy information preferably represents the next line to be updated, the first column to be updated and the number of pixels to follow, followed by the colour of each pixel.

Further, as described above, if groups of pixels are all of the one colour, then each 546904.doc -66individual pixel does not need to be transmitted, simply a proxy representing a particular mutually exclusive region and the colour to paint the region need to be transmitted.

Further, where the attributes of two or more further regions are the same, then preferably one set of attributes referencing the two or more further regions need to be transmitted.

Also included in the rendering instructions is a proxy configured to mark out the region formed by the rectangle within which the video image is to be displayed.

The process of flow chart 3000 continues at the next step 3015, where the CPU 2809, under control of a computer program stored in ROM 2811 or running out of general purpose RAM 2813, transfers image data from the local frame buffer 2827 to memory positions in the local display buffer 2829, such that the memory positions correspond to the rectangle region of the local display 2807. For example, referring to Fig. 29, this causes the image captured by the video camera to appear in the rectangle region 2904 of the local display 2807, whilst the remainder of the display area 2902 will be filled with i :the further image defined by rendering instructions to be received from the host computer 2702 via the communication interface 2703. At the next step 3017, the rendering instructions determined at step 3013 are transmitted to the remote graphical display, via the bandwidth limited communications link, so that the further image can be constructed.

The process of flow chart 3000 concludes when the entire further image has been constructed.

In accordance with a further arrangement, where a video image captured by a video camera 2803) is to be rendered on a remote multimedia terminal along with a further image to be provided by a host computer, the space in which the images are to be defined can be divided into a number of mutually exclusive regions and each of the S. regions can be processed independently. For example, Fig. 31 shows the local display 2807 of the multimedia terminal 2805 with a rectangular region 3101 within which a captured video image can be displayed. Once the rectangle region 3101 has been defined as discussed above with reference to step 3005, the local display 2807 can be divided into four regions comprising the areas 3105, 3107, 3109 and the rectangle region 3101, defined by the broken lines 3111, 3113. The further image provided by the host computer will be displayed on the local display 2807, with each of the areas 3105, 3107 and 3109 defining a corresponding part of the further image. Each of the areas 3105, 3107 and 3109 can have a separate compositing tree each representing the part of the further image to be display within the respective area. The separate compositing trees can each be independently processed in accordance with the method of Fig. 23 described above, in order to construct the further image on the remote graphical display. The video image 546904.doc -67can subsequently be rendered within the rectangular region 3101 in the manner described above.

In accordance with a still further arrangement, the method utilised for rendering a digital image captured by a video camera of a multimedia terminal that is remotely positioned with respect to a host computer associated with the multimedia terminal, can be determined dynamically based on predetermined user defined criteria. For example, if the rectangle region 3101) within which a captured video image is to be displayed, is positioned in one comer of the display, as shown in Fig. 31, then the remaining image space can be divided and independently processed as described above with reference to Fig. 31. Alternatively, if the rectangle region 3101) within which a captured video image is to be displayed, is positioned in the centre (not shown) of the display then the remaining image space can rendered in accordance with the method of Fig. Preferred Apparatus The methods described herein can be implemented using a general-purpose computer system 2100 as the host computer, such as that shown in Fig. 19, wherein the process described with reference to Figs. 2 to 18 and 21 to 31 can be implemented as software recorded on a computer readable medium that can be loaded into and carried out by the computer. In particular, the steps of the described methods are effected by instructions in the software that are carried out by the computer. The instructions may be formed as one or more code modules, each for performing one or more particular tasks.

The computer system 2100 includes a computer module 2101, input devices 2118, 2120, a display device 2116 and a remote graphics display device 2130.

S. With reference to Fig. 19, the computer module 2101 includes at least one processor unit 2105, a memory unit 2106 which typically includes random access memory (RAM) and read only memory (ROM), input/output interfaces including a high speed communication interface 2107, a video interface 2104, a keyboard and mouse interface 2108 and an I/O interface 2110. The storage device 2109 can include one or more of the following devices: a floppy disk, a hard disk drive, a CD-ROM drive or similar a non-volatile storage device known to those skilled in the art. The components, 2104 to 2110 of the computer module 2101, typically communicate via an interconnected bus 2114 and in a manner which results in a usual mode of operation of the computer system 2100 known to those in the relevant art. Examples of computer systems on which the described methods can be practised include IBM-PCI ATs and compatibles, Sun Sparcstations or alike computer system. In particular, the pseudocode described herein 546904.doc -68can be programmed into any suitable language and stored for example on the HDD and executed in the RAM 2106 under control of the processor 2105 with the results being transmitted via the high speed communication interface 2107 to the remote graphics display 2130. The programs can be supplied to the system 2100 on a pre-programmed floppy disk or CD-ROM or accessed via a connection with a computer network, such as the Internet.

The aforementioned preferred methods comprise a particular control flow.

There are many other variants of the preferred methods which use different control flows without departing the spirit or scope of the invention. Furthermore one or more of the steps of the preferred methods may be performed in parallel rather sequentially.

The foregoing describes only several examples of the described methods, and modifications, obvious to those skilled in the art, can be made thereto without departing from the scope of the present invention. For example, in accordance with a still further example of the described methods, the portions of an image which need to be rendered or re-rendered to update a remote graphical display can be determined without considering the opacity or region dependency of particular regions of the image. In this instance, those regions which are required to be updated can then be packaged and transmitted to a remote display.

In the context of this specification, the word "comprising" means "including principally but not necessarily solely" or "having" or "including" and not "consisting only of'. Variations of the word comprising, such as "comprise" and "comprises" have corresponding meanings.

S.

99 546904.doc

Claims (3)

1. A method of rendering at least a first and second image on a graphical display that is remotely spaced from a computer, said computer being connected to said display via a communications link, said second image being formed by rendering and compositing at least a plurality of graphical primitives according to a compositing expression, said method comprising the steps of: capturing said first image on a camera connected to said display; determining a bounding area within which said first image is to be rendered; transmitting a description of said bounding area to said computer; determining at least one region for one or more of said graphical primitives, said region representing at least a portion of a corresponding primitive outside of which said corresponding primitive is transparent; manipulating said regions to determine a plurality of further regions, said

7. 15 further regions being dependent on said description of said bounding area, wherein each said further region comprises at least one contributing primitive; encoding said further regions into at least one rendering instruction substantially representing said contributing primitives; rendering said first image within said bounding area utilising said graphical 20 display; and transmitting said rendering instructions via said communications link to said graphical display in order to render said second image. 2. The method according to claim 1, said method comprising the further steps, in 25 response to at least one change to said second image, of: identifying any graphical primitives affected by said change; determining at least one region for one or more of said affected graphical primitives, said region representing at least a portion of a corresponding affected primitive outside of which said corresponding affected primitive is transparent; manipulating said regions representing at least a portion of a corresponding affected primitive to determine a plurality of still further regions, wherein each said still further region comprises at least one contributing affected primitive; encoding said still further regions into at least one further rendering instruction substantially representing said contributing affected primitives; and transmitting said further rendering instructions via said communications link

546904.doc to said graphical display. 3. The method according to claim 2, wherein each of said further regions and said still further regions are converted into a corresponding rectangle substantially representing a corresponding further region or still further region. 4. The method according to claim 3, wherein said corresponding rectangle comprises at least a header section and a body section. 5. The method according to claim 4, wherein said header section substantially represents the position of the rectangle in the image. 6. The method according to claim 4, wherein said body section comprises attributes for displaying said rectangle on said display screen. S7. The method according to claim 3, wherein said corresponding rectangle is classified as one of three types. S8 The method according to claim 7, wherein said corresponding rectangle is 20 displayed based on said classification. 9. The method according to any one of claims 3 to 7, wherein said corresponding rectangle comprises a plurality ofpixels. 25 10. The method according to claim 1, wherein said manipulating said regions comprises applying set operations to said regions. 11. The method according to claim 10, wherein said set operations include difference and/or intersection operations. 12. The method according to any one of claims 1 to 11, wherein the compositing expression is a hierarchically structured representation of the second image. 13. The method according to any one of the preceding claims, wherein a flag is stored to indicate whether data of a primitive is opaque or ordinary. 546904.doc -71 14. The method according to claim 13, wherein said compositing expression is optimised based on a value of said flag for contributing primitives. 15. The method according to any one of the preceding claims, wherein a wholly opaque primitive in a region acts to eliminate one or more primitives within a region from a compositing expression representing said region. 16. A method according to any one of the preceding claims, wherein a wholly transparent primitive in a region eliminates at least itself from a compositing expression representing said region. 17. A method according to any one of claims 1 to 16, wherein a region defined by said bounding area is classified as a wholly opaque primitive. 18. A method of creating an image on a graphical display that is remotely spaced from a computer, said computer being connected to said display via a communications link, said image being formed by rendering and compositing at least a plurality of graphical primitives according to a compositing expression, said method comprising the steps of: determining at least one region for one or more of said graphical primitives, said region representing at least a portion of a corresponding primitive outside of which said corresponding primitive is transparent; S:i manipulating said regions to determine a plurality of further regions, wherein 25 each said further region comprises at least one contributing primitive; encoding said further regions into at least one rendering instruction substantially representing said contributing primitives; and transmitting said rendering instructions via said communications link to said graphical display in order to create said image. 19. The method according to claim 18, said method comprising the further steps, in .response to at least one change to said image, of: identifying any graphical primitives affected by said change; determining at least one region for one or more of said affected graphical 35 primitives, said region representing at least a portion of a corresponding affected 546904.doc 72- primitive outside of which said corresponding affected primitive is transparent; manipulating said regions representing at least a portion of a corresponding affected primitive to determine a plurality of still further regions, wherein each said still further region comprises at least one contributing affected primitive; encoding said still further regions into at least one further rendering instruction substantially representing said contributing affected primitives; and transmitting said further rendering instructions via said communications link to said graphical display in order to create a further image. 20. The method according to claim 19, wherein each of said further regions and said still further regions are converted into a corresponding rectangle substantially representing a corresponding further region or still further region. 21. The method according to claim 20, wherein said corresponding rectangle comprises at least a header section and a body section. 22. The method according to claim 21, wherein said header section substantially represents the position of the rectangle in the image. 23. The method according to claim 21, wherein said body section comprises attributes for displaying said rectangle on said display screen. 24. The method according to claim 21, wherein said corresponding rectangle is "classified as one of three types. 25. The method according to claim 24, wherein said rectangle is displayed based on gsaid classification. 26. The method according to any one of claims 20 to 24, wherein said rectangle comprises a plurality of pixels. 27. The method according to claim 18, wherein said manipulating said regions oooo comprises applying set operations to said regions. .oolite a 35 28. The method according to claim 27, wherein said set operations include S546904.d 546904.doc 73 difference and/or intersection operations. 29. The method according to any one of claims 18 to 28, wherein the compositing expression is a hierarchically structured representation of the image. The method according to any one of claims 18 to 29, wherein a flag is stored to indicate whether data of a primitive is opaque or ordinary. 31. The method according to claim 30, wherein said compositing expression is optimised based on a value of said flag for contributing primitives. 32. The method according to any one of claims 18 to 31, wherein a wholly opaque primitive in a region acts to eliminate one or more primitives within a region from a compositing expression representing said region. 33. A method according to any one of claims 18 to 32, wherein a wholly transparent primitive in a region eliminates at least itself from a compositing expression representing said region. 34. A method of creating an image on a graphical display that is remotely spaced from a computer associated with said display, said computer being connected to said display via a communications link, said image being formed by rendering and compositing at least a plurality of graphical primitives, each said primitive having a i predetermined outline, said method comprising the steps of: dividing a space in which said outlines are defined into a plurality of regions, each said region being defined by at least one region outline substantially following at least one of said predetermined outlines or parts thereof; manipulating said regions to determine a plurality of further regions, wherein each said further region has a corresponding compositing expression; classifying said further regions according to at least one attribute of said graphical primitives within said further regions; modifying each said corresponding compositing expression according to a classification of each said further region to form an augmented compositing expression o for each said further region; and encoding said further regions, according to said augmented compositing 546904.doc -74- expressions, into a plurality of rendering instructions; and transmitting said plurality of rendering instructions via said communications link to said graphical display in order to create said image. 35. A method of creating a series of images on a graphical display that is remotely spaced from a computer associated with said display, said computer being connected to said display via a communications link, each member of said series being related to a preceding member, said images being formed by rendering a plurality of graphical primitives to be composited according to a hierarchical structure representing a compositing expression, said hierarchical structure including a plurality of nodes each representing a component of at least one of said images, each of said primitives having a predetermined outline, said method comprising the steps of: for each said node: dividing a component image space in which said outlines are defined into at least one mutually exclusive region, each said region being related to at least one graphical primitive; (ii) examining each said region to determine those primitives that contribute to the region; creating internodal dependency information identifying those said regions that will be affected by a change in any one of said regions; encoding all of those primitives that contribute to said regions, substantially according to said hierarchical structure, into a plurality of rendering instructions representing said first image; :i transmitting said plurality of rendering instructions via said 1 25 communications link to said graphical display in order to create said first image; S(e) in response to at least one change to at least one of said nodes; examining said internodal dependency information to identify those of said regions affected by said at least one change; (ii) for each node with affected regions, updating the corresponding identified regions and incorporating into said node those (any) new regions arising from the change and/or removing any of said regions that are no longer relevant; (iii) updating said intemrnodal dependency information to reflect o:o changes to said hierarchical structure; (iv) examining each said identified region to determine those 35 primitives that contribute to the identified region; 546904.doc 75 encoding all of those primitives that contribute to said identified regions, substantially according to said hierarchical structure, into a plurality of rendering instructions representing said identified regions; (vi) transmitting said rendering instructions representing said identified regions, via said communications link, to said graphical display in order to render a further image of said series; and repeating step for further changes to at least one of said nodes. 36. The method according to claim 35, wherein each of said primitives contributing to a region is converted into a corresponding rectangle substantially representing said region. 37. The method according to claim 36, wherein said corresponding rectangle comprises at least a header section and a body section. 38. The method according to claim 37, wherein said header section substantially represents the position of the rectangle in the image. 39. The method according to claim 37, wherein said body section comprises attributes for displaying said rectangle on said display screen. The method according to claim 36, wherein said corresponding rectangle is classified as one of three types. 41. The method according to claim 40, wherein said rectangle is displayed based on said classification. 42. The method according to any one of claims 36 to 41, wherein said rectangle comprises a plurality ofpixels. 43. The method according to claim 35, wherein said manipulating said regions 4. comprises applying set operations to said regions. o 44. The method according to claim 43, wherein said set operations include oooo 35 difference and/or intersection operations. 546904.doc 76- The method according to any one of claims 35 to 44, wherein the compositing expression is a hierarchically structured representation of the image. 46. The method according to any one of claims 35 to 45, wherein a flag is stored to indicate whether data of a primitive is opaque or ordinary. 47. The method according to claim 46, wherein said compositing expression is optimised based on a value of said flag for contributing primitives. 48. The method according to any one of claims 35 to 47, wherein a wholly opaque primitive in a region acts to eliminate one or more primitives within said region from said compositing expression. 49. The method according to any one of claims 35 to 48, wherein a wholly transparent primitive in a region eliminates at least itself from said compositing expression. A method of creating a series of images on a graphical display that is remotely spaced from a computer associated with said display, said computer being connected to said display via a communications link, each member of said series being related to a preceding member, said images being formed by rendering a plurality of graphical primitives to be composited according to a compositing expression, each said primitive having a predetermined outline, said method comprising the steps of: 25 dividing a component image space in which said outlines are defined into *at least one mutually exclusive region, each said region being related to at least one graphical primitive; examining each said region to determine those primitives that contribute to the region; encoding all of those primitives that contribute to said regions into a plurality of rendering instructions representing a first image; transmitting said plurality of rendering instructions, via said communications link, to said graphical display in order to create said first image; iet in response to at least one change to at least one of said nodes; *5SS S. 5 546904.doc 77 identifying those of said regions affected by said at least one change; (ii) for each node with affected regions, updating the corresponding identified regions and incorporating into said node those (any) new regions arising from the change and/or removing any of said regions that are no longer relevant; (iii) examining each said identified region to determine those primitives that contribute to the identified region; (iv) encoding all of those primitives that contribute to said identified regions into a plurality of rendering instructions representing said identified regions; (vi) transmitting said rendering instructions representing said identified regions, via said communications link, to said graphical display in order to render a further image of said series; and repeating step for further changes to at least one of said nodes. 51. The method according to any one of the preceding claims, wherein said graphical display is serially connected to said remotely spaced computer. 52. A computer program for a computer, comprising software code portions for performing the steps of any one of claims 1 to 51. 53. A computer readable medium storing a computer program, wherein said computer program comprises software code portions for performing the steps of any one of claims 1 to 51. 54. An apparatus for creating an image on a graphical display that is remotely spaced from a computer, said computer being connected to said display via a communications link, said image being formed by rendering and compositing at least a plurality of graphical primitives according to a compositing expression, said apparatus comprising: means for determining at least one region for one or more of said graphical primitives, said region representing at least a portion of a corresponding primitive outside •of which said corresponding primitive is transparent; o means for manipulating said regions to determine a plurality of further regions, wherein each said further region comprises at least one contributing primitive; 3gsr 35 means for encoding said further regions into at least one rendering instruction 666666 6o 6 546904.doc -78- substantially representing said contributing primitives; means for transmitting said rendering instructions via said communications link to said graphical display in order to create said image; means for capturing a further image on a camera connected to said display; means for displaying said further image and a still further image on said display. An apparatus according to claim 54, said apparatus further comprising: means for identifying any graphical primitives affected by at least one change to said image; means for determining at least one region for one or more of said affected graphical primitives, said region representing at least a portion of a corresponding affected primitive outside of which said corresponding affected primitive is transparent;. means for manipulating said regions representing at least a portion of a corresponding affected primitive to determine a plurality of still further regions, wherein each said still further region comprises at least one contributing affected primitive; means for encoding said still further regions into at least one further rendering instruction substantially representing said contributing affected primitives; and means for transmitting said further rendering instructions via said communications link to said graphical display. 56. An apparatus for creating an image on a graphical display that is remotely spaced from a computer, said computer being connected to said display via a communications link, said image being formed by rendering and compositing at least a plurality of graphical primitives according to a compositing expression, said apparatus 25 comprising: means for determining at least one region for one or more of said graphical primitives, said region representing at least a portion of a corresponding primitive outside of which said corresponding primitive is transparent; means for manipulating said regions to determine a plurality of further regions, wherein each said further region comprises at least one contributing primitive; means for encoding said further regions into at least one rendering instruction •substantially representing said contributing primitives; and means for transmitting said rendering instructions via said communications link o to said graphical display in order to create said image. 546904.doc -79- 57. An apparatus according to claim 56, said apparatus further comprising: means for identifying any graphical primitives affected by at least one change to said image; means for determining at least one region for one or more of said affected graphical primitives, said region representing at least a portion of a corresponding affected primitive outside of which said corresponding affected primitive is transparent; means for manipulating said regions representing at least a portion of a corresponding affected primitive to determine a plurality of still further regions, wherein each said still further region comprises at least one contributing affected primitive; means for encoding said still further regions into at least one further rendering instruction substantially representing said contributing affected primitives; and means for transmitting said further rendering instructions via said communications link to said graphical display in order to create a further image. 58. An apparatus according to claim 57, wherein each of said further regions and said still further regions are converted into a corresponding rectangle substantially representing a corresponding further region or still further region. 59. An apparatus according to claim 58, wherein said corresponding rectangle comprises at least a header Section and a body section. An apparatus according to claim 59, wherein said header section substantially represents the position of the rectangle in the image. S 25 61. An apparatus according to claim 59, wherein said body section comprises *o0* attributes for displaying said rectangle on said display screen. 62. An apparatus for creating an image on a graphical display that is remotely spaced from a computer associated with said display, said computer being connected to said display via a communications link, said image being formed by rendering and compositing at least a plurality of graphical primitives, each said primitive having a ao° predetermined outline, said apparatus comprising: ooo .means for dividing a space in which said outlines are defined into a plurality of oa. a regions, each said region being defined by at least one region outline substantially 3ado 35 following at least one of said predetermined outlines or parts thereof; 546904.doc means for manipulating said regions to determine a plurality of further regions, wherein each said further region has a corresponding compositing expression; means for classifying said further regions according to at least one attribute of said graphical primitives within said further regions; means for modifying each said corresponding compositing expression according to a classification of each said further region to form an augmented compositing expression for each said further region; and means for encoding said further regions, according to said augmented compositing expressions, into a plurality of rendering instructions; and means for transmitting said plurality of rendering instructions via said communications link to said graphical display in order to create said image. 63. An apparatus for creating a series of images on a graphical display that is remotely spaced from a computer associated with said display, said computer being connected to said display via a communications link, each member of said series being related to a preceding member, said images being formed by rendering a plurality of graphical primitives to be composited according to a hierarchical structure representing a compositing expression, said hierarchical structure including a plurality of nodes each representing a component of at least one of said images, each of said primitives having a predetermined outline, said apparatus comprising means for performing the following steps: for each said node: dividing a component image space in which said outlines are :i defined into at least one mutually exclusive region, each said region being related to at 25 least one graphical primitive; (ii) examining each said region to determine those primitives that contribute to the region; creating internodal dependency information identifying those said regions that will be affected by a change in any one of said regions; encoding all of those primitives that contribute to said regions, substantially according to said hierarchical structure, into a plurality of rendering S•instructions representing said first image; transmitting said plurality of rendering instructions via said communications link to said graphical display in order to create said first image; go• in response to at least one change to at least one of said nodes; 546904.doc -81- examining said intemrnodal dependency information to identify those of said regions affected by said at least one change; (ii) for each node with affected regions, updating the corresponding identified regions and incorporating into said node those (any) new regions arising from the change and/or removing any of said regions that are no longer relevant; (iii) updating said internodal dependency information to reflect changes to said hierarchical structure; (iv) examining each said identified region to determine those primitives that contribute to the identified region; encoding all of those primitives that contribute to said identified regions, substantially according to said hierarchical structure, into a plurality of rendering instructions representing said identified regions; (vi) transmitting said rendering instructions representing said identified regions, via said communications link, to said graphical display in order to render a further image of said series; and repeating step for further changes to at least one of said nodes. 64. A method of rendering at least a first and second image on a graphical display that is remotely spaced from a computer, said computer being connected to said display via a communications link, said second image being formed by rendering and compositing at least a plurality of graphical primitives according to a compositing expression, said method comprising the steps of: capturing said first image on a camera connected to said display; :i determining a bounding area within which said first image is to be rendered; transmitting a description of said bounding area to said computer; dividing a space in which the second image is to be rendered into at least one mutually exclusive region depending on said bounding area, and for each of the mutually exclusive regions performing the following steps to determining at least one further region for one or more of said graphical primitives falling within said further region representing at least a portion of a corresponding primitive outside of which said corresponding primitive is transparent; manipulating said further regions to determine a plurality of still further regions, wherein each said still further region comprises at least one 35 contributing primitive; •seS 546904.doc 82 (d)(iii) encoding said still further regions into at least one rendering instruction substantially representing said contributing primitives; transmitting said rendering instructions via said communications link to said graphical display in order to render a portion of said second image within said mutually exclusive region; and rendering said first image within said bounding area utilising said graphical display. The method according to any one of claims 1, 18, 34, 35, 50 or 64, wherein said communications link is bandwidth limited. 66. An apparatus according to any one of claims 54, 56, 62 or 63, wherein said communications link is bandwidth limited. 67. A method of rendering at least a first and second image on a graphical display that is remotely spaced from a computer, said method being substantially as herein described with reference to any one of the embodiments as that embodiment is described in Figs. 4 to 31. 68. A method of creating an image, substantially as herein described with reference to any one of the embodiments as that embodiment is described in Figs. 4 to 31. 69. An apparatus for creating an image, substantially as herein described with reference to any one of the embodiments as that embodiment is described in Figs. 4 to 31. @SWS 70. A multimedia terminal, connected to a remote host computer through a connection and for displaying on a local display an image captured on a camera connected to said multimedia terminal, said multimedia terminal being substantially as herein described with reference to any one of the embodiments as that embodiment is described in Figs. 4 to 31. 546904.doc S. 55 S OS.. 54694.do -83 71. A computer readable medium having a program recorded thereon, said program comprising a plurality of software modules adapted for interactive operation on at least one computer platform, said program being substantially as herein described with reference to any one of the embodiments as that embodiment is described in Figs. 4 to 31. DATED this the Twelfth Day of September 2003 Canon Kabushiki Kaisha Patent Attorneys for the Applicant SPRUSON FERGUSON 9 0 @0 0 0*0S *0 S* 546904.doc