AU2009243389A1 - Rewind edge placement - Google Patents

Description

S&F Ref: 920391 AUSTRALIA PATENTS ACT 1990 COMPLETE SPECIFICATION FOR A STANDARD PATENT Name and Address Canon Kabushiki Kaisha, of 30-2, Shimomaruko 3 of Applicant: chome, Ohta-ku, Tokyo, 146, Japan Actual Inventor(s): Richard John Vagg Address for Service: Spruson & Ferguson St Martins Tower Level 35 31 Market Street Sydney NSW 2000 (CCN 3710000177) Invention Title: Rewind edge placement The following statement is a full description of this invention, including the best method of performing it known to me/us: 5845c(2413634_1) - 1 REWIND EDGE PLACEMENT TECHNICAL FIELD The current invention relates to rendering systems and, in particular, to rendering systems having limited and/or fixed resource constraints. BACKGROUND 5 in modem 2D computer graphics, rendering is the process of generating raster images based on the description of a desired artwork, often made of graphical objects. The description is usually written in a strictly defined graphical description language, such as POSTSCRIPT or SVG. Such graphical description languages are commonly used in UI (user interface) creation and the printing arts, for example. There are a 10 number of alternative algorithms that may be used for rendering, such algorithms varying based on the rendering sequence of the graphical objects. In one such rendering algorithm, for graphical objects received in z-order, the entire description of the artwork is interpreted as a whole. Each graphical object contained in the artwork is decomposed into low level primitives (e.g. points or edges), a 15 process sometimes referred to as vectorization. These primitives are stored in a list constructed to hold all data and operations needed for rendering, called a display list. Upon completion of the construction of the display list, the renderer renders the display list and generates a final image, typically in raster scan order (i.e. processing pixels sequentially within a scan line and progressing sequentially through scan lines upon a 20 screen or page). A drawback of this rendering algorithm is that it can be resource hungry. Since the display list contains the data of the entire artwork, as the artwork becomes complex, the size of the display list can grow to be very large. Memory resources, such as required to store the display list, can easily run out, especially in an 2413251_1 920391_speci_lodge -2 embedded environment, before rendering even takes place. As a result, renderer design often has to be dependent on a set of predefined artworks to prevent an out-of-resource memory condition. Predefined artworks may use a known amount of memory resources and may be processed in manner that takes into account the known amount of memory 5 resources to be used.. One technique intended to solve this issue is called fallback rendering. Fallback rendering renders the display list, containing a portion of the artwork, to a temporary bitmap image once available memory resources are consumed. The temporary bitmap image, often referred to as a fallback image, is re-inserted into the display list as an 10 object to be rendered with the remaining artwork. This process is repeated until the entire artwork is completely rendered to a final bitmap image. Though fallback rendering has proven to be effective in rendering relatively complex images using limited memory, it has to be constrained to the object-level. In other words, fallback rendering will only work when the display list contains complete objects. If memory 15 resources run out in the middle of decomposing an object to primitives, triggering a fallback is likely to result in undefined behaviour and consequently a distorted or erroneous final image. Several approaches have improved the fallback technique to better address the resource problem. One approach is to make estimates before adding each object to the 20 display list to determine if there are enough resources to accommodate the object to be added. If resources are likely to run out, the display list is fallback rendered. This approach has proven to be useful, but at a cost to performance. The process of estimation is complex and typically impacts on the overall speed, especially in the non fallback case. Another approach is to allocate memory from elsewhere, often a "swap" 2413251_1 920391_speci_lodge -3 space, to compensate for the resource shortage. However, both these approaches may use more memory than is needed for the actual render, which is undesirable in an embedded environment. Moreover, the second approach is also likely to be slow as swapping is often done with a memory hard drive, where such is available. 5 Therefore, what is needed is a solution that is both speed efficient and memory efficient. SUMMARY It is an object of the present invention to substantially overcome, or at least ameliorate, one or more deficiencies of existing rendering arrangements. 10 In accordance with one aspect of the present disclosure, there is provided a computerized method of rendering a plurality of objects to form an image, the method being performed by a processor in association with a memory, said method comprising the steps of: (a) forming a display list describing the image in the memory by processing 15 the objects sequentially and in z-order, said processing comprising, for each object: (aa) generating at least one primitive for the object, said generating comprising determining if a resource in the memory for the primitive is available and adding the primitive to the display list if the memory resource is determined as available; (ab) where the memory resource is determined as not available: 20 (aba) reversing effects upon the display list of any added primitives of the object; (abb) rendering the display list to pixel data as a fallback image; (abc) clearing the display list; (abd) adding the fallback image to the display list; and 2413251_1 920391_speci_lodge -4 (abe) repeating step (aa) for the received object; and (b) rendering the display list as pixel data. According to anther aspect, there is disclosed a computerized method of rendering a plurality of objects to form an image, the method being performed by a 5 processor in association with a memory, said method comprising the steps of: (a) forming a display list describing the image in the memory by processing the objects sequentially and in z-order, said processing comprising: (aa) receiving a z-ordered object; (ab) generating at least one primitive for the received object, said 10 generating comprising, for each primitive generated: (aba) determining if a resource in the memory for the primitive is available; (abb) adding, if the memory resource is determined as available, the primitive to the display list; 15 (abc) if the memory resource is determined as not available: (abca) reversing effects upon the display list of any added primitives of the received object; (abcb) rendering the display list to pixel data as a fallback image; 20 (abec) clearing the display list; (abcd) adding the fallback image to the display list; and (abce) repeating step (ab) for the received object; and (b) rendering the display list as pixel data. 2413251_1 920391_spedlodge -5 According to anther aspect, there is provided a computerized method of rendering a plurality of objects to form an image, the method being performed by a processor in association with a memory, said method comprising the steps of: (a) forming a display list describing the image in the memory by processing 5 the objects sequentially and in z-order, said processing comprising: (aa) receiving a z-ordered object; (ab) generating at least one primitive for the received object, said generating comprising, for each primitive generated: (aba) determining if a resource in the memory for the primitive 10 is available; (abb) adding, if the memory resource is determined as available, the primitive to the display list; (abc) setting, if the memory resource is determined as not available, a flag indicating fallback render is required; 15 (ac) testing the flag, and where the flag is set: (aca) reversing effects upon the display list of any added primitives of the received object; (acb) rendering the display list to pixel data as a fallback image; (acc) clearing the display list; 20 (acd) adding the fallback image to the display list; and (ace) repeating step (ab) for the received object; and (b) rendering the display list as pixel data. Other aspects are also disclosed. 2413251_1 920391_speci_lodge -6 BRIEF DESCRIPTION OF THE DRAWINGS At least one embodiment of the present invention will now be described with reference to the following drawings, in which: Fig. I is a schematic flow diagram illustrating a method of rendering objects, in 5 which the arrangements to be described may be practised; Fig. 2 is a schematic flow diagram illustrating a conventional method of adding an object to a display list, corresponding to step 120 in Fig. 1; Figs. 3A and 3B schematically illustrate the out of resource problem during edge placement; 10 Figs. 4A to 4G schematically illustrate an exemplary process, presented with reference to Figs. 3A and 3B, to avoid the out of resource problem; Fig. 5 is a schematic flow diagram illustrating a first method, of adding an object to a display list; Fig. 6 is a schematic flow diagram illustrating an out of edges fallback render 15 method in accordance with the first method; Fig. 7 illustrates a simplified function call tree in which an edge is added in four different locations during processing an object. Fig. 8 is a schematic flow diagram illustrating a second method, of adding an object to a display list; 20 Fig. 9 is a schematic flow diagram illustrating a third method, of adding an object to a display list including sorting bins; Fig. 10 is a schematic flow diagram illustrating an out of edges fallback render method including sorting bins in accordance with the third method; 2413251_1 920391_specilodge -7 Fig. II is a schematic flow diagram illustrating a fourth method, of finalising the display list in which placing the final clip edges uses reserved edges; Fig. 12 is a schematic block diagram of a rendering apparatus for implementing the described methods; 5 and Figs. 13A and 13B collectively form a schematic block diagram of a general purpose computer on which the arrangements described may be practised. DETAILED DESCRIPTION INCLUDING BEST MODE The following description is presented, for the sake of simplicity, using straight 10 line edges as low level primitives. However, the arrangements and methods described herein are not limited in use to straight-line edges, but however are applicable to any type of primitive, such as points or curves. Fig. I is a schematic flow diagram illustrating a processing method 100 of a typical renderer, in which the arrangements described herein may be applied. In an entry 15 step 110, the rendering process 100 commences by receiving descriptions of a graphic object of a desired artwork. The descriptions are typically received sequentially in z order. The received description often contains a vector path outlining the object shape and an object color. In the following step 120, the received object is added to the display list, which is stored in a memory associated with a computerized device implementing 20 the process 100. The process of adding includes decomposing the object into straight line edges and storing them in the display list, this sub-process often being referred to as edge placement. For example, a Bezier curve is typically vectorized into straight line edges, and each edge is stored in memory in the display list. A test 130 is then made to determine if there are any more objects to be added. If there are, the processing goes 2413251_1 920391_speci_lodge -8 back to the entry step 110 to receive a next object, otherwise, when the accumulation of objects in the display list is complete to thus describe a single image to be rendered, the process 100 continues to the next step 140 to finalise the display list in preparation for render. Typically, the finalising process 140 includes a step of sorting the primitives in 5 y-order and x-order, so that each primitive can be easily tracked and rendered to an output, such as a memory, a display, or a printer, in an orderly fashion. Upon completion of the construction of the display list, a final step 150 renders the display list to a pixel based (bitmap) image, which is then output, for example to a display or to a storage in a memory. 10 Fig. 2 is a detailed flow diagram of a conventional (prior art) method 200 of the process 120 of adding an object to the display list. The processing of the method 200 commences with an entry step 210, that performs a series of setup operations on the received object to initiate the object decomposition. The setup operations typically include initialising data structures and transforming object paths. A loop is then started 15 by making a first test 220 to determine whether there is any edge left in the object to be added to the display list. If there is no edge left the processing exits. If there is an edge left, being a current edge, then a second test 240 is made to see if resources are available for storing the current edge. If resources have run out, the processing quits thus omitting the current edge from the display list. Otherwise the processing proceeds to a next step 20 260 which places the edge in the display list. Once the edge is placed, the processing then loops back to the first test 220 for a next edge. A predefined edge data structure is used to store the edge data. The edge data structure is usually allocated from a pool of resources and initialised with predefined 2413251_1 920391_speci_lodge -9 values. The data structure may also include other rendering data needed for render, such as object color and z-order. The display list is a collection of such data structures. Resources can run out, especially in an embedded environment, in the midst of processing an object. This presents a problem, as a portion of the edges of the 5 unfinished object still remain in the display list, which is likely to result in an undefined rendering behaviour. An example of such an occurrence of the problem is seen in Figs. 3A and 3B, which illustrate three objects, 321, 322 and 323 of an input artwork 320 in Fig. 3A, being added to a display list 340 seen in Fig. 3B. A bottom object 321 has five edges and the bottom z-order, followed by a middle object 322 with four edges, and a top 10 object 323 has top z-order, also with four edges. In this example, the display list 340 has a maximum capacity of eleven edges, given the resources available. When the bottom and middle objects, 321 and 322, are added, they are decomposed into edges, 301 and 302 and occupy nine edge spaces in the display list 340. As the top object 323 is being added to the display list 340, only two edges 303 of its total four edges are able to be 15 accommodated, and no resources are available to accommodate the last two edges 304. As a result, the display list 340 can only be rendered with the bottom object 321, middle object 322 and only a portion of the top object 323. If the portion of the top object 323 is rendered, the result may be an image with an incomplete object. An incomplete object may lead to undefined results in the image that can result in visible errors in the rendered 20 image. The present disclosure provides methods to solve the problem of rendering incomplete objects. By way of summary, the methods operate as follows according to two steps: (i) Re-winding edges, and 2413251_1 920391_specilodge -10 (ii) Fallback rendering. When an out-of-edge-resource condition is encountered, all placed edges of the unfinished object are rewound. By rewinding, the state of the renderer is reverted to the state before the unfinished object was added. In other words, rewinding effects a 5 reversing or reversal of processes that have been performed in relation to the unfinished object. To achieve this, any operations performed for placing these edges are undone, this includes removing or discarding their corresponding data structures from the display list. Following rewinding, the display list contains only complete objects, and is fallback rendered to release the memory resources. Depending on the z-order of the receiving 10 sequence of the objects, the rendered fallback image is inserted into the display list with either the top or bottom z-order. For example, if the objects are received in the bottom up z-order, the fallback image is added to the display list as the bottom-most object. In a first implementation, edge rewinding and fallback rendering are performed within the process 120 of adding an object to the display list and executed immediately after an out 15 of-edge-resource condition. Figs. 4A through 4G demonstrate an example of how the out of edge resource condition is dealt with by the arrangements according to the present disclosure. Fig. 4A depicts the same out-of-edge-resources condition as explained in Figs. 3A and 3B, in which two edges 304 of the top object 323 are not placed into the display list 340, as 20 there are not enough resources available to the top object 323 after the bottom object 321 and the middle object 322 are added to the display list 340. When this happens, the placed edges 303 of the top object 323 are rewound from the display list 340 and as a result, the display list 340, as seen in Fig. 4B, only contains edges 301 and 302 of the bottom object 321 and the middle object 322. The display list 340 is then fallback 2413251_1 920391_speci_lodge - 11 rendered to a fallback image 460, as illustrated in Fig. 4C, before being cleared as seen in Fig. 4D. Following the fallback rendering, the edges 401 of the fallback image 460 are inserted into the display list 340 as the bottom object, as seen in Fig. 4E. The previously unfinished top object 323 is then (again) added to the display list 340, as seen 5 in Fig. 4F, however on this occasion there are sufficient resources available in the display list 340 to store all the edges 402 of the object 323. The final raster scan image of the artwork 320, as seen in Fig. 4G, is then obtained by rendering the display list 340 of Fig. 4F. Fig. 12 is a schematic block diagram of a rendering apparatus 1200 that may be 10 used to implement the arrangements described herein. The apparatus 1200 includes a display list generator module 1210, an edge sorter module 1220, a rendering engine 1230, an edge store 1240, an array of sort bins 1250 and a fill data store 1260. The display list generator module 1210 includes an interface that receives an input 1205 containing a sequence of graphical object descriptions. A source of the input 1205 may 15 include read only memory, random access memory, storage device or a streaming data connection such as a network or a data bus. The received object descriptions are interpreted to generate edge and fill data, which are then stored in the edge store 1240 and fill store 1260 respectively. A reference to the edge is also added to the sort bins 1250. A portion 1245 of the edge store 1240 is reserved to store additional edges 20 required for a render when out of edge resource condition is met. Edges are partially sorted in the display list generator module 1210 by using the sort bins 1250. The edge sorter module 1220 sorts the edges in the edge store 1240 in y-order and x-order. The rendering engine 1230 performs rendering by tracking the sorted edges in the edge store 1240 and compositing colors from the fill data store 1260. The rendering engine 1230 2413251_1 920391_speci_lodge - 12 also includes an interface that outputs streams of pixels 1235 to the intended destination. The output destination may include frame buffer, random access memory, storage device or a streaming data connection such as a network or a data bus. The arrangement of Fig. 12 may be implemented in hardware, firmware or both, particularly for embedded 5 rendering applications, such as used in mobile telephone handsets, portable computing devices such as PDAs, portable gaming devices, satellite navigation devices, or other products having embedded computing resources, such a printers or multifunction devices. Figs. 13A and 13B collectively form a schematic block diagram of a general 10 purpose computer system 1300, upon which the functionality of Fig. 12 may be implemented in whole or part by software as an application program. As seen in Fig. 13A, the computer system 1300 is formed by a computer module 1301, input devices such as a keyboard 1302, a mouse pointer device 1303, a scanner 1326, a camera 1327, and a microphone 1380, and output devices including a 15 printer 1315, a display device 1314 and loudspeakers 1317. An external Modulator Demodulator (Modem) transceiver device 1316 may be used by the computer module 1301 for communicating to and from a communications network 1320 via a connection 1321. The network 1320 may be a wide-area network (WAN), such as the Internet or a private WAN. Where the connection 1321 is a telephone line, the modem 20 1316 may be a traditional "dial-up" modem. Alternatively, where the connection 1321 is a high capacity (eg: cable) connection, the modem 1316 may be a broadband modem. A wireless modem may also be used for wireless connection to the network 1320. The computer module 1301 typically includes at least one processor unit 1305, and a memory unit 1306 for example formed from semiconductor random access 2413251_1 920391_speci_lodge - 13 memory (RAM) and semiconductor read only memory (ROM). The module 1301 also includes an number of input/output (1/O) interfaces including an audio-video interface 1307 that couples to the video display 1314, loudspeakers 1317 and microphone 1380, an 1/0 interface 1313 for the keyboard 1302, mouse 1303, 5 scanner 1326, camera 1327 and optionally a joystick (not illustrated), and an interface 1308 for the external modem 1316 and printer 1315. In some implementations, the modem 1316 may be incorporated within the computer module 1301, for example within the interface 1308. The computer module 1301 also has a local network interface 1311 which, via a connection 1323, permits coupling of the computer system 1300 to a 10 local computer network 1322, known as a Local Area Network (LAN). As also illustrated, the local network 1322 may also couple to the wide network 1320 via a connection 1324, which would typically include a so-called "firewall" device or device of similar functionality. The interface 1311 may be formed by an Ethernetm circuit card, a Bluetooth M wireless arrangement or an IEEE 802.11 wireless arrangement. 15 The interfaces 1308 and 1313 may afford either or both of serial and parallel connectivity, the former typically being implemented according to the Universal Serial Bus (USB) standards and having corresponding USB connectors (not illustrated). Storage devices 1309 are provided and typically include a hard disk drive (HDD) 1310. Other storage devices such as a floppy disk drive and a magnetic tape drive (not 20 illustrated) may also be used. An optical disk drive 1312 is typically provided to act as a non-volatile source of data. Portable memory devices, such optical disks (eg: CD-ROM, DVD), USB-RAM, and floppy disks for example may then be used as appropriate sources of data to the system 1300. 2413251_1 920391_specijlodge -14 The components 1305 to 1313 of the computer module 1301 typically communicate via an interconnected bus 1304 and in a manner which results in a conventional mode of operation of the computer system 1300 known to those in the relevant art. Examples of computers on which the described arrangements can be 5 practised include IBM-PC's and compatibles, Sun Sparcstations, Apple MacTM or alike computer systems evolved therefrom. The methods of rendering using rewind edge placement may be implemented using the computer system 1300 wherein the processes of Figs. 4 to 12, to be described, may be implemented as one or more software application programs 1333 executable 10 within the computer system 1300. In particular, the steps of the methods of rendering using rewind edge placement are effected by instructions 1331 in the software 1333 that are carried out within the computer system 1300. The software instructions 1331 may be formed as one or more code modules, each for performing one or more particular tasks. The software may also be divided into two separate parts, in which a first part and the 15 corresponding code modules performs the rendering methods and a second part and the corresponding code modules manage any user interface between the first part and the user, if required. The software 1333 is generally loaded into the computer system 1300 from a computer readable medium, and is then typically stored in the HDD 1310, as illustrated 20 in Fig. 13A, or the memory 1306, after which the software 1333 can be executed by the computer system 1300. In some instances, the application programs 1333 may be supplied to the user encoded on one or more CD-ROM 1325 and read via the corresponding drive 1312 prior to storage in the memory 1310 or 1306. Alternatively the software 1333 may be read by the computer system 1300 from the networks 1320 2413251_1 920391_specijlodge - 15 or 1322 or loaded into the computer system 1300 from other computer readable media. Computer readable storage media refers to any storage medium that participates in providing instructions and/or data to the computer system 1300 for execution and/or processing. Examples of such storage media include floppy disks, magnetic tape, CD 5 ROM, a hard disk drive, a ROM or integrated circuit, USB memory, a magneto-optical disk, or a computer readable card such as a PCMCIA card and the like, whether or not such devices are internal or external of the computer module 1301. Examples of computer readable transmission media that may also participate in the provision of software, application programs, instructions and/or data to the computer module 1301 10 include radio or infra-red transmission channels as well as a network connection to another computer or networked device, and the Internet or Intranets including e-mail transmissions and information recorded on Websites and the like. The second part of the application programs 1333 and the corresponding code modules mentioned above may be executed to implement one or more graphical user 15 interfaces (GUls) to be rendered or otherwise represented upon the display 1314. Through manipulation of typically the keyboard 1302 and the mouse 1303, a user of the computer system 1300 and the application may manipulate the interface in a functionally adaptable manner to provide controlling commands and/or input to the applications associated with the GUI(s). Other forms of functionally adaptable user interfaces may 20 also be implemented, such as an audio interface utilizing speech prompts output via the loudspeakers 1317 and user voice commands input via the microphone 1380. Fig. 13B is a detailed schematic block diagram of the processor 1305 and a "memory" 1334. The memory 1334 represents a logical aggregation of all the memory 2413251_1 920391_speci_lodge - 16 devices (including the HDD 1310 and semiconductor memory 1306) that can be accessed by the computer module 1301 in Fig. 13A. When the computer module 1301 is initially powered up, a power-on self-test (POST) program 1350 executes. The POST program 1350 is typically stored in a 5 ROM 1349 of the semiconductor memory 1306. A program permanently stored in a hardware device such as the ROM 1349 is sometimes referred to as firmware. The POST program 1350 examines hardware within the computer module 1301 to ensure proper functioning, and typically checks the processor 1305, the memory (1309, 1306), and a basic input-output systems software (BIOS) module 1351, also typically stored in 10 the ROM 1349, for correct operation. Once the POST program 1350 has run successfully, the BIOS 1351 activates the hard disk drive 1310. Activation of the hard disk drive 1310 causes a bootstrap loader program 1352 that is resident on the hard disk drive 1310 to execute via the processor 1305. This loads an operating system 1353 into the RAM memory 1306 upon which the operating system 1353 commences operation. 15 The operating system 1353 is a system level application, executable by the processor 1305, to fulfil various high level functions, including processor management, memory management, device management, storage management, software application interface, and generic user interface. The operating system 1353 manages the memory (1309, 1306) in order to ensure 20 that each process or application running on the computer module 1301 has sufficient memory in which to execute without colliding with memory allocated to another process. Furthermore, the different types of memory available in the system 1300 must be used properly so that each process can run effectively. Accordingly, the aggregated memory 1334 is not intended to illustrate how particular segments of memory are 2413251_1 920391_speci_lodge - 17 allocated (unless otherwise stated), but rather to provide a general view of the memory accessible by the computer system 1300 and how such is used. The processor 1305 includes a number of functional modules including a control unit 1339, an arithmetic logic unit (ALU) 1340, and a local or internal memory 1348, 5 sometimes called a cache memory. The cache memory 1348 typically includes a number of storage registers 1344 - 1346 in a register section. One or more internal buses 1341 functionally interconnect these functional modules. The processor 1305 typically also has one or more interfaces 1342 for communicating with external devices via the system bus 1304, using a connection 1318. 10 The application program 1333 includes a sequence of instructions 1331 that may include conditional branch and loop instructions. The program 1333 may also include data 1332 which is used in execution of the program 1333. The instructions 1331 and the data 1332 are stored in memory locations 1328-1330 and 1335-1337 respectively. Depending upon the relative size of the instructions 1331 and the memory 15 locations 1328-1330, a particular instruction may be stored in a single memory location as depicted by the instruction shown in the memory location 1330. Alternately, an instruction may be segmented into a number of parts each of which is stored in a separate memory location, as depicted by the instruction segments shown in the memory locations 1328-1329. 20 In general, the processor 1305 is given a set of instructions which are executed therein. The processor 1305 then waits for a subsequent input, to which it reacts to by executing another set of instructions. Each input may be provided from one or more of a number of sources, including data generated by one or more of the input devices 1302, 1303, data received from an external source across one of the 2413251_1 920391_speci_lodge -18 networks 1320, 1322, data retrieved from one of the storage devices 1306, 1309 or data retrieved from a storage medium 1325 inserted into the corresponding reader 1312. The execution of a set of the instructions may in some cases result in output of data. Execution may also involve storing data or variables to the memory 1334. 5 The disclosed rendering arrangements use input variables 1354, that are stored in the memory 1334 in corresponding memory locations 1355-1358. The rendering arrangements produce output variables 1361, that are stored in the memory 1334 in corresponding memory locations 1362-1365. Intermediate variables may be stored in memory locations 1359, 1360, 1366 and 1367. 10 The register section 1344-1346, the arithmetic logic unit (ALU) 1340, and the control unit 1339 of the processor 1305 work together to perform sequences of micro operations needed to perform "fetch, decode, and execute" cycles for every instruction in the instruction set making up the program 1333. Each fetch, decode, and execute cycle comprises: 15 (a) a fetch operation, which fetches or reads an instruction 1331 from a memory location 1328; (b) a decode operation in which the control unit 1339 determines which instruction has been fetched; and (c) an execute operation in which the control unit 1339 and/or the ALU 1340 20 execute the instruction. Thereafter, a further fetch, decode, and execute cycle for the next instruction may be executed. Similarly, a store cycle may be performed by which the control unit 1339 stores or writes a value to a memory location 1332. 2413251_1 920391_speci_lodge - 19 Each step or sub-process in the processes of Figs. 3 to 12 is associated with one or more segments of the program 1333, and is performed by the register section 1344 1347, the ALU 1340, and the control unit 1339 in the processor 1305 working together to perform the fetch, decode, and execute cycles for every instruction in the instruction 5 set for the noted segments of the program 1333. The methods of rendering including rewind edge placement may alternatively be implemented in dedicated hardware such as one or more integrated circuits performing the functions or sub functions of the configuration illustrated in Fig. 12. Such dedicated hardware may include graphic processors, digital signal processors, or one or more 10 microprocessors and associated memories. Such may be directly incorporated into an embedded device or supplemented into the general purpose computer 1300. In an embedded device implementation, the general functionality of the computer 1300 may be realised however with typically much more limited resources. For example, embedded devices are typically used in hand-portable computerised devices such PDAs, mobile 15 cellular telephones and media players, which typically do not have hard disk drives, and only fixed limited dynamic RAM. In a first method according to the present disclosure, step 120 of adding an object to the display list from Fig. 1, is performed according to a process step 500 seen in Fig. 5. In a software implementation, this may involve the processor 1305 retrieving an 20 object from the memory 1306 and processing that object to update the display list that is also retained in the memory 1306. In Fig. 5, processing commences with an entry step 510, that processes the received object in preparation for edge decomposition. A loop is then started in a first test 520 where the processor 1305 determines whether there is any edge left in the object to be added to the display list. If there is no edge left the 2413251_1 920391_speci-lodge - 20 processing exits and rendering continue according to the process of Fig. 1. If there is an edge left in the object to be added to the display list, then a second test 540 is executed by the processor 1305 to determine if there is an edge resource available for storing the current edge. If there is an edge resource available, the edge is placed into the memory 5 1306 in the display list in the next step 560 and the processing loops back to the first test 520 for a next edge. If there is no edge resource available, the processing exits the placement loop formed by steps 520, 540 and 560, and proceeds to step 550, where the display list is out-of-edge-resource fallback rendered. A more detailed flow diagram of processing method of out-of-edge-resource 10 fallback rendering 550 is illustrated in Fig. 6. In an entry step 610, the remaining edges in the display list of the current object that triggered fallback are rewound in the manner as seen in Fig. 4B. This includes removing edge data structures from the display list and undoing other operations by which those edges were placed in the display list. In a next step 140a, the display list is finalised in preparation for fallback render. The operation of 15 step 140a within step 550 corresponds with that of step 140 discussed above, except that step 140a operates upon what is essentially an intermediate display list applicable to fallback render, whereas step 140 operates on the final display list to conclude rendering of the image in question. The finalised/prepared display list is then rendered at step 630 to form a fallback image, for example as seen in Fig. 4C, which may be stored in 20 memory 1306. Upon the completion of the fallback render, the display list is then cleared in step 640, as seen in Fig. 4D, to free up or otherwise release memory resources in preparation for the next render. At that point, the display list should have sufficient resources to add the current unfinished object. In the final step 650, the rendered fallback image, or at least a memory reference to the fallback image, is added to the 2413251_1 920391_speci_lodge -21 otherwise cleared or empty display list, as seen in Fig. 4E, as either the top-most or the bottom-most object, depending on the z-order of the object sequence. Upon completing the fallback render 550, the processing then goes back to the process object step 510 and the particular object that triggered the fallback render is 5 reprocessed. The display list should now have sufficient resources, as seen by example in Fig. 4F. Typically, graphic object image formation and rendering may be expressed as a function call tree. In some renderer implementations, the condition of running out of resources to accommodate edges may be detected many levels down the function call 10 tree, and in several different places. Fig. 7 illustrates a simplified function call tree 700 in which an edge is added 720 in four different locations during processing an object. The condition of running out of resources to accommodate edges is determined when adding 720 an edge. In a complete function call tree the add edge 720 function may occur many more times. In such cases, for efficient management, and to maintain 15 performance for non fallback cases, a second method is proposed. In the case where the resource condition is checked many levels down a function call tree, in order to terminate the edge processing loop and start fallback rendering, the resource condition needs to be passed back to the function processing the current object 710. In order to pass the resource condition up the call tree, computational and 20 processing overhead occurs at each level of the function call tree. When the number of edges is large, this cost can be significant, and also undesirable in cases that do not need fallback rendering, for example where the large number of objects are relatively simple and can often thus be accommodated. Therefore, the second method may be implemented to reduce this cost by using a fallback flag and performing fallback 2413251_1 920391_speci_lodge -22 rendering at the end of the edge placement loop. This will however introduce a performance cost in the fallback cases. The second method is similar to the first method and can be practised using a renderer process as shown in Fig. I within the arrangements of Figs. 12 and 13. In the 5 second method, step 120 of adding an object to the display list is performed as described below in relation to Fig. 8. Additionally, step 550 from Fig. 8 for out-of-edge-resource fallback rendering is performed as described in relation to Fig. 6. Fig. 8 illustrates a detailed flow diagram of the processing method 800 for adding an object to the display list, which substitutes for step 120 in the process of Fig. 1. 10 Initially, the received object is processed in an entry step 810 to prepare for edge decomposition. In a first test 820 the processor 1305 determines if there is any edge that needs to be added to the display list. If there is an edge left in the current object to be added to the display list, a second test 840 is made by the processor 1305 to determine if there is an edge resource available for storing the current edge. If there is an edge 15 resource available, the edge is stored in step 860 into the display list in the memory 1306, and the processing continues to the next edge by looping back to the first test 820. If there is no edge resource available as determined at step 840, a process 870 then sets a "need fallback flag" to indicate the need for fallback rendering. Processing then continues to the next edge by going back to the first test 820. This looping is repeated 20 until the first test 820 indicates there are no edges left in the current object to be placed and, in turn, ends the loop formed by steps 820, 860 and 870. Upon reaching the loop end, the processor 1305 executes a third test 880 to determine if fallback rendering is needed. This is done by checking the status of the need fallback flag. If the flag is not set, fallback is not needed, and the processing of step 800 exits. Otherwise, if the flag is 2413251_1 920391_speci_lodge -23 set, the processing proceeds to the next step 550 to perform out-of-edge-resource fallback rendering of the display list. This may be performed in the same fashion described above with reference to Fig. 6. The processing then goes back to the process object step 810 and the object that triggered the fallback render is reprocessed. Based on 5 the above arrangement, every edge of the object is processed regardless of the availability of resources. Further, no extra processing occurs on a per edge basis, thereby minimising the impact on processing speed when fallback rendering is not needed. The extra processing time, for example to processed edges that do not get stored, in the fallback case is insignificant when compared with the total time to render in the fallback 10 case. In the finalising of the display list step 140 of Figs. I and 6, edges need to be sorted. To facilitate sorting, edges may be bin sorted during the adding of an edge to the display list. An array of sorting bins, each representing an evenly divided screen or display region for the image being rendered, are allocated from memory locations of the 15 memory 1306 at the start of rendering. Each edge has a unique identifier or ID. For example an edge ID may be a pointer to the data structure of the edge. As edges are placed in the display list, their corresponding IDs are added to the corresponding sorting bins. In finalising the display list step 140, edges in each sorting bin are then sorted in y order and x-order. This is typically performed by the processor 1305 accessing the bins 20 in the memory 1306. One example of an arrangement of sorting bins, is the allocation of one sorting bin per scanline. In this example, the edges in each sorting bin then only need to be sorted in x-order, as the sorting bins are already allocated and thus sorted in y order. 2413251_1 920391_speci_lodge - 24 The number of memory locations in each sorting bin may be varied depending upon the particular implementation. It is possible for all edges to be arranged in the one sorting bin. In a specific implementation, each bin has only the first and last edge of the bin. An alternate implementation is with just the first edge, with the rest of the edges in 5 the bin pointing to subsequent edges thus forming a linked list. A third method of rendering with rewind edge placement may be implemented to include the process of re-winding bin sort operations. An array of restore bins, also formed in the memory 1306 and with the same number of entries as the sorting bins, is allocated and initially clear or cleared. Each restore bin corresponds to one sorting bin 10 and is used to keep track of the state of the sorting bin. Whenever fallback rendering occurs, the state of the sorting bins is restored to the state before the current object was placed, based on the state information recorded in the restore bins. The third method is also similar to the first and can be practised within the renderer process as shown in Fig. I and with the processing arrangements of Figs. 12 and 15 13. In the third method, step 120 of adding an object to the display list, is performed according to the process 900 of Fig. 9. The process 900 of Fig. 9 commences with an entry step 910 that processes the received object in preparation for edge decomposition. An edge placement loop is then started by the processor 1305 executing a first test 920 to determine if there is any edge 20 that needs to be added to the display list. If there is an edge left in the current object to be added to the display list, a second test 930 is then made by the processor 1305 to determine if there is an edge resource available for storing the current edge. If there is an edge resource available, the processor 1305 executes a third test 935 to determine the status of a restore bin for the current edge. If the restore bin is empty, meaning the 2413251_1 920391_speci_lodge -25 restore bin has not yet been saved for the current object, the restore bin is updated 940 with the state of the corresponding sorting bin, indicating the state in which the sorting bin was left after processing the previous object. Regardless of the result of the third test, in step 945 the edge is stored in the display list in the memory 1306. The process of 5 step 945 also then adds the edge ID to the sorting bin and continues to the next edge by going back to the first test 920. In the second test 930, if there is no edge resource available, the process 970 sets a flag to indicate the need for fallback rendering and continues to the next edge by going back to the first test 920. When there are no edges left as determined in the first test 920, the processor 1305 executes a fourth test 980 to 10 determine if fallback rendering is needed by checking the status of the fallback flag. If fallback is not needed, then step 990 clears the restore bins and the process 900 exits. If fallback is needed, the process 900 proceeds to the next step 950 which performs out-of edge-resource fallback rendering of the display list. The processing then goes back to the process object step 910 and the object that triggered the fallback render is 15 reprocessed. A more detailed flow diagram of the out-of-edge-resource fallback rendering 950 is illustrated in Fig. 10. The process 950 has two extra steps added to the process of Fig. 6, so as to manage the restoration of the sort bins. The processing of step 950 commences with an entry step 610 corresponding to that of Fig. 6, that re-winds edges 20 from the display list as previously described. In a following step 1020, the state of the sorting bins is restored back to the state in which the sorting bins were left after processing the previous object, based on the content in the restore bins. The restore bins are cleared in the next step 1030 in preparation for the next round of processing. The display list is then finalised at step 140b and rendered at step 630 to form a fallback 2413251_1 920391_speci_lodge - 26 image. Upon completing the fallback rendering, the display list is cleared at step 640 and the fallback image is inserted at step 650 to the display list for the next render. The processing of step 950 then exits. Sometimes additional edges are needed to carry out preparatory operations before 5 rendering can take place. Preparatory operations are operations that are required for the renderer to generate correct output. For example, in an alternate method, clipping is implemented in such a way that a clipper is added to the display list as the top object just before rendering. The clipper is typically formed from a user defined clipping shape and clips objects located below to a level defined by a user. When fallback rendering is 10 enabled, rendering will fail to take place as no edge resources are available in which to place the clipper. Thus, a fourth method may be implemented to reserve edge resources at the start of rendering. These edge resources are separated from the main edge resource pool and are not available to regular edge placement. Only the preparatory operations are able to allocate these edge resources. 15 The fourth method is similar to those above and also can be practised within the renderer process of Fig. I and with the arrangements of Figs. 12 and 13. The fourth method differs from the third in that the finalisation of the display list step 140b from Fig. 10 and step 140 of Fig. l, is performed as will be described in Fig. II below. Step 140b, like step 140a described above mirrors the functionality of step 140. 20 Fig. 11 is a detailed flow diagram of a method 1100 of finalising the display list, which can substitute for step 140 of Figs. I and 10. Processing of the method 1100 commences with a first test 1110 where the processor 1305 determines if there is any clipping edge left in a given clipper to be placed into the display list. If there is a clip edge left, a second test 1130 is performed by the processor 1305 to ascertain if there is an 2413251_1 920391_specilodge - 27 edge resource available from the main resource pool to place the current clipping edge. If there is an edge resource available, the clipping edge is placed in step 1140 into the display list and the processing continues to the next clipping edge by going back to the first test 1110. If there is no edge resource available, the clipping edge is placed at step 5 1150 into the display list using a reserved edge resource and the processing also continues to the next clipping edge by going back to the first test 1110. If there are no edges left in the first test 1110, step 1160 sorts the edges in y-order and x-order, so that each edge can be tracked and rendered on the screen during render. The processing of step 1100 then exits. 10 A background image, possibly a resultant image from a previous render, is placed into the display list with bottom z-order to form the background for further objects. In some implementations it may be desirable to delay the placing of a background image until just before rendering takes place. This is desirable because if an opaque shape is placed with higher z-order that completely covers the rendering area, this background 15 image may be skipped. Placing the background image is another case of a preparatory operation that must be completed before a rendering can take place. The display list may be implemented using either a linked list or an array. When edges are placed, their data is added in an orderly fashion. Edges cluster together in succession in the display list. The state of the display list is often tracked by three 20 pointers, a start pointer, an end pointer and a current pointer. The start pointer indicates the position of a starting edge in the display list and the end pointer indicates the position of an end edge. The current pointer indicates the position of a current edge being added. After the current edge is added, the current pointer moves to a next position, which is later used by a next edge. When the display list is rendered, only edges in between the 2413251_1 920391_speci_lodge -28 start pointer and the end pointer are processed. Thus, the described process of re winding edges is, for the most part, simply adjusting the end pointer to the position of the last edge of a previous object. This greatly reduces the performance cost of edge rewinding, as mostly pointer manipulations are involved. 5 Moreover, fallback rendering is always performed on previously allocated complete objects, since the current unfinished object is re-wound from the display list. Therefore, no additional resources are needed in order to complete the rendering. As a result, rendering can be performed in a known amount of memory, which is desirable in an embedded environment. 10 INDUSTRIAL APPLICABILITY The arrangements described are applicable to the computer and data processing industries and particularly for the efficient rendering of object-based images in embedded environments. The foregoing describes only some embodiments of the present invention, and 15 modifications and/or changes can be made thereto without departing from the scope and spirit of the invention, the embodiments being illustrative and not restrictive. (Australia Only) In the context of this specification, the word "comprising" means "including principally but not necessarily solely" or "having" or "including", and not "consisting only of'. Variations of the word "comprising", such as "comprise" and 20 "comprises" have correspondingly varied meanings. 2413251_1 920391_specilodge

Claims (19)

1. A computerized method of rendering a plurality of objects to form an image, the method being performed by a processor in association with a memory, said method 5 comprising the steps of: (a) forming a display list describing the image in the memory by processing the objects sequentially and in z-order, said processing comprising: (aa) receiving a z-ordered object; (ab) generating at least one primitive for the received object, said 10 generating comprising, for each primitive generated: (aba) determining if a resource in the memory for the primitive is available; (abb) adding, if the memory resource is determined as available, the primitive to the display list; 15 (abc) if the memory resource is determined as not available: (abca) reversing effects upon the display list of any added primitives of the received object; (abcb) rendering the display list to pixel data as a fallback image; 20 (abcc) clearing the display list; (abcd) adding the fallback image to the display list; and (abce) repeating step (ab) for the received object; and (b) rendering the display list as pixel data. 25

2. A method according to claim 1, wherein the primitive is an edge. 2413251_1 920391_speci_lodge - 30

3. A method according to claim 1, wherein the primitive is a straight line edge.

4. A method according to claim 1, wherein step (abca) of reversing effects upon the 5 display list of any added primitives of the received object includes removing the primitives of the received object from the display list.

5. A computerized method of rendering a plurality of objects to form an image, the method being performed by a processor in association with a memory, said method 10 comprising the steps of: (a) forming a display list describing the image in the memory by processing the objects sequentially and in z-order, said processing comprising: (aa) receiving a z-ordered object; (ab) generating at least one primitive for the received object, said 15 generating comprising, for each primitive generated: (aba) determining if a resource in the memory for the primitive is available; (abb) adding, if the memory resource is determined as available, the primitive to the display list; 20 (abc) setting, if the memory resource is determined as not available, a flag indicating fallback render is required; (ac) testing the flag, and where the flag is set: (aca) reversing effects upon the display list of any added primitives of the received object; 2413251_1 920391_specilodge -31 (acb) rendering the display list to pixel data as a fallback image; (acc) clearing the display list; (acd) adding the fallback image to the display list; and (ace) repeating step (ab) for the received object; and 5 (b) rendering the display list as pixel data.

6. A method according to claim 5 wherein step (abc) comprises discarding the primitive. 10

7. A method according to claim 5, wherein the primitive is an edge.

8. A method according to claim 5, wherein the primitive is a straight line edge.

9. A method according to claim 5, wherein step (acb) of reversing effects of any 15 added primitives of the received object includes removing the primitives of the received object from the display list.

10. A method according to claim 5, wherein before step (b) the primitives in the display list are sorted according to y-order and x-order associated with raster scan 20 rendering.

11. A method according to claim 10 further comprising establishing restore bins and sorting bins in the memory, wherein: 2413251_1 920391_specilodge - 32 (i) step (abb) comprises (abba) testing if the restore bins are empty and where so, updating the restore bins with the state of the sorting bins before processing of the current object, and (abbb) updating the sorting bins with an identifier of the primitive and adding the primitive to the display list; and 5 (ii) step (ac) comprises, where the flag is not set, clearing the restore bins.

12. A method according to claim 11, wherein step (aca) comprises removing primitives associated with the received object from the display list, removing said primitives from the sorting bins and updating the restore bins with a revised state of the 10 sorting bins.

13. A method according to claim 1 or 5, further comprising reserving memory resources for preparatory operations for render for use when memory resources are otherwise determined as not available. 15

14. A method according to claim 13, wherein the preparatory operations include at least one of the set of placing of a clipper object, and a placing of a background image.

15. A computerized method of rendering a plurality of objects to form an image, the 20 method being performed by a processor in association with a memory, said method comprising the steps of: (a) forming a display list describing the image in the memory by processing the objects sequentially and in z-order, said processing comprising, for each object: 2413251_1 920391_speci_lodge - 33 (aa) generating at least one primitive for the object, said generating comprising determining if a resource in the memory for the primitive is available and adding the primitive to the display list if the memory resource is determined as available; (ab) where the memory resource is determined as not available: 5 (aba) reversing effects upon the display list of any added primitives of the object; (abb) rendering the display list to pixel data as a fallback image; (abc) clearing the display list; (abd) adding the fallback image to the display list; and 10 (abe) repeating step (aa) for the received object; and (b) rendering the display list as pixel data.

16. A method according to claim 15 wherein the primitives are sorted according to at least one of an y-order and an x-order associated with raster scan rendering, said method 15 further comprising establishing restore bins and sorting bins in the memory, wherein: (i) step (aa) comprises (aaa) testing if the restore bins are empty and where so, updating the restore bins with the state of the sorting bins before processing of the current object, and (aab) updating the sorting bins with an identifier of the primitive and adding the primitive to the display list; and 20 (ii) step (ab) comprises, clearing the restore bins.

17. A method of rendering a plurality of objects to form an image, the method being substantially as described herein with reference to any one of the embodiments as that embodiment is illustrated in the drawings. 2413251_1 920391_speci_lodge -34

18. A computer readable storage medium having a computer program recorded thereon, the program being executable by a computerized device having an associated memory to perform the method of any one of claims I to 17. 5

19. Computer apparatus comprising a processor and an associated memory, said apparatus having means adapted to perform the method of any one of claims I to 17. 10 Dated this 27th day of November 2009 CANON KABUSHIKI KAISHA Patent Attorneys for the Applicant Spruson & Ferguson 2413251_1 920391_specilodge