AU2008202867A1 - Recognition of disjoint shapes from document images - Google Patents

Description

S&F Ref: 862314 AUSTRALIA PATENTS ACT 1990 COMPLETE SPECIFICATION FOR A STANDARD PATENT Name and Address Canon Kabushiki Kaisha, of 30-2, Shimomaruko 3 of Applicant: chome, Ohta-ku, Tokyo, 146, Japan Actual Inventor(s): Kjetil Wormnes Address for Service: Spruson & Ferguson St Martins Tower Level 35 31 Market Street Sydney NSW 2000 (CCN 3710000177) Invention Title: Recognition of disjoint shapes from document images The following statement is a full description of this invention, including the best method of performing it known to me/us: 5845c(1295150_1) RECOGNITION OF DISJOINT SHAPES FROM DOCUMENT IMAGES Field of the Invention The present invention relates generally to document analysis and, in particular, to a method and apparatus for creating a document. The present invention also relates to a 5 computer program product including a computer readable medium having recorded thereon a computer program for creating a document. Background The proliferation of scanning technology combined with ever increasing computational processing power has led to many advances in the area of document 10 analysis systems. These systems may be used to extract semantic information from a scanned document, for example, by means of Optical Character Recognition (OCR) technology. OCR technology is used in a growing number of applications such as automated form reading. Document analysis systems can also be used to improve compression of an electronic representation of a document by selectively using an 15 appropriate compression method depending on the content of each part of a page of the document. Improved document compression lends itself to applications such as archiving and electronic distribution. A significant proportion of documents are generated using structured text/graphics editing applications such as Microsoft TM WordTM, MicrosoftTM PowerpointTM , and the like. 20 In addition to formatted text editing, these text/graphics editing applications include basic figure drawing tools and options. An important class of document analysis applications process a bitmap representation of a document to generate an electronic version of the document that can be -2 viewed and edited using the above editing applications. Such document analysis applications will be referred to as "scan-to-editable" document analysis applications. The basic figure drawing options in a typical structured text/graphics editing application include freeform line drawing, "template" shapes and connectors (i.e., dynamic 5 line objects that connect to and/or between template shapes within a document). The text/graphics editing applications may also include colouring, filling, layering and grouping options for sets of objects. Freeform line drawing can be used to draw open and closed objects with straight or curved sections by defining a set of points along the path of the object. A closed N-point 10 polygon may be drawn by defining an ordered set of vertices. For example, Fig. I shows a generalised line polygon 100 including vertices (e.g., 101) which are represented by black squares on the line polygon 100. Freeform drawn objects may be filled or empty, with a range of possible fill options including solid colours, transparency, blends and patterns. Many commonly used geometric shapes can be created using template shapes. A 15 user may prefer to use a template shape rather than drawing the shape using freeform lines as using a template shape can be faster, more accurate in terms of representation of the desired shape, and easier to edit at a later time. The well known MicrosoftTM AutoShapes set includes a number of examples of template shapes which can be manipulated within editing applications such as MicrosoftTM WordTM and PowerPointTM. Other template 20 shapes may be found in OpenOfficeTM and editing applications such as the Writerrm and Impress applications. Template shapes are commonly found in documents, in particular in figures and flow charts. Some example template shapes 405 to 460 are shown in Fig. 4. A range of fill options are generally available including unfilled line objects 405 to 420, solid filled -3 objects of the same colour 425 to 440 or different colour 445 to 460 to the line, transparency, blends or patterned fills. Existing scan-to-editable applications tend to be biased towards processing text and tables and typically do not deal effectively with the processing of figures. Such scan-to 5 editable applications may use optical character recognition (OCR) processing to recognise characters and simple symbols from an image of a document. Many basic scan-to-editable applications simply embed the parts of the image that are not recognised as text or tables as a bitmapped image typically in a compressed format and sometimes at a low resolution. Such basic scan-to-editable applications are clearly disadvantageous to a user as the 10 embedded bitmapped images can not be readily edited using geometric drawing tools. Further, the overall file size, including the bitmapped images, can be large. Other applications employ vectorisation methods to identify line objects and solid filled objects. The line objects and solid filled objects may be represented as freeform line drawings in the output rather than instances of specific template shapes. Specific editing 15 options defined for a template shape are typically not available, limiting the usefulness of a corresponding editable electronic version. Such applications also provide less accurate semantic information regarding the content of a document and may therefore be less useful in database applications that rely on this data. There are a number of known methods for recognising graphical elements within a 20 bitmap representation. Some of these known methods use global shape features including Fourier descriptors, Curvature Scale Space descriptors, Zernike Moments, Grid Descriptors, Concavity Trees and other features. These known methods generally will oily recognise and assign a label to a single graphical object.

-4 Thus a need clearly exists for an improved method of creating a document from an image. Summary According to one aspect of the present disclosure there is provided a method of 5 creating a document comprising a modifiable template shape, the method comprising the steps of: analysing an image to detect at least two graphical objects within the image; matching each of the detected graphical objects with a template shape; matching at least two of the matched template shapes with a multi-part disjoint 10 template shape, one or more parts of said multi-part disjoint template shape being modifiable through adjustment of one or more control parameters; and creating a document comprising the at least one modifiable multi-part disjoint template shape. Preferably the multi-part disjoint template shape is non-textual. The method may 15 further comprise the step of merging one or more further template shapes with the matched template shapes to form the multi-part disjoint template shape. Desirably one or more of the template shapes is matched to more than one multi-part disjoint template shape. Advantageously each of the at least two template shapes have a single outer contour. Desirably the multi-part disjoint template shape is modifiable in a non-affine manner. In 20 some implementations the multi-part disjoint template shape is a striped arrow disjoint template shape. Preferably the multi-part disjoint template shape is a donut disjoint template shape. Also the multi-part disjoint template shape may be a cloud callout template shape.

-5 According to another aspect of the present disclosure there is provided a method of creating a document comprising a modifiable multi-part disjoint template shape, the method comprising the steps of: analysing an image to detect at least two graphical objects within the image; 5 matching each of the detected graphical objects with a template shape; matching at least two of the matched template shapes with a multi-part disjoint template shape that may be modifiable in a non-affine manner through adjustment of one or more control parameters; creating a document comprising the at least one modifiable multi-part disjoint 10 template shape. Other aspects including apparatus and computer readable media are also disclosed. Brief Description of the Drawings Some aspects of the prior art and one or more embodiments of the present invention will now be described with reference to the drawings and appendices, in which: 15 Fig. I shows a generalised line polygon; Fig. 2 is intentionally blank; Fig. 3 shows a set of unfilled template shapes; Fig. 4 shows some example template shapes commonly found in documents: Fig. 5(a) shows a pentagon template shape changing as scaling, sx and sy, rotation, 0, 20 and offset, Ax and Ay, affine parameters of the template shape are modified; Fig. 5(b) shows a number of different triangles that may be obtained as the skew parameter of a triangle is modified; Fig. 6 shows a set of example template shapes with one control parameter; -6 Fig. 7 shows an example of a template shape with two control parameters as the parameters are modified; Fig. 8 shows another example of a template shape with two control parameters as the parameters are modified; 5 Fig. 9 shows a rounded rectangle template shape as the scaling of the rounded rectangle template shape is modified; Fig. 10 shows an unfilled hexagon template shape with a set of increasing line thickness parameters from left to right; Fig. 11 is a flowchart showing a method of creating a document comprising a 10 modifiable template shape (i.e., a modifiable closed-form non-textual template shape); Fig. 12 is a flowchart showing a method of analysing graphics regions to identify (or detect) graphical objects within the graphics regions, as executed in the method of Fig. 11; Fig. 13 is a flowchart showing a method of matching a detected graphical object with a predetermined template shape, as executed in the method of Fig. 11; 15 Fig. 14 is a flowchart showing another method of creating a document comprising a modifiable template shape; Fig. 15 is intentionally blank; Fig. 16 is intentionally blank; Fig. 17 is a flowchart showing a method of determining a set of candidate template 20 shapes for a graphical object, as executed in the method of Fig. 13; Fig. 18 is a flowchart showing a method of determining a set of parameters for modifying a candidate template shape to match a graphical object, as executed in the method of Fig. 13; -7 Fig. 19 is a flowchart showing a method of determining optimal control parameter values that transform a template shape to match a graphical object, as executed in the method of Fig. 18; Fig. 20 is a flowchart showing a method of creating an editable document, as 5 executed in the method of Fig. 11; Fig. 21 is a flowchart showing a method of extracting non-line objects, as executed in the method of Fig. 12; Fig. 22 is intentionally blank; Fig. 23 is intentionally blank; 10 Fig. 24 is a flowchart showing a method of determining rotation and scaling parameters for use in modifying a template shape to match a graphical object, as executed in the method of Fig. 18; Fig. 25 is a schematic block diagram of a general purpose computer upon which arrangements described can be practiced; 15 Fig. 26 is intentionally blank; Figs. 27(a) and (b) show a selection of the MicrosoftTM AutoShapes (i.e., template shapes); Fig. 28 is a flowchart showing a method of merging template shapes as executed in the method of Fig. 11; 20 Fig. 29 is a flowchart showing a method creating a list of subsets of graphical objects, as executed in the method of Fig. 28; Fig. 30 is a flowchart for determining a match score for a set of template shapes, as executed in the method of Fig. 28; -8 Fig. 31 shows a striped arrow mulit-part disjoint template shape consisting of three separate template shapes; Fig. 32 shows an example of ambiguity in relabelling pairs of shapes. Fig. 33(a) shows an exemplary set of label pairs of template shapes that can be 5 combined (or merged) to form a donut multi-part disjoint template shape; Fig. 33(b) shows another exemplary set of label pairs of template shapes that can be combined to form a striped arrow multi-part disjoint template shape; Fig. 33(c) shows another exemplary set of label pairs of template shapes that can be combined to form a cloud callout multi-part disjoint template shape; 10 Fig. 34 shows a donut multi-part disjoint template-shape and its constituent parts; Fig 35(a) shows a cloud callout multi-part template-shape and its constituent parts; Fig. 35(b) shows the cloud callout multi-part template shape of Fig. 35(a) where the circle template shapes form a "cloud caterpillar"; Fig. 35(c) shows the cloud callout multi-part template shape of Fig. 35(a) 15 corresponding to a particular rule of Fig. 33(c); Fig 36(a) shows a striped arrow multi-part disjoint template shape; Fig. 36(b) shows the striped arrow multi-part disjoint template shape corresponding to a particular rule of Fig. 33(b); Fig. 37(a) shows the cloud callout main contour template-shape of Fig. 35(b); 20 Fig. 37(b) shows the cloud caterpillar template-shape Fig. 35(b); Appendix A is a table listing the names of each of the MicrosoftTM AutoShapes of Figs. 27(a) and (b) together with their drawing reference. Detailed Description including Best Mode -9 Where reference is made in any one or more of the accompanying drawings to steps and/or features, which have the same reference numerals, those steps and/or features have for the purposes of this description the same function(s) or operation(s), unless the contrary intention appears. 5 It is to be noted that the discussions contained in the "Background" section and that above relating to prior art arrangements relate to discussions of documents or devices which form public knowledge through their respective publication and/or use. Such should not be interpreted as a representation by the present inventor(s) or patent applicant that such documents or devices in any way form part of the common general knowledge in the 10 art. The term "template shape" as used below refers to a predetermined model that defines the form of a modifiable non-textual shape object comprising one or more closed contours each having a closed outer boundary within a specified editing or viewing environment. Most of the template shapes described herein are "modifiable single closed 15 form non-textual template shapes." Such template shapes typically consist of a single closed outer boundary (or outer contour). However, some template shapes, such as the "can" template shape 420 shown in Fig. 4, include other internal lines. Further, a number of modifiable "multi-part disjoint" template shapes, such as those shown in Figs. 31, 34, 35(a)-(b) and 37, will also be described. These multi-part disjoint template shapes 20 comprise at least two separate single template shapes each comprising a single closed outer boundary. Again, the multi-part disjoint template shapes are non-textual. The template shapes described herein, including the multi-part disjoint template shapes, are specified by a plurality of parameters the interpretation of which is defined within the corresponding predetermined model. The plurality of parameters can include -10 affine parameters (i.e., width, height, rotation and skew), control parameters and line thickness parameters. The parameters defining a template shape are described in further detail below. The term "label" as used herein refers to a description of which one of a plurality of 5 template shapes (including the multi-part disjoint templates shapes) a particular graphical object or group of graphical objects has been recognised (or identified) as, together with control-parameters and affine-parameters that transform the template shape to match the graphical object or group of graphical objects. Further, the term "labelling" or "relabelling" of a graphical object or group of graphical objects refer to addition or 10 alteration of a "label" assigned to the graphical object or group of graphical objects. For clarity purposes, such a "label" is referred to below as the "recognised shape label". In some cases a single graphical object may have more than one recognised shape label. Accordingly, each recognised shape label represents an "interpretation" of a graphical object or group of graphical objects. 15 Methods of creating documents are described below. The described methods may be implemented using a computer system 2500, such as that shown in Fig. 25 wherein the processes of Figs. I to 37(b) may be implemented as software, such as one or more application programs executable within the computer system 2500. In particular, the steps of the described methods are affected by instructions in the software that are carried out 20 within the computer system 2500. The instructions may be formed as one or more code modules, each for performing one or more particular tasks. The software may also be divided into two separate parts, in which a first part and the corresponding code modules perform the described methods and a second part and the corresponding code modules manage a user interface between the first part and the user. The software may be stored in - 11 a computer readable medium, including the storage devices described below, for example. The software is loaded into the computer system 2500 from the computer readable medium, and then executed by the computer system 2500. A computer readable medium having such software or computer program recorded on it is a computer program product. 5 The use of the computer program product in the computer system 2500 preferably affects an advantageous apparatus for implementing the described methods. As seen in Fig. 25, the computer system 2500 is formed by a computer module 2501, input devices such as a keyboard 2502 and a mouse pointer device 2503, and output devices including a printer 2515, a display device 2514 and loudspeakers 2517. An 10 external Modulator-Demodulator (Modem) transceiver device 2516 may be used by the computer module 2501 for communicating to and from a communications network 2520 via a connection 2521. The network 2520 may be a wide-area network (WAN), such as the Internet or a private WAN. Where the connection 2521 is a telephone line, the modem 2516 may be a traditional "dial-up" modem. Alternatively, where the connection 2521 is a 15 high capacity (eg: cable) connection, the modem 2516 may be a broadband modem. A wireless modem may also be used for wireless connection to the network 2520. The computer module 2501 typically includes at least one processor unit 2505, and a memory unit 2506 for example formed from semiconductor random access memory (RAM) and read only memory (ROM). The module 2501 also includes a number of 20 input/output (1/0) interfaces including an audio-video interface 2507 that couples to the video display 2514 and loudspeakers 2517, an I/O interface 2513 for the keyboard 2502 and mouse 2503 and optionally a joystick (not illustrated), and an interface 2508 for the external modem 2516 and printer 2515. In some implementations, the modem 2516 may be incorporated within the computer module 2501, for example within the interface 2508.

- 12 The computer module 2501 also has a local network interface 2511 which, via a connection 2523, permits coupling of the computer system 2500 to a local computer network 2522, known as a Local Area Network (LAN). As also illustrated, the local network 2522 may also couple to the wide network 2520 via a connection 2524, which 5 would typically include a so-called "firewall" device or similar functionality. The interface 2511 may be formed by an EthernetTM circuit card, a wireless BluetoothTM or an IEEE 802.11 wireless arrangement. The interfaces 2508 and 2513 may afford both serial and parallel connectivity, the former typically being implemented according to the Universal Serial Bus (USB) standards 10 and having corresponding USB connectors (not illustrated). Storage devices 2509 are provided and typically include a hard disk drive (HDD) 2510. Other devices such as a floppy disk drive and a magnetic tape drive (not illustrated) may also be used. An optical disk drive 2512 is typically provided to act as a non-volatile source of data. Portable memory devices, such as optical disks (eg: CD-ROM, DVD), USB-RAM, and floppy disks 15 for example may then be used as appropriate sources of data to the system 2500. The components 2505 to 2513 of the computer module 2501 typically communicate via an interconnected bus 2504 and in a manner which results in a conventional mode of operation of the computer system 2500 known to those in the relevant art. Examples of computers on which the described arrangements can be practised include IBM-PC's and 20 compatibles, Sun Sparcstations, Apple MacTM or alike computer systems evolved therefrom. Typically, the application programs discussed above are resident on the hard disk drive 2510 and read and controlled in execution by the processor 2505. Intermediate storage of such programs and any data fetched from the networks 2520 and 2522 may be - 13 accomplished using the semiconductor memory 2506, possibly in concert with the hard disk drive 2510. In some instances, the application programs may be supplied to the user encoded on one or more CD-ROM and read via the corresponding drive 2512, or alternatively may be read by the user from the networks 2520 or 2522. Still further, the 5 software can also be loaded into the computer system 2500 from other computer readable media. Computer readable media refers to any storage medium that participates in providing instructions and/or data to the computer system 2500 for execution and/or processing. Examples of such media include floppy disks, magnetic tape, CD-ROM, a hard disk drive, a ROM or integrated circuit, a magneto-optical disk, or a computer readable 10 card such as a PCMCIA card and the like, whether or not such devices are internal or external of the computer module 2501. Examples of computer readable transmission media that may also participate in the provision of instructions and/or data include radio or infra-red transmission channels as well as a network connection to another computer or networked device, and the Internet or Intranets including e-mail transmissions and 15 information recorded on Websites and the like. The second part of the application programs and the corresponding code modules mentioned above may be executed to implement one or more graphical user interfaces (GUIs) to be rendered or otherwise represented upon the display 2514. Through manipulation of the keyboard 2502 and the mouse 2503, a user of the computer system 20 2500 and the application may manipulate the interface to provide controlling commands and/or input to the applications associated with the GUI(s). The described methods may alternatively be implemented in dedicated hardware such as one or more integrated circuits performing the functions or sub functions of the -14 methods. Such dedicated hardware may include graphic processors, digital signal processors, or one or more microprocessors and associated memories. Figs. 27(a) and (b) show a selection of MicrosoftTM Autoshapes (i.e., template shapes). Further, Appendix A is a Table listing the names of each of the MicrosoftTM 5 AutoShapes of Figs. 27(a) and (b) together with their drawing reference in Figs. 27(a) and (b). The affine parameters of a template shape forming part of a multi-part disjoint template shape, define modifications to the multi-part disjoint template shape. Such modifications may be described by a linear matrix transform. The modifications include 10 scaling, rotation and offset, and for certain shape classes may also include skew. For scaling of s,, and sy along the x- and y-axes respectively, a rotation of 0, and offsets of Ax and Ay along the x- and y-axes respectively, the linear matrix transform may be represented in accordance with Equation (1) below: (-sinO cos& (s, 0 ()(AX 0) cos 0 sin 0 0 s, 0 Ay 15 If the transform also includes a skew, c, the matrix transform may be represented in accordance with Equation (2) below: -sinO coso).(s, 0)1 c0(Ax 0( cos0 sin 0 0 s, 0 1 0 Ay Objects of the template class that differ only in terms of affine parameters may be modified to match using the appropriate matrix transform. Fig. 5(a) illustrates a pentagon 20 template shape 500 changing firstly to the template shape 501, secondly to the template shape 502 and thirdly to the template shape 503, as scaling, s,, and sy, rotation, 0, and offset, Ax and Ay, affine parameters of the template shape 500 are modified.

- 15 Fig. 5(b) shows a number of different triangles 505 to 507 that may be obtained as the skew parameter of the triangle 504 is modified. It is noted that the skew parameter for specific shapes may appear as a control parameter within some user interfaces. However, as described herein, the skew parameter is considered as an affine parameter. 5 Control parameters define further geometric modifications to the template shape that in general cannot be defined in terms of a linear matrix transform of the template shape. Each template shape, including the multi-part disjoint template shapes, has a defined set of N control parameters, where N is specified by the model and may be zero In particular, template shapes typically comprise at most four (4) (i.e., four or less) control parameters. 10 However, some template shapes may comprise more than four control parameters. Template shapes may be defined entirely by straight edges (ie. polygons) or curves, or may comprise a mixture of straight edges and curve sections between a set of vertices. Each control parameter typically modifies a plurality of vertices, a plurality of curve sections, or both a plurality of vertices and curve sections. 15 A closed-form template shape can alternatively be defined by parameters that denote the coordinates of individual vertices of the particular template shape. Still further, a closed-form template shape can be defined by parameters that independently modify each curve section of the template shape. Commonly used curve parameterisation methods are cubic splines and Bezier curves. The advantage of defining template shapes using control 20 parameters that modify more than one vertex and/or curve section is that substantially fewer parameters are required, and the parameters describe modifications better suited to that particular shape. The effect of modifying the control parameters of a template shape is defined by the template model for the template shape. The control parameters may be used for modifying - 16 the template shapes, including the multi-part disjoint template shapes, in a non-affine manner. In one embodiment, each part of the multi-part disjoint template shape is modifiable simultaneously through manipulation of one or more control parameters. A set 600 of example template shapes 601 to 624 with one control parameter are 5 shown in Fig. 6. Adjacent to each template shape (e.g., 606) is a black square (e.g., 625), the x-coordinate of which represents the value of the control parameter. Fig. 6 includes a set of round rectangles (i.e., 601 to 604), for which the control parameter modifies the radius of curvature of the corners. Fig. 6 also includes trapezoids (i.e., 605 to 608), hexagons (i.e., 609 to 612) and octagons (i.e., 613 to 616) for which the control parameter 10 (e.g., 625) defines the point of intersection of a line on the left side of the template shape with the upper horizontal section. Fig. 6 also includes 16-point seal (i.e., 617 to 620) for which the control parameter (e.g., 626) defines the relative radii of the convex and concave vertices. Finally, Fig. 6 includes partially occluded moon shapes (i.e., 621 to 624) for which the control parameter (e.g., 627) defines the phase of the moon shape. 15 Fig. 7 shows a set 700 of example arrow template shapes 701 to 716 with two control parameters. The black squares represent the values of control parameters, in this case the two parameters are represented by the x- and y- coordinates respectively. Similarly, Fig. 8 shows a set 800 of example round callout template shapes 801 to 816 with two control parameters. Again, the black squares represent the values of control 20 parameters, in this case the two parameters are represented by the x- and y- coordinates respectively. Template shapes generally have less control parameters than the number of sections that define the outer boundary (or boundaries, or contour(s)) of the template shape. For example, the set 600 of example shapes in Fig. 6 are defined in terms of a single control - 17 parameter. However, the trapezoids (605 to 608) consist of four (4) sections, the hexagons (609 to 612) consists of six (6) sections, the octagons (613 to 616) consist of eight (8) sections, the 16-point seals (617 to 620) consist of thirty two (32) sections and the moons (621 to 624) consist of tow (2) sections. Similarly, the template shapes shown in Figs. 7 5 and 8 are defined in terms of two (2) control parameters however each template shape consists of more than two (2) sections. Most template shapes in the MicrosoftTM AutoShapes set have four (4) or less control parameters. It is noted that in some cases the scaling parameters may not behave as affine parameters for specific template shapes. For example, as seen in Fig. 9, the corners of a 10 rounded rectangle template shape 910 may retain their form as arcs of a circle while the scaling for the rounded rectangle template shape 910 is modified to form the rounded rectangle shape 920, rather than taking an elliptical form as would be expected with an affine scaling. Such behaviour can be modelled by adding an extra control parameter to the template shape corresponding to the aspect ratio (i.e., the ratio between the vertical and 15 horizontal scaling parameters) of the template shape. Line thickness parameters simply define the thickness of the lines that define a given template shape. In general there is only one line thickness parameter which defines the uniform thickness of the line for the entire template shape. Fig. 10 shows an unfilled hexagon template shape 1001 with a set of increasing line thickness parameters from left to 20 right on the page. The methods described herein may be implemented using a number of moments, such as the geometric moments, the central geometric moments and the Zernike moments.

- 18 The (p, q)lh order geometric moments are defined in terms of an integral over the area, A, covered by the interior of a given shape object (e.g., a graphical object) in accordance with Equation 3 as follows: Mpq = xpyqdxdy (3) A 5 where p and q are integers ;>0. The geometric moments may be determined in terms of a boundary line integral for speed. The moment Moo represents the area of the shape object and the point (xo, yo) = (Mio / Moo, MoI / Moo) represents its centroid. Central geometric moments are geometric moments of shape objects, for example, that have been translated such that their centroids are located at the origin in accordance 10 with Equation 4 below: mq = J -xo)Py-yo) dxdy. (4) A Zernike moments are often used in image processing shape recognition processing. Zernike moments have excellent properties in terms of distinguishing shapes using a small number of terms. Zernike moments are defined in terms of a set of basis functions that are 15 orthogonal over the unit circle. These basic functions are given in accordance with Equation (5) as follows: V,,q(r,6) = Rpq(r)e''', (5) where: 2 R,q(r)= S(k,p,q)r-2* k=0 {kJ'~ k - k)! ! q| p,(p - Jq|)even S(k,p,q)=- [k ot-k k h! (2 2 0 otherwise - 19 The Zernike moments are determined in accordance with Equation (6) as follows: zpq = JjrdrdoVpq(r, 0) f(r, 0) (6). unit circle The methods described below make use of an alternative representation of a shape 5 object which will be referred to as a "normalised shape object". The normalised shape object is based on the original shape object shifted such that the centroid of the shape object coincides with the origin and modified to reduce the effects of rotation and stretching. The normalisation is preferably represented by an affine transformation. Preferably, the general affine normalisation, which is a series of linear transformations to 10 map an arbitrary two dimensional (2D) shape object to its canonical form, is used to perform the normalisation. Specifically, the original shape object is shifted so that the centroid of the shape object is at the origin, stretched along the x and y axes so that the second order moments become I (M 20 = M 02 = 1), rotated by 45 degrees and then stretched again so that the second order moments are again I (M2 0 = M 02 = 1). Alternative 15 normalisation methods may be used, for example based on the principal axis and bounding box of the original shape object. The affine normalisation described above may be represented in terms of the following matrix Equation 7, as follows: = N. , (7) Y'f y - Ay 20 where (x, y)T and (X', Y')T define a particular point in the shape object before and after the transform respectively, (Ax, Ay)T is the location of the centroid of the original shape object, and the normalisation matrix N is defined in terms of the second order central geometric moments using Equation 8, as follows: -20 1(n,+ n_) 1.(n.- n_)' N = m_2 2 m0 2 (8) 1 (n.--) I (n, +n-)' where 1 1 n+n 1+ mn 1 i_ m 1 M20M02: m 20 m 02 5 The methods described below may be used to analyse a bitmap image of a document with graphical regions to generate a document with modifiable template shapes and connectors. Many of the described methods may also be applicable to the analysis of hand drawn digital ink inputs, for example, in tablet PCs and whiteboard applications. Fig. 11 is a flowchart showing a method 1100 of creating a document comprising a 10 modifiable shape (i.e., a closed-form non-textual template shape), according to one embodiment. The method 1100 creates the document by converting an input image with graphical regions into a document with modifiable template shapes and connectors (i.e., dynamic line objects that connect template shapes within a document). The method 1100 may be implemented as software resident on the hard disk drive 2510 and being controlled 15 in its execution by the processor 2505. As shown in Fig. 11, the method 1100 begins in step 1110, where a bitmap image undergoes a low-level image segmentation. In this manner, the bitmap image is segmented by the processor 2505 into one or more regions according to colour. At the next step 1120, the processor 2505 performs a high-level document layout analysis on the segmented 20 regions of the bitmap image to identify regions of various document content types. In the exemplary embodiment, the regions are rectangular and the content types include the -21 following classifications: text, photographs (halftone) and graphical objects. Each of these content types is made up of pixel data. In alternate embodiments other document content types such as table and line-drawing may be included. The classified regions may also be referred to as zones, blocks or regions of interest. The bounding boxes of the regions may 5 overlap. At the next step 1130, the detected graphics regions are further analysed to detect graphical objects within the graphics regions, in accordance with a method 1200 which will be described in more detail below with reference to Fig. 12. Accordingly, at step 1130, the processor 2505 performs the step of analysing graphics regions of the bitmap 10 image to detect bitmap representations of graphical objects and a bitmap representation of a line object. The method 1100 continues at the next step 1140, where the processor 2505 performs the step of matching each of the detected graphical objects with one of a plurality of predetermined template shapes, as described above, to determine the identities of the 15 individual detected graphical objects. In this instance, the template shapes are all modifiable closed-form non-textual template shapes. Each of the detected graphical objects is assigned a label representing the template shape that is matched with that graphical object. This label may be referred to as a "recognised shape label", as described above. 20 Each template shape may have a finite set of one or more predetermined non contiguous connection points. As described above, each of the template shapes typically comprises at most four control parameters for modifying the closed-form non-textual template shape in a non-affine manner. The number of control parameters of the predetermined template shape is typically less than the number of sections making up the -22 template shape. A method 1300 of matching a detected graphical object with a predetermined template shape, as executed at step 1140, will be described in detail below with reference to Fig. 13. The output of the method 1300 is one or more candidate template shapes and associated shape parameters. 5 The method 1100 continues at the next step 1150, where the processor 2505 performs the step of merging a plurality of the template shapes matched in step 1140, to form one or more multi-part disjoint template shapes. In particular, at step 1150, the processor 2505 identifies any group of matched template shapes that may be interpreted as a multi-part disjoint template shape. All the template shapes in an identified group are 10 assigned a single label (i.e., a recognised shape label) representing a particular multi-part disjoint template shape. For example, Fig. 31 shows three closed contours 3110, 3120 and 3130, individually labelled as two rectangle template shapes 3110 and 3120 and an arrow template shape 3130. The template shapes 3110, 3120 and 3130 may alternatively be labelled as a "striped arrow" multi-part disjoint template shape 3140. A method 2800 of 15 merging template shapes, as executed at step 1150, will be described in more detail below with reference to Fig. 28. The method 2800 assigns a single label to any identified group of template shapes where the single label represents a particular multi-part disjoint template shape. At the next step 1160, the processor 2505 performs the step of creating an editable 20 version of the document comprising the template shapes with any line objects (or connectors) connected thereto. As described below, the template shapes within the document may also include multi-part disjoint template shapes. The matched template shapes and multi-part disjoint template shapes are all closed-form non-textual template shapes, as described above. The editable version of the document comprises any - 23 recognised text, document elements, and the in-painted background. A method 2000 of creating an editable document comprising the matched template shapes, and other document elements, as executed at step 1160, will be described below with reference to Fig. 20. 5 The method 1200 of analysing the graphics regions (i.e., as identified in step 1120) to identify (or detect) graphical objects within the graphics regions, as executed at step 1130, will now be described with reference to Fig. 12. The method 1200 may be implemented as software resident on the hard disk drive 2510 and being controlled in its execution by the processor 2505. 10 The method 1200 employs a loop structure to generate line and solid filled graphical objects from the graphics regions identified in Step 1120. The method 1200 begins at step 1210, where the processor 2505 selects a segmentation colour mask/layer with graphics regions for processing. Then at the next step 1220, the processor 2505 extracts both open and closed line objects as vectors from 15 the colour layer. An open line object consists of one or more connected line segments with two unconnected ends. A closed line object consists of a number of connected line segments that form a closed polygon. Preferably a line segment is a straight line. In alternate embodiments a line segment may be a curve. Any suitable method of extracting open and closed line segments may be used at step 1220. 20 The colour layer is further processed at step 1230 to extract non-line objects such as filled shapes. The extracted filled shapes are preferably represented as polygon objects. A method 2100 of extracting non-line objects, as executed at step 1230, will be described in more detail below with reference to Fig. 21. At the next step 1240, if there are any more - 24 colour layers, then the method 1200 returns to Step 1210, where the next colour layer is selected and processed. Otherwise, the method 1200 concludes. The method 1300 of matching a detected graphical object with a predetermined template shape, as executed at step 1140, will now be described with reference to Fig. 13. 5 The method 1300 may be implemented as software resident on the hard disk drive 2510 and being controlled in its execution by the processor 2505. The method 1300 employs a loop structure to select a best matched template shape for each detected graphical object. The method 1300 begins at step 1310, where the processor 2505 selects a next detected graphical object for processing. Then at the next step 1320, a set of candidate 10 template shapes for the graphical object is determined using a machine learning classifier. A method 1700 of determining a set of candidate template shapes for a graphical object, as executed at step 1320, will be described in more detail below with reference to Fig. 17. A second loop structure is employed in the method 1300 to process all selected candidate template shapes in turn to identify the best matching template shape for the 15 current graphical object. At step 1330 a next candidate template shape is selected, and then in step 1340, the set of shape parameters of the candidate template shape that modifies the candidate template shape to best match the current graphical object and a match score that quantifies how well the graphical object and the modified template shape match each other are determined. A method 1800 of determining a set of parameters for modifying a 20 candidate template shape to match a graphical object as executed at step 1340 will be described in more detail below with reference to Fig. 18. From step 1350, the method 1300 returns to step 1330 if there are more candidate template shapes to be analysed. Otherwise, the method 1300 continues at step 1360, where the candidate template shape with the highest matching score to the current graphical - 25 object is selected. Then at step 1370, if that score is greater than a predetermined threshold for shape recognition, then the method 1300 proceeds to step 1380. If the score is greater than the predetermined threshold, the candidate template shape selected at step 1360 with associated shape parameters is stored in memory 2506 for output at step 1380, then the 5 method 1300 continues to step 1390. Otherwise, processing continues directly to step 1390 from step 1370. At step 1390, if there are any more graphical objects to be processed, then the method 1300 returns to step 1310. Otherwise, the method 1300 is complete. The method 1700 begins at step 1710, where the processor 2505 determines a 10 normalised shape object of a current graphical object. At step 1720 the processor 2505 extracts a set of normalised shape feature values from the normalised shape object. These feature values may be based on simple moments, Fourier descriptors, or simple radial distance functions. Then at step 1730, a current normalised shape feature vector is analysed using a 15 previously trained machine learning classifier to generate candidate template shapes. The classifier is preferably trained using the Support Vector Machine (SVM) algorithm. In alternate embodiments other machine learning algorithms may be implemented for training and classification. Examples of other suitable machine learning algorithms may include artificial neural networks, k-nearest neighbour and decision tree. 20 Optionally, step 1730 of the method 1700 may be followed by step 1740 to improve the classification rates. Step 1740 uses a confusion matrix generated by the classifier to rationalise the set of candidate template shapes. The confusion matrix specifies the likelihood that a given template shape may be confused with (or misclassified as) another template shape. If a candidate template shape shares a high confusion likelihood with -26 another template shape, then the latter template shape is also selected as a candidate template shape for further processing. The method 1800 of determining a set of parameters for modifying a candidate template shape to match a graphical object, as executed at step 1340, will be described in 5 more detail below with reference to Fig. 18. The method 1800 determines a set of transform parameters that optimally modifies the candidate template shape to match the graphical object. The transform parameters preferably consist of two scaling parameters s, and sy, two offset parameters Ax and Ay, a rotation angle, and, depending on the graphical object, some additional control parameters. The method 1800 may be carried out using sets 10 of features or descriptors of the graphical object and its normalised form such as moments. In the exemplary embodiment geometric, central geometric and Zernike moments of the normalised graphical object are used to estimate the template shape parameters. At step 1810, if the processor 2505 determines that the template shape being matched is modifiable by at least one control parameter, then the method 1800 proceeds to step 15 1840. Otherwise, the method 1800 proceeds to step 1820. At step 1820, if the processor 2505 determines that the template shape has a shearing parameter, then the method 1800 proceeds to step 1830. Otherwise, the method 1800 proceeds to step 1850. At step 1830, the processor 2505 determines the affine transform parameters for a template shape with a skew parameter by specifying a set of possible scaling, rotation and 20 skew parameters for the graphical object. Preferably, at step 1830, a set of potential rotation parameters for the normalised template shape is determined, and then a matrix equation is solved in terms of the four unknown parameters that define the scaling in x and y- axes (s, and sy), the rotation (0) and the skew control parameter, c of the template shape respectively. If the normalised template shapes are rotationally symmetric and the - 27 order of symmetry of this normalised template shape is k, the set of possible normalised space rotation angles on may be determined by comparing the Fourier phases of the normalised shape and a normalised template shape. The Fourier phase of a given shape is given by Equation 9: 5 Argff e-k(cosa+isin a)r da drjJ k(A) =(9) k where A is the area covered by the shape. The angles on are then given by Equation 10: 10 #, = c(A') -(A7')+ 2nn /k (10) where A' is the area of the normalised shape, AT is the area of the normalised template shape, and n is an arbitrary integer in the range (0,..., k-1 }. If the normalised template shape is not symmetric then alternative rotation methods may be used such as those 15 described below. For each value of the angle on, the following matrix equation can be solved in terms of the four unknown parameters that define the scaling in x- and y- axes (s, and sy), the rotation (0) and the skew control parameter, c in accordance with matrix Equation 11: 2 sinO -cosO s, 0 1 c _ sin#, -cos#,, 201 1 = N 1)i ycosO sinO )y 0 s, )y0 1) cos#,~ sin#~)(1 - 28 where N is the affine normalisation matrix of the template shape. A set of k potential affine transform parameter sets corresponding to the k possible values of the angle o" are found. In the exemplary embodiment, solutions which are outside the valid range for the parameters are removed, while the remaining solutions are passed on to step 1860. 5 Parameter determination for shapes with control parameters is performed in step 1840. A method 1900 of determining optimal control parameter values that transform a template shape to match a graphical object, as executed at step 1840, will be described in further detail with reference to Fig. 19. At the next step 1850, a set of potential rotation and scaling parameters are 10 determined for the template shape. A method 2400 of determining rotation and scaling parameters for use in modifying a template shape to match a graphical object, as executed at step 1850, will be described in detail below with reference to Fig. 24. The method 1800 continues at the next step 1860 where the x- and y-offsets of the graphical object, (Ax,Ay), are determined for each set of estimated parameters from 15 previous processing steps. In particular, at step 1860, the processor 2505 determines the centre of mass of the graphical object and shifts the template shape so that the centre of mass of the template shape is in the same location as the graphical object. The method 1800 then continues to step 1870, at which a score for each possible set of parameters is determined. The score is a measure of how well the graphical object 20 matches the template shape for the given set of parameters. In the exemplary embodiment, the score, f, is determined by first determining the integral, d, of the square distance between each (transformed) template shape contour and the closest point on the matched graphical object contour, computed around the boundary of the template shape contour. If the template shape has more contours than the matched graphical object then for each - 29 additional contour not present in the matched graphical object, then the integral, d, is over the distance from the points on the boundary of the template contour to its geometric centroid instead. The actual score,f, is normalised in accordance with Equation 11, as follows: 5 f= I (11) 1+6 ' Where di is the square distance integral, A; is the area and pi the perimeter of the i-th contour. In alternative embodiments, other distance metrics may be used such as the Hausdorff distance. The method 1800 concludes at the next step 1880, where the processor 2505 10 determines the set of parameters having the highest score. Returning to step 1840, the best parameter set is determined to be the parameter set with the highest match score. This match score is then passed back for use in step 1360 for comparing the candidate template shape to the other candidate template shapes. The method 1900 of determining optimal control parameter values that transform a 15 template shape to match a graphical object, as executed at step 1840, will now be described in detail with reference to Fig. 19. The method 1900 may be implemented as software resident on the hard disk drive 2510 and being controlled in its execution by the processor 2505. The method 1900 begins at step 1910, where a set of descriptors is determined for 20 the graphical object. This can be any set of shape features that can be represented as a function of the graphical object control parameters. In the exemplary embodiment, the Zernike moments of the graphical object are used, but other descriptors such as Fourier descriptors, central geometric moments, or more specific features such as the boundary -30 length, or the location of identifiable features such as corners may be used, as well as any combination of different types of features. In step 1920 the descriptors of the graphical object are compared with those of the template shape. A distance between the template shape descriptors for a given set of 5 control parameters and the graphical object descriptors is defined. In the exemplary embodiment this distance is the sum of the squares of the differences between the graphical object descriptors and the corresponding template shape descriptors. The value of this distance measure is a function of the control parameters for the graphical object. If there are N control parameters then the distance measure defines an N-dimensional function. 10 Estimates of the control parameters can be determined by finding the minima of this function, typically using function minimisation methods. The method 2000 of creating an editable document comprising the matched template shapes, including multi-part disjoint template shapes, and connectors (i.e., line objects that connect to and/or between template shapes within a document), and other document 15 elements, as executed at step 1160, will now be described with reference to Fig. 20. The method 2000 may be implemented as software resident on the hard disk drive 2510 and being controlled in its execution by the processor 2505. The method 2000 begins at the first step 2010, where the processor 2505 opens a document file in memory 2506 for writing and prepares the document file for entering the 20 template shapes. At the next step 2020, the background image layer is inpainted. Inpainting removes the recognised text and shapes from the original image. Pixels which are part of a shape or recognised as text are replaced with the surrounding background colour. At the next step 2030, this inpainted background image is optionally written to the document file, to serve as the background of the document. This ensures that any objects -31 not recognised, which may include things such as photographic images, are still visible in the document. In step 2040, the matched template shapes, including any multi-part disjoint template shapes, and classified straight and elbow connectors are written to the document file with the corresponding parameters. The connectors are output as connecting to the 5 template shapes that the connectors have been recognised as connecting to. At step 2050, any remaining line objects are output as lines to the document file. Then at the next step 2060, the text and tables recognised earlier are added to the document file. The method 2000 concludes at the next step 2070 where the document is complete and the document file is closed. 10 The method 2100 of extracting non-line objects from a colour layer, as executed at step 1230, will now be described with reference to Fig. 21. The method 2100 may be implemented as software resident on the hard disk drive 2510 and being controlled in its execution by the processor 2505. The method 2100 begins at the first step 2110, where the outside boundary of the 15 non-line object to be extracted is determined. Step 2110 may be performed by starting at a point of the object known to be on the boundary of the object, such as the first object pixel in raster order. Then the boundary of the non-line object is followed by moving from each boundary pixel to the next eight (8)-connected pixel in a clockwise fashion. Once the starting point is reached again the outside boundary of the non-line object is complete. 20 At the next step 2120, a tree of critical points is generated for the non-line object to be extracted. First a pair of starting points is found. The first starting point is the furthest point from the centre of the non-line object to be extracted, and the second is the furthest point on the boundary of the non-line object from the first point. This defines a simple polygon of two lines, one starting at the first point and ending at the second, and the other -32 starting at the second point and ending at the first. Each line represents a summary of a section of the boundary of the non-line object to be extracted, where the section of boundary is made up of the boundary points found by moving between the start and the end of the line in a clockwise direction. Each line is then recursively split into two more 5 lines, until a minimum line length is reached. In one embodiment the minimum line length is four (4) pixels. The splitting point used in the recursive splitting operation is the furthest point from the line on the section of boundary that line represents. Both end points of each line are therefore on the boundary of the non-line object, and each line represents a section of the boundary of the non-line object to be extracted. All of the lines formed during step 10 2120 are stored in memory 2506 and are represented in a tree structure, where each line is a parent of the two lines that the line is split into. At the next step 2130 a measure of significance is given to each line in the tree. In one embodiment the significance is defined as the length of the line divided by the distance of the furthest point on the section of the boundary of the object that line represents from 15 the line. In one embodiment, the distance of the furthest point is increased to a minimum value if the distance is too low. Further, in one embodiment this minimum value is a distance of one (1) pixel. The method 2100 concludes at the next step 2140, where the dominant lines are written to a list configured within memory 2506. The dominant lines are those which have 20 a higher significance than all the lines underneath them in the tree. An ordered set of dominant lines is formed which represents the entire boundary of the object in the form of a polygon, and so the polygon creation step 1230 is complete. The method 2400 of determining rotation and scaling parameters for use in modifying a template shape to match a graphical object, as executed at step 1850, will now - 33 be described with reference to Fig. 24. The method 2400 estimates the scaling parameters (s, and sy) and rotation, 0, in more detail. These parameters may be estimated using sets of features of a current graphical object. In the exemplary embodiment, the scaling parameters are estimated using combinations of central geometric moments that are 5 invariant to rotation. Let (me, m 20 ,mo 2 ) and (fiio, i20,1;o2} denote the sets of central geometric moments of the untransformed shape template and graphical object, respectively. The scaling parameters, denoted by s,, and sy, are determined in accordance with Equations 12 and 13: inoo =| s Moo (12) 10 m 20 + Mi 0 2 =|s~s, (s m 2 o + s(' 2m 2 ) (13) At step 2410, two pairs of solutions for the absolute values of the scaling parameters that satisfy the Equations (12) and (13), are determined in accordance with Equations 15, 16, 17 and 18: s ni +nio 2 + (m 2 o +nio 2 ) -4K'm 20 mo 2 Is,|, 2Km 2 , (15) K 15 is , (16) ni2 +m~o -Q(S20+ mo 2 )2 -4Kmmo s2= ,2 2Km 2 0 4K M 20 m 0 2 (17) is~ Km 2o s K (18) where K = .__ moo -34 Once the two pairs of solutions are found for the scaling parameters, at the next step 2420 the processor 2505 determines values for the rotation angle. In the exemplary embodiment, fourth order Zernike moments are used to determine the base rotation angles. Zernike moments are complex (comprising a real part and an 5 imaginary part) shape descriptors which may be used for shape detection. Fourth order Zernike moments may be expressed in terms of zeroth, second and fourth order central geometric moments in accordance with Equations 19 and 20: z 44 =- [(m 40 -6m 22 + m4)+ 4i(m 3 - M 3 1 )] (19) z 42 =5[(4m 4 , -4m4 -3m2o +3mo2) + 2i(3m 1 - 4m3 -4m 3 )] (20) 10 where i -1. Let mpq denote the central geometric moments of the untransformed template shape, let m' denote those of the template shape after scaling along the x- and y-axis by s, and sy Pq respectively, and let TiPq denote those of the graphical object. Similarly let z', and 4, , n = 2, 4, denote the fourth order Zernike moments of the scaled template shape and the 15 graphical object respectively. In the exemplary embodiment, the rotation angles used are the roots of Equation 21: ao + a, cos 20 + a 2 cos2 20 + a 3 cos3 20 + a 4 cos 4 20 (21) where ao = A 2

-C

2 20 a, =2(AB-CD) a 2 = B 2

+C

2 -D2 + 2AE a 3 = 2(CD+ BE) - 35 a 4 = D 2 + E 2 A = -8(-z'4,z44, + Z' 4 rz 4 4 i) B =4(,z - Z 4 2 i 4 2,,) E = -2A 5 C =4(z' iz2 + Z 2 rz 42 r) D=16(' 4 if 44 i + Z 4 r z 44 r) and Z,,r = Re[z' ] z',,, = Im[z',] 10 Z 4 nr = Re[z 4 ,,] Z4,,, = Im[z 4 ,,] Equation 21 may be derived by finding the local minima of Equation 22: err =|Z' - z 44 +||Z' - z 42 (22) 15 using the relationship between the moments of a shape and the rotation of the shape in accordance with Equations 23, 24, 25 and 26: mq = sssIs,''mpq (23) in 0 0 = m' (24)

'

20 c -2cs s nm 20 Ci S = cs c 2 -s 2 -CS m (25) S s2 2cs c2 m' in 0 2 S, C5 C i 0 2

,

-36 -ii 4 4 0 c -4c's 6c 2 s 2 -4cs 3 s 'm 40 Mi3 cs c2(c 2 -3s 2 ) -3cs(c 2 -s 2 ) s 2 (3c 2 -s 2 ) - cs 3 m in 22 = c 2 s 2 2cs(c 2 -s 2 ) 1-6c 2 s 2 -2cs(c 2 -s 2 ) c 2 s 2 M2 2 (26) Fin 3 cs 3 s 2 (3c 2 - s 2 ) 3cs(c 2 -s 2 ) c 2 (c 2 -3s 2 ) - c 3 s M 1 3 S4 4 C362243 n 0 4 s 4cs 3 6c 2 s 2 4c's c 4 m 04 where c=cosO and s =sin9. The solutions of Equation 21 give base rotation angles for 6 for each of the solutions for the absolute values of the scaling parameters. For each value of 6 there are then up to 5 four solutions added corresponding to the four solutions for the signed scaling parameters: {sx, sy}, {s., -sy}, {-sx, sy}, {-sx,-sy}. Noting that taking the negative of both s,, and sy is the same as a one-hundred and eighty (180) degree rotation, the options may be written as 10 {s, |sy|, 0}, {s, |Sy1, 0 + 7r}, {sx, -Isyl, 0}, {s", -Isyl, 0 + 7r}. 15 For template shapes with x or y symmetry, the negatives of s,, and sy have no effect on the template shape so those possibilities may be ignored. Also, template shapes with one-hundred and eighty (180) degree rotational symmetry do not need to consider the rotated possibilities. Template shapes with both sorts of symmetry only need one solution possibility for each base angle.

-37 The method 2400 concludes at the next step 2430, where all of the solutions for each value of 8, as described above, are added to a list. The best solution will be decided in step 1870 based on the fit scores. The method 2800 of merging template shapes, as executed at step 1150, will now be 5 described with reference to Fig. 28. The method 2800 recognises (or identifies) groups of template shapes and relabels each of the template shapes in a particular group with a recognised shape label representing a multi-part disjoint template shape. The method 2800 may be implemented as software resident on the hard disk drive 2510 and being controlled in its execution by the processor 2505. 10 The method 2800 contains a loop structure to determine if each relabelled group of template shapes corresponds to a multi-part disjoint template shape. The method 2800 begins at step 2810, where the processor 2505 analyses the template shapes matched with the graphical objects in step 1140 and creates a list of subsets of the matched template shapes. Each of the subsets of template shapes are created 15 based on certain geometric criteria. In particular, each of the subsets of template shapes corresponds to a particular multi-part disjoint template shape. A method 2900 of creating a list of subsets of template shapes, as executed at step 2810, will be described in more detail below with reference to Figure 29. At the next step 2820, the processor 2505 determines a match score for a first one of 20 the subsets of template shapes in the list. The match score represents how well the first subset of template shapes matches the particular multi-part disjoint template shape corresponding to the first subset. A method 3000 of determining a match score for a set of template shapes, as executed at step 2820, will be described in more detail below with reference to Fig. 30.

- 38 The match score determined at step 2820 is a numerical match score in the form of a percentage representing how closely the particular subset of template shapes matches the particular multi-part disjoint template shape. This match score is compared, at step 2830 of method 2800, with a predetermined threshold which may be template specific. In one 5 embodiment, the threshold is 94% when the threshold is not template specific. If the match score determined at step 2820 is above the threshold, then the method 2800 proceeds to step 2840 of method 2800. However, if the match score is below the threshold, then the method 2800 proceeds to step 2850 of the method 2800. At step 2840 of method 2800, the processor 2505 analyses all of the template shapes 10 in the subset of template shapes and updates a recognised shape label list corresponding to each template shape in the subset. The recognised shape label list corresponding to a particular template shape contain recognised shape labels corresponding to that template shape. In particular, at step 2840, a label is added to the recognised shape label list for each template shape, where the label represents (or points to) the particular disjoint template 15 shape corresponding to the subset. At step 2850, the method 2800 analyses all of the template shapes in the subset of template shapes and updates the recognised shape label list associated with each of the template shapes to include the recognised shape label assigned to the template shape at step 1140. 20 At a following step 2860, if the processor 2505 determines that there are more sets of template shapes to be processed in the matched template shape list created at step 2810, then the method 2800 returns to step 2820 with the next set of template shapes as input. Otherwise, the method 2800 concludes.

- 39 The method 2900 of creating a list of subsets of template shapes, as executed at step 2810, will now be described with reference to Fig. 29. The method 2900 may be implemented as software resident on the hard disk drive 2510 and being controlled in its execution by the processor 2505. 5 The input to the method 2900 is the matched template shape list containing template shapes represented by closed contours with associated recognised shape labels . The template shapes in the matched template shape list are those template shapes that where matched with the graphical objects at step 1140. Each template shape may have an associated recognised shape label list containing one or more recognised shape labels 10 associated with the template shape. Preferably, each template shape has one label at a first step 2901 of the method 2900. The method 2900 employs several loop-structures inside each other in order to consider all template shapes in the matched template shape list. The method 2900 begins at first step 2901, where the processor 2505 selects a first template shape in the matched template shape list. Then at the next step 2903, the 15 processor 2505 selects a first recognised shape label (RS1) associated with the selected first template shape. The method 2900 continues at a next step 2905, where the processor 2505 selects a second template shape from the remaining template shapes in the matched template shape list. Then at the next step 2907, the processor 2505 selects a second recognised shape label 20 (RS2) associated with the selected second template shape. For example, if there are two shape templates (CC I and CC 2), and CC I has two labels denoted A and B, while CC2 has one label, C, then, at the end of a first step 2907, the first recognised shape label, RS 1, would first be A, while the second recognised shape - 40 label, RS 2, would be C. Then, after a second pass through the loop structure, at the end of step 2907, RS I would be B and RS 2 would be C. Once the template shapes are selected from the list, at the next step 2930, the labels associated with each of the selected template shapes are compared with a set of 5 predetermined pairs of potential labels to determine the compatibility of the selected labels. This step avoids having to do a full geometric comparison for all different pairs of template shapes. Figs. 33(a) to (c) show three exemplary sets 3330, 3350 and 3370 of label pairs of template shapes (represented by their corresponding closed contours for clarity). Each of the label pairs in the sets 3330, 3350 and 3370 represent template shapes that can be 10 merged to form disjoint template shapes in the form of a donut, a striped arrow and a cloud, respectively. The sets 3330, 3360 and 3370 of pairs of labels of Figs. 33(a) to (c) form rules for creating the list of subsets of template shapes, where each of the pairs (i.e., rules) are numbered in Figs. 33(a) to (c) for clarity. The sets 3330, 3350 and 3370 may be extended to include any further suitable combination of labels. 15 If the processor 2505 determines that the labels associated with the selected template shapes are not compatible (i.e., there is no corresponding label pair in any one of the sets 3330, 3350 and 3370), at step 2930, then the method 2900 proceeds to step 2980. If the labels are compatible, then the method 2900 proceeds to step 2940. For example, if the labels selected at steps 2903 and 2907 both represent circle template shapes, and thus 20 satisfy rule 3302 of set 3330, then the method 2900 proceeds to step 2940. Otherwise, if the selected labels represent a circle template shape and an arrow template shape, then the method 2900 proceeds to step 2980 as there is no rule in the sets 3330, 3350 or 3370 of Figs. 33(a) to (c) for such a case.

-41 The disjoint template shape (e.g., donut 3340, striped arrow 3360, cloud callout 3380) which the selected labels are compatible with will be referred to below as the "compatible disjoint template shape". The method 2900 continues at the next step 2940, where the template shapes selected 5 at steps 2901 and 2905 are compared in terms of their geometry. If the selected template shapes satisfy predetermined template-specific merging rules then the geometry of the selected template shapes is considered to be compatible and the method 2900 proceeds to step 2950. Otherwise, the method 2900 proceeds to step 2980 of method 2900. The template specific merging rules are manually selected for each of a plurality of disjoint 10 template shapes, as will be described in more detail below with reference to Figs. 33(a)-(c), 34, 35(a)-(c) and 36(a)-(b). In another embodiment, the template specific merging rules may be automatically learned using machine learning methods. The method 2900 continues at the next step 2950, where if the processor 2505 determines that the first recognised shape label selected at step 2903 already represents a 15 disjoint template shape (i.e., the first template shape is already a disjoint template shape), then the method 2900 proceeds to step 2970. Otherwise, the method 2900 proceeds to step 2960. At step 2960, control parameters and affine parameters (i.e., width, height, skew and rotation) associated with the second template shape are updated to correspond to the compatible disjoint template shape (e.g., a striped arrow) . That is, a first interpretation of 20 the second template shape is considered to be the compatible disjoint template shape. The manner in which the control parameters and affine parameters are updated for the second template shape at step 2960 is template-specific as will be described in more detail below with reference to Figs. 34, 35(a)-(c) and 36(a)-(b). In another embodiment, a general method is used to update the parameters of the second template shape. Such a method may -42 use machine learning techniques to update the parameters based on a large set of examples. The recognised shape label of the second selected template, as it was before it was updated by step 2960, is backed up and stored as an additional interpretation of the second template shape at step 2960. Also at step 2960, the recognised shape label of the first selected 5 template shape is updated to correspond to the compatible disjoint template shape. The recognised shape label of the first selected template shape, as it was before it was updated, is backed up and stored as an additional interpretation of the first template shape. If the processor 2505 determines that the first template shape is already a disjoint template shape, at step 2970, then the processor 2505 first updates the control parameters 10 and affine parameters of the recognised shape label associated with the first template shape (ie of the compatible disjoint template shape) instead. This update is only performed since it is possible to achieve a better estimate of the control parameters and affine parameters with the added information given by the second template shape. The recognised shape label associated with the first template shape is not necessarily backed up since this would 15 already have been performed when the first template shape became a disjoint template shape. The processor 2505 then updates the recognised shape label associated with the second template shape to correspond to the same compatible disjoint template shape (e.g., a striped arrow). Also at step 2970, the recognised shape label of the second template shape, as it was before it was updated by step 2970, is backed up and stored as an 20 additional interpretation of the second template shape at step 2970. Appending the original interpretation (i.e., originally assigned recognised shape label) of the template shapes to the end of the recognised shape label list for the template shapes ensures that the original interpretation is also considered when trying to create other disjoint template shapes. For example, rectangle 3210 in Fig. 32 may be part of either a - 43 right striped arrow template shape together with the arrow 3240, or part of a left striped arrow template shape together with the rectangle 3220 and arrow 3230. At the next step 2980, if the processor 2505 determines that all of the labels associated with the second selected template shape have been considered, then the method 5 proceeds to step 2985. Otherwise, the method 2900 returns to step 2907 and another recognised shape label associated with the second template shape is selected. At step 2985, if the processor 2505 determines that all of the remaining template shapes in the matched template shape list have been processed as the "second" template shape, then the method 2900 proceeds to the step 2990. Otherwise, the method 2900 10 proceeds to step 2905 where another template shape is selected from the matched template shape list. At step 2990, if the processor 2505 determines that all of the labels associated with the first selected template shape have been considered then the method 2900 proceeds to step 2995. Otherwise, the method 2900 returns to step 2910 to select another label 15 associated with the first selected template shape. At the next step 2995, if the processor 2505 determines that all the template shapes detected at step 1140 have been processed as the "first" template shape, then the method 2900 concludes. Otherwise, the method 2900 returns to step 2905 where another template shape is selected from the matched template shape list. 20 Once all pairs of recognised shape labels, both pre-existing and newly created, have been compared, all the template shapes that are potentially compatible with a multi-part disjoint template shape are passed on to the processing step 2820 of the method 2800 in the form of the matched template shape list.

-44 The method 3000 of determining a match score for a set of template shapes, as executed at step 2820, will now be described. The method 3000 determines a match score for each of the subsets of template shapes determined at step 2810. However, the method 3000 will be described with reference to the first subset of template shapes. The method 5 3000 determines whether the first subset of template shapes match a multi-part disjoint template shape. The method 3000 may be implemented as software resident on the hard disk drive 2510 and being controlled in its execution by the processor 2505. The result of the method 3000 is the match score and the corresponding subset of template shapes that matches the multi-part disjoint template shape. 10 The fist subset of template shapes generated in accordance with the method 2900 does not necessarily have the same number of template shapes as the corresponding multi part disjoint template-shape has parts. For example, a subset of template shapes determined at step 1140 may have fewer template shapes than the corresponding multi-part disjoint template shape has parts. Alternatively, the subset of template shapes may have the same 15 number of template shapes as the multi-part disjoint template shape has parts. In still a further alternative, the subset of template shapes determined at step 1140 may have more template shapes than the multi-part disjoint template shape has parts. The method 3000 begins at step 3010, where the processor 2505 forms potential sets of template shapes from the first subset of template shapes. The potential sets of template 20 shapes are matched against the particular multi-part disjoint template shape. The number of potential sets of template shapes formed in step 3010 depends on the number of template shape parts (n) in the multi-part disjoint template shape and the number of template shapes in the first subset of template shapes (m). If the number of template shapes in the first subset of template shapes is less than or equal to the number of template - 45 shape parts in the multi-part disjoint template shape (i.e. m n ) then there is a single potential set of template shapes formed at step 3010. Otherwise, at step 3010, the processor 2505 determines all possible sets of n of template shapes from the m templates, of which there are a total of M! 5 n!(m - n)! The method 3000 continues at the next step 3020, where the processor 2505 selects a next potential set of template shapes and determines how well the selected set of template shapes matches the multi-part disjoint template shape. The numerical match-score determined at step 3020 is equivalent to the score described above with reference to step 10 1870 of the method 1800 and is determined in the same manner. The match score may be interpreted as a percentage as described above. At the next step 3030, if the match score determined at step 3020 is the highest score determined for the selected set of template shapes, then the method 3000 proceeds to step 3040. Otherwise, the method 3000 proceeds to step 3050. At step 3040, the processor 15 2505 sets a variable "best match" to the value of the match score for the set of template shapes. At step 3050, if the processor 2505 determines that a match score has been determined for all of the potential sets of template shapes determined at step 3010 then the method 3000 concludes. Otherwise, the method 3000 returns to step 3020. 20 Some examples of pairs of template shapes that can be merged into a multi-part disjoint template shape will now be described with reference to Figs. 33(a)-(c), 34, 35(a) (c) and 36(a)-(b). As described above, Figs. 33(a) to (c) show three exemplary sets of label pairs 3330, 3360 and 3370 of template shapes (represented by their corresponding closed contours for -46 clarity) that may be merged to form a multi-part disjoint template shape in the form of a donut 3340, a striped arrow 3360 and a cloud callout, respectively. The set 3330 of Fig. 33(a) shows that two template shapes may potentially be relabelled as the donut multi-part disjoint template shape 3340 if the template shapes have previously been labelled as an 5 ellipse and a rounded rectangle (i.e., rule 3301). The reason for this is that circles can be represented as rounded rectangles, and so there is potential for ellipses to be mis recognised (or wrongly identified) as being rounded rectangles at the earlier stages of processing. Two ellipses (i.e., rule 3302) and two rounded rectangles (i.e., rule 3303) may similarly be relabelled as the donut multi-part disjoint template shape 3340. 10 For pairs of template shapes that satisfy any one of rules 3301, 3302 or 3303, the earlier stages of processing will have recognised the two template shapes and labelled them with the appropriate recognised shape labels and updated the control and affine parameters associated with the template shapes. Fig. 34 shows two template shapes (i.e., represented by closed contours) 3430 and 3440, which have been recognised (or identified) and stored 15 as two elliptical polygons 3420 and 3410, respectively. The polygon 3420 has a grey fill and the polygon 3410 has a white fill. In accordance with the exemplary embodiments, the two elliptical polygons 3420 and 3410 will be relabelled as a donut multi-part disjoint template shape if the aspect ratio of the two ellipses is the same and the two ellipses are concentric to within some predetermined tolerance. The resulting donut multi-part disjoint 20 template shape 3400 will have a control-parameter p that describes band thickness of the donut multi-part disjoint template shape 3400 as follows: w h 1 w 1 h, P=2 4 (27) - 47 where w is the width and h is the height of the small (s) polygon 3410 and large () polygon 3420. In accordance with rules 3304 and 3305, a donut multi-part disjoint template shape can already have been recognised (or identified) when another template shape satisfies the 5 geometric relationships described in the rules 3304 and 3305. In those cases, the new template shape is merged with the donut multi-part disjoint template shape without changing the parameters associated with the multi-part disjoint template shape. The set 3350 of Fig. 33(b) shows examples of pairs of template shapes (or recognised shape labels) that can be relabelled as the striped arrow multi-part disjoint template shape 10 3360. The striped arrow multi-part disjoint template shape 3360 is also shown in Fig. 36(a). As seen in Fig. 36(a) the striped arrow multi-part disjoint template shape 3360 comprises three template shapes (or parts) represented by closed contours 3610, 3630 and 3650, each represented by a polygon (the boundary) 3620, 3640 and 3660, respectively. The rectangular template shapes 3610 and 3630 may be misrecognised (or wrongly 15 identified) (e.g., in step 1140) as parallelogram template shapes. Template shape pairs that can be relabelled as a striped arrow multi-part disjoint template shape similar to the striped arrow multi-part disjoint template shape 3360 are therefore, a left-arrow and a parallelogram (i.e., rule 3306), a left-arrow and a rectangle (i.e., rule 3307), an up-arrow and a parallelogram (i.e., rule 3308), an up-arrow and a rectangle (i.e., 3309), a right-arrow 20 and a parallelogram (i.e., 3310), a right-arrow and a rectangle (i.e., 3311), a down-arrow and a parallelogram (i.e., 3312) and a down-arrow and a rectangle (i.e., 3313). In each case the parameter of the resulting striped arrow multi-part disjoint template shape will be different to those of the arrow and rectangle / parallelogram template shape, and so a - 48 mapping is performed. For rules 3306, 3307, 3308, 3309, 3312 and 3313 the rotation of the resulting striped arrow multi-part disjoint template shape is also different. As an example, Fig. 36(b) shows a striped arrow multi-part disjoint template shape 3600 corresponding to rule 3311 of Fig. 33(b). The striped arrow multi-part disjoint 5 template shape 3600 comprises a rectangular template shape 3670 which may be stored in memory 2510 as its boundary 3675 with a recognised shape label "rectangle". The striped arrow multi-part disjoint template shape 3600 also comprises an arrow-shaped template shape 3680 which may be stored in memory 2510 as its boundary 3685 with a recognised shape label "arrow". The rectangle template shape 3670 and the arrow template shape 3680 10 may be merged into a potential striped right arrow disjoint template shape if the rectangle template shape 3670 is sufficiently close (in terms of its Euclidean distance) to the arrow template shape 3680, and if the height of the rectangle template shape 3670 equals the width of the arrow template shape 3680 (i.e., without the arrowhead). Mapped scaling and parameters for the striped arrow multi-part disjoint template shape 3600 may be 15 determined as follows: w =1.185w , sa a a h =h, sa a p0 =1 1.185(1- pOa)' (28) plsa a ' where w represents width and h represents height, p0 represents a first parameter and p] is represents a second parameter, and where subscript a indicates original recognised arrow template shape 3680, sa represents the striped arrow disjoint template shape 3600. The two 20 parameters p0 and p1 correspond to coordinates of point 3601 where the main body of the - 49 arrow template shape 3680 meets the arrowhead. This is the case for both the striped arrow multi-part disjoint template shapes and normal arrow template shapes. After the striped arrow multi-part disjoint template shape 3600 has been created from the rectangle template shape 3670 and arrow template shape 3680, there will generally be 5 another rectangle remaining (e.g., rectangle template shape 3630). The rectangle template shape 3630 may in later processing be merged with the newly created striped arrow multi part disjoint template shape 3600. Accordingly, the processor 2505 may perform the step of merging one or more further rectangle template shapes with the matched template shapes (i.e., the template shape 3670 and the arrow template shape 3680) to form the 10 striped arrow multi-part disjoint template shape 3600. In the exemplary embodiment, the rectangle template shape 3630 is merged with the striped arrow multi-part disjoint template shape 3600 based on proximity to the rectangle template shape 3670 and/or the arrow template shape 3680 and no mapping of parameters is necessary. In another embodiment, the parameters associated with the striped arrow 15 template shape 3680 has its parameters updated. The set 3370 of label pairs of template shapes of Fig. 33(c) may be relabelled as the cloud callout multi-part disjoint template shape 3380. The cloud callout multi-part disjoint template shape 3380 is also shown in Fig. 35(a). As seen in Fig. 35(a), the cloud callout multi-part disjoint template shape 3380 comprises a main cloud template shape 3511 and three circle template shapes 20 3513, 3515 and 3517. The template shapes 3511, 3513, 3515 and 3517 are represented by their contours 3510, 3512, 3514 and 3516, respectively, and have previously been recognised and parameterised. The circle template shapes 3513, 3515 and 3517 may be incorrectly recognised as rounded rectangles in some cases. A new cloud callout multi part disjoint template shape may be created from a circle and a main cloud contour (i.e., -50 rule 3316) if the dimensions of the circle are a predetermined fraction of a dimension of the main cloud contour. The main cloud contour of rule 3316 has previously been recognised as the cloud callout multi-part disjoint template shape main contour 3710 as seen in Fig. 37(a). 5 In the exemplary embodiment, as seen in Fig. 35(a), the multi-part disjoint template shape 3380 has a largest circle template shape 3512 having a width and height which is point one six five (.165) that of a main cloud contour 3510. The disjoint template shape 3380 has another circle template shape 3514 having a width and height which is point one one zero (.110) that of the main cloud contour 3510. Finally, the disjoint template shape 10 3380 has a smallest circle template shape 3516 having width and height which is point zero five five (.055) that of the main contour 3510. If a pair of template shapes corresponds to either a main cloud contour template shape and a circle template shape (i.e., rule 3316) or a main cloud contour template shape and a rounded rectangle template shape (i.e., rule 3318) and the main cloud contour 15 template shape and the rounded rectangle template shape have any of the three size relationships described immediately above, then the main cloud contour template shape and the rounded rectangle template shape can be relabelled as a cloud callout multi-part disjoint template shape. Fig. 35(c) shows a main cloud contour template shape 3523 and a circle template shape 3524, having one of the size relationships described above, and 20 which can be relabelled as a cloud callout multi-part disjoint template shape. Relabelling a template shape as a cloud callout multi-part disjoint template shape requires two control-parameters to be added to the template shape. The two control parameters correspond to a relative position of the centre of the smallest ellipse in the cloud callout multi-part disjoint template shape relative to the centre of the main cloud -51 contour template shape when normalised so that the main cloud contour template shape has dimensions 1 x 1. The required normalisation of the main cloud contour template shape may be determined by rotation, 0, offsets, Ax and Ay, and scaling parameters, w and h, of a cloud callout multi-part disjoint template shape matched to the main cloud contour 5 template shape. All of the template shapes being matched are first negatively shifted by the offset parameters, Ax and Ay, along the x- and y- axes so that the multi-part disjoint template shape will be centred at the origin. The template shapes being matched are then rotated back by 0 so that the template shapes are oriented with respect to the x- and y-axes. The template shapes being matched may then be scaled by a factor of I / w with respect to 10 the x-axis and by a factor of I / h with respect to the y-axis. After the normalisation, the main cloud contour template shape is centred at the origin and roughly fill a unit square, while the other template shapes of the subset should be circular. It is possible to determine which of three circle template shapes of the cloud callout multi-part disjoint template shape a given template shape represents based on the 15 dimensions of the normalised main cloud contour template shape as described above. In the exemplary embodiment, if the average dimension of the circle template shape compared to the normalised main cloud contour template shape is closest to the value 0.165 then the circle template shape is the largest circle in the cloud callout multi-part disjoint template shape. If the average dimension of the circle template shape compared to 20 the normalised main cloud contour template shape is closest to the value 0.110 then the circle template shape is the middle circle. Finally, if the average dimension of the circle template shape compared to the normalised main cloud contour template shape is closest to 0.055 then the template shape is the smallest circle.

- 52 Three cases need to be considered depending on which circle template shape is being considered with the main cloud contour template shape. In the equations below, the circle template shapes and the main cloud contour template shape have been normalised so that the main cloud contour template shape has size lxI and is centred on the origin. 5 If the circle template shape 3517 of Fig. 35(a) is being considered, then the mapping is straight-forward: p0 x, ,(29) p1 ye. where x, and ye represent the x and y coordinates of the centre of the considered circle template shape after normalising everything as described above. 10 If the circle template shape 3512 of Fig. 35(a) is being considered, then the mapping is given by: p0 = (3 x + y 0.945)cos tan' ('3, p 1 +y -0.945)sin tan 21 Finally, if the circle template shape 3514 of Fig. 35(a) is being considered, then the mapping is given by: p0 = K x, +y - 0.278 cos tan' , 15 2 C(31) = x +y -0.278 sin tan- . After a circle template shape has been merged with the main cloud contour template shape as described above, and the template shapes have been relabelled as "cloud", there will generally be two remaining circle template shapes (e.g., the circle template shapes -53 3515 and 3513). The two remaining circle template shapes are merged with the cloud callout multi-part disjoint template shape so long as the remaining circle template shapes conform geometrically to the other template shapes in the cloud callout multi-part disjoint template shape (i.e. the above equations yield the same parameters to within a given 5 tolerance. In the exemplary embodiment, the tolerance is nought point three (.3). Fig. 35(b) shows a cloud callout multi-part disjoint template shape where three circle template shapes 3520 overlap. In this case, the three circle template shapes 3520 will not be recognised in previous processing stages as ellipses, but rather as a single multi-contour template shape called a "cloud caterpillar". Fig. 37(b) also shows a cloud caterpillar 3720. 10 In Fig 35(b), the cloud caterpillar 3520 is represented by multiple contours 3521. Accordingly, the rule-set 3370 of Fig. 33(c) comprises a rule (ie., rule 3320) for relabelling the cloud caterpillar 3520 and a main cloud contour template shape 3519 as the cloud callout multi-part disjoint template shape 3380. The set of rules 3370 also comprise a rule (i.e., rule 3321) for relabelling the cloud caterpillar template shape 3520 and a cloud 15 callout multi-part disjoint template shape as the cloud callout template shape 3380. The process for relabelling the cloud caterpillar template shape 3520 and a cloud callout multi part disjoint template shape as the cloud callout template shape 3380 is the same as when merging the circle template shape 3512 with the main cloud contour template shape 3510 of Fig. 35(a) since the biggest part of the cloud caterpillar template shape 3521 20 corresponds to the circle template shape 3512. A method 1400 of creating a document comprising a modifiable template shape, will now be described with reference to Fig. 14. The modifiable template shape is a multi-part disjoint template shape, such as those shown in Figs. 31, 34, 35(a)-(b) and 37. In one embodiment, the multi-part disjoint template shape is non-textual.

- 54 The method 1400 creates the document by converting an input image with graphical regions into a document with modifiable template shapes and connectors (i.e., dynamic line objects that connect template shapes within a document). The method 1400 may be implemented as software resident on the hard disk drive 2510 and being controlled in its 5 execution by the processor 2505. The method 1400 begins at step 1401, where the processor 2505 analyses an image to detect at least two graphical objects within the image. Graphical objects are detected within the image at step 1401 in accordance with the method 1200 described above. At the next step 1403, the processor 2505 matches each of the detected graphical 10 objects with a template shape. The detected graphical objects are matched with the template shapes at step 1403 in accordance with the method 1300. Then at step 1404, the processor 2505 matches at least two of the matched template shapes with a multi-part disjoint template shape. One or more parts of the multi-part disjoint template shape are modifiable through adjustment of one or more control parameters. The template shapes are 15 matched with the multi-part disjoint template shape in accordance with the method 2800 described above. The method 1400 concludes at the next step 1405, where the processor 2505 creates a document comprising the at least one modifiable multi-part disjoint template shape. The document is created at step 1405 in accordance with the method 2000. The template 20 shapes used in the described methods may be one of the template shapes defined by the MicrosoftTm AutoShape application. The template shapes may also be one of the template shapes defined by the OpenOffice.org editing applications such as WriterTM and ImpresSTM. Further, the template shapes may also be one of the template shapes defined by the VISIOW editing application. Still further, the template shapes may also be one of the -55 template shapes defined by the Smart DrawTM editing application developed by SmartDraw.com. Finally, the template shapes may be custom defined template-shapes such as, for example, the cloud callout main contour template shape or the cloud caterpillar template shape described above 5 Industrial Applicability It is apparent from the above that the arrangements described are applicable to the computer and data processing industries). The foregoing describes only some embodiments of the present invention, and modifications and/or changes can be made thereto without departing from the scope and 10 spirit of the invention, the embodiments being illustrative and not restrictive. In the context of this specification, the word "comprising" means "including principally but not necessarily solely" or "having" or "including", and not "consisting only of'. Variations of the word "comprising", such as "comprise" and "comprises" have correspondingly varied meanings. 15 - 56 Appendix A TABLE 1 SHAPE NAME REFERENCE Rectangle 27001 Round Rectangle 27002 Ellipse 27003 Diamond 27004 Isosceles Triangle 27005 Right Triangle 27006 Parallelogram 27007 Trapezoid 27008 Hexagon 27009 Octagon 27010 Plus Sign 27011 Star 27012 Arrow 27013 Thick Arrow 27014 Home Plate 27015 Balloon 27016 Seal 27017 Plaque 27018 Chevron 27019 Pentagon 27020 Seal8 27021 Seal16 27022 Sea132 27023 Wedge Rectangle Callout 27024 Wedge Rrect Callout 27025 Wedge Ellipse Callout 27026 Wave 27027 Left Arrow 27028 Down Arrow 27029 Up Arrow 27030 Left Right Arrow 27031 Up Down Arrow 27032 Irregularseal1 27033 Irregularseal2 27034 Lightning Bolt 27035 Heart 27036 Quad Arrow 27037 Left Arrow Callout 27038 Right Arrow Callout 27039 Up Arrow Callout 27040 Down Arrow Callout 27041 Left Right Arrow Callout 27042 Up Down Arrow Callout 27043 Quad Arrow Callout 27044 Left Up Arrow 27045 - 57 Bent Up Arrow 27046 Bent Arrow 27047 SeaI24 27048 Notched Right Arrow 27049 Block Arc 27050 Circular Arrow 27051 U Turn Arrow 27052 Flow Chart Process 27053 Flow Chart Decision 27054 Flow Chart Input Output 27055 Flow Chart Document 27056 Flow Chart Terminator 27057 Flow Chart Preparation 27058 Flow Chart Manual Input 27059 Flow Chart Manual Operation 27060 Flow Chart Connector 27061 Flow Chart Punched Card 27062 Flow Chart Punched Tape 27063 Flow Chart Extract 27064 Flow Chart Merge 27065 Flow Chart Online Storage 27066 Flow Chart Magnetic Tape 27067 Flow Chart Display 27068 Flow Chart Delay 27069 Flow Chart Alternate Process 27070 Flow Chart Off Page Connector 27071 Left Right Up Arrow 27072 Moon 27073 Seal4 27074 Double Wave 27075 Cube 27076 Can 27077 Donut 27078 Ribbon 27079 Ribbon2 27080 No Smoking 27081 Folded Corner 27082 Bevel 27083 Striped Right Arrow 27084 Vertical Scroll 27085 Horizontal Scroll 27086 Curved Right Arrow 27087 Curved Left Arrow 27088 Curved Up Arrow 27089 Curved Down Arrow 27090 Cloud Callout 27091 Ellipse Ribbon 27092 Ellipse Ribbon 2 27093 Flow Chart Predefined Process 27094 Flow Chart Internal Storage 27095 Flow Chart Multidocument 27096 Flow Chart Summing Junction 27097 - 58 Flow Chart Or 27098 Flow Chart Collate 27099 Flow Chart Sort 27100 Flow Chart Offline Storage 27101 Flow Chart Magnetic Disk 27102 Flow Chart Magnetic Drum 27103 Sun 27104 5

Claims (13)

1. A method of creating a document comprising a modifiable template shape, the method comprising the steps of: 5 analysing an image to detect at least two graphical objects within the image; matching each of the detected graphical objects with a template shape; matching at least two of the matched template shapes with a multi-part disjoint template shape, one or more parts of said multi-part disjoint template shape being modifiable through adjustment of one or more control parameters; and 10 creating a document comprising the at least one modifiable multi-part disjoint template shape.

2. The method according to claim 1, wherein said multi-part disjoint template shape is non-textual. 15

3. The method according to claim 1, further comprising the step of merging one or more further template shapes with the matched template shapes to form the multi-part disjoint template shape. 20

4. The method according to claim 1, wherein one or more of the template shapes is matched to more than one multi-part disjoint template shape.

5. The method according to claim 1, wherein each of the at least two template shapes have a single outer contour. - 60

6. The method according to claim 1, wherein the multi-part disjoint template shape is modifiable in a non-affine manner. 5

7. The method according to claim 1, wherein the multi-part disjoint template shape is a striped arrow disjoint template shape.

8. The method according to claim 1, wherein the multi-part disjoint template shape is a donut disjoint template shape. 10

9. The method according to claim 1, wherein the multi-part disjoint template shape is a cloud callout template shape.

10. A method of creating a document comprising a modifiable multi-part disjoint template 15 shape, the method comprising the steps of: analysing an image to detect at least two graphical objects within the image; matching each of the detected graphical objects with a template shape; matching at least two of the matched template shapes with a multi-part disjoint template shape that may be modifiable in a non-affine manner through adjustment of one 20 or more control parameters; creating a document comprising the at least one modifiable multi-part disjoint template shape. - 61

11. A method of creating a document substantially as described herein with reference to any one of the embodiments as that embodiment is illustrated in the drawings.

12. Computer apparatus configured to perform the method of any one of the preceding 5 claims.

13. A computer readable medium having a computer program record3ed thereon, the program being executable by a computer apparatus to perform the method of any one of claims I to 11. 10 Dated this 30th day of June 2008 CANON KABUSHIKI KAISHA Patent Attorneys for the Applicant Spruson&Ferguson 15