AU1754300A - Method and apparatus for modification of a natural image - Google Patents

Description

S&F Ref: 493010

AUSTRALIA

PATENTS ACT 1990 COMPLETE SPECIFICATION FOR A STANDARD PATENT

ORIGINAL

f t** f oft ro "i o Name and Address of Applicant: Actual Inventor(s): Canon Kabushiki Kaisha 30-2, Shimomaruko 3-chome, Ohta-ku Tokyo Japan Cameron Bolitho Browne, Paul Quentin Scott and Michael Richard Arnold Spruson Ferguson St Martins Tower 31 Market Street Sydney NSW 2000 Method and Apparatus for Modification of a Natural Image Address for Service: a Invention Title: ASSOCIATED PROVISIONAL APPLICATION DETAILS [33] Country [31] Applic. No(s) AU PP8911 [32] Application Date 26 Feb 1999 The following statement is a full description of this invention, including the best method of performing it known to me/us:- 5815c -1- METHOD AND APPARATUS FOR MODIFICATION OF A NATURAL IMAGE The present invention relates to the computer creation of characters and, in particular, to the creation of characters from components of natural images.

Background Art In the field of computer graphics there is a continuing demand for innovative techniques by means of which information can be conveyed in a personalised, highly distinctive fashion. Often large amounts of data do not need to be conveyed, however, generally some form of message content is to be included. Examples include screen savers where some form of personalised message, possibly a name, a slogan or similar, is to be incorporated in an image.

In order to make the image more personalised, the user wishes to select parts of an existing natural image which is some favourite of the user. For example, in order to construct a screen saver, a user may wish to incorporate portions of an existing image such as a photograph of the user, or a friend of the user, a photograph of a landscape or other similar scene, or like image.

It is the object of the present invention to provide a means by which components 20 of a natural image can be incorporated into characters which enable information to be conveyed.

Summary of the Invention In accordance with a first aspect of the present invention there is disclosed a method of modifying a character string having one or more characters said method comprising steps of: segmenting a source image into one or more image segments; identifying one or more source image features in said image segments; (Natural) (CFP1624AU Open 43) (493010) 1:%ELECCISRAOPEN\OPEN43\493010AU.DOC extracting a corresponding skeletal representation for each said source image feature; identifying one or more character stroke segments for each character in the character string; and matching one or more of said skeletal representations to each character stroke segment, thereby forming a modified character string comprising modified characters each said modified character having a visual impression corresponding to a visual impression of the source image.

According to another aspect of the invention, there is provided an apparatus adapted for modifying a character string having one or more characters said apparatus S: comprising: segmenting means for segmenting a source image into one or more image segments; first identifying means responsive to said image segments, said first identifying means adapted for identifying one or more source image features in said image segments; extracting means responsive to said source image features, said extracting means adapted for extracting a corresponding skeletal representation for each said source image °o feature; second identifying means for identifying one or more character stroke segments 20 for each character in the character string; and matching means for matching one or more of said skeletal representations to each character stroke segment, thereby forming a modified character string comprising modified characters each said modified character having a visual impression corresponding to a visual impression of the source image.

According to yet another aspect of the invention there is provided a computer readable memory medium for storing a program for apparatus which modifies a character string, said program comprising: code for a segmenting step for segmenting a source image into one or more image segments; (Natural) (CFP1624AU Open 43) (493010) I:\ELEC\CISRA\OPEN\OPEN43\493010AU.DOC code for a first identifying step for identifying one or more source image features in said image segments; code for an extracting step for extracting a corresponding skeletal representation for each said source image feature; code for a second identifying step for identifying one or more character stroke segments for each character in the character string; and code for a matching step for matching one or more of said skeletal representations to each character stroke segment, thereby forming a modified character string comprising modified characters each saud modified character having a visual impression corresponding to a visual impression of the source image.

Brief Description of the Drawings Embodiments of the present invention will now be described with reference to the drawings in which: i Fig. 1 is a natural image in the form of a photograph ofa person, Fig. 2 is a word in which each of the characters is formed from components of the natural image of Fig. 1, Fig. 3 is a representation of each of the characters of Fig. 2 divided into So, individual strokes, Fig. 4 is a street scene in which the word CAB is formed from images of taxi 20 cabs aligned so as to form the letters, Fig. 5 is a flow chart illustrating creation of characters from components of natural images, Fig. 6 depicts the processing of a source image, Fig. 7 illustrates an exemplary source image, Fig. 8 depicts feature extraction and skeletonisation of the source image, Fig. 9 depicts a text string process, Fig. 10 illustrates type setting for an input text string, Fig. 11 depicts stroke extraction for the type set input text string, Fig. 12 illustrates segmentation of stroke primitives, (Natural) (CFP1624AU Open 43) (493010) ]:%ELEC\CISRAOPENOpEN43\4930 1OAU. DO Fig. 13 illustrates "closest-fit" instantiation, with scale preserved between elements, Fig. 14 depicts "closest-fit" instantiation, with no scaling constraints, Fig. 15 illustrates "favourite element" instantiation, Fig. 16 illustrates a background image, Fig. 17 depicts a simple overlay, and Fig. 18 shows a general purpose computer upon which an embodiment of the present invention can be practiced.

Detailed Description Where reference is made in any one or more of the accompanying drawings to steps and/or features, which have the same reference numerals, those steps and/or features have for the purposes of this description the same function(s) or operation(s), unless the oo contrary intention appears.

In the context of this specification, the word "comprising" means "including principally but not necessarily solely" or "having" or "including" and not "consisting only of'. Variations of the word comprising, such as "comprise" and "comprises" have corresponding meanings.

As seen in Fig. 1, a natural image, typically a photograph, can be used as a source of components from which characters can be created. It is apparent from considering the natural image that it is possible to identify various portions of the natural image that can constitute straight lines, circles, curves, and like components which can be used to form building blocks from which characters can be constituted.

In Fig. 2 the word "Kit", being a person's nickname has been created from the various constituent parts of the natural image of Fig. 1.

(Natural) (CFP1624AU Open 43) (493010) I:kELECCISPA\OPENOPEN43IA93010AU.DOC The strokes which go to make up each of the characters of Fig. 2 are indicated in Fig. 3. It can be seen that there are eight such strokes counting the dot of the lower case i as a stroke and also dividing the crossbar of the lower case t into two strokes because of the junction of the upright of the t. Each of these strokes has then been identified with a particular component of the natural image of Fig. 1 and that component has then been used to fill or overlie the stroke of Fig. 3 in order to create the character of Fig. 2.

Whilst in Fig. 2 the closest feature of the natural image to the intended stroke is used to fill or overly the stroke, in Fig. 4 a different technique is used. Here the word CAB (being short for taxi cab) is created in which the strokes of the characters are filled by repeated identical images of an actual taxi cab in which the longitudinal axis of each taxi cab image is constrained to lie within a few degrees of the "line" upon which the letters are written. In the photograph this line is aligned with the street axis. This "tiling" technique differs from the segmentation/matching approach described in relation to Figs.

1 to 3, however the tiling technique achieves a similar visual flavour to that of the segmentation/matching approach.

o Turning now to Fig. 5, a process is described whereby a text string is overlayed onto a background image, where the text is made up of natural images extracted from a source image. The process begins with step 500, whereafter a number of processes are performed in parallel. The background image is loaded in process 508, and held in 20 readiness while the text string is being processed. The text string is processed in process 504, this processing including stroke extraction as described above in relation to Figs. 2 and 3. This process 504 is explained in more detail with reference to Figs. 9 to 12 below.

The source image from which the natural features are extracted to compose natural-look text characters, is processed in block 502, whereby various portions of the natural image are extracted as described above in relation to Figs. 1 and 2. This is described in more detail below with reference to Figs. 6 to 8. Also, as described further in relation to Fig. 6, image feature extraction can can make use of image segments derived from cut-out images which have been prepared off-line and stored in a segment library. An instantiation process 510 applies the features extracted from the source image in process (Natural) (CFP1624AU Open 43) (493010) 1:\ELECCISRAOPEN\OPEN43\493010AU.DOC 502 to the text string processed in block 504. This application is performed in a manner dictated by the instantiation and merge styles provided by the user in step 506. The instantiation process 510 produces natural-look text which is composed of the features extracted from the source image in process 502. This has been described above in relation to Figs. 1 and 2, and will be described in more detail below in relation to Figs. 13 to 15. At this point, instantiation process 510 has produced natural-look text which is now ready to be overlaid into the background image which was loaded in step 508. The natural-look text string is overlayed onto the background image by process 516. This composite image is now available for printing and/or saving and/or display as indicated by block 520, whereafter the process comes to an end at step 522.

S. ."Fig. 6 depicts the source image processing block 502 in more detail. A source image, exemplified by Fig. 7, is loaded in step 600. Thereafter, this image is segmented in process 602, this segmentation process being an initial coarse decomposition of the *source image into various feature zones. Image segmentation is an initial step in many image processing tasks, such as patent recognition, image coding, and image interpretation, and segmentation can be performed using either or both automatic and manual processes. Image segments can optionally thereafter be stored in a segment library for later retrieval. One method of segmentation utilises a region splitting and merging procedure, whereby an image is initially subdivided into a set of arbitrary, disjointed regions, which are then merged and/or split depending on whether the regions satisfy a certain homogeneity criteria. Another method for segmentation of images is based on a region growing principle of selecting a pixel adjacent to a region of pixels, which is most similar to the region of pixels. This method is controlled by choosing a small number of pixels called seeds. After seed selection, the segmented regions are grown in an interactive fashion.

Fig. 7 illustrates a segment 700 determined by the segmentation process 602.

After image segmentation 602, feature extraction step 604 extracts individual features from each segment. This is illustrated in Fig. 7, wherein individual features 702 and 704 have been extracted from segment 700. Once features are extracted in process 604, the (Naturai) (CFPI624AU Open 43) (493010) I:XELECCISRAOPEN\OPEN43493010AU.DOC -7individual features are skeletonised in process 606. Skeletonisation comprises a process whereby each feature is represented by a minimal "stick-man" representation of the feature. In this manner, feature 702 in Fig. 7 is skeletonised into skeleton 802 in Fig. 8.

Similarly, feature 704 in Fig. 7 is skeletonised to skeleton 804 in Fig. 8. The output of skeletonisation process 606 is output as depicted by arrow 524 in Fig. 6, which, making reference to Figure 5 is directed to the instantiation process 510.

Figure 9 describes the text string processing block 504 (see Fig. 5) in more detail. The text string of interest is loaded in step 900. Thereafter, the text string is typeset by process 902, this involving arranging the text in a desired order, as represented in Fig. 10. Thereafter, a stroke extraction process 904 is performed on the typeset text i string, the unordered strokes being represented in Fig. 11. After stroke extraction, stroke segmentation process 906 segments individual stroke primitives 1100 to 1104 (see Fig.

11), into segments 1200 to 1210 (see Fig. 12). This can be performed in various ways, for ~example by breaking individual primitives at points where curvature of the primitives changes (for example, at point 1212 in Fig. 12). The stroke segmentation process 906 °a also assigns an orientation to each segment as shown in Fig. 12. Primitives such as 1106 in Fig. 11 whose segmented representation is the point 1210 in Fig. 12, are also given a °onotional orientation, as indicated by arrow 1214 in Fig. 12.

Figs. 13 to 15 provide examples of images output by instantiation process 510 20 (see Fig. which is performed once source image process 502 and text string process 504 are completed. The instantiation process is performed, for example, by matching primitive stroke segments (see Fig. 12) to similarly-shaped image features (see Fig. 8).

This instantiation process 510 thus involves either or both "stroke/skeleton" matching (see Figs. 13 and 14) and tiling (see Fig. Fig. 13 presents a first example of "closest-fit" instantiation, where primitive stroke segments are matched to similarly shaped image features. An instantiation style constraint (see 506 in Fig. 5) dictates that the relative scale of the features are maintained according to the original source image (see Fig. 7).

(Natural) (CFP1624AU Open 43) (493010) I:IELEC\CISRAOPENOPEN43\93010A.DO Fig. 14 presents another example of "closest-fit" instantiation with no scaling constraints. Thus the scale relationship between features 1400 and 1402 has not been maintained according to the scale in the original source image (see features 708 and 706 in Fig. 7).

Fig. 15 provides yet another example of instantiation, this time however "favourite element" instantiation having been performed. Thus a particular feature in the source image is selected as a favourite (in this instance 702 in Fig. and this feature in the present example is fitted substantially perpendicular to the medial axis of the individual stroke primitives of the text.

The aforementioned instantiation examples all result in production of a text string with text characters having a natural-look. This look is drawn from features in the selected source image, in this instance illustrated in Fig. 7.

Fig. 16 depicts the background image, which is to have the text string overlayed therein according to process 516 in Fig. 15 Fig. 17 illustrates an application of the present embodiment to a conventional *o o graphic art technique, whereby an overlay of an opaque text string comprising a 1700 and an 1702 onto the image 1704. A natural-look text string, where the look is dictated by features selected from the desired source image Fig. is thus overlayed onto the desired background image as described in Fig. 16.

The text string 1700, 1702 is overlayed onto the image 1704 in a manner determined purely by the scale factor applied to the text string and the location and/or orientation of the text string within the boundaries of the image 1704. In other words, the scale factor and positioning of the text 1700, 1702 is not a function of the contents of the image 1704.

Fig. 18 shows how the system is preferably practised using a conventional general-purpose computer 1800 wherein the various processes described above are implemented as software executing on the computer 1800. In particular, the various process steps are effected by instructions in the software that are carried out by the computer 1800. The software is stored in a computer readable medium, is loaded onto the (Natural) (CFP1624AU Open 43) (493010) 1: rLECCISRAOPEN\OPEN43\493010AU.DOC computer 1800 from the medium, and is then executed by the computer 1800. The use of the computer program product in the computer creates an apparatus for processing source image and text string data, performing instantiation according to user directions, and thereby incorporating natural-look text into the selected background image. The computer system 1800 includes a computer module 1802, a graphics input card 1816, and input devices 1818 and 1820. In addition, the computer system 1800 can have any of a number of other output devices including a graphics output card 1810 and output display 1824. The computer system 1800 can be connected to one or more other computers using an appropriate communication channel such as a modem communications path, a computer network, or the like. The computer network can include a local area network (LAN), a wide area network (WAN), an Intranet, and/or Intemrnet.

Thus, for example, image frames selected in process 600 (Fig. 6) and 508 (Fig. can be input via graphics input card 1816. Control commands and the text string can be input via keyboard 1818, and/or mouse 1820, and the background image, incorporating the natural-look text can be output via graphics output card 1810. The computer 1802 itself includes one or more central processing unit(s) (simply referred to as a processor hereinafter) 1804, a memory 1806 which can include random access memory (RAM) and read-only memory (ROM), an input/output interface 1808, a graphics input interface *too., 1822, and one or more storage devices generally represented by a block 1812. The storage device(s) 1812 can include one or more of the following: a floppy disk, a hard o disk drive, a magneto-optical disk drive, CD-ROM, magnetic tape or any other of a number of non-volatile storage devices well known to those skilled in the art. Each of the components 1804, 1806, 1808, 1812 and 1822, is typically connected to one or more of the other devices via a bus 1814 that in turn can include data, address, and control buses.

The graphics interface 1822 is connected to the graphics input 1816 and graphics output 1810 cards, and provides graphics input from the graphics input card 1816 to the computer 1802 and from the computer 1802 to the graphics output card 1810.

The method of producing a character string may alternatively be implemented in dedicated hardware such as one or more integrated circuits performing the functions or (Natural) (CFP1624AU Open 43) (493010) I:\ELEC\CISRA\OPEN\OPEN43\493010AU.DOC sub functions of character string production. Such dedicated hardware may include graphic processors, digital signal processors, or one or more microprocessors and associated memories.

The foregoing describes only some embodiments of the present invention and modifications, can be made thereto without departing from the scope of the present invention, the embodiments being illustrative and not restrictive.

0* 0 0 0 0@e S9 5 0

S

S

0 0* 0 O 0 0505 060000 0 I:%ELEC\CISRA\OPEN\OPEN43\4931 OAU.DOC (Natural) (CFP1624AU Open 43) (493010)

Claims (9)

1. A method of modifying a character string having one or more characters said method comprising steps of: segmenting a source image into one or more image segments; identifying one or more source image features in said image segments; extracting a corresponding skeletal representation for each said source image feature; identifying one or more character stroke segments for each character in the character string; and :matching one or more of said skeletal representations to each character stroke segment, thereby forming a modified character string comprising modified characters each said modified character having a visual impression corresponding to a visual impression of the source image.

2. A method according to claim 1 whereby the source image is a natural ••go image. *•2

3. A method according to claim 1 whereby said character stroke segment identification step comprises sub-steps of: ooooo extracting one or more character stroke primitives for each said character; and segmenting the one or more character stroke primitives to form character stroke segments.

4. A method according to claim 1 whereby said skeletal representation matching step comprises one of: matching character stroke segments to skeletal representations, said matching maintaining a relative scale between skeletal representations; (Natural) (CFP1624AU Open 43) (493010) I:\ELECCISRAOPEN\OPEN43A93010AU.DOC -12- matching character stroke segments to skeletal representations, said matching not maintaining a relative scale between skeletal representations; and matching character stroke segments to skeletal representations on the basis of a favourite skeletal representations. A method according to claim 1 whereby the modified character string is overlayed onto a background image.

6. An apparatus adapted for modifying a character string having one or more characters said apparatus comprising: segmenting means for segmenting a source image into one or more image segments; :.:"-first identifying means responsive to said image segments, said first identifying means adapted for identifying one or more source image features in said image segments; extracting means responsive to said source image features, said extracting means .adapted for extracting a corresponding skeletal representation for each said source image feature; second identifying means for identifying one or more character stroke segments for each character in the character string; and 20 matching means for matching one or more of said skeletal representations to S-each character stroke segment, thereby forming a modified character string comprising modified characters each said modified character having a visual impression corresponding to a visual impression of the source image.

7. A computer readable memory medium for storing a program for apparatus which modifies a character string, said program comprising: code for a segmenting step for segmenting a source image into one or more image segments; (Natural) (CFP1624AU Open 43) (493010) I:\ELECCISRA\OPEN\OPEN43\493010AU.DOC -13- code for a first identifying step for identifying one or more source image features in said image segments; code for an extracting step for extracting a corresponding skeletal representation for each said source image feature; code for a second identifying step for identifying one or more character stroke segments for each character in the character string; and code for a matching step for matching one or more of said skeletal representations to each character stroke segment, thereby forming a modified character string comprising modified characters each saud modified character having a visual impression corresponding to a visual impression of the source image.

8. A computer readable memory medium according to claim 7, whereby said code for the second identifying step comprises: oo.o code for an extracting step for extracting one or more character stroke primitives for each said character; and code for a segmenting step for segmenting the one or more character stroke primitives to form character stroke segments.

9. A computer readable memory medium according to claim 7, whereby 20 said code for the skeletal representation matching step comprises one or more of: i code for a matching step for matching character stroke segments to skeletal representations, said matching maintaining a relative scale between skeletal representations; code for a matching step for matching character stroke segments to skeletal representations, said matching not maintaining a relative scale between skeletal representations; and code for a matching step for matching character stroke segments to skeletal representations on the basis of a favourite skeletal representations. (Natural) (CFP1624AU Open 43) (493010) I:\ELEC\CISRA\OPEN\OPEN43,493010AU.DOC -14- A method of producing a character string substantially as described herein with reference to accompanying Fig. Nos. 1 to 3, and 5 to 14, or with reference to accompanying Fig. Nos. 4 and 15, or with reference to accompanying Fig. Nos. 16 and

17. 11. An apparatus adapted for producing a character string substantially as described herein with reference to accompanying Fig. No. 18. 12. A computer readable memory medium for storing a program for apparatus which produces a character string, substantially as described herein with reference to accompanying Fig. Nos. 1 to 3, and 5 to 14, or with reference to accompanying Fig. Nos. 4 and 15, or with reference to accompanying Fig. Nos. 16 and 17. DATED this Fifteenth Day of February 2000 Canon Kabushiki Kaisha Patent Attorneys for the Applicant SPRUSON FERGUSON (Natural) (CFP1624AU Open 43) (493010) I:\ELEC\CISRA\OPEN\OPEN43\493010AU.DOC