WO2010139007A1

WO2010139007A1 - Apparatus and methods for storing image with embedded information

Info

Publication number: WO2010139007A1
Application number: PCT/AU2010/000676
Authority: WO
Inventors: Peter Bayley
Original assignee: Peter Bayley
Priority date: 2009-06-04
Filing date: 2010-06-01
Publication date: 2010-12-09

Abstract

Systems and methods for embedding ancillary data in an image and indexing the ancillary data to the image pixels. The specific ancillary data accessed depends on the indexing scheme and the pixel coordinates supplied to the ancillary data request. Various indexing schemes are described allowing flexible interaction between the image and the ancillary data. Ancillary data can be compressed and encrypted and can include procedural code allowing the image to respond to external events such as user input.

Description

Description Title of Invention: Apparatus and methods for storing image with embedded information

Technical Field

[1] The present invention relates to digital image processing. More particularly, the present invention relates to the embedding and indexing of ancillary information or data in a digital image file. This application claims the benefit of Australian Provisional Patent Application Number 2009902582 - "Smartlmages" filed on 4th June 2009. Background Art

[2] Previously, early digital image files were not designed to store ancillary data. Any information relevant to the content of the image needed to be stored separate to the image. Any connection between the ancillary data and the image itself needed to be established using external means. For example, the HTML language supports the MAP and AREA elements which can be used to associate regions of the image with links. The digital image, itself, remains simply an array of coloured pixels and is not able to provide any information relevant to the image, being only able to display itself on screen or for printing.

[3] Some modern digital image formats may have the ability to store some ancillary data.

For example, the JPEG or PNG digital image file formats provide a number of fields for storing ancillary data. One type of ancillary data currently available is EXIF (Exchangeable Image File Format) which is incorporated which is based on the Tagged metadata fields of the TIFF (Tagged Image File Format) format. Some image formats also allow for the storage of textual dat as comment fields. All these ancillary data storage options have been designed to hold relatively small amounts of text of relevance to the image in its entirety. For example, a comment field might contain the contact details of the image creator or copyright holder so that viewers of the image could contact the creator or copyright holder. The EXIF data might contain in- fomration on hte various setting s of the camera (F-Stop, Shutter speed etc) that were used to make the image.

[4] There is no system for defining a relation between the comment fields or other ancillary data stored in the image and the pixel image data. It would therefore be advantageous to augment an image file format to incorporate information relevant to the pixel data that constitutes the image. In particular, it would be advantageous to be able to associate specific parts of the image with specific parts of the ancillary or comment data. [5] It is an object of the present invention to provide a system or method for defining a relationship between the image pixel data and ancillary information stored inside the same image or inside a different image.

[6] Preferably, another object of the present invention is to provide a mechanism that associates parts of the image with corresponding parts of the ancillary information. Summary of Invention

[7] The present invention relates to the incorporation and indexing of ancillary information into an image and of the indexing of the ancillary information to the pixels of the image such that specific image pixels can be linked, through the indexes, with specific parts of the ancillary information. While some existing image formats support the incorporation of ancillary information that information is only linked to the image a whole. There is no provision for the association of specific ancillary information with specific image pixels. Disclosure of Invention Technical Problem

[8] Digital Image files are a widely used and supported method of supplying information. While a number of schemes exist for providing extra information (known as metadata or comments) in and on behalf of an entire image, these systems do not support the indexing of image pixels to specific parts of the ancillary data such that contextual data specific to a particular subset of the image can be accessed through reference to the indexed pixels. Technical Solution

[9] According to a first aspect of the invention there is provided a method for creating a digital image file including a plurality of ancillary data elements, wherein the digital image file comprises a plurality of pixel data elements, the method comprising the steps of:

• recording ancillary data elements into the digital image file according to an index scheme, such that each pixel data element is coupled to a corresponding ancillary data element through the index scheme.

[10] Preferably, the index scheme comprises the steps of:

• indexing the ancillary data element with a set of index numbers;

• associating one index number to at least one pixel data element. [11] Alternatively, the index scheme comprises the steps of:

• recording each pixel data element from the digital image file on a second digital image file;

• recording each ancillary data element that is associated with a corresponding pixel data element at a location next to the corresponding pixel data element on the second digital file. [12] Alternatively, the index scheme comprises the steps of:

• sorting each ancillary data element that is associated with a corresponding pixel data element according to a relative location of the corresponding pixel data element;

• recording the sorted ancillary data elements in the digital image file. [13] Alternatively, the indexing scheme comprises the steps of:

• defining a plurality of points comprising relative locations of the pixel data elements on the image, wherein the plurality of points define a polygon area;

• associating at least one ancillary data element to the polygon area.

[14] In another aspect of the invention, there is provided a system for creating a digital image file including a plurality of ancillary data elements, wherein the digital image file comprises a plurality of pixel data elements, the method comprising the steps of:

[15] In this specification, unless the context clearly indicates otherwise, the term

'comprising' has the non-exclusive meaning of the word, in the sense of 'including at least' rather than the exclusive meaning in the sense of 'consisting only of. The same applies with corresponding grammatical changes to other forms of the word such as 'comprise', 'comprises' and so on.

[16] Referring to Figures 1 to 3, there is shown an internal format of a modern digital image file, in this case, a Portable Network Graphics (PNG) file. The PNG file was specifically developed for use on the Internet and as a patent- free alternative to the Graphics Interchange Format (GIF) file that was encumbered by a patented compression algorithm. While the GIF format, which allows a maximum of 256 different colours in an image was appropriate to the early days of the Internet, modern image formats such as PNG and JPEG can support a much greater range of colours and even transparencies. They do this by supporting more data per pixel (short for Picture Element - a single 'dot' in an image). While the GIF image supports 8 -bits (1 byte or 256 values) linked to a colour table, the PNG image format supports up to 64 bits per pixel. Figure 1 shows the various pixel depths supported by the PNG format. While the PNG format also supports indexed images (where the pixel value represents an offset into a colour lookup table) it is the formats with greater pixel depth that are of interest.

[17] Pixel Depths A 24 bit PNG image with 8 bits per channel (i.e. 8 bits for Red, 8 bits for Green and 8 bits for Blue) provides a total of 16,777,216 colours. This format is normally called RGB (Colour type 2 in Figure 1). An additional 8_bit Alpha channel can be added for a total pixel depth of 32 bits. The Alpha channel stores transparency information for each pixel, allowing every pixel to vary between fully transparent (invisible) to fully opaque.

[18] Another supported PNG format doubles each of the pixel channels (Red, Green, Blue and possibly Alpha) to 16 bits each for a total of 48 bits per pixel (RGB) or 64 bits per pixel (RGBA). A 48-bit image can store 281,474,976,710,656 (281 Trillion) colours while the 64-bit format can store 18,446,744,073,709,551,616 (18,447 Trillion) colours. Obviously, this is more pixel storage than is needed for the display of colours, so the invention provides a mechanism for re-purposing some of the image pixels as either data themselves or, more commonly, as indexes into the ancillary data.

[19] Storing Ancillary data Both PNG and JPEG digital image formats support areas in the image file for the storage of text comments. Both the JPEG and PNG image file formats are implemented as a sequence of blocks of storage. The JPEG block is called a segment while the PNG block is called a chunk. JPEG segments can be approximately 64Kb long (2^Λ16) while PNG Chunks can be approximately 2Gb in size 2^Λ31). The JPEG image supports multiple COM (comment) segments and the PNG image supports multiple Text chunks in ASCII (tEXt), Unicode (iTXt) and compressed ISO-9959-1 Text (zTXt). In addition, the PNG image allows the insertion of additional, custom chunk types. The invention uses these comment areas and or additional chunks or segments in the digital image files as storage for the ancillary data and/or the indexing structures.

[20] The ancillary information or data is any form of digital information relevant to specific parts of the image. The ancillary data is stored as a sequence of bytes in the appropriate image file areas (originally designed for comments or in the form of additional chunk or segment types). While the invention supports the storage of text- based information in the ancillary data, it also supports the storage of binary data representing any other format. For example, other image files, video clips, audio clips, 2D and 3D vector data, compiled or source software programs or scripts can all be placed into the ancillary data areas and optionally indexed to the image pixels. In the case where the image format requires the stored data to be in textual format (eg ASCII), and the ancillary data is to be stored in an existing comment chunk or segment, then one of the available encoding systems such as Uuencoding, Base64, Hex, or BinHex can be used to convert the binary data (an audio wave file, for example) to an ASCII text equivalent for subsequent storage. In the case where the additional chunk or segment types are used to store the ancillary data, no ASCII conversion is necessary and the data can be stored directly in the native binary format. In the case where the data has been converted to ASCII for storage, once it has been retrieved, it can be converted back to binary for return to the requesting external program. The data can also be compressed where, as in the zTXt Chunk type of PNG, this is supported. An example of the use of this ability of the invention would be an image of an orchestra where the sound of each instrument is accessed as a stored audio file and played when the pixel or pixels depicting that instrument are referenced.

[21] In one case, the ancillary data may include the name 'Jane Smith' and be linked to those 20-30 pixels in the image that code for the face of Jane Smith. The invention allows the name 'Jane Smith' to be stored in the image itself (along with the other 299 names) and linked to the pixels in the image that represents that person.

[22] Following are some examples of the ancillary data, however the present invention is not limited to these examples.

[23] In one embodiment, the ancillary information is the actual estimated bone density of a particular pixel location in an X-ray image. The dual-energy x-ray absorptiometry (DXA) or bone densitometry is an enhanced form of x-ray technology that is used to measure bone loss. Normally the bone density is displayed on a computer screen. Using the invention, the actual bone density measurements could be incorporated into the image so that the software displaying the X-ray image could also provide the bone density at that pixel location.

[24] In another embodiment, the ancillary information is the Geographic measures of pixel locations in an aerial photograph. This kind of information can be associated with geographic areas represented by an aerial image. For example, the information specific to a particular point might be altitude or elevation, the slope or steepness of the ground, the aspect or orientation of the ground (the direction of the slope - is it to the North?), the type of geology, the type of vegetation cover (forest, swamp, grassland), the ownership of the ground, the average rainfall, average daytime temperature and so on. Again, the invention allows any or all of these values to be incorporated into the image and linked to the pixel representation through the indexing scheme.

[25] In yet another embodiment, the ancillary data can be an array of images relevant to the main image. The ancillary information does not need to be text. It can be any kind of data such as images, audio, binary data etc. Where the particular image format allows only text format, the binary data can be encoded using one of the available binary-to-text encoding schemes such as Uuencoding, BinHex or Base64. This process would make the final image slightly larger but ensure the text areas contain only text. If the main image contained a portrait (head and shoulders) of a person, then the series of images stored as ancillary data could contain other head and shoulders portrait images of the same person but taken from a range of slightly different viewpoints. As the user moves the cursor over the image and the underlying software requests ancillary data for different rows and columns, different images would be extracted from the ancillary data and used to replace the main image. The net effect would be that the portrait would appear to 'turn' as the user's mouse cursor passes over the image. [26] In the present invention, a PNG image is used as an example for storing the ancillary information in the pixel data block.

[27] A PNG image can consist of up to 64 bits of information, but usually has 32 bits (8 each for the Red, Green and Blue Channels and 8 bits more for the transparency-controlling Alpha channel).

[28] In one embodiment, an existing PNG RGBA image with 32 bits per 5 pixel is used for expanding to construct a 64-bits per pixel image. However, the extra 32 bits need not be used to code for colour information, but be used as an index for the pixel into its associated ancillary information. In another example, the extra 32 bits will be used to code directly for the ancillary information.

[29] Figure 2 shows a 32 bits RGBA pixel storage with 8 bits each for the Red, Green,

Blue and Alpha channels. Figures 3 and 4 show a 64 bits RGBA pixel storage with 16 bits each for the Red, Green, Blue and Alpha channels. This is a common format for PNG images as it allows up to 16,777,216 colours as well as 256 levels of transparency.

[30] In order to separate the bits allocated to a colour channel from those to be used for ancillary pixel data, we can 'mask' and 'shift' the relevant bits. Taking the 16-bit red channel above, for example, the following sequence extracts the two values, as shown in Figures 5, 6, and 7. In this example, some or all of the image pixels are re-purposed to provide either indexing structures or data directly, or both. The re-purposing is accomplished using a bit-mask operation with a possible subsequent bit-shift operation to isolate the normal pixel information from the re-purposed data. This is illustrated in Figures 6 (Masking and Shifting to extract the value in the 'red' field) and Figure 7 (Masking to retrieve the value in re-purposed field or the 'spare' field).

[31] In the present invention, a PNG image is also used as an example for storing the ancillary information in custom chunk types. The indexes which associate the image pixels with the ancillary information can also be stored in the custom chunk types. In the case where both the ancillary information and the indexing information that associates specific ancillary information with specific pixel locations (rows and columns), there is not need to re-purpose teh image pixel data for ancillary storage.

[32] After ancillary information is stored into the digital image format, an indexing scheme is required for building up the contextual relation between the various image pixels and the ancillary data. In the present invention, the contextual relation of the image coordinates with the ancillary data is accomplished by using different indexing schemes. The appropriate indexing scheme can be selected depending on the nature of the image pixels and the ancillary data being stored in order to optimise the size of the image.

[33] Different indexing schemes, such as but not limited to discrete data indexing, continuous data indexing or direct data indexing, are implemented for mapping a pixel row and column to the ancillary information associated with that pixel.

[34] In one embodiment, the ancillary data consists of images, each associated with one or more of the image pixels. The ancillary images can be presented to replace the original image pixels. An example of this method would be a picture of a face. As the user moves the cursor across the image, different ancillary images are extracted, based on the pixel row and column, and displayed as the image itself. If each ancillary image were of the same face from a different viewpoint, the effect would be one of the head turning left and right, up and down as the cursor is moved.

[35] In another embodiment, the ancillary information is in the form of digital audio streams. An example of the use of this method would be where the image is of a number of singers. When the cursor is over a particular singer, the voice of that singer is made available and could be played using the computer's audio system.

[36] In yet another embodiment, the ancillary information is in the form of URLs

(hyperlinks). An example of this method would be where the user accesses different web content depending on where the cursor is clicked on the image.

[37] Indexing The index is a structured data storage area supported either in the pixel fields or in the ancillary storage area. The index entries are each of fixed length and are stored in order so that the first index element represents the first pixel (left-most) or the first row (top-most) in the image, while the next index entry represents the second pixel (to the immediate right of the first pixel) in the same, top row but in the second column. Index entries are stored in order from the left-most column along the row to the right-most column at the end of the row. The next index entry represents the first pixel in the second row (immediately below the first row) with subsequent entries completing the second row and so on to the last column of the last row (at the extreme bottom right of the image).

[38] The invention interface which is part of this invention takes an image coordinates

(consisting of two numbers representing the column (left to right) and row (top to bottom) and calculates where to find the index entry. In the case where the index entry is stored in the image's pixel data, then the index information is accessed directly by accessing the pixel in that row and column. In the case where the index is stored in the ancillary data storage area, the index entry location is calculated as:

[39] image Width x (row - 1) + (column - 1) x indexSize.

[40] For example, if the image is 256 x 256 pixels, each index entry is two bytes and the invention interface has received a request for information for the image coordinates row=107, column=94, then the index value is stored at

[41] 256 x (107 - 1) + (94 -1) = 54,458.

[42] So the index value would be in bytes 54,459 and 54,460. [43] Indexing Schemes Various indexing schemes may be used for the present invention.

Different indexing schemes are chosen in each situation to suit a particular type of image and type of ancillary information. The essential difference between the indexing schemes rests with the relationship between individual image pixels and the ancillary information associated with those pixels. The invention includes, but is not limited to, indexing schemes discussed and illustrated below.

[44] Following are some examples of indexing schemes used in the present invention.

[45] Discrete data Indexing In one embodiment, discrete data indexing scheme may be used when the number of pixels is much bigger than the number of items for indexing. Discrete data indexing is characterised by a relatively small number of different values. If the image is a map of a set of roads and the properties between them, then there would be a set of data (owner, address, valuation, contact details etc) associated with each property. So all pixels that are over a particular property, for example 32 High St, will all point to the same set of ancillary attribute data. As there are a large number of pixels pointing to a relatively small number of attribute sets, the best way to connect the pixels to the attributes is to use the ancillary pixel data as an index into the ancillary attribute data.

[46] As shown in Figure 8, the value held in the ancillary pixel data is interpreted as an index into a table of Ancillary Attribute Data. So the ancillary pixel value 11 would link to row 11 in the ancillary attribute table, while a value 14 would link to row 14 etc.

[47] In another example, an image of property parcels and roads contains information on the properties (such as owner, valuation etc). Firstly, there is no useful information to be stored for all the pixels that do not cover properties (roads, lakes etc). Also there may be many (100's) of pixels all representing a single property. It would be extremely inefficient to store all the ownership information against each of these pixels. It is more efficient to store the ownership once, elsewhere, and store only a reference to the ownership information in the index itself. For example, if a cadastral image showed 37 properties, then we would start by creating an ancillary storage area of the 37 ownership records, one after the other. The storage areas would start with the data for the first owner (0). At the end of that information, which could be of any length, then the information for owner 1 would start. When the data for owner 0 took 187 bytes (bytes 0 to 186), then the information for owner 1 would start from byte 187 - and so on to owner 36. Each set of owner data would start with a length value - so the first two bytes of the data for owner 0 would be 187 - meaning the total length is 187 bytes.

[48] In some cases, there is not sufficient pixel depth available in an image to re-purpose enough pixels to provide indexing or ancillary data storage. In these cases, the index can be stored directly with the ancillary data leaving the image pixels untouched. Reference is now made to Figure 9, a two-byte integer is used for storing the offset in the index. So all the pixels that code for owner 0 would have an index value of 0 - meaning the data for owner 0 starts at byte 0. The software will be able to read the first two bytes (187) and then extract 187 bytes (including the 2-byte length) as the owner 0 data. Those pixels that code for owner 2 would have the value 187. So the reading software would start reading the ownership data from byte 187. It would read the first two bytes to get the total length (say 356) and then read a total of 356 bytes to extract all the ownership data.

[49] This indexing method is more efficient than the direct method because there are only

37 ownership records once. Referring to Figure 9B, here is another level of indirection that can be introduced to make the indexing scheme even more efficient. The indirect scheme above would need a 2-byte offset value for each of the 65,536 index values for a 256 x 256 cadastral image, that is 131,072 bytes.

[50] When there are 37 properties to index, then it is possible use a smaller index entry of

6 bits. This can hold 64 different values (a 5-bit value can only hold 32 values - not enough). So now we would need only 65,536 * 6/8 bytes or 49,152 bytes - a saving of 62%. What we do need, however, is a short additional index of 37 x 2 or 74 bytes that provides the offsets for each owner record. For example, the index values for pixels representing owner 1 would each have a value of 1. This would index the second 2-byte value of the secondary index that would contain the value 187 - the offset into the ownership data. The total size saving is 81,846 bytes.

[51] Continuous data indexing Continuous data is characterised by nearly every value being different. Normally the values are numeric. For example, when the elevation above sea level for pixels in an aerial image is to be stored in the image, there will be many different values, with only occasionally two values being identical. Where there might be as many different values as there are pixels, it is inefficient to use the discrete information indexing for indexing the ancillary information.

[52] In this situation, the ancillary data is used in its own right. The system or module of the present invention will interpret the ancillary data as any data type that uses 32 bits. These include

[53] • a 32-bit unsigned integer.

[54] • a 32-bit signed integer.

[55] • a 32-bit floating point number.

[56] • a 32-bit binary number (i.e. 32 flags).

[57] • a 4-byte string.

[58] Reference is now made to Figure 11. When the ancillary data is elevation information as integer metres, for example, it is possible to have pixel data, such as the values which can be read out directly from the pixel data without any need for data storage in the ancillary attribute data area.

[59] Direct data indexing Another way of indexing is known as direct data indexing, as shown in Figure 12. In this indexing method, ancillary data can be indexed into the image with direct reference to a pixel's row and column. In direct indexing, the index entry actually contains data. This method is only efficient when each and every pixel needs to be associated with a different attribute dataset. The ancillary attribute data would carry its own row/column indexing and would be able to access a particular set of data directly given a particular pixel context (row and column).

[60] In one embodiment, wherein there are many different attributes, there is no advantage in indexing in order to optimise storage space. It is preferable to simply use the position of data in the Text Chunk to act as the index to data. This will require each attribute value to be of the same length. For example, an 8 byte floating point value is used to represent each of the ground elevation at each pixel in an image. The Text Chunk would be used to keep these data as an array of 8-bytes block. For a 256 pixel by 256 pixel image, the storage would be 256 rows of 256 pixels x 8_bytes = 524288 or 524Kb. To find the value for the pixel at row 120 column 88, the software need to access the (119 x 256 x 8) + (87 x 8) _ 243,712 + 696 = 244,408th byte of 8 byte block.

[61] This indexing method can provide a faster access time than the other methods.

Obviously, the data could be both larger and compound (i.e. made up of multiple fields. For example, three values of elevation, slope

[62] (angle of the ground to the horizontal) and aspect (angle of the slope of the ground to

North) for each pixel location, could each be stored as an 8-byte block floating point number. The data is now 24 bytes block but can be laid out in similar way. The data size now becomes 1,572,864 bytes or l.SMb and to read the data for the previous example (row 120 column 88), the system need to read the 24 bytes from byte offset 733,224.

[63] In another example, when the ancillary data is the elevation of the ground at a particular image coordinate of an aerial image, then it is best to store the elevation as the contents of the index entry as the chances of multiple pixels having exactly the same elevation value are small. The elevation might be represented in a single precision floating point representation using 4 bytes. If the index uses the image pixels, then 4 bytes (32 bits) of each image pixel would be repurposed as the elevation at that point. If the index was stored as part of the ancillary information, then the elevation value would be stored at a calculated offset from the beginning of the index.

[64] The index consists of a long sequence of entries starting with the entry for row 0 column 0, then the value for row 0 column 1 and so on until the end of the row (row 0 column n where n is the right-most column of the image. The very next index entry would be the data for row 1 column 0, then row 1 column 1 and so on to the end of row 1. Then data for row 2 follows, then row 3 and so on until we have the last (bottom) row. The very last index entry is for the last column on the last row.

[65] Once the data has been laid out like this, the offset within the sequence needed to extract a particular entry can be easily calculated. Because the whole sequence will be in memory, access via the offset is immediate. Note also that the data at a particular index entry does not have to represent a single value. If we wanted to store elevation, slope and aspect, each as a 4-bytes float, then the index data entry would be (4 + 4 + 4) or 12 bytes long.

[66] Referring to Figures 13 and 14, a 256 pixel rows by 256 pixel columns image is used to illustrate this example. Assume also that the data stored in the index entry consists of a 4-byte single -precision floating-point value for each of elevation, slope and aspect for a total of 12 bytes.

[67] In order to locate the ancillary data for the pixel at image coordinates 103,129 (row

103, column 129), the offset of that ancillary data is required. The pixel is on the 104th row (index starts with 0) the offset will of 103 full rows or 103 x 256 * 12 = 316,416 bytes. To find the offset of the 129th column with 12 bytes each, that will be 129 x 12 = 1,548 bytes. Added to the first total, the offset will be 317,964. So the data required is in bytes 317,965 to 317,976 of the index data.

[68] Another example that could use a direct encoding would be where the image might be of a person's head. Each attribute could be an image of the person from a particular viewpoint. As the cursor moves over the image, the changing rows and columns are fed to the rendering application as representing the image.

[69] Polygon indexing In another embodiment, the invention supports indexes implemented as a set of 2D spatial geometries in the image coordinate system (i.e. the coordinates are the row/column coordinates of the image pixels). When indexes are stored in a Text Chunk, it is possible to use other formats for the indexing process. The indexes can be stored as a series of shapes.

[70] As shown in Figure 15, the indexes are defined as a polygon area (shape=poly coords='127, 57, 202, 60, 191, 131, 160, 127'). The coordinates are simply pairs of pixel row-column - So '127,57,202,60' is a line from pixel column 127, row 57 to pixel column 202 row 60.

[71] The use of geometric shapes defined in the pixel coordinate system can, in some cases, be more efficient than direct pixel indexes. For example, if each pixel index needed 2 bytes (16 bits), and a particular rectangular area on the image was, say, 100 pixels by 75 pixels, we would need 100 x 75 x 2 or 15,000 bytes of index storage. If we used the geometry shape index, we would simply need a text string similar to:

[72] 100,100,175,100,175, 200,100,200,100,100 [73] which requires only 39 bytes. If, however, the image has many small, intricate shapes, the direct pixel index will become smaller than trying to represent complex shapes using geometries.

[74] The shapes can be polygons (irregular multi-side area figures) or simpler geometries such as circles (where only the centre point row and column and a radius in pixels need be stored) or rectangles (where only the bottom- left and top-right image coordinates need be stored.

[75] The same principle of storing variable-length entries, which define an area containing all pixels associated with a particular attribute set can also be applied in this situation. If the index consists of a relatively small number of shapes, then the polygon definition of an index may be smaller than the index method mentioned above. On the other hand, when the number of attributes becomes large, polygon indexing will become less efficient.

[76] Entire data indexing Another way of using the digital image for the storage, indexing access and presentation of ancillary data is to treat the entire image as an index and ancillary data store. A separate image of the same width and height (and of any format) would be the image displayed by the application and would also provide feedback as to the image context of the various pixels in the image. As there is no need to represent or display an image, the entire pixel storage area can be given over to indexes (for indirect data) and/or data itself (for continuous data). By maintaining a consistent relationship between the image and its attribute image file, it should be relatively easy to coordinate systematic image sets (for example, those in a tile-based mapping system such as those of popular web mapping products with the ancillary data and indexing images supported by the invention).

[77] Indexing Preferences An index for ancillary data is simply a number that indicates an offset pointing to a list of attribute sets. The size of storage required for the index depends on how many attribute entries are in the image. When there are a lot of entries, a larger number of bits is required to store an index value. When there are only a few entries, a smaller index entry size is required.

[78] Reference now is made to Figures 16 and 17. This tile has 275 Cadastral properties.

[79] In order to uniquely index each of the properties, a binary number is required to hold more than a value as big as 275. Since an 8-bit block can only hold a value from 0 to 255, it is not good enough to represent all 275 different properties. In this situation a 9-bit value which can hold 512 different values is required. Assuming the tile is 256 pixels by 256 pixels image, then the index will consist of 256 x 256 pixels, each pixels has a 9_bit (1.125 bytes) value. Total size need for indexing the image will be 256 x 256 x 1.125 bytes or 73,728 bytes (Around 72Kb).

[80] Reference is now made to Figures 18 and 19, there are only 15 different properties. Therefore only 4-bits long block is required for indexing all properties (4 bits can hold 16 different values).

[81] Assuming the image is 256 pixels wide and 256 pixels high, the index will need to be

256 x 256 x 0.5 bytes or 32,768 bytes- (32Kb) - less than half the storage required in the example above. By varying the size of the block for storing the index value, it is then possible to optimise the storage requirements based on the complexity of the indexing required.

[82] Composite Interpretation Using the 64-bit RGBA PNG format, there provides ancillary pixel data of 32 bits that can index 232 or 4,294,967,296 (about 4 Billion) values. This is far more than it would need for most purposes (and normally far more than the number of pixels in an image - it would have to be an image of 65,536 * 65,536 pixels in size). Consequently, it is possible to subdivide the 32 bits into a series of fields, each of which either indexes a different attribute in the discrete data indexing, or a different data value - in the Composite case.

[83] Reference is now made to Figures 20 and 21, the 32 bits can be sub-divided into different field widths. For example, the Pool attribute above indicating if a property has a pool or not has only two values (yes and no)

[84] and so can be represented using a single bit. If, for example, there were less than

1,000 suburbs in a country, then allocating 10 bits would be enough to encode all suburbs. The valuation is in 4 bands and so can be encoded with 2 bits leaving 19 bits (524,288 values) available for addresses.

[85] Offset and Scaling In another embodiment of the invention, to optimise precision of values across the image or to reduce the storage requirements, we can also define a minimum or base value and a scale factor to be applied to the ancillary data as it is stored and then recovered when the ancillary data is read. For example, elevation across a country might vary from 0 (sea level) to the highest mountain at several thousand metres. However, a single aerial terrain image will probably not have the full range of elevation values. For example, if the image is of a mountainous area and the minimum elevation in the image is 786.25m then we can establish that value as a Base and subtract it from each elevation value before storing them. If the resolution of the elevation model was to lcm (i.e. metres to 2 decimal places), then we can use integers and an implied decimal place of two. If the maximum elevation in the image is 1012.56 metres then the range of values is

[86] • Min: 786.25 * 100 = 78,625 (as an integer).

[87] • Max: 1012.56 * 100 = 101,256 (as an integer)

[88] • Range is 101,256 - 78,625 = 22,631.

[89] This ancillary data value can be represented using less 15 bits (which can store

32,768 values) which is more efficient than using a full 4-byte real number for each elevation value. In a similar way, the data values could be scaled up or down before storage and reversed during the reading process.

[90] System Interface The invention includes a software system to create the enhanced images and an invention interface to interpret the enhanced images (image plus ancillary data and indexing) and support requests from external systems returning ancillary data appropriate to a particular image coordinate.

[91] System and Module In hte embodiment of the invention in which pixel storage is appropriated for the storage of ancillary data, ancillary index, or both, a special system or module is required to intervene between the normal image rendering code of the browser and the actual image sitting in memory. The system or module is required to recognise, parse and separate the ancillary information from the pixel information and then reconstruct the image pixels so that they repesent only pixel information and can be processed further, such as by passing them to normal image software (which interprets all the pixel bits as representing colours) to display the image properly.

[92] One embodiment has a system or module to take the normal 24/32 bits-per-pixel image and expand it to 48/64 bits with the extra bits being for data or indexes. When the software intervenes, it will filter the extra bits and render a proper image for browser image software to display as a 24/32bit image. So the image is rendered as a 24/32bit image (the usual format for images) while the software for the present image format now has the balance of the pixel bits available as data or as pointers.

[93] In one embodiment, there is no special tag required in the image to differentiate normal image and the image of the present invention. The software or module of the present invention will start by searching for the Text Chunk. If the Text Chunk is found, the module will inspect at the first 6 data / content bytes of the first Text Chunk. If there is a predetermined keyword, such as 'ENHIMG' found in the data, then the system / module can confirm that the image is one that needs special processing. In another embodiment, the software or module wouold search for Text Chunks of a specific type that denote that the image has been enhanced and the chunk contains anciallry information.

[94] The system or module then starts interpreting the rest of the Text Chunk which would be a header chunk, signalling the system or module all about this image format and where to find other data (in other Text Chunks).

[95] Image Writer

[96] One embodiment of the present invention includes a software system that creates and/or enhances the image by combining the image and ancillary data and creating the appropriate indexing scheme to connect the two together.

[97] Reference is now made to Figure 22. This describes one process of creating an image in accordance with the present invention. The process is implemented by a software system which is part of the invention and that is [98] designed to create enhanced images. It can be a stand-alone application or add-on to another application. This example shows how an image tile used in web mapping can be enhanced using the image writer. [99] One process of creating an image in accordance with the present invention is described. This process can be in a form of system or module. It can be a stand-alone application or an add-on to another application.

• Start by loading the original image in to the system memory for processing;

• Load the attribute data that applies to this image. In this example, because the original image represents a known location, the system can select all the attribute data in that area. In other words, the system can select all the information on the properties covered or partially covered by the image tile. This data may include one or more polygons which represents the extent of each parcel.

• Determine how many bits the index will have to have. In this example, the image covers 24 properties to index (plus the zero value meaning no-data) so we will need 5 bits per index entry (2^Λ5 = 32).

• Create an attribute metadata structure. This provides information on each field of the attribute record including:

• Size of field

• Type of data (integer, float, string, date, boolean etc).

• Label for the data

• Create a memory space of image width x image height x 5 bits. For example, if the image width and image height are both 256, we will need a memory space of 40,960 bytes to be allocated. The image width and image height of the memory space are filled with O's (no-data).

• Take the first property and identify which image pixels it is over. For each of these pixels, put the value T into the corresponding 5 bit index entries of the index structure.

• Place the attributes for the first property into the first of the ancillary storage areas.

• Continue with the second property. Wherever its polygon covers the image pixels, set the corresponding index values to a predetermined value. In this example, the next index value is '2'.

• Add the attributes for the second property into a second ancillary storage area.

• Continue this process until all 24 properties have been indexed. • Construct a header, consisting of:

• a predetermined identifier, sometimes known as 'magic number', such as the keyword 'ENGIMG';

• information on the type of indexing in this case, the index value is an offset into a set of fixed length ancillary data elements;

• the location of the index and the number of bits per index. In this case 5 -bits;

• the location of the attribute metadata Text Chunk; or

• the location of the first attribute Text Chunk.

• add the header, ancillary metadata, ancillary data and index data into the existing image structure. The existing image header, pixel data etc will remain the same but index offsets and pointers may need to be changed to accommodate the new content to ensure the image is 5 properly structured.

• write out the new image data.

• optionally, close the image file.

[100] The image is now an enhanced image and can be detected as such and read by an image reader.

[101] Image Reader A part of the invention is an invention interface that can read the enhanced image. The reader extracts the pixel-based indexing data, if any, from the image pixels, then repairs the pixels so they properly represent the original image pixel data. The image data for display is then provided to the normal image display software of the application environment the image is part of. In the case where ancillary information and indexes have been stored in custom image extensions (such as the "chunks" of hte PNG specification), then the reader can extract the ancillary data and indexes directly without needing to access or repair the pixel data.

[102] Combined Image Reader / Writer The writer can also form part of the Image Reader software such that the ancillary data for an image can be read and displayed, altered, enhanced, added to or updated and then written back to the image. The invention interface would provide an additional set of functions that would allow ancillary data to be changed and the new, modified image written out.

[103] Dynamic Highlighting In another embodiment of the invention, the invention interface provides a function essentially the reverse of those previously described. In this case, the external software environment provides the invention interface a portion of the ancillary data, optionally as a pattern that potentially matches multiple ancillary data instances. The invention interface searches the ancillary data, identifying those ancillary data items matching the supplied data or data pattern and alters the pixels indexed to those data items. The effect is that the external software environment can cause specific parts of the image to be altered (e.g. 'highlighted') to indicate which parts of the image pertain to the ancillary data or data patterns. The alteration to the pixels could be by hue (blue turns to yellow, for example) or transparent. By changing the alteration over a regular time cycle, a flashing effect can also be produced.

[104] Dynamic Labelling In another embodiment of the invention, the system interface supports a function that would temporarily change some of the pixels in the image data such that a small display area like a tool tip could be added to the display close to or with an arrow pointing to the pixel coordinates associated with the anciallary information. The display area could contain any of the ancillary data. The pixel images under the display area would be saved and restored when the location of the display area moves on. In another embodiment of the invention, the reader could display the labelling over the image pixels which would be unaltered.

[105] Enhanced Image detection The enhanced image, in the case where the image pixels are completely dedicated to image display, can be displayed by the software systems that read and display normal images. For example, a browser could read in an enhanced PNG image and displayed just as if it is not enhanced. It cannot, of course, access the extra content contained in the comment areas. The Image Reader that forms part of this invention needs to detect the fact that a particular image has been enhanced and contains both indexing and ancillary information. It does this by identifying Comment areas and searching for a set series of characters (known as a 'Magic Number') such as 'ENHIMG' at the beginning of a comment area. When that sequence is found, the remainder of the comment area is identified as a header for an enhanced image. The header contains Metadata - that is information about the structure and location of all the other information - the indexes and the ancillary data. It includes information describing:

• The type of indexing being used.

• The location of the index, index element size, etc.

• Nature of the ancillary data, number of fields, data types, storage sizes etc.

• Display metadata - this is information on how to display a particular ancillary data field. It might include labels (in various languages), interpretation (does the field data represent a date? - if so, what format, time zone etc.)

[106] In the case where the ancillary information is stored in the image extensions (such as custom "chunks"), then a unique chunk type could be used to allow the image reader to detect if the image is enhanced. The image reader could look through each chunk of the image, looking for a particular unique chunk type. If it detected that chunk type then it recognises that the image has been enhanced. It can then parse and interprete the extension information from the custom chunk or chunks.

[107] Implementations [108] The invention includes an invention interface and software systems to write and read enhanced images. Some (but not all) of the suggested ways the software could be implemented are as:

• A stand-alone application that can Write, Read or Read and Write without external software requirements

• As a plug-in to existing software systems such as web browsers. In this implementation, the software would read the image to extract the enhanced content and supply the un-enhanced image-equivalent to the normal software system for display.

• As an add-on or extension to an existing software application (for example a web browser).

• A library that can be linked into other software systems and accessed through a series of functions.

• As a hardware device with the functionality implemented in the circuits of the device or in a set of low-level bios-like software functions. An example would be a digital photo-frame showing pictures and able to respond to the screen being touched with information on that particular area of the image.

[109] Ancillary Data Printing The ancillary data can also be provided to software preparing the image to be printed. For example, the software could allow selected points in an image to be labelled with the elevation. The ancillary data could also be used to construct labels with arrows pointing to particular parts of the image.

[110] Scripting As the ancillary data can include data of any type that can be serialised and stored in a binary format, the invention also supports the inclusion of procedural instructions (scripts or programs) in the ancillary data. The scripts would be read and interpreted by the image reading software of the invention interface and used to provide customised functionality. Alternatively, the procedural instructions could be passed to an external target which would undertake the interpretation of the instructions. In one embodiment of the invention, the scripts could be provided in the JavaScript (or EC- MAScript) language. In the case where the Image Reader has been implemented as an add-on or extension to an existing web-capable application such as a browser, then the Image Reader environment could utilise the fact that the application can read and in- terprete JavaScript source. The ancillary information could be extracted from the image and interpreted as JavaScript by the Image Reader.

[I l l] Security and encryption The invention includes the option of encrypting the ancillary information using a keysuch that it can only be read by supplying a corresponding decryption key. Existing encryption schemes such as PGP would be used to encrypt the data, including both indexes, and ancillary data. Once a valid key is supplied, the index and ancillary data are decrypted and then used as in the non-encrypted case. [112] Some of the embodiments described here refer to the PNG image format. However, the invention is not constrained to using only PNG images. Other image file formats with the ability to store user-defined content, such as the TIFF and JPEG image formats can also be used for constructing the enhanced images created by the present invention.

[113] It will be apparent that obvious variations or modifications may be made which are in accordance with the spirit of the invention and which are intended to be part of the invention, and any such obvious variations or modifications are therefore within the scope of the invention.

[114] The invention includes a design for software to make the necessary changes to an existing image to incorporate the new information. It also includes the design for software to extract the information from the image when requested and to present the information to the application accessing the image.

[115] The invention includes a design for the reader to be implemented as an extension, add-on or plug-in to browser, thereby allowing an existing web browser to become a digital image reader for the present invention and thus able to read, decode and display digital images described in this invention.

[116] With the image reader for the present invention, a particular image pixel forms part of the depiction of a particular property parcel in a map tile, the invention would enable contextual information on that exact property (owner, address, area etc) to be retrieved and displayed. If another pixel becomes the new context (for example, if a user moves a mouse cursor over the image to a new property area), then information relevant to the property over that pixel location would be provided.

[117] The invention is also designed to be implemented by making a small change, or developing an add-on or extension to current browsers allowing the extra information contained in the image to be made available to the browser environment via an extension to its API. The images created by the writer for the present invention do not need to be used in a web browser and could be incorporated into any application software that uses images.

[118]

[119] EXAMPLES

[120] Patient Passport

[121] An example might be an image that incorporates a multitude of medical information in a multitude of formats that all pertain to a particular patient. Examples of the type of medical data that might be incorporated into the image are doctor's textual notes, x- rays, functional MRI (fMRI) scans, the tabular results of blood tests, a history of prescriptions written and fulfilled for that patient and many other types of medical information. Different types of information could be indexed and retrieved using the the indexing scheme best suited to the particular infomration type. The "patient passport" would then be analogous to a normal "travel" passport in that it would contain a comprehensive set of medical information in a variety of formats on behalf of a single patient. Any sensitive or personal data could be encrypted before being stored and indexed into the image and could only be decrypted using a supplied password. So the information would be safe until "unlocked" by the patient or an authorized representative for use by authorised medical personnel. While the "Patient Passport" could contain a comprehensive set of medical information and history, it would still continue to be a single image file and could be stored conveniently, for example, in one embodiment, in the memory of a USB key or similar memory device that could be carried by the patient.

[122] High School Year photograph

[123] For example, if an image depicts a high-school class of 30 people, then the ancillary information might be those 30 peoples' names and contact information. Then the parts of the image (the pixels) that form the face of, say, John Doe, would correlate with stored information on John Doe's name and contact details, while those pixels that represent Jane Smith would -be linked to Jane Smith's name and contact details.

[124] Terrain Profiling

[125] This example assumes there is an aerial image of some mountainous terrain. The external software makes a call with two image coordinates representing each end of a straight line across the image. The invention interface determines all the image pixels lying along the line and extracts the elevation information for each of them. These elevation values along the line are then used to create a Profile or Cross-Section of the image - the equivalent of cutting through the terrain along the line and viewing the exposed surface. The same principle can be used to profile any ancillary data.

[126] Aerial image profiling

[127] In another example, the image might depict an aerial photo of a mountainous area. The image, once enhanced using the invention, would be able to return, say, the elevation (height), slope (steepness) and aspect (orientation to the north) of each pixel in the image. The image user could move the mouse cursor over the image and receive a continuous update of elevation, slope and aspect. Alternatively, the user could draw a line across the image; the elevation of all the pixels along the line could be extracted and a diagram displayed profiling the elevation along the line.

[128] In yet another example, an image showing an aerial photograph or satellite image could, using the invention, be augmented with information on the elevation (height above sea level) of the terrain represented by each part of the image. As the user of an application moves the mouse cursor over the image, the elevation of the ground at the location represented by the pixel can be accessed and displayed. [129] X-ray with Bone Density Data

[130] In one embodiment, the image is an X-ray while the ancillary information is the actual estimated bone density at each point of the image. The dual-energy x-ray absorptiometry (DXA) or bone densitometry is an enhanced form of X-ray technology that is used to measure bone loss.

[131] Normally the bone density is displayed on a computer screen. Using the invention, the actual bone density measurements could be incorporated into the image so that the software displaying the X-ray image could also provide the bone density at every pixel location.

[132] Moving Portrait

[133] In yet another embodiment, the ancillary data can be an array of images relevant to the main image. If the main image contained a portrait (head and shoulders) of a person, then the series of images stored as ancillary data could contain other head and shoulders portrait images of the same person but taken from a range of slightly different viewpoints. As the user moves the cursor over the image and the underlying software requests ancillary data for different rows and columns, different images would be extracted from the ancillary data and used to replace the main image. The net effect would be that the portrait would appear to 'turn' as the user's mouse cursor passed over the image.

[134] Image Application

[135] In yet another embodiment, the ancillary data could comprise procedural programs or scripts that would be executed as part of the actions the image reader could perform on the image. The reader would be able to evaluate specific anciallry data as programs which would then cause further actions. The image would, in effect, become able to perform actions both spontaneously (immediately upon being read) and based on specific events occurring. It would thus be able to "interact" with its environment by running a specific procedure or script when encountering a specific event. Advantageous Effects

[136]

Description of Drawings

[137] The invention is now discussed with reference to drawings, where:

[138] Figure 1 is a diagram showing different PNG file formats;

[139] Figure 2 is a conceptual diagram showing a 32 bits PNG file format;

[140] Figure 3 is a conceptual diagram showing a 64 bits PNG file format;

[141] Figure 4 is a conceptual diagram showing a PNG file format of Figure 3 with a 32 bits pixel data;

[142] Figure 5 is a conceptual diagram showing a PNG file format of Figure 4 with a 32 bits ancillary data;

[143] Figure 6 is a conceptual diagram showing the extraction of pixel data from a colour component of the PNG file of Figure 3;

[144] Figure 7 is a conceptual diagram showing the extraction of ancillary data from a PNG file of Figure 3;

[145] Figure 8 is a conceptual diagram showing one indexing scheme in 10 accordance with one embodiment of the invention;

[146] Figure 9 is a conceptual diagram showing an example of the indexing scheme as shown in Figure 8;

[147] Figure 10 is a conceptual diagram showing another example of the indexing scheme as shown in Figure 8;

[148] Figure 11 is a conceptual diagram showing another data indexing scheme in accordance with another embodiment of the invention;

[149] Figure 12 is a conceptual diagram showing a data structure for storing ancillary data using the indexing scheme as shown in Figure 11 ;

[150] Figure 13 is a conceptual diagram showing another data indexing scheme in accordance with yet another embodiment of the invention;

[151] Figure 14 is a conceptual diagram showing an example of the indexing scheme as shown in Figure 13;

[152] Figure 15 is a conceptual diagram showing yet another indexing scheme in accordance with yet another embodiment of the invention;

[153] Figure 16 is a conceptual diagram showing one example of a digital image in accordance with an embodiment of the invention;

[154] Figure 17 is a conceptual diagram showing another example of a digital image in accordance with an embodiment of the invention;

[155] Figure 18 is a conceptual diagram showing one example of a digital image in accordance with an embodiment of the invention;

[156] Figure 19 is a conceptual diagram showing another example of a digital image in accordance with an embodiment of the invention;

[157] Figure 20 is a conceptual diagram showing an indexing method with composite ancillary data in accordance with yet another embodiment of the invention;

[158] Figure 21 is a conceptual diagram showing another indexing method with composite ancillary data in accordance with yet another embodiment of the invention;

[159] Figure 22 is a flow chart showing a process for creating a digital image file in according with an embodiment of the present invention. Best Mode

[160] Mode for Invention [161]

Industrial Applicability

[162]

Sequence List Text

[163]

Claims

[Claim 1] L A method for creating a digital image file including a plurality of ancillary data elements, wherein the digital image file comprises a plurality of pixel data elements, the method comprising the steps of: recording ancillary data elements into the digital image file according to an index scheme, such that each pixel data element is coupled to a corresponding ancillary data element through the index scheme.

2. A method of claim 1, wherein the index scheme comprises the steps of: indexing the ancillary data element with a set of index numbers; associating one index number to at least one pixel data element.

3. A method of claim 1, wherein the index scheme comprises the steps of recording each pixel data element from the digital image file on a second digital image file; recording each ancillary data element that is associated with a corresponding pixel data element at a location next to the corresponding pixel data element on the second digital file.

4. A method of claim 1, wherein the index scheme comprises the steps of: sorting each ancillary data element that is associated with a corresponding pixel data element according to a relative location of the corresponding pixel data element; recording the sorted ancillary data elements in the digital image file.

5. A method of claim 1, wherein the indexing scheme comprises the steps of: defining a plurality of points comprising relative locations of the pixel data elements on the image, wherein the plurality of points define a polygon area; associating at least one ancillary data element to the polygon area.

6. A system for creating a digital image file including a plurality of ancillary data elements, wherein the digital image file comprises a plurality of pixel data elements, the method comprising the steps of: recording ancillary data elements into the digital image file according to an index scheme, such that each pixel data element is coupled to a corresponding ancillary data element through the index scheme.

7. A system of claim 6, wherein the index scheme comprises: indexing means for indexing the ancillary data element with a set of index numbers; associating means for associating one index number to at least one pixel data element.

8. A system of claim 6, wherein the index scheme comprises: recording means for recording each pixel data element from the digital image file on a second digital image file; recording means for recording each ancillary data element that is associated with a corresponding pixel data element at a location next to the corresponding pixel data element on the second digital file.

9. A system of claim 6, wherein the index scheme comprises the steps of: sorting means for sorting each ancillary data element that is associated with a corresponding pixel data element according to a relative location of the corresponding pixel data element; recording means for recording the sorted ancillary data elements in the digital image file.

10. A system of claim 6, wherein the indexing scheme comprises the steps of: defining means for defining a plurality of points comprising relative locations of the pixel data elements on the image, wherein the plurality of points define a polygon area; associating means for associating at least one ancillary data element to the polygon area.