GB2403305A - Image and date information processing - Google Patents

Abstract

An image processing apparatus 2 processes image data defining an image or a sequence of images to read the recording date of the image(s), to access an electronic diary in accordance with the recording date to determine the photographer's activities at the time the image/image sequence was recorded, and to generate annotation data describing the image subject matter based upon the determined activities of the photographer. Each input image or sequence of images processed by the apparatus is stored in an image database together with the generated annotation data. The database can then be searched in accordance with text keywords input by a user to retrieve images having those keywords in the associated annotation data.

Description

IMAGE PROCESSING

The present invention relates to the field of image processing and, more particularly, to the processing of image data by an image processing apparatus to generate annotation data comprising text data describing the image. The image data and text data can then be stored in a database linked together to facilitate text-based searching and retrieval of images therefrom by a user.

Many apparatus which generate annotation data for archiving images in a database are known. For example, GB-B-2,349,761 and GB-A-2,349,764 describe apparatus for generating annotation data for still images and video images, comprising data defining, for each person or animal in the image, the person, animal or object at which they are looking.

Similarly, systems are known which perform face recognition processing on an image to identify the people appearing in the image, and to generate annotation data defining the names of the identified people for storage in a database in association with the image data.

It is an object of the present invention to facilitate the automatic generation of reliable and accurate annotation data for images. It is a further object of the invention to facilitate, through electronic processing, an image annotation system which is quick and easy for a person to use.

According to the present invention, there is provided an apparatus for generating annotation data for an image for storage in a database with the image data, comprising an electronic diary accessible by a user to record personal activity data therein arranged by date, means for reading personal activity data stored in the electronic diary for the recording date of an image to be archived in the database, and means for generating annotation data defining the subject matter of the image using the personal activity data.

The present invention also provides a method of generating annotation data for an image for storage in a database with the image data, in which personal activity data stored in an electronic diary for the recording date of the image is read and used to generate annotation data defining the subject matter of the image.

In this way, provided the content of the electronic diary is accurate and complete, reliable and accurate image annotation data describing where the user may remember the image being recorded and/or the subject matter of the image can be generated quickly without user input to facilitate subsequent retrieval of images from a database.

The present invention also provides a computer program product, embodied for example as a storage medium carrying instructions or as a signal carrying instructions, having instructions for causing a programmable processing apparatus to become configured as an apparatus as set out above or to perform a method as set out above.

Embodiments of the invention will now be described, by way of example only, with reference to the accompanying drawings, in which: Figure 1 schematically shows the components of an embodiment of the invention, together with the notional functional processing units and data stores into which the processing apparatus component may be thought of as being configured when programmed by programming instructions; and Figure 2, comprising Figures 2a and 2b, shows the processing operations performed by the apparatus of Figure 1 in an embodiment.

Referring to Figure 1, an embodiment of the invention comprises a programmable processing apparatus 2, such as a personal computer, containing, in a conventional manner, one or more processors, memories, graphics cards etc., together with a display device 4 and user input devices 6, such as a keyboard, mouse etc. The processing apparatus 2 is programmed to operate in accordance with programming instructions input, for example, as data stored on a data storage medium 12 (such as an optical CD ROM, semiconductor ROM, or magnetic recording medium, etc.), and/or as a signal 14 (for example an electrical or optical signal input to the processing apparatus 2, for example from a remote database, by transmission over a communication network such as the Internet or by transmission through the atmosphere)' and/or entered by a user via a user input device 6 such as a keyboard.

As will be described in detail below, the programming instructions comprise instructions to program the processing apparatus 2 to become configured to process image data defining an image or a sequence of images to read the recording date of the image(s), to access an electronic diary in accordance with the recording date to determine the photographer's activities on the date the image/image sequence was recorded, and to generate annotation data describing the image subject matter based upon the determined activities of the photographer. For example, if the electronic diary indicates that the photographer was on a family holiday in Scotland on the date at which an image was recorded, then the text keywords "family holiday Scotland" would be generated to describe the subject matter of the image. Each input image or sequence of images processed by the apparatus is stored in an image database together with the generated annotation data. The database can then be searched in accordance with text keywords input by a user to retrieve images having those keywords in the associated annotation data.

When programmed by the programming instructions, the processing apparatus 2 can be thought of as being configured as a number of functional units for performing processing operations and a number of data stores configured to store data. Examples of such functional units and data stores together with their interconnections are shown in Figure 1. The functional units, data stores and interconnections illustrated in Figure 1 are, however, notional, and are shown for illustration purposes only to assist understanding; as will be appreciated by the skilled person, they do not necessarily represent the units, data stores and connections into which the processors, memories, etc. of the processing apparatus 2 actually become configured.

Referring to the functional units shown in Figure 1, central controller 20 is arranged to process inputs from the user input devices 6, and also to provide control and processing for the other functional units. Working memory 30 is provided for use by central controller 20 and the other functional units.

A plurality of electronic diaries 40-1 to 40-n are provided, each for a respective different user. Each electronic diary 40 comprises, in a conventional way, an interface 42 to enable a user to access dates stored in a memory 44 and to enter and view text data describing personal activities undertaken or to be undertaken on those dates. In addition, each electronic diary 40 stores data defining the name of the user to which it belongs.

Input data interface 50 is arranged to receive, and write to memory, image data defining images to be stored in an image database and for which annotation data is to be generated for storage in association therewith in the database. The input image data may be input to processing apparatus 2 as data stored on a storage medium 42, or as data carried by a signal 44.

Image data store 60 is configured to store the input image data for subsequent processing.

Diary selector 70 is operable to select one of the electronic diaries 401 to 40-n for use in generating annotation date, with the selection being carried out in this embodiment in accordance with a photographer's name entered by a user.

Image date reader 80 is operable to read the image data of an image stored in image data store 60, to identify the pixels on which is overlaid the date of recording of the image, and to read the recording date.

Annotation data generator 90 is operable to access the electronic diary selected by diary selector 70 in accordance with the image date read by image date reader 80, to read text data stored in the diary for the recording date of the image, and to generate text annotation data for the image using the data read from the electronic diary 40.

Image database 110 is configured to store image data from image data store 60 linked to the corresponding annotation data generated by annotation data generator 90, thereby allowing text-based search and retrieval of images from the database.

Database search engine 120 is operable to search the annotation data in the image database 110 in accordance with one or more keywords entered by a user using a user input device 6 such as a keyboard, so as to identify each image in the image database 110 which is linked to annotation data containing the keyword(s). Database search engine 120 is further operable to enable a user to select one or more of the identified images from image database 110 and to display the selected image(s) on display device 4.

Display controller 130, under the control of central controller 20, is operable to control display device 4 to display image data received as input image data, and to display image data retrieved from image database 110.

Output data interface 140 is operable to output data from processing apparatus 2 for example as data on a storage medium 142 (such as an optical CD ROM, semiconductor ROM or magnetic recording medium, etc.) and/or as a signal 144 (for example an electrical or optical signal transmitted over a communication network such as the Internet or through the atmosphere). In this embodiment, the output data comprises data defining an image from image data store 60 and the associated annotation data generated by annotation data generator 90. The output data may, for example, be stored in an external database.

A recording of the output data may be made by recording the output signal 144 either directly or indirectly (for example by making a recording and then making a subsequent copy recording) using recording apparatus (not shown).

Figure 2 shows the processing operations performed by processing apparatus 2 in this embodiment.

Referring to Figure 2, at step S2-2, input data interface stores image date in image data store 60 for each image to be stored in image database 110 and for which annotation data is to be generated. In this embodiment, each input image contains pixel data on which has been overlaid the recording date of the input image (this information being provided in a conventional way by recording each image with a camera having a so-called "databack" which overlays the recording date on the image). Each image stored at step S2-2 may comprise a frame of video data, a "still" image, or a sequence of video images for which one respective set of annotation data is to be generated to be associated with all of the images in the sequence.

At step S2-4, central controller 20 controls display processor 130 to display a message on display device 4 requesting the user to enter the name of the photographer who recorded the image(s) stored at step S2-2.

In response to the entry of the photographer's name by the user, diary selector 70 selects at step S2-6 the electronic diary 40 of the photographer from among the plurality of electronic diaries 40-1 to 40-n.

At step S2-8, image date reader 80 reads the pixel data of the next image (or the first image of the next image sequence) to be stored in image database 110 (this being the first image or image sequence stored at step S2-2 the first time step S2-8 is performed) to determine the recording date of the image. This processing comprises identifying the pixels on which is overlaid the recording date, and reading the recording date from the identified pixels.

At step S2-10, annotation data generator 90 reads the text entries stored in the electronic diary 40 selected at step S2-6 for the date read at step S2-8.

At step S2-12, annotation data generator 90 generates text annotation data for the image (or image sequence) currently being processed using the text information read from the electronic diary at step S2-10. The annotation data comprises one or more text annotations, with each annotation containing one or more keywords.

By way of example, if the diary information read at step S2-10 comprises the text "Simon's wedding", then annotation data is generated at step S212 comprising the words "Simon wedding This annotation data may be generated as two separate words, namely "Simon" and "wedding" or may be generated as one phrase containing both words, namely "Simon wedding". By way of further example, if the diary information read at step S2-10 comprises "fishing trip with children to Scotland", then annotation data is generated at step S2-12 comprising the words "fishing children Scotland". If the diary information read at step S2-10 comprises "dinner party with Matthew", then annotation is generated at step S2-12 comprising the words "dinner party Matthew". Similarly, if the diary information read at step S2-10 merely comprises the word "holiday", then annotation data is generated at step S2-12 comprising the word ''holiday".

It should be noted that additional annotation data may be generated in dependence upon the diary information read at step S2-10. For example, information may be stored in each user's electronic diary 40 defining the names of the user's wife, children, pets, etc. In this case, when diary information is read at step S2-10 such as "fishing trip with children to Scotland", then the names of the user's children may be read and additional annotation data generated comprising those names to supplement the annotation data "fishing children Scotland".

At step S2-14, the image data for the image (or image sequence) currently being processed and the annotation data generated at step S2-12 are stored in the image database 110. The annotation data is stored in the database so that it is linked to the image data, thereby facilitating text-based search and retrieval of images by database search engine 120. More particularly, the annotation data may be searched by the database search engine 120 in accordance with one or more keywords input by a user, to identify annotation data which contains the keyword(s) and the associated image(s).

At step S2-16, a check is carried out to determine whether annotation data has been generated for all of the images and image sequences stored at step S2-2. Steps S2-8 to S2-16 are repeated until annotation data has been generated for each image/image sequence, and each image/image sequence has been stored in image database linked to the generated annotation data.

Many modifications and variations can be made to the embodiment described above within the scope of the claims.

For example, in the embodiment described above, each image stored at step S2-2 includes data identifying the recording date of the image, and image date reader 80 is arranged to read the image data to determine the recording date at step S2-8. However, instead, the user may enter data defining the recording date of one or more images at step S2-2, and image date reader 80 may be arranged to read the date entered by the user at step S2-8.

In the embodiment described above, the date at which an image was recorded is read at step S2-8 and diary information is read at step S2-10 in accordance with the date. In addition, the time at which an image was recorded may be read at step S2-8 as well as the date on which the image was recorded, and diary information may be read at step S2-lO in accordance with the date and time. As a result, annotation data may be generated at step S2-12 in accordance with the recording date and time of the image.

In the embodiment described above, diary selector 70 is arranged to select an electronic diary 40 at step S2-6 in accordance with the photographer's name entered by a user. However, instead, a respective camera may be assigned to each user of processing apparatus 2, step S2-4 may be omitted so that the user does not enter the name of the photographer, and diary selector 70 may be arranged to perform processing at step S2-6 to process the image data to identify the camera which recorded the image, and to select the electronic diary for the user associated with the identified camera.

More than one electronic diary 40 may be stored for each user in the embodiment described above, diary selector 70 may be arranged to select all of the electronic diaries 40 belonging to a photographer at step S2-6, and annotation data generator 90 may be arranged to read diary information at step S2-10 for all of the selected electronic diaries of the photographer. If each respective electronic diary 40 of a photographer is for a different year, then diary selector 70 may be arranged to perform additional processing after step S2-8 has been performed to read the image recording date and before step S2-10 is performed, to select the electronic diary 40 of the named photographer corresponding to the year in which the image was recorded, this diary then being used at steps S2-10 and S2-12 to generate the annotation data for the image.

In the embodiment described above, processing apparatus 2 may include, in addition to or as part of each electronic diary 40, an electronic calendar storing pre defined but non-user-defined national calendar information for each date of the year comprising, for example, national events (such as public holidays, religious festivals, New Year's eve, New Year's day, Halloween, Bonfire night, etc) occurring on that date and the season of the year (spring, summer, autumn, winter) on that date. Annotation data generator 90 may then be arranged to read the stored calendar information for the image recording date and to generate the annotation data in dependence thereon in addition to or instead of the information from the electronic diary 40.

In the embodiment described above, each electronic diary is part of processing apparatus 2. However, instead, one or more of the electronic diaries 40 may be stored in a separate apparatus (such as a personal digital assistant (PDA), etc) connected to processing apparatus 2 for access by the diary selector 70.

In the embodiment described above, image database 110 is part of processing apparatus 2. However, instead, image database 110 may be provided in a separate apparatus, and image data and annotation data may be output from processing apparatus 2 via output data interface 140 for storage in the external image database 110. Of course, the image database may comprise a recording medium, such as a writeable compact disk, and image data and annotation data may be stored thereon using a recording device, such as a CD writer.

In the embodiment described above, processing is performed by a computer using processing routines defined by software programming instructions. However, some or all, of the processing could, of course, be performed using hardware or firmware.

The present application incorporates the full contents of the applicant's UK patent application being filed concurrently herewith (entitled "Image Processing" and having attorney reference 2876901).

Claims (22)

1. An image processing apparatus, comprising: means for receiving image data defining an image; means for reading the recording date of the image; means for accessing an electronic diary accessible by a user to store user-defined information therein and for reading user-defined information therefrom in accordance with the recording date of the image; means for generating annotation data for the image in dependence upon the userdefined information read from the electronic diary; and means for writing the image data and the annotation data to a database configured to store the annotation data linked to the image data.

2. An apparatus according to claim 1, wherein the means for generating the annotation data is operable to generate annotation data comprising text data.

3. An apparatus according to claim 2, wherein the means for generating the annotation data is operable to select keywords from information read from the electronic diary and to use the keywords as annotation data.

4. An apparatus according to any preceding claim, wherein the means for reading the recording date of the image is operable to process the image data to read the recording date thereof.

5. An apparatus according to any preceding claim, wherein the means for reading the recording date of the image is operable to read a recording date entered by a user.

6. An apparatus according to any preceding claim, wherein: the means for reading the recording date of the image is further operable to read the recording time of the image; and the means for accessing the electronic diary and for reading information therefrom is operable to access the electronic diary and read information therefrom for use in generating the annotation data in accordance with both the recording date and recording time of the image.

7. An apparatus according to any preceding claim, further comprising means for selecting an electronic diary from amongst a plurality of electronic diaries as the diary to be accessed to read information to generate the annotation data.

8. An apparatus according to claim 7, wherein the means for selecting an electronic diary is operable to process the image data to determine therefrom which electronic diary to select.

9. An apparatus according to any preceding claim, further comprising means for searching the annotation data stored in the database in accordance with one or more keywords to identify each stored image having annotation data containing the keyword(s).

lo. An image processing apparatus, comprising: means for receiving image data defining an image; means for reading the recording date of the image; means for accessing electronic storage means storing data indexed by date and for reading data therefrom for the recording date of the image; means for generating annotation data for the image using the data read from the storage means; and means for writing the image data and the annotation data to a database configured to store the annotation data linked to the image data.

11. A method of processing image data, comprising: receiving image data defining an image; reading the recording date of the image; accessing an electronic diary previously accessed by a user to store user-defined information therein and reading user-defined information therefrom in accordance with the recording date of the image; generating annotation data for the image in dependence upon the user-defined information read from the electronic diary; and writing the image data and the annotation data to a database configured to store the annotation data linked to the image data.

12. A method according to claim 11, wherein the generated annotation data comprises text data.

13. A method according to claim 12, wherein the generation of annotation data comprises selecting at least one keyword from information read from the electronic diary and using the keyword(s) as the annotation data.

14. A method according to any of claims ll to 13, wherein the image data is processed to read the recording date thereof.

15. A method according to any of claims 11 to 13, wherein a date entered by a user is read as the image recording date.

16. A method according to any of claims 11 to 15, wherein: the recording date of the image is read as well as the recording time of the image; and the electronic diary is accessed and information is read therefrom for use in generating the annotation data in accordance with both the recording date and recording time of the image.

17. A method according to any of claims 11 to 16, further comprising selecting an electronic diary from amongst a plurality of electronic diaries as the diary to be accessed to read information to generate the annotation data.

18. A method according to claim 17, wherein the electronic diary is selected by processing the image data to determine therefrom which electronic diary to select.

19. A method according to any of claims 11 to 18, further comprising searching the annotation data stored in the database in accordance with one or more keywords to identify each stored image having annotation data containing the keyword(s).

20. A method of processing image data, comprising: receiving image data defining an image; reading the recording date of the image; accessing electronic storage means storing data indexed by date and reading data therefrom for the recording date of the image; generating annotation data for the image using the data read from the storage means; and writing the image data and the annotation data to a database configured to store the annotation data linked to the image data.

21. A storage medium storing computer program instructions for programming a programmable processing apparatus to become configured as an apparatus as set out in at least one of claims 1 to 10.

22. A signal carrying computer program instructions for programming a programmable processing apparatus to become configured as an apparatus as set out in at least once of claims 1 to 10.