GB2403304A

GB2403304A - Image and date information processing

Info

Publication number: GB2403304A
Application number: GB0314825A
Authority: GB
Inventors: Simon Michael Rowe
Original assignee: Canon Inc
Current assignee: Canon Inc
Priority date: 2003-06-25
Filing date: 2003-06-25
Publication date: 2004-12-29
Also published as: GB0314825D0

Abstract

An image processing apparatus 2 processes image data defining an image or a sequence of images to read the recording date of the image(s), to access an electronic calendar in accordance with the recording date to determine national calendar information such as national events and the season of the year on the date that the image/image sequence was recorded, and to generate annotation data describing the image subject matter based upon the determined national calendar information. Each input image or sequence of images processed by the apparatus is stored in an image database together with the generated annotation data. The database can then be searched in accordance with text keywords input by a user to retrieve images having those keywords in the associated annotation data.

Description

IMAGE PROCESSING

The present invention relates to the field of image processing and, more particularly, to the processing of image data by an image processing apparatus to generate annotation data comprising text data describing the image. The image data and text data can then be stored in a database linked together to facilitate text-based searching and retrieval of images therefrom by a user.

Many apparatus which generate annotation data for archiving images in a database are known. For example, GB-B-2,349,761 and GB-A-2,349,764 describe apparatus for generating annotation data for still images and video images, comprising data defining, for each person or animal in the image, the person, animal or object at which they are looking.

Similarly, systems are known which perform face recognition processing on an image to identify the people appearing in the image, and to generate annotation data defining the names of the identified people for storage in a database in association with the image data.

It is an object of the present invention to facilitate the automatic generation of reliable and accurate annotation data for images. It is a further object of the invention to facilitate, through electronic processing, an image annotation system which is quick and easy for a person to use.

According to the present invention, there is provided an apparatus and method for generating annotation data for an image for storage in a database with the image data, in which data defining national calendar information stored in an electronic calendar for the recording date of an image is read and used to generate annotation data defining the subject matter of the image.

In this way, reliable and accurate image annotation data describing when the user may remember the image being recorded and/or the subject matter of the image can be generated quickly without user input to facilitate subsequent retrieval of images from a database.

The present invention also provides a computer program product, embodied for example as a storage medium carrying instructions or as a signal carrying instructions, having instructions for causing a programmable processing apparatus to become configured as an apparatus as set out above or to perform a method as set out above.

Embodiments of the invention will now be described, by way of example only, with reference to the accompanying drawings, in which: Figure 1 schematically shows the components of an embodiment of the invention, together with the notional functional processing units and data stores into which the processing apparatus component may be thought of as being configured when programmed by programming instructions; Figure 2 schematically illustrates part of the data content stored in the electronic calendar in the apparatus of Figure l; and Figure 3 shows the processing operations performed by the apparatus of Figure 1.

Referring to Figure 1, an embodiment of the invention comprises a programmable processing apparatus 2, such as a personal computer, containing, in a conventional manner, one or more processors, memories, graphics cards etc., together with a display device 4 and user input devices 6, such as a keyboard, mouse etc. The processing apparatus 2 is programmed to operate in accordance with programming instructions input, for example, as data stored on a data storage medium 12 (such as an optical CD ROM, semiconductor ROM, or magnetic recording medium, etc.), and/or as a signal 14 (for example an electrical or optical signal input to the processing apparatus 2, for example from a remote database, by transmission over a communication network such as the Internet or by transmission through the atmosphere), and/or entered by a user via a user input device 6 such as a keyboard.

As will be described in detail below, the programming instructions comprise instructions to program the processing apparatus 2 to become configured to process image data defining an image or a sequence of images so as to read the recording date of the image(s), to access an electronic calendar in accordance with the recording date to determine national calendar information (comprising, for example, events of national applicability and season of the year) on the date that the image/image sequence was recorded, and to generate annotation data describing the image subject matter using the calendar information. For example, if an image was recorded on 25 December and the electronic calendar indicates that this date was Christmas Day, a public holiday and in winter then the text keywords "Christmas Day public holiday winter" would be generated as annotation data together with the month and year in which the image was recorded to describe the subject matter of the image. Each input image or sequence of images processed by the apparatus is stored in an image database together with the generated annotation data. The database can then be searched in accordance with text keywords input by a user to retrieve images having those keywords in the associated annotation data.

When programmed by the programming instructions, the processing apparatus 2 can be thought of as being configured as a number of functional units for performing processing operations and a number of data stores configured to store data. Examples of such functional units and data stores together with their interconnections are shown in Figure 1. The functional units, data stores and interconnections illustrated in Figure 1 are, however, notional, and are shown for illustration purposes only to assist understanding; as will be appreciated by the skilled person, they do not necessarily represent the units, data stores and connections into which the processors, memories, etc. of the processing apparatus 2 actually become configured.

Referring to the functional units shown in Figure 1, central controller 20 is arranged to process inputs from the user input devices 6, and also to provide control and processing for the other functional units. Working memory 30 is provided for use by central controller 20 and the other functional units.

A plurality of electronic calendars 40-1 to 40-n are provided, one for each respective year. Each electronic calendar 40 comprises, in a conventional way, a memory 44 storing dates and associated text data describing national calendar information for those dates, and an interface 42 to enable the data defining the dates and calendar information to be read. The data may be stored in each electronic diary 40 so as to Reread only".

In this embodiment, each electronic calendar 40 stores national calendar information comprising, firstly, national events occurring on each date of the year, and, secondly, the season of the year (spring, summer, autumn, winter) for each date. The national events defined in each electronic calendar 40 may comprise, inter alla, public holidays, Christmas Eve, Christmas Day, Boxing Day, New Year's Eve, New Year's Day, Mothering Sunday/Mother's Day, Pancake Day/Shrove Tuesday, Valentines Day, Saints days (such as St Patrick's Day), religious festivals (such as Easter, Good Friday, Easter Sunday, Easter Monday, etc), Father's Day, Halloween, and Bonfire Night/Guy Fawkes Day.

Figure 2 schematically illustrates some of the text data stored in an electronic calendar 40.

Referring to Figure 2, text data for the dates December 20 to December 31 is illustrated. For December 20, the word "autumn'' is stored defining the season for that date, while the word "winter" is stored for each of the dates December 21 to December 31. The word. 'Christmas" is stored for each of the dates December 20 to December 30 (either as a word on its own or as part of the words Christmas Eve or Christmas Day) because a user may associate each of these dates, and hence images recorded on one of these dates, with the Christmas period. The words ''Christmas Eve" are stored for December 24, the words "'Christmas Day.' are stored for December 25, the words "Boxing Day" are stored for December 26, and the words "New Year's Eve" are stored for December 31. In addition, the words "Public Holiday'' are stored for December 25 and December 26 because these dates are national holidays. Of course, the text data stored in an electronic calendar will vary dependent upon country.

Input data interface 50 is arranged to receive, and write to memory, image data defining images to be stored in an image database and for which annotation data is to be generated for storage in association therewith in the database. The input image data may be input to processing apparatus 2 as data stored on a storage medium 42, or as data carried by a signal 44.

Image data store 60 is configured to store the input image data for subsequent processing.

Image date reader 70 is operable to read the image data of an image stored in image data store 60, to identify the pixels on which is overlaid the date of recording of the image, and to read the recording date.

Calendar selector 80 is operable to select one of the electronic calendars 40-1 to 40-n for use in generating annotation date, with the selection being carried out in accordance with the year read by the image date reader of the image for which the annotation data is to be generated.

Annotation data generator 90 is operable to access the electronic calendar selected by calendar selector 80 in accordance with the image date read by image date reader 70, to read text data stored in the calendar for the recording date of the image, and to generate text annotation data for the image using the data read from the electronic calendar 40.

Image database 110 is configured to store image data from image data store 60 linked to the corresponding annotation data generated by annotation data generator 90, thereby allowing text-based search and retrieval of images from the database.

Database search engine 120 is operable to search the annotation data in the image database 110 in accordance with one or more keywords entered by a user using a user input device 6 such as a keyboard, so as to identify each image in the image database 110 which is linked to annotation data containing the keyword(s). Database search engine 120 is further operable to enable a user to select one or more of the identified images from image database 110 and to display the selected image(s) on display device 4.

Display controller 130, under the control of central controller 20, is operable to control display device 4 to display image data received as input image data, and to display image data retrieved from image database 110.

Output data interface 140 is operable to output data from processing apparatus 2 for example as data on a storage medium 142 (such as an optical CD ROM, semiconductor ROM or magnetic recording medium, etc.) and/or as a signal 144 (for example an electrical or optical signal transmitted over a communication network such as the Internet or through the atmosphere). In this embodiment, the output data comprises data defining an image from image data store 60 and the associated annotation data generated by annotation data generator 90. The output data may, for example, be stored in an external database.

A recording of the output data may be made by recording the output signal 144 either directly or indirectly (for example by making a recording and then making a subsequent copy recording) using recording apparatus (not shown).

Figure 3 shows the processing operations performed by processing apparatus 2 in this embodiment.

Referring to Figure 3, at step S3-2, input data interface stores image date in image data store 60 for each image to be stored in image database 110 and for which annotation data is to be generated. In this embodiment, each input image contains pixel data on which has been overlaid the recording date of the input image (this information being provided in a conventional way by recording each image with a camera having a so-called "databack" which overlays the recording date on the image). Each image stored at step S3-2 may comprise a frame of video data, a "still" image, or a sequence of video images for which one respective set of annotation data is to be generated to be associated with all of the images in the sequence.

At step S3-4, image date reader 70 reads the pixel data of the next image (or the first image of the next image sequence) to be stored in image database 110 (this being the first image or image sequence stored at step S3-2 the first time step S3-4 is performed) to determine the recording date of the image. This processing comprises identifying the pixels on which is overlaid the recording date, and reading the recording date from the identified pixels.

At step S3-6, calendar selector 80 selects the electronic calendar 40 corresponding to the year in which the image currently being processed was recorded (read at step S3-4) from among the plurality of electronic calendars 40-1 to 40-n.

At step S3-8, annotation data generator 90 reads the text entries describing national calendar information stored in the electronic calendar 40 selected at step S3-6 for the date (that is, month and day) read at step S3-4.

At step S3-10, annotation data generator 90 generates text annotation data for the image (or image sequence) currently being processed using the text information read from the electronic calendar 40 at step S3-8. More particularly, annotation data generator 90 uses the text information read from the electronic calendar 40 as annotation data for the image (or image sequence) and, in addition, adds the month and year of the recording date of the image (read at step S3-4) to the annotation data. The annotation data therefore comprises one or more text annotations, with each annotation containing one or more keywords.

At step S3-12, the image data for the image (or image sequence) currently being processed and the annotation data generated at step S3-10 are stored in the image database 110. The annotation data is stored in the database so that it is linked to the image data, thereby facilitating text-based search and retrieval of images by database search engine 120. More particularly, the annotation data may be searched by the database search engine 120 in accordance with one or more keywords input by a user, to identify annotation data which contains the keyword(s) and the associated image(s).

At step S3-14, a check is carried out to determine whether annotation data has been generated for all of the images and image sequences stored at step S3-2. Steps S3-4 to S3-14 are repeated until annotation data has been generated for each image/image sequence, and each image/image sequence has been stored in image database linked to the generated annotation data.

Many modifications and variations can be made to the embodiment described above within the scope of the claims.

For example, in the embodiment described above, each image stored at step S3-2 includes data identifying the recording date of the image, and image date reader 70 is arranged to read the image data to determine the recording date at step S3-4. However, instead, the user may enter data defining the recording date of one or more images at step S3-2, and image date reader 70 may be arranged to read the date entered by the user at step S3-4.

In the embodiment described above, the date at which an image was recorded is read at step S3-4 and national calendar information is read at step S3-8 in accordance with the date. In addition, the time at which an image was recorded may be read at step S3-4 as well as the date on which the image was recorded, and national calendar information specifying time-of-day-dependent events such as "dawn", "dusk'', eta may be stored for each date in each electronic calendar 40 and may be read at step S3-8 in accordance with the date and time. As a result, annotation data may be generated at step S3-10 in accordance with the recording date and time of the image.

In the embodiment described above, a plurality of electronic calendars 40 may be provided for each year, each respective electronic calendar 40 storing calendar information for a different country. The user may then select an electronic calendar 40 to be used to generate the annotation data, and may change the selection if images recorded in a different country are to be processed.

In the embodiment described above, each electronic calendar 40 stores national calendar information comprising national events and also the season at each date, and annotation data generator 90 generates annotation data for each image defining both the national events and season when the image was recorded. However, the season data may be omitted from the national calendar information in each electronic calendar 40 and annotation data generator 90 may be arranged to generate annotation data defining national events but not season.

Alternatively, national event data may be omitted from the national calendar information in each electronic calendar 40 and annotation data generator may be arranged to generate annotation data defining season but not national events.

In the embodiment described above, each electronic calendar 40 is part of processing apparatus 2. However, instead, one or more of the electronic calendars 40 may be stored in a separate apparatus (such as a personal digital assistant (PDA), eta) connected to processing apparatus 2 for access by the calendar selector 80.

In the embodiment described above, image database 110 is part of processing apparatus 2. However, instead, image database 110 may be provided in a separate apparatus, and image data and annotation data may be output from processing apparatus 2 via output data interface 140 for storage in the external image database 110. Of course, the image database may comprise a recording medium, such as a writeable compact disk, and image data and annotation data may be stored thereon using a recording device, such as a CD writer.

In the embodiment described above, processing is performed by a computer using processing routines defined by software programming instructions. However, some or all, of the processing could, of course, be performed using hardware or firmware.

Claims

1. An image processing apparatus, comprising: means for receiving image data defining an image; means for reading the recording date of the image; means for accessing an electronic calendar storing predefined national calendar information and for reading information therefrom in accordance with the recording date of the image; means for generating annotation data for the image in dependence upon the national calendar information read from the electronic calendar; and means for writing the image data and the annotation data to a database configured to store the annotation data linked to the image data.

2. An apparatus according to claim l, wherein the means for generating the annotation data is operable to generate annotation data comprising text data.

3. An apparatus according to claim 2, wherein the means for generating the annotation data is operable to use words read from the electronic calendar as annotation data.

4. An apparatus according to any preceding claim, wherein the means for reading the recording date of the image is operable to process the image data to read the recording date thereof.

5. An apparatus according to any preceding claim, wherein the means for reading the recording date of the image is operable to read a date entered by the user.

6. An apparatus according to any preceding claim, wherein: the means for reading the recording date of the image is further operable to read the recording time of the image; and the means for accessing the electronic calendar and for reading information therefrom is operable to access the electronic calendar and read information therefrom for use in generating the annotation data in accordance with both the recording date and recording time of the image.

7. An apparatus according to any preceding claim, further comprising means for selecting an electronic calendar from amongst a plurality of electronic calendars as the calendar to be accessed to read information to generate the annotation data.

8. An apparatus according to claim 7, wherein the means for selecting an electronic calendar is operable to select an electronic calendar in dependence upon the year in which the image for which annotation data is to be generated was recorded.

9. An apparatus according to any preceding claim, further comprising means for searching the annotation data stored in the database in accordance with one or more keywords to identify each stored image having annotation data containing the keyword(s).

10. A method of processing image data, comprising: receiving image data defining an image; reading the recording date of the image; accessing an electronic calendar storing predefined national calendar information and reading information therefrom in accordance with the recording date of the image; generating annotation data for the image in dependence upon the national calendar information read from the electronic calendar; and writing the image data and the annotation data to a database configured to store the annotation data linked to the image data.

11. A method according to claim 10, wherein the generated annotation data comprises text data.

12. A method according to claim 11, wherein the generation of annotation data comprises using at least one word read from the electronic calendar as annotation data.

13. A method according to any of claims 10 to 12, wherein the image data is processed to read the recording date thereof.

14. A method according to any of claims 10 to 12, wherein a date entered by the user is read as the image recording date.

15. A method according to any of claims 10 to 14, wherein: the recording time of the image is read as well as the recording date; and the electronic calendar is accessed and information is read therefrom for use in generating the annotation data in accordance with both the recording date and recording time of the image.

16. A method according to any of claims 10 to 15, further comprising selecting an electronic calendar from amongst a plurality of electronic calendars as the calendar to be accessed to read information to generate the annotation data.

17. A method according to claim 16, wherein an electronic calendar is selected in dependence upon the year in which the image for which annotation data is to be generated was recorded.

18. A method according to any of claims 10 to 17, further comprising searching the annotation data stored in the database in accordance with one or more keywords to identify each stored image having annotation data containing the keyword(s).

19. A storage medium storing computer program instructions for programming a programmable processing apparatus to become configured as an apparatus as set out in at least one of claims 1 to 9.

20. A signal carrying computer program instructions for programming a programmable processing apparatus to become configured as an apparatus as set out in at least one of claims 1 to 9.