Classifications

• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
  • G06V20/00—Scenes; Scene-specific elements
  • G06V20/35—Categorising the entire scene, e.g. birthday party or wedding scene
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06F—ELECTRIC DIGITAL DATA PROCESSING
  • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
  • G06F16/50—Information retrieval; Database structures therefor; File system structures therefor of still image data
  • G06F16/58—Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06F—ELECTRIC DIGITAL DATA PROCESSING
  • G06F18/00—Pattern recognition
  • G06F18/20—Analysing
  • G06F18/23—Clustering techniques
  • G06F18/232—Non-hierarchical techniques
  • G06F18/2321—Non-hierarchical techniques using statistics or function optimisation, e.g. modelling of probability density functions
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
  • G06V20/00—Scenes; Scene-specific elements
  • G06V20/30—Scenes; Scene-specific elements in albums, collections or shared content, e.g. social network photos or video

Definitions

• the invention relates generally to the field of digital image processing, and in particular to a method for identifying groups of digital images that portray recurring events in consumer image collections.
• the images in this group can be organized appropriately with links to previous years' vacations. Travel and tourism-related advertising can be targeted to fall within the planning phase of this time period, and the images can be shared with contacts with which the user regularly shares this type of images.
• a method of detecting recurring events in a digital image collection taken over a pre-determined period of time comprising, using a processor for:
• Collections are described in terms of events that are represented in an appropriate multidimensional space. Density-based clustering at different neighborhood sizes is used, filtering the neighborhood based on event characteristics to reduce the number of false matches in the group. An event signature based on event classification, location, and temporal characteristics is created to characterize events. This invention detects personal special dates such as birthdays and anniversaries, seasonal activities and holidays celebrated customized to a user's personal collection.
• the present invention is applicable for two of the commonly occurring classes of calendar-based recurring events in consumer collections— events that typically occur around the same date every year, e.g., birthdays, anniversaries, and some holidays; and events that are loosely tied to the calendar date. Although events in this second class have similar temporal characteristics, the exact date is often not followed year-over-year. These include holidays that do not follow the exact date, e.g., those that are on a certain day of the week during a specified week and month (such as Labor Day in the US;
• FIG. 1 is a block diagram of a system that practices the present invention
• FIG. 2 is an overall flowchart of the method of the present invention
• FIG. 3 is a more detailed flowchart of the event signature generation shown in block 130 of FIG. 2;
• FIG. 4 shows a specific example of the 2D representation of events produced in accordance with the present invention
• FIG. 5 shows a specific example of a 3D representation of events produced in accordance with the present invention.
• FIGS. 6A and 6B show two examples of displaying an organized collection showing the recurring event groups detected in the collection.
• the present invention can be implemented in computer systems as will be well known to those skilled in the art.
• some embodiments of the present invention will be described as software programs. Those skilled in the art will readily recognize that the equivalent of such a method can also be constructed as hardware or software within the scope of the invention.
• the present invention can be implemented in computer hardware and computerized equipment.
• the method can be performed in a digital camera, a multimedia smart phone, a digital printer, on an internet server, on a kiosk, and on a personal computer.
• FIG. 1 there is illustrated a computer system for implementing the present invention.
• the computer system is shown for the purpose of illustrating a preferred embodiment, the present invention is not limited to the computer system shown, but can be used on any electronic processing system such as found in digital cameras, home computers, kiosks, or any other system for the processing of digital images.
• a computer 10 includes a microprocessor-based unit 20 (also referred to herein as a processor) for receiving and processing software programs and for performing other processing functions.
• a microprocessor-based unit 20 also referred to herein as a processor
• a memory unit 30 stores user-supplied and computer- generated data which can be accessed by the processor 20 when running a computer program.
• a display device (such as a monitor) 70 is electrically connected to the computer 10 for displaying information and data associated with the software, e.g., by a graphical user interface.
• a keyboard 60 is also connected to the computer 10. As an alternative to using the keyboard 60 for input, a mouse can be used for moving a selector on the display device 70 and for selecting an item on which the selector overlays, as is well known in the art.
• Input devices 50 such as compact disks (CD) and DVDs can be inserted into the computer 10 for inputting the software programs and other information to the computer 10 and the processor 20.
• the computer 10 can be programmed, as is well known in the art, for storing the software program internally.
• media files (such as images, music and videos) can be transferred to the memory unit 30 of the computer 10 by use of input devices 50 such as memory cards, thumb drives, CDs and DVDs, or by connecting a capture device (such as camera, cell phone, video recorder) directly to the computer 10 as an input device.
• the computer 10 can have a network connection, such as a telephone line or wireless connection 80, to an external network, such as a local area network or the Internet.
• Software programs and media files can be transferred to the computer 10 from other computers or the Internet through the network connection.
• the present invention can be implemented in a combination of software or hardware and is not limited to devices which are physically connected or located within the same physical location.
• One or more of the devices illustrated in FIG. 1 can be located remotely and can be connected via a network.
• One or more of the devices can be connected wirelessly, such as by a radio-frequency link, either directly or via a network.
• a user's multi-year digital image collection 110 resides in the memory unit 30 of the computer 10.
• the digital image collection 110 spans at least 5 years of time.
• the other blocks in the figure are implemented by a software program and are executed by the processor 20 of the computer 10.
• the digital image collection 110 is provided to an event clustering algorithm 120 that groups the images in the digital image collection 110 into temporal events.
• a collection of images is classified into one or more events determining one or more largest time differences of the collection of images based on time or date clustering of the images and separating the plurality of images into the events based on having one or more boundaries between events where one or more boundaries correspond to the one or more largest time differences.
• sub-events can be determined (if any) by comparing the color histogram information of successive images as described in U.S. Patent No. 6,351 ,556. This is accomplished by dividing an image into a number of blocks and then computing the color histogram for each of the blocks.
• a block-based histogram correlation procedure is used as described in U.S. Patent No. 6,351,556 to detect sub-event boundaries.
• an event clustering method uses foreground and background segmentation for clustering images from a group into similar events. Initially, each image is divided into a plurality of blocks, thereby providing block-based images. Using a block-by-block comparison, each block-based image is segmented into a plurality of regions including at least a foreground and a background. One or more luminosity, color, position or size features are extracted from the regions and the extracted features are used to estimate and compare the similarity of the regions including the foreground and background in successive images in the group. Then, a measure of the total similarity between successive images is computed, thereby providing image distance between successive images, and event clusters are delimited from the image distances.
• each event forms an event point 310 in the space defined by the year number on the y- axis 320 and the day of the year on the x-axis 330, also referred to as event points in this application.
• the years are simply numbered chronologically to generate the year number e.g. if the collection spans 2005 to 2010, 2005 would correspond to year number 1, 2006 to year number 2 and so on with 2010 corresponding to year number 6.
• the day of the year is counted from the beginning of the year with January 1st as day 1.
• the week of year refers to the sequential number of the week starting with the first week of the year as 1.
• the last week of the year is week 53 when the first and last weeks are partial weeks.
• the day of the week are numbered sequentially from -3 to + 3 (including 0) starting with Monday. This representation is useful for detecting recurring events that are associated with the day of the week e.g. school sporting leagues, regular weekly gatherings, Easter, Thanksgiving and other holidays.
• spatial clustering 150 is performed on the event representation in the multi-dimensional space generated in 140.
• the density-based clustering approach Data Mining Concepts and Techniques by Han and Kamber, Elsevier, 2006, pg. 418-420
• This algorithm grows regions with sufficiently high point density into clusters.
• the neighborhood around any given central event point (x, y) is defined as (x ⁇ 2, y ⁇ 2) for detecting events closely tied to the calendar date.
• Core objects are identified that have greater than a threshold of points (5 points in this embodiment) in their neighborhood.
• the density-based clustering algorithm iteratively collects directly density-reachable objects from these core objects, terminating when no new points can be added.
• a larger neighborhood of (x ⁇ 7, y ⁇ 2) is selected around the central event point (x, y) with the same threshold (5 points) for qualifying as a core point.
• event points 310 that pass the event signature filtering process described in the next paragraph are included to compute the neighborhood points for any given event point 310.
• filtering based on event signature 130 can be used to refine the spatial clustering 150. This additional step is especially useful when using larger neighborhoods or when detecting recurring events within a year.
• the event signature 130 is used as a filter to determine whether points can be considered to be in the same neighborhood as any given central event point 310.
• the event signature 130 captures the commonality of features between events, and can be derived from content-based analysis at the image level or event-based analysis at the event level or both. In one embodiment, three main features obtained at the event level are used— day of the week, event category, and location— that show good correlation within events from the same recurring group to perform the event signature based filtering as shown in FIG. 3.
• Event category matching 220 determines if the potential neighboring event point 210 has the same event category label as the central point 205. In the preferred embodiment, the method described in Event Classification in Personal Image Collections by Das and Loui, IEEE Intl.
• a location matching module 230 checks if the potential neighboring event point 210 can be co-located with the central event point 205.
• the location where an event occurs is an important factor when determining whether it forms a recurring group with other events. Many recurring groups contain events that occur in the same locality.
• the event locations are matched using SIFT features as described by Das et al in "Event-based Location Matching for Consumer Image Collections " in the Proceedings of the ACM Int. Conf. on Image and Video Retrieval, 2008.
• SIFT features as described by Das et al in "Event-based Location Matching for Consumer Image Collections " in the Proceedings of the ACM Int. Conf. on Image and Video Retrieval, 2008.
• SIFT features as described by Das et al in "Event-based Location Matching for Consumer Image Collections " in the Proceedings of the ACM Int. Conf. on Image and Video Retrieval, 2008.
• SIFT features as described by Das et al in "Event-based Location Matching for Consumer Image
• the region in which a recurring group can be said to be located can be very broad, e.g., Florida during spring break. Conversely, there can be distinctions in the event groups based on much finer granularity than the town where they are located, e.g., the user may consider "School” to be a different location than "Home,” both of which are in the same town. In some instances, the location information can be irrelevant. For example, birthday parties are usually celebrated in the user's home town, but some can be at home and others can be at some special location.
• the day of week is used as a part of the event signature-based filter because in studies of consumer media collections, a significant association was found between members of the same recurring event group and the day of the week the event occurred, e.g. events from the same group could all occur on
• the day of week match 240 determines if two events have the same day of week label described above. For multi-day events, any overlap of day of the week is considered to be a match.
• features derived from content-based analysis of images in the event can also be included in the event signature.
• One such example is people- based matching where the presence of common people in both events is determined using available facial recognition technology (such as "OKAO Vision" Face Sensing Technology from Omron).
• Matching a common object can provide another matching criterion.
• Common scene classification such as beach, urban scene, or field of the images in the two events can also be used as a matching criterion.
• the event signature comparison block 250 generates the final decision on whether the potential neighboring event point 210 should be considered to be in the neighborhood of the central event point 205.
• the features described above that are included in the event signature 130 are not combined into a single value, as that is not meaningful in the given context.
• mismatches are not necessarily significant for any of the three features 220, 230, 240 discussed above. Instead, positive matches are meaningful, and these are noted.
• Equal weight is assigned to positive matches from any of the three features. For example, two events that occur on the same day of the week, have the same event category and the same location, would have an event signature- based match score of 3; whereas two events that occur on the same day of the week but have different event categories and no location match was found would have a score of 1.
• the clusters generated by the spatial clustering process 150 are output as recurring events 160 detected in the multi-year collection 110.
• the interpretation of these recurring events is based on the axes used in the multidimensional representation of the events.
• the images belonging to each recurring event are indexed so that they are linked to other images in the group.
• the recurring events are displayed to the user in an organized multi-year collection view.
• the events can be represented by a representative image or a collage of images from the event. Referring to FIGS. 6A and 6B, two common
• visualizations of an organized collection are shown - the timeline view 440 in FIG. 6A, and the calendar view 450 in FIG. 6B.
• the non-recurring events 420 are displayed on the timeline and in the calendar based on their date of occurrence.
• the recurring events 400 appear with icons 425, 430 that link to events in the previous and next time period in that recurring group. For example, a person's birthday event in 2010 would be linked to his birthday in 2009 and 2011. This formulation allows the user an easy access to related events that are separated by large time differences.

Landscapes

• Engineering & Computer Science (AREA)
• Theoretical Computer Science (AREA)
• Physics & Mathematics (AREA)
• General Physics & Mathematics (AREA)
• Data Mining & Analysis (AREA)
• Multimedia (AREA)
• General Engineering & Computer Science (AREA)
• Databases & Information Systems (AREA)
• Library & Information Science (AREA)
• Life Sciences & Earth Sciences (AREA)
• Evolutionary Computation (AREA)
• Evolutionary Biology (AREA)
• Computer Vision & Pattern Recognition (AREA)
• Bioinformatics & Computational Biology (AREA)
• Bioinformatics & Cheminformatics (AREA)
• Artificial Intelligence (AREA)
• Probability & Statistics with Applications (AREA)
• Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
• Image Analysis (AREA)

Priority Applications (3)

Applications Claiming Priority (2)

Publications (1)

Family

ID=44509719

Family Applications (1)

Country Status (5)

Families Citing this family (24)

Citations (7)

Family Cites Families (23)

• 2010
  • 2010-08-25 US US12/862,806 patent/US8634662B2/en not_active Expired - Fee Related
• 2011
  • 2011-08-18 JP JP2013525998A patent/JP5632084B2/ja not_active Expired - Fee Related
  • 2011-08-18 EP EP11748535.9A patent/EP2609527A1/en not_active Ceased
  • 2011-08-18 CN CN201180040728.1A patent/CN103069420B/zh not_active Expired - Fee Related
  • 2011-08-18 WO PCT/US2011/048169 patent/WO2012027178A1/en not_active Ceased
• 2013
  • 2013-12-18 US US14/133,013 patent/US8811755B2/en not_active Expired - Fee Related

Patent Citations (7)

Non-Patent Citations (6)

Also Published As

Similar Documents

Legal Events