Classifications

• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06F—ELECTRIC DIGITAL DATA PROCESSING
  • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
  • G06F16/50—Information retrieval; Database structures therefor; File system structures therefor of still image data
  • G06F16/58—Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
  • G06V20/00—Scenes; Scene-specific elements
  • G06V20/40—Scenes; Scene-specific elements in video content

Definitions

• the invention relates generally to the field of digital image processing, and in particular to a method for identifying groups of digital images that portray significant events in consumer image collections.
• Event and "sub-event” are used in an objective sense to indicate the products of a computer mediated procedure that attempts to match a user's subjective perceptions of specific occurrences (corresponding to events) and divisions of those occurrences (corresponding to sub-events).
• Another method of automatically organizing images into events is disclosed in U.S. Patent No. 6,915,011, assigned to A. Loui, M. Jeanson, and Z. Sun, entitled “Event clustering of images using foreground and background segmentation” issued July 5, 2005. The events detected are chronologically ordered in a timeline from earliest to latest.
• a method for determining significant events in a digital image collection comprising, using a processor for
• Fig. 1 is a block diagram of a system that practices the present invention
• Fig. 2 is an overall flowchart of the method practiced by Fig. 1;
• Fig. 3 is a more detailed flowchart of the time-series generator shown in block 110 of Fig. 2;
• Fig. 4 is a more detailed flowchart of the time-series modeling block 115 of Fig. 2;
• Figs. 5(a), (b) and (c) show a specific example of the image counts time- series and analysis produced in accordance with the present invention
• Fig. 6 is a more detailed flowchart of the significant event detector shown in block 120 of Fig. 2; and Fig. 7 is a flowchart of the time granularity selection method in accordance with the present invention.
• the present invention can be implemented in computer systems as will be well known to those skilled in the art.
• some embodiments of the present invention will be described as software programs.
• the equivalent of such a method may also be constructed as hardware or software within the scope of the invention.
• image manipulation algorithms and systems are well known, the present description will be directed in particular to algorithms and systems forming part of, or cooperating more directly with, the method in accordance with the present invention.
• Other aspects of such algorithms and systems, and hardware or software for producing and otherwise processing the image signals involved therewith, not specifically shown or described herein can be selected from such systems, algorithms, components, and elements known in the art.
• all software implementation thereof is conventional and within the ordinary skill in such arts. Videos in a collection are included in the term "images" in the rest of the description.
• the present invention can be implemented in computer hardware and computerized equipment.
• the method can be performed in a digital camera, a multimedia smart phone, a digital printer, on an internet server, on a kiosk, and on a personal computer.
• FIG. 1 there is illustrated a computer system for implementing the present invention.
• the computer system is shown for the purpose of illustrating a preferred embodiment, the present invention is not limited to the computer system shown, but can be used on any electronic processing system such as found in digital cameras, home computers, kiosks, or any other system for the processing of digital images.
• the computer 10 includes a microprocessor-based unit 20 (also referred to herein as a processor) for receiving and processing software programs and for performing other processing functions.
• a microprocessor-based unit 20 also referred to herein as a processor
• a memory unit 30 stores user-supplied and computer- generated data which may be accessed by the processor 20 when running a computer program.
• a display device (such as a monitor) 70 is electrically connected to the computer 10 for displaying information and data associated with the software, e.g., by means of a graphical user interface.
• a keyboard 60 is also connected to the computer. As an alternative to using the keyboard 60 for input, a mouse can be used for moving a selector on the display device 70 and for selecting an item on which the selector overlays, as is well known in the art.
• Input devices 50 such as compact disks (CD) and DVDs can be inserted into the computer 10 for inputting the software programs and other information to the computer 10 and the processor 20.
• the computer 10 can be programmed, as is well known in the art, for storing the software program internally, hi addition, media files (such as images, music and videos) can be transferred to the memory 30 of the computer 10 by means of input devices 50 such as memory cards, thumb drives, CDs and DVDs, or by connecting a capture device (such as camera, cell phone, video recorder) directly to the computer 10 as an input device.
• the computer 10 can have a network connection, such as a telephone line or wireless connection 80, to an external network, such as a local area network or the Internet.
• Software programs and media files can be transferred to the computer from other computers or the Internet through the network connection.
• the present invention can be implemented in a combination of software or hardware and is not limited to devices which are physically connected or located within the same physical location.
• One or more of the devices illustrated in FIG. 1 can be located remotely and can be connected via a network.
• One or more of the devices can be connected wirelessly, such as by a radio-frequency link, either directly or via a network.
• a user's digital image collection 105 resides in the memory 30 of a computer 10.
• the other blocks in the figure are implemented by a software program and are executed by the processor 20 of the computer 10.
• the digital image collection 105 is provided to a time-series generator 110.
• Fig. 3 shows the steps in the time-series generation process.
• the image capture date and time information are extracted 205 from the EXIF metadata stored in the image files of the digital image collection 105 by the capture device (such as a camera).
• a set of time units are determined - these time units could be a few months (capturing a season), a month, a week, a day, or hours - the size of the unit being referred to as granularity.
• the range of time units chosen is limited by the size of the image collection. Since at least 40 to 50 data points are needed for producing reliable results, a collection spanning one year is limited to a maximum granularity of one week; and season- level granularity would require a collection spanning 10 years or more. In the preferred embodiment, the finest granularity used is parts of a day (morning, afternoon, evening), with the maximum granularity being determined by the size of the image collection. Using at least one year of the user's collection is recommended.
• the accumulators 215 count the number of images in each time unit with the beginning of the collection being at unit zero, producing a set of image counts time-series 225.
• Fig. 4 shows the steps in time-series modeling to generate a suitable model.
• time-series modeling There are many well established methods for time-series modeling (ref. "Introduction to Time Series and Forecasting", Brockwell and Davis, Springer- Verlag 2002).
• the image counts time-series are typically non-stationary (i.e. their mean and standard deviation may vary over time). Since pictures are often taken in groups, e.g. there may be consecutive days of picture-taking during vacations and family gatherings, and consecutive days of zero image counts during week-days, the model needs to include an auto- regressive component that captures the correlation with previous values of the data.
• the model also needs to include a moving average component that can capture gradual changes in picture-taking behavior.
• the model that is appropriate in this situation is the ARIMA (Auto-Regressive Integrated Moving Average) model (Brockwell and Davis, supra, pp 179-187).
• the model ARIMA (p.d.q) has three main parameters —p being the order of the autoregressive component, q being the order of the moving average component and d being the order of differencing required for dealing with the deviations from stationarity.
• An ARBVLA. (p,d, q) model is given by where L is the lag operator, ⁇ are the parameters of the autoregressive part of the model, ⁇ are the parameters of the moving average part, and the ⁇ are the error terms.
• the error terms are generally assumed to be independent, identically distributed (iid) variables sampled from a normal distribution with zero mean.
• the first step is to estimate the order of the time-series model 310.
• the initial values for p and q are determined based on the autocorrelation plot (ACF) and partial autocorrelation plot (PACF) of the data (Brockwell and Davis, supra, pp 94-95). If the ACF exponentially decays to zero or shows damped oscillations while decaying to zero, the initial value of/? is chosen to be one less than the lag at which the PACF drops to zero, where it is typical to consider a 95% confidence interval band around zero instead of exactly zero value.
• the initial value of q is chosen to be one less than the lag at which the ACF becomes zero (typically, within a 95% confidence band around zero). Based on experiments on a large number of consumer collections, the case where the ACF does not decay to zero or decays very slowly indicating severe non- stationarity that requires high order of differencing was not found in this domain. So the initial estimate o ⁇ d is set to 1.
• An example is shown in Fig. 5.
• Fig. 5(a) shows an image counts time-series covering one year with a calendar day as the time unit.
• Fig. 5(b) shows the ACF plot and Fig. 5(c) shows the PACF plot. The solid vertical lines show the 95% confidence interval. Both plots show significant values till a lag of 2 and decay to zero after that. Based on this information, the/? and q values are set to 1.
• the value of d is at the default level of 1.
• a set of ARIMA models are fitted 315 to the image counts time-series to include variations around the initial estimates of the order parameters.
• the model fitting process determines the values of ⁇ and ⁇ in equation (1). Note that the number of parameters that need to be determined equals p+q+J since there are/?
• ⁇ parameters, q ⁇ parameters and the additional parameter is the standard deviation of the error term, ⁇ .
• the model fitting process is implemented in most commercially available statistical analysis software packages (e.g. JMP from SAS Corporation). hi practice, mixed models (that include auto-regressive and moving average components) such as ARIMA are difficult to identify and involve much trial and error. Some models fitted in 315 may have to be discarded on the basis of poor parameter estimation or poor fit to the data. This is done in step 320 based on the following two checks: (1) some of the model parameters for the ARIMA models fitted in 315 may have a t-Ratio less than 2, which indicates that the probability that the parameter value is zero is greater than 5%.
• the residuals are computed between the models fitted in 315 and the image counts time-series.
• the residuals are defined as the difference between the value predicted by the model and the actual value at that time instant. In the ideal case, where the model fits the data well, the residual is approximately iid (independent and identically distributed). If the residuals are iid, the ACF plot of the residuals will have values that are within the 95% confidence level for all lags. If this is not the case for more than 3 out of 40 lags, or if one falls far outside the 95% bounds, the iid hypothesis can be rejected (Brockwell and Davis, supra, pp 166-167). The ARIMA model does not fit the data very well in this case, and it is discarded.
• the remaining models all fit the data and are viable.
• the identification of the best model can be based on a number of criteria used to determine the goodness-of-fit (Brockwell and Davis, supra, pp 171-174).
• the Akaike Information Criterion (AICC) is used as a goodness-of-fit measure, as defined in Brockwell and Davis, supra, pp 171. This criterion is computed 325 for the ARIMA models remaining after step 320.
• the model showing the best (lowest) value of AICC is selected 330.
• the ARIMA model is very commonly used in forecasting for economic and financial markets, and it is well known to persons skilled in the art that most commercially available statistical analysis software packages include tools for fitting ARIMA models by specifying the/?, d and q values (e.g. JMP from SAS corporation, Autobox from Automatic Forecasting Systems and Forecast Pro from Business Forecast Systems Inc.), as well as tools for analyzing the ACF and PACF plots and performing the tests mentioned above.
• the selected ARIMA model in step 330 is provided to the significant event detector 120 that is used to determine significant events in the collection.
• Fig. 6 shows the steps followed in the significant event detector 120. Referring to Fig.
• the predicted output of the selected ARIMA model 410 is compared with the image counts time-series 405 that was used to generate the model. Residuals are computed 415 as the difference between the predicted output of the model and the image counts time-series at each time step.
• the variance ( ⁇ ) of the residuals is computed and a threshold is determined 420 based on this variance. In the preferred embodiment, a threshold of ⁇ /3 is used.
• the time steps where the absolute value of the residual is greater than the threshold are identified as "time steps of interest" 430.
• Significant events are identified 440 by merging adjacent time steps of interest and by retaining only the time steps or merged group of time steps that have image counts above a minimum threshold. In the preferred embodiment, this minimum threshold is the mean image count of the image counts time-series. Referring to Fig. 7, each of the image counts time-series generated in step
• time-series modeling 115 and significant event detection 120 are passed through time-series modeling 115 and significant event detection 120 to produce significant events at different time granularities 510. These significant events 510 are made available to the time granularity selector 530 which selects the set of significant events to use based on additional inputs 520.
• the additional inputs can include user actions, system requirements or user preferences.
• significant events can be selected at the time granularity at which the user selects to view the collection. For example, if the user is viewing a short time-span of a single day, significant events at the finest granularity are shown; whereas, if the user is viewing the collection over five years, significant events at the weekly time granularity is appropriate.
• the system requirement in terms of display capability can also dictate the number of significant events, and therefore, the granularity selected. For example, if approximately 10 events will fit the display, then the granularity is selected so that the number of significant events is close to that number. The user may also set the preference for viewing significant events at a certain granularity.
• Time-series generator Time-series modeling step
• Significant event detector Extract date/time step Accumulators for different time units
• Estimate initial parameters step
• Fit ARIMA models Choose viable models step
• Compute goodness-of-fit measures Choose best ARIMA model step
• Image counts time-series ARIMA model Compute residuals step Determine threshold step Identify time steps of interest step Identify significant events step Significant events Additional inputs Time granularity selector

Landscapes

• Engineering & Computer Science (AREA)
• Theoretical Computer Science (AREA)
• Physics & Mathematics (AREA)
• General Physics & Mathematics (AREA)
• Multimedia (AREA)
• Library & Information Science (AREA)
• Data Mining & Analysis (AREA)
• Databases & Information Systems (AREA)
• General Engineering & Computer Science (AREA)
• Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
• Processing Or Creating Images (AREA)
• Image Analysis (AREA)

Priority Applications (3)

Applications Claiming Priority (2)

Publications (1)

Family

ID=42646272

Family Applications (1)

Country Status (5)

Families Citing this family (14)

Citations (5)

Family Cites Families (7)

• 2009
  • 2009-06-19 US US12/487,686 patent/US8340436B2/en not_active Expired - Fee Related
• 2010
  • 2010-06-04 JP JP2012516050A patent/JP5548260B2/ja not_active Expired - Fee Related
  • 2010-06-04 EP EP10726666A patent/EP2443568A1/en not_active Withdrawn
  • 2010-06-04 CN CN201080026591.XA patent/CN102804178B/zh not_active Expired - Fee Related
  • 2010-06-04 WO PCT/US2010/001637 patent/WO2010147623A1/en not_active Ceased

Patent Citations (6)

Non-Patent Citations (1)

Also Published As

Similar Documents

Legal Events