Classifications

• • G—PHYSICS
  • G11—INFORMATION STORAGE
  • G11B—INFORMATION STORAGE BASED ON RELATIVE MOVEMENT BETWEEN RECORD CARRIER AND TRANSDUCER
  • G11B27/00—Editing; Indexing; Addressing; Timing or synchronising; Monitoring; Measuring tape travel
  • G11B27/02—Editing, e.g. varying the order of information signals recorded on, or reproduced from, record carriers
  • G11B27/031—Electronic editing of digitised analogue information signals, e.g. audio or video signals
  • G11B27/034—Electronic editing of digitised analogue information signals, e.g. audio or video signals on discs
• • G—PHYSICS
  • G11—INFORMATION STORAGE
  • G11B—INFORMATION STORAGE BASED ON RELATIVE MOVEMENT BETWEEN RECORD CARRIER AND TRANSDUCER
  • G11B27/00—Editing; Indexing; Addressing; Timing or synchronising; Monitoring; Measuring tape travel
  • G11B27/02—Editing, e.g. varying the order of information signals recorded on, or reproduced from, record carriers
  • G11B27/031—Electronic editing of digitised analogue information signals, e.g. audio or video signals
  • G11B27/036—Insert-editing

Definitions

• the present invention relates to a method of integrating a still photo within a video sequence.
• the invention moreover relates to a device and a computer program product for integrating a still photo within a video sequence.
• the photo set can also be exploited to enrich the video by adding some photos inside the video.
• some small video clips can be artificially generated, and included in the video.
• the basic assumption is that, by adding some content-related photos inside the video stream, the resulting video will be much more informative to the user rather than considering the single video source.
• the Muvee Reveal AutoProducer from Muvee Technologies is an example of a commercial software package that allows inserting photos into video.
• the photos are animated with some virtual camera motion. The idea is that, by doing so, watching a photo can become a more pleasant experience for the user.
• an alternative method of combining video and still photos would be advantageous, and in particular a method wherein the combination of photos and video provides enhanced user experience due to a visually more appealing experience would be advantageous.
• the invention preferably seeks to mitigate, alleviate or eliminate one or more of the above mentioned disadvantages singly or in any combination.
• This object and several other objects are obtained in a first aspect of the invention by providing a method of integrating a still photo within a video sequence, the method comprising the steps of obtaining a video sequence and a still photo, creating a synthesized video sequence, wherein the step of creating a synthesized video sequence includes the steps of: - estimating a video camera motion in the video sequence at an insertion position in the video sequence at which the still photo is to be included, creating a virtual video sequence of sub frames of the still photo, said virtual video sequence having a virtual camera motion, said virtual camera motion of the virtual video sequence being correlated to said video camera motion at said insertion position, and - combining the video sequence and the virtual video sequence into the synthesized video sequence.
• the virtual camera motion is correlated to video camera motion
• the video camera motion is taken into account during determination of the virtual camera motion, and that the virtual camera motion to some degree is made to conform to the video camera motion.
• the virtual camera motion is made to be similar or identical to the video camera motion.
• virtual video sequence is meant to denote a sequence of sub frames of the still photo, each sub frame being rescaled to the dimensions of the images within the video sequence, and the sequence of sub frames together possibly representing scenes in motion.
• the "virtual camera motion” is a visual effect in the virtual video sequence that corresponds to a simulated camera motion in a sequence of frames obtained from parts of the photo, e.g. by cropping and/or rescaling.
• the frame of the virtual camera motion is denoted "sub frames” or "sub frames of the still photo”.
• sub frame is meant to denote a continuous subpart of the still photo; the sub frame is typically interpolated up to the pixel dimensions of the still photo, corresponding to a zooming in on the sub frame.
• insertion position is meant to denote the position within the range of frames of the video sequence, viz. a timely position. Thus, the term is not related to inserting part of a still photo within a single frame of the video sequence.
• the virtual camera motion at the beginning of the virtual video sequence is correlated to the estimated camera motion in the video sequence prior to the insertion position.
• a smooth transition from the video sequence to the virtual video sequence of sub frame of the still photo is achieved.
• the virtual camera motion at the end of the virtual video sequence is correlated to the estimated camera motion in the video sequence subsequent to said insertion position.
• a smooth transition from the virtual video sequence of sub frame of the still photo to the video sequence is achieved.
• the method moreover comprises the step of determining said insertion position. Such a determination of the insertion position may be based on synchronization, e.g. based on time stamp information stored together with the still photo and the video sequence.
• the insertion position may advantageously be determined such that the still photo is inserted between video segments recorded at a point in time close to the instant in time of taking the still photo.
• the insertion position may also be based on similar objects identified on the video sequence and the still photo, such as a tourist attraction occurring on both the video sequence and the still photo.
• the determination comprises segmenting said video sequence into video segments and determining said insertion position in the video sequence as a position between two video segments.
• the segmenting of the video sequence into video segments may be a division of the video sequence into segments of related video frames, where different segments relate to different scenes, camera angles, instants in time, or other.
• the insertion of the virtual video sequence corresponding to the still photo does not create extra breakups within the video sequence, in that the virtual video sequence is inserted in a position of an existing breakup within the original video sequence.
• one or more specific regions of interest of the photo is/are taken into account in the step of creating said virtual video sequence.
• the virtual video sequence may e.g. comprise a zoom- in of a part of particular interest within the still photo.
• the estimation of the video camera motion may comprise estimating the pan, tilt and/or zoom. However, other camera motion parameters might also be taken into account.
• it may also comprise selecting sub frames of the still photo for the virtual video sequence, wherein none of the sub frames of the still photo exceeds the boundaries of the still photo for the zoom factors to be applied to the sub frames.
• the selection of such sub frames may comprise determination and application of a differential zoom factor for a plurality of sub frames of the still photo.
• the differential zoom factor is an auxiliary zoom factor which may be added to the zoom factor in a situation where a zooming out of a still photo otherwise would result in exceeding the boundary of the still photo.
• the invention moreover relates to a device for integrating a still photo within a video sequence.
• the invention relates to a computer program product being adapted to enable a computer system comprising at least one computer having data storage means associated therewith to control device according to the invention.
• This aspect of the invention is particularly, but not exclusively, advantageous in that the present invention may be implemented by a computer program product enabling a computer system to perform the method of the invention.
• some known device such as a video camera device, may be changed to operate according to the present invention by installing a computer program product on a computer system controlling the device .
• Such a computer program product may be provided on any kind of computer readable medium, e.g. magnetically or optically based medium, or through a computer based network, e.g. the Internet.
• Fig. 1 is a flow-chart of a method according to the invention
• Fig. 2 is a schematic drawing of an example of the method according to the invention.
• Fig. 3 a schematic drawing of an example of the steps performed in video transition synthesizing of Fig. 2;
• Fig. 4 shows diagrams of three camera motion parameters
• Fig. 5 shows diagrams of a zoom parameter, an adjusted zoom parameter and the resulting zoom parameter after adjustment.
• Fig. 6 indicating allowed translation values for different zoom values.
• Figure 1 is a flow-chart of a method 10 according to the invention.
• the method 10 starts in 11 and continues to step 12, wherein a video sequence and a still photo are obtained.
• step 13 a synthesized video sequence is created.
• Step 13 comprises steps 14-16, whereof step 14 comprises estimating a video camera motion in the video sequence at an insertion position in the video sequence at which the still photo is to be included.
• the camera motion estimation typically comprises the pan, tilt and zoom parameters of the video sequence.
• step 15 a virtual video sequence of sub frames of the still photo is created, where the virtual video sequence has a virtual camera motion and where the virtual camera motion of the virtual video sequence is correlated to the video camera motion at said insertion position.
• step 16 the video sequence and the virtual video sequence are combined into the synthesized video sequence.
• the method ends in step 17.
• Figure 2 is a schematic drawing of an example of the method 100 according to the invention.
• the input consist of an original video 20 and a number of still photos from a photo set 21.
• the camera motion in the original video is estimated.
• Camera motion may be estimated by estimating the camera motion parameters: pan, tilt and zoom.
• the following camera motion parameters could also be estimated: rotation, tracking, booming and dolling.
• Block 22 results in a set of estimated motion parameters 25 which are subsequently exploited in order to synthesize coherent transitions between video and photos.
• other low- level features of the images in the video are estimated, resulting in a set of estimated low level features 26.
• Such low-level features may comprise one or more of the following: edge detection, corner detection, blob detection, ridge detection, scale-invariant feature transform.
• the set of estimated low- level features 26 may subsequently be used during segmentation of the video, a summarization process and in the process of determining start and end parameters for the video transition synthesis.
• the set of estimated motion parameters 25 and the set of estimated low level features 26 are employed in a segmentation process, wherein the original video 20 is divided into a set of video segments 29.
• time synchronization between the original video 20 and the photo set 21 is performed, e.g. by exploiting timestamp information.
• a summarization process is performed, wherein the video segments suitable for forming a video summary are selected:
• the suitability of the video segments for the summary may be determined by e.g. the low- level features of the segment and the number of photos synchronized in that segment.
• the result of the process 30 is a video-photo composition segment set 31.
• the transitions between the video and the photos are synthesized.
• the process 32 uses the estimated motion parameters 25, a photo from the photo set 21 or from the synchronized photo set 27, a portion of video from the original video 20 or from the video-photo composition segment set 31 as well as information on the point in time of insertion of the photo inside the video. Moreover, the process 32 may take one or more regions of interest of the photo into account.
• the process 32 is described in further detail in relation to Figure 3.
• the result of the process 32 is a set of transition segments 33, used in an optional subsequent re-encoding process in block 34.
• the video stream is re-encoded in order to provide an synthesized video sequence 35 which may include part of the original video 20 enriched with photo(s) from the photo set 21 and the transitions between the original video and the inserted photo(s).
• the re- encoding process is optional in that it is conceivable that all the operations were performed in real-time while rendering the sequence.
• a description of the transformations and transitions that have to be applied to obtain the final video may be stored, in the form of playlist, multimedia presentation description, or code.
• an appropriate player may be arranged for interpreting the description and create the final sequence on-the- fly. Examples of languages and players that can be used are Macromedia Flash (proprietary) and SMIL (standard).
• FIG 3 a schematic drawing of an example of the steps performed in video transition synthesizing 32 of Figure 2.
• the video synthesizing 32 relates to any transition, viz. a transition between two still photos, a transition from a still photo to a video segment or a transition from a video segment to a still photo.
• step 40 the step of determining whether the transition is a transition from a photo to a photo. In the affirmative, the method proceeds to step 50, wherein a fading effect between the two still photos is performed, and the method ends subsequently. In the negative, the method continues to step 41, wherein it is determined whether the transition is a fade-out transition, viz.
• step 42 the estimated motion parameters obtained in block 25 as a result of step 22 of the method 100 in Figure 2 are rearranged.
• a transition that is neither a transition from one still photo to another nor from a video segment to a still photo must be a transition from a still photo to a video segment, and such a transition may be handled by mirroring the estimated motion parameters and treating the transition as a transition from a video segment to a still photo.
• the method continues from block 42 to 43; the method also continues from block 41 to block 43 in case of the result in block 41 being positive.
• three motion parameter curves for the sub frames are calculated.
• the three motion parameters viz. pan, tilt, zoom, need to correspond to the detected motion of the video segments surrounding the photo.
• the virtual camera motion may also depend on the regions of interest of the photo (not shown in the Figures).
• FIG. 4 shows an example of diagrams of the motion parameters pan, tilt and zoom.
• the points along the X-axis denote frames of a video in chronological order.
• frame a is prior to frame b, etc.
• Figure 4 shows an example where a customizable interval of motion values, from point a to b in the graphs in Figure 4 are the same as for the video frames.
• a linear decrease is applied, from point b to c, and in a later part, from point c to d, the values are set to 0 corresponding to a situation where a photo stands still.
• the example of Figure 4 corresponds to a situation where the estimated motion parameters for the video frames are followed in the beginning, slowing down smoothly, and stopped at the end.
• the calculation of the graph value at point b, for each curve, can be performed by weighted mean of the values in the interval [a,b] of the data, using the following formula where x represents the motion parameter (pan, tilt or zoom) and n is equal to b—a+1 :
• an additional zoom factor to be added to the estimated motion parameter, zoom is determined.
• the an additional zoom factor is calculated in order to ensure that no sub frame to be included in the virtual video sequence exceeds the boundaries of the still photo of which it is a part. For example, in the case where a high zoom factor is applied to a sub frame being close to the boundary of the still photo and a zoom-out is initiated, a situation might occur wherein sufficient information for all parts of the zoomed out sub frame does not exist. In order to avoid this situation, which might lead to unwanted black stripes appearing in the sub- photo frame, an algorithm for choosing appropriate sub-photo frames from the motion parameters is used.
• widthi ⁇ ) width ⁇ 1 SCaIe 1-1
• the sub-photo frame will represent a limited portion of the photo.
• a specific nominal value of translation corresponds to a lower absolute value. For example in the case of a sub- photo corresponding to a zoom factor of 1, a pan throughout the entire sub-photo will correspond to a pan through only half of the still photo of which the sub-photo frame is a part.
• Figure 5 shows diagrams of three zoom parameters, viz. an initial zoom parameter, zoom, a differential zoom parameter and the resulting zoom parameter, zoom', after adjustment.
• the zooms parameter are calculated backwards from point d to point b.
• the differential zoom parameter is calculated and applied in order to avoid any situation wherein a zoom-out could lead to a sub frame exceeding the boundaries of the still photo.
• Figure 5 shows a situation wherein a differential zoom parameter different from zero has been determined.
• a differential zoom factor would be necessary in order to avoid the situation wherein a necessary sub frame would exceed the boundaries of the still photo. This is indicated at the middle curve of Figure 5 as the apex at point e.
• the calculated differential zoom parameter should not be added in isolation at the frame e for which it is needed, since this concentrated value would create an abrupt change in the zoom factor curve.
• the differential zoom parameter is applied at the zoom factor curve at point f, which is half the way between the frame for which it is needed and the last frame, point d and point e in Figure 5.
• zoom' lower curve in Figure 5
• zoom' upper curve in Figure 5
• the differential zoom factor must be calculated and completely added to the zoom factor before the frame for which it is needed.
• the application of the resultant zoom parameter to the sub-frames between the points d and e corresponds to a change in the sub- frames of the still photo disclosed in the virtual video sequence.
• the sub frame is the same in all three graphs of Figure 5; however at point e and at the points between frame d and frame e, the sub frame are different after application of the differential zoom parameter compared with the sub frames to which the zoom parameters of the upper curve in Figure 5 were applied.
• step 47 a fading effect is effected between the video frames and the sub frames of the still photo.
• step 48 it is determined whether the transition is a fade-out transition, viz. a transition from a video segment to a photo.
• the method continues to step 49 wherein the frame order for the synthesized transition is inverted.
• the method ends.
• the method also ends.
• the method 32 in Figure 3 is an example of the method step 32 of the method 100 in Figure 2.
• the result of the method step 32 is a set of transition segments 33 which may be re-encoded to obtain a synthesized video sequence 35.
• the invention can be implemented in any suitable form including hardware, software, firmware or any combination of these.
• the invention or some features of the invention can be implemented as computer software running on one or more data processors and/or digital signal processors.
• the elements and components of an embodiment of the invention may be physically, functionally and logically implemented in any suitable way. Indeed, the functionality may be implemented in a single unit, in a plurality of units or as part of other functional units. As such, the invention may be implemented in a single unit, or may be physically and functionally distributed between different units and processors.

Landscapes

• Engineering & Computer Science (AREA)
• Multimedia (AREA)
• Television Signal Processing For Recording (AREA)
• Studio Devices (AREA)
• Studio Circuits (AREA)
• Processing Or Creating Images (AREA)

Priority Applications (4)

Applications Claiming Priority (2)

Publications (1)

Family

ID=41571358

Family Applications (1)

Country Status (6)

Families Citing this family (13)

Citations (1)

Family Cites Families (13)

• 2009
  • 2009-11-13 WO PCT/IB2009/055051 patent/WO2010058334A1/en active Application Filing
  • 2009-11-13 JP JP2011536977A patent/JP5451769B2/ja active Active
  • 2009-11-13 US US13/130,077 patent/US8649660B2/en active Active
  • 2009-11-13 KR KR1020117013924A patent/KR101588031B1/ko active IP Right Grant
  • 2009-11-13 CN CN200980146397.2A patent/CN102224545B/zh not_active Expired - Fee Related
  • 2009-11-13 EP EP09774739.8A patent/EP2359368B1/en active Active

Patent Citations (1)

Non-Patent Citations (5)

Also Published As

Similar Documents

Legal Events