USRE47421E1 - Apparatus, method and computer program product for generating a thumbnail representation of a video sequence - Google Patents

Apparatus, method and computer program product for generating a thumbnail representation of a video sequence Download PDF

Info

Publication number
USRE47421E1
USRE47421E1 US14/046,795 US201314046795A USRE47421E US RE47421 E1 USRE47421 E1 US RE47421E1 US 201314046795 A US201314046795 A US 201314046795A US RE47421 E USRE47421 E US RE47421E
Authority
US
United States
Prior art keywords
objects
video sequence
identified
frames
caused
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Expired - Fee Related, expires
Application number
US14/046,795
Inventor
Antonio Haro
Vidya Setlur
Tolga Capin
George Chen
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Convesant Wireless Licensing SA RL
Conversant Wireless Licensing SARL
Original Assignee
Convesant Wireless Licensing SA RL
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Convesant Wireless Licensing SA RL filed Critical Convesant Wireless Licensing SA RL
Priority to US14/046,795 priority Critical patent/USRE47421E1/en
Assigned to CONVERSANT WIRELESS LICENSING S.A R.L. reassignment CONVERSANT WIRELESS LICENSING S.A R.L. CHANGE OF NAME (SEE DOCUMENT FOR DETAILS). Assignors: CORE WIRELESS LICENSING S.A.R.L.
Assigned to CPPIB CREDIT INVESTMENTS, INC. reassignment CPPIB CREDIT INVESTMENTS, INC. AMENDED AND RESTATED U.S. PATENT SECURITY AGREEMENT (FOR NON-U.S. GRANTORS) Assignors: CONVERSANT WIRELESS LICENSING S.A R.L.
Application granted granted Critical
Publication of USRE47421E1 publication Critical patent/USRE47421E1/en
Assigned to CONVERSANT WIRELESS LICENSING S.A R.L. reassignment CONVERSANT WIRELESS LICENSING S.A R.L. RELEASE BY SECURED PARTY (SEE DOCUMENT FOR DETAILS). Assignors: CPPIB CREDIT INVESTMENTS INC.
Expired - Fee Related legal-status Critical Current
Adjusted expiration legal-status Critical

Links

Images

Classifications

    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/048Interaction techniques based on graphical user interfaces [GUI]
    • GPHYSICS
    • G11INFORMATION STORAGE
    • G11BINFORMATION STORAGE BASED ON RELATIVE MOVEMENT BETWEEN RECORD CARRIER AND TRANSDUCER
    • G11B27/00Editing; Indexing; Addressing; Timing or synchronising; Monitoring; Measuring tape travel
    • G11B27/10Indexing; Addressing; Timing or synchronising; Measuring tape travel
    • G11B27/102Programmed access in sequence to addressed parts of tracks of operating record carriers
    • G11B27/105Programmed access in sequence to addressed parts of tracks of operating record carriers of operating discs
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/70Information retrieval; Database structures therefor; File system structures therefor of video data
    • G06F16/73Querying
    • G06F16/738Presentation of query results
    • G06F16/739Presentation of query results in form of a video summary, e.g. the video summary being a video sequence, a composite still image or having synthesized frames
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/70Information retrieval; Database structures therefor; File system structures therefor of video data
    • G06F16/74Browsing; Visualisation therefor
    • G06F16/743Browsing; Visualisation therefor a collection of video files or sequences
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/70Information retrieval; Database structures therefor; File system structures therefor of video data
    • G06F16/78Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually
    • G06F16/783Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually using metadata automatically derived from the content
    • G06F16/7837Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually using metadata automatically derived from the content using objects detected or recognised in the video content
    • GPHYSICS
    • G11INFORMATION STORAGE
    • G11BINFORMATION STORAGE BASED ON RELATIVE MOVEMENT BETWEEN RECORD CARRIER AND TRANSDUCER
    • G11B27/00Editing; Indexing; Addressing; Timing or synchronising; Monitoring; Measuring tape travel
    • G11B27/10Indexing; Addressing; Timing or synchronising; Measuring tape travel
    • G11B27/19Indexing; Addressing; Timing or synchronising; Measuring tape travel by using information detectable on the record carrier
    • G11B27/28Indexing; Addressing; Timing or synchronising; Measuring tape travel by using information detectable on the record carrier by using information signals recorded by the same method as the main recording
    • GPHYSICS
    • G11INFORMATION STORAGE
    • G11BINFORMATION STORAGE BASED ON RELATIVE MOVEMENT BETWEEN RECORD CARRIER AND TRANSDUCER
    • G11B27/00Editing; Indexing; Addressing; Timing or synchronising; Monitoring; Measuring tape travel
    • G11B27/10Indexing; Addressing; Timing or synchronising; Measuring tape travel
    • G11B27/19Indexing; Addressing; Timing or synchronising; Measuring tape travel by using information detectable on the record carrier
    • G11B27/28Indexing; Addressing; Timing or synchronising; Measuring tape travel by using information detectable on the record carrier by using information signals recorded by the same method as the main recording
    • G11B27/32Indexing; Addressing; Timing or synchronising; Measuring tape travel by using information detectable on the record carrier by using information signals recorded by the same method as the main recording on separate auxiliary tracks of the same or an auxiliary record carrier
    • G11B27/327Table of contents
    • G11B27/329Table of contents on a disc [VTOC]
    • GPHYSICS
    • G11INFORMATION STORAGE
    • G11BINFORMATION STORAGE BASED ON RELATIVE MOVEMENT BETWEEN RECORD CARRIER AND TRANSDUCER
    • G11B27/00Editing; Indexing; Addressing; Timing or synchronising; Monitoring; Measuring tape travel
    • G11B27/10Indexing; Addressing; Timing or synchronising; Measuring tape travel
    • G11B27/34Indicating arrangements 
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04NPICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N5/00Details of television systems
    • H04N5/76Television signal recording
    • H04N5/765Interface circuits between an apparatus for recording and another apparatus
    • H04N5/77Interface circuits between an apparatus for recording and another apparatus between a recording apparatus and a television camera
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04NPICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N9/00Details of colour television systems
    • H04N9/79Processing of colour television signals in connection with recording
    • H04N9/80Transformation of the television signal for recording, e.g. modulation, frequency changing; Inverse transformation for playback
    • H04N9/82Transformation of the television signal for recording, e.g. modulation, frequency changing; Inverse transformation for playback the individual colour picture signal components being recorded simultaneously only
    • H04N9/8205Transformation of the television signal for recording, e.g. modulation, frequency changing; Inverse transformation for playback the individual colour picture signal components being recorded simultaneously only involving the multiplexing of an additional signal and the colour video signal
    • H04N9/8227Transformation of the television signal for recording, e.g. modulation, frequency changing; Inverse transformation for playback the individual colour picture signal components being recorded simultaneously only involving the multiplexing of an additional signal and the colour video signal the additional signal being at least another television signal

Definitions

  • the present invention generally relates to systems and methods of browsing video sequences and, more particularly, relates to systems and methods for generating thumbnail representations of video sequences to facilitate browsing of those sequences.
  • a number of software applications facilitating a user browsing cataloged or otherwise stored video sequences are capable of presenting those sequences in a number of different manners.
  • a number of software applications are capable of presenting the names of stored video sequences in a list from which the user may browse the listed video sequences.
  • Other software applications are capable of additionally or alternatively presenting thumbnail images representative of the video sequences, where each thumbnail representation corresponds to either a single frame or a collection of multiple frames.
  • thumbnail representations of a video sequence By presenting a thumbnail representation of a video sequence, these software applications facilitate the user identifying a desired video sequence from among a number of similar thumbnail representations of other video sequences. Further, a thumbnail representation may facilitate more efficient memory recall of video sequences since the user may more readily identify a desired video clip.
  • thumbnail representations that correspond to one or more frames of video sequences facilitates users browsing a number of video sequences
  • this manner of browsing video sequences has drawbacks.
  • conventional software applications that present thumbnail representations in this manner do so without regard to the extent the represented frame(s) may or may not be descriptive of the entire video sequence.
  • the respective thumbnail representations may not adequately identify those video sequences to the user.
  • exemplary embodiments of the present invention provide an improved apparatus, method and computer program product for generating an image (e.g., thumbnail) representation of a video sequence.
  • an image or thumbnail representation of a video sequence that comprises a composite of selected objects in one or more frames of the video sequence.
  • the objects of the composite may be selected based upon a ranking of those objects in relation to other objects in the frames of the video sequence.
  • exemplary embodiments of the present invention can generate an image representation that is semantically meaningful of the contents of the video sequence, particularly as compared to conventional thumbnail representations.
  • a method for generating an image representation of a video sequence, where the video sequences includes a plurality of frames, at least some of which include an image of a scene that may include one or more objects.
  • the method can include identifying a plurality of objects from the video sequence, and selecting at least some of the identified objects. Before identifying the objects, however, the video sequence may be decomposed such that one or more transitions in the video sequence are identified, where each transition includes at least one frame. In such instances, the plurality of objects may be identified from the identified transitions in the video sequence.
  • the method can also include extracting the selected objects from the frames of the video sequence including the respective objects, and combining the selected and extracted objects into an image representation of the video sequence.
  • the selected objects may be extracted such that at least one of the selected objects is extracted from a frame different from the frame from which another of the selected objects is extracted.
  • one or more of the identified objects may be altered to thereby denote a relative importance of the respective object(s) in the frames of the video sequence. In such instances, at least one of the altered objects may be combined with at least one other object into the image representation of the video sequence.
  • the method may include assigning a priority ranking to the identified objects, where the selecting step comprises selecting at least some of the identified objects based upon the assigned priority rankings of the respective objects.
  • the priority rankings can be assigned based upon one or more variables associated with each of the identified objects. Such variables may include, for example, the size of the object, the number of frames including the object, a motion characterization of the object, and/or the number of video transitions including the object.
  • the step of assigning priority rankings may include classifying at least some of the identified objects as a key object or a context object.
  • a object may be classified as a key object based upon the respective object being included in a measurable amount of motion within at least some of the frames of the video sequence, and/or being in the center of focus of at least some of the frames.
  • the identified objects may then be selected such that at least one of the selected objects comprises a key object.
  • an apparatus and computer program product are provided for generating an image representation of a video sequence.
  • Exemplary embodiments of the present invention therefore provide an improved system, method and computer program product for generating an image representation of a video sequence.
  • the system, method and computer program product of exemplary embodiments of the present invention may solve the problems identified by prior techniques and may provide additional advantages.
  • FIG. 1 is a block diagram a system for generating a thumbnail representation of a video sequence, in accordance with exemplary embodiments of the present invention
  • FIG. 2 is a schematic block diagram of a processing element of the system of FIG. 1 , in accordance with exemplary embodiments of the present invention
  • FIG. 3 is a flowchart illustrating various steps in a method of generating a thumbnail representation of a video sequence, in accordance with exemplary embodiments of the present invention
  • FIGS. 4a, 4b and 4c are three exemplary frames of a first video sequence, in accordance with exemplary embodiments of the present invention.
  • FIGS. 5a, 5b and 5c are identified objects extracted from the respective frames of FIGS. 4a, 4b and 4c , in accordance with exemplary embodiments of the present invention
  • FIGS. 6a and 6b are thumbnail representations of the first video sequence including the frames of FIGS. 4a, 4b and 4c , the representations generated in accordance with exemplary embodiments of the present invention
  • FIGS. 7a, 7b, 7c and 7d are four exemplary frames of a second video sequence, in accordance with exemplary embodiments of the present invention.
  • FIGS. 8a, 8b, 8c and 8d are identified objects extracted from the respective frames of FIGS. 7a, 7b, 7c and 7d , in accordance with exemplary embodiments of the present invention.
  • FIGS. 9a and 9b are thumbnail representations of the second video sequence including the frames of FIGS. 7a, 7b, 7c and 7d , the representations generated in accordance with exemplary embodiments of the present invention.
  • FIG. 1 an illustration of one system that would benefit from the present invention is provided.
  • the system, method and computer program product of exemplary embodiments of the present invention will be primarily described without respect to the environment within which the system, method and computer program product operate. It should be understood, however, that the system, method and computer program product may operate in a number of different environments, including mobile and/or fixed environments, wireline and/or wireless environments, standalone and/or networked environments or the like.
  • the system, method and computer program product of exemplary embodiments of the present invention can operate in mobile communication environments whereby mobile terminals operating within one or more mobile networks include or are otherwise in communication with one or more sources of video sequences.
  • the system 10 includes a video source 12 and a processing element 14 .
  • a single entity may support both the video source and processing element, logically separated but co-located within the respective entity.
  • a mobile terminal may support a logically separate, but co-located, video source and processing element.
  • the video source can comprise any of a number of different entities capable of providing one or more sequences of video.
  • the processing element can comprise any of a number of different entities capable of processing video sequences from the image source to automatically generate thumbnail image representations of those sequences, such as for facilitating a user browsing a library of video sequences, as explained below.
  • each sequence of video provided by the video source can include a plurality of frames, each of which comprises an image of a scene that may include one or more objects.
  • the image source can comprise, for example, an image capture device (e.g., video camera), a video cassette recorder (VCR), DVD player, a video file stored in memory, or the like.
  • the image source can be capable of providing one or more video sequences in a number of different formats including, for example, Third Generation Platform (3GP), AVI (Audio Video Interleave), Windows Media®, MPEG (Moving Pictures Expert Group), QuickTime®, RealVideo®, Shockwave® (Flash®) or the like.
  • the processing element can comprise, for example, a personal computing (PC) device such as a desktop or laptop computer, a server, a workstation, a media center device or other PC derivative, a personal video recorder, portable media consumption device (mobile terminal, personal digital assistant (PDA), gaming and/or media console, etc.), dedicated entertainment device, television, digital television set-top box, other consumer electronic device or the like.
  • PC personal computing
  • the processing element includes various means for performing one or more functions in accordance with exemplary embodiments of the present invention, including those more particularly shown and described herein.
  • the processing element may include alternative means for performing one or more like functions, without departing from the spirit and scope of the present invention. More particularly, for example, as shown in FIG. 2 , the processing element can include a processor 20 connected to a memory 22 .
  • the memory can comprise volatile and/or non-volatile memory, and typically stores content, data or the like.
  • the memory can store client applications, instructions or the like for the processor to perform steps associated with operation of the entity in accordance with exemplary embodiments of the present invention.
  • the memory can store one or more video sequences, such as those received from the video source 12 . And to facilitate browsing of one or more of those sequences, the memory can further store thumbnail representations of one or more respective sequences, where one or more of those thumbnail representations may include one or more objects identified or otherwise recognized from the respective sequences.
  • the client application(s), instructions or the like may comprise software operated by the processing element 14 . It should be understood, however, that any one or more of the client applications described herein can alternatively comprise firmware or hardware, without departing from the spirit and scope of the present invention.
  • the processing element can include one or more logic elements for performing various functions of one or more client application(s), instructions or the like. As will be appreciated, the logic elements can be embodied in any of a number of different manners. In this regard, the logic elements performing the functions of one or more client applications, instructions or the like can be embodied in an integrated circuit assembly including one or more integrated circuits integral or otherwise in communication with the processing element or more particularly, for example, the processor 20 of the processing element.
  • the design of integrated circuits is by and large a highly automated process.
  • complex and powerful software tools are available for converting a logic level design into a semiconductor circuit design ready to be etched and formed on a semiconductor substrate.
  • These software tools automatically route conductors and locate components on a semiconductor chip using well established rules of design as well as huge libraries of pre-stored design modules.
  • the resultant design in a standardized electronic format (e.g., Opus, GDSII, or the like) may be transmitted to a semiconductor fabrication facility or “fab” for fabrication.
  • the processor 20 can also be connected to at least one interface or other means for displaying, transmitting and/or receiving data, content or the like.
  • the interface(s) can include at least one communication interface 24 or other means for transmitting and/or receiving data, content or the like.
  • the interface(s) can also include at least one user interface that can include one or more earphones and/or speakers, a display 26 , and/or a user input interface 28 .
  • the user input interface can comprise any of a number of devices allowing the entity to receive data from a user, such as a microphone, a keypad, a touch display, a joystick, image capture device (e.g., digital camera) or other input device.
  • a microphone e.g., a microphone, a keypad, a touch display, a joystick, image capture device (e.g., digital camera) or other input device.
  • a number of current software applications facilitating a user browsing cataloged or otherwise stored video sequences are capable of presenting those sequences in a number of different manners, such as by presenting thumbnail images representative of the video sequences, where each thumbnail representation corresponds to either a single frame or a collection of multiple frames.
  • presenting thumbnail images representative of the video sequences where each thumbnail representation corresponds to either a single frame or a collection of multiple frames.
  • Exemplary embodiments of the present invention are therefore capable of generating a thumbnail representation that comprises a composite of one or more objects in one or more frames of the video sequence, where the object(s) may be selected based upon a ranking of those objects in relation to other objects in the frames of the video sequence.
  • exemplary embodiments of the present invention are capable of generating a thumbnail representation semantically meaningful of the contents of the video sequence, as compared to conventional thumbnail representations including the first or some randomly selected frame of the sequence.
  • FIG. 3 illustrates a flowchart including steps in a method of generating a thumbnail representation of a video sequence, in accordance with exemplary embodiments of the present invention.
  • the flowchart includes a number of steps for performing the functions of a processing element 14 in the method of generating a thumbnail.
  • the steps may be implemented by various means, such as hardware and/or firmware, including those described above.
  • the hardware and/or firmware may implement respective steps alone and/or under control of one or more computer program products.
  • such computer program product(s) can include at least one computer-readable storage medium (e.g., memory 22 ) and software including computer-readable program code portions, such as a series of computer instructions, embodied in the computer-readable storage medium.
  • the illustrated method of generating a thumbnail representation of a video sequence includes the processing element 14 , or more particularly for example a client application of the processing element, receiving or otherwise being provided with a video sequence from the video source 12 .
  • the video sequence can include a plurality of frames, each of which comprises an image of a scene that may include one or more objects.
  • the processing element is capable of decomposing the frames of the video sequence to identify one or more three-dimensional (spatial and temporal) planes of texturally similar regions, as shown in block 32 .
  • the processing element 14 can first identify one or more transitions in the video sequence.
  • a video transition such as a scene change generally separates parts of the video sequence and may itself include a number of frames of the sequence. Transitions can be identified in any of a number of different manners, such as based on pair-wise image comparisons in a manner similar to that disclosed in S. Uchihashi et al., Video Manga: Generating Semantically Meaningful Video Summaries, in MULTIMEDIA '99: P ROC. OF THE S EVENTH sACM INT'L C ONF.
  • the processing element can perform motion analysis to measure the motion of each pixel of each frame in the respective transitions, from which the processing element can perform a graph cut analysis in the spatial and temporal dimensions across the respective transitions to form three-dimensional planes segmenting the transition into one or more three-dimensional (spatial and temporal) planes of texturally similar regions.
  • the processing element 14 can thereafter identify one or more objects in one or more of the video transitions based upon the measured motion and identified planes of texturally similar regions, as shown in block 34 .
  • the identified texturally similar regions can be clustered based on textural similarity to acquire clusters of textured regions (e.g. outlined strawberries). Redundant textured regions can then be removed to leave only single instances, such as based on size (e.g., out of all the regions texturally similar to a detected strawberry, keep only the largest one).
  • the processing element may process the identified objects to identify and remove any duplicate objects, thereby leaving visually and semantically distinct objects from which a composite may be generated, as explained below.
  • the identified objects can be processed in this regard in accordance with any of a number of different techniques, such as by performing a cluster analysis. For more information on such cluster analysis techniques, see Duda et al., P ATTERN C LASSIFICATION (2001); and Gersho & Gray, V ECTOR Q UANTIZATION AND S IGNAL C OMPRESSION (1992), the content of both of which are hereby incorporated by reference in their entireties.
  • the processing element 14 can thereafter rank or otherwise assign a priority ranking to the identified objects, and select a predetermined number of the objects based on the priority rankings, such as by selecting a predetermined number of the highest ranked objects, as shown in block 36 .
  • the priority rankings can be assigned to the identified objects in accordance with any of a number of different algorithms based upon any of a number of different variables.
  • the variables of an object from which the processing element can assign a ranking to the object can include, for example, the size of the object, the number of frames including the object, a motion characterization of the object (e.g.
  • objects can be ranked by area, number of objects clustered about a particular cluster center (e.g., before removing redundant objects—based on number of times it appeared in the video), periodicity of motion, or the like.
  • the algorithm can then sort the identified objects based on their rankings (e.g., highest ranked to lowest ranked) and keep only N objects (e.g., N highest), where N can be pre-specified (e.g., 2-3).
  • the processing element 14 can compose an image (e.g., thumbnail) representation of the video sequence based upon the selected objects, such as by extracting one or more of the selected objects from their respective frames of the video sequence and forming a composite image of the extracted objects, as shown in block 38 .
  • the composite image can be formed by combining one or more of the selected objects in an aesthetically pleasing and visually distinctive manner to maximize user recall of the video's content and to automatically maximize differentiation between similar representations of other video sequences.
  • the processing element can compose the thumbnail representation in any of a number of different manners, such as based upon one or more artistic techniques for generating digital montages as briefly explained below.
  • the processing element 14 can composite the selected objects either directly (photorealistic), or in an abstracted manner by reducing the color space of one or more of the objects and then compositing the objects (non-photorealistic).
  • many compositing techniques utilized by the processing element can be derived from how the human visual system perceives and processes visual information.
  • the processing element can apply a number of rules to alter one or more objects to thereby denote the relative importance of those objects in the frames of the video sequence.
  • the processing element can lighten one or more-important objects (key objects that may correspond, e.g., to the higher ranked objects), and darken one or more less-important objects (context objects that may correspond, e.g., to the lower ranked objects) to guide the user to the more-important objects in the composite.
  • the processing element can use selective softening and sharpening image filtering techniques to guide the user's line of focus.
  • the processing element can add more detail and/or texture to one or more important objects to attract the user's attention to those objects.
  • the processing element can add color to one or more-important objects, and/or reduce the color of one or more less-important objects.
  • the processing element can increase the size of one or more-important objects, and/or reduce the size of one or more less-important objects.
  • the processing element 14 can be configured to observe a number of general rules to compose a more aesthetically pleasing representation of the video sequence.
  • the processing element can reduce alignment artifacts by matching and/or aligning the field of view of two adjacent objects.
  • the processing element may determine two-point and/or three-point perspective based upon the objects themselves or through information from the video source 12 . From the determined perspective, then, the processing element can align objects by performing operations such as object rotation operations, scale operations and/or more complex image-based rendering operations.
  • the processing element 14 can compose the thumbnail representation such that foreground objects are larger than background objects, and/or such that the scale of the objects is consistent throughout the thumbnail representation, the front object of any overlapping object thereby being intended to be closer in perspective as the human visual system may infer as much.
  • the processing element can compose the thumbnail representation to maintain consistency in color hues and contrast in the included objects to reduce “patchiness.”
  • the processing element can, for example, compose the thumbnail representation to harmonize any shadows and/or reflections associated with the objects as it may be desirable for any shadows and reflections accompanying any extracted objects to be consistent in the environment (i.e., the thumbnail representation) into which those objects are composited.
  • the processing element can perform one or more additional operations on one or more of the extracted objects to enhance the thumbnail representation, such as by performing an anti-aliasing operation on one or more segmented edges of one or more objects to increase visual coherence in the thumbnail representation.
  • additional operations may include the use of motion lines in the icons, cartoonized objects and other placement rules.
  • the thumbnail representation can be saved and associated with the respective video sequence for use in an appropriate application domain.
  • the processing element may be configured to compose a number of alternative thumbnail representations such that the user may optionally select a representation from a set of alternative representations to associate with the video sequence. Irrespective, the user may thereafter be capable of more quickly identifying a desired video sequence based upon its associated thumbnail representation, such as by browsing a number of thumbnail representations of a number of video sequences and selecting the thumbnail of the desired video.
  • exemplary embodiments of the present invention may also be used by online services, search engines or the like to enable users (of the processing element 14 ) to search for video sequences local to and/or remote from (e.g., across the Internet from) the respective services, search engines or the like.
  • the processing element 14 can identify one or more objects in the three identified transitions (see block 34 ).
  • the processing element may identify a plant, an automobile and a building, as shown extracted from their respective frames in FIGS. 5a, 5b and 5c .
  • the processing element can compose an image (e.g., thumbnail) representation of the video sequence based upon the selected objects.
  • the processing element can extract the selected objects from their respective frames of the video sequence and form a composite image of the extracted objects (see block 38 ).
  • the processing element can extract the plant and automobile from their respective frames, as shown in FIGS. 5a and 5b , and form a composite image of the extracted objects, such as shown in FIG. 6a .
  • the processing element selects the identified automobile and building.
  • the processing element can extract the automobile and building from their respective frames, as shown in FIGS. 5b and 5c , and form a composite image of the extracted objects, such as shown in FIG. 6b .
  • the processing element 14 can identify one or more objects in the four identified transitions (see block 34 ).
  • the processing element may identify a skater in a first pose, a skateboard, the skater and a second pose, and the skater in a third pose, as shown extracted from their respective frames in FIGS. 8a, 8b, 8c and 8d .
  • the processing element 14 can rank or otherwise assign a priority ranking to the identified objects, and select a predetermined number of the objects based on the priority rankings (see block 36 ).
  • ranking one or more identified objects may include classifying those objects as key objects or context objects, where the key objects represent the main actors in the story (e.g., the skater) and the context objects provide scene information (e.g., skateboard).
  • key objects may be classified as such based upon on their role in the video sequence, which may be evidenced for example by being involved in an appreciable or at least a measurable amount of motion, and/or by being in the center of focus of a number of frames.
  • the remaining objects not otherwise classified as key objects then, may be classified as context objects. It should be understood, however, that key objects of one or more transitions may be classified as context objects in one or more other transitions, and vice versa.
  • the processing element can compose an image (e.g., thumbnail) representation of the video sequence based upon the selected objects.
  • the processing element can extract the selected objects from their respective frames of the video sequence and form a composite image of the extracted objects (see block 38 ).
  • the objects may be selected such that the composed image includes at least one key object, which may be increased in size over other included objects (key or context objects) in the image.
  • key object which may be increased in size over other included objects (key or context objects) in the image.
  • the processing element can extract the skater in the second pose and the skater in the third pose from their respective frames, as shown in FIGS. 8c and 8d , and form a composite image of the extracted objects, such as shown in FIG. 9b with the second key object being increased in size of the first key object.
  • the processing element 14 can extract selected objects from their respective frames, and combine those objects into the image representation of the video sequence.
  • the processing element may extract one or more, if not all, identified objects from their respective frames of the video sequence at any point after the processing element identifies the respective frames.
  • the identified objects may be extracted from their respective frames after being identified but before being ranked and/or selected (see block 36 ).
  • the image representation of a video sequence generated in accordance with exemplary embodiments of the present invention may comprise or otherwise be considered a thumbnail image, which may be small or otherwise miniaturized in nature.
  • the image representation generated in accordance with exemplary embodiments of the present invention may be small or otherwise of a size classifying the image representation as a thumbnail. It should be understood, however, that the image representation may comprise any of a number of different sizes, some of which may be larger than that required to classify the image representation as a thumbnail.
  • the functions performed by one or more of the entities of the system may be performed by various means, such as hardware and/or firmware, including those described above, alone and/or under control of a computer program product (client application).
  • the computer program product for performing one or more functions of exemplary embodiments of the present invention includes a computer-readable storage medium, such as the non-volatile storage medium, and software including computer-readable program code portions, such as a series of computer instructions, embodied in the computer-readable storage medium.
  • FIG. 3 is a flowchart of methods, systems and program products according to the invention. It will be understood that each block or step of the flowchart, and combinations of blocks in the flowchart, can be implemented by various means, such as hardware, firmware, and/or software including one or more computer program instructions. As will be appreciated, any such computer program instructions may be loaded onto a computer or other programmable apparatus (i.e., hardware) to produce a machine, such that the instructions which execute on the computer or other programmable apparatus create means for implementing the functions specified in the flowchart's block(s) or step(s).
  • a computer or other programmable apparatus i.e., hardware
  • These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means which implement the function specified in the flowchart's block(s) or step(s).
  • the computer program instructions may also be loaded onto a computer or other programmable apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart's block(s) or step(s).
  • blocks or steps of the flowchart support combinations of means for performing the specified functions, combinations of steps for performing the specified functions and program instruction means for performing the specified functions. It will also be understood that one or more blocks or steps of the flowchart, and combinations of blocks or steps in the flowchart, can be implemented by special purpose hardware-based computer systems which perform the specified functions or steps, or combinations of special purpose hardware and computer instructions.

Landscapes

  • Engineering & Computer Science (AREA)
  • Multimedia (AREA)
  • Theoretical Computer Science (AREA)
  • General Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Data Mining & Analysis (AREA)
  • Databases & Information Systems (AREA)
  • Signal Processing (AREA)
  • Library & Information Science (AREA)
  • Human Computer Interaction (AREA)
  • Computational Linguistics (AREA)
  • Television Signal Processing For Recording (AREA)

Abstract

A system, method and computer program product are provided for generating an image, such as a thumbnail, representation of a video sequence, where the video sequences includes a plurality of frames, at least some of which include an image of a scene that may include one or more objects. The method can include identifying a plurality of objects from the video sequence, and selecting at least some of the identified objects. In addition, the method can include extracting the selected objects from the frames of the video sequence including the respective objects, and combining the selected and extracted objects into an image representation of the video sequence.

Description

FIELD OF THE INVENTION
The present invention generally relates to systems and methods of browsing video sequences and, more particularly, relates to systems and methods for generating thumbnail representations of video sequences to facilitate browsing of those sequences.
BACKGROUND OF THE INVENTION
As mobile data storage increases and camera-imaging quality improves, users are increasingly capturing and sharing video with their mobile devices. One major drawback of the increasing use of video, however, arises while browsing a graphical user interface for a desired video clip or sequence. Currently, a number of software applications facilitating a user browsing cataloged or otherwise stored video sequences are capable of presenting those sequences in a number of different manners. For example, a number of software applications are capable of presenting the names of stored video sequences in a list from which the user may browse the listed video sequences. Other software applications are capable of additionally or alternatively presenting thumbnail images representative of the video sequences, where each thumbnail representation corresponds to either a single frame or a collection of multiple frames. By presenting a thumbnail representation of a video sequence, these software applications facilitate the user identifying a desired video sequence from among a number of similar thumbnail representations of other video sequences. Further, a thumbnail representation may facilitate more efficient memory recall of video sequences since the user may more readily identify a desired video clip.
Although presenting thumbnail representations that correspond to one or more frames of video sequences facilitates users browsing a number of video sequences, this manner of browsing video sequences has drawbacks. In this regard, conventional software applications that present thumbnail representations in this manner do so without regard to the extent the represented frame(s) may or may not be descriptive of the entire video sequence. Thus, for example, for represented frame(s) that do not have a meaningful relation to the remainder of the video sequence, or for the first frame(s) of a video sequence with a number of significant cut frames, the respective thumbnail representations may not adequately identify those video sequences to the user.
SUMMARY OF THE INVENTION
In light of the foregoing background, exemplary embodiments of the present invention provide an improved apparatus, method and computer program product for generating an image (e.g., thumbnail) representation of a video sequence. In accordance with exemplary embodiments of the present invention, an image or thumbnail representation of a video sequence that comprises a composite of selected objects in one or more frames of the video sequence. In this regard, the objects of the composite may be selected based upon a ranking of those objects in relation to other objects in the frames of the video sequence. Thus, exemplary embodiments of the present invention can generate an image representation that is semantically meaningful of the contents of the video sequence, particularly as compared to conventional thumbnail representations.
According to one aspect of the present invention, a method is provided for generating an image representation of a video sequence, where the video sequences includes a plurality of frames, at least some of which include an image of a scene that may include one or more objects. The method can include identifying a plurality of objects from the video sequence, and selecting at least some of the identified objects. Before identifying the objects, however, the video sequence may be decomposed such that one or more transitions in the video sequence are identified, where each transition includes at least one frame. In such instances, the plurality of objects may be identified from the identified transitions in the video sequence.
The method can also include extracting the selected objects from the frames of the video sequence including the respective objects, and combining the selected and extracted objects into an image representation of the video sequence. In this regard, the selected objects may be extracted such that at least one of the selected objects is extracted from a frame different from the frame from which another of the selected objects is extracted. Further, if so desired, one or more of the identified objects may be altered to thereby denote a relative importance of the respective object(s) in the frames of the video sequence. In such instances, at least one of the altered objects may be combined with at least one other object into the image representation of the video sequence.
Further, the method may include assigning a priority ranking to the identified objects, where the selecting step comprises selecting at least some of the identified objects based upon the assigned priority rankings of the respective objects. In such instances, the priority rankings can be assigned based upon one or more variables associated with each of the identified objects. Such variables may include, for example, the size of the object, the number of frames including the object, a motion characterization of the object, and/or the number of video transitions including the object. Additionally or alternatively, the step of assigning priority rankings may include classifying at least some of the identified objects as a key object or a context object. A object may be classified as a key object based upon the respective object being included in a measurable amount of motion within at least some of the frames of the video sequence, and/or being in the center of focus of at least some of the frames. The identified objects may then be selected such that at least one of the selected objects comprises a key object.
According to other aspects of the present invention, an apparatus and computer program product are provided for generating an image representation of a video sequence. Exemplary embodiments of the present invention therefore provide an improved system, method and computer program product for generating an image representation of a video sequence. And as indicated above and explained in greater detail below, the system, method and computer program product of exemplary embodiments of the present invention may solve the problems identified by prior techniques and may provide additional advantages.
BRIEF DESCRIPTION OF THE DRAWINGS
Having thus described the invention in general terms, reference will now be made to the accompanying drawings, which are not necessarily drawn to scale, and wherein:
FIG. 1 is a block diagram a system for generating a thumbnail representation of a video sequence, in accordance with exemplary embodiments of the present invention;
FIG. 2 is a schematic block diagram of a processing element of the system of FIG. 1, in accordance with exemplary embodiments of the present invention;
FIG. 3 is a flowchart illustrating various steps in a method of generating a thumbnail representation of a video sequence, in accordance with exemplary embodiments of the present invention;
FIGS. 4a, 4b and 4c are three exemplary frames of a first video sequence, in accordance with exemplary embodiments of the present invention;
FIGS. 5a, 5b and 5c are identified objects extracted from the respective frames of FIGS. 4a, 4b and 4c, in accordance with exemplary embodiments of the present invention;
FIGS. 6a and 6b are thumbnail representations of the first video sequence including the frames of FIGS. 4a, 4b and 4c, the representations generated in accordance with exemplary embodiments of the present invention;
FIGS. 7a, 7b, 7c and 7d are four exemplary frames of a second video sequence, in accordance with exemplary embodiments of the present invention;
FIGS. 8a, 8b, 8c and 8d are identified objects extracted from the respective frames of FIGS. 7a, 7b, 7c and 7d, in accordance with exemplary embodiments of the present invention; and
FIGS. 9a and 9b are thumbnail representations of the second video sequence including the frames of FIGS. 7a, 7b, 7c and 7d, the representations generated in accordance with exemplary embodiments of the present invention.
DETAILED DESCRIPTION OF THE INVENTION
The present invention now will be described more fully hereinafter with reference to the accompanying drawings, in which preferred exemplary embodiments of the invention are shown. This invention may, however, be embodied in many different forms and should not be construed as limited to the exemplary embodiments set forth herein; rather, these exemplary embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the invention to those skilled in the art. Like numbers refer to like elements throughout.
Referring to FIG. 1, an illustration of one system that would benefit from the present invention is provided. The system, method and computer program product of exemplary embodiments of the present invention will be primarily described without respect to the environment within which the system, method and computer program product operate. It should be understood, however, that the system, method and computer program product may operate in a number of different environments, including mobile and/or fixed environments, wireline and/or wireless environments, standalone and/or networked environments or the like. For example, the system, method and computer program product of exemplary embodiments of the present invention can operate in mobile communication environments whereby mobile terminals operating within one or more mobile networks include or are otherwise in communication with one or more sources of video sequences.
The system 10 includes a video source 12 and a processing element 14. Although shown as separate entities, it should be understood that in some embodiments, a single entity may support both the video source and processing element, logically separated but co-located within the respective entity. For example, a mobile terminal may support a logically separate, but co-located, video source and processing element. Irrespective of the manner of implementing the system, however, the video source can comprise any of a number of different entities capable of providing one or more sequences of video. Like the image source, the processing element can comprise any of a number of different entities capable of processing video sequences from the image source to automatically generate thumbnail image representations of those sequences, such as for facilitating a user browsing a library of video sequences, as explained below. In this regard, each sequence of video provided by the video source can include a plurality of frames, each of which comprises an image of a scene that may include one or more objects. The image source can comprise, for example, an image capture device (e.g., video camera), a video cassette recorder (VCR), DVD player, a video file stored in memory, or the like. In this regard, the image source can be capable of providing one or more video sequences in a number of different formats including, for example, Third Generation Platform (3GP), AVI (Audio Video Interleave), Windows Media®, MPEG (Moving Pictures Expert Group), QuickTime®, RealVideo®, Shockwave® (Flash®) or the like.
Referring now to FIG. 2, a block diagram of an entity capable of operating as a processing element 14 is shown in accordance with one exemplary embodiment of the present invention. As shown and described herein, the processing element can comprise, for example, a personal computing (PC) device such as a desktop or laptop computer, a server, a workstation, a media center device or other PC derivative, a personal video recorder, portable media consumption device (mobile terminal, personal digital assistant (PDA), gaming and/or media console, etc.), dedicated entertainment device, television, digital television set-top box, other consumer electronic device or the like. As shown, the processing element includes various means for performing one or more functions in accordance with exemplary embodiments of the present invention, including those more particularly shown and described herein. It should be understood, however, that the processing element may include alternative means for performing one or more like functions, without departing from the spirit and scope of the present invention. More particularly, for example, as shown in FIG. 2, the processing element can include a processor 20 connected to a memory 22. The memory can comprise volatile and/or non-volatile memory, and typically stores content, data or the like. For example, the memory can store client applications, instructions or the like for the processor to perform steps associated with operation of the entity in accordance with exemplary embodiments of the present invention. Also, for example, the memory can store one or more video sequences, such as those received from the video source 12. And to facilitate browsing of one or more of those sequences, the memory can further store thumbnail representations of one or more respective sequences, where one or more of those thumbnail representations may include one or more objects identified or otherwise recognized from the respective sequences.
As described herein, the client application(s), instructions or the like may comprise software operated by the processing element 14. It should be understood, however, that any one or more of the client applications described herein can alternatively comprise firmware or hardware, without departing from the spirit and scope of the present invention. Generally, then, the processing element can include one or more logic elements for performing various functions of one or more client application(s), instructions or the like. As will be appreciated, the logic elements can be embodied in any of a number of different manners. In this regard, the logic elements performing the functions of one or more client applications, instructions or the like can be embodied in an integrated circuit assembly including one or more integrated circuits integral or otherwise in communication with the processing element or more particularly, for example, the processor 20 of the processing element. The design of integrated circuits is by and large a highly automated process. In this regard, complex and powerful software tools are available for converting a logic level design into a semiconductor circuit design ready to be etched and formed on a semiconductor substrate. These software tools automatically route conductors and locate components on a semiconductor chip using well established rules of design as well as huge libraries of pre-stored design modules. Once the design for a semiconductor circuit has been completed, the resultant design, in a standardized electronic format (e.g., Opus, GDSII, or the like) may be transmitted to a semiconductor fabrication facility or “fab” for fabrication.
In addition to the memory 22, the processor 20 can also be connected to at least one interface or other means for displaying, transmitting and/or receiving data, content or the like. In this regard, the interface(s) can include at least one communication interface 24 or other means for transmitting and/or receiving data, content or the like. In addition to the communication interface(s), the interface(s) can also include at least one user interface that can include one or more earphones and/or speakers, a display 26, and/or a user input interface 28. The user input interface, in turn, can comprise any of a number of devices allowing the entity to receive data from a user, such as a microphone, a keypad, a touch display, a joystick, image capture device (e.g., digital camera) or other input device.
As explained in the background section, a number of current software applications facilitating a user browsing cataloged or otherwise stored video sequences are capable of presenting those sequences in a number of different manners, such as by presenting thumbnail images representative of the video sequences, where each thumbnail representation corresponds to either a single frame or a collection of multiple frames. And although such a technique for facilitating browsing of video sequences facilitates a user identifying a desired video sequence, it has a drawback at least insofar as the represented frame(s) are not very descriptive of the entire video sequence. Exemplary embodiments of the present invention are therefore capable of generating a thumbnail representation that comprises a composite of one or more objects in one or more frames of the video sequence, where the object(s) may be selected based upon a ranking of those objects in relation to other objects in the frames of the video sequence. Thus, exemplary embodiments of the present invention are capable of generating a thumbnail representation semantically meaningful of the contents of the video sequence, as compared to conventional thumbnail representations including the first or some randomly selected frame of the sequence.
Reference is now made to FIG. 3, which illustrates a flowchart including steps in a method of generating a thumbnail representation of a video sequence, in accordance with exemplary embodiments of the present invention. As shown, the flowchart includes a number of steps for performing the functions of a processing element 14 in the method of generating a thumbnail. It should be understood that the steps may be implemented by various means, such as hardware and/or firmware, including those described above. In such instances, the hardware and/or firmware may implement respective steps alone and/or under control of one or more computer program products. In this regard, such computer program product(s) can include at least one computer-readable storage medium (e.g., memory 22) and software including computer-readable program code portions, such as a series of computer instructions, embodied in the computer-readable storage medium.
As shown in block 30, the illustrated method of generating a thumbnail representation of a video sequence includes the processing element 14, or more particularly for example a client application of the processing element, receiving or otherwise being provided with a video sequence from the video source 12. As indicated above, the video sequence can include a plurality of frames, each of which comprises an image of a scene that may include one or more objects. After receiving the video sequence, the processing element is capable of decomposing the frames of the video sequence to identify one or more three-dimensional (spatial and temporal) planes of texturally similar regions, as shown in block 32. For a brief description of one technique suitable for decomposing the video sequence, see Kwatra et al., Graphcut Textures: Image and Video Synthesis Using Graph Cuts, ACM TRANSACTIONS ON GRAPHICS (TOG), July 2003, at 277-286, the contents of which are hereby incorporated by reference in its entirety.
In accordance with one technique for decomposing the video sequence, the processing element 14 can first identify one or more transitions in the video sequence. In this regard, a video transition such as a scene change generally separates parts of the video sequence and may itself include a number of frames of the sequence. Transitions can be identified in any of a number of different manners, such as based on pair-wise image comparisons in a manner similar to that disclosed in S. Uchihashi et al., Video Manga: Generating Semantically Meaningful Video Summaries, in MULTIMEDIA '99: PROC. OF THE SEVENTH sACM INT'L CONF. ON MULTIMEDIA (PART 1), at 383-392, the content of which is hereby incorporated by reference in its entirety. For one or more of the identified transitions, then, the processing element can perform motion analysis to measure the motion of each pixel of each frame in the respective transitions, from which the processing element can perform a graph cut analysis in the spatial and temporal dimensions across the respective transitions to form three-dimensional planes segmenting the transition into one or more three-dimensional (spatial and temporal) planes of texturally similar regions.
Irrespective of exactly how the video sequence is decomposed, the processing element 14 can thereafter identify one or more objects in one or more of the video transitions based upon the measured motion and identified planes of texturally similar regions, as shown in block 34. In this regard, the identified texturally similar regions can be clustered based on textural similarity to acquire clusters of textured regions (e.g. outlined strawberries). Redundant textured regions can then be removed to leave only single instances, such as based on size (e.g., out of all the regions texturally similar to a detected strawberry, keep only the largest one). Further, if so desired, the processing element may process the identified objects to identify and remove any duplicate objects, thereby leaving visually and semantically distinct objects from which a composite may be generated, as explained below. The identified objects can be processed in this regard in accordance with any of a number of different techniques, such as by performing a cluster analysis. For more information on such cluster analysis techniques, see Duda et al., PATTERN CLASSIFICATION (2001); and Gersho & Gray, VECTOR QUANTIZATION AND SIGNAL COMPRESSION (1992), the content of both of which are hereby incorporated by reference in their entireties.
After and irrespective of exactly how one or more objects in one or more of the video transitions are identified, the processing element 14 can thereafter rank or otherwise assign a priority ranking to the identified objects, and select a predetermined number of the objects based on the priority rankings, such as by selecting a predetermined number of the highest ranked objects, as shown in block 36. In this regard, the priority rankings can be assigned to the identified objects in accordance with any of a number of different algorithms based upon any of a number of different variables. The variables of an object from which the processing element can assign a ranking to the object can include, for example, the size of the object, the number of frames including the object, a motion characterization of the object (e.g. linear, periodic, etc.), and/or number of video transitions including the object (calculated, e.g., during the duplicate removal step). In accordance with one exemplary algorithm, then, objects can be ranked by area, number of objects clustered about a particular cluster center (e.g., before removing redundant objects—based on number of times it appeared in the video), periodicity of motion, or the like. The algorithm can then sort the identified objects based on their rankings (e.g., highest ranked to lowest ranked) and keep only N objects (e.g., N highest), where N can be pre-specified (e.g., 2-3).
After selecting a predetermined number of the objects, the processing element 14 can compose an image (e.g., thumbnail) representation of the video sequence based upon the selected objects, such as by extracting one or more of the selected objects from their respective frames of the video sequence and forming a composite image of the extracted objects, as shown in block 38. The composite image can be formed by combining one or more of the selected objects in an aesthetically pleasing and visually distinctive manner to maximize user recall of the video's content and to automatically maximize differentiation between similar representations of other video sequences. The processing element can compose the thumbnail representation in any of a number of different manners, such as based upon one or more artistic techniques for generating digital montages as briefly explained below. For more information on a number of such techniques, see Eismann, PHOTOSHOP: MASKING AND COMPOSITING (2004); and Knight & Glaser, THE GRAPHIC DESIGNER'S GUIDE To EFFECTIVE VISUAL COMMUNICATION: CREATING HIERARCHIES WITH TYPE, IMAGE, AND COLOR (2005), the content of both of which are hereby incorporated by reference in their entireties.
The processing element 14 can composite the selected objects either directly (photorealistic), or in an abstracted manner by reducing the color space of one or more of the objects and then compositing the objects (non-photorealistic). In this regard, many compositing techniques utilized by the processing element can be derived from how the human visual system perceives and processes visual information. Thus, before, after or as the processing element assigns priority rankings to the objects, the processing element can apply a number of rules to alter one or more objects to thereby denote the relative importance of those objects in the frames of the video sequence. For example, as the human eye may move from dark images to light images, the processing element can lighten one or more-important objects (key objects that may correspond, e.g., to the higher ranked objects), and darken one or more less-important objects (context objects that may correspond, e.g., to the lower ranked objects) to guide the user to the more-important objects in the composite. Also, for example, as the human eye may move from soft images to sharp images, the processing element can use selective softening and sharpening image filtering techniques to guide the user's line of focus. In addition, for example, as the human eye may move away from less detailed images to more detailed images (or portions thereof), the processing element can add more detail and/or texture to one or more important objects to attract the user's attention to those objects. Further, for example, as the human eye may be attracted to color images, the processing element can add color to one or more-important objects, and/or reduce the color of one or more less-important objects. And in yet another example, as the human visual system may perceive larger objects as being more important, the processing element can increase the size of one or more-important objects, and/or reduce the size of one or more less-important objects.
Further, for example, as the processing element 14 performs either photorealistic and non-photo realistic compositing, the processing element can be configured to observe a number of general rules to compose a more aesthetically pleasing representation of the video sequence. For example, the processing element can reduce alignment artifacts by matching and/or aligning the field of view of two adjacent objects. In this regard, the processing element may determine two-point and/or three-point perspective based upon the objects themselves or through information from the video source 12. From the determined perspective, then, the processing element can align objects by performing operations such as object rotation operations, scale operations and/or more complex image-based rendering operations.
Also for example, the processing element 14 can compose the thumbnail representation such that foreground objects are larger than background objects, and/or such that the scale of the objects is consistent throughout the thumbnail representation, the front object of any overlapping object thereby being intended to be closer in perspective as the human visual system may infer as much. In addition, for example, the processing element can compose the thumbnail representation to maintain consistency in color hues and contrast in the included objects to reduce “patchiness.” The processing element can, for example, compose the thumbnail representation to harmonize any shadows and/or reflections associated with the objects as it may be desirable for any shadows and reflections accompanying any extracted objects to be consistent in the environment (i.e., the thumbnail representation) into which those objects are composited. Further, for example, the processing element can perform one or more additional operations on one or more of the extracted objects to enhance the thumbnail representation, such as by performing an anti-aliasing operation on one or more segmented edges of one or more objects to increase visual coherence in the thumbnail representation. Other implementations may include the use of motion lines in the icons, cartoonized objects and other placement rules.
After the thumbnail representation is composed, the thumbnail representation can be saved and associated with the respective video sequence for use in an appropriate application domain. In this regard, the processing element may be configured to compose a number of alternative thumbnail representations such that the user may optionally select a representation from a set of alternative representations to associate with the video sequence. Irrespective, the user may thereafter be capable of more quickly identifying a desired video sequence based upon its associated thumbnail representation, such as by browsing a number of thumbnail representations of a number of video sequences and selecting the thumbnail of the desired video. It should also be noted that exemplary embodiments of the present invention may also be used by online services, search engines or the like to enable users (of the processing element 14) to search for video sequences local to and/or remote from (e.g., across the Internet from) the respective services, search engines or the like.
To illustrate various benefits of exemplary embodiments of the present invention, consider a video sequence including the three frames of FIGS. 4a, 4b and 4c, those frames being within three identified transitions in the video sequence. In such an instance, after the processing element 14 receives and decomposes the video sequence (see FIG. 3, blocks 30, 32), the processing element can identify one or more objects in the three identified transitions (see block 34). In first, second and third transitions including the respective frames of FIGS. 4a, 4b and 4c, for example, the processing element may identify a plant, an automobile and a building, as shown extracted from their respective frames in FIGS. 5a, 5b and 5c. After ranking the identified objects and selecting one or more of the identified objects (see block 36), the processing element can compose an image (e.g., thumbnail) representation of the video sequence based upon the selected objects. In this regard, the processing element can extract the selected objects from their respective frames of the video sequence and form a composite image of the extracted objects (see block 38). Consider, for example, that the identified plant and automobile are selected. In such an instance, the processing element can extract the plant and automobile from their respective frames, as shown in FIGS. 5a and 5b, and form a composite image of the extracted objects, such as shown in FIG. 6a. Alternatively, consider, for example, that the processing element selects the identified automobile and building. In that instance, the processing element can extract the automobile and building from their respective frames, as shown in FIGS. 5b and 5c, and form a composite image of the extracted objects, such as shown in FIG. 6b.
To further illustrate various benefits of exemplary embodiments of the present invention, consider a video sequence including the four frames of FIGS. 7a, 7b, 7c and 7d, those frames being within four identified transitions in the video sequence. Here, after the processing element 14 receives and decomposes the video sequence (see FIG. 3, blocks 30, 32), the processing element can identify one or more objects in the four identified transitions (see block 34). In first, second, third and fourth transitions including the respective frames of FIGS. 7a, 7b, 7c and 7d, for example, the processing element may identify a skater in a first pose, a skateboard, the skater and a second pose, and the skater in a third pose, as shown extracted from their respective frames in FIGS. 8a, 8b, 8c and 8d.
After identifying objects in the video transitions, the processing element 14 can rank or otherwise assign a priority ranking to the identified objects, and select a predetermined number of the objects based on the priority rankings (see block 36). In this example, ranking one or more identified objects may include classifying those objects as key objects or context objects, where the key objects represent the main actors in the story (e.g., the skater) and the context objects provide scene information (e.g., skateboard). In this regard, key objects may be classified as such based upon on their role in the video sequence, which may be evidenced for example by being involved in an appreciable or at least a measurable amount of motion, and/or by being in the center of focus of a number of frames. The remaining objects not otherwise classified as key objects, then, may be classified as context objects. It should be understood, however, that key objects of one or more transitions may be classified as context objects in one or more other transitions, and vice versa.
After ranking the identified objects and selecting one or more of the identified objects, the processing element can compose an image (e.g., thumbnail) representation of the video sequence based upon the selected objects. In this regard, the processing element can extract the selected objects from their respective frames of the video sequence and form a composite image of the extracted objects (see block 38). The objects may be selected such that the composed image includes at least one key object, which may be increased in size over other included objects (key or context objects) in the image. Consider, for example, that the identified skater in the first pose (key object) and the skateboard (context object) are selected. In such an instance, the processing element can extract the skater in the first pose and skateboard from their respective frames, as shown in FIGS. 8a and 8b, and form a composite image of the extracted objects, such as shown in FIG. 9a. Alternatively, consider, for example, that the identified skater in the second pose (first key object) and the skater in the third pose (second key object) are selected. In that instance, the processing element can extract the skater in the second pose and the skater in the third pose from their respective frames, as shown in FIGS. 8c and 8d, and form a composite image of the extracted objects, such as shown in FIG. 9b with the second key object being increased in size of the first key object.
As explained above, the processing element 14 can extract selected objects from their respective frames, and combine those objects into the image representation of the video sequence. However, it should be understood that the processing element may extract one or more, if not all, identified objects from their respective frames of the video sequence at any point after the processing element identifies the respective frames. Thus, for example, the identified objects may be extracted from their respective frames after being identified but before being ranked and/or selected (see block 36).
Further, as explained above the image representation of a video sequence generated in accordance with exemplary embodiments of the present invention may comprise or otherwise be considered a thumbnail image, which may be small or otherwise miniaturized in nature. Thus, the image representation generated in accordance with exemplary embodiments of the present invention may be small or otherwise of a size classifying the image representation as a thumbnail. It should be understood, however, that the image representation may comprise any of a number of different sizes, some of which may be larger than that required to classify the image representation as a thumbnail.
According to one aspect of the present invention, the functions performed by one or more of the entities of the system, such as the source processing element 14, may be performed by various means, such as hardware and/or firmware, including those described above, alone and/or under control of a computer program product (client application). The computer program product for performing one or more functions of exemplary embodiments of the present invention includes a computer-readable storage medium, such as the non-volatile storage medium, and software including computer-readable program code portions, such as a series of computer instructions, embodied in the computer-readable storage medium.
In this regard, FIG. 3 is a flowchart of methods, systems and program products according to the invention. It will be understood that each block or step of the flowchart, and combinations of blocks in the flowchart, can be implemented by various means, such as hardware, firmware, and/or software including one or more computer program instructions. As will be appreciated, any such computer program instructions may be loaded onto a computer or other programmable apparatus (i.e., hardware) to produce a machine, such that the instructions which execute on the computer or other programmable apparatus create means for implementing the functions specified in the flowchart's block(s) or step(s). These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means which implement the function specified in the flowchart's block(s) or step(s). The computer program instructions may also be loaded onto a computer or other programmable apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart's block(s) or step(s).
Accordingly, blocks or steps of the flowchart support combinations of means for performing the specified functions, combinations of steps for performing the specified functions and program instruction means for performing the specified functions. It will also be understood that one or more blocks or steps of the flowchart, and combinations of blocks or steps in the flowchart, can be implemented by special purpose hardware-based computer systems which perform the specified functions or steps, or combinations of special purpose hardware and computer instructions.
Many modifications and other embodiments of the invention will come to mind to one skilled in the art to which this invention pertains having the benefit of the teachings presented in the foregoing descriptions and the associated drawings. Therefore, it is to be understood that the invention is not to be limited to the specific exemplary embodiments disclosed and that modifications and other embodiments are intended to be included within the scope of the appended claims. Although specific terms are employed herein, they are used in a generic and descriptive sense only and not for purposes of limitation.

Claims (19)

What is claimed is:
1. An apparatus comprising a processor and a memory storing executable instructions that in response to execution by the processor cause the apparatus or another apparatus to at least:
decompose a video sequence including a plurality of frames, wherein at least some of the frames include an image of a scene, at least some of the scenes including at least one object, the images of the at least some of the scenes including at least one image of the respective at least one object, wherein being caused to decompose a video sequence includes being caused to identify one or more transitions in the video sequence, each of the identified one of or more transitions including at least one frame;
identify a plurality of objects from within the at least one frame of the identified one or more transitions in the video sequence;
assign a priority ranking to the identified objects, wherein being caused to assign a priority ranking includes being caused to assign a priority ranking to the identified objects based upon at least one variable associated with each of the identified objects, the at least one variable of an identified object comprising at least one of a size of the object, a number of frames including the object, a motion characterization of the object, or a number of video transitions including the object;
select at least some of the identified objects based upon the assigned priority rankings of the respective objects;
extract the selected objects from the frames of the video sequence including the respective objects; and
combine the selected and extracted objects into an image representation of the video sequence.
2. An apparatus according to claim 1, wherein being caused to extract the selected objects includes being caused to extract the selected objects such that at least one of the selected objects is extracted from a frame different from the frame from which another of the selected objects is extracted.
3. An apparatus according to claim 1, wherein the memory stores executable instructions that in response to execution by the processor cause the apparatus or the other apparatus to further:
alter at least one of the identified objects to thereby denote a relative importance of the respective at least one object in the frames of the video sequence, at least one of the altered objects being combined with at least one other object into the image representation of the video sequence.
4. An apparatus according to claim 1, wherein being caused to assign a priority ranking includes being caused to classify at least some of the identified objects as a key object or a context object, at least one object being classified as a key object based upon at least one of the respective object being included in a measurable amount of motion within at least some of the frames of the video sequence, or being in a center of focus of at least some of the frames, and
wherein being caused to select at least some of the identified objects includes being caused to select at least some of the identified objects such that at least one of the selected objects comprises a key object.
5. An apparatus according to claim 1, wherein the apparatus is adapted for implementation by at least one of a personal computing device, a media center device, a personal video recorder, a portable media consumption device, a dedicated entertainment device, a television, or a digital television set-top box.
6. An apparatus comprising:
a first means for decomposing a video sequence including a plurality of frames, wherein at least some of the frames include an image of a scene, at least some of the scenes including at least one object, the images of the at least some of the scenes including at least one image of the respective at least one object, wherein the first means being for decomposing a video sequence includes being for identifying one or more transitions in the video sequence, each of the identified one of or more transitions including at least one frame;
a second means for identifying a plurality of objects from within the at least one frame of the identified one or more transitions in the video sequence;
a third means for assigning a priority ranking to the identified objects, wherein the third means is for assigning the priority ranking to the identified objects based upon at least one variable associated with each of the identified objects, the at least one variable of an identified object comprising at least one of a size of the object, a number of frames including the object, a motion characterization of the object, or a number of video transitions including the object;
a fourth means for selecting at least some of the identified objects based upon the assigned priority rankings of the respective objects;
a fifth means for extracting the selected objects from the frames of the video sequence including the respective objects; and
a sixth means for combining the selected and extracted objects into an image representation of the video sequence.
7. An apparatus according to claim 6, wherein the fifth means is for extracting the selected objects such that at least one of the selected objects is extracted from a frame different from the frame from which another of the selected objects is extracted.
8. An apparatus according to claim 6 further comprising:
a sixth means for altering at least one of the identified objects to thereby denote a relative importance of the respective at least one object in the frames of the video sequence, at least one of the altered objects being combined with at least one other object into the image representation of the video sequence.
9. An apparatus according to claim 6, wherein the third means is for assigning the priority ranking including classifying at least some of the identified objects as a key object or a context object, at least one object being classified as a key object based upon at least one of the respective object being included in a measurable amount of motion within at least some of the frames of the video sequence, or being in a center of focus of at least some of the frames, and
wherein the fourth means is for selecting at least some of the identified objects such that at least one of the selected objects comprises a key object.
10. An apparatus according to claim 6, wherein the apparatus is adapted for implementation by at least one of a personal computing device, a media center device, a personal video recorder, a portable media consumption device, a dedicated entertainment device, a television, or a digital television set-top box.
11. A method comprising:
decomposing a video sequence including a plurality of frames, wherein at least some of the frames include an image of a scene, at least some of the scenes including at least one object, the images of the at least some of the scenes including at least one image of the respective at least one object, wherein decomposing a video sequence includes identifying one or more transitions in the video sequence, each of the identified one of or more transitions including at least one frame;
identifying a plurality of objects from within the at least one frame of the identified one or more transitions in the video sequence;
assigning a priority ranking to the identified objects, wherein assigning a priority ranking comprises assigning a priority ranking to the identified objects based upon at least one variable associated with each of the identified objects, the at least one variable of an identified object comprising at least one of a size of the object, a number of frames including the object, a motion characterization of the object, or a number of video transitions including the object;
selecting at least some of the identified objects based upon the assigned priority rankings of the respective objects;
extracting the selected objects from the frames of the video sequence including the respective objects; and
combining the selected and extracted objects into an image representation of the video sequence.
12. A method according to claim 11, wherein extracting the selected objects comprises extracting the selected objects such that at least one of the selected objects is extracted from a frame different from the frame from which another of the selected objects is extracted.
13. A method according to claim 11 further comprising:
altering at least one of the identified objects to thereby denote a relative importance of the respective at least one object in the frames of the video sequence, at least one of the altered objects being combined with at least one other object into the image representation of the video sequence.
14. A method according to claim 11, wherein assigning a priority ranking includes classifying at least some of the identified objects as a key object or a context object, at least one object being classified as a key object based upon at least one of the respective object being included in a measurable amount of motion within at least some of the frames of the video sequence, or being in a center of focus of at least some of the frames, and
wherein selecting at least some of the identified objects comprises selecting at least some of the identified objects such that at least one of the selected objects comprises a key object.
15. A computer program product comprising a non-transitory computer-readable storage medium having computer-readable program code portions stored therein that in response to execution by a processor cause an apparatus to at least:
decompose a video sequence including a plurality of frames, wherein at least some of the frames include an image of a scene, at least some of the scenes including at least one object, the images of the at least some of the scenes including at least one image of the respective at least one object, wherein being caused to decompose a video sequence includes being caused to identify one or more transitions in the video sequence, each of the identified one of or more transitions including at least one frame;
identify a plurality of objects from within the at least one frame of the identified one or more transitions in the video sequence;
assign a priority ranking to the identified objects, wherein being caused to assign a priority ranking includes being caused to assign a priority ranking to the identified objects based upon at least one variable associated with each of the identified objects, the at least one variable of an identified object comprising at least one of a size of the object, a number of frames including the object, a motion characterization of the object, or a number of video transitions including the object;
select at least some of the identified objects based upon the assigned priority rankings of the respective objects;
extract the selected objects from the frames of the video sequence including the respective objects; and
combine the selected and extracted objects into an image representation of the video sequence.
16. A computer program product according to claim 15, wherein being caused to extract the selected objects includes being caused to extract the selected objects such that at least one of the selected objects is extracted from a frame different from the frame from which another of the selected objects is extracted.
17. A computer program product according to claim 15, wherein the computer-readable storage medium has computer-readable program code portions stored therein that in response to execution by the processor cause the apparatus to further:
alter at least one of the identified objects to thereby denote a relative importance of the respective at least one object in the frames of the video sequence, at least one of the altered objects being combined with at least one other object into the image representation of the video sequence.
18. A computer program product according to claim 15, wherein being caused to assign a priority ranking includes being caused to classify at least some of the identified objects as a key object or a context object, at least one object being classified as a key object based upon at least one of the respective object being included in a measurable amount of motion within at least some of the frames of the video sequence, or being in a center of focus of at least some of the frames, and
wherein being caused to select at least some of the identified objects includes being caused to select at least some of the identified objects such that at least one of the selected objects comprises a key object.
19. An apparatus comprising a processor and a memory storing executable instructions that in response to execution by the processor cause the apparatus or another apparatus to at least:
decompose a video sequence including a plurality of frames, wherein being caused to decompose a video sequence includes being caused to identify one or more transitions in the video sequence, and each of the identified one or more transitions includes at least one frame;
identify a plurality of objects within the at least one frame of the identified one or more transitions in the video sequence;
assign a priority ranking to the identified objects, wherein being caused to assign a priority ranking includes being caused to assign a priority ranking to the identified objects based upon at least one variable associated with each of the identified objects, the at least one variable of an identified object comprising at least one of a size of the object, a number of frames including the object, a motion characterization of the object, or a number of video transitions including the object;
select at least one of the identified objects based on the assigned priority ranking of the plurality of objects;
extract the at least one selected object from the plurality of frames of the video sequence; and
combine the at least one extracted object into an image representation of the video sequence.
US14/046,795 2006-01-10 2013-10-04 Apparatus, method and computer program product for generating a thumbnail representation of a video sequence Expired - Fee Related USRE47421E1 (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
US14/046,795 USRE47421E1 (en) 2006-01-10 2013-10-04 Apparatus, method and computer program product for generating a thumbnail representation of a video sequence

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
US11/328,617 US8032840B2 (en) 2006-01-10 2006-01-10 Apparatus, method and computer program product for generating a thumbnail representation of a video sequence
US14/046,795 USRE47421E1 (en) 2006-01-10 2013-10-04 Apparatus, method and computer program product for generating a thumbnail representation of a video sequence

Related Parent Applications (1)

Application Number Title Priority Date Filing Date
US11/328,617 Reissue US8032840B2 (en) 2006-01-10 2006-01-10 Apparatus, method and computer program product for generating a thumbnail representation of a video sequence

Publications (1)

Publication Number Publication Date
USRE47421E1 true USRE47421E1 (en) 2019-06-04

Family

ID=38234182

Family Applications (2)

Application Number Title Priority Date Filing Date
US11/328,617 Ceased US8032840B2 (en) 2006-01-10 2006-01-10 Apparatus, method and computer program product for generating a thumbnail representation of a video sequence
US14/046,795 Expired - Fee Related USRE47421E1 (en) 2006-01-10 2013-10-04 Apparatus, method and computer program product for generating a thumbnail representation of a video sequence

Family Applications Before (1)

Application Number Title Priority Date Filing Date
US11/328,617 Ceased US8032840B2 (en) 2006-01-10 2006-01-10 Apparatus, method and computer program product for generating a thumbnail representation of a video sequence

Country Status (2)

Country Link
US (2) US8032840B2 (en)
WO (1) WO2007080465A1 (en)

Families Citing this family (27)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2007189473A (en) * 2006-01-13 2007-07-26 Hitachi Ltd Moving picture reproducing device
JP2007266800A (en) * 2006-03-28 2007-10-11 Hitachi Ltd Information reproducing device
US8750115B2 (en) * 2006-11-21 2014-06-10 Verizon Patent And Licensing Inc. Priority-based buffer management
JP4978324B2 (en) * 2007-06-14 2012-07-18 株式会社日立製作所 Image recording apparatus, image recording system, and image reproducing method
JP5194776B2 (en) * 2007-12-21 2013-05-08 株式会社リコー Information display system, information display method and program
JP5106271B2 (en) * 2008-06-27 2012-12-26 キヤノン株式会社 Image processing apparatus, image processing method, and computer program
US8259177B2 (en) * 2008-06-30 2012-09-04 Cisco Technology, Inc. Video fingerprint systems and methods
US20090327334A1 (en) * 2008-06-30 2009-12-31 Rodriguez Arturo A Generating Measures of Video Sequences to Detect Unauthorized Use
JP4623201B2 (en) * 2008-10-27 2011-02-02 ソニー株式会社 Image processing apparatus, image processing method, and program
US9565479B2 (en) * 2009-08-10 2017-02-07 Sling Media Pvt Ltd. Methods and apparatus for seeking within a media stream using scene detection
US8897603B2 (en) * 2009-08-20 2014-11-25 Nikon Corporation Image processing apparatus that selects a plurality of video frames and creates an image based on a plurality of images extracted and selected from the frames
US8571330B2 (en) * 2009-09-17 2013-10-29 Hewlett-Packard Development Company, L.P. Video thumbnail selection
JP2011254240A (en) * 2010-06-01 2011-12-15 Sony Corp Image processing device, image processing method and program
US8621351B2 (en) * 2010-08-31 2013-12-31 Blackberry Limited Methods and electronic devices for selecting and displaying thumbnails
US9609236B2 (en) 2013-09-16 2017-03-28 Kyle L. Baltz Camera and image processing method
EP2869236A1 (en) * 2013-10-31 2015-05-06 Alcatel Lucent Process for generating a video tag cloud representing objects appearing in a video content
JP6257337B2 (en) * 2014-01-14 2018-01-10 日本放送協会 Image selection device
EP2960811A1 (en) * 2014-06-26 2015-12-30 Thomson Licensing Method and device for analyzing a database comprising a large number of video images
KR102375864B1 (en) * 2015-02-10 2022-03-18 한화테크윈 주식회사 System and method for browsing summary image
CN105704570B (en) * 2016-03-08 2019-05-07 上海小蚁科技有限公司 Method and apparatus for generating one or more preview frames of video
GB201818647D0 (en) * 2018-11-15 2019-01-02 Spectral Edge Ltd Image enhancement system and method
US10887542B1 (en) 2018-12-27 2021-01-05 Snap Inc. Video reformatting system
US11665312B1 (en) * 2018-12-27 2023-05-30 Snap Inc. Video reformatting recommendation
US11183219B2 (en) * 2019-05-01 2021-11-23 Sony Interactive Entertainment Inc. Movies with user defined alternate endings
CN113840169B (en) * 2020-06-23 2023-09-19 中国移动通信集团辽宁有限公司 Video processing method, device, computing equipment and storage medium
GB2600156A (en) * 2020-10-23 2022-04-27 Canon Kk Computer-implemented method, computer program and apparatus for generating a thumbnail from a video sequence
WO2022083990A1 (en) * 2020-10-23 2022-04-28 Canon Kabushiki Kaisha Computer-implemented method, computer program and apparatus for video processing and for generating a thumbnail from a video sequence, and video surveillance system comprising such an apparatus

Citations (23)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US4737921A (en) * 1985-06-03 1988-04-12 Dynamic Digital Displays, Inc. Three dimensional medical image display system
US5894333A (en) * 1996-01-30 1999-04-13 Mitsubishi Denki Kabushiki Kaisha Representative image display method, representative image display apparatus, and motion image search appratus employing the representative image display apparatus
US6052492A (en) * 1997-12-09 2000-04-18 Sun Microsystems, Inc. System and method for automatically generating an image to represent a video sequence
US20010020981A1 (en) 2000-03-08 2001-09-13 Lg Electronics Inc. Method of generating synthetic key frame and video browsing system using the same
US20010040924A1 (en) * 2000-05-11 2001-11-15 Osamu Hori Object region data describing method and object region data creating apparatus
US20020028026A1 (en) 1998-06-11 2002-03-07 Chen Shenchang Eric Extracting photographic images from video
US6389168B2 (en) * 1998-10-13 2002-05-14 Hewlett Packard Co Object-based parsing and indexing of compressed video streams
US20020069218A1 (en) * 2000-07-24 2002-06-06 Sanghoon Sull System and method for indexing, searching, identifying, and editing portions of electronic multimedia files
US20020093591A1 (en) * 2000-12-12 2002-07-18 Nec Usa, Inc. Creating audio-centric, imagecentric, and integrated audio visual summaries
US20030137546A1 (en) * 2001-12-28 2003-07-24 Lg Electronics Inc. Apparatus and method for generating thumbnail images
US20030189588A1 (en) * 2002-04-03 2003-10-09 Andreas Girgensohn Reduced representations of video sequences
US20030210261A1 (en) 2002-05-07 2003-11-13 Peng Wu Scalable video summarization
US6665342B1 (en) * 1999-07-02 2003-12-16 International Business Machines Corporation System and method for producing a still image representation of a motion video
US20040095396A1 (en) 2002-11-19 2004-05-20 Stavely Donald J. Video thumbnail
EP1465196A1 (en) 2003-04-01 2004-10-06 Microsoft Corporation Generating visually representative video thumbnails
US20050010955A1 (en) 2003-05-15 2005-01-13 Elia Eric J. Method and system for playing video
US20050185823A1 (en) * 2004-02-24 2005-08-25 International Business Machines Corporation System and method for generating a viewable video index for low bandwidth applications
US6973628B2 (en) * 2000-08-31 2005-12-06 Sony Corporation Image displaying apparatus and image displaying method and program medium
US20060064716A1 (en) * 2000-07-24 2006-03-23 Vivcom, Inc. Techniques for navigating multiple video streams
US20060155684A1 (en) * 2005-01-12 2006-07-13 Microsoft Corporation Systems and methods to present web image search results for effective image browsing
US20070036518A1 (en) * 2003-09-27 2007-02-15 Jeong Dong S Apparatus and method for extracting representative still images from mpeg video
US20070041707A1 (en) * 2005-08-19 2007-02-22 Timothy Edmunds Video receiver, transmission apparatus and method of providing a representative image
US20080019610A1 (en) * 2004-03-17 2008-01-24 Kenji Matsuzaka Image processing device and image processing method

Patent Citations (26)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US4737921A (en) * 1985-06-03 1988-04-12 Dynamic Digital Displays, Inc. Three dimensional medical image display system
US5894333A (en) * 1996-01-30 1999-04-13 Mitsubishi Denki Kabushiki Kaisha Representative image display method, representative image display apparatus, and motion image search appratus employing the representative image display apparatus
US6052492A (en) * 1997-12-09 2000-04-18 Sun Microsystems, Inc. System and method for automatically generating an image to represent a video sequence
US20020028026A1 (en) 1998-06-11 2002-03-07 Chen Shenchang Eric Extracting photographic images from video
US6389168B2 (en) * 1998-10-13 2002-05-14 Hewlett Packard Co Object-based parsing and indexing of compressed video streams
US6665342B1 (en) * 1999-07-02 2003-12-16 International Business Machines Corporation System and method for producing a still image representation of a motion video
US20010020981A1 (en) 2000-03-08 2001-09-13 Lg Electronics Inc. Method of generating synthetic key frame and video browsing system using the same
US20010040924A1 (en) * 2000-05-11 2001-11-15 Osamu Hori Object region data describing method and object region data creating apparatus
US7091988B2 (en) * 2000-05-11 2006-08-15 Kabushiki Kaisha Toshiba Object region data describing method and object region data creating apparatus
US20020069218A1 (en) * 2000-07-24 2002-06-06 Sanghoon Sull System and method for indexing, searching, identifying, and editing portions of electronic multimedia files
US20060064716A1 (en) * 2000-07-24 2006-03-23 Vivcom, Inc. Techniques for navigating multiple video streams
US6973628B2 (en) * 2000-08-31 2005-12-06 Sony Corporation Image displaying apparatus and image displaying method and program medium
US20020093591A1 (en) * 2000-12-12 2002-07-18 Nec Usa, Inc. Creating audio-centric, imagecentric, and integrated audio visual summaries
US20030137546A1 (en) * 2001-12-28 2003-07-24 Lg Electronics Inc. Apparatus and method for generating thumbnail images
US20030189588A1 (en) * 2002-04-03 2003-10-09 Andreas Girgensohn Reduced representations of video sequences
US7149974B2 (en) * 2002-04-03 2006-12-12 Fuji Xerox Co., Ltd. Reduced representations of video sequences
US20030210261A1 (en) 2002-05-07 2003-11-13 Peng Wu Scalable video summarization
US20040095396A1 (en) 2002-11-19 2004-05-20 Stavely Donald J. Video thumbnail
US20040197071A1 (en) 2003-04-01 2004-10-07 Microsoft Corporation Generating visually representative video thumbnails
EP1465196A1 (en) 2003-04-01 2004-10-06 Microsoft Corporation Generating visually representative video thumbnails
US20050010955A1 (en) 2003-05-15 2005-01-13 Elia Eric J. Method and system for playing video
US20070036518A1 (en) * 2003-09-27 2007-02-15 Jeong Dong S Apparatus and method for extracting representative still images from mpeg video
US20050185823A1 (en) * 2004-02-24 2005-08-25 International Business Machines Corporation System and method for generating a viewable video index for low bandwidth applications
US20080019610A1 (en) * 2004-03-17 2008-01-24 Kenji Matsuzaka Image processing device and image processing method
US20060155684A1 (en) * 2005-01-12 2006-07-13 Microsoft Corporation Systems and methods to present web image search results for effective image browsing
US20070041707A1 (en) * 2005-08-19 2007-02-22 Timothy Edmunds Video receiver, transmission apparatus and method of providing a representative image

Non-Patent Citations (38)

* Cited by examiner, † Cited by third party
Title
Byrne, M. D., Using Icons to Find Documents: Simplicity is Critical; Proceedings of the SIGCHI Conference on Human Factors in Computing Systems; 1993; pp. 446-453; AMC Press.
Changick, K. et al., Object-Based Video Abstraction for Video Surveillance Systems; IEEE Transactions on Circuits and Systems for Viedo Technology; Dec. 2002; pp. 1128-1138; vol. 12, No. 12.
Chen, H. W., J. H. Kuo, W. T. Chu, J. L. Wu; Action Movies Segmentation and Summarization Based on Tempo Analysis; Proceedings of the 6th ACM SIGMM International Workshop on Multimedia Information Retrieval; 2004; pp. 251-258; ACM Press, New York, NY.
Corridoni, J. M., A. Del Bimbo, S. DeMagistris; Image Query by Semantical Color content; Proceedings of the Workshop on Advanced Visual Interfaces; 1996; pp. 213-222.
Cristel, M. G., A. G. Hauptmann, A. S. Warmack, S. A. Crosby; Adjustable Filmstrips and Skims as Abstractions for a Digital Video Library; May 1999; 6 pages.
Dekel, A., O. Bergman; Synopsus: A Personal Summary Tool for Video; CHI 2000 Extended Abstracts on Human Factors in Computing Systems; 2000; pp. 4-5; ACM Press, New York.
Duda, R. O., P. E. Hart, D. G. Stork; Pattern Classification; International Journal of Computational Intelligence and Applications; 2001; pp. 335-339; vol. 1, No. 3; Imperial College Press.
Fayzullin, M., V. S. Subrahmanian, A. Piccariello, M. L. Sapino; The CPR Model for Summarizing Video; Proceedings of the 1st SCM International Workshop on Multimedia Databases; 2003; pp. 2-9; ACM Press, New York, NY.
Flickner, M., H. Sawhney, W. Niblack, J. Ashley, Q. Huang, B. Dom, M. Gorkani, J. Hafner, D. Lee, D. Petkovic, D. Steele; Query by Image and Video Content: The QBIC System; Sep. 1995; pp. 23-32; IEEE.
He, T. R., S. E. Hudson; Multidimensional Icons; ACM Transactions on Graphics; 1990; pp. 133-137.
Henry, L., E. Sanocki, A. Gupta, J. Grudin; Auto-Summarization of Audio-Video Presentations; Proceedings of the Seventh ACM International Conference on Multimedia (Part 1); 1999; pp. 489-498; ACM Press, New York, NY.
Huang, Q., Z. Liu, A. Rosenberg; Automated Semantic Structure Reconstruction and Representation Generation for Broadcast News; Proceedings of SPIE on Storage and Retrieval for Still Image and Video Databases; Jan. 1999; vol. 3656; pp. 50-62.
Hyun Sung Chang, Sanghoon Sull, and Sang Uk Lee. "Efficient video indexing scheme for content-based retrieval", Dec. 1999, IEEE, Transactions on Circuit and System for Video Technology, vol. 9 No. 8, pp. 1269-1279. *
Jonathan D. Courtney, "Automatic Video Indexing via Object Motion Analysis", May 12, 1998, Texas Instruments, Pattern Recognition, vol. 30, Issue 4, pp. 607-625. *
Kwatra, V., A. Schodli, I. Essa, G. Turk, A. Bobick; Graphcut Textures: Image and Video Systehsis Using Graph Cuts; Proceedings: ACM Transactions on Graphics; 2003; 10 pages.
Lam, H., P. Baudisch; Summary Thumbnails: Readable Overviews for Small Screen Web Browsers; Proceeding of the SIGCHI Conference on Human Factors in Computing Systems; 2005; pp. 681-690; SCM Press, New York, NY.
Lewis, J. P., R. Rosenholtz, N. Fong, U. Neumann; Visual IDs: Automatic Distinctive Icons for Deskktop Interfaces; 2004; pp. 416-423; ACM Transactions on Graphics.
Linehart, R., S. Pfeiffer, W. Effelsberg; Video Abstracting; SCM Journal of Communications; 1997; pp. 54-62.
M. G. Cristel, A. G. Hauptmann, H. D. Wactlar, T. D. Ng; Colleges as Dynamic Summaries for News Video; Mullltimedia 2002: Proceedings of the Tenth ACM International Conference on Multimedia; 2002; pp. 561-569; ACM Press, New York, NY.
M. M. Yeung, B. L. Yeo; Video Visualization for Compact Presentation and Fast Browsing of Pictorial Content; IEEE Trans. Circuits Syst. Video Technology; Oct. 1997; 15 pages; vol. 7, No. 5.
Ma, Y., H, Zhang; Video Snapshot: A Bird View of Video Sequence; 11th International Multimedia Modelling Conference; 2005; 8 pages; IEEE Computer Society.
Manske, K., Video Browsing Using 3d Video Content Trees; Proceedings of the 1998 Workshop on New Paradigms in Information Visualization and Manipulation; 1998; pp. 20-24; ACM Press, New York, NY.
Moriyama, T., M. Sakauchi; Video Summarization Based on the Psychological Content in the Track Structure; Proceedings of the 2000 ACM Workshops on Multimedia; 2000; pp. 191-194; ACM Press, New York, NY.
Moyes, J., When Users Do and Don't Rely on Icon Shape; Conference Companion on Human Factors in Computing Systems; 1994; pp. 283-284; ACM Press.
Nam, J., A. H. Tewfik; Dynamic Video Summarization and Visulaization;Proceedings of the Seventh ACM International Conference on Multimedia (part 2); 1999; pp. 53-56; ACM Press, New York, NY.
PCT/IB2007/000010 International Search Report dated May 6, 2007.
Ponceleon, D., S. Srinivasan, A. Amir, D. Petkovic, D. Diklic; Key to Effective Video Retrieval: Effective Cataloging and Browsing; Proceedings of the Sixth ACM International conference on Multimedia; 1998; pp. 99-107; ACM Press, New York, NY.
R. O. Duda, P. E. Hart, D. G. Stork; Pattern Classification; International Journal of Computational Intelligence and Applications; 2001; pp. 335-339; vol. 1, No. 3; Imperial College Press.
Rother, C., S. Kumar, V. Kolmogorov, A. Blake; Digital Tapestry; 2005; 8 pages; Proceedings of the 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition.
Schodl, A., R. Szeliski, D. H. Salesin, I. Essa; Video Textures; Oct. 2000; 10 pages.
Shipman, Frank et al., Creating navigable multi-level video summaries; Proceedings of the 2003 International Conference on Multimedia and Expo: Jul. 6-9, 2003; Page(s): 753-756; vol. 2.
Taniguchi, Y., A. Akutsu, Y. Tonomura; PanoramaExcerpts: Extracting and Packing Panoramas for Video Browsing; Proceedings of the Fifth ACM International Conference on Multimedia; Nov. 1997; pp. 427-436.
Tonomura, Y., A. Akutso, Y. Taniguchi,G. Suzuki; Structured Video Computing; IEEE Multimedia; Sep. 1995; pp. 34-43; vol. 3, Issue 3.
Tsang, M., N. Morris, R. Balakrishnan; Temporal Thumbnails: Rapid Visualization of Time-Based Viewing Data; Proceedings of the Working Conference on Advanced Visual Interfaces; 2004; pp. 175-178; ACM Press, New York, NY.
Uchihashi, S., J. Foote, A. Girgensohn, J. Boreczky; Video Manga: Generating Semantically Meaningful Video Summaries; Proceedings of the Seventh ACM International conference on Multimedia (Part 1); 1999; pp. 383-392; ACM Press, New York, NY.
Woodruff, A., A. Faulring; R. Rosenholtz, J. Morrison, P. Pirolli; Using Thumbnails to Search the Web; Proceedings of the SIGCHI Conference on Human Factors in Computing Systems; 2001; pp. 198-205; ACM Press.
Yeung, M. M., B. L. Yeo; Video Visualization for Compact Presentation and Fast Browsing of Pictorial Content; IEEE Trans. Circuits Syst. Video Technology; Oct. 1997; 15 pages; vol. 7, No. 5.
Zhong, D., R, Kumar, S. F. Chang; Real-Time Personalized Sports Video filtering and Summarization; Proceedings of the Ninth ACM International Conference on Multimedia; 2001; pp. 623-625; ACM Press, New York, NY.

Also Published As

Publication number Publication date
WO2007080465A1 (en) 2007-07-19
US8032840B2 (en) 2011-10-04
US20070162873A1 (en) 2007-07-12

Similar Documents

Publication Publication Date Title
USRE47421E1 (en) Apparatus, method and computer program product for generating a thumbnail representation of a video sequence
US12051160B2 (en) Tagging virtualized content
De Avila et al. VSUMM: A mechanism designed to produce static video summaries and a novel evaluation method
US10930071B2 (en) Adaptive decimation using 3D video features
US9478054B1 (en) Image overlay compositing
US20120263433A1 (en) Detecting Key Roles and Their Relationships from Video
US11620480B2 (en) Learning method, computer program, classifier, and generator
US20090119583A1 (en) Image displaying apparatus, image display method, and image display system
US9224156B2 (en) Personalizing video content for Internet video streaming
US20100057722A1 (en) Image processing apparatus, method, and computer program product
JP2006333453A (en) Method and system for spatially and temporally summarizing video
CN104394422A (en) Video segmentation point acquisition method and device
JP3634266B2 (en) Color video processing method and apparatus
CN107750369A (en) For showing the electronic equipment of multiple images and method for handling image
CN116132711A (en) Method and device for generating video template and electronic equipment
US20090132462A1 (en) Distributed metadata extraction
EP3652704B1 (en) Systems and methods for creating and displaying interactive 3d representations of real objects
Hu et al. Video summarization via exploring the global and local importance
Kurzhals et al. AOI transition trees.
CN113596574A (en) Video processing method, video processing apparatus, electronic device, and readable storage medium
CN103503469B (en) The categorizing system of element stage by stage
US11170043B2 (en) Method for providing visualization of progress during media search
CN112749660B (en) Method and device for generating video content description information
Ejaz et al. Video summarization by employing visual saliency in a sufficient content change method
US10930039B2 (en) Systems and methods for personalized banner generation and display

Legal Events

Date Code Title Description
AS Assignment

Owner name: CONVERSANT WIRELESS LICENSING S.A R.L., LUXEMBOURG

Free format text: CHANGE OF NAME;ASSIGNOR:CORE WIRELESS LICENSING S.A.R.L.;REEL/FRAME:043814/0274

Effective date: 20170720

AS Assignment

Owner name: CPPIB CREDIT INVESTMENTS, INC., CANADA

Free format text: AMENDED AND RESTATED U.S. PATENT SECURITY AGREEMENT (FOR NON-U.S. GRANTORS);ASSIGNOR:CONVERSANT WIRELESS LICENSING S.A R.L.;REEL/FRAME:046897/0001

Effective date: 20180731

AS Assignment

Owner name: CONVERSANT WIRELESS LICENSING S.A R.L., LUXEMBOURG

Free format text: RELEASE BY SECURED PARTY;ASSIGNOR:CPPIB CREDIT INVESTMENTS INC.;REEL/FRAME:055910/0584

Effective date: 20210302

FEPP Fee payment procedure

Free format text: MAINTENANCE FEE REMINDER MAILED (ORIGINAL EVENT CODE: REM.); ENTITY STATUS OF PATENT OWNER: LARGE ENTITY

LAPS Lapse for failure to pay maintenance fees

Free format text: PATENT EXPIRED FOR FAILURE TO PAY MAINTENANCE FEES (ORIGINAL EVENT CODE: EXP.); ENTITY STATUS OF PATENT OWNER: LARGE ENTITY