US8683326B2 - Spatiotemporal media object layouts - Google Patents

Spatiotemporal media object layouts Download PDF

Info

Publication number
US8683326B2
US8683326B2 US12/991,431 US99143108A US8683326B2 US 8683326 B2 US8683326 B2 US 8683326B2 US 99143108 A US99143108 A US 99143108A US 8683326 B2 US8683326 B2 US 8683326B2
Authority
US
United States
Prior art keywords
spatiotemporal
determinate
media objects
spatiotemporal layout
computer
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Expired - Fee Related, expires
Application number
US12/991,431
Other languages
English (en)
Other versions
US20110060979A1 (en
Inventor
Eamonn O'Brien-Strain
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Hewlett Packard Development Co LP
Original Assignee
Hewlett Packard Development Co LP
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Hewlett Packard Development Co LP filed Critical Hewlett Packard Development Co LP
Publication of US20110060979A1 publication Critical patent/US20110060979A1/en
Assigned to HEWLETT-PACKARD DEVELOPMENT COMPANY, L.P. reassignment HEWLETT-PACKARD DEVELOPMENT COMPANY, L.P. ASSIGNMENT OF ASSIGNORS INTEREST (SEE DOCUMENT FOR DETAILS). Assignors: O'BRIEN-STRAIN, EAMONN
Application granted granted Critical
Publication of US8683326B2 publication Critical patent/US8683326B2/en
Expired - Fee Related legal-status Critical Current
Adjusted expiration legal-status Critical

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/40—Information retrieval; Database structures therefor; File system structures therefor of multimedia data, e.g. slideshows comprising image and additional audio data
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T11/00—2D [Two Dimensional] image generation
    • G06T11/60—Editing figures and text; Combining figures or text
    • G—PHYSICS
    • G09—EDUCATION; CRYPTOGRAPHY; DISPLAY; ADVERTISING; SEALS
    • G09G—ARRANGEMENTS OR CIRCUITS FOR CONTROL OF INDICATING DEVICES USING STATIC MEANS TO PRESENT VARIABLE INFORMATION
    • G09G5/00—Control arrangements or circuits for visual indicators common to cathode-ray tube indicators and other visual indicators
    • G09G5/14—Display of multiple viewports
    • G—PHYSICS
    • G09—EDUCATION; CRYPTOGRAPHY; DISPLAY; ADVERTISING; SEALS
    • G09G—ARRANGEMENTS OR CIRCUITS FOR CONTROL OF INDICATING DEVICES USING STATIC MEANS TO PRESENT VARIABLE INFORMATION
    • G09G2340/00—Aspects of display data processing
    • G09G2340/12—Overlay of images, i.e. displayed pixel being the result of switching between the corresponding input pixels
    • G—PHYSICS
    • G09—EDUCATION; CRYPTOGRAPHY; DISPLAY; ADVERTISING; SEALS
    • G09G—ARRANGEMENTS OR CIRCUITS FOR CONTROL OF INDICATING DEVICES USING STATIC MEANS TO PRESENT VARIABLE INFORMATION
    • G09G2340/00—Aspects of display data processing
    • G09G2340/12—Overlay of images, i.e. displayed pixel being the result of switching between the corresponding input pixels
    • G09G2340/125—Overlay of images, i.e. displayed pixel being the result of switching between the corresponding input pixels wherein one of the images is motion video

Definitions

  • the invention features a method in accordance with which a determinate spatiotemporal layout specification automatically is generated in accordance with a relative spatiotemporal layout specification.
  • the relative spatiotemporal layout specification describes relative spatial positions and temporal order of media object types.
  • the determinate spatiotemporal layout specification describes a layout of media objects in a display area over time.
  • the process of generating the determinate spatiotemporal layout specification includes determining for each of the media objects a respective spatiotemporal slot corresponding to a respective window in the display area over a respective rendering period in which the media object is scheduled to be rendered.
  • the determinate spatiotemporal layout specification is outputted.
  • the invention also features apparatus and a computer-readable medium storing computer-readable instructions causing a computer to implement the method described above.
  • FIG. 1 is a block diagram of an embodiment of a spatiotemporal layout generation system that includes a spatiotemporal layout generator that processes a set of media objects in accordance with a relative spatiotemporal layout specification to produce a determinate spatiotemporal layout specification.
  • FIG. 2 is a flow diagram of an embodiment of a spatiotemporal layout generation method.
  • FIG. 3 is a diagrammatic view of an embodiment of a relative spatiotemporal layout specification.
  • FIG. 4 is a diagrammatic view of an embodiment of a determinate spatiotemporal layout of media objects generated in accordance with the relative spatiotemporal layout specification of FIG. 3 .
  • FIG. 5A is a diagrammatic view of a set of image frames extracted from an embodiment of a determinate spatiotemporal layout of media objects.
  • FIG. 5B shows the correspondences between the elements of the image frames shown in FIG. 5A and the elements of a corresponding relative spatiotemporal layout specification.
  • FIG. 6 is a diagrammatic view of two representations of an embodiment of a relative spatiotemporal layout specification.
  • FIG. 7 is a diagrammatic view of two representations of an embodiment of a relative spatiotemporal layout specification.
  • FIG. 8 is a diagrammatic view of two representations of an embodiment of a relative spatiotemporal layout specification.
  • FIG. 9A is a diagrammatic view of two representations of an embodiment of a relative spatiotemporal layout specification.
  • FIG. 9B is a diagrammatic view of an embodiment of a determinate spatiotemporal layout of media objects generated in accordance with the relative spatiotemporal layout specification represented in FIG. 9A .
  • FIG. 10 is a block diagram of an embodiment of the spatiotemporal layout generation system shown in FIG. 1 .
  • FIG. 11 is a block diagram of an embodiment of a multidimensional optimization process used in an embodiment of the spatiotemporal layout generation method of FIG. 2 .
  • FIG. 12 is a block diagram of an embodiment of an adaptive scheduling process used together with the multidimensional optimization process of FIG. 11 in an embodiment of the spatiotemporal layout generation method of FIG. 2 .
  • FIG. 13 is a block diagram of an embodiment of a computer system that is programmed to implement an embodiment of the spatiotemporal layout generation system of FIG. 1 .
  • the embodiments that are described in detail below are capable of organizing a collection of media objects into a spatiotemporal layout in which each media object is allocated to a respective slot in a scheduled rendering (or presentation) space that is divided both spatially and temporally.
  • the spatiotemporal layout typically is generated in accordance with a relative spatiotemporal layout specification that guides the spatial and temporal divisions of the presentation space into spatiotemporal slots and guides the allocation of media objects into the slots.
  • the relative spatiotemporal layout specifications are generic specifications of relative spatial layouts of media object types and schedules for ordering the media object types in a particular rendering sequence.
  • the relative spatiotemporal layout specifications specify relative spatiotemporal layouts without regard to any media objects or media object metadata (e.g., duration, aspect ratio, resolution, etc).
  • the relative spatiotemporal layout specifications are generated independently of any particular media objects by skilled multimedia artisans. In this way, the relative spatiotemporal layout specifications may embody the craft and aesthetics of professional multimedia artisans in a way that may be leveraged by unskilled users to produce high-quality presentations of their collections of media objects.
  • media object refers broadly to any form of digital content, including text, audio, graphics, animated graphics, still images, full-motion video, and electronic proxies for physical objects. This content may be packaged and presented individually or in some combination in a wide variety of different forms, including documents, annotations, presentations, music, still photographs, commercial videos, home movies, and metadata describing one or more associated digital content files.
  • Image-based media objects may be complete or partial versions of any type of digital or electronic image, including: an image that was captured by an image sensor (e.g., a video camera, a still image camera, or an optical scanner) or a processed (e.g., filtered, reformatted, enhanced or otherwise modified) version of such an image; a computer-generated bitmap or vector graphic image; a textual image (e.g., a bitmap image containing text); and an iconographic image.
  • image sensor e.g., a video camera, a still image camera, or an optical scanner
  • a processed e.g., filtered, reformatted, enhanced or otherwise modified
  • textual image e.g., a bitmap image containing text
  • iconographic image e.g., a textual image
  • the assignment of single-element media objects to a particular multi-element media object signifies that the constituent single-element media objects are related.
  • the type of single-element media objects in a multi-element media object may be the same or different.
  • the media objects typically are stored in one or more databases on one or more computer-readable media.
  • the media objects may be stored physically in a local database or in one or more remote databases that may be accessed over a local area network and a global communication network. Some media objects also may be stored in a remote database that is accessible over a peer-to-peer network connection.
  • relative spatiotemporal layout refers to a relative spatial arrangement and temporal sequence of media object types, where the absolute positions of the media object types and the absolute rendering times of the media object types are not specified.
  • a relative spatiotemporal layout specification describes the relative spatial positions of the media object types over time.
  • the term “determinate spatiotemporal layout” refers to a layout of media objects in a display area in a particular sequence in accordance with a determinate spatiotemporal layout specification that describes the positions, dimensions, and scheduled rendering periods of the media objects.
  • data structure refers broadly to the physical layout (or format) in which data is organized and stored.
  • a “computer” is a machine that processes data according to computer-readable instructions (e.g., software) that are stored on a computer-readable medium either temporarily or permanently.
  • computer-readable instructions e.g., software
  • a set of such instructions that performs a particular task is referred to as a program or software program.
  • computer-readable medium refers to any medium capable of storing information that is readable by a computer.
  • Examples of computer-readable media are storage devices suitable for tangibly embodying instructions and data include, but are not limited to, all forms of computer-readable memory, including non-volatile forms, for example, semiconductor memory devices, such as EPROM, EEPROM, and Flash memory devices, magnetic disks such as internal hard disks and removable hard disks, magneto-optical disks, DVD-ROM/RAM, and CD-ROM/RAM.
  • FIG. 1 shows an embodiment of a spatiotemporal layout generation system 10 that includes a spatiotemporal layout generator 12 that processes a set 16 of media objects 18 in accordance with a relative spatiotemporal layout specification 14 to produce a determinate spatiotemporal layout specification 20 .
  • the relative spatiotemporal layout specification describes relative spatial positions and temporal order of media object types
  • the determinate spatiotemporal layout specification 20 describes a layout of the media objects 18 in a display area over time.
  • FIG. 2 shows an embodiment of a method that is implemented by the spatiotemporal layout generator 12 .
  • the spatiotemporal layout generator 12 automatically generates the determinate spatiotemporal layout specification 20 in accordance with the relative spatiotemporal layout specification 14 ( FIG. 2 , block 22 ). In this process, the spatiotemporal layout generator 12 determines for each of the media objects 18 a respective spatiotemporal slot corresponding to a respective window in the display area over a respective rendering period in which the media object is scheduled to be rendered.
  • the spatiotemporal layout generator 12 outputs the determinate spatiotemporal layout specification 20 ( FIG. 2 , block 24 ). In some embodiments the spatiotemporal layout generator 12 outputs the determinate spatiotemporal layout specification 20 by storing it on a computer-readable medium. In these embodiments, the spatiotemporal layout generator 12 typically outputs the determinate spatiotemporal layout specification 20 in the form of a specification that includes a data structure (e.g., a table or a list) that describes the allocation of the media objects 18 to slots in a scheduled rendering (or presentation) space that is divided both spatially and temporally. In some embodiments, the specification is stored on a computer-readable medium in an XML (eXtensible Markup Language) file format.
  • XML eXtensible Markup Language
  • the spatiotemporal layout generation system 10 renders a determinate spatiotemporal layout of the media objects 18 in accordance with the determinate spatiotemporal layout specification 20 .
  • the spatiotemporal layout generation system 10 renders the determinate spatiotemporal layout of the media objects 18 on a display.
  • the display may be, for example, a flat panel display, such as a LCD (liquid crystal display), a plasma display, an EL display (electro-luminescent display) and a FED (field emission display).
  • the spatiotemporal layout generation system 10 renders the determinate spatiotemporal layout of the media objects 18 on a print medium (e.g., one or more sheets of paper).
  • the determinate spatiotemporal layout specification 20 corresponds to an output video file that can be rendered by a video player to present the corresponding spatiotemporal layout of the media objects 18 .
  • the output video file is stored on a computer-readable medium in accordance with a video file format (e.g., AVI, MOV, MPEG-2, MPEG-4, Ogg, ASF, RealMedia, and 3gp).
  • the determinate spatiotemporal layout specification 20 corresponds to parsable video playback instructions that cause a machine (e.g., a computer) to present a composite video corresponding to the spatiotemporal layout of the media objects 18 .
  • the instructions are stored on a computer-readable medium in accordance with a multimedia authoring scripting language (e.g., Adobe Flash)® that can by run or parsed by a script interpreter (e.g., an Adobe Flash player) to render the spatiotemporal layout of the media objects 18 .
  • a multimedia authoring scripting language e.g., Adobe Flash
  • a script interpreter e.g., an Adobe Flash player
  • the determinate spatiotemporal layout specification 20 corresponds to a video compositing specification (e.g., a script) that describes the way in which the spatiotemporal layout of the media objects 18 are to be presented in the display area.
  • the video compositing specification is processed by a video authoring tool (e.g., Adobe Flash or AviSynth) that produces an output video file (e.g., an AVI file) or a set of parsable video playback instructions (e.g., an Adobe Flash script or an AviSynth script) that can be processed to render the spatiotemporal layout of the media objects 18 .
  • a video authoring tool e.g., Adobe Flash or AviSynth
  • an output video file e.g., an AVI file
  • a set of parsable video playback instructions e.g., an Adobe Flash script or an AviSynth script
  • the relative spatiotemporal layout specification 14 describes a spatial layout of media object types in a particular temporal sequence, where the absolute positions of the media object types and the absolute rendering periods of the media object types are not specified.
  • the relative spatial positions of the media object types may be described, for example, in accordance with any type of floor plan model that describes the relative spatial positions of the media object types either in relation to each other or in relation to a common reference point (e.g., a corner point or an edge point of a common coordinate system).
  • the relative spatiotemporal layout specification 14 describes a decomposition of a relative rendering space into slots each of which contains exactly one of the media object types.
  • FIG. 3 shows an embodiment of a relative spatiotemporal layout specification 14 that corresponds to a recursive partitioning (or subdividing) model of the relative rendering space.
  • the partitioning model is a binary spatiotemporal partitioning model that is organized into a tree structure 26 .
  • the tree structure 26 has leaf nodes 28 , 30 , 32 , 34 corresponding to respective media object types and interior nodes 36 , 38 , 40 corresponding to partitions of the relative rendering space that is partitioned by the tree structure 26 .
  • ” denotes a vertical spatial division (or split) of the relative rendering space
  • the em dash “—” denotes a horizontal division of the relative rendering space
  • the much greater than sign “>>” denotes a temporal division of the relative rendering space in which the left child node precedes the right child node in a relative rendering sequence.
  • the recursive partitioning of the relative rendering space that is specified by the tree structure 26 corresponds to a first instance of a video media object type that is allocated to a spatiotemporal slot to the left of two successive instances of a photo media object type and a second instance of the video media object type, where the successive instances of the photo media object type are rendered in a top right spatiotemporal slot over a bottom right spatiotemporal slot containing the second instance of the video media object type.
  • FIG. 4 shows an exemplary implementation of a determinate spatiotemporal layout of two videos (i.e., video_ 1 and video_ 2 ) and two photos (i.e., photo_ 1 and photo_ 2 ) that are allocated to respective slots in a scheduled rendering space 42 in accordance with the spatiotemporal partitioning specification that is represented by the binary tree structure 26 .
  • the spatiotemporal partitioning specification that is represented by the tree structure 26 also can be specified using an analogous textual schema that defines a recursive spatiotemporal partitioning of the relative rendering space.
  • an analogous textual schema that defines a recursive spatiotemporal partitioning of the relative rendering space.
  • the schema additionally includes tags or other metadata that allows a designer of the relative spatiotemporal layout specification to specify one or more media object selection criteria for a designated one of the slots.
  • the spatiotemporal layout generator 12 assigns one of the media objects 18 in the set 16 to the designated slot based on a user's indication that the assigned media object matches the media object selection criterion.
  • tags or other metadata that can be included in the schema are the following:
  • the spatiotemporal relative partitioning specification that is represented by the tree structure 26 shown in FIG. 3 is equivalently specified by the following textual description in a computer language consisting of a single expression: video
  • FIG. 5A shows a set of image frames that have been extracted from an embodiment of a determinate spatiotemporal layout of media objects at successive times t 1 , t 2 , t 3 , t 4 , t 5 , t 6 .
  • This embodiment was generated from a user-selected set of ten photo media objects and two video media objects in accordance with the following relative partitioning specification:
  • FIG. 5B shows the correspondences between the elements of the image frames shown in FIG. 5A and the elements of the corresponding relative partitioning specification.
  • the photo 44 rendered in the window in the upper left corner of the display area 46 was selected by the user as the thematic photo of the determinate spatiotemporal layout and the video 48 in the upper right corner of the display area 46 was selected by the user as the climatic video of the determinate spatiotemporal layout.
  • FIG. 6 shows embodiments of a textual specification 50 and a graphical specification 52 of the same relative spatiotemporal layout.
  • two instances of a video media object type are positioned over each other in respective slots that are to the left of a thematic instance of the photo media object type in a slot over two side-by-side instances of the photo media object type.
  • FIG. 7 shows embodiments of a textual specification 54 and a graphical specification 56 of the same relative spatiotemporal layout. These embodiments specify a first allocation of media object types to a set of spatiotemporal slots followed in time by a second allocation of media object types to a set of spatiotemporal slots. Each of these allocations consists of a respective instance of a video media object type positioned in a slot that is to the left of three vertically distributed slots containing respective instances of the video media object type.
  • FIG. 8 shows embodiments of a textual specification 58 and a graphical specification 60 of the same relative spatiotemporal layout.
  • the textual description shows that the computer language can have a sequence of statements before the single expression. These statements can set the values of variables to sub-expressions that can then be combined in the final expression.
  • These embodiments specify a first arrangement of media object types that is positioned to the left of a second arrangement of media object types. In the first arrangement, an instance of a video media object type is allocated to a slot positioned over a pair of side-by-side slots respectively containing a left thematic instance of a photo media object type and a right instance of the photo media object type.
  • the second arrangement consists of a vertical arrangement of three sequences of slots, where
  • FIG. 9A shows embodiments of a textual specification 62 and a graphical specification 64 of the same relative spatiotemporal layout. These embodiments specify a first (“TOP”) arrangement of media object types that is positioned over a second arrangement (“PHOTOS”) of media object types.
  • the first arrangement consists of a horizontal arrangement of a first instance of a text media object type to the left of an instance of a blank media object that is to the left of a second instance of the text media object type.
  • the second arrangement consists of a horizontal arrangement of a left group of media object types to the left of a middle group of media objects types that is to the left of a right group of media object types.
  • Each of the left and right groups consists of a vertical arrangement of three slots each containing a respective instance of a photo media object type, and the middle group consists of a vertical arrangement of two slots each containing a respective instance of the photo media object type.
  • FIG. 9B shows an exemplary embodiment of a determinate spatiotemporal layout 66 of media objects corresponding to the relative spatiotemporal layout specification shown in FIG. 9A .
  • FIG. 10 shows an embodiment 70 of the spatiotemporal layout generation system 10 that includes an embodiment 72 of the spatiotemporal layout generator 12 , a display 74 , and a database 76 storing a set of relative spatiotemporal layout specifications 78 .
  • the spatiotemporal layout generator 72 includes a metadata extraction module 80 , an optimization module 82 , and an output generation module 84 .
  • the spatiotemporal layout generator 72 operates on a collection 86 of media objects 88 , which may be designated by the user or may be identified automatically by the spatiotemporal layout generator 72 .
  • the metadata extraction module 80 extracts values for various parameters, including the aspect ratio and duration (if applicable), from each of the media objects 88 , and passes the extracted values to the optimization module 82 .
  • the optimization module 82 determines values of the spatial and temporal parameters that define the slots in a determinate spatiotemporal layout of the media objects 88 in accordance with a multidimensional optimization process.
  • the optimization module 82 may use any one of a wide variety of multidimensional optimization methods in the process of determining the values of the spatial and temporal parameters that define the slots in a determinate spatiotemporal layout of the media objects 88 .
  • optimization methods include, but are not limited to, simulated annealing optimization methods, hill climbing optimization methods, downhill simplex optimization methods, steepest descent optimization methods, and genetic optimization methods.
  • the optimization module 82 passes the parameter values to the output generation module 84 , which generates a determinate spatiotemporal layout specification 90 from the received parameters values.
  • the spatiotemporal layout generation system 10 renders a determinate spatiotemporal layout of the media objects 88 corresponding to the determinate spatiotemporal layout specification 90 on the display 74 .
  • This section describes an exemplary embodiment of a simulated annealing method that includes a main calling process and a primary simulated annealing routine.
  • the simulated annealing method involves ascertaining a series of successive candidate determinate spatiotemporal layouts of the media objects 88 from an initial candidate determinate spatiotemporal layout of the media objects 88 , through successive candidate determinate spatiotemporal layouts of the media objects 88 defined by different respective sets of spatial and temporal parameter values, to a final determinate spatiotemporal layout corresponding to the determinate spatiotemporal layout specification 90 in accordance with a process of optimizing an objective function characterizing the candidate determinate spatiotemporal layouts.
  • FIG. 11 shows an embodiment of a primary simulated annealing optimization routine that is used in an embodiment of the spatiotemporal layout generation method of FIG. 2 in generating the determinate spatiotemporal layout specification 20 .
  • the optimization module 82 initializes the value of an Accept variable to 0 ( FIG. 11 , block 92 ).
  • the optimization module 82 determines a random candidate layout ( FIG. 11 , block 94 ).
  • the candidate layout typically is specified by a state vector containing values of the spatial and temporal parameters that define a respective determinate layout of the media objects 88 in a display area.
  • the optimization module 82 determines the difference ( ⁇ score ) between the scores of the objective function characterizing the candidate layout and the current score ( FIG. 11 , block 96 ).
  • the optimization module 82 sets the current score equal to the candidate score ( FIG. 11 , block 100 ), increments the Accept value ( FIG. 11 , block 102 ), and sets the current determinate spatiotemporal layout equal to the candidate determinate spatiotemporal layout ( FIG. 11 , block 104 ). If the number of iterations is not equal to N (e.g., 100 ) ( FIG. 11 , block 106 ), the process is repeated; otherwise, the optimization module 82 returns the value of the Accept variable to the main calling process ( FIG. 11 , block 107 ). If ⁇ score ⁇ 0 ( FIG.
  • the optimization module 82 determines whether a move acceptance probability function ⁇ ( ⁇ score ,t) is greater than P ( FIG. 11 , block 108 ), where P is a parameter that has a respective pseudo randomly generated probability value.
  • the random move function is given by e ⁇ score /t and P has a random value in the range [0,1). If ⁇ ( ⁇ score ,t)>P ( FIG. 11 , block 108 ), then the optimization module 82 sets the current score equal to the candidate score ( FIG. 11 , block 100 ), increments the Accept value ( FIG. 11 , block 102 ), and sets the current determinate spatiotemporal layout equal to the candidate determinate spatiotemporal layout ( FIG.
  • the simulated annealing optimization method of FIG. 11 typically is called multiple times by a main calling process that controls the number of iterations of the primary simulated annealing routine and the cooling schedule that sets a temperature parameter, t, which regulates the likelihood that any particular candidate layout will be accepted despite having a lower objective function score.
  • the main calling process changes the temperature parameter, t, each time after the optimization module 82 returns from the primary simulated annealing routine.
  • a variety of different annealing schedules may be used to change the temperature parameter.
  • the temperature parameter is reduced each time after the optimization module 82 returns from the primary simulated annealing routine.
  • the amount by which the temperature parameter is reduced may be a fixed amount or it may vary as a function of the fraction of the time budget that has been expended or as a function of the current temperature value.
  • FIG. 12 shows an embodiment of an adaptive cooling schedule that is used together with the primary simulated annealing routine of FIG. 11 in an embodiment of the spatiotemporal layout generation method of FIG. 2 .
  • the optimization module 82 initializes the value of the temperature parameter, t, to an initial (typically high) value.
  • the optimization module 82 then begins the execution of a FOR loop in which the loop counter R accept decreases incrementally from a high value, H, to a low value, L ( FIG. 12 , block 110 ). If the current value of the Accept variable is equal to the current value of the loop counter ( FIG. 12 , block 112 ), the optimization module 82 proceeds to the next iteration ( FIG. 12 , block 110 ).
  • the value of the Accept variable is set by the primary simulated annealing routine of FIG. 11 . If the current value of the Accept variable is not equal to the current value of the loop counter ( FIG. 12 , block 112 ), the optimization module 82 sets the value of the Accept variable by executing the primary simulated annealing routine of FIG. 11 with the current temperature value, t ( FIG. 12 , block 114 ). If the returned value of the Accept variable is greater than the current loop counter value ( FIG. 12 , block 116 ), the optimization module 82 reduces the temperature value ( FIG. 12 , block 118 ). In some embodiments, the optimization module 82 reduces the current temperature value by a fixed percentage (e.g., 1%).
  • the optimization module 82 increases the temperature value ( FIG. 12 , block 118 ). In some embodiments, the optimization module 82 increases the current temperature value by a fixed percentage (e.g., 1%).
  • the optimization module 82 terminates the simulated annealing method after exiting the FOR loop in block 110 of FIG. 12 .
  • the optimization module 82 continues to run the primary simulated annealing routine of FIG. 11 using a non-adaptive cooling schedule.
  • the value of the temperature parameter, t is reduced from its value at the end of the adaptive cooling process of FIG. 12 by a fixed percentage (e.g., 1%) for a specified number of iterations (e.g., 1000 ).
  • the optimization module 82 terminates the simulated annealing method after the specified number of iterations.
  • the optimization module 82 continues to run the primary simulated annealing routine of FIG. 11 after the specified number of iterations. In this process, the optimization module 82 runs the primary simulated annealing routine of FIG. 11 a specified number of iterations (e.g., 1000 ) with the value of the temperature parameter, t, set to 0 for each of the iterations.
  • the optimization module 82 calculates a respective score for each of the candidate determinate spatiotemporal layouts.
  • the score is the weighted geometric mean of individual matching scores that measure of how close the parameters of each media object match the corresponding parameters of the spatiotemporal slots allocated to the media object.
  • the respective matching score for each media object is calculated from various factors, at least some of which measure the closeness of the media object to its current slot in terms of a respective ratio of the values of a particular parameter (e.g., aspect ratio or duration) for the media object and its current slot.
  • the matching score for a graphical media object is a function of a distort factor and an area factor.
  • the distort factor measures how close the aspect ratio of the media object matches the aspect ratio of its current slot.
  • the area factor measures how close the fraction of the display area that is allocated to the media object corresponds to an equal division of the available display area.
  • the matching score corresponds to a weighted average of the distort factor and the area factor. In some of these embodiments, the distort factor is weighted more than the area factor.
  • the matching score of each non-time-based media object (e.g., a photo) additionally incorporates a duration factor that measures how close the duration of its current slot matches a preferred duration specified for the media object.
  • the preferred duration may be specified either by the user or by default.
  • the duration factor typically is included in the weighted average of the distort factor and the area factor.
  • the matching score of each time-based media object (e.g., a video) additionally incorporates a duration factor that measures how close its duration matches the duration of its current slot matches.
  • the duration factor typically is included in the weighted average of the distort factor and the area factor.
  • the matching score for text-based media objects is a function of preferred values for the height, width, and duration, which may be set by the user or by default.
  • the matching score for text-based media objects corresponds to the product of a width factor, a height factor, and a duration factor.
  • the width factor corresponds to the smaller of the ratio of the preferred width to the slot width or the ratio of the slot width to the preferred width.
  • the height factor corresponds to the smaller of the ratio of the preferred height to the slot height or the ratio of the slot height to the preferred height.
  • the duration factor measures how close the slot duration matches a preferred duration specified for text-based media objects.
  • the matching scores additionally incorporate one or more penalization factors that reduce the matching scores of media objects that are allocated to slots with one or more spatial or temporal dimensions that are below specified threshold dimensions.
  • Embodiments of the spatiotemporal layout generation system 10 may be implemented by one or more discrete modules (or data processing components) that are not limited to any particular hardware, firmware, or software configuration.
  • the modules may be implemented in any computing or data processing environment, including in digital electronic circuitry (e.g., an application-specific integrated circuit, such as a digital signal processor (DSP)) or in computer hardware, firmware, device driver, or software.
  • DSP digital signal processor
  • the functionalities of the modules are combined into a single data processing component.
  • the respective functionalities of each of one or more of the modules are performed by a respective set of multiple data processing components.
  • process instructions e.g., computer-readable code, such as computer software
  • process instructions for implementing the methods that are executed by the embodiments of the spatiotemporal layout generation system 10 , as well as the data is generates, are stored in one or more computer-readable media.
  • Storage devices suitable for tangibly embodying these instructions and data include all forms of non-volatile computer-readable memory, including, for example, semiconductor memory devices, such as EPROM, EEPROM, and flash memory devices, magnetic disks such as internal hard disks and removable hard disks, magneto-optical disks, DVD-ROM/RAM, and CD-ROM/RAM.
  • semiconductor memory devices such as EPROM, EEPROM, and flash memory devices
  • magnetic disks such as internal hard disks and removable hard disks, magneto-optical disks, DVD-ROM/RAM, and CD-ROM/RAM.
  • embodiments of the spatiotemporal layout generation system 10 may be implemented in any one of a wide variety of electronic devices, including computers (e.g., laptop or notebook computers, desktop computers, workstation computers, and server computers).
  • computers e.g., laptop or notebook computers, desktop computers, workstation computers, and server computers.
  • FIG. 13 shows an embodiment 138 of the spatiotemporal layout generation system 10 that is implemented by one or more software modules operating on a computer 140 .
  • the computer 140 includes a processing unit 142 , a system memory 144 , and a system bus 146 that couples processing unit 142 to the various components of the computer 140 .
  • the processing unit 142 typically includes one or more processors, each of which may be in the form of any one of various commercially available processors.
  • the system memory 144 typically includes a read only memory (ROM) that stores a basic input/output system (BIOS) that contains start-up routines for the computer 140 and a random access memory (RAM).
  • ROM read only memory
  • BIOS basic input/output system
  • RAM random access memory
  • the system bus 146 may be a memory bus, a peripheral bus or a local bus, and may be compatible with any of a variety of bus protocols, including PCI, VESA, Microchannel, ISA, and EISA.
  • the computer 140 also includes a persistent storage memory 148 (e.g., a hard drive, a floppy drive, a CD ROM drive, magnetic tape drives, flash memory devices, and digital video disks) that is connected to the system bus 146 and contains one or more computer-readable media disks that provide non-volatile or persistent storage for data, data structures and computer-executable instructions.
  • a persistent storage memory 148 e.g., a hard drive, a floppy drive, a CD ROM drive, magnetic tape drives, flash memory devices, and digital video disks
  • a user may interact (e.g., enter commands or data) with the computer 30 using one or more input devices 150 (e.g., a keyboard, a computer mouse, a microphone, joystick, and touch pad). Information may be presented through a graphical user interface (GUI) that is displayed to the user on a display monitor 152 , which is controlled by a display controller 154 .
  • GUI graphical user interface
  • the computer 30 also typically includes peripheral output devices, such as speakers and a printer.
  • One or more remote computers may be connected to the computer 140 through a network interface card (NIC) 156 .
  • NIC network interface card
  • the system memory 144 also stores the spatiotemporal layout generation system 138 , a GUI driver 158 , and other data 160 including the media objects 18 , intermediate processing data, and output data.
  • the spatiotemporal layout generation system 138 interfaces with the GUI driver 158 and the user input 150 to control the creation of the determinate spatiotemporal layout specification.
  • the spatiotemporal layout generation system 138 additionally includes at least one of a video player and a script interpreter that are configured to render the spatiotemporal layout of the media objects 18 that is specified by the determinate spatiotemporal layout specification 20 by processing the specification 20 .
  • the spatiotemporal layout generation system 138 interfaces with the GUI driver 158 , the user input 150 , the relative spatiotemporal layout specification 14 , and other data structures in producing a graphical user interface that guides the user through the process of generating the determinate spatiotemporal layout specification 20 .
  • the spatiotemporal layout generation system 138 also interfaces with the GUI driver 158 , the determinate spatiotemporal layout specification 20 , and other data structures to control the presentation of determinate spatiotemporal layout of the media objects 18 to the user on the display monitor 152 .
  • the various media objects 18 that are used to render the presentation may be stored locally in persistent storage memory 148 or stored remotely and accessed through the NIC 156 , or both.
  • the embodiments that are described herein are capable of organizing a collection of media objects into a spatiotemporal layout in which each media object is allocated to a respective slot in a scheduled rendering (or presentation) space that is divided both spatially and temporally.
  • the spatiotemporal layout typically is generated in accordance with a relative spatiotemporal layout specification that guides the spatial and temporal divisions of the presentation space into spatiotemporal slots and guides the allocation of media objects into the slots.
  • the relative spatiotemporal layout specifications are generated independently of any particular media objects by skilled multimedia artisans. In this way, the relative spatiotemporal layout specifications may embody the craft and aesthetics of professional multimedia artisans in a way that may be leveraged by unskilled users to produce high-quality presentations of their collections of media objects.
  • the embodiments that are described herein provide significant advantages in the consumer application space where they allow complex events to be documented in an appropriate form for media objects with contents that are inherently choppy and are in widely varying formats and resolutions.

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Computer Hardware Design (AREA)
  • Multimedia (AREA)
  • Data Mining & Analysis (AREA)
  • Databases & Information Systems (AREA)
  • General Engineering & Computer Science (AREA)
  • Processing Or Creating Images (AREA)
  • User Interface Of Digital Computer (AREA)
  • Document Processing Apparatus (AREA)
US12/991,431 2008-05-06 2008-05-06 Spatiotemporal media object layouts Expired - Fee Related US8683326B2 (en)

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
PCT/US2008/005842 WO2009136888A1 (en) 2008-05-06 2008-05-06 Spatiotemporal media object layouts

Publications (2)

Publication Number Publication Date
US20110060979A1 US20110060979A1 (en) 2011-03-10
US8683326B2 true US8683326B2 (en) 2014-03-25

Family

ID=41264802

Family Applications (1)

Application Number Title Priority Date Filing Date
US12/991,431 Expired - Fee Related US8683326B2 (en) 2008-05-06 2008-05-06 Spatiotemporal media object layouts

Country Status (6)

Country Link
US (1) US8683326B2 (ja)
JP (1) JP5325977B2 (ja)
CN (1) CN102084337A (ja)
DE (1) DE112008003854T5 (ja)
GB (1) GB2473370A (ja)
WO (1) WO2009136888A1 (ja)

Families Citing this family (8)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US10282391B2 (en) 2008-07-03 2019-05-07 Ebay Inc. Position editing tool of collage multi-media
US8893015B2 (en) 2008-07-03 2014-11-18 Ebay Inc. Multi-directional and variable speed navigation of collage multi-media
US9639505B2 (en) 2008-07-03 2017-05-02 Ebay, Inc. System and methods for multimedia “hot spot” enablement
JP5501915B2 (ja) * 2010-09-24 2014-05-28 シャープ株式会社 レイアウト選択装置、レイアウト選択方法、レイアウト選択プログラムおよびコンピュータ読み取り可能な記録媒体、ならびに、コンテンツ表示装置およびコンテンツ作成方法
JP6045232B2 (ja) * 2012-07-09 2016-12-14 キヤノン株式会社 画像処理装置、画像処理方法、及びプログラム
KR102069538B1 (ko) * 2012-07-12 2020-03-23 삼성전자주식회사 멀티미디어 요소의 배치를 위한 마크업을 구성하는 방법
US10121270B2 (en) * 2013-07-01 2018-11-06 Facebook, Inc. Flexible image layout
JP6701207B2 (ja) * 2015-08-24 2020-05-27 株式会社日立製作所 情報処理システム

Citations (20)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JPH0830637A (ja) 1994-07-14 1996-02-02 Ricoh Co Ltd 自動レイアウトシステム
US5669006A (en) * 1995-02-23 1997-09-16 International Business Machines Corporation Method for automatically obtaining spatial layout for multimedia presentations
JPH11219369A (ja) 1998-02-03 1999-08-10 Fujitsu Ltd 情報提示装置
JP2000149045A (ja) 1998-11-05 2000-05-30 Matsushita Electric Ind Co Ltd タイトル情報の編集及び再生方法と編集装置
US6223183B1 (en) * 1999-01-29 2001-04-24 International Business Machines Corporation System and method for describing views in space, time, frequency, and resolution
US20020122067A1 (en) 2000-12-29 2002-09-05 Geigel Joseph M. System and method for automatic layout of images in digital albums
US20030192049A1 (en) 2002-04-09 2003-10-09 Schneider Tina Fay Binding interactive multichannel digital document system
US20040186723A1 (en) * 2003-03-19 2004-09-23 Fujitsu Limited Apparatus and method for converting multimedia contents
US20050012743A1 (en) * 2003-03-15 2005-01-20 Thomas Kapler System and method for visualizing connected temporal and spatial information as an integrated visual representation on a user interface
US20050071783A1 (en) * 2003-09-30 2005-03-31 Atkins C. Brian Automatic photo album page layout
US6907563B1 (en) 1999-05-27 2005-06-14 International Business Machines Corporation System and method for composing heterogeneous media components into a unified environment for rich spatio-temporal hotlink authoring and action enablement in low-bandwidth presentations
US20050177593A1 (en) 2004-01-23 2005-08-11 Geodesic Dynamics Dynamic adaptive distributed computer system
US20050286738A1 (en) 2004-05-27 2005-12-29 Sigal Leonid Graphical object models for detection and tracking
JP2006114013A (ja) 2004-10-18 2006-04-27 Microsoft Corp チャート上の自動ラベル配置のためのシステムおよび方法
US7143083B2 (en) 2001-06-12 2006-11-28 Lucent Technologies Inc. Method and apparatus for retrieving multimedia data through spatio-temporal activity maps
US20070033612A1 (en) 2005-08-08 2007-02-08 Princeton Server Group, Inc. Method and apparatus for scheduling delivery of video and graphics
US20070033632A1 (en) * 2005-07-19 2007-02-08 March Networks Corporation Temporal data previewing system
US7231144B2 (en) 2003-08-22 2007-06-12 Seiko Epson Corporation Element layout apparatus, element layout program and element layout method
US20070171716A1 (en) * 2005-11-30 2007-07-26 William Wright System and method for visualizing configurable analytical spaces in time for diagrammatic context representations
US7499046B1 (en) * 2003-03-15 2009-03-03 Oculus Info. Inc. System and method for visualizing connected temporal and spatial information as an integrated visual representation on a user interface

Patent Citations (24)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JPH0830637A (ja) 1994-07-14 1996-02-02 Ricoh Co Ltd 自動レイアウトシステム
US5669006A (en) * 1995-02-23 1997-09-16 International Business Machines Corporation Method for automatically obtaining spatial layout for multimedia presentations
JPH11219369A (ja) 1998-02-03 1999-08-10 Fujitsu Ltd 情報提示装置
JP2000149045A (ja) 1998-11-05 2000-05-30 Matsushita Electric Ind Co Ltd タイトル情報の編集及び再生方法と編集装置
US6223183B1 (en) * 1999-01-29 2001-04-24 International Business Machines Corporation System and method for describing views in space, time, frequency, and resolution
US6907563B1 (en) 1999-05-27 2005-06-14 International Business Machines Corporation System and method for composing heterogeneous media components into a unified environment for rich spatio-temporal hotlink authoring and action enablement in low-bandwidth presentations
US20020122067A1 (en) 2000-12-29 2002-09-05 Geigel Joseph M. System and method for automatic layout of images in digital albums
US7143083B2 (en) 2001-06-12 2006-11-28 Lucent Technologies Inc. Method and apparatus for retrieving multimedia data through spatio-temporal activity maps
US20030192049A1 (en) 2002-04-09 2003-10-09 Schneider Tina Fay Binding interactive multichannel digital document system
US7062712B2 (en) 2002-04-09 2006-06-13 Fuji Xerox Co., Ltd. Binding interactive multichannel digital document system
US7180516B2 (en) * 2003-03-15 2007-02-20 Oculus Info Inc. System and method for visualizing connected temporal and spatial information as an integrated visual representation on a user interface
US7499046B1 (en) * 2003-03-15 2009-03-03 Oculus Info. Inc. System and method for visualizing connected temporal and spatial information as an integrated visual representation on a user interface
US20050012743A1 (en) * 2003-03-15 2005-01-20 Thomas Kapler System and method for visualizing connected temporal and spatial information as an integrated visual representation on a user interface
US20040186723A1 (en) * 2003-03-19 2004-09-23 Fujitsu Limited Apparatus and method for converting multimedia contents
US7702996B2 (en) * 2003-03-19 2010-04-20 Fujitsu Limited Apparatus and method for converting multimedia contents
US7231144B2 (en) 2003-08-22 2007-06-12 Seiko Epson Corporation Element layout apparatus, element layout program and element layout method
US20050071783A1 (en) * 2003-09-30 2005-03-31 Atkins C. Brian Automatic photo album page layout
US7743322B2 (en) * 2003-09-30 2010-06-22 Hewlett-Packard Development Company, L.P. Automatic photo album page layout
US20050177593A1 (en) 2004-01-23 2005-08-11 Geodesic Dynamics Dynamic adaptive distributed computer system
US20050286738A1 (en) 2004-05-27 2005-12-29 Sigal Leonid Graphical object models for detection and tracking
JP2006114013A (ja) 2004-10-18 2006-04-27 Microsoft Corp チャート上の自動ラベル配置のためのシステムおよび方法
US20070033632A1 (en) * 2005-07-19 2007-02-08 March Networks Corporation Temporal data previewing system
US20070033612A1 (en) 2005-08-08 2007-02-08 Princeton Server Group, Inc. Method and apparatus for scheduling delivery of video and graphics
US20070171716A1 (en) * 2005-11-30 2007-07-26 William Wright System and method for visualizing configurable analytical spaces in time for diagrammatic context representations

Non-Patent Citations (3)

* Cited by examiner, † Cited by third party
Title
International Search Report and Written Opinion received in counterpart International Patent Application No. PCT/US2008/005842 (date mailed: Jan. 22, 2009).
Tina Schneider et al., "Description and Narrative in Hypervideo," Proceedings of the 34th Hawaii International Conference on System Sciences-2001.
Tina Schneider et al., "Description and Narrative in Hypervideo," Proceedings of the 34th Hawaii International Conference on System Sciences—2001.

Also Published As

Publication number Publication date
JP2011524035A (ja) 2011-08-25
GB201020315D0 (en) 2011-01-12
WO2009136888A1 (en) 2009-11-12
DE112008003854T5 (de) 2011-06-22
GB2473370A (en) 2011-03-09
JP5325977B2 (ja) 2013-10-23
US20110060979A1 (en) 2011-03-10
CN102084337A (zh) 2011-06-01

Similar Documents

Publication Publication Date Title
US8683326B2 (en) Spatiotemporal media object layouts
US10984295B2 (en) Font recognition using text localization
US10699166B2 (en) Font attributes for font recognition and similarity
US9875229B2 (en) Template-based page layout for web content
CN108122264B (zh) 促进草图到绘画变换
US9824304B2 (en) Determination of font similarity
CN101379485B (zh) 数字标牌系统及为数字标牌网络开发内容的方法
US20160118080A1 (en) Video playback method
US6919910B2 (en) Apparatus and method for distributing representative images in partitioned areas of a three-dimensional graphical environment
US20090265611A1 (en) Web page layout optimization using section importance
US20100058213A1 (en) Display controlling apparatus and display controlling method
WO2011053602A1 (en) Arranging graphic objects on a page
CN111415396A (zh) 一种图像生成方法、装置和存储介质
US20070188520A1 (en) 3D presentation process and method
CN113625863A (zh) 自主式导览虚拟场景的创建方法、系统、设备和存储介质
CN108780377A (zh) 使用计算设备的对象管理和可视化
DE60220082T2 (de) Vorrichtung und verfahren zum entwerfen des layouts einer dreidimensionalen graphischen umgebung
US20150149875A1 (en) Image processing device, image processing device control method, program, and information storage medium
US8988423B2 (en) Electronic album generating apparatus, stereoscopic image pasting apparatus, and methods and programs for controlling operation of same
US10482173B2 (en) Quality distributions for automated document
US8687876B2 (en) Stereoscopic image pasting system, and method and program for controlling operation of same
US8933999B2 (en) Stereoscopic image display control apparatus, and method and program for controlling operation of same
US20240249387A1 (en) Information-processing device, information-processing method and non-transitory computer-readable information recording medium
CN104391829A (zh) 一种实现多媒体信息与文本混排的方法及装置
CN115082702A (zh) 图像和电商图像处理方法、设备及存储介质

Legal Events

Date Code Title Description
AS Assignment

Owner name: HEWLETT-PACKARD DEVELOPMENT COMPANY, L.P., TEXAS

Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNOR:O'BRIEN-STRAIN, EAMONN;REEL/FRAME:027502/0863

Effective date: 20080429

STCF Information on status: patent grant

Free format text: PATENTED CASE

FPAY Fee payment

Year of fee payment: 4

FEPP Fee payment procedure

Free format text: MAINTENANCE FEE REMINDER MAILED (ORIGINAL EVENT CODE: REM.); ENTITY STATUS OF PATENT OWNER: LARGE ENTITY

LAPS Lapse for failure to pay maintenance fees

Free format text: PATENT EXPIRED FOR FAILURE TO PAY MAINTENANCE FEES (ORIGINAL EVENT CODE: EXP.); ENTITY STATUS OF PATENT OWNER: LARGE ENTITY

STCH Information on status: patent discontinuation

Free format text: PATENT EXPIRED DUE TO NONPAYMENT OF MAINTENANCE FEES UNDER 37 CFR 1.362

FP Lapsed due to failure to pay maintenance fee

Effective date: 20220325