JP7317189B2 - automated media publishing - Google Patents

Description

Cross-citation to related applications
[001] The present application is based on 35 U.S.C. S. C. claiming priority and rights under § 120, 20
This is a continuation of pending US patent application Ser. No. 10/025,043 filed Jul. 2, 18. By reference herein to this patent application, its content is also incorporated herein.

Prior art

[002] Media is consumed worldwide in theaters, offices, homes, and many other venues. Media is viewed on large fixed screens, on handheld mobile devices, and on a wide range of intermediate platforms. As the number of media consumption preferences and platforms increases, a given item of media content will be delivered in versions and formats appropriate for each of the venues and platforms on which the media is intended to be delivered to consumers. There is an increasing demand for In addition, the means of publication and distribution have also diversified. As a result, a given media project may call for dozens or even hundreds of different deliverables.

[003] Today's media publishing workflows are time consuming, complex, error prone and ad hoc. Those involved in media creation could have dealt with such inefficiencies when only a small number of versions, renditions, and consumption vehicles were required; became unwieldy and too costly as the deliverables for a given media project became diverse. Additionally, creative editors involved in the story-telling stage of media creation do not have the ability to express their creative choices in the downstream rendering process. Therefore, media creation and publishing pipelines need new ways to extend content, improve the technical quality of media deliverables, and further boost productivity and control costs.

[0004] Generally, the methods, systems, and computer program products described herein enable rationalization and automation of portions of media creation and publishing workflows. At any stage of the editing process, editors are provided with the means to tag composite objects with rules. A rule should determine what rendering decisions are made to adhere to the editor's rendition-related choices while producing the best possible quality output rendering as specified by the rendition profile. to specify how the input source should be treated. These tags provide rules that are interpreted downstream by the rendition engine when rendering the composition. These rules specify rendition requirements1
Specifies how renditions should be generated in response to a set of profiles.

[0005] In general, in one aspect, a method of editing a media composition comprises, using a media composition application executing on a media composition host system, an editor editing a media composition; The media composition includes a compositing object, the compositing object references a source media asset, specifying editing rendition rules for the compositing object, and associating the editing rendition rule with the compositing object of the media composition. and when a rendition of a given media composition is generated, the rendition engine generates a rendition of the given media composition and the editor-specified given media composition composition object and associated editing rendition rules, and for a rendition of a given media composition, a rendition profile that specifies media essence encoding parameters, and Inputs the source media assets for a position and renders a given media composition according to the given composition's edit rendition rules and the media essence encoding parameters for the given composition's renditions. generating renditions of a given media composition from the source media assets.

[0006] Various embodiments include one or more of the following features. A media composition application includes a rendition engine. A rendition engine is hosted on a platform external to the media composition host. A rendition profile specifies the version of a given media composition that should be used to generate a rendition of the given composition. The editing rendition rules specified by the editor and associated with the composite object of a given media composition are the spatial editing rendition rules and have at least the media essence encoding parameters specified by the rendition profile. One is the spatial rendition parameter. The spatial rendition parameter is a framing box that defines a portion of a video frame for a composite object of a given media composition, and the rendition engine is a The first portion of the source media asset is preferentially included in the rendition. If the target display aspect ratio for the rendition is greater than the framing box aspect ratio, the rendition engine may further Portions are preferentially included in renditions, with a second portion of the source media assets of a given media composition containing media essence data. The editing rendition rules specified by the editor and associated with the composite objects of a given media composition are temporal editing rendition rules and have at least the media essence encoding parameters specified by the rendition profile. One is the temporal rendition parameter. The temporal rendition parameters include a temporal framing range that bounds the time within the source media assets of a given composition, and the rendition engine logic determines that a given composition exists within the temporal framing range. The time portion of the source media asset of the media composition is preferentially included in the rendition. The editing rendition rule specified by the editor and associated with the composite object of a given media composition is the dynamic range editing rendition rule, and the media essence encoding parameters specified by the rendition profile is a dynamic range rendition parameter. The source media asset for a given composition is a video asset and the dynamic range rendition parameter specifies the range of pixel brightness values within the source media asset for the given composition. A legal dynamic range that the rendition engine defines that causes pixels of the source media asset of a given composition that have luminance values that lie within the legal range to be preferentially included in the rendition. A given composition's source media asset is an audio asset, and the dynamic range rendition parameter defines the range of audio sample intensity values within the given composition's source media asset Legal dynamic range, so that the rendition engine prefers audio samples of source media assets for a given composition that have intensity values that lie within the legal range to be included in the rendition . The editing rendition rule specified by the editor and associated with the composite object of a given media composition is a color editing rendition rule and has at least the media essence encoding parameters specified by the rendition profile. One is the gamut rendition parameter. Legal color where the source media asset for a given composition is one of a graphics asset and a video asset, and the gamut rendition parameter defines a multidimensional region within a multidimensional color space • Include a range so that the rendition engine prioritizes pixels of the source media asset for a given composition that have color values that lie within the legal color range for inclusion in the rendition. The editing rendition rules specified by the editor and associated with the composite object of the given media composition are spatial editing rendition rules for the layout of the title text on the frames of the rendition, and at least one A media essence encoding parameter is a spatial framing box, a source media asset for a given composition is a text titling effect, a rendition engine is a rendering Render the text of the text titling effect in the framing box on the frame of the distion. The editing rendition rules specified by the editor and associated with the composition object of a given media composition are composition editing rendition rules, and at least one media essence encoding parameter is a spatial framing box , where the source media asset of a given composition is a compositing effect, and the rendition engine renders frames of the rendition with the composite image into a framing box. The rendition profile and editorial rendition rules specified by the editor and associated with the composite object of a given media composition are received by the rendition engine in standardized form, and the rendition engine , which contains the generic logic for executing the rules. From editorial rendition decisions that the rendition engine uses deep learning methods to capture from a cohort of editors working with a wide range of media content, source media types, and rendition requirements Contains learned logic.

[0007] In general, in another aspect, a computer program product includes non-transitory computer readable storage having computer program instructions stored therein which, when executed by the computer, cause a computer to: , to execute a method for editing a media composition. The method uses a media composition application running on a media composition host system to allow an editor to edit a media composition, the media composition to include a composition object, the composition object to represent a source media asset. , specifying editing rendition rules for the compositing object and enabling associating the editing rendition rules with the compositing object of the media composition; When generated, a rendition engine receives a given media composition and editing rendition rules specified by an editor and associated with composite objects of the given media composition, and generates a given For renditions of a media composition, take a rendition profile that specifies media essence encoding parameters, input source media assets for a given composition, and edit a given composition Renders a rendition of a given media composition from the given media composition's source media assets according to the rendition rules and the media essence encoding parameters for the given composition's renditions. and generating.

[0008] In general, in another aspect, a media composition editing system includes a memory for storing computer readable instructions and a processor coupled to the memory, the processor operable to perform computer readable Execution of the instructions instructs the system to edit a media composition using a media composition application running on the system, the media composition containing the composition object, the composition object containing the source media asset. to specify edit rendition rules for a compositing object, and to allow associating edit rendition rules with compositing objects of a media composition such that a rendition of a given media composition is generated. When the rendition engine receives a given media composition and the editing rendition rules specified by the editor and associated with the composite object of the given media composition, the rendition engine renders the given media composition For renditions of a composition, take a rendition profile that specifies media essence encoding parameters, input source media assets for a given composition, and edit renditions for a given composition Generate a rendition of a given media composition from a given media composition's source media assets according to rules and media essence encoding parameters for the given composition's rendition .

[0009] In general, in another aspect, a method of generating a rendition of a media composition comprises receiving a media composition, the media composition referencing a source media asset. for a rendition of a media composition, a step comprising a composition object, wherein the composition object is associated with specified editing rendition rules by an editor of the media composition using a media composition application; Receiving a Rendition Profile that specifies Media Essence Encoding Parameters; From a Source Media Asset according to the Editorial Rendition Rules and the Media Essence Encoding Parameters for the renditions of the Media Composition; , and generating a rendition of the media composition.

[0010] Various embodiments include one or more of the following features. The media composition has multiple versions, and the rendition profile further specifies which of the multiple versions of the media composition should be used to generate the rendition of the composition. The editing rendition rules are spatial editing rendition rules, and at least one of the media essence encoding parameters specified by the rendition profile is a spatial rendition parameter. The spatial rendition parameter is a framing box that defines a portion of a video frame of a composite object of the media composition, and the first portion of the composition's source media assets that resides within the framing box is , to be included in renditions with priority. If the target display's aspect ratio for the rendition is greater than the framing box's aspect ratio, the second portion of the composition's source media asset that is outside the framing box is also given preference to the rendition. A second portion of the source media assets included in the media composition includes media essence data. The editorial rendition rule is a temporal editorial rendition rule, and at least one of the media essence encoding parameters specified by the rendition profile is a temporal rendition parameter. The time rendition parameters include a time framing range that defines a range of time within the source media assets of the composition, and the time within the source media assets of the media composition that falls within the time framing range. parts are preferentially included in renditions. The editing rendition rule is a dynamic range editing rendition rule, and at least one of the media essence encoding parameters specified by the rendition profile is a dynamic range rendition parameter. The composition's source media asset is a video asset, and the dynamic range rendition parameter is the legal dynamic range that defines the range of pixel luminance values within the composition's source media asset , pixels of the composition's source media assets with luminance values that lie within the legal range are preferentially included in the rendition. The composition's source media asset is an audio asset, and the dynamic range rendition parameter is the legal dynamic range that defines the range of audio sample intensity values within the composition's source media asset and audio samples of the composition's source media assets with intensity values that lie within the legal range are preferentially included in the rendition. The editing rendition rules are color editing rendition rules, and at least one of the media essence encoding parameters specified by the rendition profile is a gamut rendition parameter. The composition is one of a graphics asset and a video asset, the color gamut rendition parameter includes a legal color range that defines a multidimensional region within a multidimensional color space, and the legal color Pixels of the composition's source media assets with color values that lie within the range are preferentially included in the rendition. the editing rendition rules are spatial editing rendition rules for layout of title text on frames of the rendition; at least one media essence encoding parameter is a spatial framing box; The media asset is a text titleing effect, and the text of the text titleing effect is rendered within the framing box on the frame of the rendition. the editing rendition rule is a composition editing rendition rule, the at least one media essence encoding parameter is a spatial framing box, the source media asset of the composition is a compositing effect, and framing • A frame of the rendition is rendered with the composite image inside the box. Generating includes interpreting the edit rendition rules using generic logic to execute the rules. Logic in which the generating step uses deep learning methods to learn from editorial rendition decisions captured from a collective of editors working with a wide range of media content, source media types, and rendition requirements including the step of performing

[0011] In general, in another aspect, a computer program product includes non-transitory computer readable storage having computer program instructions stored thereon, wherein when the computer program instructions are processed by a computing system, a , directs a computing system to perform a method of generating a rendition of a media composition. The method includes receiving a media composition, the media composition including a composition object that references source media assets, the composition object being processed by an editor of the media composition in a media composition application. receiving a rendition profile that specifies media essence encoding parameters for a rendition of a media composition associated with a specified editing rendition rule using generating renditions of the media composition from the source media assets according to the rendition rules and media essence encoding parameters for the renditions of the media composition.

[0012] Various embodiments include one or more of the following features. A computing system is remote from where the media composition is received. A computing system is implemented in the cloud.

[0013] In general, in another aspect, a media synthesis rendition engine includes a memory for storing computer readable instructions, and a processor coupled to the memory. The processor executes the computer readable instructions to cause a media composition rendition engine to receive a media composition, the media composition including a composition object that references source media assets, the composition object that references the media A rendition template that is associated with a specified editorial rendition rule by a composition editor using a media composition application and that specifies media essence encoding parameters for a rendition of the media composition. Subjecting the profile to generating a rendition of the media composition from the source media asset according to the editorial rendition rules and the media essence encoding parameters for the rendition of the media composition.

[0014] Various embodiments include one or more of the following features. A media composition rendition engine is located remotely from where the media composition is received. A media synthesis rendition engine is implemented in the cloud.

FIG. 1 is a high-level block diagram illustrating the described media publishing workflow. FIG. 2 shows a media composition tagged with editorial rendition rules. FIG. 3 shows a set of rendition profiles for a given composition. FIG. 4A shows the generation of multiple renditions from one composition. FIG. 4B illustrates rendition generation in a workflow that includes multiple versions of a composition. FIG. 5 shows three boxes defining spatial rendition parameters. FIG. 6 shows how two different spatial rendition rules cause different renditions to be generated for two renditions with the same spatial parameters in their rendition profiles. FIG. 7 shows an idealized timeline showing temporal rendition parameters. FIG. 8 shows the distinction between time range and time resolution. FIG. 9 shows dynamic range rendition parameters. FIG. 10 is a CIE chromaticity diagram showing color space rendition parameters. FIG. 11 is a high level block diagram showing a system for automated media publishing. FIG. 12 is a high-level block diagram illustrating an automated media publishing system with multiple rendition engines.

[0028] The systems and methods described herein provide a new paradigm for generating a set of media deliverables derived from a given composition or set of related compositions. offer. It describes the rendering rules inserted during the editing stage and the workflow by which compositions are tagged. Rules are automatically or semi-automatically interpreted downstream when a deliverable media-based product is rendered. A typical end-to-end media workflow consists of (i) creating a composition; creating a family of versions based on one or more compositions with diversity in content catering; (iii) renditions directly from the family of versions or, if no versions are generated, from the composition; and (iv) generating and distributing a deliverable media package from the renditions. The focus here is on optimizing and automating the rendering of the published media product, ie the generation of renditions, stage (iii) in the workflow. As used herein, a rendition is a "flattened" rendering of a media composition in a form that can be consumed or played by a target platform. Common rendition formats include Audio Interchange File Format (.AIFF), WAVE Audio File (.WAV), MP3 Audio File (.MP3), MPEG Movie (.MPG), Apple QuickTime Movie (. MOV), Windows, Media Video File (.WMV), Material Exchange Format (MXF), Interoperable Mastering Format (IMF), and Digital Cinema Package (DCP).

[0029] In general, a given media composition needs to be adapted and optimized for multiple contexts in which it is intended to be consumed. Previously, this was done by a transcoding process. The transcoding process involves spatial, temporal and color reformatting and compression of a high quality flattened file, commonly referred to as a "master". A master represents the final mix where multiple tracks and effects are mixed into one media stream. When transcoding from a master, the quality of the resulting program is limited to that of the master. This is because, at best, the transcoding process only preserves the quality of the master, and often degrades it.

[0030] The term "master" is also used when referring to a distribution master. A distribution master is a package assembled for distribution to various media outlets and businesses. The package contains multiple alternatives for aspects such as language, program length, and target display parameters. The media (ie, video and audio) selections contained within the distribution master are mixed-down files generated by transcoding the original master into the required format. That is, such a distribution master can contain multiple alternative tracks, each of which has been "flattened" rather than as a programmatic composition element defined with respect to the original source asset. Included as a substitute. In fact, the Source Assets are not even included in the distribution package. As an example, a distribution master created for Eastern Canada would include dialog tracks in both English and French, and video files formatted for smart phones and HD televisions. Digital Cinema Package (DCP) and Interoperable Mastering Format (IMF) are two examples of distribution master types.

[0031] In contrast, the methods and systems described herein apply the irreversible application of editorial decisions to the publication of a media product, where the specific requirements of the product are known and the product is produced. postpone until Since the original edited composition and original source assets are made available, the application of rendition choices such as track mixing, compositing, and effects, rather than a mix-down version Allows to run against the original media source. For example, to produce a 4K resolution output after editing 4K source media in HD mezzanine format, the traditional method is to resample the edited HD master to produce a 4K output. , resulting in lower quality than the original. In contrast, using the methods described herein, the deliverables are generated directly from the 4K source material after editing is complete. Text layouts such as titles or closed captions provide other examples. With existing methods, if the text does not fit in the requested destination screen size, the only options available are to scale or crop the entire image, but doing so renders the text illegible, or cropped out of the artifact
of) there is a risk. With the approach described here, the original media and titling effects are still available even when the destination screen size is specified, so perhaps the text is If it doesn't fit, run the inscription effect again and the text is laid out so that it fits best on the destination screen. Yet another example involves color correction. If final output to a high dynamic range (HDR) capable monitor is specified, an 8-bit or 10-bit master cannot supply the full luminance range available for display. However, since the original media composition and source media assets are still available, the full dynamic range of the original can be included in the distributed rendition.

[0032] To enable such a workflow, additional functionality is added to the editing environment by which editors specify rules and corresponding parameters that define how renditions are to be generated, and how these rules are defined. with a media composition, and more particularly with a compositional object of the media composition. The rules are exposed to automatic rendering software when renditions are generated from the composition. The methods described herein provide editors with a mechanism for verifying their ability to embody the stories they want to tell in the media. That is, their full creative intent is automatically captured downstream when renditions are generated, rather than being recreated through laborious manual sub-editing. Such rules are called "editorial rendition rules". Because they are essentially editorial in nature, they must be distinguished from media format rendition rules. The latter serves to optimize the structure and format of media files, as well as technical media quality, by specifying the conditions under which various codec parameters are applied when generating renditions. In existing workflows, common format decisions are specified by people downstream in the creative editing and assembly process. In some cases, format determination and encoding may be done manually. The methods described herein support automation of this stage of the workflow as well. However, media format rules and parameters are supplied as part of the Rendition specification and are provided within the Rendition Profile (described below) rather than within the Media Composition.

[0033] The methods described herein serve to enhance the ability of media publishers and distributors to monetize their assets by providing the ability to reuse stored assets. Creative judgment is made on the original source media assets and stored in the archive as metadata associated with these assets, so that the full range of media scope and quality is It remains available for generating renditions at that point in time. This applies to the spatial domain (original large raster), temporal domain (original frame rate), and color domain (original dynamic range).

[0034] Figure 1 is a high-level diagram illustrating the described media publishing workflow and the data objects involved in this process. Source media assets 102 such as video clips, audio clips, computer-generated images, and special effects provide sources of material that is used by editors 104 using media editing applications 106. . Media editing application 106 may be a non-linear video editor, digital audio workstation (DAW), or special effects application for creating composition 108 . Examples of such applications include Avid® Technology, Inc., both of Burlington, Massachusetts. , MEDIA CO from
MPOSER®, a nonlinear video editor, and PRO TOOLS®, DAW, and AFTER-EFFECTS® visualization, motion graphics, and compositing applications from Adobe® Systems, Inc. included. The composition 108 is represented by multiple tracks containing clips of the source media assets 102 arranged in a row, along with rich metadata specifying editing decisions, parameters, and settings. This composition contains all source media assets, references to effects, and metadata needed to play or render this composition. Such an "unflattened" composition is a form that allows for further editing, and can be compared with the traditional flattened file "master" cited above, or distribution masters such as DCP and IMF. must be distinguished. The composition structure varies from application to application. For example, MEDIA COMPOSER calls a video composition a "sequence" and includes settings, bin files, and clips, and sequence metadata includes references to sources. PRO TOOLS refers to an audio composition as a "session", which contains metadata that specifies audio editing and mixing decisions and settings, as well as references to source assets such as audio files and effects. . graphics and compositing
A composition, represented as a scene graph, specifies the editing decisions applied to a graphics source asset.

[0035] As mentioned above, the editing phase is extended beyond its traditional scope by allowing editors to specify editing rendition rules 110 . When an editor specifies an editorial rendition rule, the media compositing application attaches this rule, along with the values of any parameters it may require, to the appropriate compositional object in the composition. FIG. 2 shows a composite data model for a composition 202 containing three tracks, each track containing one or more clips. Tracks and clips are examples of composite objects. Track 1 204 is associated with editing rendition rule A 206 and Clip 1 208 is associated with editing rendition rule 210 . The step of associating edit rendition rules with a composition object is also referred to herein as tagging the composition object with the rule. A media composition application provides a user interface. This user interface allows editors to create and enter editorial rendition rules and parameters for selected composite objects of the media composition. Once specified by the editor, the editor can tag the composite object with editorial rendition rules, which become part of the media composition, as shown in FIG. Examples of edit rendition rules are described below.

[0036] When one or more renditions of a composition are to be generated, editorial rendition rules 110 apply media delivery requirements specified by rendition profile 114 to generate rendition 116. are interpreted by the Rendition Engine 112 according to the deliverable requirements). The Rendition Engine Logic attempts to produce each Rendition with quality commensurate with its corresponding Rendition Profile. Since the Rendition Engine starts from the original source assets, without resorting to intermediate files, this quality can be less than or equal to the quality of those assets. Also, access to source effects and source media provides a high degree of flexibility as to how effects are applied given the specified parameters.

[0037] Rendition profiles 114 include one or more individual rendition profiles, one for each rendition that is generated. Each Rendition Profile specifies: (i) for workflows that include a version stage, which version to use, (ii) a structural description of the media package to be generated from that version, e.g., an Interoperable Master Format (IMF) ), material exchange formats (MXF) OP-Atom and OP1a, (iii) spatial parameters such as spatial framing boxes, temporal framing
essence encoding parameters, including but not limited to temporal parameters such as range, and color parameters such as dynamic range and color legal ranges; (iv) The distribution method that should be deployed. Additionally, the Rendition Profile may also include one or more editorial rendition rules, possibly already specified by the editor when the rendition requirements were originally defined. These can complement the editing rendition rules embedded within the composition. In addition, certain rendition parameters, such as valid boxes or framing boxes (see below), may be present within the source media asset 102, and according to the editorial rendition rules 110, the rendition • May be given priority over corresponding parameters present in the profile. However, the logic used by the Rendition Engine to interpret parameters and generate renditions is the same no matter what the source of the rendition parameters. FIG. 3 is an exemplary illustration of a set of rendition profiles for a given composition.

[0038] In some workflows, renditions may be generated from media assets that include one or more partially mixed down media files. Such intermediate files are then combined with other original media and/or effects during the rendition phase. In other examples, the background track is flattened but the title track is left untouched during the rendition phase, or a mixed down mono audio track is added just before the rendition. Used in the rendition phase to blend into the dialog in the specified language. That is, some renditioning decisions remain deferred, preserving flexibility in final renditioning, while other decisions are committed early in the workflow. be.

[00391] In the final phase of the workflow, one or more renditions are packaged into a distribution master, which is then distributed to where they are intended to be consumed.

[0040] FIG. 4A illustrates the generation of multiple renditions 406 from a composition 402. FIG. Each rendition is specified by a corresponding member of a set of rendition profiles 404 . FIG. 4B illustrates the generation of renditions in a workflow in which multiple versions of a composition are generated. Each version can then be used as the basis for a set of renditions. A version refers to a variation of a composition, including variations in content and/or timing of particular elements. Other sources or compositions in the form of submasters or encapsulated clips or partial mixdowns can also provide content for the various versions. For example, a set of versions authoring an episode of a continuous program, content from a composition representing the individual content of that episode, and an opening scene reproduced in each episode. It can contain content from a second master that contains the scenes (opening sequence). Multiple versions of a composition can each contain dialog tracks in different languages. In addition to dialog tracks, the transformations required to achieve different language versions can also involve titling, subtitling, and scene selection. Other version types have one or more scenes added or removed based on the intended consumption venue. For example, scenes involving aircraft crashes are removed from the in-flight airline version. The version shown to a general audience may contain different scenes than those intended for an adult audience. Other types of versioning involve the insertion of advertisements within the program, or a range of placeholder lengths for advertisements added at the point of distribution. The set of versions to be generated can be specified by a set of version profiles, and the set of version profiles can be supplied either during composition editing or after it is completed. Referring to FIG. 4B, composition 408 is used as the basis for version A 410 and version B 412 . According to rendition profile A 414, version A is used as the basis for compositing, from which renditions A1, A2, A3, and A4 416 are generated, and version B is used for compositing according to rendition profile B 418. used as a basis from which renditions B1 and B2 420 are generated.

[0041] When generating a rendition according to a given member of a set of supplied Rendition Profiles, the Rendition Engine takes the Media Composition and its Composite Objects (or if versions exist, which version of that composition) and uses it to apply one or more of the rendition logic related to space, time, or color to a given rendering. Converts the source media assets, effects into the specified rendition, calling them according to the editorial rendition rules when applied to the essence encoding parameters supplied by the dition profile. In addition, rendition rules can also allow editors to specify rules to optimize the selection of source media assets when multiple alternative assets are available. Alternate assets may differ from each other in one or more of their spatial, temporal, or color resolutions or ranges. The rendition engine receives the editorial rendition rules from the tagged composition itself, and the rules' corresponding parameters from one or more of the compositing rules themselves, the rendition profiles, and the source media assets. receive from

[0042] Three categories of rendition rules and their parameters, spatial, color, and temporal parameters, will now be described. Spatial parameters relate to the x,y extent within a video frame. As shown in Figure 5, three boxes define the spatial parameters. For video frame 502, essence box 504 defines the extent of the image in which the sample resides. This can include black samples previously introduced as padding. Valid box 506 delimits pixels that contain image content, as opposed to pixels previously introduced for padding. One or more framing boxes defined by the editor constitute the editor's assessment of where the most important parts of the image are located. The editor uses framing boxes to define safe action areas, safe title areas, or to tell the system where actors are in the image. The framing box is invoked according to spatial type editorial rendition rules to determine which parts of the frame should be given priority for inclusion in the rendition. Other rules constrain the outline of the framing box to be used, for example, depending on the aspect ratio of the target playout display specified by the rendition profile. FIG. 5 shows framing box 508 . This illustrates an example where the aspect ratios of framing box 508 and target screen 510 are the same and the entire extent defined by the framing box is contained within rendition frame 512 . Framing boxes are editorial in nature, unlike essence and effect boxes that are defined within a source media asset. The media composition application used by the editor provides the user interface. This user interface allows the editor to specify framing boxes for individual frames, clips, or other portions of the composition. In an exemplary user interface, the application allows an editor to draw a framing box on the display of video frames displayed in a playback window within the application's user interface.

[0043] Spatial rendition parameters are independent of resolution. That is, the framing box defines the portion of the image that the editor wishes to call out in terms of x,y ranges of pixels. Figure 6 shows how two different spatial rendition rules generate different renditions for two renditions that have the same spatial parameters (in this case resolution parameters) in their rendition profiles. The source image 502 has a resolution of 3840 by 2160. The framing box 604 defines x,y ranges of (1000-2920), (540-1620), resulting in varying resolutions. without the H
Obtained in D resolution. In contrast, the full range of source image 602 can be scaled to HD resolution image 608 by simply changing the resolution without changing the extent. The decision to scale or crop (or, more generally, to choose a combination of cropping and scaling) to produce a 1920x1080 HD image is essentially an editorial in nature. . Because the overall quality of the rendered image is more important than whether the source image should be scaled, or if the image is cropped in order to focus the consumer's attention on the image's vital content. This is because it may rely on a qualitative assessment as to whether the loss of quality is less significant. An example of this is when a sub-editor or automated renderer makes poor choices, such as inadvertently cropping the most important parts of a frame. Indicates ambiguity that can arise without the guidance of

[0044] Hinging rules are another type of spatial rendition rule.
Hinging rules specify how interactive elements, typically graphical objects, are laid out on a rendered display. These rules include hinge position parameters. For example, a bar graph, which can be adjusted by a television presenter, can hinge at its base and its vertical extent changes with respect to a fixed hinge position. Other spatial rules and parameters are the placement of displays within a set of displays, such as a video wall, and, for example,
Defines compositing rules for pictures within picture compositing.

[0045] The following is an example of rendition engine logic for using a spatial framing box. If the rendition target display has the same aspect ratio as the framing box, the following rules apply. If the resolution of the target display is the same as the resolution inside the framing box, the rendition engine simply extracts the extent of the framing box within the source media asset for rendition. If the target display has a lower resolution than the framing box, downsample the range of pixels within the framing box to the target resolution. If the target display has a higher resolution than the resolution in the framing box, the range of pixels included in the rendition is spread symmetrically around the framing box in the source media asset. If such expansion results in the rendition box reaching the valid box, the additional resolution required beyond the point at which the valid box is reached is obtained by upsampling the range limited by the valid box, Or the target display is allowed to contain non-essence parts that are outside the valid box but inside the essence box. In the latter case, if the essence box limit is also reached before the target resolution is reached, the remaining required resolution is obtained by upsampling the range bounded by the essence box.

[0046] If the rendition target display has a higher aspect ratio than the framing box aspect ratio (ie, a higher ratio of x resolution to y resolution), the following rules apply. Expand the x-extent of the rendition box symmetrically on each side of the framing box without changing the y-resolution until the target aspect ratio is reached. If the resulting resolution matches that of the target display, this becomes the rendition box. If the target display has a lower resolution, downsample the boxes to match the target. If the valid box is reached before the target aspect ratio is obtained, then if the valid box is reached at only one end of the range, then extend the range within the valid box at the other end of the range. If both ends of the x-range reach the active box before the target x-resolution is achieved, the still required aspect ratio is reduced by including pixels outside the active box but inside the essence box. Gain extra augmentation. If there are insufficient media samples even in the essence box to accommodate the desired aspect ratio, insert additional padding at the bottom and top of the x-range or crop the y-range. Once the desired aspect ratio is reached, the resulting box becomes a rendition box if the resolution meets the requirements, and downsamples if it exceeds the target resolution. Similar logic applies if the target aspect ratio is lower than that of the framing box.

[0047] Text may be laid out using rendition engine logic to take into account the target aspect ratio. To align closed captions or subtitles, the line length of the text can be specified as a percentage, eg, 75% of the horizontal extent of the target display. Similarly, the font size can be scaled to match the resolution available on the target display. Text can interfere with images that they overlap, so the rendition engine logic is bounded by the framing box if bounds are available outside of the framing box on the target display. You can try to avoid or minimize outlying subtitle placement. The renditioning engine then arranges the text anew for each rendition without having to introduce text scaling artifacts that can interfere with readability. Similar rules for the aesthetic layout of text also apply to titles, such as titles displayed at the beginning and end of a video composition, or titles demarcating individual scenes.

[0048] The temporal rendition parameters are similar to the spatial parameters described above. As shown in the idealized timeline of Figure 7, which represents time along the horizontal axis, one full clip corresponding to the temporal essence range 702 is a portion of the original video. may include headers 704 and footers 706, such as countdown clocks and color bars that are not An entire clip that spans the entire essence range is equivalent to the essence box defined above. Valid range 708 excludes headers and footers that were not originally present, but includes head 710 and tail 712 of the clip. These are typically designed for use in transitions, for example in crossfades to the previous or successive clip. Typically such heads and tails comprise 520 video frames. The framing range 714 is the portion of the clip that is within the essence range, excluding both the header 704 and footer 706, and the header 710 and tail 712, which were not originally present. The framing range indicates the time range within which the most important content is present. Additional time framing ranges can also be inserted to give guidance to the hierarchy of content within the clip. For example, a second framing range 716 is specified by the editor from within the clip and used to drop the original clip length to about 50% during on-the-fly editing downstream. Indicates a temporal portion. This allows for automated downstream editing of different length compositions without the risk of losing the most important content. Like the spatial rendition parameters, the temporal essence range and valid range are properties of the source media asset, but the framing range can be specified by the editor. The user interface of the media compositing application allows the editor to mark the framing range by, for example, inserting markers 718 and 720 onto the displayed timeline representation of the clip to indicate the start and end of the framing range. allows you to specify

[0049] The rendition engine logic for temporal rendition parameters is similar to that for spatial framing boxes. If the clip's target duration specified in the rendition profile is the same as that specified by the framing range, then that portion of the clip is included in the rendition. If a duration longer than the duration of the framing box is requested in the rendition, additional content beyond the framing range but within the valid range is added. In a specific use case, the duration of a broadcast rendition may depend on the number of ad-like inserts being broadcast. This is because the combined duration of renditions and inserts must fit within a given total allotted time. This can be determined just before show time, making automatic rendering with already created editorial rendition rules particularly valuable. Supplemental framing ranges (eg, 716 in FIG. 7) are represented by the length shortening of the framing range that the rendition engine specifies in the edit, such as shortening by 30 seconds, 1 minute, or 2 minutes. allows content to be cut according to various predicted insert durations. In other use cases, as part of the final edit, new effects are applied that require duration extensions, such as in fade transitions. Then, in this situation, the additional content extends beyond the framing box and into the effective range. Extending the duration, such as frame duplication, padding the tail of the clip with graphics or black frames, if the duration extension cannot be fully supplied by the available range and no additional material is present in the source media. Other techniques can also be used for

[0050] The distinction between temporal range and temporal resolution, ie video frames per second (FPS) or audio samples per second, is shown in FIG. FIG. 8 shows two video clips 802 and 804 having the same time range but different temporal resolutions. Clip 802 has twice the temporal resolution of clip 804 . For example, 50 FPS versus 25 FPS.

[0051] For some renditions, the requested frame rate deviates from that of the source media asset. For example, a film frame rate of 24 frames per second (FPS) may need to be rendered at a video frame rate of 25 FPS (for PAL encoding) or 29.97 frames per second (for NTSC encoding). . When performing such transformations, the editing rendition rules determine whether to preserve the duration of the clip or whether to allow the duration of the clip to be changed. When trying to preserve the duration of a clip when converting to a higher frame rate, either by interpolation methods, e.g., by 3:2 pulldown, or by adding extra content to the clip. You get an extra frame. If you are trying to preserve the duration of a clip when converting to a lower frame rate, remove frames from the clip, optionally using the removed frames to adjust/blend the retained frames. scales the clip to fewer frames. If the duration of the clip is allowed to vary, the speed of the video can be changed, ie, slowed down or sped up somewhat, while retaining the number of frames in the clip content. When rendering a 24FPS source to a 25FPS rendition, a simple tempo scaling is usually considered acceptable, resulting in a slight shortening corresponding to the clip duration, but rendering a 24FPS source When rendering to the 29.97 FPS rendition, simple scaling results in excessive speedup. Therefore, in this situation, we insert a new interpolated frame to preserve tempo and duration.

[0052] Color rendition rules and their associated parameters are
Instruct the rendition engine to select the portion of the range that should be mapped to the target display, or the portion of the multi-dimensional color range that should be mapped to the target display in the video encoding. For example, the legal range specified by the compilation may correspond to the color range that can be displayed on an SD or HD television receiver. The legal range specified by other compilations may correspond to a wider color range that can be displayed on an HDR display.

[0053] The first set of color parameters defines the dynamic range. The essence range corresponds to the range of values that the pixel value encoding of the image can take. This is a function of the bit length, ie the number of bits allocated to represent each pixel. The valid range corresponds to the range of values in which the media sample resides. The range of actual encoded values in an image is the legal range, which is usually a subset of the essence range.
reach the range. The legal range, corresponding to the framing range defined above, is the range of encoded values that the editor maps to the range between the reference black and white points, often denoted 0 to 100% RE.

[0054] Figure 9 illustrates the dynamic range parameter on a histogram plot 902 of pixel values for a given video frame. Essence range 904 spans the entire range that can be represented by the encoding scheme in which the frame is represented. Valid range 906 spans the extent to which the essence range is populated by the pixel values of the frame, while legal range 908 spans the reference black and white points, i.e., the IRE from 0 to 100%. Indicates the range of pixel values that are mapped to ranges that span ranges. The legal range is the broadcast safe range, i.e. standard definition (SD) or high definition (HD) television, as illustrated in FIG. Any range that can be represented on the John monitor can be accommodated. When creating a color editing rendition rule, the editor can specify a legal range to indicate what portion of the source dynamic range is to be rendered. For example, for a video composition, if the editor wants to show subtle details in the shadows of the frame and not in the more brightly lit areas, the editor may change the legal range to the effective range. Slide towards the bottom edge. Additional legal ranges can also be specified to accommodate high dynamic range monitors. Such monitors are rated at 4,000% IRE or 10
,000% IRE in some cases. To specify what should be mapped on such a display, a correspondingly wider legal range is specified by the editor from within the valid range. The user interface of the media compositing application allows the editor to specify the legal range, for example by sliding lines 912 and 914, and the lower and upper limits of the legal range, respectively, to the pixel value histogram. Denote left or right on plot 902 . A standard dynamic range corresponding to the display device standard may be selected, ie the separation of lines 912 and 914 may be adjusted to correspond to such standard, eg 0-100% IRE (SD, HD) Or it can be constrained to correspond to 0-4,000% IRE (HDR).

[0055] A second set of rendition-related color parameters defines a color gamut in a multidimensional space. Like dynamic range, essence range corresponds to the range of values that can be represented by the encoding used to represent the image data, but in this case a multi-dimensional color space such as RGB or YCBCR. The valid range corresponds to the range of color space values in which media samples exist in the represented color image. That is, the essence range can contain values that were not sampled from the original scene. For example, if 16-bit encoding is used, not all possible 16-bit code words will be captured by the camera.

[0056] The legal range is the range determined by editing, a subset of the valid range, which part of the color space should be selected for representation in the rendition when the entire valid range cannot be displayed. reflect the editor's choice about FIG. 10 is a Commission on Illumination (CIE) chromaticity diagram showing an example of color space parameter selection used by color editing rendition rules. The essence range is contained within outer envelope 1002 . The outer envelope 1002 spans all color values represented by the image encoding scheme and depends on the multidimensional bit depth of the pixel representation. The valid range 1004 is a subset of the essence range and corresponds to the International Telecommunications Union (ITU)-RBT2020 color gamut 1004 and the editor-specified legal range corresponds to the ITU-709 color gamut 1006. The user interface of the media composition application, for example, allows the editor to plot the standard color gamut (ITU-709 (1006 ) or allows the envelope of this color gamut to be drawn, allowing the editor to specify the legal color gamut range. In this way, editors can specify where to concentrate color resolution on output displays that cannot faithfully display the entire BT2020 color gamut.

[0057] In a manner similar to that described above for spatial and temporal editing rendition rules, color rendition rules and their parameters are used by the automated rendition engine to rendition according to rendition profiles. determines whether to generate For example, when the target color space legal range specified for a rendition occupies a smaller color space volume than the source media essence range, the edit rendition rule will apply the essence range Determine whether to crop the volume to the legal range volume, or whether to shrink the pixel values in the source media to fit within the legal range volume. Conversely, when the target color space legal range occupies a larger volume than the source media essence range, the rendition rule expands the essence range volume and uses the inner color space to avoid aliasing artifacts. Specifies whether to use interpolated values or whether to use value ranges.

[0058] In addition to specifying spatial, temporal, and color parameters, a rendition profile may also include auxiliary parameters required to fully specify a rendition. Auxiliary parameters include codec used, bandwidth intended for streaming distribution, file wrapper type, digital rights management information, and slate. For example, one rendition may be specified as 1080P in the MXF file wrapper and distribution rights as US only. In general, such auxiliary rendition parameters are required for the complete specification of the rendition to be generated, but are irrelevant to the edit in nature and are therefore not specified by the edit for these parameters. It is not necessary to provide an upstream editing interface for tagging media with values that

[0059] Distribution channels for consumption of the renditions described herein include theatrical, broadcast, cable, satellite television and radio, computers, tablets, and over-the-air programming streamed to smartphones. the-top programming), and even live streams. Consumption environments include entertainment and educational settings, public spaces or private institutional and corporate settings. These environments may be located in one or more geographic regions. Each channel, environment, and geography can create specific requirements for media consumption and is encapsulated in a correspondingly specially crafted Rendition Profile.

[0060] Referring again to FIG. In other embodiments, the rendition engine includes a generic logic engine. This overarching logic engine takes as input editorial rendition rules 110 and executes the rules to generate renditions and effects from source media assets 102 according to rendition profiles 114 (shown in FIG. 1). It has not been). In various embodiments, the rendition engine includes artificial intelligence techniques. Artificial intelligence techniques involve learning from editorial rendition decisions by a collective of editors for a wide range of synthetic content, source media types, and rendition requirements. How deep learning methods deploy multi-level neural networks to select rules and parameters, and sources to optimize the rendered output with respect to the available source media can decide whether to

[0061] Figure 11 is a high-level block diagram illustrating a system for automated media publishing. Media composition host system 1102 hosts media composition application 1104 including composition 1106 and rendition engine 1108 . One or more editors 1110 edit the composition. A composition includes specifying editing rendition rules 1112 and tagging one or more composite objects of composition 1106 with the rules. A rendition profile 1114 is provided to the media composition host system. When a rendition specified by rendition profile 1114 is to be generated, rendition engine 1108 accesses source media asset 1116 from storage 1118 and generates output rendition 1120 for output to storage 1122. . In the illustrated embodiment, the rendition engine 1108 is implemented as part of a media composition application and has access to the media composition application's media player software. A media compositing application can read the composition in its original, unflattened form.

[0062] In another embodiment, shown in Figure 12, the rendition engine is implemented in one or more platforms external to the media synthesis host system. In these embodiments, the rendition engine includes functionality similar to media player software used by media compositing applications so that the rich composition can be read in its original, unflattened form. can. Referring to FIG. 12, which illustrates the use of two rendition engines external to the media composition host system, the media composition host system 1202 includes a composition 1206 tagged with editorial rendition rules 1208, media It hosts the compositing application 1204 . When a rendition is about to be generated, Rendition Engine 1 1210 and Rendition Engine 2 1212 receive the Rendition Profile 1214 and Composition 1206 and, starting with the Source Media Asset 1218, create a Generate renditions 1216 specified by profile and edit rendition rules 1208 . The rendition engine outputs renditions for storage in storage 1220 . The external platform on which the Rendition Engine is implemented may be co-located with the host 1202, for example connected by a local area network, or by one or more remote servers, or by a private cloud. Or it may be hosted by a public cloud.

[0063] Various components of the systems described herein can be implemented as computer programs using a general-purpose computer system. Such computer systems typically include a main unit connected to both an output device for displaying information to a user and an input device for receiving input from the user. A main unit generally includes a processor connected to a memory system via an interconnection mechanism. Input and output devices are also connected to processors and memory systems via interconnection mechanisms.

[0064] One or more output devices can be connected to the computer system. Examples of output devices include liquid crystal displays (LCDs), plasma displays, various stereoscopic displays including displays that require viewer glasses and displays that do not require glasses, cathode ray tubes, video projection systems, and other video displays. Devices for communicating over low or high bandwidth networks including, but not limited to, output devices, printers, network interface devices, cable modems, and storage devices such as disks or tapes. not One or more input devices can be connected to the computer system. Examples of input devices include, but are not limited to, keyboards, keypads, trackballs, mice, pens and tablets, touchscreens, cameras, communication devices, and data entry devices. The invention is not limited to any particular input or output device used in conjunction with the computer system, nor is it limited to those described herein.

[0065] The computer system may be a general-purpose computer system, programmable using a computer programming language, scripting language, or even assembly language. The computer system may also be specially programmed, special purpose hardware. In general-purpose computer systems, the processor is typically an off-the-shelf processor. A general purpose computer also typically has an operating system. The operating system controls the execution of other computer programs and provides scheduling, debugging, input/output control, account management, editing, storage allocation, data and memory management, and communications control and communications-related services ( provide). The computer system may also be connected to local networks and/or to wide area networks such as the Internet. The connected network can transfer media data, such as program instructions, video data, still image data, or audio data to and from the computer system for execution in the computer. Opinion and endorsement information for the composition, media annotations
annotation), as well as other data can be transferred.

[0066] The memory system typically includes a computer-readable medium. The medium may be volatile or non-volatile, writable or non-writable, and/or overwritable or non-writable. Memory systems typically store data in binary form. Such data may define application programs executed by the microprocessor or information stored on the disk for processing by the application programs. The invention is not limited to any particular memory system. Time-based media can be stored and input from magnetic, optical, or solid state devices. These drives may include arrays of local or network attached disks.

[0067] A system as described herein may be implemented in software, hardware, firmware, or a combination of the three. The various elements of this system, either individually or in combination, may be implemented as one or more computer program products, in which computer program instructions are non-transitory for execution by a computer. Stored on a computer readable medium or transferred to a computer system over a local area or wide area network connected. Various steps of a process can be performed by a computer executing such computer program instructions. The computer system may be a multiprocessor computer system, or may include multiple computers connected via a computer network, or may be implemented in the cloud. The components described herein may be separate modules of a computer program, or may be separate computer programs and operable on separate computers. The data generated by these components can be stored in memory systems or can be transmitted between computer systems by various communication media such as carrier wave signals.

[0068] While example embodiments have been described above, it should be apparent to those skilled in the art that the foregoing has been presented by way of illustration only and not limitation. Numerous modifications and other embodiments will come within the purview of those skilled in the art and are considered to fall within the scope of the invention.

Claims (18)

A method of generating a rendition of a media composition, comprising:
receiving the media composition, wherein the media composition includes a composition object that references source media assets; the composition object is created by an editor of the media composition in a media composition application; a step associated with a specified edit rendition rule using
receiving a rendition profile that specifies media essence encoding parameters for a rendition of the media composition;
generating renditions of the media composition from the source media assets according to the editorial rendition rules and the media essence encoding parameters for the renditions of the media composition;
A method, including 2. The method of claim 1, wherein the media composition has a plurality of versions, and the rendition profile further comprises a version of the media composition among the plurality of versions of the media composition. How to specify the version that should be used to generate the rendition. 2. The method of claim 1, wherein the editing rendition rules are spatial editing rendition rules, and at least one of the media essence encoding parameters specified by the rendition profile is a spatial rendition parameter. there is a way. 4. The method of claim 3, wherein the spatial rendition parameter is a framing box that defines a portion of a video frame of the composite object of the media composition, and the source media composition of the media composition. A method, wherein a first portion of an asset that resides within the framing box is preferentially included in the rendition. 5. The method of claim 4, wherein the aspect ratio of the target display for the rendition is greater than the aspect ratio of the framing box, and the framing box among the source media assets of the media composition. is also preferentially included in the rendition, the second portion of the source media assets of the media composition comprising media essence data. 2. The method of claim 1, wherein the editing rendition rules are temporal editing rendition rules, and at least one of the media essence encoding parameters specified by the rendition profile is a temporal rendition parameter. there is a way. 7. The method of claim 6, wherein the temporal rendition parameters include a temporal framing range that defines a temporal range within the source media assets of the media composition, and wherein the source media assets of the media composition. A method wherein temporal portions of a media asset that fall within said time framing range are preferentially included in said rendition. 2. The method of claim 1, wherein the editing rendition rules are dynamic range editing rendition rules, and at least one of the media essence encoding parameters specified by the rendition profile is a dynamic range editing rendition rule. • A method, which is a rendition parameter. 9. The method of claim 8, wherein the source media asset of the media composition is a video asset, and the dynamic range rendition parameter is the source media asset of the media composition. A legal dynamic range defining a range of pixel luminance values within an asset, wherein pixels of the source media asset of the media composition having luminance values lying within the legal dynamic range are prioritized. and included in said renditions. 9. The method of claim 8, wherein the source media asset of the media composition is an audio asset, and the dynamic range rendition parameter is the source media asset of the media composition. A legal dynamic range defining a range of audio sample intensity values within an asset, wherein audio samples of said source media asset of said media composition having intensity values lying within said legal dynamic range. is preferentially included in said rendition. 2. The method of claim 1, wherein the editing rendition rules are color editing rendition rules, and at least one of the media essence encoding parameters specified by the rendition profile is a gamut rendition rule. A method, which is a parameter. 12. The method of claim 11, wherein
the source media asset of the media composition is one of a graphics asset and a video asset;
the gamut rendition parameter includes a legal color range defining a multidimensional region within a multidimensional color space;
A method wherein pixels of the source media assets of the media composition with color values that lie within the legal color range are preferentially included in the rendition. The method of claim 1, wherein
the editing rendition rules are spatial editing rendition rules for layout of title text on frames of the rendition;
at least one media essence encoding parameter is a spatial framing box;
wherein the source media asset of the media composition is a text titling effect;
A method, wherein the text of the text titling effect is rendered within the spatial framing box on a frame of the rendition. The method of claim 1, wherein
the editing rendition rules are composition editing rendition rules;
at least one media essence encoding parameter is a spatial framing box;
wherein the source media assets of the media composition are composite effects;
A method wherein a frame of said rendition is rendered having a composite image within said spatial framing box. 2. The method of claim 1, wherein the generating step includes interpreting the edit rendition rules using generic logic to execute rules. 2. The method of claim 1, wherein the generating step uses deep learning methods to capture from a pool of editors working with a wide range of media content, source media types, and rendition requirements. A method that includes executing logic that learns from edit rendition decisions. A computer program,
comprising non-transitory computer readable storage having computer program instructions stored thereon, said computer program instructions being processed by said computing system to generate a rendition of a media composition in said computing system. command to perform a method of
receiving the media composition, wherein the media composition includes a composition object that references source media assets; the composition object is created by an editor of the media composition in a media composition application; a step associated with an edit rendition rule specified using
receiving a rendition profile that specifies media essence encoding parameters for a rendition of the media composition;
generating renditions of the media composition from the source media assets according to the editorial rendition rules and the media essence encoding parameters for the renditions of the media composition;
A computer program, including A media synthesis rendition engine, comprising:
a memory for storing computer readable instructions;
a processor connected to the memory;
and when the processor executes the computer readable instructions, to the media synthesis rendition engine:
receiving a media composition, said media composition including a composition object that references source media assets, said composition object specified by an editor of said media composition using a media composition application associated with the edited rendition rule and
receiving a rendition profile specifying media essence encoding parameters for a rendition of the media composition;
generating renditions of the media composition from the source media assets according to the editorial rendition rules and the media essence encoding parameters for the renditions of the media composition;
A media composition rendition engine that runs

Images