BR112016001639B1 - SYSTEM FOR GENERATING STABILIZED SCENE METADATA - Google Patents

Abstract

SYSTEM AND METHODS FOR GENERATING STABILIZED SCENE METADATA. The invention relates to methods and systems for generating and applying stable scene metadata to a video data stream. A video data stream is divided or partitioned into scenes and a first set of metadata can be generated for a given video data scene. The first set of metadata can be any metadata known as a desired function of video content (e.g., luminance). The first set of metadata can be generated on a frame-by-frame basis. In one example, stable scene metadata may be generated that may be different from the first set of metadata for the scene. Stable scene metadata can be generated by tracking a desired feature with the scene and can be used to keep the desired feature within an acceptable range of values. This can help avoid noticeable and possibly objectionable visual artifacts when rendering video data.

Description

CROSS REFERENCE TO RELATED ORDERS

[001] This application claims the priority of Provisional Patent Application No. US 61/859,956, filed on July 30, 2013, which is incorporated by reference in this document in its entirety.

TECHNICAL FIELD

[002] The present invention relates to the creation and rendering of video content, and particularly to systems and methods for generating associated metadata to enhance the rendering of video data on target displays.

BACKGROUND

[003] Metadata in a video file is typically generated on a per-frame or key-frame basis. However, in many cases, it is possible for video playback to have artifacts that are objectionable to a viewer of the video content. These artifacts may be noticeable between scenes - for example, for scenes that may have certain common features. For example, the camera may be capturing video of an individual actor moving through space and time - for example, a moment in a darkened room and then moving to a sunlit outdoor space.

[004] Such a change in ambient conditions may cause noticeable artifacts to a viewer (e.g., a change in the aforementioned actor's facial color tones). This can be especially true when video content must be displayed on a target display that may have limitations on its performance - for example, for luminance, gamma rendering, or the like. For a content creator (such as a director or post-production professional), it may be possible to mitigate such artifacts by generating scene-based metadata.

SUMMARY

[005] Methods and systems for generating and applying stable scene metadata to a desired video data stream are disclosed herein. Systems and/or methods are data in which a video data stream is divided or partitioned into scenes and a first set of metadata can be generated for a given video data scene. The first set of metadata can be any metadata known as a desired function of video content (e.g. luminance, color gamut, etc.). The first set of metadata can be generated on a frame-by-frame basis. In one embodiment, stable scene metadata may be generated that may be different from the first set of metadata for the scene. Stable scene metadata can be generated by tracking a desired feature with the scene and can be used to keep the desired feature within an acceptable range of values. This can help avoid noticeable and possibly objectionable visual artifacts when rendering video data.

[006] In one embodiment, a method for using stable scene metadata in a video data stream, the method comprising: dividing the video data stream into a set of scenes; generating first metadata associated with a first scene within the scene set; generate stable scene metadata; and associating the stable scene metadata with said first scene.

[007] In another embodiment, a system for using stable scene metadata for video data, said system comprising: a processor; a memory associated with said processor and said memory additionally comprising processor-readable instructions, such that when said processor reads the processor-readable instructions, the processor performs the following instructions: receive video data streams, wherein said video data stream comprises a set of scenes; for the scene set, generate first metadata associated with the scene set; generate a set of stable scene metadata; and for at least one scene, associate stable scene metadata with said at least one scene.

[008] In yet another embodiment, a video processor comprising: a processor; a memory associated with said processor and said memory additionally comprising computer-readable instructions, such that when said processor reads the computer-readable instructions, the processor performs the following instructions: receive an incoming video data stream, said video data stream comprising a set of scenes; receiving a first set of metadata associated with at least one scene; receiving an indication that a scene cut is substantially the next frame of the received video data stream; receive stable scene metadata; and substantially associating the stable scene metadata with the next frame of the received video data stream.

[009] Other features and advantages of the present system are presented below in the Detailed Description when read in combination with the drawings presented within this application.

BRIEF DESCRIPTION OF THE DRAWINGS

[0010] Exemplary embodiments are illustrated in reference figures in the drawings. It is understood that the embodiments and Figures disclosed in this document should be considered illustrative rather than restrictive.

[0011] Figure 1 depicts an embodiment of an environment and an architecture of a video chaining system made in accordance with the principles of the present application.

[0012] Figures 2A and 2B depict two embodiments of video threading flowcharts that may be suitable for the purposes of the present application.

[0013] Figure 3 depicts one embodiment of a high-level flowchart of video processing that may occur in display management on an exemplary target display.

[0014] Figure 4 is a video processing modality for generating and associating stable scene metadata for a video file.

[0015] Figure 5 is an embodiment of a flowchart that incorporates advance notice of scene changes in a video thread.

[0016] Figure 6 depicts an exemplary video file, partitioned into scenes and a frame within a scene that comprises a scene change indication.

DETAILED DESCRIPTION

[0017] As used herein, the terms "component", "system", "interface", and the like are intended to refer to a computer-related entity, both hardware and software (e.g., running), and/or firmware. For example, a component may be a process running on a processor, a processor, an object, an executable, a program, and/or a computer. By way of illustration, both an application running on a server and the server can be a component. One or more components may reside within a process and a component may be located on a computer and/or distributed between two or more computers. A component may also be intended to refer to a communications-related entity, both hardware and software (e.g., running), and/or firmware, and may additionally comprise wired or wireless hardware sufficient to affect communications.

[0018] Throughout the following description, specific details are presented in order to provide a more complete understanding for those skilled in the art. However, well-known elements may not be shown or described in detail to avoid unnecessarily obscuring the description. Accordingly, the description and drawings should be considered in an illustrative and non-restrictive sense.

INTRODUCTION

[0019] To ensure temporal stability (e.g., no flickering, pulsing, dimming, etc.) and to mitigate a potentially unpleasant visual artifact in video playback on a target display, it may be desired that metadata associated with the video data be generally stable over time. In various embodiments, this can be achieved by applying stability to the metadata for the duration of a scene. Metadata can be allowed to change with each scene cut. In such a case, changes in metadata in adaptation to the content may not be noticeable to a viewer.

[0020] For merely one example, it may be possible to estimate video/image data metadata on a frame-by-frame basis just before display. However, this can result in unpleasant and noticeable changes to the appearance of a scene - possibly in the middle of a scene.

[0021] In various embodiments of the present application, systems and methods for generating or otherwise creating metadata that refer to, and/or are associated with, video data are described herein. In many embodiments of the present application, associated metadata may be generated on a scene-by-scene basis, as will be described in more detail herein. Such metadata can be created at the front end of a video data stream - or in any other suitable part of the video data stream. The video data may be transmitted and/or sent to a user/client/viewer of the video data - whether in a cinema, a home viewing environment, video conferencing, or wherever the video data is viewed and/or consumed.

[0022] Various metadata generation and/or metadata consumption techniques are described in the following patents assigned to the same assignee and/or in patent applications:

[0023] Patent Application No. U.S. 20130076763 by Messmer, published on March 28, 2013 and entitled "TONE AND GAMUT MAPPING METHODS AND APPARATUS";

[0024] Patent Application No. U.S. 20130162666 by Messmer et al., published on June 27, 2013 and entitled "DATA TRANSMISSION USING OUT-OF-GAMUT COLOR COORDINATES";

[0025] Patent Application No. U.S. 20130141647 by Longhurst et al., published on June 6, 2013, entitled "METADATA FOR USE IN COLOR GRADING"; It is

[0026] Patent Application No. U.S. 20120315011 by Messmer et al., published on December 13, 2012 and entitled "VIDEO DELIVERY AND CONTROL BY OVERWRITING VIDEO DATA"

[0027] -- all of which are hereby incorporated by reference in their entireties.

[0028] Figures 1, 2A and 2B depict several general environmental systems (100, 200, 206, respectively) in which the systems and/or methods of the present application may reside. These systems represent possible end-to-end video generation/transmission/view video threads - for example, where video can be captured, scene-to-scene metadata extracted, and placed into a video stream for distribution and sent to a target display for viewing.

[0029] In Figure 1, system 100 (or portions thereof) can be configured to perform one or more of the methods described herein. The components of system 100 may be implemented as software, firmware, hardware, and/or a combination thereof. System 100 includes a video capture subsystem 102, a post-production subsystem 103, and a display subsystem 104. A stream of video data 123 (more specifically, 1231, 123-2, 123-3 to different points in the video stream thread) is generated by the video capture subsystem 102 and is delivered to the post-production subsystem 103 for processing and editing. Video images may be displayed and viewed on a reference display 111 of the post-production subsystem 103 in the course of editing the video data 123. The edited video data 123 is delivered to the display subsystem 104 (via a distribution means 125 via encoder 127A and decoder 127B) for further processing and display. Each of subsystems 102, 103, and 104 (and encoder 127A) may be configured to encode metadata 225 into video data 123. Downstream subsystems may be configured to receive video data 123 from an upstream device and decode the metadata 225 that were incorporated into it. Metadata 225 may be used by downstream subsystems (e.g., subsystems 103 and 104) to guide the processing and/or display of video data 123. Metadata 225 may be used by display subsystem 104 along with characterization parameters of display 120 for controlling and/or guiding video playback on a display 118 of display subsystem 104.

[0030] As seen in Figure 1, subsystems 102, 103 and 104 may comprise a processor 106, 108 and 116 respectively and a program memory 107, 109 and 117 accessible to the respective processors. Each processor (described here and elsewhere) may comprise a central processing unit (CPUs), one or more microprocessors, one or more field-programmable gate arrays (FPGAs), or any combination thereof, or any other unit suitable processing(s) comprising hardware and/or software capable of functioning as described herein. In each subsystem, the processor executes instructions provided by software stored in program memory. The software may include routines for performing the metadata generation, encoding, decoding and processing steps described herein, such as, for example, routines for:

[0031] generate metadata 225 or receive parameters for metadata 225;

[0032] encoding metadata 225 into video data 123 before such video data is communicated to a downstream device;

[0033] decode metadata 225 of video data 123 received from an upstream device;

[0034] processing and applying metadata 225 to guide the processing and/or display of video data;

[0035] selecting a method for encoding metadata 225, based on image and/or video data 123;

[0036] and/or similar.

[0037] System 100 may include a repository 110 accessible to subsystems 102, 103, and 104. Repository 110 may include a library of metadata definitions 112 (e.g., which tells metadata encoders and decoders how to generate and/or read the metadata) and a list of reserved words 114 (e.g., protected pixel values or reserved metadata words). The library of metadata definitions 112 can be accessed by subsystems 102, 103 and 104 in generating, encoding and/or processing metadata. In encoding or decoding metadata 225, reserved words 114 may be compared to encoded/decoded metadata bits to identify sequences of protection bits to insert (or that have been inserted) into the metadata stream to prevent communication of a reserved word. Although a shared repository 110 is shown in the illustrated embodiment of Figure 1, in other embodiments, each of subsystems 102, 103, and 104 may incorporate a local repository 110 stored on a storage medium accessible to that subsystem.

[0038] Figure 2A is a flowchart showing the flow of data through a video delivery thread 200 according to a particular embodiment. Video delivery thread 200 incorporates stages similar to those depicted in video delivery thread 100 of Figure 1. In one or more stages of video delivery thread 200, metadata 225 may be generated and incorporated into a video data stream. video 123 for use in a downstream stage. Metadata 225 is transmitted along with video data 123 through video delivery thread 200 to guide downstream devices in processing the video data and/or to guide video playback in a display subsystem in block 210. The data video content 123, which includes embedded metadata 225, may be delivered to the display subsystem at block 206 using systems, apparatus, and methods suitable for the type of video content delivery (e.g., broadcast satellite television, cable , or high definition networks; streaming multimedia over IP or wireless networks; DVD playback or other storage media, etc.).

[0039] In the embodiment of Figure 2A, camera metadata 225A can be generated and incorporated into video data 123-1 in block 202. Camera metadata 225A can be generated based on camera definitions and the video capture environment. video frame. Camera metadata 225A may comprise, for example, camera parameters that provide a snapshot of camera settings during video frame capture. Such camera parameters may include aperture (f-stops), lens, shutter speed, sensitivity (ISO ratings), and the like. These camera parameters can be used to guide subsequent steps in video delivery thread 200, such as color adjustments (e.g., color timing) during post-production editing in block 204, or display configuration in block 210. .

[0040] In block 204, post-production metadata 225B is generated and incorporated into video data 123-2. Post-production metadata 225B may include: a reference view and environment metadata 225B1 and source video content characterization metadata 225B2. Post-production metadata 225B may be used to guide subsequent steps in video delivery thread 200, such as display configuration in block 210.

[0041] Reference display and environment metadata 225B1 may describe the reference display configuration and studio or preview environment used in post-production editing of block 204. For example, in relation to the reference display used to display video data 123 during post-production editing of block 204, display and reference environment metadata 225B1 may include parameters such as:

[0042] a 3D color gamut mapping that describes the tone and gamma limits of the reference display at a detailed resolution;

[0043] a reduced set of parameters that define the tone and gamma limits of the reference display (which can be used to estimate a 3D color gamut mapping);

[0044] system tonal response parameters that describe the tonal response of the reference display for each chrominance channel;

[0045] screen size

[0046] and/or similar.

[0047] The 225B1 display and reference environment metadata may also include parameters that describe the studio environment in which the video content was color-timed or edited in a reference display during post-production editing of the block 204. Such parameters may include ambient luminance and ambient color temperature.

[0048] Source video content characterization metadata 225B2 may describe post-production edited video content that includes information that may identify or provide:

[0049] tone mapping (e.g., custom tone mapping parameters or curves that can be used to guide tone expansion on the display); and gamma mapping (e.g., custom gamma mapping parameters that can be used to guide gamma expansion in the display);

[0050] the level of the minimum black level considered important in the scene (for example, the shadow under a car);

[0051] the level that corresponds to the most significant portion of a scene (for example, an actor's face);

[0052] the level of the maximum white level considered important in the scene (for example, the center of a lamp);

[0053] the most chromatic color in the scene (e.g. neon light, etc.);

[0054] a map of the location of light sources in an image, or of reflective or emissive objects in an image;

[0055] range of video source content;

[0056] areas of an image that are purposely color timed outside the range of the reference display;

[0057] protected colors that must not be changed during preview processing by the video processor or during display configuration;

[0058] an image histogram that characterizes the image in terms of luminance or gamma (e.g., such information can be used by downstream devices to determine average luminance to refine tone and gamma mapping);

[0059] a change in scene or a reset flag, to alert downstream devices that any statistics or hysteresis from previous video frames are no longer valid;

[0060] a motion map that characterizes video content to identify moving objects, which can be used by downstream devices in combination with the light source location map to guide tone and gamma mapping;

[0061] an indication of the source of content subjected to color timing (e.g., straight from the camera, or post-production editing);

[0062] director's creative intent settings that can be used to control downstream devices such as a set-top box/television or other display. For example, such settings may include: display mode control that provides the ability to control the display to operate in a particular mode (e.g., vivid, cinema, standard, professional, etc.); type of content (e.g. animation, drama, sports, games, etc.) that can be used to determine appropriate gamma or tone mapping or the like;

[0063] 15 and/or similar.

[0064] Video data 123-2 is delivered to a display subsystem in block 206. As seen in Figure 2B, delivery thread 206 may include an encoder platform 127A for triggering the distribution, broadcast or transmission of video data. video 123 by a video distribution medium 125 such as a satellite, cable, or high definition networks; IP or wireless networks; or DVD or other storage media, etc. A decoder stage 127B may be provided at the display end in block 206 to decode video data 123 distributed over medium 125. Decoder stage 127B may be implemented by a signal decoder, for example, or by a decoder within the display subsystem. In blocks 206 and/or 208, viewing environment metadata 225C and/or other metadata 225 may be incorporated into video data 123. Viewing environment metadata 225C may comprise, for example:

[0065] Advanced Video Coding (AVC) VDR encoder data that provides reference monitor tone mapping or bed curves or ambient luminance of the reference environment. At least some of this information can be determined by the video processor with knowledge of the display characteristics (e.g., by reading the display's Extended Display Identification Data (EDID)) and the environment of the display subsystem. In some embodiments, at least some of this information may be determined in the studio during post-production processing of the video data.

[0066] The parameters describe the environment in which the display subsystem display is situated. Such parameters may include, for example, ambient luminance and/or tone or color temperature.

[0067] Viewing environment metadata 225C may be used to guide video data processing in block 208 and/or display configuration in block 210.

[0068] The display subsystem comprises a video processor for processing video data 123-3 received in block 208. The video processor of the display subsystem may perform signal processing on video data 123-3 based on metadata 225 extracted from video data 123 (e.g. metadata 225A) and/or known display characteristics associated with the display subsystem display. Video data 123 may be processed and adjusted for the display according to display characterization parameters 226 and/or metadata 225.

[0069] Other metadata 225 that may be incorporated into video data 123 in blocks 206 and/or 208, or in other stages of video delivery chain 200, includes cleanup metadata 225D (to manage distribution rights and the like). such as, for example:

[0070] watermark data indicating where the video content was generated, distributed, modified, etc.;

[0071] fingerprint data that provides a description of video content for search or indexing purposes and the like;

[0072] protection data indicating who owns the video content and/or who has access to it;

[0073] and/or similar.

[0074] Display environment metadata 225C may be generated based at least in part on display characterization parameters 206 associated with the display subsystem display. In some embodiments, viewing environment metadata 225C, source video content characterization metadata 225B2, and/or cleanup metadata 225D may be created or provided by analyzing video data 103 on the encoder platform 127A. in the decoder platform 127B and/or by the video processor in block 208.

[0075] In block 210, display configuration can be performed on a display of the display subsystem. Appropriate parameters for display configuration may be determined based on display characterization parameters 226 and/or metadata 225, such as camera metadata 225A, post-production metadata 225B (including display and reference environment metadata 225B1, and source video content characterization metadata 225B2) and viewing environment metadata 225C. The display is configured according to these parameters. Video data 123 is output to the display.

[0076] The metadata 225 that is used for processing the video data 123 in block 208 and for display configuration in block 210 is delivered into the video data stream so that the metadata 225 is received in the display subsystem ( including video processor and display) prior to application. In some embodiments, the metadata 225 is delivered such that it is received by the display subsystem at least one video frame ahead of the frame to which the metadata 225 is to be applied. In certain embodiments, metadata 225 is delivered one video frame ahead and application of metadata 225 in blocks 208 and/or 210 may be triggered upon detection of a new video frame in the received video stream. "STABLE" SCENE-BY-SCENE METADATA

[0077] As mentioned previously, it may be desirable to capture metadata in a video file on a scene-by-scene basis. As described herein, various embodiments of the present application may capture metadata (e.g., based on luminance, color gamut, or the like) on a scene-by-scene basis. In particular, an embodiment may provide a set of "stable" metadata that can be applied across the same and/or similar scenes.

[0078] In one embodiment, each scene may be associated with global scene metadata that may be generated in response to frame-dependent characteristics within a scene - for example, such as the minimum, average and maximum luminance values in each frame . Scenes with similar characteristics can also be forced to share the same metadata, so that during viewing they can maintain the same look and feel. In another embodiment, a receiver may also receive "advance notification metadata" - for example, that is, metadata for a future scene, so that it can prepare, in advance, parameters related to DM processing.

[0079] To verify the concept of "stable" scene metadata, the following descriptions are offered for expository purposes only and are not intended to limit the scope of the present application. It may be desirable to have stable color and brightness over some scenes. In one example, suppose there are two actors in a "scene", but the camera cuts to one actor and then to the other actor in a sequence of video frames - for example, in an extended dialogue between the two actors on a set . While this may constitute a "scene" theatrically, the two different camera cuts may cause color and/or luminance shifts that may be both noticeable and objectionable to the viewer. In some embodiments, it may be possible to have different metadata for each cut - for example, in order to produce a stable appearance of the entire scene.

[0080] For another example, consider that there is a single actor in a "scene" - but the actor is moving and the camera follows the actor. Again, although this may be a single scene theatrically, there may be changes in luminance and/or color that are both noticeable and/or objectionable to the viewer. For yet another example, the director might utilize a "dissolving" (or "cross-fading") technique in which one scene reduces its luminance (perhaps to zero), while another scene may start low (e.g., zero). luminance for maximum luminance over the period of a few frames. Such dissolves or cross-fades can be used to illustrate a flashback to an actor in the scene or for other purposes.

[0081] These situations may become relevant in cases where a director may be engaged in post-production processing of captured video. Such a director may be applying color grading, mapping video luminance on a professional-grade monitor - for example, which has luminance on the order of up to 5,000 nits. However, a movie may be viewed on a home video set or some other target display) which may have much lower luminance. Knowing this in advance can allow a director or other content creator the opportunity to enhance the content viewer's experience.

[0082] Based on these few examples (as well as others not mentioned herein), it may be desirable from a viewer's perspective (if not the content creator/video director's perspective) to apply metadata on a scene basis the scene and/or have a process in place that can determine when to apply "stable" metadata to a scene and/or sequence of frames - which may have employed different metadata, perhaps based on frames, to the current frame/scene.

[0083] For a home video situation, it may often be the case that there is a Display Management (DM) processor that may seek to provide a "better" (or "superior") mapping of the video data to the home display . Often, DM’s provide dynamic range mapping to provide a satisfactory luminance that matches the target display from the available video data. Dynamic range mapping can use metadata based on luminance statistics - for example, maximum luminance, average luminance, and/or minimum luminance to provide the mapping.

[0084] Several patent applications assigned to the same assignee disclose Display Management (DM) systems and techniques and may be useful for the systems and methods of the present application:

[0085] Patent Application No. U.S. 20110194618 by Gish et al., published on August 11, 2011 and entitled "COMPATIBLE COMPRESSION OF HIGH DYNAMIC RANGE, VISUAL DYNAMIC RANGE, AND WIDE COLOR GAMUT VIDEO";

[0086] Patent Application No. U.S. 20120229495 by Longhurst, published on September 13, 2012 and entitled "INTERPOLATION OF COLOR GAMUT FOR DISPLAY ON TARGET DISPLAY";

[0087] Patent Application No. U.S. 20120321273 by Messmer, published on December 20, 2012 and entitled "VIDEO DISPLAY CONTROL USING EMBEDDED METADATA"; It is

[0088] Patent Application No. U.S. 20130038790 by Seetzen et al., published on February 14, 2013 and entitled "DISPLAY MANAGEMENT METHODS AND APPARATUS";

[0089] -- all of which are hereby incorporated by reference in their entireties.

[0090] A Metadata Modality With Stable Scene

[0091] Figure 3 depicts an embodiment of a high-level block flowchart of the present application. Video thread 300 may receive an encoded bitstream 301 that may further comprise video/image data, along with metadata that may be in some usable format - e.g., frame-by-frame, scene-by-scene, and which comprises metadata with based on luminance statistics, color mapping metadata or similar.

[0092] This encoded bit stream 301 may be received by a decoder 302, which may further comprise an analyzer 304. The decoder 302 may decode the received bit stream, which may be encrypted, compressed or otherwise encoded in any known manner. in technique. Once decoded, the received bit stream can be analyzed by an analyzer 304. The analyzer 304 can split the metadata of the video/image data.

[0093] The extracted video/image data may be sent forward, together with its associated metadata as an intermediate bitstream 303. The bitstream 303 may also comprise one or more flags (or some other indication, signal, or the like). 305 that can inform the downstream processor(s) which metadata to apply or similar, as will be further described in this document.

[0094] The intermediate bit stream 303 and/or which flags 305 may be received by a Display Management (DM) module 306. The DM module 306 may apply any desired image/video mapping before the image data - final gem/video is sent to the target display 308. The target display 308 can be any suitable device that can display image and/or video data to a viewer. Merely for a few examples, such target displays 308 may be HD televisions, movie projectors, desktop monitors, laptop-type computers, tablet-type computers, smart devices, or the like.

[0095] As mentioned, various embodiments of the present application may involve the calculation and/or derivation of scene-by-scene metadata - for example, possibly a set of "stable" scene metadata. Such scene-stable metadata may be employed judiciously by the thread during a time when scene-stable metadata may be employed (possibly overriding other available metadata both scene-based and frame-based) to mitigate possible noticeable and/or objectionable artifacts. to the viewer.

[0096] For merely one example, consider a scene in a dark cave. The image can show all the dark detail of the cave. However, if the camera pans to the cave opening (which is bright), adaptive mapping can adjust the image accordingly - for example, it can decrease the dark details of the cave walls to accommodate the new brighter pixels. By generating and using stable scene metadata, mapping can be optimized for the entire scene - for example, so that there is no noticeable change in the middle of the scene.

[0097] Figure 4 is an embodiment of a high-level flowchart 400 of processing stable scene metadata. At 402, video data can be divided into a set of scenes. This division and/or partitioning of a video into a set of scenes can be performed in several ways. First, the partitioning can be done by a human user - for example, a director, a film editor, someone in post-production, or the like. For example, in one embodiment, the scene cuts may already be known from the Edit Decision List (EDL) -- which may be used to create the film from several different takes. It is possible in one embodiment to extract this EDL and use it to delimit the scene boundaries. This way, there is little to no effort required. Furthermore, the user has the option to overwrite the determined (or extracted) scene cuts automatically.

[0098] Alternatively, scene partitioning identification can be made automatically by a video processor that can make such determinations by analyzing video data on a frame-by-frame basis. For example, if there is a measurably large variation in luminance data, color data, or other image data metrics between frames, then a video processor may decide that this difference can mark the boundary of two scenes. Such automatic determination may be enhanced in a look-ahead, or multi-pass, process - where several frames may be analyzed and if an initial difference in an image data metric is not observed and if that metric in many frames thereafter conform substantially to such an initial difference, then a high degree of probability can be assessed that a scene change has occurred.

[0099] For purposes of the present application, scenes may be identified in the video data in any known manner. At 404, metadata may be computed, measured, or otherwise extracted on a scene-by-scene basis. Just one example, if there are 50 frames that comprise a given scene, then luminance data can be measured and extracted for the entire scene. Scene metadata such as minimum luminance, mean and/or median luminance and maximum luminance can be calculated. Other image/video metrics can also be measured and/or extracted to form other scene-based metadata.

[00100] The following is an embodiment of generating stable scene metadata within a video stream:

[00101] MIN, MID and MAX luminance is computed for each frame within a scene. The results are then combined for the entire scene.

[00102] For MIN, take the minimum of all minimums of all frames in a scene;

[00103] For MID, take the average (Median) of all averages of all frames in a scene;

[00104] For MAX, take the maximum of all maximums of all frames in a scene.

[00105] It will be found that similar statistics can be derived for other video/image metrics - for example, color gamut data or the like. In another embodiment, it may be possible to generate other scene-dependent metadata - for example, how much sharpening or smoothing to apply to image data within a scene.

[00106] At 406, a set of "stable" metadata can be computed for a scene. Stable metadata may differ from older computed scene-based (or frame-based) metadata - depending on the use of such metadata. Stable scene metadata may be computed and/or generated for a scene, possibly according to some monitored features, aspects, and/or metrics that may potentially produce a noticeable and/or objectionable change in the video data - e.g., even if The earliest computed scene-based metadata was used in rendering the scene for preview. For example, an actor moving in space and time across a different background (e.g., going from a dark indoor setting to a bright, sunlit outdoor setting in one cut) may produce noticeable and/or objectionable in color or hue to the actor's face or otherwise in skin tone. In some embodiments, metadata for a second scene may also be replaced with metadata computed for a first scene (e.g., different from the second scene) if according to a feature, aspect, and/or monitored metric the two scenes can be considered perceptually. similar. The second scene can be subsequent or prior to the first scene.

[00107] Other features, aspects, and/or metrics are possible - for example, skin tone, luminous objects/features, dark features/objects, colored features/objects, and the like. Such changes can be mitigated with stable scene metadata. Stable scene metadata can be computed and/or generated such that the feature, aspect, and/or metric that is monitored throughout a scene is brought back to and/or maintained within a range of acceptable values. At 408, the process may associate and/or replace this stable scene metadata with any other metadata that may or may not previously be associated with the scene. Such association and/or replacement of stable scene metadata may be provided to bring such features, aspects and/or metrics back within an acceptable range - for example, possibly if other metadata allows such features, aspects and/or metrics outside the acceptable range. The range of acceptable values for a feature, aspect, and/or metric may be determined manually (e.g., by a director and/or film editor), or according to certain rules and/or heuristics involving image processing/rendering and/or film editing.

[00108] It should be noted that the processing set out in Figure 4 can occur at many different points in the video/image chain. For example, partitioning video into scenes can be done in post-production by a human - or otherwise by a processor in the thread. Additionally, then, computation and/or scene-based metadata extraction can be done in post-production or elsewhere in the pipeline. Likewise, the association of "stable" scene metadata may occur in post-production - or may be performed additionally downstream, for example, by the DM or other video processor prior to sending the final video/image data to the target viewport for rendering.

ALTERNATIVE MODALITIES

[00109] In some embodiments, mapping operations may be dependent on image content in order to achieve maximum performance. Such image-dependent mapping can be controlled by metadata that is generated from the source content. To ensure temporal stability (e.g., no flickering, pulsing, fading, etc.) it may be desirable for metadata to be generally stable over time. In one embodiment, this may be affected by applying stability to the meta-data for the duration of a scene. Metadata can be allowed to change with each scene cut. A sudden change in metadata in adaptation to the content may not be noticeable to a viewer in such a case.

[00110] In one embodiment, the steps for generating stable scene metadata may comprise the following:

[00111] Acquire location of scene cuts in video data. In one embodiment, this may be derived from the Edit Decision List (EDL). Alternatively, this can be entered manually by a human -- or detected automatically by a processor.

[00112] Compute and/or generate metadata for each frame in the scene:

[00113] Optionally reduce image resolution. (This may tend to speed up processing and minimize the effect of some peripheral pixel values).

[00114] Convert an image to desired color space (e.g. IPT-PQ)

[00115] Compute minimum image value (e.g. channel I)

[00116] Compute maximum image value (e.g. channel I)

[00117] Compute the average value of the image (e.g. channel I)

[00118] Combine the per-frame results into one per-scene result:

[00119] Compute the minimum of each of the frame minimums.

[00120] Compute the maximum of each of the frame maximums.

[00121] Compute the average of each of the frame averages.

[00122] Associate metadata with the scene -- or alternatively each frame within the scene

[00123] It will be seen that variations of the above embodiments are possible and are contemplated within the scope of this present application. For example, instead of analyzing each frame of the scene in step (2), a single representative frame can be chosen and used to compute and/or generate metadata, which is then associated with the entire scene.

[00124] Additionally, cross-fades can be supported by indicating metadata for the scenes on either side of the cross-fade and then interpolating to the intermediate frames. Such interpolation may be linear or asymptote at both ends via a cosine or similar function.

[00125] After step (4), the metadata can be inserted into an encoded bitstream with appropriate synchronization with the correct video frame. Metadata can be repeated regularly to allow random entry into the stream.

[00126] In yet another embodiment, it is possible to include in the metadata some pre-computed values to help convert the decoded video to a desired color space (e.g., IPT-PQ). This may be desirable since conversion is often done on devices with fixed-point processors, which may not do well with certain mathematical operations such as divisions and exponents. Using pre-computed values and incorporating them into the metadata stream can be beneficial.

[00127] Decoding of Stabilized Scene Metadata / "Advance Notice" Metadata

[00128] In a video decoder, new scene metadata may arrive in the same frame as the first frame of a new scene. Alternatively, the metadata may arrive before the first frame of the scene to provide time to decode and interpret the metadata in time for it to be applied to process the video. Such "early warning metadata" and other techniques may be desirable to improve the robustness of stabilized scene metadata through bitstream transmission. Multiple enhancements may comprise the following, either as individual enhancements or as multiple enhancements in combination:

[00129] Repeat metadata every frame within the same scene;

[00130] Add an indicator/flag of a scene cut that occurs substantially in the following frame in the metadata body;

[00131] Add an indicator/flag of a scene cut that occurs in the current frame in the metadata body;

[00132] Add an indicator/flag that the metadata of the next frame is substantially identical to (or substantially different from) the metadata of the current frame; and/or

[00133] Add a data integrity field in the metadata body for error checking (e.g., CRC32).

[00134] Figure 5 depicts an embodiment of such an early warning metadata flowchart 500. At 502, the system/thread may compute and/or generate the metadata on a per-scene basis. This metadata - stable scene metadata or otherwise - can be associated with the scene in the video data bitstream at 504. The system/thread can then add an indication of an impending scene change in advance by a desired number ( e.g., one or more) of frames preceding the first actual frame of the new scene at 506. This indication and/or signaling may comprise a portion of the bit stream and be perceived by the DM (or other suitable processor in the thread). At 508, the system may allow the DM (or other suitable processor) time to install the parameters and/or mappings before the scene change. This additional time may allow the system an opportunity to avoid any noticeable artifacts that might be objectionable to a viewer of the video content.

[00135] This may tend to be an improvement over conventional methods of stabilizing metadata that may not have access to scene cut locations in advance. For example, if scene cuts are not known in advance, metadata can be estimated on the fly by analyzing video data, or by allowing metadata to change smoothly over time. This may cause image artifacts such as flickering, pulsation, fading, etc. In another embodiment, by computing the metadata at the source (e.g., before video compression), it may be possible to reduce the computations and, consequently, the cost required of less capable consumer devices.

[00136] Figure 6 is an example of video data 600 that is partitioned into a number of scenes (Scene 1 to Scene N) - which in turn comprises a number of frames (e.g., frame 602a). Frame 602m of Scene 1 can have an early warning flag associated with the frame - such that the DM can have time to set parameters and/or mapping to better render Scene 2, which follows in turn.

[00137] A detailed description of one or more embodiments of the invention, read together with the accompanying figures, which illustrate the principles of the invention has now been given. It should be noted that the invention is described in combination with such embodiments, but the invention is not limited to any embodiment. The scope of the invention is limited to embodiments only and the invention encompasses several alternatives, modifications and equivalents. Several specific details have been defined in this description in order to provide a thorough understanding of the invention. These details are provided for exemplary purposes and the invention may be practiced in accordance with the embodiments without some or all of these specific details. For the purpose of clarity, the technical material that is known in the art in relation to the invention has not been described in detail so that the invention is not unnecessarily obscured.

Claims (4)

1. System for using stable scene metadata for video data comprising: a processor; a memory associated with the processor and wherein the memory further comprises a processor-readable method, such that when the processor reads the processor-readable method, the processor performs the following steps: receiving a stream of video data, wherein the stream video data comprises a set of scenes; generating first metadata associated with a first scene within the set of scenes, wherein generating first metadata associated with the first scene within the set of scenes comprises: computing a minimum, average and maximum luminance for each frame in a scene; and compute a minimum, average and maximum luminance for the scene; generate stable scene metadata; and associating the stable scene metadata with said first scene, wherein associating the stable scene metadata with said first scene further comprises: repeating the stable scene metadata for each frame within said first scene. 2. System, according to claim 1, characterized by the fact that the step of generating stable scene metadata further comprises: monitoring a resource within a scene to determine the stable scene metadata for the scene. 3. System according to claim 2, characterized by the fact that the step of generating stable scene metadata further comprises: computing stable scene metadata so that the monitored resource within a scene is maintained within a range of values acceptable. 4. System, according to claim 3, characterized by the fact that the monitored resource is one of a group, the group comprising: skin tone, luminance resources, dark resources and colored resources.