WO2024173258A1

WO2024173258A1 - Object-based display mapping for dynamic content

Info

Publication number: WO2024173258A1
Application number: PCT/US2024/015410
Authority: WO
Inventors: William Anthony Rozzi
Original assignee: Dolby Laboratories Licensing Corp
Current assignee: Dolby Laboratories Licensing Corp
Priority date: 2023-02-15
Filing date: 2024-02-12
Publication date: 2024-08-22
Anticipated expiration: 2025-08-15

Abstract

Techniques for object-based display mapping for dynamic content are disclosed. The techniques can include determining a presence of one or more objects in a content frame, determining frame-level display-mapping parameters for the content frame based on object level display-mapping parameters for at least one of the one or more objects, applying the frame-level display-mapping parameters to the content frame to obtain a display-mapped content frame, and rendering the display-mapped content frame on a display device.

Description

OBJECT-BASED DISPLAY MAPPING FOR DYNAMIC CONTENT

CROSS-REFERENCE TO RELATED APPLICATIONS

[0001] This application claims priority from U.S. Provisional Application No. 63/485,1 18, filed on 15 February 2023, and European Patent Application No. 23168992.8 filed on 20 April 2023, each of which is incorporated by reference herein in its entirety.

BACKGROUND

[0002] This application relates generally to systems and methods of content generation and rendering.

SUMMARY

[0003] At least some aspects of the present disclosure may be implemented via methods. Some methods may involve determining a presence of one or more objects in a content frame, determining frame-level display-mapping parameters for the content frame based on object level display-mapping parameters for at least one of the one or more objects, applying the frame-level display -mapping parameters to the content frame to obtain a display-mapped content frame, and rendering the display-mapped content frame on a display device.

[0004] In some examples, some methods may involve composing present object information based on one or more object event notifications received from the content engine, and determining the presence of the one or more objects based on the present object information.

[0005] In some examples, each of the object event notifications may include an object identifier associated with a respective object.

[0006] In some examples, some methods may involve receiving an object exit notification associated with an object, modifying the present object information based on the object exit notification, and determining an absence of the object from a second content frame based on the modified present object notification.

[0007] In some examples, at least one of the frame-level display-mapping parameters for the content frame may be determined based on respective object level display mapping parameters for multiple objects in the content frame. [0008] In some examples, the at least one of the frame-level display-mapping parameters for the content frame may be determined as an average of the respective object level display mapping parameters for the multiple objects in the content frame.

[0009] In some examples, the at least one of the frame-level display-mapping parameters for the content frame may be determined as a median or mode of the respective object level display mapping parameters for the multiple objects in the content frame.

[0010] In some examples, the at least one of the frame-level display-mapping parameters for the content frame may be determined as a maximum or minimum of the respective object-level display-mapping parameters for the multiple objects in the content frame.

[0011] In some examples, the at least one of the frame-level display-mapping parameters for the content frame may be determined as a weighted summation of the respective object level display mapping parameters for the multiple objects in the content frame.

[0012] In some examples, the frame-level display-mapping parameters may include an ambient color temperature parameter.

[0013] In some examples, the frame-level display-mapping parameters may include a contrast parameter.

[0014] In some examples, the frame-level display-mapping parameters may include a luminance parameter.

[0015] In some examples, the frame-level display-mapping parameters may include a hue parameter.

[0016] In some examples, the frame-level display-mapping parameters may include a color saturation parameter.

[0017] In some examples, the dynamic content can include content of a game engine.

[0018] In some examples, the dynamic content can include content of a live broadcast.

[0019] In some examples, the dynamic content can include augmented reality (AR) content, virtual reality (VR) content, or a combination of both.

[0020] Some or all of the operations, functions and/or methods described herein may be performed by one or more devices according to instructions (e.g., software) stored on one or more non-transitory media. Such non-transitory media may include memory devices such as those described herein, including but not limited to random access memory (RAM) devices, read-only memory (ROM) devices, etc. Accordingly, some innovative aspects of the subject matter described in this disclosure can be implemented via one or more non-transitory media having software stored thereon.

[0021] At least some aspects of the present disclosure may be implemented via an apparatus. For example, one or more devices may be capable of performing, at least in part, the methods disclosed herein. In some implementations, an apparatus is, or includes, an image processing system having an interface system and a control system. The control system may include one or more general purpose single- or multi-chip processors, digital signal processors (DSPs), application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs) or other programmable logic devices, discrete gates or transistor logic, discrete hardware components, or combinations thereof.

[0022] Details of one or more implementations of the subject matter described in this specification are set forth in the accompanying drawings and the description below. Other features, aspects, and advantages will become apparent from the description, the drawings, and the claims. Note that the relative dimensions of the following figures may not be drawn to scale.

DESCRIPTION OF THE DRAWINGS

[0023] These and other more detailed and specific features of various embodiments are more fully disclosed in the following description, reference being had to the accompanying drawings, in which:

[0024] FIG. 1 depicts an example operating environment in which object-based display mapping for dynamic content may be performed according to aspects of the disclosure.

[0025] FIG. 2 depicts an example of a method for object-based display mapping according to aspects of the disclosure.

[0026] FIG. 3 is a block diagram that shows examples of components of an apparatus capable of implementing various aspects of this disclosure.

DETAILED DESCRIPTION [0027] This disclosure and aspects thereof can be embodied in various forms, including hardware, devices or circuits controlled by computer-implemented methods, computer program products, computer systems and networks, user interfaces, and application programming interfaces; as well as hardware-implemented methods, signal processing circuits, memory arrays, application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), and the like. The foregoing is intended solely to give a general idea of various aspects of the present disclosure, and does not limit the scope of the disclosure in any way.

[0028] In the following description, numerous details are set forth, such as optical device configurations, timings, operations, and the like, in order to provide an understanding of one or more aspects of the present disclosure. It will be readily apparent to one skilled in the art that these specific details are merely exemplary and not intended to limit the scope of this application.

[0029] Display mapping is a technique used to convert high dynamic range (HDR) images into lower dynamic range images suitable for display on a monitor or screen. Display mapping can be implemented in conjunction with the generation and presentation of video content to improve the appearance of the presented content to the viewer. According to some approaches, display mapping can be applied to video content using a frame-based approach, according to which display mapping parameters are determined and applied on a per- frame basis. According to some approaches, display mapping can be applied to video content on a per-scene basis, such that the same display-mapping parameters are applied to each of the frames that make up a given scene within the video content.

[0030] In the case of static video content (such as linear cinematic content), with respect to which the composition and contents of each frame are predefined and not subject to change, suitable display mapping parameters can be determined during development. For instance, a colorist can manually tune the display mapping parameters for frames (or scenes) of static video content during color grading. Metadata that indicates the display mapping parameters determined during development can be stored in the content data stream, and during subsequent playback, the content can be display mapped according to the display mapping parameters indicated by the metadata.

[0031] In the case of dynamic video content, with respect to which the composition and contents of each frame are not predefined, it may not be possible to determine suitable display mapping parameters ahead of time. For example, the composition and contents of frames of video content associated with a gaming application may depend on user inputs received from the user as the user plays the game, and thus may be unknown during the content development process. In the example of video content associated with a live broadcast, there may not even be a content development process. As such, display mapping parameters for dynamic video content may need to be determined and applied on-the-fly during content rendering and presentation.

[0032] According to one approach to on-the-fly determination and application of display mapping parameters for dynamic video content, display mapping parameters for frames of dynamic video content can be determined based on similarities between those frames and reference frames (“exemplars”). During content development, respective display-mapping parameters can be adopted for each of a collection of reference frames. In conjunction with display mapping a content frame during subsequent content rendering and playback, the content frame can be compared to each of the collection of reference frames to determine how similar it is to each of those frames. The degrees of similarity between the content frame and the various reference frames can be quantified, such as based on a characteristic vector of the content frame and respective characteristic vectors of the reference frames. Display-mapping parameters for the content frame can then be determined by using the quantified similarities between the content frame and the various reference frames as weights to compute a weighted sum of the respective display-mapping parameters for the various reference frames.

[0033] Such an approach can be inconvenient in a dynamic content development setting, in that it may be difficult or tedious for the developer to identify or create suitable exemplars, due to the dynamic nature of the content. Furthermore, as the display mapping parameters for content frames are determined according to only indirect consideration of the contents and compositions of those frames, the extent of visual quality improvement yielded by such an approach may vary widely across the frames of the dynamic content, and a lesser degree of overall improvement in the viewing experience may be achieved than may be desirable.

[0034] Described herein are object-based display mapping techniques for dynamic content. According to techniques described herein, display-mapping parameters for frames of dynamic content can be determined on-the-fly during content rendering and playback, based on the presence of objects within those frames. Respective sets of object-level display mapping parameters can be defined for various types of objects, and the presence of such objects can be tracked over the course of content rendering and playback. In conjunction with display mapping a given content frame, respective object-level display mapping parameters can be identified for each of one or more objects present in the content frame. Frame- level display-mapping parameters can then be determined for the content frame based on the identified object-level display-mapping parameters, and can be applied to the content frame.

[0035] FIG. 1 depicts an example operating environment 100 in which object-based display mapping for dynamic content may be performed according to aspects of the disclosure. In operating environment 100, a content engine 102 generates content frames 104. Content frames 104 can be frames of dynamic content. In some examples, content engine 102 can correspond to a gaming engine, and content frames 104 can be frames of gaming content. In some examples, content engine 102 can generate content of a live broadcast, and content frames 104 can be frames of live broadcast content. In some examples, content engine 102 can correspond to a virtual reality (VR) engine, an augmented reality (AR) engine, or a VR/AR engine, and content frames 104 can be frames of VR content, frames of AR content, or a combination of both.

[0036] A display-mapping (DM) module 125 receives content frames 104 from content engine 102 and conducts display-mapping of content frames 104 to generate display-mapped content frames 126, which are rendered on a display device 140. Display device 140 can include any device configured to display or project light; for example, computer displays, televisions, OLED displays, LCD displays, quantum dot displays, cinema, consumer, and other commercial projection systems, heads-up displays, virtual reality displays, and the like.

[0037] Content engine 102 can be configured to recognize the presence of objects within content frames 104. The nature of objects that content engine 102 may recognize can vary widely, depending on the implementation. In an example implementation, content engine 102 can correspond to a gaming engine, and may recognize the presence of objects such as buildings, vehicles, characters, explosions, and lights in content frames 104. In a second example implementation, content engine 102 can generate content of a live concert broadcast, and may recognize the presence of objects such as a crowd, performers, lights, instruments, amplifiers, fog, and flames in content frames 104.

[0038] In some implementations, content engine 102 can be configured during development to recognize various types of graphics objects. In some implementations, content engine 102 can keep track of the graphics objects that it draws as it renders content frames 104 in order to maintain awareness of the presence of such graphics objects in content frames 104. For example, if content engine 102 is a gaming engine, it may keep track of which space ship objects it draws in content frames 104 depicting a space battle. In some implementations, as content engine 102 generates content frames 104, it can analyze content frames 104 to detect the presence of various types of objects dynamically. For example, if content engine 102 generates content frames 104 of a live concert broadcast, then as it generates content frames 104, content engine 102 can analyze content frames 104 to determine whether various types of objects are present within them. In some implementations, content engine 102 can analyze content frames 104 to dynamically detect the presence of predefined objects. In some implementations, content engine 102 can dynamically detect new objects by analyzing content frames 104 as well, and can analyze subsequent content frames 104 to detect the new objects in addition to - or in lieu of - checking content frames 104 for the presence of predefined objects.

[0039] Content engine 102 can maintain object information 106 associated with objects that it recognizes (or looks for) within content frames 104. Object information 106 can include object identifiers (IDs) 107 and object-level display-mapping parameter information 108. Each object identifier 107 can be a logical value, such as a number, name, text string, or code, that uniquely identifies an object that is recognizable to content engine 102 and that may appear within content frames 104 generated by content engine 102. In some implementations a given object identifier 107 can be defined to correspond to a collective set of multiple instances of a same object (for example, to a school of fish) rather than to just a single instance of that object (for example, to a single fish among those in the school).

[0040] Object-level display-mapping parameter information 108 can include information indicating object-level display mapping parameters associated with objects identified by object identifiers 107. Such object- level display mapping parameters can include display-mapping parameters (or sets of display-mapping parameters) that are specific to particular objects, display-mapping parameters (or sets of display-mapping parameters) that are shared among multiple objects, or both. In some implementations, object-level display-mapping parameter information 108 may not specify any display mapping parameters for some objects identified by object identifiers 107. In some implementations, with respect to some objects identified by object identifiers 107, object-level display-mapping parameter information 108 may specify subsets of a collective set of possible display-mapping parameters, such that object- specific or default values are used on a parameter-by-parameter basis. In some implementations, sets of object-level display mapping parameters can be associated with various objects during the development process. In some implementations, default object-level display mapping parameters can be defined during development that are to be applied during content rendering to objects for which object-level display mapping parameters were not previously defined. [0041] In some implementations, the object- level display-mapping parameters for some objects may be made programmable, enabling automatic updates to changes in global game state, graphics context, or object attributes like size, color, or reflectivity. Such programmability may allow effects like an explosion object changing its color parameters as it evolves over time, or a space vehicle changing its brightness parameters based on its relative orientation to light sources and the virtual camera when it exposes its engines. In such, content engine 102 can modify object-level display-mapping parameter information 108 to reflect changes in such programmable object-level display-mapping parameters as they occur.

[0042] The object-level display-mapping parameters indicated by object-level displaymapping parameter information 108 can define requirements or constraints relating to various visual characteristics of frames of content to be displayed on display 140. Examples of such visual characteristics can include ambient color temperature, contrast, luminance, hue, colorsaturation, highlights, shadows, white clipping, and black clipping.

[0043] Content engine 102 can inform display mapping engine 125 of objects that are present within content frames 104 and of object-level display-mapping parameters for those objects. In some implementations, content engine 102 can so inform display mapping engine 125 by sending object event notifications 110. Each object event notification 1 10 can generally indicate a change, with respect to a given content frame 104 relative to a preceding content frame 104, in the composition of the present objects or in the object- level display-mapping parameters applicable to the present objects.

[0044] Examples of object event notifications 110 according to aspects of the disclosure can include object entry notifications 112, object exit notifications 116, and object update notifications 118. Content engine 102 can send an object entry notification 112 to display mapping engine 125 to notify display mapping engine 125 that an object that was not present in previous content frames 104 is now present. The object entry notification 112 can specify an object identifier 113 associated with the newly -present object, and can indicate object- level display -mapping parameters 114 applicable to the newly-present object. Content engine 102 can send an object exit notification 116 to display mapping engine 125 to notify display mapping engine 125 that an object that was present in previous content frames 104 is no longer present. The object exit notification 116 can specify an object identifier 117 associated with the newly- absent object. Content engine 102 can send an object update notification 118 to display mapping engine 125 to provide display mapping engine 125 with updated display-mapping parameters for a currently-present object, such as may reflect changes made to programmable display-mapping parameters . The object update notification 118 can specify an object identifier 119 associated with the object in question, and can indicate updated object-level display-mapping parameters 120 applicable to that object.

[0045] Display mapping engine 125 can maintain present object information 130 based on object event notifications 110 that it receives from content engine 102. Present object information 130 can generally comprise information based on which display mapping engine 125 can identify which objects are present within content frames 104 received from content engine 102, and the object-level display-mapping parameters associated with those present objects. Present object information 130 can include present object identifiers 132, present object object-level display-mapping parameters 134, and present object counts 136. Present object identifiers 132 can include the respective object identifiers for all of the objects understood (based on received object event notifications 110) to be present in the current frame. Present object object- level display-mapping parameters 134 can include respective sets of object- level display-mapping parameters for the various present objects.

[0046] In response to receipt of an object entry notification 112 from content engine 102, display mapping engine 125 can add an object identifier 113 specified by the object entry notification 1 12 to present object identifiers 132, and can add object- level display-mapping parameters 114 indicated by the object entry notification 112 to present object object-level display -mapping parameters 134.

[0047] In response to receipt of an object exit notification 116 from content engine 102, display mapping engine 125 can remove an object identifier 117 specified by the object exit notification 116 from present object identifiers 132. In response to receipt of an object update notification 118 from content engine 102, display mapping engine 125 can identify, among present object object- level display-mapping parameters 134, object-level display-mapping parameters associated with an object identifier 119 specified by the object update notification 118, and can replace those object-level display-mapping parameters with object-level displaymapping parameters 120 indicated by the object update notification 118.

[0048] In some implementations, to avoid redundancy in present object information 130, multiple instances of an object (for example, the various fish in a school of fish) can use a same object identifier among present object identifiers 132 and share a same set of object-level display-mapping parameters among present object object-level display-mapping parameters 134. In some implementations, display mapping engine 125 can maintain a present object count for such an object. The present object count can be incremented when an additional instance of the object becomes present, and can be decremented when an instance of the object becomes no longer present. If the present object count reaches zero, indicating that no instances of the object are present, display mapping engine 125 can remove the corresponding object identifier from present object identifiers 132, and can remove the associated object-level display-mapping parameters from present object object-level display-mapping parameters 134.

[0049] Display mapping engine 125 can use an object-based display mapping procedure to generate display-mapped content frames 126 based on content frames 104 that it receives from content engine 102. According to the object-based display mapping procedure, display mapping engine 125 can generally conduct display mapping of content frames 104 based on object-level display mapping parameters for objects that appear within content frames 104. This can generally involve adjustments to various visual characteristics of the content frames. Examples of such adjustments can include adjustments to ambient color temperature, contrast, luminance, hue, color-saturation, highlights, shadows, white clipping, black clipping, slope (multiplying brightness and/or one or more chromatic channels by a factor), and offset (adding an offset to brightness and/or one or more chromatic channels).

[0050] In order to perform display mapping for a given content frame 104, display mapping engine 125 can consult present object information 130 to identify objects that are present within the content frame 104 and object-level display-mapping parameters applicable to those objects. Based on the objects that are present within the content frame 104 and their applicable objectlevel display-mapping parameters, display mapping engine 125 can derive frame-level displaymapping parameters 128 for the content frame 104. Display mapping engine 125 can apply the derived frame-level display-mapping parameters 128 to the content frame 104 to obtain a display -mapped content frame 126 for presentation on display device 140. In some implementations, display mapping engine 125 can use frame-level display-mapping parameters 128 to construct a 3D lookup table for mapping high dynamic range RGB values to lower dynamic range RGB values, and the table can be applied to a content frame 104 to generate a display -mapped content frame 126. In some implementations, a shader can apply a display mapping computation directly to each pixel of a content frame 104 in conjunction with generation of a display-mapped content frame 126.

[0051] Display mapping engine 125 may use various possible approaches to determine frame-level display-mapping parameters 128 based on object-level display-mapping parameters for objects present within content frames 104, and may use different approaches for different types of display-mapping parameters. According to some implementations, display mapping engine 125 may determine some frame-level display-mapping parameters 128 based on statistical characteristics of the corresponding object-level display-mapping parameters for the present objects. For example, display mapping engine 125 may determine a given frame-level display -mapping parameter 128 as an average, median, mode, minimum, or maximum of the respective corresponding object-level display mapping parameters for the objects present in the frame. Which such statistic is most suitable may depend on the nature of the particular displaymapping parameter in question. For instance, display mapping engine 125 may determine a frame-level maximum luminance as a maximum of the object-level maximum luminances for the objects present in the frame, but may determine a frame-level minimum luminance as a minimum of the object-level minimum luminances for the present objects.

[0052] According to some implementations, display mapping engine 125 may determine some frame-level display-mapping parameters 128 based on weighted summations of the corresponding object-level display-mapping parameters for the present objects. The weights applied to the individual object-level display-mapping parameters for the present objects may be computed from, for example, a measure of the object's area coverage within the frame, such measures including but not limited to pixel counts and percentage of total screen area.

[0053] In some implementations, display mapping can be applied separately to various objects within content frames 104, by segmenting content frames 104 according to the locations of the present objects and applying each object’s object-level display-mapping parameters only within the region occupied by that object. In some implementations, content engine 102 can provide display mapping engine 125 with respective object- level display-mapping parameters for various objects present within a content frame 104 and a segmentation map that indicates the respective regions occupied by those objects, and display mapping engine 125 can perform a display-mapping operation according to which it applies the object-level display-mapping parameters for the various objects to the respective regions that they occupy. In some other implementations, display mapping engine 125 may conduct display mapping of such a content frame 104 via a series of display-mapping operations, during each one of which it may apply the object-level display-mapping parameters for a different respective object within a region occupied by that object.

[0054] FIG. 2 depicts an example of a method 200 for object-based display mapping according to aspects of the disclosure. Method 200 may be representative of operations that may be performed in operating environment 100 of FIG. 1 according to some implementations. According to method 200, a presence of one or more objects in a content frame generated by a content engine may be determined at 202. For example, in operating environment 100 of FIG. 1 , display mapping engine 125 may determine a presence of one or more objects in a content frame 104. In some examples, present object information may be composed based on one or more object event notifications received from the content engine, and the presence of the one or more objects may be determined based on the present object information. In some examples, each of the object event notifications may include an object identifier associated with a respective object. In some examples, an object exit notification associated with an object may be received, the present object information may be modified based on the object exit notification, and an absence of the object from a second content frame may be determined based on the modified present object notification.

[0055] In some examples, the dynamic content may include content of a game engine. In some examples, the dynamic content may include content of a live broadcast. In some examples, the dynamic content may include augmented reality (AR) content, virtual reality (VR) content, or a combination of both.

[0056] At 204, frame-level display-mapping parameters for the content frame may be determined based on object-level display-mapping parameters for at least one of the one or more objects. For example, in operating environment 100 of FIG. 1, display mapping engine 125 may determine frame-level display-mapping parameters 128 for a content frame 104 based on objectlevel display-mapping parameters (such as may be included among present object object-level display-mapping parameters 134) for at least one object within the content frame 104.

[0057] In some examples, at least one of the frame-level mapping parameters may be determined based on respective object-level display-mapping parameters for multiple objects in the content frame. In some examples, the at least one of the frame-level display-mapping parameters for the content frame may be determined as an average of the respective object-level display-mapping parameters for the multiple objects in the content frame. In some examples, the at least one of the frame-level display-mapping parameters for the content frame may be determined as a median or mode of the respective object-level display-mapping parameters for the multiple objects in the content frame. In some examples, the at least one of the frame-level display-mapping parameters for the content frame may be determined as a maximum or minimum of the respective object-level display-mapping parameters for the multiple objects in the content frame. In some examples, the at least one of the frame-level mapping parameters for the content frame may be determined based on weighted summations of the respective object- level display-mapping parameters for the multiple objects in the content frame. In some such examples, the weights applied to the individual object-level display-mapping parameters for the present objects may be computed from, for example, measures of the objects’ respective area coverages within the frame, such measures including but not limited to pixel counts and percentage of total screen area.

[0058] In some examples, the frame-level display-mapping parameters may include an ambient color temperature parameter. In some examples, the frame-level display-mapping parameters may include a contrast parameter. In some examples, the frame-level displaymapping parameters may include a luminance parameter. In some examples, the frame-level display-mapping parameters may include a hue parameter. In some examples, the frame-level display-mapping parameters may include a color saturation parameter.

[0059] At 206, the frame-level display-mapping parameters may be applied to the content frame to obtain a display-mapped content frame. For example, in operating environment 100 of FIG. 1, display mapping engine 125 may apply frame-level display-mapping parameters 128 to a content frame 104 to obtain a display-mapped content frame 126. At 208, the display-mapped content frame may be rendered on a display device. For example, in operating environment 100 of FIG. 1 , a display-mapped content frame 126 obtained by applying frame-level displaymapping parameters 128 to a content frame 104 may be rendered on display device 140.

[0060] FIG. 3 is a block diagram that shows examples of components of an apparatus 300 capable of implementing various aspects of this disclosure. According to some aspects of the disclosure, apparatus 300 can be capable of implementing operations performed by one or both of content engine 102 and display-mapping engine 125 in operating environment 100 of FIG. 1. According to some aspects of the disclosure, apparatus 300 can be capable of implementing operations associated with method 200 of FIG. 2. As with other figures provided herein, the types and numbers of elements shown in FIG. 3 are merely provided by way of example. Other implementations may include more, fewer and/or different types and numbers of elements.

According to some examples, the apparatus 300 may be configured for performing at least some of the methods disclosed herein. In some implementations, the apparatus 300 may be, or may include, a television, one or more components of an audio, video, or multimedia system, a mobile device (such as a cellular telephone), a laptop computer, a tablet device, a smart speaker, or another type of device. [0061] According to some implementations the apparatus 300 may be, or may include, a server. In some such examples, the apparatus 300 may be, or may include, an encoder. Tn some implementations the apparatus 300 may be a device that is configured for use within an audio, video, or multimedia environment, such as a home audio, video, or multimedia environment, whereas in other instances the apparatus 300 may be a device that is configured for use in "the cloud," e.g., a server.

[0062] In this example, the apparatus 300 includes an interface system 305 and a control system 310. The interface system 305 may, in some implementations, be configured for communication with one or more other devices of an audio, video, or multimedia environment. The audio, video, or multimedia environment may, in some examples, be a home audio, video, or multimedia environment. In other examples, the audio, video, or multimedia environment may be another type of environment, such as an office environment, an automobile environment, a train environment, a street or sidewalk environment, a park environment, etc. The interface system 305 may, in some implementations, be configured for exchanging control information and associated data with audio, video, or multimedia devices of the audio, video, or multimedia environment. The control information and associated data may, in some examples, pertain to one or more software applications that the apparatus 300 is executing.

[0063] The interface system 305 may, in some implementations, be configured for receiving, or for providing, a content stream. The content stream may include audio, video, or multimedia data. The audio, video, or multimedia data may include, but may not be limited to, audio, video, or multimedia signals. In some instances, the content stream can include audio data comprising spatial data, such as channel data and/or spatial metadata. In some examples, the content stream may include video data and audio data corresponding to the video data.

[0064] The interface system 305 may include one or more network interfaces and/or one or more external device interfaces (such as one or more universal serial bus (USB) interfaces). According to some implementations, the interface system 305 may include one or more wireless interfaces. The interface system 305 may include one or more devices for implementing a user interface, such as one or more microphones, one or more speakers, a display system, a touch sensor system and/or a gesture sensor system. In some examples, the interface system 305 may include one or more interfaces between the control system 310 and a memory system, such as the optional memory system 315 shown in FIG. 3. However, the control system 310 may include a memory system in some instances. The interface system 305 may, in some implementations, be configured for receiving input from one or more microphones in an environment. [0065] The control system 310 may, for example, include a general purpose single- or multichip processor, a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic device, discrete gate or transistor logic, and/or discrete hardware components.

[0066] In some implementations, the control system 310 may reside in more than one device. For example, in some implementations a portion of the control system 310 may reside in a device within one of the environments depicted herein and another portion of the control system 310 may reside in a device that is outside the environment, such as a server, a mobile device (e.g., a smartphone or a tablet computer), etc. In other examples, a portion of the control system 310 may reside in a device within one environment and another portion of the control system 310 may reside in one or more other devices of the environment. For example, a portion of the control system 310 may reside in a device that is implementing a cloud-based service, such as a server, and another portion of the control system 310 may reside in another device that is implementing the cloud-based service, such as another server, a memory device, etc. The interface system 305 also may, in some examples, reside in more than one device.

[0067] In some implementations, the control system 310 may be configured for performing, at least in part, the methods disclosed herein. According to some examples, the control system 310 may be configured for implementing methods of utilizing an aggressiveness control parameter when training a machine learning model, utilizing an aggressiveness control parameter in post-processing, or the like.

[0068] Some or all of the methods described herein may be performed by one or more devices according to instructions (e.g., software) stored on one or more non-transitory media. Such non-transitory media may include memory devices such as those described herein, including but not limited to random access memory (RAM) devices, read-only memory (ROM) devices, etc. The one or more non-transitory media may, for example, reside in the optional memory system 315 shown in FIG. 3 and/or in the control system 310. Accordingly, various innovative aspects of the subject matter described in this disclosure can be implemented in one or more non-transitory media having software stored thereon. The software may, for example, include instructions for utilizing an aggressiveness control parameter when training a machine learning model, utilizing an aggressiveness control parameter in post-processing, etc. The software may, for example, be executable by one or more components of a control system such as the control system 310 of FIG. 3. [0069] In some examples, the apparatus 300 may include the optional microphone system 320 shown in FIG. 3. The optional microphone system 320 may include one or more microphones. In some implementations, one or more of the microphones may be part of, or associated with, another device, such as a speaker of the speaker system, a smart audio, video, or multimedia device, etc. In some examples, the apparatus 300 may not include a microphone system 320. However, in some such implementations the apparatus 300 may nonetheless be configured to receive microphone data for one or more microphones in an audio, video, or multimedia environment via the interface system 310. In some such implementations, a cloudbased implementation of the apparatus 300 may be configured to receive microphone data, or a noise metric corresponding at least in part to the microphone data, from one or more microphones in an audio, video, or multimedia environment via the interface system 310.

[0070] According to some implementations, the apparatus 300 may include the optional loudspeaker system 325 shown in FIG. 3. The optional loudspeaker system 325 may include one or more loudspeakers, which also may be referred to herein as "speakers" or, more generally, as "audio reproduction transducers." In some examples (e.g., cloud-based implementations), the apparatus 300 may not include a loudspeaker system 325. In some implementations, the apparatus 300 may include headphones. Headphones may be connected or coupled to the apparatus 300 via a headphone jack or via a wireless connection (e.g., Bluetooth).

[0071] According to some implementations, apparatus 300 can include or be coupled with a display 340. Display 340 can comprise a display device capable of displaying visual information, content, and/or effects, such as, for example, display-mapped content frames 126 of operating environment 100 of FIG. 1. Examples of display 340 can include, without limitation, a television, a monitor, a laptop computer screen, a projector, and a touchscreen.

[0072] With regard to the processes, systems, methods, heuristics, etc. described herein, it should be understood that, although the steps of such processes, etc. have been described as occurring according to a certain ordered sequence, such processes could be practiced with the described steps performed in an order other than the order described herein. It further should be understood that certain steps could be performed simultaneously, that other steps could be added, or that certain steps described herein could be omitted. In other words, the descriptions of processes herein are provided for the purpose of illustrating certain embodiments, and should in no way be construed so as to limit the claims. [0073] Accordingly, it is to be understood that the above description is intended to be illustrative and not restrictive. Many embodiments and applications other than the examples provided would be apparent upon reading the above description. The scope should be determined, not with reference to the above description, but should instead be determined with reference to the appended claims, along with the full scope of equivalents to which such claims are entitled. It is anticipated and intended that future developments will occur in the technologies discussed herein, and that the disclosed systems and methods will be incorporated into such future embodiments. In sum, it should be understood that the application is capable of modification and variation.

[0074] All terms used in the claims are intended to be given their broadest reasonable constructions and their ordinary meanings as understood by those knowledgeable in the technologies described herein unless an explicit indication to the contrary in made herein. In particular, use of the singular articles such as “a,” “the,” “said,” etc. should be read to recite one or more of the indicated elements unless a claim recites an explicit limitation to the contrary.

[0075] The Abstract of the Disclosure is provided to allow the reader to quickly ascertain the nature of the technical disclosure. It is submitted with the understanding that it will not be used to interpret or limit the scope or meaning of the claims. In addition, in the foregoing Detailed Description, it can be seen that various features are grouped together in various embodiments for the purpose of streamlining the disclosure. This method of disclosure is not to be interpreted as reflecting an intention that the claimed embodiments incorporate more features than are expressly recited in each claim. Rather, as the following claims reflect, inventive subject matter lies in less than all features of a single disclosed embodiment. Thus, the following claims are hereby incorporated into the Detailed Description, with each claim standing on its own as a separately claimed subject matter.

[0076] Various aspects of the present invention may be appreciated from the following Enumerated Example Embodiments (EEEs):

1. A method, comprising: obtaining a content frame generated by a content engine; determining a presence of one or more objects in the content frame; determining frame-level display-mapping parameters for the content frame based on object-level display-mapping parameters for at least one of the one or more objects; applying the frame-level display-mapping parameters to the content frame to obtain a display-mapped content frame; and rendering the display-mapped content frame on a display device.

2. The method of EEE1, further comprising: composing present object information based on one or more object event notifications received from the content engine; and determining the presence of the one or more objects based on the present object information.

3. The method of EEE2, wherein each of the object event notifications includes an object identifier associated with a respective object.

4. The method of EEE 2 or 3, further comprising: receiving an object exit notification associated with an object; modifying the present object information based on the object exit notification; and determining an absence of the object from a second content frame based on the modified present object notification.

5. The method of any of EEEs 1 to 4, wherein at least one of the frame-level display-mapping parameters for the content frame is determined based on respective object-level display-mapping parameters for multiple objects in the content frame.

6. The method of EEE5, wherein the at least one of the frame-level display-mapping parameters for the content frame is determined as an average of the respective object-level display-mapping parameters for the multiple objects in the content frame.

7. The method of EEE5, wherein the at least one of the frame-level display-mapping parameters for the content frame is determined as a median or mode of the respective object-level display-mapping parameters for the multiple objects in the content frame.

8. The method of EEE5, wherein the at least one of the frame-level display-mapping parameters for the content frame is determined as a maximum or minimum of the respective object-level display-mapping parameters for the multiple objects in the content frame. 9. The method of EEE5, wherein the at least one of the frame-level display-mapping parameters for the content frame is determined as a weighted summation of the respective object-level display-mapping parameters for the multiple objects in the content frame.

10. The method of any of EEEs 1 to 9, wherein the frame-level display-mapping parameters include an ambient color temperature parameter.

11. The method of any of EEEs 1 to 10, wherein the frame-level display-mapping parameters include a contrast parameter.

12. The method of any of EEEs 1 to 11, wherein the frame-level display-mapping parameters include a luminance parameter.

13. The method of any of EEEs 1 to 12, wherein the frame-level display-mapping parameters include a hue parameter.

14. The method of any of EEEs 1 to 13, wherein the frame-level display-mapping parameters include a color saturation parameter.

15. The method of any of EEEs 1 to 14, wherein the content frame comprises content of a game engine.

16. The method of any of EEEs 1 to 14, wherein the content frame comprises content of a live broadcast.

17. The method of any of EEEs 1 to 14, wherein the content frame comprises augmented reality (AR) content, virtual reality (VR) content, or a combination of both.

18. An apparatus configured to perform the method of any one of EEEs 1 to 17.

19. A system, comprising: the apparatus of EEE18; and a display. 20. A computer-readable medium having stored therein instructions that, when executed by one or more processors, cause the one or more processors to perform the method of any one of EEEs 1 to 17.

Claims

CLAIMS What is claimed is:

1. A method, comprising: obtaining a content frame generated by a content engine; determining a presence of one or more objects in the content frame; determining frame-level display-mapping parameters for the content frame based on object-level display-mapping parameters for at least one of the one or more objects; applying the frame-level display-mapping parameters to the content frame to obtain a display -mapped content frame; and rendering the display -mapped content frame on a display device.

2. The method of claim 1, further comprising: composing present object information based on one or more object event notifications received from the content engine; and determining the presence of the one or more objects based on the present object information.

3. The method of claim 2, wherein each of the object event notifications includes an object identifier associated with a respective object.

4. The method of claim 2s or 3, further comprising: receiving an object exit notification associated with an object; modifying the present object information based on the object exit notification; and determining an absence of the object from a second content frame based on the modified present object notification.

5. The method of any of claims 1 to 4, wherein at least one of the frame-level display-mapping parameters for the content frame is determined based on respective object-level display-mapping parameters for multiple objects in the content frame.

6. The method of claim 5, wherein the at least one of the frame- level display-mapping parameters for the content frame is determined as an average of the respective object-level display-mapping parameters for the multiple objects in the content frame.

7. The method of claim 5, wherein the at least one of the frame-level display-mapping parameters for the content frame is determined as a median or mode of the respective object-level display-mapping parameters for the multiple objects in the content frame.

8. The method of claim 5, wherein the at least one of the frame- level display-mapping parameters for the content frame is determined as a maximum or minimum of the respective object-level display-mapping parameters for the multiple objects in the content frame.

9. The method of claim 5, wherein the at least one of the frame- level display-mapping parameters for the content frame is determined as a weighted summation of the respective object-level display-mapping parameters for the multiple objects in the content frame.

10. The method of any of claims 1 to 9, wherein the frame-level display-mapping parameters include an ambient color temperature parameter.

11. The method of any of claims 1 to 10, wherein the frame-level display-mapping parameters include a contrast parameter.

12. The method of any of claims 1 to 11, wherein the frame-level display-mapping parameters include a luminance parameter.

13. The method of any of claims 1 to 12, wherein the frame-level display-mapping parameters include a hue parameter.

14. The method of any of claims 1 to 13, wherein the frame-level display-mapping parameters include a color saturation parameter.

15. The method of any of claims 1 to 14, wherein the content frame comprises content of a game engine.

16. The method of any of claims 1 to 14, wherein the content frame comprises content of a live broadcast.

17. The method of any of claims 1 to 14, wherein the content frame comprises augmented reality (AR) content, virtual reality (VR) content, or a combination of both.

18. An apparatus configured to perform the method of any one of claims 1 to 17.

19. A system, comprising: the apparatus of claim 18; and a display.

20. A computer-readable medium having stored therein instructions that, when executed by one or more processors, cause the one or more processors to perform the method of any one of claims 1 to 17.