WO2025019443A1

WO2025019443A1 - Multi-sensory (ms) spatial mapping and characterization for ms rendering

Info

Publication number: WO2025019443A1
Application number: PCT/US2024/038079
Authority: WO
Inventors: Benjamin SOUTHWELL; Jaclyn Anne Pytlarz; David GUNAWAN; Yin-Lee HO; Carlos Eduardo Medaglia DYONISIO; Robin Atkins
Original assignee: Dolby Laboratories Licensing Corporation
Priority date: 2023-07-17
Filing date: 2024-07-15
Publication date: 2025-01-23

Abstract

Some methods may involve obtaining controllable actuator location information for a set of one or more controllable actuators in an environment, the controllable actuators including one or more light fixtures, one or more haptic sensors, one or more airflow control devices, or combinations thereof. Some methods may involve obtaining controllable actuator capability information for each controllable actuator of the set of controllable actuators and obtaining environment information corresponding to the environment. Some methods may involve generating, based at least in part on the controllable actuator location information, the controllable actuator capability information, and the environment information, an actuator-room response (ARR) that summarizes responses of the environment to controllable actuator activations. Some methods may involve modifying the ARR to produce an actuator map (AM). The modifying may involve regularization, filling gaps in the ARR, reducing one or more overlapping volumes of the environment affected by multiple actuator responses, or combinations thereof.

Description

MULTI-SENSORY (MS) SPATIAL MAPPING AND CHARACTERIZATION FOR MS RENDERING

CROSS-REFERENCE TO RELATED APPLICATIONS

[0001] This application claims the benefit of priority of U.S. Provisional Application No. 63/669,236 filed July 10, 2024 and U.S. Provisional Application No. 63/514,105 filed July 17, 2023, each of which is hereby incorporated by reference in their entireties.

TECHNICAL FIELD

[0002] The present disclosure relates to providing multi-sensory (MS) experiences, and is more specifically directed to mapping and characterizing actuators and the environment in which the actuators are deployed.

BACKGROUND

[0003] Unless otherwise indicated herein, the approaches described in this section are not prior art to the claims in this application and are not admitted as prior art by inclusion in this section.

[0004] Media content delivery has generally focused on audio and screen-based visual experiences. There has been limited delivery of multi-sensory content due to the bespoke nature of actuation. Luminaires, for example, are used extensively as an expression of art and function for concerts. However, each installation is designed specifically for a unique set of luminaires. Delivering a lighting design beyond the set of fixtures the system was designed for is generally not feasible. Other systems that attempt to deliver light experiences more broadly simply do so by extending the screen visuals algorithmically, but are not specifically authored. Haptics content is designed for a specific haptics apparatus. If another device, such as a game controller, mobile phone or even a different brand of haptics device is used, there has been no way to translate the creative intent of content to the different actuators.

SUMMARY

[0005] At least some aspects of the present disclosure may be implemented via methods, such as audio processing methods. In some instances, the methods may be implemented, at least in part, by a control system such as those disclosed herein. [0006] Some methods may involve obtaining, by a control system, controllable actuator location information for a set of one or more controllable actuators in an environment. The one or more controllable actuators may include one or more light fixtures, one or more haptic sensors, one or more airflow control devices, or combinations thereof. Some methods may involve obtaining, by the control system, controllable actuator capability information for each controllable actuator of the set of one or more controllable actuators. Some methods may involve obtaining, by the control system, environment information corresponding to the environment. Some methods may involve generating, by the control system and based at least in part on the controllable actuator location information, the controllable actuator capability information, and the environment information, an actuatorroom response (ARR) that summarizes responses of the environment to one or more activations of the set of one or more controllable actuators.

[0007] In some examples, the environment information may include at least one or more locations, one or more orientations and one or more dimensions of one or more environment features. The one or more environment features may, in some examples, include one or more structural elements of the environment. The one or more structural elements may, in some examples, include one or more walls, a ceiling, a floor, or a combination thereof. The one or more environment features may, in some examples, include one or more furniture items. In some examples, the environment information may include furniture position information. According to some examples, the environment information may include environment feature color information, environment feature reflectivity information, or a combination thereof.

[0008] In some examples, obtaining the controllable actuator location information may involve receiving, by the control system, camera data from one or more cameras. The camera data may, in some examples, include one or more images of the set of one or more controllable actuators. In some examples, obtaining the controllable actuator location information may involve determining, by the control system, the controllable actuator location information based at least in part on the camera data. Some methods may involve determining, by the control system, controllable actuator actuation direction information based at least in part on the camera data. The controllable actuator actuation direction information may, in some examples, include light direction information, air movement direction information, or both.

[0009] According to some examples, the camera data may include one or more optical images, one or more depth images, or a combination thereof. Some methods may involve obtaining, by the control system, inertial measurement data. The inertial measurement data may, in some examples, include gyroscope data, accelerometer data, or both. The controllable actuator location information may, in some examples, be determined according to a simultaneous location and mapping (SLAM) process based at least in part on the camera data and the inertial measurement data.

[0010] In some examples, the one or more controllable actuators may include one or more controllable light fixtures. Some methods may involve sending, by the control system, one or more calibration signals to the one or more controllable light fixtures while the one or more cameras are obtaining the camera data. The one or more calibration signals may cause the one or more controllable light fixtures to modulate light intensity, light color, or both. Obtaining the controllable actuator capability information and obtaining the environment information may be based, at least in part, on the camera data and the inertial measurement data. In some examples, the one or more calibration signals may be, or may include, a sequence having bounded cross-correlations within a set.

[0011] According to some examples, the one or more calibration signals may control the one or more controllable light fixtures to emit light according to a code or a pattern that allows the control system to isolate one or more lighting effects caused by each of the one or more controllable light fixtures. Some methods may involve color matching of light emitted by two or more of the controllable light fixtures.

[0012] According to some examples, obtaining the controllable actuator capability information may involve a light characterization process of determining characteristics of one or more lights within the environment. Some methods may involve a light localization and segmentation process based at least on part on the camera data. The light characterization process may be based, at least in part, on the light localization and segmentation process. The light localization and segmentation process may, in some examples, involve analyzing pixels of the camera data and determining which pixels correspond with individual lights of the environment.

[0013] In some examples, obtaining the environment information may involve an environment characterization process of determining environment feature color information and determining environment feature reflectivity information. According to some examples, obtaining the controllable actuator capability information and obtaining the environment information may involve an iterative process of determining one or more characteristics of lights within the environment, determining environment feature color information and determining environment feature reflectivity information. [0014] Some methods may involve modifying, by the control system, the ARR to produce an actuator map (AM). According to some examples, the modifying may include regularization, filling gaps in the ARR by increasing one or more volumes of the environment affected by an actuator response, reducing one or more overlapping volumes of the environment affected by multiple actuator responses, or combinations thereof. In some examples, the AM may indicate an effect of each controllable actuator on the environment. According to some examples, the AM may be mathematically suitable for inversion. In some examples, the AM may allow rendering of object-based sensory data to actuator commands.

[0015] Some methods may involve identifying, by the control system, one or more ill-posed spatial coordinates of the AM. Some such methods may involve generating, by the control system, one or more repeller functions configured to deflect a trajectory of sensory objects from the one or more ill-posed spatial coordinates. Some such methods may involve saving or providing the one or more repeller functions with the AM.

[0016] In some examples, the ARR may be represented as a matrix and the AM may be represented as a modified version of the matrix. According to some examples, the matrix may include values corresponding to one or more actuator indices, one or more actuation commands and spatial coordinates.

[0017] In some examples, obtaining the controllable actuator location information may involve receiving controllable actuator capability information via an interface system. According to some examples, the interface system may be, or may include, a user interface system.

[0018] Some or all of the operations, functions and/or methods described herein may be performed by one or more devices according to instructions (e.g., software) stored on one or more computer-readable non-transitory media. Such non-transitory media may include one or more memory devices such as those described herein, including but not limited to one or more random access memory (RAM) devices, read-only memory (ROM) devices, etc. Accordingly, some innovative aspects of the subject matter described in this disclosure can be implemented in one or more computer-readable non-transitory media having software stored thereon.

[0019] At least some aspects of the present disclosure may be implemented via apparatus. For example, one or more devices may be capable of performing, at least in part, the methods disclosed herein. In some implementations, an apparatus may include an interface system and a control system. The control system may include one or more general purpose single- or multi-chip processors, digital signal processors (DSPs), application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs) or other programmable logic devices, discrete gates or transistor logic, discrete hardware components, or combinations thereof. The control system may be configured to perform some or all of the disclosed methods.

[0020] Details of one or more implementations of the subject matter described in this specification are set forth in the accompanying drawings and the description below. Other features, aspects, and advantages will become apparent from the description, the drawings, and the claims. Note that the relative dimensions of the following figures may not be drawn to scale.

BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Disclosed embodiments now be described, by way of example only, with reference to the accompanying drawings.

[0022] Figure 1 A is a block diagram that shows examples of components of an apparatus capable of implementing various aspects of this disclosure.

[0023] Figure IB shows example elements of an endpoint.

[0024] Figure 2 shows examples of actuator elements.

[0025] Figure 3 shows example elements of a system for the creation and playback of multi- sensory (MS) experiences.

[0026] Figure 4A shows example elements of a multi-sensory (MS) Tenderer.

[0027] Figure 4B is a flow diagram that outlines one example of a method that may be performed by an apparatus or system such as those disclosed herein.

[0028] Figure 5 shows example elements of another system for the creation and playback of MS experiences.

[0029] Figure 6 depicts the light-room-response (LRR) of a single light fixture in egocentric polar coordinates.

[0030] Figure 7 shows example elements of a light mapping system. [0031] Figure 8 shows an example of a color volume.

[0032] Figure 9 depicts an alternative method of producing an LRR.

[0033] Figure 10 shows examples of information that may be provided to the simulator of Figure 9.

[0034] Figures 11 A and 1 IB show examples of augmented LRRs.

[0035] Figure 12 is a flow diagram that outlines one example of a method that may be performed by an apparatus or system such as those disclosed herein.

DETAILED DESCRIPTION

[0036] Described herein are techniques related to providing multi-sensory media content. In the following description, for purposes of explanation, numerous examples and specific details are set forth in order to provide a thorough understanding of the present disclosure. It will be evident, however, to one skilled in the art that the present disclosure as defined by the claims may include some or all of the features in these examples alone or in combination with other features described below, and may further include modifications and equivalents of the features and concepts described herein.

[0037] In the following description, various methods, processes and procedures are detailed. Although particular steps may be described in a certain order, such order is mainly for convenience and clarity. A particular step may be repeated more than once, may occur before or after other steps (even if those steps are otherwise described in another order), and may occur in parallel with other steps. A second step is required to follow a first step only when the first step must be completed before the second step is begun. Such a situation will be specifically pointed out when not clear from the context.

[0038] In this document, the terms “and”, “or” and “and/or” are used. Such terms are to be read as having an inclusive meaning. For example, “A and B” may mean at least the following: “both A and B”, “at least both A and B”. As another example, “A or B” may mean at least the following: “at least A”, “at least B”, “both A and B”, “at least both A and B”. As another example, “A and/or B” may mean at least the following: “A and B”, “A or B”. When an exclusive-or is intended, such will be specifically noted (e.g., “either A or B”, “at most one of A and B”). [0039] This document describes various processing functions that are associated with structures such as blocks, elements, components, circuits, etc. In general, these structures may be implemented by one or more processors controlled by one or more computer programs.

[0040] As noted above, media content delivery has generally been focused on audio and video experiences. There has been limited delivery of multi-sensory (MS) content due to the customized nature of actuation.

[0041] This application describes methods for extending the creative palette for content creators, allowing spatial, MS experiences to be created and delivered at scale. Some such methods involve the introduction of new layers of abstraction, in order to allow authored MS experiences to be delivered to different endpoints, with different types of fixtures or actuators. As used herein, the term “endpoint” is synonymous with “playback environment” or simply “environment,” meaning an environment that includes one or more actuators that may be used to provide an MS experience. Such endpoints may include a room, such as the living room of a home, a car, a cinema, a night club or other venue, etc. Some disclosed methods involve the creation, delivery and/or rendering of object-based sensory data, which may include sensory objects and corresponding sensory metadata. This abstraction allows creative intent to be implemented in an object-based format that does not require prior knowledge of the specific controller types, numbers and layout of any particular playback environment or “endpoint,” thereby enabling greater flexibility and scalability. An MS experience provided via object-based sensory data may be referred to herein as a “flexibly- scaled MS experience.”

[0042] Acronyms

MS - multisensory

MSIE - MS Immersive Experience AR - Augmented Reality VR - Virtual Reality PC - personal computer

[0043] Figure 1 A is a block diagram that shows examples of components of an apparatus capable of implementing various aspects of this disclosure. As with other figures provided herein, the types and numbers of elements shown in Figure 1 A are merely provided by way of example. Other implementations may include more, fewer and/or different types and numbers of elements. According to some examples, the apparatus 101 may be, or may include, a device that is configured for performing at least some of the methods disclosed herein, such as a smart audio device, a laptop computer, a cellular telephone, a tablet device, a smart home hub, etc. In some such implementations the apparatus 101 may be, or may include, a server that is configured for performing at least some of the methods disclosed herein.

[0044] In this example, the apparatus 101 includes at least an interface system 105 and a control system 110. In some implementations, the control system 110 may be configured for performing, at least in part, the methods disclosed herein. According to some examples, the control system 110 may be configured for determining environment and actuator data, such as the environment and actuator data 004 that is described herein with reference to Figures 3, 4 A and 5.

[0045] In some examples, the control system 110 may be configured for obtaining — for example, via the interface system 105 — controllable actuator location information for a set of controllable actuators in an environment, which is a playback environment in this example. The playback environment may, for example, be a room of a house, a vehicle, etc. According to this example, the controllable actuators include one or more light fixtures, one or more haptic sensors, one or more airflow control devices, or combinations thereof. According to some examples, the control system 110 may be configured for obtaining controllable actuator actuation direction information, such as light direction information, air movement direction information, etc.

[0046] According to some examples, the control system 110 may be configured for obtaining controllable actuator capability information for each controllable actuator of the set of controllable actuators. Each controllable actuator may require a specific type and/or level of control to produce a desired output. For example, a Philips Hue™ light bulb may need to receive control information in a particular format to turn the light on, with a particular saturation, brightness and hue, and a digital representation of the desired drive level. Accordingly, the controllable actuator capability information may indicate the controllable actuator type, required control signal format, if any, the drive level required to obtain a particular actuator response, etc. Obtaining the controllable actuator capability information may, in some examples, involve a light characterization process of determining characteristics of controllable lights within the environment.

[0047] In some examples, the control system 110 may be configured for obtaining environment information corresponding to the environment, in other words corresponding to the environment in which the controllable actuators are located. The environment information may include locations, orientations and/or dimensions of one or more environment features. The one or more environment features may include one or more structural elements of the environment, such as one or more walls, a ceiling, a floor, or a combination thereof. In some examples, the environment information may include environment feature color information, environment feature reflectivity information, or a combination thereof. The one or more environment features may include one or more furniture items. In some such examples, the environment information may include furniture position information.

[0048] Various methods of obtaining the controllable actuator location information, the controllable actuator capability information and the environment information are disclosed herein. Such methods include automated methods, manual/user input-based methods, and “hybrid” methods that are partly automated and partly manual.

[0049] According to some examples, the control system 110 may be configured for generating an actuator-room response (ARR) that summarizes responses of the environment to activations of the set of controllable actuators.

[0050] The matrix representation of ARR is often ill-posed for inversion. For example, according to some matrix representations of ARR, if there are no actuator effects at a particular location of the playback environment, the ARR matrix values of a number of columns or rows may be zero.

[0051] In some examples, the control system 110 may be configured for modifying an ARR — for example modifying a matrix representation of the ARR — to produce an actuator map (AM) that indicates an effect of each controllable actuator on the environment. The AM may allow rendering of object-based sensory data to actuator commands. According to some examples, the matrix representation of the AM is mathematically suitable for inversion. Modifying the ARR may involve regularization, filling gaps in the ARR — for example, eliminating zero values of the ARR matrix — by increasing one or more volumes of the environment affected by an actuator response, reducing one or more overlapping volumes of the environment affected by multiple actuator responses, or combinations thereof.

[0052] The interface system 105 may include one or more network interfaces and/or one or more external device interfaces (such as one or more universal serial bus (USB) interfaces). According to some implementations, the interface system 105 may include one or more wireless interfaces. The interface system 105 may include one or more devices for implementing a user interface, such as one or more microphones, one or more speakers, a display system, a touch sensor system and/or a gesture sensor system. In some examples, the interface system 105 may include one or more interfaces between the control system 110 and a memory system, such as the optional memory system 115 shown in Figure 1 A. However, the control system 110 may include a memory system in some instances.

[0053] The control system 110 may, for example, include a general purpose single- or multichip processor, a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic device, discrete gate or transistor logic, and/or discrete hardware components.

[0054] In some implementations, the control system 110 may reside in more than one device. For example, a portion of the control system 110 may reside in a device within an environment (such as a laptop computer, a tablet computer, a smart audio device, etc.) and another portion of the control system 110 may reside in a device that is outside the environment, such as a server. In other examples, a portion of the control system 110 may reside in a device within an environment and another portion of the control system 110 may reside in one or more other devices of the environment.

[0055] Some or all of the methods described herein may be performed by one or more devices according to instructions (e.g., software) stored on one or more non-transitory media. Such non-transitory media may include memory devices such as those described herein, including but not limited to random access memory (RAM) devices, read-only memory (ROM) devices, etc. The one or more non-transitory media may, for example, reside in the optional memory system 115 shown in Figure 1 A and/or in the control system 110. Accordingly, various innovative aspects of the subject matter described in this disclosure can be implemented in one or more non-transitory media having software stored thereon. The software may, for example, include instructions for controlling at least one device to process audio data. The software may, for example, be executable by one or more components of a control system such as the control system 110 of Figure 1A.

[0056] In some examples, the apparatus 101 may include the optional microphone system 120 shown in Figure 1 A. The optional microphone system 120 may include one or more microphones. In some implementations, one or more of the microphones may be part of, or associated with, another device, such as a speaker of the speaker system, a smart audio device, etc.

[0057] According to some implementations, the apparatus 101 may include the optional actuator system 125 shown in Figure 1 A. The optional actuator system 125 may include one or more loudspeakers, one or more haptic devices, one or more light fixtures, also referred to herein as luminaires, one or more fans or other air-moving devices, one or more display devices, including but not limited to one or more televisions, one or more positional actuators, one or more other types of devices for providing an MS experience, or combinations thereof. The term “light fixture” as used herein refers generally to any actuator that is configured to provide light. The term “light fixture” encompasses various types of light sources, including individual light sources such as light bulbs, groups of light sources such as light strips, light panels such as light-emitting diode (LED) panels, projectors, display devices such as television (TV) screens, etc. A “light fixture” may be moveable, and therefore the word “fixture” in this context does not mean that a light fixture is necessarily in a fixed position in space. The term “positional actuators” as used herein refers generally to devices that are configured to change a position or orientation of a person or object, such as motion simulator seats. Loudspeakers may sometimes be referred to herein as “speakers.” In some implementations, the optional actuator system 125 may include a display system including one or more displays, such as one or more light-emitting diode (LED) displays, one or more organic light-emitting diode (OLED) displays, etc. In some examples wherein the apparatus 101 includes a display system, the optional sensor system 130 may include a touch sensor system and/or a gesture sensor system proximate one or more displays of the display system. According to some such implementations, the control system 110 may be configured for controlling the display system to present a graphical user interface (GUI), such as a GUI related to implementing one of the methods disclosed herein. [0058] In some implementations, the apparatus 101 may include the optional sensor system 130 shown in Figure 1 A. The optional sensor system 130 may include a touch sensor system, a gesture sensor system, one or more cameras, etc.

[0059] Figure IB shows example elements of an endpoint. In this example, the endpoint is a living room 1001 containing multiple actuators 008, some furniture 1010 and a person 1000 — also referred to herein as a user — who will consume a flexibly-scaled MS experience. Actuators 008 are devices capable of altering the environment 1001 that the user 1000 is in. Actuators 008 may include one or more haptic devices, one or more light fixtures, also referred to herein as luminaires, one or more fans or other air-moving devices, one or more display devices, including but not limited to one or more televisions, one or more positional actuators, one or more other types of devices for providing an MS experience, or combinations thereof.

[0060] The number of actuators 008, the arrangement of actuators 008 and the capabilities of actuators 008 in the space 1001 may vary significantly between different endpoint types. For example, the number, arrangement and capabilities of actuators 008 in a car or other vehicle will generally be different from the number, arrangement and capabilities of actuators 008 in a living room, a night club, a patio, etc. In many implementations, the number, arrangement and/or capabilities of actuators 008 may vary significantly between different instances of the same type, e.g., between a small living room with 2 actuators 008 and a large living room with 16 actuators 008. The present disclosure describes various methods for creating and delivering flexibly-scaled MSIEs to these non-homogenous endpoints.

[0061] Figure 2 shows examples of actuator elements. In this example, the actuator is a luminaire 1100, which includes a network module 1101, a control module 1102 and a light emitter 1103. The terms “luminaire” and “light fixture” may be used synonymously herein. According to this example, the light emitter 1103 includes one or more light-emitting devices, such as light-emitting diodes, which are configured to emit light into an environment in which the luminaire 1100 resides. In this example, the network module 1101 is configured to provide network connectivity to one or more other devices in the space, such as a device that sends commands to control the emission of light by the luminaire 1100. According to this example, the network module 1101 is an instance of the interface system 105 of Figure 1A. In this example, the control module 1102 is configured to receive signals via the network module 1101 and to control the light emitter 1103 accordingly. According to this example, the control module 1102 is an instance of the control system 110 of Figure 1A.

[0062] Other examples of actuators also may include a network module 1101 and a control module 1102, but may include other types of actuating elements. Some such actuators may include one or more haptic devices, one or more fans or other air-moving devices, one or more positional actuators, etc.

[0063] Figure 3 shows example elements of a system for the creation and playback of multi- sensory (MS) experiences. As with other figures provided herein, the types and numbers of elements shown in Figure 3 are merely provided by way of example. Other implementations may include more, fewer and/or different types and numbers of elements. According to some examples, system 300 may be, or may include, one or more devices configured for performing at least some of the methods disclosed herein. In some examples, system 300 may include one or more instances of the control system 110 of Figure 1 A that are configured for performing at least some of the methods disclosed herein.

[0064] According to the examples in the present disclosure, creating and providing an object-based MS Immersive Experience (MSIE) approach involves the application of a suite of technologies for creation, delivery and rendering of object-based sensory data, which may include sensory objects and corresponding sensory metadata, to the actuators 008. Some examples are described in the following paragraphs.

[0065] Object-Based Representation: In various disclosed implementations, multi-sensory (MS) effects are represented using what may be referred to herein as multi-sensory (MS) objects, or simply as “sensory objects.” According to some such implementations, properties such as layer-type and priority may be assigned to and associated with attached to each sensory object, enabling content creators’ intent to be represented in the rendered experiences. Detailed examples of sensory object properties are described below.

[0066] In this example, system 300 includes a content creation tool 000 that is configured for designing multi-sensory (MS) immersive content and for outputting object-based sensory data 005, either separately or in conjunction with corresponding audio data 011 and/or video data 012, depending on the particular implementation. The object-based sensory data 005 may include time stamp information, as well as information indicating the type of sensory object, the sensory object properties, etc. In this example, the object-based sensory data 005 is not “channel -based” data that corresponds to one or more particular sensory actuators in a playback environment, but instead is generalized for a wide range of playback environments with a wide range of actuator types, numbers of actuators, etc. In some examples, the object-based sensory data 005 may include object-based light data, object-based haptic data, object-based air flow data, or object-based positional actuator data, object-based olfactory data, object-based smoke data, object based data for one or more other types of sensor effects, or combinations thereof. According to some examples, the object-based sensory data 005 may include sensory objects and corresponding sensory metadata. For example, if the object-based sensory data 005 includes object-based light data, the object-based light data may include light object position metadata, light object color metadata, light object size metadata, light object intensity metadata, light object shape metadata, light object diffusion metadata, light object gradient metadata, light object priority metadata, light object layer metadata, or combinations thereof. In some examples, the object-based sensory data 005 may include time data, such as time stamp information. Although the content creation tool 000 is shown providing a stream of object-based sensory data 005 to the experience player 002 in this example, in alternative examples the content creation tool 000 may produce object-based sensory data 005 that is stored for subsequent use. Examples of graphical user interfaces for a light-object-based content creation tool are described below.

Examples of MS Object Properties

[0067] Following is a non-exhaustive list of possible properties of MS objects:

• Priority;

• Layer;

• Mixing Mode;

• Persistence;

• Effect; and

• Spatial Panning Law.

[0068] Effect

As used herein, the “effect” of an MS object is a synonym for the type of MS object. An “effect” is, or indicates, the sensory effect that the MS object is providing. If an MS object is a light object, its effect will involve providing direct or indirect light. If an MS object is a haptic object, its effect will involve providing some type of haptic feedback. If an MS object is an air flow object, its effect will involve providing some type of air flow. As described in more detail below, some examples involve other “effect” categories.

[0069] Persistence

Some MS objects may contain a persistence property in their metadata. For example, as an moveable MS object moves around in a scene, the moveable MS object may persist for some period of time at locations that the moveable MS object passes through. That period of time may be indicated by persistence metadata. In some implementations, the MS Tenderer is responsible for constructing and maintaining the persistence state.

[0070] Layers

According to some examples, individual MS objects may be assigned to “layers,” in which MS objects are grouped together according to one or more shared characteristics. For example, layers may group MS objects together according to their intended effect or type, which may include but are not limited to the following:

Mood/ Ambience

Informational

Punctuati onal/ Attenti on

Alternatively, or additionally, in some examples, layers may be used to group MS objects together according to shared properties, which may include but are not limited to the following:

Color

Intensity

Size

Shape

Position

Region in space

[0071] Priority

In some examples, MS objects may have a priority property that enables the Tenderer to determine which object(s) should take priority in an environment in which MS objects are contending for limited actuators. For example, if the volumes corresponding to multiple light objects overlap with a single light fixture at a time during which all of the light objects are scheduled to be rendered, a Tenderer may refer to the priority of each light object in order to determine which light object(s) will be rendered. In some examples, priority may be defined between layers or within layers. According to some examples, priority may be linked to specific properties such as intensity. In some examples, priority may be defined temporally: for example, the most recent MS object to be rendered may take precedence over MS objects that have been rendered earlier. According to some examples, priority may be used to specify MS objects or layers that should be rendered regardless of the limitations of a particular actuator system in a playback environment.

[0072] Spatial Panning Laws

Spatial panning laws may define an MS object’s movement across a space, how an MS object affects actuators as it moves between them, etc.

[0073] Mixing Mode

The mixing mode may specify how multiple objects are multiplexed onto a single actuator. In some examples, mixing modes may include one or more of the following:

Max mode: select the MS object which activates an actuator the most;

Mix mode: mix in some or all the objects according to a rule set, for example by summing activation levels, taking the average of activation levels, mixing color according to activation level or priority level, etc.;

MaxNmix: mix in the top N MS objects (by activation level), according to a rule set.

[0074] According to some examples, more general metadata for an entire multi-sensory content file, instead of (or in addition to) per-object metadata may be defined. For example, MS content files may include metadata such as trim passes or mastering environment.

[0075] Trim Controls

[0076] What are referred to in the context of Dolby Vision™ as “trim controls” may act as guidance on how to modulate the default rendering algorithm for specific environments or conditions at the endpoint. Trim controls may specify ranges and/or default values for various properties, including saturation, tone detail, gamma, etc. For example, there may be automotive trim controls, which provide specific defaults and/or rule sets for rendering in automotive environments, for example guidance that includes only objects of a certain priority or layer. Other examples may provide trim controls for environments with limited, complex or sparse multisensory actuators. [0077] Mastering Environment

A single piece of multisensory content may include metadata on the properties of the mastering environment such as room size, reflectivity and ambient bias lighting level. The specific properties may differ depending on the desired endpoint actuators. Mastering environment information can aid in providing reference points for rendering in a playback environment.

[0078] MS Object Renderer: Various disclosed implementations provide a Tenderer that is configured render MS effects to actuators in a playback environment. According to this example, system 300 includes an MS renderer 001 that is configured to render object-based sensory data 005 to actuator control signals 310, based at least in part on environment and actuator data 004. In this example, the MS renderer 001 is configured to output the actuator control signals 310 to MS controllers 003, which are configured to control the actuators 008. In some examples, the MS renderer 001 may be configured to receive light objects and object-based lighting metadata indicating an intended lighting environment, as well as lighting information regarding a local lighting environment. The lighting information is one general type of environment and actuator data 004, and may include one or more characteristics of one or more controllable light sources in the local lighting environment. In some examples, the MS renderer 001 may be configured to determine a drive level for each of the one or more controllable light sources that approximates the intended lighting environment. According to some examples, the MS renderer 001 (or one of the MS controllers 003) may be configured to output the drive level to at least one of the controllable light sources. Some alternative examples may include a separate renderer for each type of actuator 008, such as one renderer for light fixtures, another renderer for haptic devices, another renderer for air flow devices, etc. In other implementations, a single renderer may be configured as an MS renderer and as an audio renderer and/or as a video renderer. In some implementations, the MS renderer 001 may be configured to adapt to changing conditions. Some examples of MS renderer 001 implementations are described in more detail below.

[0079] The environment and actuator data 004 may include what are referred to herein as “room descriptors” that describe actuator locations (e.g., according to an x,y,z coordinate system or a spherical coordinate system). In some examples, the environment and actuator data 004 may indicate actuator orientation and/or placement properties (e.g., directional and north-facing, omnidirectional, occlusion information, etc.). According to some examples, the environment and actuator data 004 may indicate actuator orientation and/or placement properties according to a 3x3 matrix, in which three elements (for example, the elements of the first row) represent spatial position (x,y,z), three other elements (for example, the elements of the second row) represent orientation (roll, pitch, yaw), and three other elements (for example, the elements of the third row) indicate a scale or size (sx, sy, sz). In some examples, the environment and actuator data 004 may include device descriptors that describe the actuator properties relevant to the MS Tenderer 001, such as intensity range and color gamut of a light fixture, the air flow speed range and direction(s) for an air-moving device, etc. Additional details about the environment and actuator data 004, including how the environment and actuator data 004 may be obtained, are described below.

[0080] In this example, system 300 includes an experience player 002 that is configured to receive object-based sensory data 005’, audio data 011’ and video data 012’, and to provide the object-based sensory data 005 to the MS Tenderer 001, to provide the audio data 011 to the audio Tenderer 006 and to provide the video data 012 to the video Tenderer 007. In this example, the reference numbers for the object-based sensory data 005’, audio data 011’ and video data 012’ received by the experience player 002 include primes (‘), in order to suggest that the data may in some instances be encoded. Likewise, the object-based sensory data 005, audio data 011 and video data 012 output by the experience player 002 do not include primes, in order to suggest that the data may in some instances have been decoded by the experience player 002. According to some examples, the experience player 002 may be a media player, a game engine or personal computer or mobile device, or a component integrated in an television, DVD player, sound bar, set top box, or a service provider media device such as a Chromecast, Apple TV device, or Amazon Fire TV. In some examples, the experience player 002 may be configured to receive encoded object-based sensory data 005’ along with encoded audio data OI L and/or encoded video data 012’. In some such examples, the encoded object-based sensory data 005’ may be received as part of the same bitstream with the encoded audio data OI L and/or the encoded video data 012’ . Some examples are described in more detail below. According to some examples, the experience player 002 may be configured to extract the object-based sensory data 005 from the content bitstream and to provide decoded object-based sensory data 005 to the MS Tenderer 001, to provide decoded audio data 011 to the audio Tenderer 006 and to provide decoded video data 012 to the video Tenderer 007. In some examples, time stamp information in the object- based sensory data 005 may be used — for example, by the experience player 102, the MS renderer 001, the audio Tenderer 106, the video Tenderer 107, or all of them — to synchronize effects relating to the object-based sensory data 005 with the audio data 111 and/or the video data 112, which may also include time stamp information.

[0081] According to this example, system 300 includes MS controllers 003 that are configured to communicate with a variety of actuator types using application program interfaces (APIs) or one or more similar interfaces. Generally speaking, each actuator will require a specific type of control signal to produce the desired output from the renderer. According to this example, the MS controllers 003 are configured to map outputs from the MS renderer 001 to control signals for each actuator. For example, a Philips Hue™ light bulb receives control information in a particular format to turn the light on, with a particular saturation, brightness and hue, and a digital representation of the desired drive level. In some alternative examples, the MS renderer 001 also may be configured to implement some or all of the MS controllers 003. For example, the MS renderer 001 also may be configured to implement one or more lighting-based APIs but not haptic-based APIs, or vice versa.

[0082] In some examples, room descriptors also may describe the size and orientation of the playback environment itself, to establish a relative or absolute coordinate system to which all objects are positioned. For example, in a living room a display screen may be regarded as the front, in some instances the front and center, and the floor and ceiling may be regarded as the vertical bounds. In some such examples, the room descriptors also may also indicate bounds corresponding with the left, right, front, and rear, walls relative to the front position. According to some examples, the room descriptor also may be provided in terms of a matrix, such as a 3x3 matrix. This room descriptor information is useful in describing the physical dimensions of the playback environment, for example in physical units of distance such as meters. In some such examples, sensory object locations, sensory object sizes, and sensory object orientations may be described in units that are relative to the room size, for example in a range from -1 to 1. Room descriptors may also describe a preferred viewing position, in some instances according to a matrix.

[0083] The types, numbers and arrangements of the actuators 008 will generally vary according to the particular implementation. In some examples, actuators 008 may include lights and/or light strips (also referred to herein as “luminaires”), vibrational motors, air flow generators, positional actuators, or combinations thereof. [0084] Similarly, the types, numbers and arrangements of the loudspeakers 009 and the display devices 010 will generally vary according to the particular implementation. In the examples shown in Figure 3, audio data 011 and video data 012 are rendered by the audio Tenderer 006 and the video Tenderer 007 to the loudspeakers 009 and display devices 010, respectively.

[0085] As noted above, according to some implementations the system 300 may include one or more instances of the control system 110 of Figure 1 A that are configured for performing at least some of the methods disclosed herein. In some such examples, one instance of the control system 110 may implement the content creation tool 000 and another instance of the control system 110 may implement the experience player 002. In some examples, one instance of the control system 110 may implement the audio Tenderer 006, the video Tenderer 007, the multi-sensory Tenderer 001, or combinations thereof. According to some examples, an instance of the control system 110 that is configured to implement the experience player 002 may also be configured to implement the audio Tenderer 006, the video Tenderer 007, the multi-sensory Tenderer 001, or combinations thereof.

[0086] Figure 4A shows example elements of a multi-sensory (MS) Tenderer. As with other figures provided herein, the types and numbers of elements shown in Figure 4A are merely provided by way of example. Other implementations may include more, fewer and/or different types and numbers of elements. According to this example, the MS Tenderer 001 is an instance of the MS Tenderer 001 that is described with reference to Figure 3. In some examples, the MS Tenderer 001 may be implemented by one or more instances of the control system 110 of Figure 1A.

[0087] According to this example, Figure 4A includes the following elements:

• 004: Environment and actuator data, which may be as described with reference to Figure 3;

• 005: Object-based sensory data 005, which may be as described with reference to Figure 3;

• 423 an actuator map (AM) that indicates the locations of at least the controllable actuators 008 in a particular playback environment;

• 450 a projection module configured to project the MS objects of the object-based sensory data 005 based, at least in part, on the AM 423. The MS objects may also be referred to herein as “sensory objects,” because in some instances only one type of sensory object — such as only haptic objects or only light objects — may be present in the object-based sensory data 005. In this example, the projection module 450 is configured to project the MS objects based, at least in part, on sensory object metadata, which may include at least sensory object location metadata and sensory object size metadata;

• 440 an actuator activation matrix (AAM) that is output by the projection module 450 according to this example. The AAM may, for example, indicate sensory objects, if any, that are currently encompassing a volume within the playback environment corresponding to one or more corresponding actuators. For example, the AAM may indicate whether the light object location metadata and light object size metadata of a light object indicate that a particular light fixture is within a volume of the playback environment corresponding to the location and size of the light object;

• 451 a mixing module configured to convert the AAM 440 into actuator control signals, based at least in part on the environment and actuator data and the Tenderer configuration data;

• 452 optional Tenderer configuration data, which may include information regarding one or more settings for the MS Tenderer 001 such as settings to indicate the desired dynamism, mode, etc. In some examples, the Tenderer configuration data 452 may be changed automatically using context-aware systems such as those described in more detail below; and

• 310: actuator control signals, which may be as described with reference to Figure 3. In some examples, the actuator control signals 310 may be sent to individual actuators 008, whereas in other examples the actuator control signals 310 may be sent to MS controllers 003, which may be configured to send appropriate control signals to various types of actuators 008.

[0088] According to some examples, the AAM 440 is a matrix describing “how much” a sensory object projects itself onto each actuator according to the actuator map 423. In some examples, the AAM 440 may be a real matrix of size No by NA, where No represents the number of sensor objects and NA represents the number of controllable actuators in the environment. In this example, the mixing module 451 is configured to produce the actuator control signals 310 based at least in part on the AAM 440 and the environment and actuator data 004. In some examples, the mixing module 451 may be configured to produce the actuator control signals 310 based at least in part on the optional Tenderer configuration data 452. According to some examples, the mixing module 451 may be configured to produce the actuator control signals 310 based at least in part on sensory object metadata — which may be received as part of the object-based sensory data 005, as shown in Figure 4A — such as mixing and panning laws associated with at least one sensory object.

[0089] In some examples, the mixing module 451 may be configured to produce the actuator control signals 310 based at least in part on one or more of the following:

1. Thresholding elements of the AAM 440;

2. Taking the maximum of a particular column of the AAM 440 — in other words, taking the sensory object that activated a particular actuator the most as the output;

3. Taking any combination of the top N and performing at least one of the following: o Mixing the objects together in the actuator channel; o Pushing objects to adjacent channels.

[0090] In some embodiments, the projection module 450 may be configured to generate a sensory object image using the sensory object’s spatial coordinates — for example, x,y,z coordinates — and the sensory object’s size to produce In(x,y,z), where In represents the sensory object image for the n^th sensor object. Then, in some examples, the projection module 450 may be configured to compute, for every column (actuator index) of the AAM 440, the n^th row (object index), by taking the inner product of this object image and the actuator map corresponding to that actuator. The object image, actuator map and dot product may be produced and performed by the projection module 450 in any spatial domain that is convenient including, without limitation, polar, cylindrical or rectangular coordinate systems.

[0091] Some implementations may involve implementing what may be referred to herein as “repellers,” which may be used to avoid potentially undesirable sensory effects, such as lighting effects, which may be caused when sensory objects are positioned in one or more areas of a playback environment. In some such examples, repeller data may be included with the environment and actuator data 004 and the AM 423, and therefore may be part of the information that is input to the projection module 450. In some such examples, the spatial coordinate of the MS objects will be augmented when projecting them onto the AM 423 to produce the AAM 440.

Multi-Sensory Rendering Synchronization [0092] Object-based MS rendering involves different modalities being rendered flexibly to the endpoint/playback environment. Endpoints have differing capabilities according to various factors, including but not limited to the following:

• The number of actuators,

• The modalities of those actuators (e.g., light fixture vs. air flow control device vs. haptic device);

• The types of those actuators (e.g., a white smart light vs. a RGB smart light, or a haptic vest vs. a haptic seat cushion) and

• The location/layout of those actuators.

[0093] In order to render object-based sensory content to any endpoint, some processing of the object signals, e.g. intensities, colors, patterns etc., will generally need to be done. The processing of each modality’s signal path should not alter the relative phase of certain features within the object signals. For example, suppose that a lightning strike is presented in both the haptics and lightscape modalities. The signal processing chain for the corresponding actuator control signals should not result in a time delay of either type of sensory object signal — haptic or light — sufficient to alter the perceived synchronization of the two modalities. The level of required synchronization may depend on various factors, such as whether the experience is interactive and what other modalities are involved in the experience. Maximum time difference values may, for example, range from approximately 10ms to 100ms, depending on the particular context.

HAPTICS

Rendering of object-based haptics content

[0094] Object-based haptics content conveys sensory aspects of the scene through an abstract sensory representation rather than a channel-based scheme only. For example, instead of defining haptics content as a single-channel time-dependent amplitude signal only, that is in turn played out of a particular haptics actuator such as a vibro-tactile motor in a vest the user wears, object-based haptics content may be defined by the sensations that it is intended to convey. More specifically, in one example, we may have a haptic object representing a collision haptic sensory effect. Associated with this object is:

• The haptic object’s spatial location;

• The spatial direction/vector of the haptic effect;

• The intensity of the haptic effect; • Haptic spatial and temporal frequency data; and

• A time-dependent amplitude signal.

[0095] According to some examples, a haptic object of this type may be created automatically in an interactive experience such as a video game, e.g. in a car racing game when another car hits a player’s car from behind. In this example, the MS Tenderer will determine how to render the spatial modality of this effect to the set of haptic actuators in the endpoint. In some examples, the Tenderer does this according to information about the following:

• The type(s) of haptic devices available, e.g., haptic vest vs. haptic glove vs. haptic seat cushion vs. haptic controller;

• The locale of each haptic device with respect to the user(s) (some haptic devices may not be coupled to the user(s), e.g., a floor- or seat-mounted shaker);

• The type of actuation each haptic device provides, e.g. kinesthetic vs. vibro-tactile;

• The on- and off-set delay of each haptic device (in other words, how fast each haptic device can turn on and off);

• The dynamic response of each haptic device (how much the amplitude can vary);

• The time-frequency response of each haptic device (what time-frequencies the haptic device can provide);

• The spatial distribution of addressable actuators within each haptic device: for example, a haptic vest may have dozens of addressable haptics actuators distributed over the user’ s torso; and

• The time-response of any haptic sensors used to render closed-loop haptic effects (e.g., an active force-feedback kinesthetic haptic device.

[0096] These attributes of the haptics modality of the endpoint will inform the render how best to render a particular haptic effect. Consider the car crash effect example again. In this example, a player is wearing a haptic vest, a haptic arm band and haptic gloves. According to this example, a haptic shockwave effect is spatially located at the place where the car has collided into the player. The shockwave vector is dictated by the relative velocity of the player’s car and the car that has hit the player. The spatial and temporal frequency spectra of the shockwave effect are authored according to the type of material the virtual cars are intended to be made of, amongst other virtual world properties. The Tenderer then renders this shockwave through the set of haptics devices in the endpoint, according to the shockwave vector and the physical location of the haptics devices relative to the user.

[0097] The signals sent to each specific actuator are preferably provided so that the sensory effect is congruent across all of the (potentially heterogenous) actuators available. For example, the Tenderer may not render very high frequencies to just one of the haptic actuators (e.g., the haptic arm band) due to capabilities lacking in other actuators.

Otherwise, as the shockwave moves through the player’s body, because the haptic vest and haptic gloves the user is wearing do not have the capability to render such high frequencies, there would a degradation of the haptic effect perceived by the user as the wave moves through the vest, into the arm band and finally into the gloves.

[0098] Some types of abstract haptic effects include:

• Shockwave effects, such as described above;

• Barrier effects, such as haptic effects which are used to represent spatial limitations of a virtual world, for example in a video game. If there are kinesthetic actuators on input devices (e.g., force feedback on a steering wheel or joystick), either active or resistive, then rendering of such an effect can be done through the resistive force applied to the users input. If no such actuators are available in the endpoint then in some examples vibro-tactile feedback may be rendered that is congruent with the collision of the in-game avatar with a barrier;

• Presence, for example to indicate the presence of a large object approaching the scene such as a train. This type of haptic effect may be rendered using a low timefrequency rumbling of some haptic devices’ actuators. This type of haptic effect may also be rendered through contact spatial feedback applied as pressure from aircuffs;

• User interface feedback, such as clicks from a virtual button. For example, this type of haptic effect may be rendered to the closest actuator on the body of the user that performed the click, for example haptic gloves that the user is wearing. Alternatively, or additionally, this type of haptic effect may also be rendered to a shaker coupled to the chair in which the user is sitting. This type of haptic effect may, for example, be defined using time-dependent amplitude signals. However, such signals may be altered (modulated, frequency-shifted, etc.) in order to best suit the haptic device(s) that will be providing the haptic effect; • Movement. These haptic effects are designed so that the user perceives some form of motion. These haptic effects may be rendered by an actuator that actually moves the user, e.g. a moving platform/seat. In some examples, an actuator may provide a secondary modality (via video, for example) to enhance the motion being rendered; and

• Triggered sequences. These haptic effects are characterized mainly by their timedependent amplitude signals. Such signals may be rendered to multiple actuators and may be augmented when doing so. Such augmentations may include splitting a signal in either time or frequency across multiple actuators. Some examples may involve augmenting the signal itself so that the sum of the haptic actuator outputs does not match the original signal.

Spatial and Non-Spatial Effects

[0099] Spatial effects are those which are constructed in a way that convey some spatial information of the multi-sensory scene being rendered. For example, if the playback environment is a room, a shockwave moving through the room would be rendered differently to each haptic device given its location within the room, according to the position and size of one or more haptic objects being rendered at a particular time.

[0100] Non-spatial effects may, in some examples, target particular locations on the user regardless of the user’s location or orientation. One example is a haptic device that provides a swelling vibration on the users back to indicate immediate danger. Another example is a haptic device that provides a sharp vibration to indicate an injury to a particular body area.

[0101] Some effects may be non-diegetic effects. Such effects are typically associated with user interface feedback, such as a haptic sensation to indicate the user completed a level or has clicked a button on a menu item. Non-diegetic effects may be either spatial or non- spatial.

Haptic Device Type

[0102] Receiving information regarding the different types of haptics devices available at the endpoint enables the Tenderer to determine what kinds of perceived effects and rendering strategies are available to it. For example, local haptics device data indicating that the user is wearing both haptic gloves and a vibro-tactile vest — or at least local haptics device data indicating that that haptic gloves and a vibro-tactile vest are present in the playback environment — allows the Tenderer to render a congruent recoil effect across the two devices when a user shoots a gun in a virtual world. The actual actuator control signals sent to the haptic devices may be different than in the situation where only a single device is available. For example, if the user is only wearing a vest, the actuator control signals used to actuate the vest may differ with regard to the timing of the onset, the maximum amplitude, frequency and decay time of the actuator control signals, or combinations thereof.

Location of the Devices

[0103] Knowledge of the location of the haptics devices across the endpoint enables the Tenderer to render spatial effects congruently. For example, knowledge of the location of the shaker motors in a lounge enables the Tenderer to produce actuator control signals to each of the shaker motors in the lounge in a way to convey spatial effects such as a shockwave propagating through the room. Additionally, knowledge of where wearable haptics devices, whilst implicit by their type, e.g. a glove is on the user’s hand, may also bd used by the Tenderer to convey spatial effects in addition to non-spatial effects.

Types of Actuation Provided by Haptic Devices

[0104] Haptic devices can provide a range of different actuations and thus perceived sensations. These are typically classed in two basic categories:

1. vibro-tactile , e.g. vibrations; or

2. Kinesthetic, e.g., resistive or active force feedback.

[0105] Either category of actuations may be static or dynamic, where dynamic effects are altered in real time according to some sensor input. Examples include a touch screen rendering a texture using a vibro-tactile actuator and a position sensor measuring the user’s finger position(s).

[0106] Moreover, the physical construction of such actuators varies widely and affects many other attributes of the device. An example of this is the onset delay or time-frequency response that varies significantly across the following haptic device types:

• Eccentric rotating mass;

• Linear resonant actuator;

• Piezoelectric actuator; and

• Linear magnetic ram. [0107] The Tenderer should be configured to account for the onset delay of a particular haptics device type when rendering signals to be actuated by the haptics devices in the endpoint.

The On- and Off-Set Delays of the Haptic Devices

[0108] The onset delay of the haptic device refers to the delay between the time that an actuator control signal is sent to the device and the device’s physical response. The off-set delay refers to the delay between the time that an actuator control signal is sent to zero the output of the device and the time the device stops actuating.

The Time-Frequency Response

[0109] The time-frequency response refers to the frequency range of the signal amplitude as a function of time that the haptic device can actuate at steady state.

The Spatial-Frequency Response

[0110] The spatial-frequency response refers to the frequency range of the signal amplitude as a function of the spacing of actuators of a haptic device. Devices with closely-spaced actuators have higher spatial-frequency responses.

Dynamic Range

[0111] Dynamic range refers to the differences between the minimum and maximum amplitude of the physical actuation.

Characteristics of Sensors in Closed-Loop Haptics Devices

[0112] Some dynamic effects use sensors to update the actuation signal as a function of some observed state. The sampling frequencies, both temporal and spatial along with the noise characteristics will limit the capability of the control loop updating the actuator providing the dynamic effect.

AIRFLOW

[0113] Another modality that some multi-sensory immersive experiences (MSIE) may use is airflow. The airflow may, for example, be rendered congruently with one or more other modalities such as audio, video, light-effects and/or haptics. Rather than only specialized (e.g. channel-based) setups for 4D experiences in cinemas which may include “wind effects,” some airflow effects may be provided at other endpoints that may typically include airflow, such as a car or a living room. Rather than a channel-based system, the airflow sensory effects may be represented as an airflow object that may include properties such as:

• Spatial location;

• Direction of the intended airflow effect;

• Intensity/airflow speed; and/or

• Air temperature.

[0114] Some examples of air flow objects may be used to represent the movement of a bird flying past. To render to the airflow actuators at the endpoint, the MS Tenderer 001 may be provided with information regarding:

• The type of airflow devices e.g. fan, air conditioning, heating;

• The position of each airflow device relative to the user’s location, or relative to an expected of the user;

• The capabilities of the airflow device, e.g., the airflow device’s ability to control direction, airflow and temperature;

• The level of control of each actuator, e.g., airflow speed, temperature range; and

• The response time of each actuator, e.g., how long does it take to reach a chosen speed.

Some Examples of Airflow Use in Different Endpoints

[0115] In a vehicle such as a car, the object-based metadata can be used to create experiences such as:

• Mimicking “chills down your spine” during a horror movie or gaming piece of content with airflow down the chair;

• Simulating the movement of a bird flying past; and/or

• Create a gentle breeze in a seascape.

[0116] In the small enclosed space of a typical vehicle, temperature changes may be possible to achieve over relatively shorter periods of time — as compared to temperature changes in a larger environment, such as a living room environment. In one example, the MS Tenderer 001 may cause an increasing air temperature as a player enters a “lava level” or other hot area during a game. Some examples may include other elements, such as confetti in the air vents to celebrate an event, such as the celebration of a goal made by the user’s your favorite football team. [0117] In a living space or other room, airflow may be synchronized to the breathing rhythm of a guided meditation in one example. In another example, airflow may be synchronized to the intensity of a workout, with increased airflow or decreased temperature as intensity increases. In some examples, there may be relatively less control over spatial aspects during rendering. For example, many existing airflow actuators are optimized for heating and/or air conditioning rather than for providing spatially diverse sensory actuation.

Combinations of Lights, Airflow and Haptics

Car Examples

[0118] The following examples are described with reference to a car, but are also applicable to other vehicles, such as trucks, vans, etc. In some examples, there may be a user interface on the steering wheel or on a touchscreen near or in the dashboard. According to some examples, the following actuators may be present in the car:

1. Individually addressable lights, spatially distributed around the car as follows: o on the dashboard; o under the footwells; o on the doors; and o in the center console.

2. Individually controllable air conditioning/heating outlets distributed around the car as follows: o In the front dashboard; o Under the footwells; o In the center console facing the rear seats; o On the side pillars; o In the seats; and o Directed to the windscreens (for defogging).

3. Individually controllable seats with vibro-tractile haptics; and

4. Individually controllable floor mats with vibro-tactile haptics.

[0119] In this example, the modalities supported by these actuators include the following:

• Lights across the individually addressable LEDs in the car, plus the indicator lights on the dash and steering wheel;

• Air flow via the controllable air conditioning vents;

• Haptics, including: o Steering wheel: tactile vibration feedback; o Dash touchscreen: tactile vibration feedback and texture rendering; and o Seats: tactile vibrations and movement.

[0120] In one example, a live music stream is being rendered to four users sitting in the front seats. In this example, the MS Tenderer 001 attempts to optimize the experience for multiple viewing positions. During the build-up before the artist has taken the stage and the previous acts have finished, the content contains:

• Interlude music;

• Low intensity lighting; and

• Haptic content representing the moshing of the crowd.

[0121] In addition to the rendered audio and video stream, the light content contains ambient light objects that are moving slowly around the scene. These may be rendered using one of the ambient layer methods disclosed herein, for example such that there is no spatial priority given to any user’s perspective. In some examples, the haptic content may be spatially concentrated in the lower time-frequency spectrum and may be rendered only by the vibro- tactile motors in the floor mats.

[0122] According to this example, pyrotechnic events during the music stream correspond to multi-sensory-sensory content including:

• Light objects that spatially correspond to the location of the pyrotechnics at the event; and

• Haptic objects to reinforce the dynamism of the pyrotechnics via a shockwave effect.

[0123] In this example, the MS Tenderer 001 renders both the light objects and the haptic objects spatially. Light objects may, for example, be rendered in the car such that each person in the car perceives the light objects to come from the left if the pyrotechnics content is located at the left of the scene. In this example, only lights on the left of the car are actuated. Haptics may be rendered across both the seats and floor mats in a way that conveys directionality to each user individually.

[0124] At the end of the concert the pyrotechnics are present in the audio content and both pyrotechnics and confetti are present in the video content. In addition to rendering light objects and haptic objects corresponding to the pyrotechnics as above, the effect of the confetti firing may be rendered using the airflow modality. For example, the individually controllable air flow vents of the HVAC system may be pulsed.

Living Room Examples

[0125] In this implementation, in addition to an audio/visual (AV) system that includes multiple loudspeakers and a television, the following actuators and related controls are available in the living room:

• A haptics vest that the user — also referred to as a player — is wearing;

• Haptics shakers mounted to the seat in which the player is sitting;

• A (haptics) controllable smart watch;

• Smart lights spatially distributed around the room;

• A wireless controller; and

• An addressable air-flow bar (AFB), which includes an array of individually controllable fans directed to the user (similar to HVAC vents in the front dashboard of a car).

[0126] In this example, the user is playing a first person shooter game and the game contains a scene in which a destructive hurricane moves through the level. As it does so, in-game objects are thrown around and some hit the player. Haptics objects rendered by the MS renderer 001 cause a shockwave effect to be provided through all of the haptics devices that the user can perceive. The actuator control signals sent to each device may be optimized according to the intensity of the impact of the in-game objects, the directi on(s) of the impact and the capabilities and location of each actuator (as described earlier).

[0127] At a time before the user is struck by an in-game object, the multi-sensory content contains a haptic object corresponding to a non-spatial rumble, one or more airflow objects corresponding to directional airflow; and one or more light objects corresponding to lightning. The MS renderer 001 renders the non-spatial rumble to the haptics devices. The actuator control signals sent to each haptics device may be rendered such that the ensemble of actuator control signals across the haptics array is congruent in perceived onset time, intensity and frequency. In some examples, the frequency content of the actuator control signals sent to the smart watch may be low-pass filtered, so that they are congruent with the frequency-limited capability of the vest, which is proximate to the watch. The MS renderer 001 may render the one or more airflow objects to actuator control signals for the AFB such that the air flow in the room is congruent with the location and look direction of the player in the game, as well as the hurricane direction itself. Lightning may be rendered across all modalities as (1) a white flash across lights that are located in suitable locations, e.g., in or on the ceiling; and (2) an impulsive rumble in the user’s wearable haptics and seat shaker.

[0128] When the user is struck by an in-game object, a directional shockwave may be rendered to the haptics devices. In some examples, a corresponding airflow impulse may be rendered. According to some examples, a damage take effect, indicating the amount of damage caused to the player by being struck by the in-game object, may be rendered by the lights.

[0129] In some such examples, signals may be rendered spatially to the haptics devices such that a perceived shockwave moves across the player’s body and the room. The MS Tenderer 001 may provide such effects according to actuator location information indicating the haptics devices locations relative to one another. The MS Tenderer 001 may provide the shockwave vector and position according to the actuator location information in addition to actuator capability information. According to some examples, a non-directional air flow impulse may be rendered, e.g., all the air vents of the AFB may be turned up briefly to reinforce the haptic modality. In some examples, at the same time, a red vignette may be rendered to the light strip surrounding the TV, indicating to the player that the player took damage in the game.

[0130] Figure 4B is a flow diagram that outlines one example of a method that may be performed by an apparatus or system such as those disclosed herein. The blocks of method 400, like other methods described herein, are not necessarily performed in the order indicated. In some implementation, one or more of the blocks of method 400 may be performed concurrently. Moreover, some implementations of method 400 may include more or fewer blocks than shown and/or described. The blocks of method 400 may be performed by one or more devices, which may be (or may include) one or more instances of control system such as the control system 110 that is shown in Figure 1 A and described above.

[0131] In this example, block 405 involves estimating — for example, by the control system 110 — an actuator-room response (ARR) that summarizes responses of an environment to activations of a set of controllable actuators that are in the environment. The playback environment may, for example, be a room of a house, a vehicle, etc. In some examples, the set of controllable actuators may include one or more light fixtures, one or more haptic devices, one or more air flow control devices, or combinations thereof. In some examples, the control system 110 may be configured for obtaining controllable actuator location information for a set of controllable actuators in the environment. According to some examples, the control system 110 may be configured for obtaining controllable actuator capability information for each controllable actuator of the set of controllable actuators. The controllable actuator capability information may, for example, indicate the controllable actuator type, required control signal format, if any, the drive level required to obtain a particular actuator response, etc.

[0132] In some examples, the control system 110 may be configured for obtaining environment information corresponding to the environment, in other words corresponding to the environment in which the controllable actuators are located. The environment information may include locations, orientations and/or dimensions of one or more environment features. The one or more environment features may include one or more structural elements of the environment, such as one or more walls, a ceiling, a floor, or a combination thereof. In some examples, the environment information may include environment feature color information, environment feature reflectivity information, or a combination thereof. The one or more environment features may include one or more furniture items. In some such examples, the environment information may include furniture position information.

[0133] Various methods of obtaining the controllable actuator location information, the controllable actuator capability information and the environment information are disclosed herein. Such methods include automated methods, manual/user input-based methods, and “hybrid” methods that are partly automated and partly manual.

[0134] The process of estimating ARR, as well as the manner of representing the ARR, may vary according to the particular implementation. In some examples, ARR may be expressed as R(j, x, y, z, a, t), where: x represents the x coordinate of the environment, which is a room in this example; y represents the y coordinate of the room; z represents the z coordinate of the room; a represents the actuation command which may be a vector, such as color; t represents time; and

J represents the actuator index. [0135] For a steady-state response — in other words, ignoring t — of all of the actuators being actuated — in other words, a is the same for all j — for actuators that overlap spatially, the ARR is not suited for inversion due to the low rank. According to some examples, R(j,x,y,z) may be reformulated as a matrix R(j, X), in which X represents the spatial coordinates (x,y,z) converted into flat indices, as x,y,z are orthogonal.

[0136] R is a projection of the actuators’ effect on the environment, which is a room in this example. Suppose that the vector A represents all of the actuators in the room. If one could invert R, one would obtain the mapping R ¹ that projected a room effector onto the vector A. In other words, one could obtain the actuator control signals from some desired room effect, which may be represented as object-based sensory data. The capability of obtaining the actuator control signals from object-based sensory data indicating a desired room effect is desirable for object-based MS rendering.

[0137] However, a “raw” or unmodified version of R is generally ill-posed (mathematically unsuitable) for inversion. On reason for this is that in many instances, there may be “dead zones” in which no actuator affects a particular X coordinate. Therefore, a corresponding number of columns or rows may have zero values. In some instances there may be correlation between columns or rows.

[0138] According to this example, block 410 involves modifying the ARR to produce an actuator map (AM). According to some examples, the matrix representation of the AM is mathematically suitable for inversion. Modifying the ARR may involve regularization, filling gaps in the ARR — for example, eliminating zero values of the ARR matrix — by increasing one or more volumes of the environment affected by an actuator response, reducing one or more overlapping volumes of the environment affected by multiple actuator responses, or combinations thereof.

[0139] The AM may allow rendering of object-based sensory data to actuator commands. According to some implementations, the AM may serve the following purposes:

Enabling the object-based sensory data 005 to be created in an end-point- agnostic manner, within reason. In other words, content creators do not need to direct MS content to particular channels that correspond to individual actuators. Instead, the content creators may create MS objects and place them spatially within a generalized representation of a playback environment; Supporting the MS Tenderer 001 in projecting spatial MS objects onto the array of actuators in the environment.

[0140] A sensory object, O, may be represented spatially at a coordinate location such as (x, y, z) or equivalently a flattened coordinate representation X. Properties of the sensory object may include sensory object location and sensory object size, which result in the sensory object being projected across a range of X of the playback environm ent/ endpoint that corresponds to the sensory object location and size. Actuators that are located within a volume of the endpoint corresponding to the sensory object size, at any given time, are candidates for actuation at that time. Accordingly, the object-based representation of multi- sensory effects can readily be represented as effects on the endpoint room. The foregoing considerations motivate not only the estimation of the ARR, but also motivate the conditioning/augmentation of ARR into an AM that is better suited for inversion. The AM can be used to map the n^th sensory object onto a vector A representing all of the actuators, e.g., as follows:

OnX -► A.

[0141] In some examples, the modification of the ARR into the AM in block 410 makes the invertibility more practical by performing one or both of the following operations:

Filling in the gaps by artificially spreading the response of actuators to complete support across X; or

Reducing the overlap, where practical, between actuators so that the response of a point-object (one with infinitesimal size) is an impulse in A.

[0142] Even with modification, the actuator map, AM, may still be rank deficient (ignoring the zero activations). In other words, the volumes corresponding to multiple sensory objects may encompass one or more of the same actuators of an environment at a particular time. The MS Tenderer 001 may be configured to use other data, such as sensory object priority metadata, in addition to the remainder of the environment and actuator data 004 in order to determine the actuator control signals 310.

Multiple Modalities

[0143] In some examples, when two or more modalities of multi-sensory effects — such as haptics and light — are scheduled to be rendered at the same time, the ARR and/or the AM corresponding to each modality may be evaluated and/or modified. Models linking the effects of the two sensory modes together should be employed in order to ensure that they are suitable for joint rendering of multi-modal-sensory effects. The ARR and/or the AM of multiple modalities may not overlap and have mutual support (regions where both modalities can actuate) across the entire endpoint because different devices are responsible for actuating different modalities. This overlap of mutual support, or lack thereof, may be in any one of a number of domains including both the frequency and spatial domain. Moreover, the transient response or dynamic response of the actuators and the according physics for different modalities should be accounted for. For example, there may be a time shift between the ARR for lighting effects and the ARR for air flow effects, due to the time lag between sending an air flow actuation signal and achieving a desired air flow.

[0144] Figure 5 shows example elements of another system for the creation and playback of MS experiences. As with other figures provided herein, the types and numbers of elements shown in Figure 5 are merely provided by way of example. Other implementations may include more, fewer and/or different types and numbers of elements. According to some examples, system 500 may be, or may include, one or more devices configured for performing at least some of the methods disclosed herein. In some examples, system 500 may include one or more instances of the control system 110 of Figure 1 A that are configured for performing at least some of the methods disclosed herein.

[0145] According to this example, the system shown in Figure 5 is an instance of the system shown in Figure 3. In this example, the system shown in Figure 5 is a “lightscape” embodiment in which video, audio and light effects are combined to create the MS experience.

[0146] In this example, system 500 includes a lightscape creation tool 100, which is an instance of the content creation tool 000 that is described with reference to Figure 3. The lightscape creation tool 100 is configured for designing and outputting object-based light data 505’, either separately or in conjunction with corresponding audio data 111’ and/or video data 112’, depending on the particular implementation. The object-based light data 505’ may include time stamp information, as well as information indicating light object properties, etc. In some instances, the time stamp information may be used to synchronize effects relating to the object-based light data 505’ with the audio data 111’ and/or the video data 112’, which also may include time stamp information. [0147] In this example, the object-based light data 505’ includes light objects and corresponding light metadata. For example, the object-based light data may include light object position metadata, light object color metadata, light object size metadata, light object intensity metadata, light object shape metadata, light object diffusion metadata, light object gradient metadata, light object priority metadata, light object layer metadata, or combinations thereof. Although the content creation tool 100 is shown providing a stream of object-based light data 505’ to the experience player 102 in this example, in alternative examples the content creation tool 100 may produce object-based light data 505’ that is stored for subsequent use. Examples of graphical user interfaces for a light-object-based content creation tool are described below.

[0148] In this example, system 500 includes an experience player 102 that is configured to receive object-based light data 505’, audio data 111’ and video data 112’, and to provide the object-based light data 505 to the lightscape Tenderer 501, to provide the audio data 111 to the audio Tenderer 106 and to provide the video data 112 to the video Tenderer 107. As noted elsewhere herein, the object-based light data 505, the audio data 111 and the video data 112 may include time stamp information that may be used to synchronize MS effects with audio and/or video effects. According to some examples, the experience player 102 may be a media player, a game engine or personal computer or mobile device, or a component integrated in an television, DVD player, sound bar, set top box, or a service provider media device such as a Chromecast, Apple TV device, or Amazon Fire TV. In some examples, the experience player 002 may be configured to receive encoded objectbased light data 505’ along with encoded audio data 111’ and/or encoded video data 112’, e.g., as part of the same bitstream with the encoded audio data 111’ and/or the encoded video data 112’. According to some examples, the experience player 102 may be configured to extract the object-based light data 505 from the content bitstream and to provide decoded object-based light data 505 to the lightscape Tenderer 501, to provide decoded audio data 111 to the audio Tenderer 106 and to provide decoded video data 112 to the video Tenderer 107. In some examples, the experience player 002 may be configured to allow control of configurable parameters in the lightscape Tenderer 501, such as immersion intensity. Some examples are described below.

[0149] As noted above, according to some implementations the system 500 may include one or more instances of the control system 110 of Figure 1 A configured for performing at least some of the methods disclosed herein. In some such examples, one instance of the control system 110 may implement the lightscape creation tool 100 and another instance of the control system 110 may implement the experience player 002. In some examples, one instance of the control system 110 may implement the audio Tenderer 006, the video Tenderer 007, the lightscape Tenderer 501, or combinations thereof. According to some examples, an instance of the control system 110 that is configured to implement the experience player 002 may also be configured to implement the audio Tenderer 006, the video Tenderer 007, the lightscape Tenderer 501, or combinations thereof.

[0150] In some examples, room descriptors of the environment and light fixture data 104 may describe the size and orientation of the playback environment itself, to establish a relative or absolute coordinate system to which all objects are positioned. Room descriptor information may indicate or describe the physical dimensions of the playback environment, for example in physical units of distance such as meters. In some such examples, sensory object locations, sensory object sizes, and sensory object orientations may be described in units that are relative to the room size, for example in a range from -1 to 1. Room descriptors may also describe a preferred viewing position. For example, in a living room a display screen may be regarded as the front, in some instances the front and center, and the floor and ceiling may be regarded as the vertical bounds. In some such examples, the room descriptors also may also indicate bounds corresponding with the left, right, front, and rear, walls relative to the front position. According to some examples, at least some room descriptor information may be provided as a matrix. In some such examples the matrix may be a 3x3 matrix, with one row or column corresponding to one dimension of a three- dimensional space.

[0151] According to this example, system 500 includes a lightscape Tenderer 501 that is configured to render object-based light data 505 to light fixture control signals 515, based at least in part on environment and actuator data 104. In this example, the lightscape Tenderer 501 is configured to output the light fixture control signals 515 to light controllers 103, which are configured to control the light fixtures 108. The light fixtures 108 may include individual controllable light sources, groups of controllable light sources (such as controllable light strips), or combinations thereof. In some examples, the lightscape Tenderer 501 may be configured to manage various types of light object metadata layers, examples of which are provided herein. According to some examples, the lightscape Tenderer 501 may be configured to render actuator signals for light fixtures based, at least in part, on the perspective of a viewer. If the viewer is in a living room, that includes a television (TV) screen, the lightscape Tenderer 501 may, in some examples, be configured to render the actuator signals relative to the TV screen. However, in virtual reality (VR) use cases, the lightscape Tenderer 501 may be configured to render the actuator signals relative to the position and orientation of the user’s head. In some examples, the lightscape Tenderer 501 may receive input from the playback environment — such as light sensor data corresponding to ambient light, camera data corresponding to a person’s location or orientation, etc. — to augment the render.

[0152] In some examples, the lightscape Tenderer 501 is configured to receive object-based light data 505 that includes light objects and object-based lighting metadata indicating an intended lighting environment, as well as environment and light fixture data 104 corresponding to light fixtures 108 and other features of a local playback environment, which may include, but are not limited to, reflective surfaces, windows, non-controllable light sources, light-occluding features, etc. In this example, the local playback environment includes one or more loudspeakers 109 and one or more display devices 510.

[0153] According to some examples, the lightscape Tenderer 501 is configured to calculate how to excite various controllable light fixtures 108 based at least in part on the object-based light data 505 and the environment and light fixture data 104. The environment and light fixture data 104 may, for example, indicate the geometric locations of the light fixtures 108 in the environment, light fixture type information, etc. In some examples, the lightscape Tenderer 501 may configured to determine which light fixtures will be actuated based, at least in part, on the position metadata and size metadata associated with each light object, e.g., by determining which light fixtures are within a volume of a playback environment corresponding to the light object’s position and size at a particular time indicated by light object time stamp information. In this example, the lightscape Tenderer 501 is configured to send light fixture control signals 515 to the light controller 103 based on the environment and light fixture data 104 and the object-based light data 505. The light fixture control signals 515 may be sent via one or more of various transmission mechanisms, application program interfaces (APIs) and protocols. The protocols may, for example, include Hue API, LIFX API, DMX, Wi-Fi, Zigbee, Matter, Thread, Bluetooth Mesh, or other protocols.

[0154] In some examples, the lightscape Tenderer 501 may be configured to determine a drive level for each of the one or more controllable light sources that approximates a lighting environment intended by the author(s) of the object-based light data 505. According to some examples, the lightscape Tenderer 501 may be configured to output the drive level to at least one of the controllable light sources.

[0155] According to some examples, the lightscape renderer 501 may be configured to collapse one or more parts of the lighting fixture map according to the content metadata, user input (choosing a mode), limitations and/or configuration of the light fixtures, other factors, or combinations thereof. For example, the lightscape Tenderer 501 may be configured to render the same control signals to two or more different lights of a playback environment. In some such examples, two or more lights may be located close to one another. For example, two or more lights may be different lights of the same actuator, e.g., may be different bulbs within the same lamp. Rather than compute a very slightly different control signal for each light bulb, the lightscape Tenderer 501 may be configured to reduce the computational overhead, increase rendering speed, etc., by render the same control signals to two or more different, but closely-spaced, lights.

[0156] In some examples, the lightscape Tenderer 501 may be configured to spatially upmix the object-based light data 505. For example, if the object-based light data 505 was produced for a single plane, such as a horizontal plane, in some instances the lightscape Tenderer 501 may be configured to project light objects of the object-based light data 505 onto an upper hemispherical surface (e.g., above an actual or expected position of the user’s head) in order to enhance the experience.

[0157] According to some examples, the lightscape Tenderer 501 may be configured to apply one or more thresholds, such as one or more spatial thresholds, one or more luminosity thresholds, etc., when rendering actuator control signals to light actuators of a playback environment. Such thresholds may, in some instances, prevent some light objects from causing the activation of some light fixtures.

[0158] Light objects may be used for various purposes, such as to set the ambience of the room, to give spatial information about characters or objects, to enhance special effects, to create a greater sense of interaction and immersion, to shift viewer attention, to punctuate the content, etc. Some such purposes may be expressed, at least in part, by a content creator according to sensory object metadata types and/or properties that are generally applicable to various types of sensory objects — such as object metadata indicating a sensory object’s location and size. [0159] For example, the priority of sensory objects, including but not limited to light objects — may be indicated by sensory object priority metadata. In some such examples, sensory object priority metadata is taken into account when multiple sensor objects map to the same fixture(s) in a playback environment at the same time. Such priority may be indicated by light priority metadata. In some examples, priority may not need to be indicated via metadata. For example, the MS Tenderer 001 may give priority to sensory objects — including but not limited to light objects — that are moving over sensory objects that are stationary.

[0160] A light object may, depending on its location and size and the locations of light fixtures within a playback environment — potentially cause the excitation of multiple lights. In some examples, when the size of a light object encompasses multiple lights, the Tenderer may apply one or more thresholds — such as one or more spatial thresholds or one or more luminosity thresholds — to gate objects from activating some encompassed lights.

Examples of Using a Lighting Map

[0161] In some implementations a lighting map, which is an instance of the of the actuator map (AM) that includes a description of lighting in a playback environment, may be provided to the lightscape Tenderer 501. In some such examples, the environment and light fixture data shown in Figure 5 may include the lighting map. According to some examples, the lighting map may be allocentric, e.g., indicating absolute spatial coordinate-based light fall-off, whereas in other examples the lighting map may be egocentric, e.g., a light projection mapped onto a sphere at an intended viewing position and orientation. In the case of a sphere, the lighting map may, in some examples, be projected onto a two-dimensional (2D) surface, e.g., in order to utilize 2D image textures in processing. In any case, the lighting map should indicate the capabilities and the lighting setup of the playback environment, such as a room. In some embodiments the lighting map may not directly relate to physical room characteristics, for example if certain user preference-based adjustments have been made.

[0162] In some examples, there may be one lighting map per light fixture, or per light, in a playback environment. According to some examples, the intensity of light indicated by the light map may be inversely correlated to the distance to the center of the light, or may be approximately (e.g., within plus or minus 5%, within plus or minus 10%, within plus or minus 15%, within plus or minus 20%, etc.) inversely correlated to the distance to the center of the light. The intensity values of the light map may indicate the strength or impact of the light object onto the light fixture. For example, as a light object approaches a lightbulb, the lightscape Tenderer 501 may be configured to determine that the lightbulb intensity will increase as the distance between the light object and the lightbulb decreases. The lightscape Tenderer 501 may be configured to determine the rate of this transition based, at least in part, on the intensity of light indicated by the light map.

[0163] Inside this common rendering space, in some examples the lightscape Tenderer 501 may be configured to use a dot product multiplication between a light object and the light map for each light to compute a light activation metric, e.g., as follows:

Y = LM ■ Objmin LM, Obj

[0164] In the foregoing equation, Y represents the light activation metric, LM represents the lighting map and Obj represents the map of a light object. The light activation metric indicates the relative light intensity for the actuator control signal output by the lightscape Tenderer 501 based on the overlap between the light object and the spread of light from the light fixture. In some examples, the lightscape Tenderer 501 may use the maximum or closest distance, or other geometric metrics, from the light object to the light fixture as part of the determination of light intensity. In some implementations, instead of computing the light activation metric, the lightscape Tenderer 501 may refer to a look-up-table to determine the light activation metric.

[0165] The lightscape Tenderer 501 may repeat one of the foregoing procedures for determining the light activation metric for all light objects and all controllable lights of the playback environment. Thresholding for light objects that produce a very low impact on light fixtures may be helpful to reduce complexity. For example, if the effect of a light object would cause an activation of less than a threshold percent of light fixture activation — such as less than 10%, less than 5%, etc. — the lightscape Tenderer 501 may disregard the effect of that light object.

[0166] The lightscape Tenderer 501 may then use the resultant light activation matrix Y, along with various other properties such as the chosen panning law (either indicated by light object metadata or Tenderer configuration) or the priority of the light object, to determine which objects get rendered by which lights and how. Rendering lights-objects into light fixture control signals may involve: • Altering the luminance of a light-object as a function of the distance it is from the light fixture;

• Mixing the colors of multiple light-objects that are simultaneously (multiplexed) rendered by a single light fixture; or

• Altering either of the above based on the light object priority.

Mapping Luminaire Actuators

[0167] This section describes devices, systems and methods for mapping and characterizing light fixtures and the environment in which the light fixtures are deployed. This process will generally involve obtaining light fixture information, such as light fixture location and capability information, as well as obtaining environment information corresponding to the environment in which the light fixtures are deployed. The light fixture information also may include light direction information indicating the direction(s) in which individual lights emit light in the environment. The environment information may include the locations, orientations and dimensions of one or more environment features, such as structural elements of the environment, furniture items, etc. The environment information may include environment feature color information, environment feature reflectivity information, or both.

[0168] In some examples, a user may manually enter the light fixture information and the environment information. According to some examples, a device or system may estimate the light fixture information and the environment information, for example by using camera data — such as a video feed — and inertial measurement data from a sensing device to obtain data corresponding to the effects caused by illuminating the environment — for example, the room — with light when the light fixtures are actuated. Detailed examples are provided below.

[0169] Some examples involve obtaining a light-room-response (LRR) based on the light fixture information and the environment information and determining a light map (LM) from the LRR. The LRR and LM are instances of the ARR and AM, respectively. The LRR describes the joint response of the lights and the environment in which the lights are deployed.

[0170] Figure 6 depicts the light-room-response (LRR) of a single light fixture in egocentric polar coordinates. The image on which Figure 6 is based was produced using a first-order reflection simulation and manually input data. The x,y,z coordinates in Figure 6 are defined as follows:

Origin at the user’ s head;

Positive x in the direction to the user’ s right;

Positive y in the direction behind the user; and Positive z in the direction above the user.

[0171] We can see from Figure 6 that the light is below and in front of the user’s head, and off to the left of the user’s head. This light was on the floor pointing up along the left-hand wall of a room, from the user’s perspective. Figure 6 shows the light’s illumination field on the room.

Video Based Analysis and Light Map Generation

[0172] The LRR indicates the response of each light in an environment, e.g., in a room. The LRR may, in some examples, be estimated by manually entering the location of each light, light characteristics, room dimensions, etc. However, some disclosed devices, methods and systems provide an automated and more accurate light mapping process, involving direct observations of the actual lights in the room using a video-based analysis. In addition to more detailed characterization, such methods also allows for compensation of slight differences between the lights as actually made and the specifications provided on a data sheet.

[0173] Figure 7 shows example elements of a light mapping system. As with other figures provided herein, the types and numbers of elements shown in Figure 7 are merely provided by way of example. Other implementations may include more, fewer and/or different types and numbers of elements. According to some examples, system 700 may be, or may include, one or more devices configured for performing at least some of the methods disclosed herein. In some examples, system 700 may include one or more instances of the control system 110 of Figure 1 A that are configured for performing at least some of the methods disclosed herein.

[0174] According to this example, the light mapping system 700 includes the following elements:

Modules 701 - A system controller, which includes a state machine in this example;

702 - Calibration signal generator: a module configured to produce calibration signals for controlling the lights 750 A, 75 OB and 750C to emit light in order to support the localization, mapping and characterization process;

703 - A light controller module configured to control the lights 750A, 750B and 750C according to signals 721 from the calibration signal generator 702;

704 - A sensing device that provides observations as input to the light mapping system 700. In some examples, the sensing device may be a cell phone;

705 - A light localization and segmentation module, which is configured to determine the location of the lights 750A, 750B and 750C in both the video feed 723 and the room 730;

706 - A room characterization module, which is configured to estimate parameters of the room 730 such as wall color(s), ceiling color(s), floor color(s), furniture color(s), surface reflectance, etc.;

707 - A light characterization module, which is configured to estimate parameters of the lights 750A, 750B and 750C, including luminance sensitivity and color matching;

708 - An analysis and fusion module, which is configured to fuse the estimated room parameters 727 and the estimated light parameters 728;

709 - A localizing module that is configured to determine the position and orientation of the sensing device 704 within the room 730. In this example, the localizing module 709 is configured to perform a simultaneous location and mapping (SLAM) process based at least in part on camera data — the video feed 723 in this example — and inertial measurement data from the sensing device 704;

710 - A camera system including one or more camera devices onboard the sensing device 704. The one or more camera devices may include one or more visible-light cameras, one or more infrared light cameras, one or more stereo cameras, one or more lidar devices, etc.;

712 - Auxiliary sensors which are part of the sensing device 704 and are physically coupled to the camera system 710 in this example. The auxiliary sensors 712 are configured to obtain inertial measurement data in this example, which may include gyroscope data, accelerometer data, or both;

730 - An environment, which is a room in this example, in which numerous luminaires can be controlled to form a flexible light rendering system; and 750A, 750B and 750C - a plurality of controllable lights in the room 730. Although there are three controllable lights in the room 730 according to this example, other examples may include different numbers of controllable lights, e.g., 5 controllable lights, 8 controllable lights, 10 controllable lights, 15 controllable lights, 20 or more controllable lights, etc.

Signals

720 - Control signals sent by the light controller 703 to the lights 750A, 750B and 750C;

721 - Calibration signals sent by the calibration signal generator to the light controller 703 for controlling the lights 750A, 750B and 750C;

723 - Video feed(s) produced by the cameras 710 of the sensing device 704;

724 - Sensor feed(s) produced by the sensors 712 of the sensing device 704;

725 - Data indicating the estimated position and orientation of the sensing device 704 within the room 730;

726 - Estimated light locations and their segmentation masks in the video feed;

727 - Estimated room parameters of the room 730;

728 - Estimated light parameters of the lights 750A, 750B and 750C;

729 - Joint estimated room and light parameters, which may include results of the analysis and fusion module 708’ s estimation of their coherent convergence on solutions; and

730 - Commands from the system controller 701 to control the generation of the calibration signals 721.

[0175] The processes of room and light characterization are iterative in this example. This iterative process is indicated in Figure 7 by the joint estimated room and light parameters 729 being sent back to the room characterization module 706 and the light characterization module 707, as well as to the system controller 701. The light sensed by a camera is the product of the object(s) an illuminant reflects off. For example, consider the pixels in the camera feed that correspond to a wall that is illuminated by one of the light-fixtures during a stage of the calibration measurement process. In this case, there is an infinite set of possible illumination and reflection spectra of the light-fixture and wall, respectively, that can produce the color sensed by the camera. For the purposes of perceptual consumption of content calibrated by this system, the control system does not need to estimate the spectral response, as the typical human is trichromatic and only 3 numbers (less for those who are color deficient) are needed to represent and reproduce a color. There are still ambiguities in the color sensed by the camera that result during the calibration process as we characterize each primary of the light-fixture individually. For example, the room characterization module 706 may, in the absence of any prior wall color information, make a first estimation that the color of a wall as being blue based on blue light reflecting from the wall that was illuminated by a light fixture. At the same time, the light characterization module 707 may make a first estimation that the radiance of the light is low, or equally the room characterization module may estimate that the wall has low albedo. However, the room characterization module 706 and light characterization module 707 may update the wall color estimate and light fixture radiance based on a different color of light that is subsequently reflected from the wall illuminated by a particular light fixture. Over time, the control system can develop joint estimates of room characteristics and light characteristics using observations made of multiple parts of the room and multiple light fixtures.

Light Calibration Signals/Coded Emissions

[0176] According to some examples, the calibration signals 121 generated by the calibration signal generator 102 may be, or may include, a set of color signals that enable the following functions:

1. Illumination of the room to provide data for the localizing module 709 to determine the position and orientation of the sensing device 704 within the room 730, for example according to a SLAM process;

2. Identification of individual lights;

3. Localization of individual lights; and

4. Estimation of individual lights’ illumination of the room.

[0177] These functions may have differing calibration signal requirements. In some examples, these functions may occur at different stages of the mapping process, which in some examples may correspond to the order of the numbers in the list above.

[0178] According to some examples, the calibration signals 121 may be configured to cause relatively low-frequency light modulation, for example configured to change the color, intensity, etc., of individual lights every 100 milliseconds (ms), every 200 ms, every 300 ms, every 400 ms, every 500 ms, etc. In such examples, the calibration signals 121 may cause the lights to be modulated at a rate that is compatible with the sampling rate of a video camera performing the calibration.

[0179] In some alternative examples, the lights may be controlled according to simple on-off keying, for example, switching each light in the room on and off sequentially, so that each controllable light in the room can be detected and localized. After this, in some examples the calibration signals 121 may be altered to aid the estimation of the light response in the room.

[0180] According to some examples, the calibration signals 121 may be configured to cause the lights to be modulated in a way that allows each of the lights to be uniquely identified. For example, each the calibration signals 121 may be configured to cause on-off keying according to a unique binary sequence. After detection of each light in the video feed 723, the task of identifying each light becomes one of identifying the binary sequence, for which there exist traditional estimators. The choice of the set of the unique codes used as the calibration signal modulator (amplitude in the case of on-off keying) may vary according to the particular implementation. Some families of codes, such as Gold codes, provide optimal separation. Gold codes are of length 2 ^I, where N is the gold code order and is above 4 and preferably odd. For N = 5, one obtains 12dB of separation between any two calibration signals derived from that code family. These sequences are of length 31 which, for a video camera performing calibration at 30 frames per second (FPS), would be a compatible length for mapping applications.

Wall Color

[0181] The wall color plays an important role in the output color of a light source. A blue light projected onto a red wall, for example, will produce a purple hue.

Bi-Directional Reflectance Measurements (BRDF)

[0182] In this context, “bi-directional” refers to reflectance measurements over a range of degrees (e.g., over 180 degrees) instead of only in a single direction. Such measurements can be useful because some surfaces within the playback environment may reflect differently from one side or direction as compared to another side or direction. The reflectance of structural elements of the environment, such as walls, floors, ceilings, etc., will determine the amount and directionality or “pointedness” of a light source. Especially if a light source is a spotlight, the reflectance or diffuseness of the wall will affect the dispersion of the light and the shape of the impulse response.

Spectral Response

[0183] The combination of direct light and reflections from structural elements, such as walls, can be measured using a typical RGB camera sensor. However, it is beneficial to capture the spectral power distribution of the lights and spectral reflectance of the structural elements. The spectral power distribution (SPD) of the light source describes the amount of power over each wavelength of the visible spectrum. The spectral reflectance (SR) of the structural elements describes how much of each wavelength will be reflected back.

[0184] Every person has different response to wavelengths of light, which is referred to as person’s color matching functions (CMFs). These CMF responses directly relate to how people perceive color. Some implementations may be configured to uniquely render a scene according to a single individual’s CMF responses, e.g., as follows:

LM = BRDF SPD ■ SR)

[0185] In the foregoing equations, the light map (LM) directionality is driven by the BRDF of from structural elements. The spectral response is driven by the multiplication of the SPD and the SR. The final response after the rendering would correspond to the human CMF.

Light Characterization

[0186] Each light as a set of colors that it can produce. In most additive systems, this set of colors can be broken down into hue and luminance. However, some light sources may have additional settings/bulbs at higher luminance, creating a non-additive system. One goal of generating light maps is to determine relationships between drive signals and light output. Hence, full characterization of the light sources is desirable. The light output can be simple XYZ, or more uniquely SPD as discussed previously. One method of efficiently characterizing the lights is described in the Coded Emissions section. It is also important to characterize the spread or directionality of the light source. A spotlight, a strip, or a bulb will affect the viewer and the room uniquely. Device Occlusion

[0187] In some instances, light may be at least partially occluded by objects such as furniture or by other lights. When working in polar coordinates, in-line lights may be merged into a single light source. If there are one or more light occlusions or “dead zones” from the observer viewpoint, such occlusions should be included in the LM. For example, a dead zone may be represented in the LM as low or zero values in coordinates corresponding to the dead zone. The Tenderer may use information regarding dead zones, e.g., for better localization of light objects.

Device Color Matching

[0188] When generating a lightscape, it is important to have all the lights and displays create similar colors. Having one light fixture either brighter or a different hue can degrade the sensory experience. In some instances, it may be easier and/or beneficial to “bake in” the color and intensity calibration into the light map and/or the light fixture itself. In the latter case the drive value to produce a certain color may be specified, instead of specifying a color and the associated drive value. In the case of a very simplistic LM that gives only a percentage of a light object to utilize, having a full characterization of a light source will allow the control system to match colors.

[0189] Figure 8 shows an example of a color volume. The colors within the color volume are indicated according to the density of the fill in a particular area, with the least dense fill corresponding to red and the most dense fill corresponding to violet. The color key 801 indicates the density of fill corresponding to red, orange, yellow, green, blue, indigo and violet (ROYGBIV). To ensure accurate matching of all colors, some implementations involve defining a color volume of all lights, reflective surfaces, etc., in the room, for example as shown in Figure 8. The control system may determine the overlapping volume of all lights, reflective surfaces, etc., in the room. Colors outside of this overlapping volume cannot be produced by all light sources. The mapping itself may be mathematical, or may be based on a three-dimensional (3D) look-up-table (LUT) for converting from drive values to hues/intensities. For color matching, it is helpful to use a perceptually uniform and decorrelated space such as ICtCp, as shown in Figure 8. External Light Sources

[0190] In many instances, one or more uncontrollable light sources may be in the environment. Whether the uncontrollable light sources include light from another room or ambient from windows, these lights affect an accurate light mapping process. In some examples, uncontrollable light sources may be characterized independently from impulse responses as a base ambient layer. According to some examples, the light object Tenderer may compensate for uncontrollable light sources by changing the light object rendering, for example by increasing the drive signals for light objects that are within a region affected by light from an uncontrollable light source. In some examples, one LM may be a “baseline” light map that is obtained with all controllable lights switched off. In some such examples, future LM map captures can cancel out the effects of the baseline light map.

Baseline Ambient

[0191] In some cases, a person may prefer to keep a baseline level of ambient light in the playback environment. This baseline level of ambient light can be characterized in a similar way to the External Light Sources. However, in some implementations the Tenderer may be configured to maintain at least that baseline level of ambient light. In some such examples, the LM may indicate baselines from controllable lights and uncontrollable lights separately.

Manual Input and Hybrid Systems

[0192] Figure 9 depicts an alternative method of producing an LRR. In this example, the LRR is produced without the use of video analysis. In this example, environment descriptor data 901 and light descriptor data 902 are provided to a simulator 903. According to this example, the simulator 903 is implemented by an instance of the control system 110 of Figure 1 A and is configured to estimate an LRR 904 based on the environment descriptor data 901 and the light descriptor data 902. Simulations may, in some examples, account for only the diffuse component of light reflections. Using Lambertian models, for example, the LRR of the j^th light fixture at location p = [x, y, z]^T can be simulated as follows:

, where:

- m = L ^0S — p represents the anti-incident vector;

- The D subscript indicates the diffuse component; - 1 j represents the intensity of the light emitted by the j^th light fixture;

- Lj^0S represents the position of the jth light-fixture;

- N represents the normal of the surface at location; and

- a and /3 represent parameters to alter the impact the attenuation caused by the distance between the light fixture and location p.

[0193] In addition to modelling the diffuse reflection, for more accurate results, especially in endpoints with smoother surfaces, some examples also involve modeling the specular component of the light reflections. This may be done by modelling the specular component on its own and simply adding it to a diffuse component simulated using, for example, the method described above. One way to simulate specular reflections is by using the Blinn model, e.g., as follows:

ARR_s[j, p] = (BN^T

, where:

B represents the normalized bisector of the reflected vector and anti-incident vector and B =

- n = U — p represents the reflected vector;

TTL

- in = - — - represents the normalized anti-incident vector; n

- n = —— represents the normalized reflected vector; IHI2

- <f> represents a parameter that describes how specular (shiny) the surface is; and

- U represents the user position.

ARR_s[j, p] should be set to zero whenever BN^T = 0.

We can combine the foregoing expressions to make a composite simulated ARR, as follows:

The model illustrated here are not based of physically accurate models. If we have sufficient accurate a priori information about the surfaces and light-fixtures in the room then we can use more accurate physically based rendering.

[0194] The environment descriptor data 901 includes what is referred to herein as “environment information” regarding a playback environment that includes controllable light fixtures. The environment descriptor data 901 may, for example, include locations, orientations and dimensions of one or more environment features, environment feature color information, environment feature reflectivity information, or combinations thereof.

According to this example, the light descriptor data 902 includes at least information about controllable light fixtures in the playback environment. The light descriptor data 902 may, for example, include light fixture location data, light parameter data, including but not limited to light color data, light directivity data indicating the direction(s) in which one or more lights of a light fixture provide light, etc. In some instances, at least some of the light descriptor data 902 may have been obtained from published specifications regarding the capabilities, parameters, etc., of one or more commercially available light products. In some alternative examples, the environment descriptor data 901 may include light fixture location data, light directivity data, or both.

[0195] In some examples, the environment descriptor data 901, the light descriptor data 902, or both, may be manually input. For example, the control system 110 may control a display (not shown) to provide a graphical user interface (GUI) with which a user may interact in order to enter the environment descriptor data 901, the light descriptor data 902, or both.

[0196] However, in some “hybrid” examples, at least some of the environment descriptor data 901, the light descriptor data 902, or both, may be generated automatically. For example, in some implementations, the locations, orientations and dimensions of one or more environment features, the locations of one or more light fixtures, etc., may be determined according to a simultaneous location and mapping (SLAM) process rather than being manually entered.

[0197] Figure 10 shows examples of information that may be provided to the simulator of Figure 9. Figure 10 depicts light fixture locations and illumination directions within an environment 1000. In these examples, individual lights are depicted as cones, with the vertices of the cones indicating light locations and the axes of the cones indicating illumination directions. For example, the downlight 1005a is located on the ceiling of the environment 1000. The illumination direction of the downlight 1005a is indicated by the direction of the arrow 1004. In this example, the environment 1000 includes ceiling downlights 1005, television (TV) gradient strip lights 1010 and floor lights 1015. Augmenting LRRs to Produce Light Maps

[0198] The LRR indicates the response of the lights within an environment. In some examples, the LRR may be augmented so that the augmented LRR also maps areas of the environment where no light can project onto, such as behind a TV or couch, and use this as an activation mechanism. Such augmentations are examples of what may be referred to herein as “filling the gaps” in the LRR. Some examples may involve other changes to the LRR. The resulting augmented LRR may be referred to herein as a light map (LM).

[0199] Figures 11 A and 1 IB show examples of augmented LRRs. Figures 11 A and 1 IB show augmented LRRs, which may be referred to as LMs, corresponding to two sections of a TV gradient light strip. Figure 11 A shows an LM 1110a that has been produced by augmenting the TV gradient light strip LRR in the positive x direction, whereas Figure 1 IB shows an LM 1110b that has been produced by augmenting the TV gradient light strip LRR in the positive z direction. In both examples, the augmented LRRs now contain a mapping from behind the TV. If a light object is placed behind the TV, using LMs such as the LM 1110a or the LM 1110b will allow the lightscape Tenderer 101 to project this light object from the spatial domain onto lights that will excite the entire TV gradient strip.

Accordingly, the LM 1110a and the LM 1110b provide natural extensions of the TV gradient light strip LRR such that the lightscape Tenderer 101 can map the entirety of the spatial domain back to lights in the environment 1100.

Regularisation of the Light Map and Repeller Generation

[0200] As noted elsewhere herein, in some examples the lightscape Tenderer 101 uses an inverse LM to project light objects from the spatial domain onto the lights of a playback environment. In some instances, even after an LRR is augmented or otherwise modified, the resulting LM may still include one or more ill-posed spatial coordinates. Some such examples can occur at locations where two lights have strong responses that overlap, but these lights also cover large areas that don’t intersect. If a light object were to be placed in the overlap region, the lightscape Tenderer 101 could cause a large area of the environment to be illuminated and could reduce the ability for another light object to be rendered if the other light object were in the larger area in which the lights do not intersect.

[0201] Some implementations involve ameliorating the effects caused by ill-posed spatial coordinates. Some examples involve regularization techniques that minimize such effects. Some such examples involve apply a SoftMax-type function, potentially to only the overlap region in the foregoing example.

[0202] Alternatively, or additionally, some examples involve implementing one or more repeller functions configured to deflect a trajectory of light objects — or other sensory objects — from one or more ill-posed spatial coordinates. Such repeller functions may be applied to the light objects themselves within the lightscape Tenderer 101, in order to prevent light objects from entering regions that include ill-posed spatial coordinates. In some such examples, repeller metadata may be appended to the LM and the lightscape Tenderer 101 may apply a corresponding repeller function. According to some examples, a control system may be configured to determine regions that include ill-posed spatial coordinates and to generate or select corresponding repeller functions for areas in which these ill-posed spatial coordinates occur. The process of determining regions that include ill-posed spatial coordinates may, for example, involve analyzing a synthetic LRR produced by simulation or analyzing an LRR produced by analyzing video feeds, e.g., as described herein with reference to Figure 7.

[0203] Figure 12 is a flow diagram that outlines one example of a method that may be performed by an apparatus or system such as those disclosed herein. The blocks of method 1200, like other methods described herein, are not necessarily performed in the order indicated. In some implementation, one or more of the blocks of method 1200 may be performed concurrently. Moreover, some implementations of method 1200 may include more or fewer blocks than shown and/or described. The blocks of method 1200 may be performed by one or more devices, which may be (or may include) one or more instances of control system such as the control system 110 that is shown in Figure 1 A and described above.

[0204] In this example, block 1205 involves obtaining, by a control system, controllable actuator location information for a set of controllable actuators in an environment. In this example, the set of controllable actuators includes one or more light fixtures, one or more haptic devices, one or more air flow control devices, or combinations thereof.

[0205] According to this example, block 1210 involves obtaining, by the control system, controllable actuator capability information for each controllable actuator of the set of controllable actuators. In some instances, controllable actuator capability information may include the type of actuation (e.g., light, haptics, air flow, etc.). For light fixtures, the controllable actuator capability information may indicate the light color(s) that can be emitted, light intensity, etc. In some examples, method 1200 may involve obtaining controllable actuator actuation direction information. For light fixtures, the controllable actuator direction information may indicate the direction(s) in which a light fixture is presently configured to transmit light.

[0206] In this example, block 1215 involves obtaining, by the control system, environment information corresponding to the environment. In some instances, the environment information may include environment feature color information, environment feature reflectivity information, or a combination thereof. According to some examples, the environment information may include at least locations, orientations and dimensions of one or more environment features. In some instances, the one or more environment features may include one or more structural elements of the environment, such as one or more walls, a ceiling, a floor, or a combination thereof. In some examples, the one or more environment features may include one or more furniture items. The environment information may include furniture position information.

[0207] According to this example, block 1220 involves generating, by the control system and based at least in part on the controllable actuator location information, the controllable actuator capability information and the environment information, an actuator-room response (ARR) that summarizes responses of the environment to activations of the set of controllable actuators.

[0208] In some examples, one or more of blocks 1205 may involve a manual entry process. The manual entry process may include one or more of the processes that are described with reference to Figure 9. In some such examples, obtaining the controllable actuator location information may involve receiving controllable actuator capability information via an interface system, such as a user interface system or a network system.

[0209] However, according to some examples one or more of blocks 1205 may involve an automated process, such as one or more of the processes that are described with reference to Figure 7. For example, obtaining the controllable actuator location information in block 1205 may involve receiving, by the control system, camera data from one or more cameras and determining, by the control system, the controllable actuator location information based at least in part on the camera data. The camera data may include images of the set of controllable actuators. The camera data may include optical images, depth images, or a combination thereof.

[0210] In some examples, method 1200 may involve determining, by the control system, controllable actuator actuation direction information based at least in part on the camera data. The controllable actuator actuation direction information may include light direction information, air movement direction information, or both.

[0211] According to some examples, method 1200 may involve obtaining, by the control system, inertial measurement data. The inertial measurement data may include gyroscope data, accelerometer data, or both. In some examples, the controllable actuator location information may be determined according to a simultaneous location and mapping (SLAM) process based at least in part on the camera data and the inertial measurement data.

[0212] In some examples, the controllable actuators may include controllable light fixtures. In some such examples, method 1200 also may involve sending, by the control system, calibration signals to the controllable light fixtures while the one or more cameras are obtaining the camera data. The calibration signals may cause the controllable light fixtures to modulate light intensity, light color, or both. In some such examples, obtaining the controllable actuator capability information and obtaining the environment information may both be based, at least in part, on the camera data and the inertial measurement data. According to some examples, the calibration signals may control the controllable light fixtures to emit light according to a code or a pattern that allows the control system to isolate lighting effects caused by each of the controllable light fixtures. In some examples, the calibration signals may be, or may include, a sequence having bounded cross-correlations within a set, such as a Gold sequence. According to some examples, method 1200 may involve color matching of light emitted by two or more of the controllable light fixtures.

[0213] In some examples, obtaining the controllable actuator capability information may involve a light characterization process, which may be, or may involve, a process of determining characteristics of lights within the environment. According to some such examples, method 1200 may involve a light localization and segmentation process based, at least on part, on the camera data. The light characterization process may be based, at least in part, on the light localization and segmentation process. For example, the light localization and segmentation process may involve analyzing pixels of the camera data and determining which pixels correspond with individual lights of the environment. [0214] According to some examples, obtaining the environment information may involve an environment characterization process of determining environment feature color information and determining environment feature reflectivity information. In some examples, obtaining the controllable actuator capability information and obtaining the environment information may involve an iterative process of determining characteristics of lights within the environment, determining environment feature color information and determining environment feature reflectivity information.

[0215] In some examples, method 1200 may involve modifying, by the control system, the ARR to produce an actuator map (AM). The modifying may include regularization, filling gaps in the ARR by increasing one or more volumes of the environment affected by an actuator response, reducing one or more overlapping volumes of the environment affected by multiple actuator responses, or combinations thereof. The AM may indicate an effect of each controllable actuator on the environment. The AM may be mathematically suitable for inversion. The AM may allow rendering of object-based sensory data to actuator commands. In some examples, the ARR may be represented as a matrix and the AM may be represented as a modified version of the matrix. The matrix may include values corresponding to actuator indices, actuation commands and spatial coordinates.

[0216] According to some examples, method 1200 may involve identifying, by the control system, one or more ill-posed spatial coordinates of the AM. Some such examples may involve generating, by the control system, one or more repeller functions configured to deflect a trajectory of sensory objects from the one or more ill-posed spatial coordinates. Some such examples may involve saving or providing the one or more repeller functions with the AM.

[0217] The above description illustrates various embodiments of the present disclosure along with examples of how aspects of the present disclosure may be implemented. The above examples and embodiments should not be deemed to be the only embodiments, and are presented to illustrate the flexibility and advantages of the present disclosure as defined by the following claims. Based on the above disclosure and the following claims, other arrangements, embodiments, implementations and equivalents will be evident to those skilled in the art and may be employed without departing from the spirit and scope of the disclosure as defined by the claims.

Claims

CLAIMS What Is Claimed Is:

1. A method, compri sing : obtaining, by a control system, controllable actuator location information for a set of one or more controllable actuators in an environment, the one or more controllable actuators including one or more light fixtures, one or more haptic sensors, one or more airflow control devices, or combinations thereof; obtaining, by the control system, controllable actuator capability information for each controllable actuator of the set of one or more controllable actuators; obtaining, by the control system, environment information corresponding to the environment; and generating, by the control system and based at least in part on the controllable actuator location information, the controllable actuator capability information, and the environment information, an actuator-room response (ARR) that summarizes responses of the environment to one or more activations of the set of one or more controllable actuators.

2. The method of claim 1, wherein the environment information includes at least one or more locations, one or more orientations and one or more dimensions of one or more environment features and wherein the one or more environment features include one or more structural elements of the environment.

3. The method of claim 2, wherein the one or more structural elements include one or more walls, a ceiling, a floor, or a combination thereof.

4. The method of claim 2 or claim 3, wherein the one or more environment features include one or more furniture items and wherein the environment information includes furniture position information.

5. The method of any one of claims 2-4, wherein the environment information includes environment feature color information, environment feature reflectivity information, or a combination thereof.

6. The method of any one of claims 1-5, wherein obtaining the controllable actuator location information involves: receiving, by the control system, camera data from one or more cameras, the camera data including one or more images of the set of one or more controllable actuators; and determining, by the control system, the controllable actuator location information based at least in part on the camera data.

7. The method of claim 6, further comprising determining, by the control system, controllable actuator actuation direction information based at least in part on the camera data and wherein the controllable actuator actuation direction information includes light direction information, air movement direction information, or both.

8. The method of claim 6 or claim 7, wherein the camera data includes one or more optical images, one or more depth images, or a combination thereof.

9. The method of any one of claims 6-8, further comprising obtaining, by the control system, inertial measurement data, wherein the inertial measurement data includes gyroscope data, accelerometer data, or both, and wherein the controllable actuator location information is determined according to a simultaneous location and mapping (SLAM) process based at least in part on the camera data and the inertial measurement data.

10. The method of any one of claims 1-9, wherein the one or more controllable actuators include one or more controllable light fixtures, further comprising sending, by the control system, one or more calibration signals to the one or more controllable light fixtures while the one or more cameras are obtaining the camera data, the one or more calibration signals causing the one or more controllable light fixtures to modulate light intensity, light color, or both, and wherein obtaining the controllable actuator capability information and obtaining the environment information are both based, at least in part, on the camera data and the inertial measurement data.

11. The method of claim 10, wherein the one or more calibration signals control the one or more controllable light fixtures to emit light according to a code or a pattern that allows the control system to isolate one or more lighting effects caused by each of the one or more controllable light fixtures.

12. The method of claim 10 or claim 11, wherein the one or more calibration signals comprise a sequence having bounded cross-correlations within a set.

13. The method of any one of claims 10-12, further comprising color matching of light emitted by two or more of the controllable light fixtures.

14. The method of any one of claims 10-13, wherein obtaining the controllable actuator capability information involves a light characterization process of determining characteristics of one or more lights within the environment.

15. The method of claim 14, further comprising a light localization and segmentation process based at least on part on the camera data, wherein the light characterization process is based, at least in part, on the light localization and segmentation process.

16. The method of claim 15, wherein the light localization and segmentation process involves analyzing pixels of the camera data and determining which pixels correspond with individual lights of the environment.

17. The method of any one of claims 10-16, wherein obtaining the environment information involves an environment characterization process of determining environment feature color information and determining environment feature reflectivity information.

18. The method of any one of claims 10-17, wherein obtaining the controllable actuator capability information and obtaining the environment information involves an iterative process of determining one or more characteristics of lights within the environment, determining environment feature color information and determining environment feature reflectivity information.

19. The method of any one of claims 1-18, wherein obtaining the controllable actuator location information involves receiving controllable actuator capability information via an interface system.

20. The method of claim 19, wherein the interface system comprises a user interface system.

21. The method of any one of claims 1-20, further comprising modifying, by the control system, the ARR to produce an actuator map (AM).

22. The method of claim 21, wherein the modifying includes regularization, filling gaps in the ARR by increasing one or more volumes of the environment affected by an actuator response, reducing one or more overlapping volumes of the environment affected by multiple actuator responses, or combinations thereof.

23. The method of claim 21 or claim 22, wherein the AM indicates an effect of each controllable actuator on the environment.

24. The method of any one of claims 21-23, wherein the AM is mathematically suitable for inversion.

25. The method of any one of claims 21-24, wherein the AM allows rendering of objectbased sensory data to actuator commands.

26. The method of any one of claims 21-25, further comprising: identifying, by the control system, one or more ill-posed spatial coordinates of the AM; generating, by the control system, one or more repeller functions configured to deflect a trajectory of sensory objects from the one or more ill-posed spatial coordinates; and saving or providing the one or more repeller functions with the AM.

27. The method of any one of claims 21-26, wherein the ARR is represented as a matrix and the AM is represented as a modified version of the matrix.

28. The method of claim 27, wherein the matrix includes values corresponding to one or more actuator indices, one or more actuation commands and spatial coordinates.

29. An apparatus configured to perform the method of any one of claims 1-28.

30. A system configured to perform the method of any one of claims 1-28.

31. One or more non-transitory, computer-readable media having instructions stored thereon for controlling one or more devices to perform the method of any one of claims 1- 28.