EP4728346A1 - A method of controlling a lighting device and an augmented reality device - Google Patents

Abstract

A method of controlling a lighting device and an augmented reality device is disclosed. The lighting device and the augmented reality device are located in a physical environment, and wherein the augmented reality device is configured to render a virtual environment on a display. The method comprises: receiving a first input indicative of a target illumination, receiving a second input indicative of a target ratio between virtual illumination and physical illumination, determining a virtual component of the target illumination based on the target ratio, determining a physical component of the target illumination based on the target ratio, rendering the virtual environment on the display as an overlay on the physical environment and applying the virtual component to the virtual environment, and controlling the lighting device according to the physical component of the target illumination.

Description

A method of controlling a lighting device and an augmented reality device

FIELD OF THE INVENTION

The invention relates to a method of controlling a lighting device and an augmented reality device. The invention further relates to a computer program product for executing the method. The invention further relates to a control system for controlling a lighting device and an augmented reality device.

BACKGROUND

Recent developments in augmented reality (AR) enable users to interact with virtual objects in virtual environments. These virtual environments may be displayed as an overlay on top of the physical environment, for example on a smartphone or on AR-glasses, thereby creating a so-called mixed reality environment. This technology enables many different types of applications, for example interaction with avatars of other users who may be virtually present in the same physical environment, or interaction with virtual characters or other objects that are rendered as an overlay on the physical environment. Users may, for example, chat or play games with virtually present users (or with artificially created characters) in such a mixed reality environment.

US 2020410725 Al discoses a system for evaluation of an augmented reality (AR) experience provided to a user. A controller is communicatively coupled to the display system and the physical light. The controller is configured to render the virtual feature, receive feedback indicative of an operational parameter of the physical light, and receive additional feedback indicative of a state of a virtual light, where the state of the virtual light defines an appearance of the virtual feature. The controller is configured to adjust the appearance of the virtual feature to an updated appearance based on the feedback indicative of the operational parameter of the physical light, adjust the operational parameter of the physical light based on the additional feedback indicative of the state of the virtual light, or both. SUMMARY OF THE INVENTION

The inventors have realized that if a user wishes to create a virtual world as an overlay on the physical world, many environmental aspects need to be set and configured in the virtual world. One of these aspects is lighting (illumination) of the virtual environment. Additionally, lighting is also required for the physical environment. The inventors have realized that it is desirable that different ratios between virtual illumination and physical illumination are required for different contextual characteristics. For instance, certain physical activities require certain physical illumination, certain virtual activities require certain virtual and physical illumination, when multiple users are present in the physical environment their illumination requirements may differ, etc. It is therefore an object to improve the balance between physical and virtual illumination for an augmented reality device.

According to a first aspect, the object is achieved by a method of controlling a lighting device and an augmented reality device operated by a user, wherein the lighting device and the augmented reality device are located in a physical environment, and wherein the augmented reality device is configured to render a virtual environment on a display, the method comprising: receiving a first input indicative of a target illumination, receiving a second input indicative of a target ratio between virtual illumination and physical illumination, determining a virtual component of the target illumination based on the target ratio, determining a physical component of the target illumination based on the target ratio, rendering the virtual environment on the display as an overlay on the physical environment and applying the virtual component to the virtual environment, and controlling the lighting device according to the physical component of the target illumination.

By receiving the target ratio between virtual illumination and physical illumination of the target illumination, and by determining the virtual and physical components of the target illumination based on the ratio, a balance between physical and virtual illumination for an augmented reality device is improved. The ratio may vary for different applications of the method. For instance, if other users are present in the environment, the virtual component may be higher compared to the physical component. In another example, a user may define the ratio, which is beneficial because it enables the user to finetune the lighting according to the user’s desires.

The second input may be a sensor input of a sensor located in the physical environment, and the method may comprise: determining the target ratio based on the sensor input from the sensor. The sensor may, for example, be a presence sensor, a light sensor, etc., and the target ratio may be determined based on the sensor value. This is beneficial, because by taking contextual and/or environmental characteristics into account when determining the ratio between physical and virtual illumination, the balance between physical and virtual illumination for an augmented reality device is further improved.

The sensor may be a light sensor for detecting an ambient light level of the physical environment. The target ratio may be determined such that when the ambient light level is higher, the physical component is lower than the virtual component. This is beneficial, because the balance between virtual and physical illumination is tuned based on the ambient illumination.

The second input may be a user input indicative of the target ratio received via a user interface. A user may provide the user input via the user interface. The user may indicate the ratio on the user interface, for instance by selecting a ratio from a set of predefined ratios. Additionally or alternatively, the user interface may comprise a slider to receive the user input indicative of the target ratio.

The method may comprise: detecting a presence of another user not operating the augmented reality device in the physical environment. The target ratio may be determined based on the presence of the other user such that the virtual component of the target illumination is higher than the physical component. This is beneficial, because the other user may be detected based on sensor input from a presence sensor, based on a presence of another user device of the other user on a network, etc.

The method may further comprise: determining the physical component of the target illumination such that it provides an intensity of the target illumination, and determining the virtual component of the target illumination such that it provides one or more colors of the target illumination. This is beneficial, because the color range and the location range of the virtual lighting is typically larger compared to the range of physical lighting (due to that the locations and the light rendering properties of physical lighting devices are typically fixed).

The method may further comprise: determining a position of a further lighting device and/or its light effect on the display, determining that the target ratio cannot be achieved due to the presence of the lighting device and/or its light effect, and rendering, on the display, one or more virtual objects as an overlay on the lighting device and/or its light effect to obscure the lighting device and/or its light effect. The one or more virtual objects are thus rendered to occlude the further lighting device and/or its light effect to reduce the illumination effect of that further lighting device. This is beneficial, because it enables that the target ratio is reached.

The method may further comprise: obtaining type information indicative of the type of the lighting device, determining if the lighting device is capable of providing the physical component of the target illumination based on the type of the lighting device, and if the lighting device is incapable of providing at least a part of the physical component of the target illumination, increasing the virtual component of the target illumination to compensate for at least the part of the physical component of the target illumination. If the target ratio cannot be achieved due to limitations of the type of the lighting device (e.g. a limited intensity, limited color range, limited beam width/shape, etc.) the virtual component may be increased to compensate therefor.

Additionally, or alternatively, the method may comprise: obtaining location information indicative of the location of the lighting device, determining if the lighting device is capable of providing the physical component of the target illumination based on the location of the lighting device, and if the lighting device is incapable of providing at least a part of the physical component of the target illumination, increasing the virtual component of the target illumination to compensate for at least the part of the physical component of the target illumination. If the target ratio cannot be achieved due to the location of the lighting device (relative to the augmented reality device) the virtual component may be increased to compensate therefore.

The method may comprise: obtaining location information indicative of locations of a plurality of lighting devices in the physical environment, obtaining location information indicative of a target area at which the target illumination is to be provided, selecting the lighting device from the plurality of lighting devices, and controlling the selected lighting device according to at least a part of the physical component of the target illumination. Lighting systems typically comprise a plurality of lighting devices. It is therefore beneficial to select a lighting device that is located at a location near a target area at which the target illumination is to be provided.

According to a second aspect, the object is achieved by a computer program product for a computing device, the computer program product comprising computer program code to perform the method of any of the preceding claims when the computer program product is run on a processing unit of the computing device.

According to a third aspect, the object is achieved by a control system for controlling a lighting device and an augmented reality device operated by a user, wherein the lighting device and the augmented reality device are located in a physical environment, and wherein the augmented reality device is configured to render a virtual environment on a display, the control system comprising one or more processors configured to: receive a first input indicative of a target illumination, receive a second input indicative of a target ratio between virtual illumination and physical illumination, determine a virtual component of the target illumination based on the target ratio, determine a physical component of the target illumination based on the target ratio, render the virtual environment on the display as an overlay on the physical environment and apply the virtual component to the virtual environment, and control the lighting device according to the physical component of the target illumination.

It should be understood that the computer program product and the control system may have similar and/or identical embodiments and advantages as the above- mentioned methods.

BRIEF DESCRIPTION OF THE DRAWINGS

The above, as well as additional objects, features and advantages of the disclosed systems, devices and methods will be better understood through the following illustrative and non-limiting detailed description of embodiments of devices and methods, with reference to the appended drawings, in which:

Fig. 1 shows schematically an example of a head-mounted augmented reality device for rendering a virtual object on a display as an overlay on a physical environment;

Fig. 2 shows schematically an example of an augmented reality device for rendering a virtual object as an overlay on a display;

Fig. 3 shows schematically examples of different ratios between virtual illumination and physical illumination; and Fig. 4 shows schematically a method of controlling a lighting device and an augmented reality device.

All the figures are schematic, not necessarily to scale, and generally only show parts which are necessary in order to elucidate the invention, wherein other parts may be omitted or merely suggested.

DETAILED DESCRIPTION

Figs. 1 and 2 show schematically examples of augmented reality devices 120. In Fig. 1, the augmented reality device 120 is depicted as a head-mounted augmented reality device (e.g. augmented reality glasses), and in Fig. 2 the augmented reality device 120 is depicted as a hand-held augmented reality device (e.g. a smartphone or a tablet pc). A plurality of lighting devices 130, 132 and the augmented reality device 120 are located in a physical environment 140. The augmented reality device 120 comprises a display 122 configured to render a virtual environment 150 as an overlay on the physical environment 140. Fig. 2 further depicts a control system 102 (not shown in Fig. 1) for controlling the lighting devices 130, 132 and the augmented reality device 120. The control system 102 comprises one or more processors 106 (e.g. circuitry, microcontrollers, microchips). The control system 102 may further comprise one or more communication units 104 for communicating with the lighting devices 130, 132 and/or the augmented reality device 120. The one or more processors 106 are configured to: receive a first input indicative of a target illumination, receive a second input indicative of a target ratio between virtual illumination and physical illumination, determine a virtual component of the target illumination based on the target ratio, determine a physical component of the target illumination based on the target ratio, render the virtual environment on the display 122 as an overlay on the physical environment 140 and applying the virtual component to the virtual environment 150, and control one or more lighting devices 130, 132 according to the physical component of the target illumination.

The control system 102 may comprise a single processor 106 for performing these steps. The processor 106 may, for example, be comprised in a (central) lighting control system (e.g. a bridge, a hub, a smartphone, etc.), in the augmented reality device 120, in a remote (cloud) server, etc. Alternatively, the control system 102 may comprise multiple processors 106 for performing these steps. The processors 106 may be located in different parts of a system 100, which system may comprise one or more lighting devices 130, 132, the control system 102 and/or the augmented reality device 120. The locations of the processors 106 and the steps performed by the respective processors 106 may depend on the system architecture of the control system 102 and/or the system architecture of the system 100. Examples thereof are explained below.

Figs. 1 and 2 further depict two lighting devices 130, 132 located in the physical environment 140. The lighting devices 130, 132 may be any type of lighting device comprising one or more (LED) light sources, and a processing unit for controlling the light output (e.g. hue, saturation and/or brightness) of the one or more light sources based on received control signals. The lighting devices 130, 132 may be arranged for providing general lighting, such as task lighting, ambient lighting, atmosphere lighting, accent lighting, indoor lighting, outdoor lighting, etc. The lighting devices 130, 132 may further comprise a communication unit (not shown) configured to receive lighting control commands (and, optionally, orientation control commands). The communication unit may comprise hardware for communicating via one or more wireless communication protocols, for example Bluetooth, Wi-Fi, Li-Fi, 3G, 4G, 5G or ZigBee. A specific communication technology may be selected based on the system architecture of the lighting system.

The augmented reality device 120 comprises a display 122 for rendering a virtual environment 150 comprising one or more virtual objects 152 as an overlay on a view of the physical environment 140. An example of the physical environment 140 is depicted in Fig. 2. The depicted physical environment 140 comprises a couch, a tv and two lighting devices 130, 132. The display 122 may be a (semi-) transparent see-through display, wherein the user can see the physical environment 140 through the display 122, and wherein the display 122 is configured to render the virtual environment 150 comprising one or more virtual objects 152 as an overlay on the physical environment 140. The display 122 may be integrated in the (semi-) transparent see-through display, or the virtual environment may be projected on the display 122. Alternatively, augmented reality device 120 may comprise a camera 124 configured to continuously capture images of the physical environment 140 and render the images on the display 122, while rendering the virtual environment comprising one or more virtual objects 152 as an overlay on the images. It should be understood that such augmented reality devices are known in the art, and will therefore not be discussed in further detail.

The augmented reality device 120 may comprise a processor configured to render, on the display 122, the virtual environment 150 as an overlay on the physical environment 140. The lighting devices 130, 132 and the augmented reality device 120 are located in the same physical environment 140 (e.g. a room such as a living room, an office, etc.). The user operating the augmented reality device 120 is also located in the physical environment 140. By rendering the virtual environment as an overlay on the physical environment 140, a so-called mixed reality environment is created, wherein the user can see both physical (real-life) objects and virtual objects. The augmented reality device 120 may comprise a processor configured to determine how to render the virtual environment as an overlay on the physical environment. The processor may, for example, determine to render the virtual environment 150 as an overlay on the physical environment 140 based on a location and/or an orientation of the augmented reality device 120. The processor may, for example, map the virtual environment onto the physical environment 140. The processor may, for example, use image analysis to analyze an image of the physical environment 140, and select one or more anchor points in the physical environment 140 based on the image analysis. The one or more processors 106 may then anchor virtual objects (e.g. virtual characters, virtual furniture, a virtual space, etc.) of the virtual environment to the physical environment 140 based on the anchor points. Alternatively, the processor may be configured to obtain a predefined mapping of the virtual environment onto the physical environment 140. The mapping may, for example, be based on the location and the orientation of the augmented reality device 120 relative to the physical environment 140. Such techniques for mapping a virtual environment onto a physical environment are known in the art and will therefore not be discussed in further detail.

The one or more processors 106 are configured to receive a first input indicative of a target illumination. The target illumination may be a target illumination of the augmented reality view of the augmented reality device 120, and it may have a physical illumination part 131, 133 and a virtual illumination part 151. The target illumination may for instance specify target lighting colors, lighting intensities, polarization, temporal light artifacts (e.g. flicker level), beam angle and/or dynamic light effects. The target illumination may further specify a target number of light sources and/or a target location of the light sources. Additionally, the one or more processors 106 are configured to receive a second input indicative of a target ratio between virtual illumination and physical illumination. The target ratio may for example be defined as a percentage of physical illumination and a percentage of virtual illumination of the (total) target illumination. The ratio may, for example, be 50:50, 70:30, 30:70, 10:90, 90: 10, etc. The ratio may be defined as a numerical value, or as non-numerical values. The target ratio may, for example, be a textual description that defines a ratio between physical lighting and virtual lighting. The textual description may be fed to a generative Al model (e.g. a large language model (LLM)), which may be configured to analyze the textual description and transform this to a virtual component and a physical component of the target illumination. For instance, the textual description may be “Transform my room into a sunset on a planet from a parallel universe”. Based on that the user asks for a parallel universe, the generative Al knows that it can violate the rules of physics on planet earth and decides that it wants to render a brown spot lighting effect in the user’s room. As the generative Al model is trained to understand that it is physically impossible to make brown-colored illumination (brown is a subtractive color), the browncolored light effect will be rendered as a virtual component of the target illumination. In other examples, the target ratio may be defined based on the transparency of the display of the AR device.

The one or more processors 106 may be configured to receive the first input and the second input via an input interface, from a lighting control software application, an augmented software reality application, etc. The input interface may be an input pin of a processor 106. The one or more processors 106 may be configured to receive the inputs from an internal memory. Alternatively, the input interface may be a receiver (or a transceiver), for instance communication unit 104, configured to receive the input signals from an external source via a (wireless) network. It should be understood that these are mere examples, and that the skilled person is able to design alternatives without departing from the scope of the appended claims.

The one or more processors 106 are further configured to determine a virtual component of the target illumination based on the target ratio and a physical component of the target illumination based on the target ratio. The one or more processors 106 may, for example, determine the physical component by selecting a subset of lighting devices (from the lighting devices of the lighting system), determining intensities of the light output of the lighting devices, determining the saturation of the light output of the lighting devices, etc. The one or more processors 106 are further configured to control the one or more lighting devices 130, 132 according to the physical component 131, 133 of the target illumination. The one or more processor 106 may communicate one or more lighting control commands to the one or more lighting devices 130, 132, for instance via communication unit 104. The one or more processors 106 are further configured to render the virtual environment 150 on the display 122 as an overlay on the physical environment 140 and apply the virtual component 151 to the virtual environment 150. The virtual component 151 may be rendered as light emitted by a virtual light source, which virtual light source may also be rendered on the display 122. Alternatively, the virtual component 151 may be rendered without rendering a virtual light source on the display 122.

Figs. 3a, 3b and 3c show schematically examples of various target ratios between virtual illumination and physical illumination. In the example of Fig. 3a, the target ratio may be 100:0 (100% virtual illumination and 0% physical illumination). Based thereon, the one or more processors 106 may determine not to control the lighting devices 130, 132 (or at least not such that these lighting devices contribute to the target illumination). The one or more processors 106 may, for example, switch these lighting devices off, or control them or according to or leave them at a default/previous light setting which does not contribute to the target illumination. The one or more processors 106 may determine the virtual component 151 of the target illumination such that the virtual illumination fully provides the target illumination, for instance by rendering virtual light sources which provide the virtual illumination and/or by rendering virtual ambient illumination of the virtual environment.

In the example of Fig. 3b, the target ratio may be 50:50 (50% virtual illumination and 50% physical illumination). Based thereon, the one or more processors 106 may determine control the lighting devices 130, 132 and to provide virtual illumination as an overlay on the display 122. The one or more processors 106 may, for example, control the lighting devices according to a reduced intensity (e.g. 50%). The one or more processors 106 may determine the virtual component 151 of the target illumination such that the virtual illumination partially provides the target illumination, for instance by rendering virtual light sources which provide the virtual illumination and/or by rendering virtual ambient illumination of the virtual environment.

In the example of Fig. 3c, the target ratio may be 0: 100 (0% virtual illumination and 100% physical illumination). Based thereon, the one or more processors 106 may determine to control the lighting devices 130, 132 to provide the full target illumination. The one or more processors 106 may, for example, control the lighting devices to provide target colors and/or target illumination levels in the physical environment 140. The one or more processors 106 may determine the virtual component 151 of the target illumination to be 0, and refrain from rendering virtual light sources which provide the virtual illumination and/or ambient illumination of the virtual environment.

More detailed examples are described below.

In a first example, the one or more processors 106 may receive a first input indicative of a target illumination for the augmented reality view of the augmented reality device 120. In this example, a software application running on the augmented reality device 120 may be configured to generate an ambient environment for the user operating the augmented reality device 120. The ambient environment may, for example, replicate an outdoor environment (e.g. a jungle environment). The target illumination may thus specify target lighting colors (e.g. green), lighting intensities (e.g. 1000 lumen) and dynamic light effects (e.g. illumination that changes over time). The target illumination may further specify a target number of light sources (e.g. 3) and/or a target location of the light sources (e.g. at the top of the field of view of the augmented reality device 120). The one or more processors 106 may further receive a second input indicative of a target ratio between virtual illumination and physical illumination. The second input may for example be received from a user via a user interface. The user may, for example, indicate a desired ratio between the virtual illumination and physical illumination. In this example, the user may have indicated a 50:50 ratio for the virtual illumination vs. the physical illumination. Based thereon, the one or more processors 106 may determine a virtual component of the target illumination and a physical component of the target illumination. The one or more processors 106 may, for example, select a subset of lighting devices (e.g. half of the lighting devices, for instance those located in the ceiling able to provide a canopy -type jungle illumination) for the physical component of the target illumination, and control the subset of lighting devices according to the dynamic light effects (e.g. by changing the intensity of the light output of the subset of lighting devices over time to create a canopy-type lighting effect). Additionally, the one or more processors 106 may, for example, provide virtual illumination as an overlay on the physical environment to create additional dynamic light effects (e.g. at locations where no (physical) lighting devices are located), or provide virtual illumination to render green colored light to provide a green jungle atmosphere. If, for example, the user would have indicated a 0: 100 ratio for the virtual illumination vs. the physical illumination, the one or more processors 106 may have selected all lighting devices of the lighting system, and controlled them to provide both dynamic light effects (intensity changes over time) and (green) colored light effects, without providing virtual illumination. If the ratio would have been 100:0, the one or more processors 106 may only apply the virtual component of the target illumination when rendering the virtual environment on the display, and the one or more lighting devices of the lighting system may, for example, be switched off or be set to a default light setting.

In another example, the one or more processors 106 may receive a first input indicative of a target illumination for the augmented reality view of the augmented reality device 120. In this example, a software application running on the augmented reality device 120 may be configured to generate an ambient environment for the user operating the augmented reality device 120. The ambient environment may, for example, replicate a game environment wherein the target illumination changes in dependence of the virtual location of the user in the virtual environment. The target illumination may specify target lighting colors, lighting intensities and dynamic light effects, which all change over time. The target illumination may further specify a target number of light sources and/or a target location of the light sources in dependence of the virtual location of the user in the virtual environment. The one or more processors 106 may further receive a second input indicative of a target ratio between virtual illumination and physical illumination. In this example, the second input may be a sensor input received from a light sensor. The sensor input may indicate that the ambient light level (environmental illumination) is low (because it is dark in the physical environment 140). In this example, the one or more processors 106 may determine the target ratio between virtual illumination and physical illumination such that the ratio for the virtual illumination vs. the physical illumination is 90: 10 (for instance to keep the ambient light level low). Based thereon, the one or more processors 106 may determine a virtual component of the target illumination and a physical component of the target illumination. The one or more processors 106 may, for example, dynamically select subsets of lighting devices (e.g. single lighting device) over time for the physical component of the target illumination, and control the subsets of lighting devices according to the target illumination in dependence on the virtual location of the user in the virtual environment (e.g. by changing the intensity of the light output of the subset of lighting devices over time). Additionally, the one or more processors 106 may, for example, provide virtual illumination as an overlay on the physical environment to create approximately 90% of the target illumination.

The second input may be an input of the carbon intensity of the electricity and/or the battery level of the augmented reality device. If the carbon intensity of electricity is right now high, the one or more processors 106 may determine the virtual illumination as an overlay on the physical environment to create, for example, approximately 60% of the target illumination such that the physical lighting devices consume less energy.

In another example, the one or more processors 106 may determine the target ratio further based on the battery level of the augmented reality device. If, for example the battery level is below a threshold level, the one or more processors 106 may assign a 10:90 ratio for the virtual illumination vs. the physical illumination. In other examples, the target ratio may be determined further as a function of the battery level of the augmented reality device. The second input may be a sensor input of a sensor located in the physical environment. The one or more processors 106 may determine the target ratio based on the sensor input from the sensor. The sensor may be comprised in augmented reality device 120, in a remote device (e.g. a sensor device), in a lighting device, etc. The one or more processors 106 may be configured to obtain the sensor data from the sensor, for instance directly or indirectly via a communication unit.

The sensor may, for example, be a light sensor for detecting an ambient light level of the physical environment 140. The one or more processors 106 may for example determine the target ratio such that when the ambient light level is higher (e.g. above a threshold light level), the physical component is lower than the virtual component. If, for example, the ambient light level is above a threshold light level, the effect of controlling the lighting devices in the physical environment may be low, and it would therefore be more beneficial to provide (more) virtual illumination. Alternatively, the one or more processors 106 may for example determine the target ratio such that when the ambient light level is lower (e.g. below a threshold light level), the physical component is lower than the virtual component.

The sensor may, for example, be a presence sensor (e.g. a PIR sensor, a camera, etc.) for detecting a presence and/or a number of people present in the physical environment 140. The one or more processors 106 may for example be configured to determine the target ratio based on a sensor signal of the sensor, for instance such that when presence of another user is detected, the physical component is lower than the virtual component. If, for example, the another user is present in the physical environment 140, the effect of controlling the lighting devices in the physical environment may affect the other user, and it would therefore be more beneficial to provide more virtual illumination compared to physical illumination.

The sensor may, for example, be a camera for detecting an activity of another user in the physical environment 140. The one or more processors 106 may for example be configured to determine the target ratio based on the activity of the other user. If, for example, the other user is reading a book, the one or more processors 106 may determine the physical component is lower compared to the virtual component.

The second input may be a user input indicative of the target ratio received via a user interface. The user interface may be a user interface of the augmented reality device 120, or be a user interface of a different device. Examples of user interfaces include but are not limited to touch user interfaces (e.g. comprised in a mobile device or the augmented reality device 120), voice user interfaces (e.g. comprised in the augmented reality device 120 or comprised in another device such as a dedicated voice assistant device), gesture user interfaces, eye tracking user interfaces, etc. The user interface may for example be a touchscreen comprising a slider to receive the user input indicative of the target ratio. The user may indicate a desired target ratio between virtual and physical illumination by sliding the slider across the touch sensitive surface of the screen. The user interface may be further configured to render a plurality of images that preview the effect of different ratios. The user interface may be provided on the display 122 of the augmented reality device 122, which may show the previews. Examples of such previews are provided in Figs. 3a-3c.

The one or more processors 106 may be further configured to detect a presence of another user not operating the augmented reality device in the physical environment, for instance based on a sensor signal of a presence sensor or a camera comprised in the augmented reality device 120. The one or more processors 106 may be further configured to determine the target ratio further based on the presence of the other user such that the virtual component of the target illumination is higher than the physical component.

The one or more processors 106 may be further configured to determine the physical component of the target illumination such that it provides an intensity of the target illumination, and determining the virtual component of the target illumination such that it provides one or more colors of the target illumination. In other words, the one or more processors may split a color component of the target illumination and an intensity component of the target illumination, and determine the physical and virtual components based thereon. For instance, if the target illumination comprises bright green illumination, the one or more processors 106 may determine the physical component such that the lighting devices are controlled according to a corresponding brightness (e.g. according to desaturated (white) light settings) and determine the virtual component such that it provides the green color (e.g. by rendering virtual green illumination on the display as an overlay on the field of view of the augmented reality device 120.

The one or more processors 106 may be further configured to obtain type information indicative of the types of the one or more lighting devices 130, 132. The one or more processors 106 may be configured to obtain the type information from the lighting devices 130, 132, from a central lighting controlled (e.g. a bridge), from a local or remote memory, etc. The one or more processors 106 may then determine if the one or more lighting devices are capable of providing the physical component of the target illumination based on the type of the one or more lighting devices 130, 132. If the one or more lighting devices are incapable of providing at least a part of the physical component of the target illumination, the one or more processors 106 may increase the virtual component of the target illumination to compensate for at least the part of the physical component of the target illumination. For instance, the one or more processors 106 may, determine that the physical component requires colored light, and determine (based on the type information) that the one or more lighting devices 130, 132 are (only) capable of providing white light, the one or more processors may adjust the virtual component such that the virtual component comprises the required color. In another example, the one or more processors 106 may, for example, determine that the physical component requires a certain target brightness, and determine (based on the type information) that the one or more lighting devices 130, 132 are incapable of providing that target brightness, the one or more processors may adjust the virtual component such that the brightness of the virtual component is increased.

The one or more processors 106 may be further configured to obtain location information indicative of the locations of the one or more lighting devices 130, 132 (relative to the augmented reality device 120). The one or more processors 106 may be configured to obtain the location information from the lighting devices 130, 132, from a central lighting controlled (e.g. a bridge), from a local or remote memory, from an (indoor) positioning system, etc. Additionally or alternatively, the field of view of the augmented reality device 120 may be analyzed to determine the locations of the lighting devices 130, 132 with respect to the augmented reality device 120. Techniques for determinig locations of lighting devices with respect to an augmented reality device are known in the art and will therefore not be discussed in further detail.

The one or more processors 106 may determine if the one or more lighting devices are capable of providing the physical component of the target illumination based on the locations of the one or more lighting devices 130, 132. If the one or more lighting devices are incapable of providing at least a part of the physical component of the target illumination based on their locations, the one or more processors 106 may increase the virtual component of the target illumination to compensate for at least the part of the physical component of the target illumination. For instance, the one or more processors 106 may, determine that the physical component requires a light effect at the bottom of the field of view of the augmented reality device, and determine (based on the location information) that the one or more lighting devices 130, 132 are located in the top of the field of view of the augmented reality device 120. The one or more processors 106 may then may adjust the virtual component such that the virtual component comprises the light effect at the bottom of the field of view of the augmented reality device 120.

The one or more processors 106 may be further configured to determine a position of a further lighting device and/or its light effect on the display 120. The one or more processors 106 may obtain location information indicative of the position (location) of the further lighting device and or its light effect (relative to the augmented reality device 120) in a similar way as described above. The one or more processors 106 may be further configured to determine that the target ratio cannot be achieved due to the presence of the lighting device and/or its light effect, and render, on the display, one or more virtual objects as an overlay on the lighting device and/or its light effect to obscure (occlude) the further lighting device and/or its light effect.

The one or more processors 106 may be further configured to obtain location information indicative of locations of a plurality of lighting devices in the physical environment. The one or more processors 106 may obtain the location information indicative of the locations of a plurality of lighting devices in a similar way as described above. The one or more processors 106 may be further configured to obtain location information indicative of a target area (e.g. at a certain area in the field of view of the augmented reality device) at which the target illumination is to be provided, and selecting a lighting device from the plurality of lighting devices which is able to provide at least a part of the physical component of the target illumination at the target area.

Fig. 4 shows schematically a method 400 of controlling a lighting device and an augmented reality device, wherein the lighting device and the augmented reality device are located in a physical environment, and wherein the augmented reality device is configured to render a virtual environment on a display. The method 400 comprises: receiving 402 a first input indicative of a target illumination, receiving 404 a second input indicative of a target ratio between virtual illumination and physical illumination, determining 406 a virtual component of the target illumination based on the target ratio, determining 408 a physical component of the target illumination based on the target ratio, rendering 410 the virtual environment on the display as an overlay on the physical environment and applying the virtual component to the virtual environment, and controlling 412 the lighting device according to the physical component of the target illumination.

The method 400 may be executed by computer program code of a computer program product when the computer program product is run on a processing unit of a computing device, such as the one or more processors 106.

It should be noted that the above-mentioned embodiments illustrate rather than limit the invention, and that those skilled in the art will be able to design many alternative embodiments without departing from the scope of the appended claims.

In the claims, any reference signs placed between parentheses shall not be construed as limiting the claim. Use of the verb "comprise" and its conjugations does not exclude the presence of elements or steps other than those stated in a claim. The article "a" or "an" preceding an element does not exclude the presence of a plurality of such elements. The invention may be implemented by means of hardware comprising several distinct elements, and by means of a suitably programmed computer or processing unit. In the device claim enumerating several means, several of these means may be embodied by one and the same item of hardware. The mere fact that certain measures are recited in mutually different dependent claims does not indicate that a combination of these measures cannot be used to advantage.

Aspects of the invention may be implemented in a computer program product, which may be a collection of computer program instructions stored on a computer readable storage device which may be executed by a computer. The instructions of the present invention may be in any interpretable or executable code mechanism, including but not limited to scripts, interpretable programs, dynamic link libraries (DLLs) or Java classes. The instructions can be provided as complete executable programs, partial executable programs, as modifications to existing programs (e.g. updates) or extensions for existing programs (e.g. plugins). Moreover, parts of the processing of the present invention may be distributed over multiple computers or processors or even the ‘cloud’.

Storage media suitable for storing computer program instructions include all forms of nonvolatile memory, including but not limited to EPROM, EEPROM and flash memory devices, magnetic disks such as the internal and external hard disk drives, removable disks and CD-ROM disks. The computer program product may be distributed on such a storage medium, or may be offered for download through HTTP, FTP, email or through a server connected to a network such as the Internet.

Claims

CLAIMS:

1. A method (400) of controlling a lighting device and an augmented reality device operated by a user, wherein the lighting device and the augmented reality device are located in a physical environment, and wherein the augmented reality device is configured to render a virtual environment on a display, the method (400) comprising: receiving a first input indicative of a target illumination, detecting, based on a sensor signal of a sensor, a presence of another user not operating the augmented reality device in the physical environment, determining a target ratio between virtual illumination and physical illumination, determining a virtual component of the target illumination based on the target ratio, determining a physical component of the target illumination based on the target ratio, rendering the virtual environment on the display as an overlay on the physical environment and applying the virtual component to the virtual environment, and controlling the lighting device according to the physical component of the target illumination, wherein the target ratio is determined based on the presence of the other user such that the virtual component of the target illumination is higher than the physical component.

2. The method (400) of any preceding claim, wherein the second input is a sensor input of a sensor located in the physical environment, and wherein the method (400) comprises: determining the target ratio based on the sensor input from the sensor.

3. The method (400) of claim 2, wherein the sensor is a light sensor for detecting an ambient light level of the physical environment.

4 The method (400) of claim 3, wherein the target ratio is determined such that when the ambient light level is higher, the physical component is lower than the virtual component.

5. The method (400) of claim 1, wherein the second input is a user input indicative of the target ratio received via a user interface.

6. The method (400) of claim 5, wherein the user interface comprises a slider to receive the user input indicative of the target ratio.

7. The method (400) of any preceding claim, wherein the method (400) comprises: determining the physical component of the target illumination such that it provides an intensity of the target illumination, and determining the virtual component of the target illumination such that it provides one or more colors of the target illumination.

8. The method (400) of any preceding claim, further comprising: obtaining type information indicative of the type of the lighting device, determining if the lighting device is capable of providing the physical component of the target illumination based on the type of the lighting device, and if the lighting device is incapable of providing at least a part of the physical component of the target illumination, increasing the virtual component of the target illumination to compensate for at least the part of the physical component of the target illumination.

9. The method (400) of any preceding claim, further comprising: obtaining location information indicative of the location of the lighting device, determining if the lighting device is capable of providing the physical component of the target illumination based on the location of the lighting device, and if the lighting device is incapable of providing at least a part of the physical component of the target illumination, increasing the virtual component of the target illumination to compensate for at least the part of the physical component of the target illumination.

10. The method (400) of any preceding claim, wherein the method (400) comprises: determining a position of a further lighting device and/or its light effect on the display, determining that the target ratio cannot be achieved due to the presence of the lighting device and/or its light effect, and rendering, on the display, one or more virtual objects as an overlay on the further lighting device and/or its light effect to obscure the lighting device and/or its light effect.

11. The method (400) of any preceding claim, wherein the method (400) further comprises: obtaining location information indicative of locations of a plurality of lighting devices in the physical environment, obtaining location information indicative of a target area at which the target illumination is to be provided, selecting the lighting device from the plurality of lighting devices, and controlling the selected lighting device according to at least a part of the physical component of the target illumination at the target area.

12. A control system (102) for controlling a lighting device (130, 132) and an augmented reality device (120), wherein the lighting device and the augmented reality device are located in a physical environment (140), and wherein the augmented reality device is configured to render a virtual environment (150) on a display (122), the control system comprising one or more processors (106) configured to: receive a first input indicative of a target illumination, detect, based on a sensor signal of a sensor, a presence of another user not operating the augmented reality device in the physical environment, determine a target ratio between virtual illumination and physical illumination,- determine a virtual component (151) of the target illumination based on the target ratio, determine a physical component (131, 133) of the target illumination based on the target ratio, render the virtual environment on the display as an overlay on the physical environment and apply the virtual component to the virtual environment, and control the lighting device according to the physical component of the target illumination, wherein the target ratio is determined based on the presence of the other user such that the virtual component of the target illumination is higher than the physical component.

13. A computer program product for a computing device, the computer program product comprising computer program code to perform the method (400) of any of claims 1- 11 when the computer program product is run on the one or more processors of the control system of claim 12.