WO2024256316A1

WO2024256316A1 - An augmented reality device for providing light effects

Info

Publication number: WO2024256316A1
Application number: PCT/EP2024/065891
Authority: WO
Inventors: Peter Deixler; Dzmitry Viktorovich Aliakseyeu
Original assignee: Signify Holding BV
Current assignee: Signify Holding BV
Priority date: 2023-06-13
Filing date: 2024-06-10
Publication date: 2024-12-19
Anticipated expiration: 2025-12-13
Also published as: CN121311857A

Abstract

A method of controlling a lighting device and an augmented reality device is disclosed. The lighting device and the augmented reality device are located in a physical environment. The method comprising rendering, on a display of the augmented reality device, a virtual environment as an overlay on the physical environment, obtaining a signal indicative of that a virtual object has entered the virtual environment, wherein the virtual object is associated with a light effect, determining a virtual location in the virtual environment where the virtual object has entered the virtual environment, determining a physical location in the physical environment that corresponds to the virtual location, determining if the physical location corresponds to a location of the lighting device based on obtained location information indicative of the location of the lighting device in the physical environment, and if the physical location corresponds to the location of the lighting device, controlling the lighting device to provide the light effect associated with the virtual object, and if the physical location does not correspond to the location of the lighting device, rendering a virtual light effect indicative of the light effect at the virtual location in the virtual environment on the display.

Description

AN AUGMENTED REALITY DEVICE FOR PROVIDING LIGHT EFFECTS

FIELD OF THE INVENTION

The invention relates to a method of controlling a lighting device and an augmented reality device. The invention further relates to a computer program product for executing the method. The invention further relates to a control system for controlling a lighting device and an augmented reality device.

BACKGROUND

Recent developments in augmented reality (AR) enable users to interact with virtual objects. These virtual objects may be displayed as an overlay on top of the physical environment, for example on a smartphone or on AR-glasses, thereby creating a so-called mixed reality environment. This technology enables many different types of applications, for example interaction with avatars of other users who may be virtually present in the same physical environment, or interaction with virtual characters or other objects that are rendered as an overlay on the physical environment. Users may, for example, chat or play games with virtually present users (or with artificially created characters) in such a mixed reality environment.

EP 3583827 Bl discloses a controller for indicating a presence of a virtual object via a lighting device. The virtual object has a virtual location corresponding to a physical location. The controller comprises a receiver configured to receive location information representative of the physical location of the virtual object relative to the lighting device. The controller further comprises a processor configured to determine if the virtual object is located within a first predetermined proximity of the lighting device based on the location information, and to control the lighting device according to a light setting when the virtual object is located within the first predetermined proximity of the lighting device.

SUMMARY OF THE INVENTION

The inventors have realized that when a certain virtual object, such as a virtual character or an avatar of another user, enters/appears in the mixed reality environment, it may be desirable to notify the user of the augmented reality device about the appearance of the virtual object. Typically, for instance in chat rooms or VR (virtual reality) games such a notification is typically done by providing a sound indicating that another user has joined the virtual environment. The inventors have realized that in a mixed reality environment, new paradigms become available for improving the notification of the appearance of a virtual object. It is therefore an object to improve notifying a user that a virtual character has entered a mixed reality environment.

According to a first aspect, the object is achieved by a method of controlling a lighting device and an augmented reality device, wherein the lighting device and the augmented reality device are located in a physical environment, the method comprising: rendering, on a display of the augmented reality device, a virtual environment as an overlay on the physical environment, obtaining a signal indicative of that a virtual object has entered the virtual environment, wherein the virtual object is associated with a light effect, determining a virtual location in the virtual environment where the virtual object has entered the virtual environment, determining a physical location in the physical environment that corresponds to the virtual location, determining if the physical location corresponds to a location of the lighting device based on obtained location information indicative of the location of the lighting device in the physical environment, and if the physical location corresponds to the location of the lighting device, controlling the lighting device to provide the light effect associated with the virtual object, and if the physical location does not correspond to the location of the lighting device, rendering a virtual light effect indicative of the light effect at the virtual location in the virtual environment on the display.

The virtual environment is rendered as an overlay on the physical environment, and when a virtual object enters the virtual environment, it is determined if a (physical) lighting device is located at/nearby the physical location which corresponds to the virtual location at which the virtual object has entered the virtual environment. The virtual environment may be mapped onto the physical environment to determine the physical location in the physical environment that corresponds to the virtual location at which the virtual object has entered the virtual environment. If the physical location (that corresponds to the virtual location at which the virtual object has entered the virtual environment) corresponds to the location of the lighting device (e.g. the location of the lighting devices is located within a threshold distance from the physical location that corresponds to the virtual location at which the virtual object has entered the virtual environment), the lighting device is controlled to provide the light effect associated with the virtual object. This indicates to the user operating the augmented reality device that the virtual object has entered the virtual environment. This is beneficial, because it enhances the notification and the user can immediately see the location at which the virtual object has entered the virtual environment (and, from a user’s perspective, the physical environment). If the physical location does not correspond to the location of the lighting device, the augmented reality device is controlled to render a virtual counterpart of the light effect at the virtual location in the virtual environment on the display.

The method may further comprise: obtaining type information indicative of the type of the lighting device, determining if the lighting device is capable of providing the light effect based on the type of the lighting device, and if the physical location corresponds to the location of the lighting device and if the lighting device is capable of providing the light effect, controlling the lighting device to provide the light effect associated with the virtual object, and if the physical location corresponds to the location of the lighting device and if the lighting device is incapable of providing the light effect, rendering the virtual light effect at the virtual location in the virtual environment on the display. Some lighting devices may not be able to provide the light effect. Therefore, it is beneficial to take the type of lighting device into account to provide a light effect that corresponds to the light effect associated with the virtual object.

The method may further comprise: obtaining location information indicative of locations of a plurality of lighting devices in the physical environment, comparing the locations of the plurality of lighting devices to the physical location, and selecting the lighting device from the plurality of lighting devices based on the comparison. The lighting system may comprise a plurality of lighting devices in the physical environment, and a lighting device for providing the (physical) light effect may be selected based on its location. Additionally or alternatively, the method may comprise: obtaining type information indicative of types of a plurality of lighting devices in the physical environment, and selecting the lighting device from the plurality of lighting devices based on the type. A lighting device may be selected based on its type such that it can provide the (physical) light effect.

The light effect may have a predefined size. The method may comprise: determining a physical area in the physical environment that corresponds to the predefined size of the light effect, and the step of determining if the physical location corresponds to the location of the lighting device may comprise: determining if the lighting device or its light output is located in the physical area. Consequently, when the lighting device is located in the physical area, it is controlled to provide the light effect associated with the virtual object.

The light effect may be a dynamic light effect that moves from the virtual location to a subsequent virtual location. The method may further comprise: determining a subsequent physical location in the physical environment that corresponds to the subsequent virtual location, obtaining location information indicative of locations of one or more further lighting devices in the physical environment, determining if one or more further lighting devices are located at the subsequent physical location, and if the subsequent physical location corresponds to the location of the one or more further lighting devices, controlling the one or more further lighting devices to provide the light effect at the subsequent physical location when the dynamic light effect has moved to the subsequent virtual location, and if the subsequent physical location does not correspond to the location of the one or more further lighting devices, rendering the virtual light effect at the subsequent virtual location in the virtual environment on the display when the dynamic light effect has moved to the subsequent virtual location. This is beneficial because the light effect propagates through the virtual and the physical environment. The light effect may be a dynamic light effect that moves to multiple subsequent virtual locations, and the steps may be repeated for the multiple subsequent virtual locations.

The method may further comprise: determining if the physical location is (fully) visible on the display, and, if the physical location is not (fully) visible on the display, extending a size of the light effect and/or repositioning the light effect to another virtual location (corresponding to another physical location) such that the light effect becomes visible on the display. This is beneficial, because the light effect (and therewith the appearance of the virtual object) becomes visible to the user operating the augmented reality device.

The method may further comprise: requesting, by the augmented reality device, a lighting system controller configured to control the lighting device to perform the step of determining if the physical location corresponds to the location of the lighting device. The augmented reality device may request the lighting system controller whether it comprises a lighting device that is located at the physical location at which the light effect is to be provided. Additionally, the method may comprise: responding, if the physical location does not correspond to the location of the lighting device, by the lighting system controller to the augmented reality device, that the physical location does not correspond to the location of the lighting device, and responding, if the physical location corresponds to the location of the lighting device, by the lighting system controller to the augmented reality device, that the physical location corresponds to the location of the lighting device.

The method may further comprise: determining the light effect based on one or more properties of the virtual object. The light effect may, for example, be determined based on the appearance (e.g. the color) or the behavior (e.g. the movement) of the virtual object. Alternatively, the light effect may be a predefined (e.g. user-defined, system-defined, etc.) light effect.

The method may further comprise: obtaining the association between the virtual object and the light effect from a memory. The memory may be located in the augmented reality device, in the cloud, in a lighting system controller, etc.

The augmented reality device may be a head mounted augmented reality device. A head mounted augmented reality device may enable full immersion in the mixed reality that is created by controlling the physical lighting and/or virtual lighting.

According to a second aspect, the object is achieved by a computer program product for a computing device, the computer program product comprising computer program code to perform any of the above-mentioned methods when the computer program product is run on a processing unit of the computing device.

According to a third aspect, the object is achieved by a control system for controlling a lighting device and an augmented reality device, wherein the lighting device and the augmented reality device are located in a physical environment, the augmented reality device comprising a display configured to render a virtual environment as an overlay on the physical environment, the control system comprising one or more processors configured to: obtain a signal indicative of that a virtual object has entered the virtual environment, wherein the virtual object is associated with a light effect, determine a virtual location in the virtual environment where the virtual object has entered the virtual environment, determine a physical location in the physical environment that corresponds to the virtual location, determine if the physical location corresponds to a location of the lighting device based on obtained location information indicative of the location of the lighting device in the physical environment, and if the physical location corresponds to the location of the lighting device, control the lighting device to provide the light effect associated with the virtual object, and if the physical location does not correspond to the location of the lighting device, render a virtual light effect indicative of the light effect at the virtual location in the virtual environment on the display.

According to a fourth aspect, the object is achieved by an augmented reality device for use in the control system. The augmented reality device comprises a display configured to render a virtual environment as an overlay on the physical environment. The augmented reality device may comprise a processor configured to obtain a signal indicative of that a virtual object has entered the virtual environment, wherein the virtual object is associated with a light effect, determine a virtual location in the virtual environment where the virtual object has entered the virtual environment, and determine a physical location in the physical environment that corresponds to the virtual location.

Optionally, the processor of the augmented reality device may be configured to determine if the physical location corresponds to a location of the lighting device based on obtained location information indicative of the location of the lighting device in the physical environment. Alternatively, the processor of the augmented reality device may be configured to request a lighting system controller configured to control the lighting device to perform the step of determining if the physical location corresponds to the location of the lighting device.

The processor of the augmented reality device is further configured to render a virtual light effect indicative of the light effect at the virtual location in the virtual environment on the display if the physical location does not correspond to the location of the lighting device. Additionally, the processor of the augmented reality device may be configured to control the lighting device to provide the light effect associated with the virtual object. Alternatively, this control may be performed by the lighting system controller. It should be understood that the computer program product, control system and the augmented reality device may have similar and/or identical embodiments and advantages as the above-mentioned methods.

BRIEF DESCRIPTION OF THE DRAWINGS

The above, as well as additional objects, features and advantages of the disclosed systems, devices and methods will be better understood through the following illustrative and non-limiting detailed description of embodiments of devices and methods, with reference to the appended drawings, in which:

Fig. 1 shows schematically an example of a head-mounted augmented reality device for rendering a virtual object on a display as an overlay on a physical environment;

Fig. 2 shows schematically an example of an augmented reality device for rendering a virtual object as an overlay on a display; and

Fig. 3 shows schematically a method of controlling a lighting device and an augmented reality device.

All the figures are schematic, not necessarily to scale, and generally only show parts which are necessary in order to elucidate the invention, wherein other parts may be omitted or merely suggested.

DETAILED DESCRIPTION

Figs. 1 and 2 show schematically examples of augmented reality devices 120. In Fig. 1, the augmented reality device 120 is depicted as a head-mounted augmented reality device (e.g. augmented reality glasses), and in Fig. 2 the augmented reality device 120 is depicted as a hand-held augmented reality device (e.g. a smartphone or a tablet pc). A plurality of lighting devices 130, 132 and the augmented reality device 120 are located in a physical environment 140. The augmented reality device 120 comprises a display 122 configured to render a virtual environment 150 as an overlay on the physical environment 140. Fig. 2 further depicts a control system 102 (not shown in Fig. 1) for controlling (and being communicatively coupled to) the lighting devices 130, 132 and the augmented reality device 120. The control system 102 comprises one or more processors 106 (e.g. circuitry, microcontrollers, microchips). The control system 102 may further comprise one or more communication units 104 for communicating with the lighting devices 130, 132 and/or the augmented reality device 120. The one or more processors 106 are configured to: obtain a signal indicative of that a virtual object 152 has entered a virtual environment 150, wherein the virtual object 152 is associated with a light effect, determine a virtual location in the virtual environment where the virtual object 152 has entered the virtual environment, determine a physical location in the physical environment 140 that corresponds to the virtual location, determine if the physical location corresponds to a location of the lighting device based on obtained location information indicative of the location of the lighting device in the physical environment, and if the physical location corresponds to the location of the lighting device, control the lighting device to provide the light effect associated with the virtual object, and if the physical location does not correspond to the location of the lighting device, render a virtual light effect indicative of the light effect at the virtual location in the virtual environment on the display.

The control system 102 may comprise a single processor 106 for performing these steps. The processor 106 may, for example, be comprised in a (central) lighting control system (e.g. a bridge, a hub, a smartphone, etc.), in the augmented reality device 120, in a remote (cloud) server, etc. Alternatively, the control system 102 may comprise multiple processors 106 for performing these steps. The processors 106 may be located in different parts of a system 100, which system may comprise one or more lighting devices 130, 132, the control system 102 and/or the augmented reality device 120. The locations of the processors 106 and the steps performed by the respective processors 106 may depend on the system architecture of the control system 102 and/or the system architecture of the system 100. Examples thereof are explained below.

Figs. 1 and 2 further depict two lighting devices 130, 132 located in the physical environment 140. The lighting devices 130, 132 may be any type of lighting device comprising one or more (LED) light sources, and a processing unit for controlling the light output (e.g. hue, saturation and/or brightness) of the one or more light sources based on received control signals. The lighting devices 130, 132 may be arranged for providing general lighting, such as task lighting, ambient lighting, atmosphere lighting, accent lighting, indoor lighting, outdoor lighting, etc. The lighting devices 130, 132 may further comprise a communication unit (not shown) configured to receive lighting control commands (and, optionally, orientation control commands). The communication unit may comprise hardware for communicating via one or more wireless communication protocols, for example Bluetooth, Wi-Fi, Li-Fi, 3G, 4G, 5G or ZigBee. A specific communication technology may be selected based on the system architecture of the lighting system.

The augmented reality device 120 comprises a display 122 for rendering a virtual environment 150 comprising one or more virtual objects 152 as an overlay on a view of the physical environment 140. An example of the physical environment 140 is depicted in Fig. 2. The depicted physical environment 140 comprises a couch, a tv and two lighting devices 130, 132. The display 122 may be a (semi-) transparent see-through display, wherein the user can see the physical environment 140 through the display 122, and wherein the display 122 is configured to render the virtual environment 150 comprising one or more virtual objects 152 as an overlay on the physical environment 140. The display 122 may be integrated in the (semi-) transparent see-through display, or the virtual environment may be projected on the display 122. Alternatively, augmented reality device 120 may comprise a camera 124 configured to continuously capture images of the physical environment 140 and render the images on the display 122, while rendering the virtual environment comprising one or more virtual objects 152 as an overlay on the images. It should be understood that such augmented reality devices are known in the art, and will therefore not be discussed in further detail.

The augmented reality device 120 may comprise a processor 106 configured to render, on the display 122, the virtual environment as an overlay on the physical environment 140. The lighting devices 130, 132 and the augmented reality device 120 are located in the same physical environment 140 (e.g. a room such as a living room, an office, etc.). The user operating the augmented reality device 120 is also located in the physical environment 140. By rendering the virtual environment as an overlay on the physical environment 140, a so- called mixed reality environment is created, wherein the user can see both physical (real-life) objects and virtual objects. The augmented reality device 120 may comprise a processor configured to determine how to render the virtual environment as an overlay on the physical environment. The processor may, for example, determine to render the virtual environment as an overlay on the physical environment 140 based on a location and/or an orientation of the augmented reality device 120. The processor 106 may, for example, map the virtual environment onto the physical environment. The processor 106 may, for example, use image analysis to analyze an image of the physical environment 140, and select one or more anchor points in the physical environment 140 based on the image analysis. The processor 106 may then anchor virtual objects (e.g. virtual characters, virtual furniture, a virtual space, etc.) of the virtual environment to the physical environment 140 based on the anchor points. Alternatively, the processor may be configured to obtain a predefined mapping of the virtual environment onto the physical environment 140. The mapping may, for example, be based on the location and the orientation of the augmented reality device 120 relative to the physical environment 140. Such techniques for mapping a virtual environment onto a physical environment are known in the art and will therefore not be discussed in further detail.

The one or more processors 106 (which may for example be comprised in the augmented reality device 120 or in the cloud) are configured to obtain a signal indicative of that a virtual object 152 has entered the virtual environment. The virtual object 152 may, for example, be an avatar of another user, which avatar enters the virtual environment. Alternatively, the virtual object 152 may be a virtual (game) character. Alternatively, the virtual object 152 may be a virtual object that appears in the virtual environment (e.g. a virtual vehicle, an environmental object, etc.). The virtual object 152 is associated with a light effect. The one or more processors 106 are configured to obtain the light effect, for instance from the augmented reality device 152, from a remote server, from a (game/software) application running on the augmented reality device 152, etc. The light effect may be a predefined light effect, and the one or more processors 106 may be configured to access the memory to retrieve the predefined light effect. Alternatively, the light effect may be defined by a user (e.g. the user operating the augmented reality device 152, a user whose avatar is the virtual object 152). Alternatively, the one or more processors 106 may be configured to determine properties of the light effect, which may for example be based on one or more properties of the virtual object. The light effect may, for example, be determined based on the appearance (e.g. the color) or the behavior (e.g. the movement) of the virtual object 152.

The one or more processors 106 are configured to determine a virtual location in the virtual environment where the virtual object has entered the virtual environment and determine a physical location in the physical environment 140 that corresponds to the virtual location. The one or more processors 106 may, for example, be configured to obtain information about the virtual location at which the virtual object 152 enters the virtual environment. The one or more processors 106 may then determine which physical location corresponds to that virtual location in the physical environment 140. For example, the virtual object 152 may enter the virtual environment on the left side of the field of view 160 of the augmented reality device 120 (as shown in Fig. 2). The one or more processors 106 may then select a physical location that corresponds to the virtual location, for instance based on the location and/or the orientation of the augmented reality device 120 in the physical environment 140. Additionally or alternatively, the one or more processors 106 may determine which physical location corresponds to the virtual location in the physical environment 140 based on a mapping of the virtual environment onto the physical environment 140. For example, the virtual object 152 may enter the virtual environment on at a set of coordinates or at an anchor point in the virtual environment, which corresponds to a set of coordinates or an anchor point in the physical environment 140. The one or more processors 106 may then determine the physical location in the physical environment that corresponds to the virtual location, and thereby the physical location of the virtual object 152.

The one or more processors 106 are configured to obtain location information indicative of the location of one or more lighting devices 130, 132 relative to the augmented reality device and/or relative to the physical environment 140. Additionally or alternatively, the one or more processors 106 may be configured to determine the location and/or the orientation of the augmented reality device 120 relative to the physical environment 140. The one or more processors 106 may be configured to receive location information indicative of the relative location/orientation of the augmented reality device 120 and/or the one or more lighting devices 130, 132 relative to the physical environment 140. The location information may, for example, be received from an (indoor) positioning system (such as an RF-based indoor positioning system or a visible light communication (VLC) based positioning system), it may be based on the signal strength of signals communicated between one or more lighting devices 130, 132 and the augmented reality device 120, based on light signals communicated between them, etc. The locations may be indicative of coordinates of the one or more lighting devices 130, 132 and the augmented reality device 120 relative to the physical environment 140. Alternatively, the one or more processors 106 may be configured to obtain the location and/or the orientation of the augmented reality device 120 relative to the physical environment 140 by analyzing the field of view (e.g. field of view 160). The one or more processors 106 may for example be configured to analyze one or more images captured by a camera 124 of the augmented reality device to determine the locations of the lighting devices relative to the physical environment 140 and relative to the augmented reality device 120. The one or more processors 106 may, for example, use image analysis techniques for identifying objects (e.g. lighting devices) in the physical environment 140 to determine their locations. Additionally, one or more depth cameras and/or depth sensors may be used to detect the dimensions of the physical environment 140. Techniques of obtaining location information of devices in an environment are known in the art and will therefore not be discussed in further detail.

The one or more processors 106 are further configured to determine if the physical location corresponds to the location of a lighting device based on obtained location information indicative of the location of the lighting device in the physical environment 140. The one or more processors 106 may compare the physical location (of the virtual object 152) to the location of a lighting device and determine if the physical location (substantially) corresponds to the location of the lighting device based on the comparison. The one or more processors 106, may, for example, determine if the physical location is located within a threshold distance from the location of the lighting device. Referring to the example of Figs. 1 and 2, the one or more processors 106 may determine that the location of lighting device 130 corresponds to the location of the virtual object 152. The augmented reality device 120 may comprise one or more processors 106 to perform the determination. Alternatively, the augmented reality device 120 may request a lighting system controller (e.g. a bridge, a smartphone, a central (home) control system, etc.) to perform the step of determining if the physical location corresponds to the location of the lighting device. One or more processors 106 of the lighting system controller may be configured to store the location information indicative of the locations of the one or more lighting devices 130, 132 relative to the physical environment 140, and respond to the augmented reality device 120 whether the physical location corresponds to the location of the lighting device or not.

The one or more processors 106 are further configured to control, if the physical location corresponds to the location of the lighting device, the lighting device to provide the light effect 131 associated with the virtual object. The one or more processors 106 may, for example, retrieve the light effect associated with the virtual object 152 from a memory (e.g. a memory comprised in the control system 102, a memory comprised in the augmented reality device 120, a memory located in the cloud, etc.). The one or more processors 106 may, for example, communicate lighting control commands to the lighting devices 130, 132 to provide the light effect in the physical environment 140. The control system 102 may comprise a communication unit configured to communicate the lighting control commands the lighting devices 130, 132. A lighting control command may comprise lighting control instructions for controlling the light output, such as the color, intensity, saturation, beam size, beam shape, etc. of one or more light sources of a lighting device. Referring to the example of Figs. 1 and 2, the one or more processors 106 may determine that the location of lighting device 130 corresponds to the location of the virtual object 152. Consequently, the one or more processors 106 may control lighting device 130 such that it renders a light effect 131 that corresponds to the light effect associated with the virtual object 152. The light effect may, for example, be a flash of light of a predefined color, or a dynamic light effect of which the color changes over time. The one or more processors 106 are further configured to render, if the physical location does not correspond to the location of the lighting device, a virtual light effect indicative of the light effect at the virtual location in the virtual environment on the display 122. The one or more processors 106 may, for example, retrieve the light effect associated with the virtual object 152 from a memory (e.g. a memory comprised in the control system 102, a memory comprised in the augmented reality device 120, a memory located in the cloud, etc.). The one or more processors 106 may then render the virtual effect 151 as an overlay on the physical environment 140. The light effect may, for example, be a flash of light of a predefined color, or a dynamic light effect of which the color changes over time, which is rendered as the overlay on the display 122.

The one or more processors 106 may be further configured to obtain type information indicative of the type of the lighting device, and determine if the lighting device is capable of providing the light effect based on the type of the lighting device. The one or more processors 106 may be configured to receive the type information from the lighting devices, from a memory (e.g. a local or a remote memory), from a lighting system controller, etc. If the physical location corresponds to the location of the lighting device and if the lighting device is capable of providing the light effect, the one or more processors 106 may control the lighting device to provide the light effect associated with the virtual object 152. The lighting device may, for example, be of a type configured to provide colored light. If the light effect associated with the virtual object 152 is a colored light effect, the one or more processors 106 may control the lighting device accordingly. If the physical location corresponds to the location of the lighting device and if the lighting device is incapable of providing the light effect, the one or more processors 106 may render the virtual light effect at the virtual location in the virtual environment on the display 122. The lighting device may, for example, be of a type configured to provide white light (and not colored light). If the light effect associated with the virtual object 152 is a colored light effect, the one or more processors 106 may refrain from controlling the lighting device, and render the virtual light effect at the virtual location in the virtual environment on the display 122. In other examples, the type of lighting device may result in that the lighting device is capable or incapable due to other light characteristics of the lighting device, for instance due to a beam size, beam shape, saturation range, intensity range, etc. For instance, linear pixelated devices like light strips, only light sources close to the virtual object may be used to provide the light effect.

The one or more processors 106 may be further configured to obtain location information indicative of locations of a plurality of lighting devices 130, 132 in the physical environment 140 and compare the locations of the plurality of lighting devices to the physical location (which corresponds to the virtual location of the virtual object 152). The one or more processors 106 may be further configured to select the lighting device from the plurality of lighting devices based on the comparison. The location information may, for example, be received from an (indoor) positioning system (such as an RF-based indoor positioning system or a visible light communication (VLC) based positioning system), it may be based on the signal strength of signals communicated between one or more lighting devices 130, 132 and the augmented reality device 120, based on light signals communicated between them, etc. referring to Figs. 1 and 2, the one or more processors 106 may obtain location information indicative of the locations of the lighting devices 130, 132, and select lighting device 130 because its location corresponds to the location of the virtual object 152.

The light effect associated with the virtual object 152 may have a predefined size (e.g. a certain diameter, a predefined shape, etc.). The one or more processors 106 may be configured to determine a physical area in the physical environment 140 that corresponds to the predefined size of the light effect. The size of the light effect may be defined as dimensions (e.g. 2d or 3d) of the light effect. The one or more processors 106 may determine the physical area (with respect to the physical location of the light effect) based on the dimensions of the light effect. Based thereon, the one or more processors 106 may determine that the lighting device is located in the physical area and control the lighting to provide the light effect associated with the virtual object 152.

The light effect may be a dynamic light effect that moves from the virtual location to one or more subsequent virtual locations. The one or more processors 106 may be further configured to determine one or more subsequent physical locations in the physical environment 140 that correspond to the one or more subsequent virtual locations (in a similar way as determining if the physical location corresponds to the location of the lighting device, as described above). The one or more processors 106 may be further configured to obtain location information indicative of locations of one or more further lighting devices in the physical environment and determining if one or more further lighting devices are located at the one or more subsequent physical locations. If the one or more subsequent physical locations correspond to the locations of the one or more further lighting devices, the one or more processors 106 may control the one or more further lighting devices to provide the light effect at the subsequent physical location when the dynamic light effect has moved to the one or more subsequent virtual locations. Alternatively, if the one or more subsequent physical locations do not correspond to the locations of the one or more further lighting devices, the one or more processors 106 may render the virtual light effect at the one or more subsequent virtual locations in the virtual environment on the display 122 when the dynamic light effect has moved to the one or more subsequent virtual locations. For example, referring to Figs. 1 and 2, the light effect may be a light effect that moves from an initial location (e.g. the location of the virtual object 152 as shown in the image) to a subsequent location (e.g. a location at the right side of the physical environment 140). Based thereon, the one or more processors 106 may first control lighting device 130 and subsequently lighting device 132 to provide the dynamic light effect. In another example, the light effect may be a light effect that moves from an initial location (e.g. the location of the virtual object 152 as shown in the image) to a first subsequent location (e.g. the ceiling) and thereafter to a second subsequent location (e.g. a location at the right side of the physical environment 140). Based thereon, the one or more processors 106 may first control lighting device 130, then render a virtual light effect as an overlay on the display 122 at the first subsequent location (at the ceiling) and thereafter control lighting device 132 to provide the dynamic light effect.

The one or more processors 106 may be further configured to determine if the physical location is visible on the display 122. It may occur that the virtual object 152 has entered the virtual environment at a location of which the physical location is not visible to the user, for instance because it is outside the field of view 160 of the augmented reality device 120. If this is the case, the one or more processors 106 may extend a size of the light effect and/or reposition the light effect to another virtual location such that the light effect becomes visible on the display 122 (e.g. such that the light effect is located the field of view of the augmented reality device 120).

Fig. 3 shows schematically an example of a method 300 of controlling a lighting device and an augmented reality device, wherein the lighting device and the augmented reality device are located in a physical environment, the method comprising: rendering 302, on a display of the augmented reality device, a virtual environment as an overlay on the physical environment, obtaining 304 a signal indicative of that a virtual object has entered the virtual environment, wherein the virtual object is associated with a light effect, determining 306 a virtual location in the virtual environment where the virtual object has entered the virtual environment, determining 308 a physical location in the physical environment that corresponds to the virtual location, determining 310 if the physical location corresponds to a location of the lighting device based on obtained location information indicative of the location of the lighting device in the physical environment, and if the physical location corresponds to the location of the lighting device, controlling 312 the lighting device to provide the light effect associated with the virtual object, and if the physical location does not correspond to the location of the lighting device, rendering 314 a virtual light effect indicative of the light effect at the virtual location in the virtual environment on the display.

The method 300 may be executed by computer program code of a computer program product when the computer program product is run on a processing unit of a computing device, such as the one or more processors 106 of the control system 102.

It should be noted that the above-mentioned embodiments illustrate rather than limit the invention, and that those skilled in the art will be able to design many alternative embodiments without departing from the scope of the appended claims.

In the claims, any reference signs placed between parentheses shall not be construed as limiting the claim. Use of the verb "comprise" and its conjugations does not exclude the presence of elements or steps other than those stated in a claim. The article "a" or "an" preceding an element does not exclude the presence of a plurality of such elements. The invention may be implemented by means of hardware comprising several distinct elements, and by means of a suitably programmed computer or processing unit. In the device claim enumerating several means, several of these means may be embodied by one and the same item of hardware. The mere fact that certain measures are recited in mutually different dependent claims does not indicate that a combination of these measures cannot be used to advantage.

Aspects of the invention may be implemented in a computer program product, which may be a collection of computer program instructions stored on a computer readable storage device which may be executed by a computer. The instructions of the present invention may be in any interpretable or executable code mechanism, including but not limited to scripts, interpretable programs, dynamic link libraries (DLLs) or Java classes. The instructions can be provided as complete executable programs, partial executable programs, as modifications to existing programs (e.g. updates) or extensions for existing programs (e.g. plugins). Moreover, parts of the processing of the present invention may be distributed over multiple computers or processors or even the ‘cloud’. Storage media suitable for storing computer program instructions include all forms of nonvolatile memory, including but not limited to EPROM, EEPROM and flash memory devices, magnetic disks such as the internal and external hard disk drives, removable disks and CD-ROM disks. The computer program product may be distributed on such a storage medium, or may be offered for download through HTTP, FTP, email or through a server connected to a network such as the Internet.

Claims

CLAIMS:

1. A method (300) of controlling a lighting device and an augmented reality device, wherein the lighting device and the augmented reality device are located in a physical environment, the method comprising: rendering (302), on a display of the augmented reality device, a virtual environment as an overlay on the physical environment, obtaining (304) a signal indicative of that a virtual object has entered the virtual environment, wherein the virtual object is associated with a light effect, determining (306) a virtual location in the virtual environment where the virtual object has entered the virtual environment, determining (308) a physical location in the physical environment that corresponds to the virtual location, determining (310) if the physical location corresponds to a location of the lighting device based on obtained location information indicative of the location of the lighting device in the physical environment, and if the physical location corresponds to the location of the lighting device, controlling (312) the lighting device to provide the light effect associated with the virtual object, and if the physical location does not correspond to the location of the lighting device, rendering (314) a virtual light effect indicative of the light effect at the virtual location in the virtual environment on the display.

2. The method (300) of claim 1, wherein the method (300) further comprises: obtaining type information indicative of the type of the lighting device, determining if the lighting device is capable of providing the light effect based on the type of the lighting device, and if the physical location corresponds to the location of the lighting device and if the lighting device is capable of providing the light effect, controlling the lighting device to provide the light effect associated with the virtual object, and if the physical location corresponds to the location of the lighting device and if the lighting device is incapable of providing the light effect, rendering the virtual light effect at the virtual location in the virtual environment on the display.

3. The method (300) of any preceding claim, wherein the method (300) further comprises: obtaining location information indicative of locations of a plurality of lighting devices in the physical environment, comparing the locations of the plurality of lighting devices to the physical location, and selecting the lighting device from the plurality of lighting devices based on the comparison.

4. The method (300) of any preceding claim, wherein the light effect has a predefined size, and wherein the method (300) comprises: determining a physical area in the physical environment that corresponds to the predefined size of the light effect, and wherein the step of determining if the physical location corresponds to the location of the lighting device comprises: determining if the lighting device or its light output is located in the physical area.

5. The method (300) of any preceding claim, wherein the light effect is a dynamic light effect that moves from the virtual location to a subsequent virtual location, wherein the method (300) further comprises: determining a subsequent physical location in the physical environment that corresponds to the subsequent virtual location, obtaining location information indicative of locations of one or more further lighting devices in the physical environment, determining if one or more further lighting devices are located at the subsequent physical location, and if the subsequent physical location corresponds to the location of the one or more further lighting devices, controlling the one or more further lighting devices to provide the light effect at the subsequent physical location when the dynamic light effect has moved to the subsequent virtual location, and if the subsequent physical location does not correspond to the location of the one or more further lighting devices, rendering the virtual light effect at the subsequent virtual location in the virtual environment on the display when the dynamic light effect has moved to the subsequent virtual location.

6. The method (300) of any preceding claim, further comprising: determining if the physical location is visible on the display, and if the physical location is not visible on the display, extending a size of the light effect and/or repositioning the light effect to another virtual location such that the light effect becomes visible on the display.

7. The method (300) of any preceding claim, further comprising: requesting, by the augmented reality device, a lighting system controller configured to control the lighting device to perform the step of determining if the physical location corresponds to the location of the lighting device.

8. The method (300) of claim 7, wherein the method (300) further comprises: if the physical location does not correspond to the location of the lighting device, responding, by the lighting system controller to the augmented reality device, that the physical location does not correspond to the location of the lighting device, and if the physical location corresponds to the location of the lighting device, responding, by the lighting system controller to the augmented reality device, that the physical location corresponds to the location of the lighting device.

9. The method (300) of claim 1, wherein the virtual object is an avatar of a user or a virtual character.

10. The method (300) of any preceding claim, wherein the method (300) further comprises: determining the light effect based on one or more properties of the virtual object.

11. The method (300) of any preceding claim, further comprising: obtaining the association between the virtual object and the light effect from a memory.

12. The method (300) of any preceding claim, wherein the augmented reality device is a head mounted augmented reality device.

13. A control system (102) for controlling a lighting device (130, 132) and an augmented reality device (120), wherein the lighting device and the augmented reality device are located in a physical environment (140), the augmented reality device comprising a display (122) configured to render a virtual environment (150) as an overlay on the physical environment, the control system (102) comprising one or more processors (106) configured to: obtain a signal indicative of that a virtual object (152) has entered the virtual environment, wherein the virtual object is associated with a light effect, determine a virtual location in the virtual environment where the virtual object has entered the virtual environment, determine a physical location in the physical environment that corresponds to the virtual location, determine if the physical location corresponds to a location of the lighting device based on obtained location information indicative of the location of the lighting device in the physical environment, and if the physical location corresponds to the location of the lighting device, control the lighting device to provide the light effect associated with the virtual object, and if the physical location does not correspond to the location of the lighting device, render a virtual light effect indicative of the light effect at the virtual location in the virtual environment on the display.

14. A computer program product for a computing device, the computer program product comprising computer program code to perform the method (300) of any of claims 1- 12 when the computer program product is run on a processing unit of the control system of claim 13.