WO2022104399A2

WO2022104399A2 - System and method to control smart lighting in the augmented reality environment by using "look-to-link" protocol with camera

Info

Publication number: WO2022104399A2
Application number: PCT/VN2021/000021
Authority: WO
Inventors: Duy Thong Nguyen; Van Trang NGUYEN; Minh Duc THIEU
Original assignee: Anh Sang So Huepress Joint Stock Company
Priority date: 2020-11-13
Filing date: 2021-11-05
Publication date: 2022-05-19
Also published as: WO2022104399A3

Abstract

This invention discloses a novel light devices selection method to control using a "look-to-link" protocol with a camera to provide an augmented reality (AR) user experience. It applies Light Communication to bring an AR experience for controlling lights. User does not need to remember the buttons or any information of the lights (such as long names). He controls the lights he sees. The proposed Light Communication method uses both visible light and IR light. Visible light is used for communications when the light device is ON. When the light device is OFF, it uses IR for communications. Novel modulation schemes are proposed, including C-FSK and C-OOK-2. Techniques for these modulation schemes, including dimming, clock rate switching, support for time- variant frame rate cameras, color control, and system security, etc., are specified. Implementation results are also provided to prove the concept.

Description

SYSTEM AND METHOD TO CONTROL SMART LIGHTING IN THE AUGMENTED REALITY ENVIRONMENT BY USING “LOOK-TO-LINK” PROTOCOL WITH CAMERA

Field of invention:

This invention relates to the field of smart lighting and wireless technologies (WiFi plus Light Communication). Especially, it discloses a novel light devices selection method to control them by using a “look-to-link” protocol with a camera. This aims to provide an augmented reality user experience. Both visible light and infrared (IR) light are used for Light Communication with a camera.

Technical Statement:

This invention is the combination of smart lighting technology and an Optical Wireless Communication (also known as Light Communication); thus, the statement of related works will consist of (1) the statement of smart lighting in the market, (2) the statement of optical wireless communication/light communication to smartphone via camera, and (3) the method to secure the right to control lighting devices.

(1) Nowadays, the lighting system that has many light sources requires user to remember the identity (ID) and buttons of every single light source in order to control them independently. Furthermore, the number of buttons in an application (App) will be the same as the number of light sources within the system. This typical lighting system or typical smart lighting system, however, is inconvenient to user as it enforces user to remember the information of every single light sources.

(2) The light communication method to deliver data to smartphone camera is also known as Optical Camera Communication (OCC). It is an emerging wireless technology that shares the burden of wireless connectivity with the traditional radio wireless technologies (such as WiFi/Bluetooth, etc.). Especially, the OCC technology is considered as the trend of the internet of things (loT) using visible light, which is friendly to environment and saves energy as it uses existing lighting infrastructure. The camera-based method, in which a camera simultaneously achieves to goals: record video to capture the reality, and receive the digital data encoded by visible light to capture the virtuality. Together, it provides a new user experience, which is the experience that combines reality and virtuality with digital information. It is called augmented reality (AR) user experience.

Up to date, several existing OCC methods and systems can be highlighted as:

• The FSK-based light modulation methods include two schemes from Kookmin University and National Taiwan University (NTU) [1], Both are within the IEEE 802.15.7-2018 Optical Wireless Communication standard, and respectively known as CM-FSK and RS-FSK schemes. Among them, RS-FSK scheme (by NTU) only supports a fixed 30fps camera and it does not support camera that has a time-variant frame rate. Also, the RS-FSK scheme has only two modes, including 8-FSK and 16-FSK. The CM-FSK can support a time-variant frame rate camera; however, it costs a half of spectral efficiency reduction for information transmission. Both existing schemes within IEEE 802.15.7-2018 standard, which target a reasonable data rate but they are well compatible with commercial cameras (due to the frame rate variation issues).

• The FSK modulation method from the Carnegie Mellon University, Pittsburgh, USA [2] uses the bandwidth of 8 kHz and the frequency separation of 0.2 kHz. This method cannot support a time-variant frame rate camera and it is not based on any IEEE standard; thus, it cannot support App- based configuration using PHY PIB attributes.

• Lastly, the method from PureVLC, Edinburgh, UK [3] uses ON-OFF-Keying (OOK) instead of FSK. However, similar to all other OOK methods, the disadvantage of OOK is that it cannot control the brightness without reducing SNR (because it must reduce the OOK signal amplitude to control the brightness) and thus, it cannot support color change at a high-resolution.

Furthermore, all existing OCC schemes have common critical disadvantages:

(i) either cannot support a time-variant frame rate camera available on the market, or

(ii) do not follow any or be recognized by any IEEE standard with App-based configuration using PHY PIB attributes, or (iii) are not specifically designed for smart lighting applications because there is no commercial product available at this moment. For example, existing OCC schemes are not optimal for transmitting ID by the color/brightness changeable light sources. Also, to the best of our knowledge, the combination of IR and visible light for Light Communication (particularly OCC) is a new technology in the field of smart lighting and smart loT control.

(3) The method to secure connectivity is the priority requirement in loT because if an anonymous device can access to your internet, it can potentially control your lights. In fact, no existing encryption method can provide such an absolute secure. We suggest using multiple-layer security protection by combining WiFi, OCC, and application authorization so that it solves the “absolute secure” problem in the context of lighting control.

Technical philosophy of invention

This invention relates to the field of smart lighting and telecommunication. Especially, it discloses a novel light source selection method to control using a “look-to-link” protocol with a camera to provide an augmented reality user experience (therefore, the camera becomes an AR-camera). The wireless technology used is a combination of WiFi (or any other similar RF tech like Bluetooth/Zigbee,...) and OCC. The AR- camera allows user selecting the light devices and controlling them in a novel and convenient manner.

In this invention, the ID-signaling scheme via light wave to a camera allows the camera identifying light devices. The light device consists of visible light LEDs, and IR LEDs. Both visible light and IR light are used to transmit data, depending on the cases of lighting. The proposed “look-to-link” protocol with camera (or the smart lights AR controlling procedure) includes the following steps: i) App allocates network-ID (net-ID) for the new light devices as follows • Light devices that do not have net-ID and connect to application (App) for the first time will send a message to app to request for net-ID;

• App receives the requests from new light devices;

• App counts the number of new light devices that are requesting net-ID;

• App generates net-ID according to the number of light devices requesting. After being connected, app can send command to light devices. If there is no command from App, light devices will switch into “sleep mode” to save energy; ii) App requests light devices to send their net-ID over light wave as follows:

• App sends “wakeup” command to the light devices of interest via WiFi before sending any other specific control commands.

• And then, App sends “request net-ID broadcasting” to light devices to request them broadcasting their net-IDs via OCC (using C-FSK modulation);

• After receiving the request, a light device modulates its net-ID based upon two cases as follows

Case 1 : Light is currently ON

The light device uses the modulation scheme called camera-FSK modulation scheme (C-FSK) to modulate its net-ID, and then the signal is sent over the visible light LEDs.

It can also uses modulation scheme called camera-OOK-version 2 (C-OOK-2) to modulate its net-ID, and then sends the signal over the infrared LEDs.

Case 2: Light is currently OFF

The light device uses the modulation scheme C-OOK-2 to modulate its net-ID, and then sends the signal over the infrared LEDs.

• Light device sends an acknowledgement frame to App to inform that it is now sending net-ID and it uses a specific modulation scheme C- FSK/C-OOK-2. iii) App selects light device(s) in a look-to-link protocol as follows:

• App interacts with user via camera preview in the AR-camera context to select a light device (or a group of lights) that user wants to control;

• App captures a set of images continuously (each image must capture the light devices so that it can receive the signals), and then demodulate the signal using the C-FSK/C-OOK-2 scheme from every single image;

• After demodulation, App lookups the IP (IP within the WiFi network) of the selected lights according to the net-ID they are broadcasting for later controlling; iv) App controls the color/brightness of lights as follows • App allows user selecting a color, a brightness, or a color temperature;

• App sends the “request new color” command to the selected light devices via WiFi to change their brightness;

• If the “absolute secure” mode is enabled by App, this controlling command must include the security key that App receives from the C- FSK/C-OOK-2 demodulation;

• The light devices receive the “request new color” command, and then they may check the security key if the “absolute secure” mode is currently enabled;

• The light devices apply PWM to adjust their pulse width at individual LEDs (red/green/blue and white if available) according to the color requested by App. v) App controls the power level of IR-LEDs as follows

• App allows user to select the brightness level of IR (or the dimming level of IR), and then sends the request to light devices

• Light devices use the proposed Algorithms (Algorithms 1 and 2) to generate a line code mapping rule based on the selected dimming level.

Figure description

Figure la (top) illustrates the steps of the “look-to-link and control” protocol using AR-camera and a light device

Figure lb (bottom) shows the components of a light device, including RGB (or RGBW) and IR LEDs.

Figure 2 shows the net-ID configuration procedure for new light devices in the smart lighting system.

Figure 3 shows the procedure of modulating the light wave (both cases of using IR and visible light)

Figure 4 shows the steps of the “look-to-link” and control protocol at the AR-camera side

Figure 5 shows Algorithm 1 - Algorithm to generate BwB LUT code table with different options of dimming parameter p.

Figure 6 shows the procedure of C-OOK-2 modulation Figure 7 shows Algorithm 2 - Algorithm to generate the C-OOK-2 mapping rule for the data packet from the given mBnB LUT code table

Figure 8 shows the procedure of C-FSK modulation

Figure 9 (top) shows 2B4B LUT as a result of Algorithm 1 in the case m =2, M=4

Figure 9 (bottom) shows 2B4B mapping rule for the data packet as a result of Algorithm 2 in the case m =2, n=4

Figure 10 describes the C-OOK-2 demodulation algorithm (Algorithm 3) by using a Matched Filter

Figure 11 shows the implementation results of C-OOK-2 transmission using 2B4B Code (BER results versus SNR, for different p)

Figure 12 shows the implementation results of C-OOK-2 transmission using 3B8B Code (BER results versus SNR, for different p)

Figure 13 shows the implementation results of C-OOK-2 transmission using 4B16B Code (BER results versus SNR, for different )

Figure 14 lists the PHY PIB attributes and their values used for C-OOK-2 modulation

Figure 15 shows Algorithm 4 - Asynchronous encoding algorithm for C-FSK using 7V-frequencies

Figure 16 shows Algorithm 5 - Asynchronous decoding algorithm for C-FSK using n-frequencies

Figure 17 shows a mapping table for C-FSK with an example of A O frequencies

Figure 18 lists the PHY PIB attributes and their values used for C-FSK modulation

Figure 19 illustrates the methods of security to control the light device from App Details of Invention

In this invention, the ID-signaling scheme over light wave to camera allows camera identifying light devices. Both visible light and infrared (IR) light are used, depending on the cases of lighting. The proposed “look-to-link” protocol with camera and the smart lights controlling procedure include the following steps: i) App allocates network-ID (net-ID) for the new light devices; ii) App requests light devices to send their net-ID over light medium; iii) App uses AR-camera to select light(s) in a look-to-link protocol; iv) App controls the color/brightness of lights; v) App controls the power level of IR-LED.

Figure 1 illustrates the steps of the look-to-link-and control procedure.

Further details of the procedure are explained as follows: i) App allocates network-ID (net-ID) for the new lights as follows

• Light devices that do not have net-IDs and connect to application (App) for the first time will send a message to App to request for net-ID;

• App receives the requests from new lights;

• App counts the number of new lights that are requesting net-ID;

• App generates net-ID according to the number of lights requesting. After being connected, app can send command to lights. If there is no command from App for a certain of time, lights will switch into “sleep mode” to save energy; ii) App requests light devices to send their net-IDs over light wave as follows:

• App sends “wakeup” command to the lights of interest via WiFi before sending any other specific control commands;

• And then, App sends “request net-ID broadcasting” to light devices to request them broadcasting their IDs via OCC (using C-FSK modulation);

• After receiving the request, a light device modulates its net-ID based upon two cases as follows:

Case 1 : Light is currently ON

It can also uses modulation scheme called camera-OOK-version 2 (C-OOK-2) to modulate its net-ID, and then sends the signal over the infrared LEDs;

Case 2: Light is currently OFF The light device uses the modulation scheme C-OOK-2 to modulate its net-ID, and then sends the signal over the infrared LEDs.

• Light device sends an acknowledgement frame to App to inform that it is now sending net-ID and it uses a specific modulation scheme C- FSK/C-OOK-2. iii) App uses camera to select light(s) in a look-to-link protocol as follows:

• App interacts with user via camera preview in the AR-camera context to select a light (or a group of light devices) that user wants to control;

• App captures a set of images continuously (each image must capture the light devices), and then demodulate the signal using the C-FSK/C-OOK- 2 scheme from every single image;

• After demodulation, App lookups the IP of the selected light devices according to the net-ID they are broadcasting for later controlling; iv) App controls the color/brightness of light devices as follows

• App allows user selecting a color, the brightness of color, or the color temperature;

• App sends the “request new color” command to the selected lights via WiFi to change their brightness;

• The light devices apply PWM to adjust their pulse width at red/green/blue (and white if available) channels according to the color requested by App. v) App controls the power level of IR-LED as follows

• App allows user to select the brightness level of IR (or the dimming level of IR), and then sends the request to lights;

The first step of the procedure is to allocate net-ID for the new light devices that are connecting to the App for the first time. After that, light devices and App are connected. Light devices can go to “sleep mode” to save energy. App sends the “wakeup” command via WiFi (t210) to the interested lights. This command wakeups the interested lights from sleeping mode.

The procedure to allocate net-ID to new lights (t240) include: i) First, new light devices will need the net-ID to connect with an App (the case Y of 210). These new lights broadcast a connection request command via WiFi to app (t220);

The connection request command (t220) includes the physical ID of the light device, which is unique for identification by the app. This physical ID is different from the net-ID. The broadcasting protocol of (t220) must follow a collision avoidance (220) in case there are two or more lights that simultaneously send the connection requests. ii) App listens the number of connection requests to count the number of new lights (230);

Based on the number of requests, app will generate individual net-IDs (240) and send them to the corresponding lights those sent requests (t230). The net-ID is as short as it can (typically 1 byte as a default value) to minimize the length but still guarantee the identification of light devices. In the case of many lights, the length of net-ID is also customizable according to the version of the WiFi command.

Figure 3 provides the procedure that an App requests light devices to broadcast their net-ID. In a normal condition without any interaction, the communication module of light device is “sleep” (310) to save energy. App will send a “Request wakeup” (t300) to wake the light ready for the next steps.

After wakeup (320), a light device that receives a “request net-ID broadcasting” from App (t310) will modulate the data (including net-ID and an optional security information) to send over LEDs (IR LEDs or visible light LEDs). There are two cases of light device (330) as follows:

Case 1 : Light is currently ON

It can also uses modulation scheme called camera-OOK-version 2 (C- OOK-2) to modulate its net-ID, and then sends the signal over the infrared LEDs.

Case 2: Light is currently OFF

The light device uses the modulation scheme C-OOK-2 to modulate its net-ID, and then sends the signal over the infrared LEDs. C-OOK-2 modulation scheme uses a line code BwB that is generated by the proposed algorithms 1 and 2.

Accordingly, the signal can be sent over visible light (t320) or IR light (t330).

Note that FSK modulation (particularly C-FSK) is selected for modulating visible light because the visible light requires color change and dimming support at a high resolution. This is crucial to the smart light. In contrast, IR light does not request dimming support, or at least does not require a high resolution of dimming. That’s why OOK modulation (particularly C-OOK-2) is chosen for IR as it provides a higher transmission rate than that of FSK.

The optical clock rate for the modulation must be considered carefully. For a short range of transmission such as several meters, C-OOK-2 uses a kHz level clock rate (10 kHz for example) so that the entire data packet (a preamble plus payload) can be captured by an image. For a long range of transmission where camera can not capture the entire data packet within an image, C-OOK-2 uses a Hz level clock rate (10 Hz for example). This Hz-level clock rate is low but it can be used because IR is imperceptible to human eyes. In this case, camera needs at least one pixel that records the intensity change of the light signal, and then it will demodulate the data packet from multiple images.

Figure 4 provides the procedure of the select-and-control protocol via AR-camera.

Light device sends an ack frame (t420) to inform App that it is now broadcasting its net-ID using what type of modulation.

At first, App opens User Interface to interact with user in an AR mode (with camera is previewing the real world), where user toughs the screen to select one or several lights/groups of lights (410) that he wants to control.

After that, App captures the picture of these lights, and then demodulate the light modulated signal (C-FSK signal or C-OOK-2 signal) (411) to extract the net-ID of lights so that app can connect to them.

Each captured image will correspond to one frequency after the demodulation; thus, the SNR is high compared to OOK at the same transmission distance. Hence, C-FSK allows for transmitting the signal further.

Finally, user can select a color/brightness/temperature of color (430) and send a color command (t430) to the selected lights.

Light device uses PWM dimming method (430, 440) to update its color as selected.

Figure 5 describes details of Algorithm 1 to produce a mB/?B LUT Code Table that contains different cases of dimming parameter p. From Algorithm 1, many codes can be created, such as 2B4B, 3B8B, 4B16B, and so on (input of m bits, output of n =2“ bits). Also, the dimming parameter p can be changed, where p = k/n ranges in between [l/2^m, 1 - l/2^m]. This allows user to select the brightness of IR and control the energy consumption of IR without reducing SNR. Figure 6 describes the procedure of C-OOK-2 modulation. The data bits of net-ID can apply FEC (610), and then splitting into smaller packages by using serial-to-parallel S/P (620). In the case that the length of data bits is shorter than the maximum allowed data packet size, there is no need for inserting a sequence number (SN) (630) for each data packet. The use of SN will support time-variant frame rate cameras, which is important in terms of hardware compatibility. From the BwB LUT (the code table), the line code mapping rule is generated for line coding (640), and the data packet is structured with preamble and payload (650). Finally, the optical clock rate is selected (660) and the OOK binaries are clocked to IR-LED

Figure 7 describes the algorithm (Algorithm 2) to produce the preamble and payload for encoding by using BzzB LUT Code Table. This is used in line coding (640) and preamble insertion (650) within the C-OOK-2 modulation procedure.

Algorithm 2 procedures a preamble that has a size double than that of a payload codeword. Hamming distance between preamble and any couple of payload codewords equals to the size of a payload codeword, despite the change of dimming parameter p This makes SNR of the system high during dimming.

In Figure 8, the C-FSK modulation process is described, which has two significant changes compared to the IEEE 802.15.7-2018 CM-FSK mode, including Asynchronous encoder (840), (910) and asynchronous n-FSK mapping (850), (920). The algorithm to implement these two features are descried in Figure 15 and Figure 17.

Additionally, the values for PHY PIB attributes (680) are included in Figure 8 to show which ones are used to configure the C-FSK modulation layer. This helps our C- FSK scheme being compatible with existing IEEE 802.15.7-2018 standard modes. The detailed values for these PHY PIB attributes are given in Figure 13.

Figure 9 (top) gives an example of mBnB LUT with m =2 and n=4 (2B4B code). Figure 9 (down) gives an example of 2B4B payload and preamble. Both are generated from Algorithm 1 and 2.

Figure 10 describes Algorithm 3 - demodulation of C-OOK-2. This algorithm uses matched filter, and it applies for all cases of dimming parameter p.

Based on the demodulation method given in Algorithm 3, Figure 11, Figure 12, and Figure 13 show the implementation results of codes 2B4B, 3B8B, 4B16B.

Figure 14 gives PHY PIB attributes Table that is used for C-OOK-2.

Note that:

• Line code and the dimming parameter p are both selected by user (via App)

• Optical clock rate has two options: kHz-level and Hz-level o At a short transmission distance, it uses a kHz level clock rate (10 kHz for example) so that the entire data packet (a preamble plus payload) can be captured by an image. o At a long range of transmission where camera can not capture the entire data packet within an image, it uses a Hz level clock rate (10 Hz for example). This Hz-level clock rate is low but it can be used because IR is imperceptible to human eyes. In this case, camera will demodulate the data packet from multiple images.

Figure 15 shows the Asynchronous Encoding algorithm (Algorithm 4) used for C- FSK modulation.

This algorithm aims to produce the dissimilarity property between any pair of two adjacent frequency symbols. If two adjacent symbols are equal, this algorithm will replace the second symbol by an asynchronous symbol.

Figure 16 shows the decoding algorithm used for C-FSK demodulation (Algorithm

5).

Figure 17 gives an example of 10-frequency C-FSK mapping table.

Among ten frequencies, the max frequency is chosen as a preamble so that it improves the calibration performance of FFT peaks in the receiver side. The lowest frequency is chosen as an asynchronous frequency.

Figure 18 highlights the necessary PHY PIB attributes used for the configuration of C-FSK modulation. The values are also newly proposed to optimize the performance of transmission in the ID-broadcasting scenario for augmented reality user experience. The proposed values for C-FSK include: o The bandwidth is limited in between 1kHz (due to flicker mitigation) and 10kHz (due to the camera cut-off frequency); o The number of data frequency is selected to be no more than 16 (not including the preamble and the asynchronous frequency) to maximize the communication distance; o The max frequency is chosen as a preamble so that it improves the calibration performance of FFT peaks in the receiver side. We linearly estimate and calibrate the FFT peaks of other frequencies based on the FFT peak of the preamble. This calibration is crucial because cameras have different rolling sampling rate (or pixel scanning rate). o Frequency separation options are 200Hz, 250Hz and so on (less frequency spacing means that the FFT peaks are denser and the transmission error performance is worse). Finally, security for accessing and controlling devices is one of critical issues. Figure 19 proposes multiple protection layers for security, including: i) Messages being sent over WiFi are protected by WiFi encryption

Different wireless security protocols are WEP, WPA, WPA2 or WPA3 can be used to protect WiFi messages. ii) All direct control commands that App sends to light devices shall include “App-key”.

Each App has an unique secured key called “App-key”, which is a local key, generated and managed by the host-App (A host- App is the App that owns the light device). All App-keys are also saved to the memory of the light device for checking the App control permission.

The “App-key” that is given to an App plays a role as a key to access the light device for controlling. An anonymous App will not have a valid “App-key”, and thus it is unable to control the light devices.

Note that, a light device has a limited memory size, thus it can support a limited number of “App-keys” being saved. iii) Cloud security checks “App-ID” of any remote control command. The App-ID is the global identity of App, which is managed by cloud.

Due to the limited number of “App-keys”, a guest App that does not have any “App-key” can also control the light device over Cloud if the permission is given (for example, permission is granted by the host-App). By this way, the cloud needs to check the ID of that App for security.

The host-App can grant the control permission to a guest-App by sharing a specific control key. This is a key that is agreed between the host- App and the Cloud in prior to any sharing to a guest-App. This key can be referred to as “remote control key”. After that, the Cloud can verify the permission for any remote control. The remote control key can be an OTC for limiting the number of times controlling from the guest-App.

Finally, an additional prompt message exchange protocol between Cloud, host-App, and guest-App can also be used for the final validation of the control. For example, when a guest-App controls the light device via Cloud, the Cloud will send a message and inform the host-App for this action, and then the Cloud may wait for the confirmation of the host-App. iv) One-Time-Code (OTC) is the security protocol within the “OTC secured mode”.

OTC is generated by the light device and sent to the App that enables the “OTC secured mode”. Accordingly, only that App (and other Apps get the OTC from sharing) can control the light. When App enables the “OTC secured mode” of a light device (this can be used for privacy purpose), only this App (and other Apps that get OTC being shared by that App) can control the enabled light device.

The procedure of “OTC secured mode” is as follows

• App enables the “OTC secured mode” of a light device;

• The light device sends an OTC to App. This OTC must be saved by App so that App can use for controlling the light device lately;

• App acknowledge that it receives the OTC. The light device enables the OTC secured mode;

• When App sends any control command to the light device, it must include the given OTC;

• The light device checks the OTC for every control command. Only commands that have valid OTC are proceeded.

• App disables “OTC secured mode” so that other Apps can control the light devices as normal.

Implementation requirements

Hardware requirement:

The lights (light devices) used in this system is the WiFi-equipped light device, that has RGB/ RGBW primary color LEDs so that the color/brightness/temperature of the color can be controlled by applying PWM. A mono-color LED (white LED) can be used to control the brightness but not the color. IR LEDs are also included so that the light can use them to transmit the net-ID when being requested.

Smartphone has camera to provide the AR user experience after installing app. However, other cameras including webcam, CCTV camera all can be used with app installed and WiFi connected. The hardware requirements for camera include:

(i) camera is rolling shutter type;

(ii) the rolling sampling rate of camera must be higher than two times of the max frequency used by the modulation (C-FSK and C-OOK-2). The maximum frequency of C-FSK band is mostly always satisfied from our 10kHz communication band, while C-OOK-2 uses a fixed clock rate of 10 kHz; and

(iii) the frame rate of camera must be at least two times the rate that the C-FSK symbols are clocked out and the rate that the C-OOK-2 packets are clocked out. For example, 10 Hz is used to clock out the C-FSK frequencies or to clock out the C-OOK-2 packets, then camera must have at least 20fps frame rate. In the case that C-OOK-2 uses 10Hz (instead of 10kHz) to clock its binaries, camera must also have at least 20fps frame rate. Time-variant frame rate camera support:

In a typical wireless system, the sampling rate of receiver must be a constant and at least two times the transmission rate. The constant sampling rate is to help receiver to down-sample the signal in a convenient way.

However, most of cameras in the market (including smartphone cameras, CCTV camera.. .) do not have any common frame rate. People think that 30fps is the fixed frame rate for all cameras, but this is not true. In the IEEE 802.15.7-2018 standard, the committee agreed that cameras do not have an absolute-3 Ofps as expected.

In practice, camera may have a nearly stable frame rate, which is roughly around 30fps (such as 29.7fps). However, when camera processes the image in realtime, it reduces the frame rate, which depends on the operating system capability. In fact, Android OS allows the image processing speed of up to 20fps or more, depending on how complex the processing is. All the problems related to the camera frame rate, including (i) the stable frame rate but is unequal to the integer times the transmission rate, and (ii) the frame rate is variant during processing, all are called a time-variant frame rate. Accordingly, commercial cameras those have time-variant frame rates should be supported.

The method to support these types of camera is crucial because of the large portion of them. This is a novel contribution of our C-FSK and C-OOK-2 modulation schemes in compared to other schemes.

To be specific,

• Our C-FSK modulation scheme supports the downsampling in time-variant frame rate cameras by producing the non- similarity property between any adjacent symbols. This is done by combining (i) asynchronous Encoding with an Asynchronous symbol, and (ii) mapping with an asynchronous frequency, fAF.

• Similarly, our C-OOK-2 modulation scheme uses the sequence number within each data packet to support the downsampling in time-variant frame rate cameras.

Hardware for controlling the brightness/color /color temperature:

Lights can change their brightness/color/color temperature by controlling the PWM of individual color channels. This is called PWM dimming. The hardware must support PWM at the clock that high much higher than the bandwidth of C-FSK modulation.

The brightness/color/color temperature can be generated by mixing R/G/B or R/G/B/W with a specific ratio. The light that contains R/G/B/W LED is typically used to support the brightness/color/color temperature control.

IR dimming for saving energy

Dimming is also applied to IR modulation to save energy. Because OOK is chosen for IR modulation for a high transmission rate, dimming will be implemented by using dimmable line codes to maintain the SNR. Algorithms 1 and 2 will produce line code with controllable ratio of ‘17 ‘0’ based on the selection of user. Accordingly, the IR energy consumption is controllable without reducing SNR or data rate.

Transmission distance and adaptive clock rate for C-OOK-2

The C-OOK-2 modulation scheme requires a short communication distance so that the image can capture the entire data packet (including a preamble and payload field). If the distance is too far, the demodulation of the data packet from a single image is error.

In the case of far distance, the light will reduce the optical clock rate, from kHz-level to Hz-level. To be specific, it uses 10Hz, the rate that is less than half the camera frame rate. In this case, camera uses a set of adjacent images to demodulate a data packet instead of using a single image like in the case of short distance.

Effectiveness of this invention (Hieu qua dat duoc cua sang che)

This invention discloses a novel light source selection method to control using a “look-to-link” protocol with a camera to provide an augmented reality user experience. It applies Light Communication to bring AR experience, convenience, and security for controlling lights. Uses do not need to remember the buttons or any information of the lights (such as long names). User controls the lights he sees.

This invention uses both visible light and IR light to transmit data, so that camera can identify light devices even if they are OFF.

Novel C-FSK modulation scheme supports long range communication. It provides a high resolution dimming capability, which is crucial for color and brightness control when visible light is used. It also supports cameras with timevariant frame rates.

Novel C-OOK-2 modulation scheme can adapt the optical clock rate to support both short range (with a fast data rate) and long range (with a slower data rate) transmission. It offers a low resolution dimming capability, however it is essential for energy saving in the case of IR. It also supports cameras with time-variant frame rates.

Claims

1. The application-based control system for smart lighting combines three wireless communication links between light devices and a smart camera device (e.g., smartphone with a camera) as follows: i) a radio frequency based wireless bidirectional link (such as WiFi); ii) a visible light based wireless downlink (to camera) is used in the case that visible light is ON; iii) an IR light based wireless downlink (to camera) is used in the case that visible light is OFF.

2. The process to select light devices using a “look-to-link” protocol with a camera (termed AR-camera) includes steps as follows: i) Application (App) allocates the network identity (net-ID) to all light devices via WiFi; ii) App sends a command to light devices over WiFi to request them for broadcasting their net-IDs via light wave; iii) Based on the current status of lighting (ON or OFF), light devices modulate their net-IDs by using either a camera-FSK (C-FSK) modulation scheme for visible light or using a camera-OOK-version 2 (C-OOK-2) modulation scheme for infrared (IR) light; iv) App turns on its AR-camera to receive the data modulated light signal, and then it demodulates the net-ID information to identify light devices; v) App selects a color and requests a light device (that its net-ID is identified) to change the color; vi) The light device uses pulse-width-modulation (PWM) method to update its color; vii) App selects a desired dimming level for the IR-LED within the light device; According to the selection, the light device generates a line code along with a specific dimming level; these are used for C-OOK-2 modulation with the IR-LED.

3. The process in claim 2, wherein the step (iii) includes the selection of different modulation methods for light communication that varies according to the current status of lighting as follows:

Case 1 : Light is currently ON,

The light device uses the modulation scheme called camera-FSK modulation scheme (C-FSK) to modulate its net-ID, and then the signal is sent over the visible light LEDs;

It can also uses modulation scheme called camera-OOK-version 2 (C- OOK-2) to modulate its net-ID, and then it sends the signal over the infrared LEDs; Case 2: Light is currently OFF,

The light device uses the C-OOK-2 modulation scheme to modulate its net-ID, and then it sends the signal over the infrared LEDs.

4. The process in claim 2, wherein the step (iv) describes the demodulation procedure of the light signal (both cases of visible light and IR), that includes the following steps: iv.l) App receives an acknowledgement frame from the light device and App knows which modulation scheme is being used by the light device; iv.2) User selects light device(s) that he wants to control via AR-camera; iv.3) App captures multiple images of the selected light device(s) in a burst mode; iv.4) From images, App demodulates the light signal and identifies the light device(s) based on the demodulated net-ID(s); and then App connects to the identified light device(s) via WiFi for controlling the color.

5. The process in claim 2, wherein the step (vi) describes the process to control the energy consumption of IR via the dimming charasteristic of the C-OOK-2 modulation scheme as follows vi.l) App allows user selecting the dimming level of IR-LEDs (Dimming level ranges in between 1% and 99%); vi.2) After the dimming level being selected, the light device uses Algorithms

1 and 2 to generate a line code lookup table (LUT) and a mapping rule for the data packet (with preamble and payload codewords);

6. The C-OOK-2 demodulation scheme at the AR-camera uses Algorithm 3 to demodulate the data, in which the dimming parameter p is one of the inputs.

7. The C-OOK-2 modulation scheme uses Algorithm 1 to produce a BwB lookup table (for any option of an integer m > 1, and n =2^m) that has multiple options for the dimming parameter p in between the range [ l/w, 1-1/n];

The codes 2B4B, 3B8B, and 4B16B are subsets of the proposed Algorithm 1;

8. The C-OOK-2 modulation scheme uses Algorithm 2, with the input of the mBnB lookup table, to produce a code table as a mapping rule for the data packet (that includes a preamble and payload codewords);

The code table as the output of Algorithm 2 also has multiple options for the dimming parameter p.

9. The C-OOK-2 modulation scheme uses a switchable optical clock rate due to the characteristic of IR light as follows: At a short distance (such as several meters), it uses a kHz-level optical clock rate (such as 10 kHz) to ensure that the entire data packet (including a preamble and payload) can be captured within an image;

At a further distance (such as over 10 meters), it uses a Hz-level optical clock rate (such as 10 Hz for a camera that has the frame rate of at least two times of that, 20fps);

10. The C-OOK-2 demodulation scheme requires the camera to capture multiple images for demodulating a single data packet in the case of Hz-level optical clock rate.

11. The C-FSK modulation scheme supports the downsampling in time- variant frame rate cameras by producing the dissimilarity property between any pair of two adjacent frequency symbols.

12. The C-FSK modulation scheme uses Algorithm 4 “asynchronous encoding” with a special symbol called “asynchronous symbol” to produce the dissimilarity property between any pair of two adjacent frequency symbols.

13. The C-FSK selects the max frequency among the chosen frequencies for a preamble; by using this preamble, a linear calibration in the frequency domain can support different cameras that have different pixel sampling rates.

14. The methods to secure the direct control commands (i.e., the commands that an App sends directly to light devices) in the smart lighting system include: i) The WiFi encryption that protects WiFi messages; ii) An “App-key” method, in which App-key is an unique secured key of an application (App), which is local, generated and managed by the host-App; All App-keys are also saved to the memory of the light device for checking the control permission of Apps; iii) An “OTC secured mode”, in which an One-Time-Code (OTC) is generated by the light device and sent to the App that enables the “OTC secured mode”; Accordingly, only that App (and other Apps that get the OTC from sharing) can control the light device.

15. The methods to secure the remote control commands (the commands that an App sends to the cloud or web services to control the light devices remotely) in the smart lighting system include: i) The verification of “App-ID” and “remote control key” of remote control commands in the cloud server, in which “App-ID” is the global identity of an App, which is managed by the cloud, and “remote control key” is an agreed key between the host-App and the cloud for the security to control remotely; ii) The “remote control key” sharing process that the host-App allows another App (guest- App) to control the light devices remotely by sharing the remote control key;

18 iii) An OTC exchange process, in which an OTC is optionally exchanged between the host-App, the guest-App, and the cloud when the remote control key is shared.

19