WO2020169172A1

WO2020169172A1 - Combined camera and vlc circuit

Info

Publication number: WO2020169172A1
Application number: PCT/EP2019/053985
Authority: WO
Inventors: Alexander Hunt
Original assignee: Telefonaktiebolaget Lm Ericsson (Publ)
Priority date: 2019-02-18
Filing date: 2019-02-18
Publication date: 2020-08-27

Abstract

A combined camera and VLC circuit (5) is disclosed. The circuit (5) comprises an image sensor (10) comprising an array of pixel sensors (70), wherein each pixel sensor has an associated color (B, Gl-2, Rl-4) and is configured to generate an analog value representative of an amount of light that the pixel sensor (70) has been exposed to. Furthermore, the circuit (5) comprises a readout circuit (20) connected to the image sensor (10) and configured to generate analog output samples representative of the analog values generated by the pixel sensors (70). In a first mode of operation, the readout circuit (10) is configured to, for each pixel sensor (70), generate an individual analog output sample representative of the analog value generated by that pixel sensor (70). In a second mode of operation, the readout circuit (10) is configured to, for each group of a plurality of groups of multiple pixel sensors (70), generate a combined analog output sample representative of a combination of the analog values generated by the pixel sensors (70) in that group. Each group of pixel sensors (70) has a related color and consists of all pixel sensors (70) having the related color as its associated color.

Description

COMBINED CAMERA AND VLC CIRCUIT

Technical field

The present invention relates to a circuit that can be used both as a camera and for visible light communication in different modes of operation.

Background

LI-FI (Light Fidelity) is a technology for wireless communication based on light. There is a part within the LI-FI area that is called VLC (Visible Light communication). VLC is wireless communication in the visible light spectrum i.e. 380nm - 800nm. The system normally uses LED (Light-Emitting Diode) that is switched on and off at high speed for transmission. The simplest way when the full spectrum of light is used (LED lamp) as the modulation carrier and a simple photo diode is used to receive the signal. Multi-channel VLC uses a photodiode with a specific color filter in-front of the photodiode for reception. VLC is standardized in the IEEE 802.15.7 standard, see IEEE Standard for Local and Metropolitan Area Networks— Part 15.7: Short-Range Wireless Optical Communication Using Visible Light," in IEEE Std 802.15.7-2011, 6 Sept. 2011.

A drawback when implementing VLC communication in a communication device, such as a mobile telephone, is the need for a dedicated sensor for VLC reception, which requires space in the device and an opening in the mechanics to let the light through.

Summary

Embodiments of the present disclosure address the above drawback with a combined camera and VLC circuit. Since communication devices, such as mobile telephones, are typically equipped with one or more cameras anyway, this approach reduces the overhead for implementing VLC communication. Although a regular camera circuit could potentially be used also for VLC reception, the limited readout speed from such a circuit would only allow relatively low bit rates. In embodiments of the present disclosure, this limitation is addressed with a mode of operation, in which a combined analog output sample is generated for pixel sensors having the same associated color. Such generation of combined analog output samples for multiple pixel sensors is typically referred to as binning and has traditionally been applied in camera sensors to groups of neighboring pixels to improve the signal-to-noise ratio of an image. The inventor has realized that binning pixels with the same associated color rather than neighboring pixels can enable relatively high bit-rate VLC reception. According to a first aspect, there is provided a combined camera and VLC circuit (below referred to simply as“the circuit”). The circuit comprises an image sensor comprising an array of pixel sensors, wherein each pixel sensor has an associated color and is configured to generate an analog value representative of an amount of light that the pixel sensor has been exposed to. Furthermore, the circuit comprises a readout circuit connected to the image sensor and configured to generate analog output samples representative of the analog values generated by the pixel sensors. In a first mode of operation, the readout circuit is configured to, for each pixel sensor, generate an individual analog output sample representative of the analog value generated by that pixel sensor. In a second mode of operation, the readout circuit is configured to, for each group of a plurality of groups of multiple pixel sensors, generate a combined analog output sample representative of a combination of the analog values generated by the pixel sensors in that group. Each group of pixel sensors has a related color and consists of all pixel sensors having the related color as its associated color.

The circuit may comprise a filter unit. The filter unit may comprise a color filter for each pixel sensor, wherein the color filter is configured to pass light of the color associated with that pixel sensor and block light of other colors.

According to some embodiments, the first mode of operation is a camera mode and the second mode of operation is a VLC mode.

The circuit may comprise an analog-to-digital converter (ADC) connected to the readout circuit and configured to convert the analog output samples to the digital domain.

The circuit may comprise a digital interface circuit for communication with a host device. The digital interface circuit may, for instance, be configured to communicate according to a Mobile Industry Processor Interface (MIPI) Camera Serial Interface (CSI) standard or a Universal Serial Bus (USB) standard.

According to some embodiments, said associated colors correspond to wavelengths defined in the IEEE 802.15.7 standard.

According to a second aspect, there is provided an electronic apparatus comprising the combined camera and VLC circuit of the first aspect.

The electronic apparatus may be a communication apparatus, such as (but not limited to) a wireless communication device for a cellular communications system.

According to a third aspect, there is provided a method of operating a combined camera and VLC circuit. The combined camera and VLC circuit comprises an image sensor comprising an array of pixel sensors, wherein each pixel sensor has an associated color and is configured to generate an analog value representative of an amount of light that the pixel sensor has been exposed to. Furthermore, the combined camera and VLC circuit comprises a readout circuit connected to the image sensor and configured to generate analog output samples representative of the analog values generated by the pixel sensors. The method comprises, in a first mode of operation, generating, by the readout circuit and for each pixel sensor, an individual analog output sample representative of the analog value generated by that pixel sensor. Furthermore, the method comprises, in a second mode of operation, generating, by the readout circuit and for each group of a plurality of groups of multiple pixel sensors, a combined analog output sample representative of a combination of the analog values generated by the pixel sensors in that group. Each group of pixel sensors has a related color and consists of all pixel sensors having the related color as its associated color.

Further embodiments are defined in the dependent claims. It should be emphasized that the term“comprises/comprising” when used in this specification is taken to specify the presence of stated features, integers, steps, or components, but does not preclude the presence or addition of one or more other features, integers, steps, components, or groups thereof.

Brief description of the drawings

Fig. 1 illustrates an environment wherein a combined camera and VLC circuit can be employed.

Fig. 2 is a block diagram of some embodiments of a combined camera and VLC circuit.

Lig. 3 illustrates an image sensor.

Lig. 4 shows filter responses for color filters.

Lig. 5 is a timing diagram for readout of an image sensor in a first mode of operation.

Lig. 6 is a timing diagram for readout of an image sensor in a second mode of operation.

Lig. 7 is a flowchart of an embodiment of a method.

Detailed description

Lig. 1 illustrates a communication environment wherein embodiments of the present disclosure may be employed. An electronic apparatus 1 is depicted having an opening 2 in its housing for letting through visible light to provide camera functionality as well as VLC functionality. The opening 2 may be equipped with an optical lens configured to focus the incoming light on an image sensor of a combined camera and VLC circuit. In one mode of operation, the combined camera and VLC circuit may be used as a camera for taking photos of photographic objects 3. In another mode of operation, the combined camera and VLC circuit may be used for receiving signals from a VLC transmitter 4. The electronic apparatus 1 is a host device for the combined camera and VLC circuit. The electronic apparatus 1 may e.g. be a communication apparatus. For instance, the

communication apparatus may be a wireless communication device for a cellular

communications system, such as a mobile telephone. However, embodiments of the combined camera and VLC circuit may be beneficially employed in other types of electronic

apparatuses as well to provide combined camera and VLC functionality.

Cameras usually have different sensor modes. For instance, a camera may have a“normal mode” where each pixel sensor is scanned one at a time, and a“binned pixel mode”, which is used when a higher dynamic range is desired. For example, 4x4 binning mode means that 4 green, red or blue (assuming an RGB sensor) neighboring pixel sensors are scanned simultaneously, and the 4 pixels work together as one larger pixel to get a better signal-to- noise ratio SNR, at the cost of a lower resolution. Embodiments of the present invention are based on, instead of just binning 4 neighboring pixels, binning all pixels with the same color filter characteristics for providing VLC functionality. If the sensor has C different distinct colors, this means that there is a total of C pixels to scan instead of several millions. By binning the pixels like this, resolution will be lost, but that is not a problem if the purpose is to detect a transmitted VLC signal. The field of view (FOV) of the receiver in this case will equal the FOV of the camera.

Fig. 2 shows a block diagram of a combined camera and visible-light communication (VLC) circuit 5 according to some embodiments. Below, the combined camera and VLC circuit 5 is refered to as“the circuit 5” for short. The circuit 5 comprises an image sensor 10, which is further described below with reference to Fig. 3.

Fig. 3 depicts an embodiment of the image sensor 10. It comprises an array of pixel sensors 70. Each pixel sensor 70 has an associated color and is configured to generate, at an output of the pixel sensor 70, an analog value representative of an amount of light (of the associated color) that the pixel sensor 70 has been exposed to. The analog value may e.g. be represented with an electrical charge. The term“color” refers to a limited range of wavelengths of light around a certain nominal wavelength of the color. The association between a pixel sensor 70 and its associated color is normally effectuated using a color filter placed over the pixel sensor that allows some wavelengths of light pass through, while blocking other wavelengths of light. In such embodiments, the color may simply be defined as the range of wavelengths that are allowed to pass through the color filter. Fig 3 further shows two alternative possible distribution patterns for the colors associated with the pixel sensors 70. In Fig. 3, the patterns are shown for a subset of 4 by 4 pixel sensors 70. The shown patterns are repeated over the whole image sensor 10.

The upper pattern is a so called Bayer pattern, which is commonly used in digital cameras. The letters R, G, and B stands for red, green, and blue, respectively.

The lower pattern is a modification of the Bayer pattern suggested by the inventor. Below, this pattern is referred to as“the modified pattern”. The modified pattern has four different red colors, labeled Rl, R2, R3, and R4. Furthermore, the modified pattern has two different green colors, labeled G1 and G2. Moreover, the modified pattern has one blue color, labeled B. According to some embodiments, the different red colors (R1-R4) are distributed over the pixels that are assigned to the red color (R) in the conventional Bayer pattern. Furthermore, according to some embodiments, The different green colors (G1-G2) are distributed over the same pixels that are assigned to the green color (G) in the conventional Bayer pattern.

Moreover, according to some embodiments, the blue color (B) is assigned to the same pixels as the blue color (B) in the conventional Bayer pattern.

The colors B, G1-G2, and R1-R4 may for instance be selected to correspond to

wavelengths defined for visible-light communication in the IEEE 802.15.7 standard. This is illustrated in Fig. 4, showing examples of filter responses for different colors. The

wavelengths for the different colors in this example are

With this example, all channels defined in IEEE 802.15.7 can be received using the circuit 5. However, other sets of colors may be used in other embodiments.

Returning to Fig. 2, the circuit 5 further comprises a readout circuit 20 connected to the image sensor 10. The readout circuit 20 is configured to generate analog output samples representative of the analog values generated by the pixel sensors 70, e.g. at an output 24 of the readout circuit. As illustrated in Fig. 2, the readout circuit 20 may have a plurality of inputs 22, where each input 22 may be connected to the output of a pixel sensor. For instance, the readout circuit may comprise a switch network configured connect one or more of its inputs 22 at a time to its output 24. If a single input 22 is connected to the output 24 at a time, then each pixel sensor 70 is scanned separately, one at a time. If more than one input 22 is connected to the output 24 at a time, then the outputs of the corresponding pixel sensors 70 are summed together, or averaged, and scanned together, whereby binned pixel operation is obtained.

According to embodiments, the readout circuit 20 has a first mode of operation and a second mode of operation. In some embodiments, the first mode of operation corresponds to normal camera operation, and may thus be referred to as a“camera mode”. In some embodiments, the second mode of operation corresponds to VLC operation, and may thus be referred to as a“VLC mode”.

In the first mode of operation, the readout circuit 10 is configured to, for each pixel sensor 70, generate an individual analog output sample representative of the analog value generated by that pixel sensor 70. Hence, the first mode is a non-binned mode.

In the second mode of operation, the readout circuit 10 is configured to, for each group of a plurality of groups of multiple pixel sensors 70, generate a combined analog output sample representative of a combination of the analog values generated by the pixel sensors 70 in that group. Each group of pixel sensors 70 has a related color and consists of all pixel sensors 70 having the related color as its associated color. The term“related color” is used for the group of pixel sensors 70, whereas the term“associated color” is used for the individual pixel sensors 70, in order to distinguish the two terms from each other.

Consider, for instance, the example with 7 different colors B, G1-G2, and R1-R4 above. For this example, there may be up to 7 different groups of pixel sensors. One group may have B as its related color, and may consist of all pixel sensors that has B as its associated color. One group may have G1 as its related color, and may consist of all pixel sensors that has G1 as its associated color. One group may have G2 as its related color, and may consist of all pixel sensors that has G2 as its associated color. One group may have R1 as its related color, and may consist of all pixel sensors that has R1 as its associated color. One group may have R2 as its related color, and may consist of all pixel sensors that has R2 as its associated color. One group may have R3 as its related color, and may consist of all pixel sensors that has R3 as its associated color. One group may have R4 as its related color, and may consist of all pixel sensors that has R4 as its associated color. Thus in the second mode, there are only C different analog output samples to be generated by the readout circuit 20 per exposure of the image sensor 10, where again C is the number of distinct colors. In the example above, C = 7. This can be compared with the total number of pixel sensors 70, which is typically in the order of several millions in an image sensor used for a digital camera. Hence, in the second mode, the analog output samples for each of the colors can be read out at a relatively high sample rate, which enables VLC reception at a relatively high data rate. This is further described below with reference to Fig. 6.

As illustrated in Fig. 2, the circuit 5 may comprise a filter unit 30. According to

embodiments, the filter unit 30 comprises a color filter for each pixel sensor 70. The color filter is configured to pass light of the color associated with that pixel sensor 70 and block light of other colors. The filter unit 30 may be placed on top of the image sensor 10, such that the color filter for a given pixel sensor 70 is aligned with that pixel sensor 70.

Furthermore, as also illustrated in Fig. 2, the circuit 5 may compris an analog-to-digital converter (ADC) 40 connected to the readout circuit 20. The ADC 40 may be configured to convert the analog output samples from the readout circuit 20 to the digital domain.

Moreover, as illustrated in Fig. 2, the circuit 5 may comprise a digital processor circuit 50 configured to process the output data from the ADC 40. The digital processor circuit 50 may, for instance, be an image processor. In the first mode of operation, the digital processor circuit 50 may be configured to perform image processing tasks, such as but not limited to image compression. In the second mode of operation, the digital processor circuit 50 may be configured to perform signal processing tasks related to VLC communication. Alternatively, the digital signal processor 50 may, in the second mode of operation, be configured to simply forward the data from the ADC 40, e.g. for further processing by another processor circuit, which may be external to the circuit 5.

As is also illustrated in Fig. 2, the circuit 5 may comprise a digital interface circuit 60. The digital interface circuit 60 may e.g. be configured for communication between the circuit 5 and a host device, such as the wireless communication device 1. According to some embodiments, digital interface circuit 60 is configured to communicate according to a Mobile Industry Processor Interface (MIPI) Camera Serial Interface (CSI) standard. However, other types of standards may be used as well. For instance, in some embodiments, the digital interface circuit may be a universal serial bus (USB) interface circuit, such as a USB 3 interface circuit.

Fig. 5 is a timing diagram illustrating the operation of the circuit 5 in the first mode of operation according to an embodiment. According to this embodiment, the image sensor 10 has N rows of pixel sensors 70. The rows are scanned in sequential order as they are physically laid out on the image sensor, which is a type of operation commonly referred to as “rolling shutter”. Each row in the timing diagram shows three different operations for the corresponding row of pixel sensors 70 of the image sensor 10. These are described in the following with reference to row 1, but the operations are the same for the other rows, but skewed in time as illustrated in Fig. 5. Operations 80 and 100 are readout operations from the pixel sensors 70 in row 1 following an exposure period where the pixel sensors 70 are exposed to light. One such exposure period is shown for each row in Fig. 5, and labeled with the reference number 90 for row 1. Thus, in operation 100, the analog values read out from the pixel sensors 70 in row 1 represent the amount of light that these pixel sensors 70 have been been exposed to during the exposure period 90. Similarly, in operation 80, the analog values read out from the pixel sensors 70 in row 1 represent the amount of light that these pixel sensors 70 have been been exposed to during a preceding exposure period, which is not explicitly shown in Fig. 5. When operation 80 has been concluded, the corresponding operation for row 2 commences, and when that operation has been concluded, the corresponding operation for row 3 commences, etc. The horizontal dotted lines in Fig. 5 are included to indicate that the process is repeated over and over again while the circuit 5 is operative in the first mode of operation.

Fig. 6 is a corresponding timing diagram for the second mode of operation according to an embodiment using the seven distinct colors B, G1-G2, and R1-R4 presented above. The analog output samples corresponding to the different colors are scanned in the sequential order B, Gl, G2, Rl, R2, R3, R4, and then starting again with B, Gl, ... etc. Each color in the timing diagram has three different operations. These are described in the following with reference to color B, but the operations are the same for the other colors, but skewed in time as illustrated in Fig. 6. Operations 110 and 130 are readout operations of the analog output sample representative of the combination of the analog values generated by the pixel sensors 70 with color B following an exposure period where these pixel sensors 70 are exposed to light. One such exposure period is shown for each color in Fig. 6, and labeled with the reference number 120 for color B. Thus, in operation 130, the analog output sample that is read out represents the amount of light that the pixel sensors 70 with associated color B have been exposed to during the exposure period 120. Similarly, in operation 110, the analog output sample that is read out represents the amount of light that the pixel sensors 70 with associated color B have been exposed to during a preceding exposure period, which is not explicitly shown in Fig. 6. When operation 110 has been concluded, the corresponding operation for color G1 commences, and when that operation has been concluded, the corresponding operation for color G2 commences, etc. The horizontal dotted lines in Fig. 6 are included to indicate that the process is repeated over and over again while the circuit 6 is operative in the first mode of operation.

Consider, as an example, an image sensor 10 with 5152 by 3864 pixel sensors 70, which is a total of 19930512 pixel sensors 70 (i.e. a 20 MP (Mega Pixel) sensor). Assume that in the first mode of operation, the circuit 5 is capable of reading out 30 frames per second. That means that it can read out samples at a sample rate of 30 19930512 * 598 10⁶ samples/second (or 598 MHz), wherein each sample represents the analog value from a single pixel sensor. If the same sample rate is used in the second mode of operation, the following sample rates per color (or“per channel” if each color is seen as a communications channel) can be obtained for the given total number of distinct colors:

According to some embodiments, there is provided a method of operating the combined camera and VLC circuit 5. A flow chart of an embodiment of the method is shown in Fig. 7. The operation is started in step 200. In step 210, it is decided whether the first or the second mode of operation should be used. If the first mode of operation should be used, step 220 is carried out. Step 220 comprises generating, by the readout circuit 20 and for each pixel sensor 70, an individual analog output sample representative of the analog value generated by that pixel sensor 70. If the second mode of operation should be used, step 230 is carried out. Step 230 comprises generating 230, by the readout circuit 20 and for each group of a plurality of groups of multiple pixel sensors 70, a combined analog output sample representative of a combination of the analog values generated by the pixel sensors 70 in that group. As above, each group of pixel sensors 70 has a related color and consists of all pixel sensors 70 having the related color as its associated color. Operation is ended in step 240. The disclosure above refers to specific embodiments. However, other embodiments than the above described are possible within the scope of the disclosure. Different method steps than those described above, performing the method by hardware or software, may be provided within the scope of the disclosure. The different features and steps of the embodiments may be combined in other combinations than those described.

Claims

1. A combined camera and visible-light communication, VLC, circuit, (5) comprising an image sensor (10) comprising an array of pixel sensors (70), wherein each pixel sensor has an associated color (B, Gl-2, Rl-4) and is configured to generate an analog value representative of an amount of light that the pixel sensor (70) has been exposed to; and

a readout circuit (20) connected to the image sensor (10) and configured to generate analog output samples representative of the analog values generated by the pixel sensors (70);

wherein

in a first mode of operation, the readout circuit (10) is configured to, for each pixel sensor (70), generate an individual analog output sample representative of the analog value generated by that pixel sensor (70);

in a second mode of operation, the readout circuit (10) is configured to, for each group of a plurality of groups of multiple pixel sensors (70), generate a combined analog output sample representative of a combination of the analog values generated by the pixel sensors (70) in that group; and

each group of pixel sensors (70) has a related color and consists of all pixel sensors (70) having the related color as its associated color.

2. The combined camera and VLC circuit (5) of claim 1, comprising a filter unit (30), the filter unit comprising a color filter for each pixel sensor (70), wherein the color filter is configured to pass light of the color (B, Gl-2, Rl-4) associated with that pixel sensor (70) and block light of other colors.

3. The combined camera and VLC circuit (5) of claim 1 or 2, wherein

the first mode of operation is a camera mode; and

the second mode of operation is a VLC mode.

4. The combined camera and VLC circuit (5) of any preceding claim, comprising an analog-to-digital converter, ADC, (40) connected to the readout circuit (20) and configured to convert the analog output samples to the digital domain.

5. The combined camera and VLC circuit (5) of any preceding claim comprising a digital interface circuit (60) for communication with a host device (1).

6. The combined camera and VLC circuit (5) of claim 5, wherein the digital interface circuit (60) is configured to communicate according to a Mobile Industry Processor Interface, MIPI, Camera Serial Interface, CSI, standard.

7. The combined camera and VLC circuit (5) of claim 5, wherein the digital interface circuit (60) is configured to communicate according to a Universal Serial Bus, USB, standard.

8. The combined camera and VLC circuit (5) of any preceding claim, wherein said associated colors (B, Gl-2, Rl-4) correspond to wavelengths defined in the IEEE 802.15.7 standard.

9. An electronic apparatus (1) comprising the combined camera and VLC circuit (5) of any one of the claims 1 - 8.

10. The electronic apparatus (1) of claim 9, wherein the electronic apparatus is a communication apparatus.

11. The electronic apparatus (1) of claim 10, wherein the communication apparatus is a wireless communication device for a cellular communications system.

12. A method of operating a combined camera and VLC circuit (5), the combined camera and VLC circuit (5) comprising

an image sensor (10) comprising an array of pixel sensors (70), wherein each pixel sensor (70) has an associated color (B, Gl-2, Rl-4) and is configured to generate an analog value representative of an amount of light that the pixel sensor (70) has been exposed to; and

the method comprising:

in a first mode of operation, generating (220) by the readout circuit (20), for each pixel sensor (70), an individual analog output sample representative of the analog value generated by that pixel sensor (70); and

in a second mode of operation, generating (230) by the readout circuit (20), for each group of a plurality of groups of multiple pixel sensors (70), a combined analog output sample representative of a combination of the analog values generated by the pixel sensors (70) in that group, each group of pixel sensors (70) having a related color and consisting of all pixel sensors (70) having the related color as its associated color.