CN105430289A - A Method of Detecting LED Flashing Frequency Based on CMOS Image Sensor - Google Patents

Abstract

The invention discloses a method for detecting the flicker frequency of an LED based on a CMOS image sensor. The method comprises the steps of: obtaining a strip image of a light source area, converting the strip image into a gray image; calculating an average gray value for all pixels in each line of the strip image, supposing that the gray image with bright and dark strips has M lines, storing M average gray values into an array IAverage(n), wherein n is equal to 0, 1, ..., M-1; performing M point DFT operation on the IAverage(n) to work out a frequency response value R(k) of the IAverage(n), wherein R(k) is equal to FORMULA, k is not less than 0 and not more than M-1, and W<nk>M is a Fourier coefficient; detecting K values corresponding to peaks in the R(k) except low frequency components, and setting K values to be KN, which namely is the number of strips in the strip image; dividing total number of lines M by KN to obtain pixel line numbers of bright and dark strip widths, which namely are the regularly varied periodic value of the pixel gray value in each line, setting Row_N to be equal to M/KN; and working out the flicker frequency of the LED fLED which is equal to frow/ROW_N, wherein frow is the line scanning frequency of the CMOS image sensor. According to the method, corresponding relationship between each characteristic value of the image with bright and dark strips and the flicker frequency of the LED is obtained, detection on the flicker frequency of the LED with high reliability is achieved, and good practical application value is provided.

Description

A Method of Detecting LED Flashing Frequency Based on CMOS Image Sensor

technical field

The invention faces the field of LED visible light communication (VLC) positioning, and proposes a method for detecting LED flickering frequency based on a CMOS image sensor.

Background technique

Using a mobile phone with a CMOS image sensor (CIS) to receive the LED light signal can realize low-rate information transmission by detecting the light and dark stripe pictures formed by the flashing LED. In order to make the blinking LEDs form a light and dark fringe picture on the imaging plane, the CMOS image sensor should use a rolling shutter. There is a prior art 1 that proposes a visible light communication system, as shown in Figure 1, the transmitter uses an LED light source driven by on-off keying (OOK) modulation, and the receiver uses a CMOS image sensor to form a picture of light and dark stripes. The picture undergoes image processing to demodulate the OOK signal. The system and decoding method proposed in this paper can achieve certain communication performance in short-distance communication (tens of centimeters) and without background ambient light interference. However, if the system and method proposed in prior art 1 are applied to the actual common indoor lighting environment, that is, to achieve a communication distance of 2-6 meters and there is ambient light interference, its communication performance will become very poor and cannot meet the requirements of practical application.

Existing technology 2 discloses a visible light signal transmission decoding method. The basic idea is that the transmitting end uses LED light source lamps to flicker at different frequencies, and the receiving end uses a CMOS image sensor to obtain flickering light signals to form fringe pictures with different widths of bright and dark fringes. The width of the light and dark stripes of the fringe picture depends on the flickering frequency of the LED light source. The transmitter drives the LED lamps to send flashing frequency information sequentially through frequency shift on-off keying (FSOOK) modulation, and each frequency represents several bits of data. The CMOS image sensor at the receiving end takes pictures at equal intervals to obtain several light and dark fringe pictures, and then detects the number of fringes on the light and dark fringe pictures. Since different numbers of stripes represent different flicker frequencies, binary data can be decoded; the visible light communication system proposed in prior art 2 is shown in FIG. 2 .

As shown in Figure 2, the key to reliable information transmission in the visible light communication (VLC) system based on FSOOK modulation is to decode the light and dark stripe pictures of different widths to obtain the LED flicker frequency. The specific idea of the method for detecting the number of stripes in the fringe picture proposed in prior art 2 is: firstly, the fringe picture in the effective light source area is further reduced to a rectangle of the light-emitting surface, and the gray-scale value of the rectangle picture of the light-emitting surface is binarized, and then the two Each row of the value rectangle picture is summed, and then the first-order or second-order partial derivative is calculated for the sum value, and finally the partial derivative value of each row is processed to obtain the number of stripes of the picture.

However, as verified by actual experiments, the reliability of the method for detecting the number of stripes in a picture proposed in prior art 2 is not ideal. Moreover, if the distance between the CMOS image sensor and the LED light source (the flickering frequency remains unchanged) changes, the number of stripes in the effective light source area will change, and the corresponding relationship between the number of stripes and the LED flickering frequency cannot be obtained.

Contents of the invention

In order to overcome at least one defect (deficiency) of the above-mentioned prior art, the present invention provides a method for detecting LED flicker frequency based on a CMOS image sensor. By mathematically modeling the visible light communication system, this method obtains the correspondence between a certain characteristic value of the bright and dark fringe picture and the LED flicker frequency, and thus realizes the reliable detection of the LED flicker frequency. practical application value.

In order to solve the problems of the technologies described above, the technical solution of the present invention is as follows:

A method for detecting LED flashing frequency based on a CMOS image sensor, comprising:

S1. Obtain the fringe image of the light source area, and convert the fringe image into a grayscale image;

S2. Calculate the average gray value for all pixels in each row of the stripe picture, assuming that the gray-scale light and dark stripe picture has M rows in total, and store M average gray values in the array I _Average (n), n=0, 1,..., M-1;

S3, carry out M point DFT operation to I _Average (n), ask the frequency response value R (k) of I _Average (n);

in $R\left(k\right)=\mathrm{D.}fT\&lsqb;I_{Average}\left(\mathrm{no}\right)\&rsqb;=\underset{\mathrm{no}=0}{\overset{m-1}{\&Sigma;}}{I}_{Average}\left(\mathrm{no}\right)W_m^{\mathrm{no}k},$ 0≤k≤M-1, is the Fourier coefficient;

S4, detect the K value corresponding to the peak value (maximum value) except the low-frequency component in R(k), set it as K _N , K _N is the number of stripes in the stripe picture; then divide the total row number M by K _N to get The number of pixel rows of the light and dark stripe width, i.e. the periodic value of the variation law of the pixel gray value of each row of the stripe picture, is set as Row_N=M/K _N ;

S5. Obtain the blinking frequency fLED of the _LED =f _row /ROW_N; wherein f _row is the row scanning frequency of the CMOS image sensor.

In the present invention, by mathematically modeling the VLC communication system combined with FSOOK modulation-driven LED lamps and a CMOS image sensor, it can be known that the change rule of the pixel gray value of each row of the captured light and dark stripe pictures is periodic, that is, using a CMOS image sensor The line scanning frequency divided by this cycle value is the LED blinking frequency.

Compared with the prior art, the beneficial effects of the technical solution of the present invention are:

The method proposed by the present invention for detecting LED flickering frequency for bright and dark stripe pictures can effectively improve the reliability of low-rate information transmission of this type of VLC communication positioning system, reduce the bit error rate, and improve the ability to resist various ambient light interferences. It is concise and clear, can greatly improve the processing speed of light and dark stripe pictures, and has great application value.

Description of drawings

FIG. 1 is a schematic diagram of a visible light communication system proposed in prior art 1.

Fig. 2 is a schematic diagram of a visible light communication system proposed in prior art 2.

Figure 3 is a light and dark fringe diagram formed by taking pictures of LED lamps with a Huawei smartphone CMOS camera.

Figure 4 is a schematic diagram of the imaging process of CIS using rolling shutter to take pictures of flashing LED lamps.

FIG. 5 is an amplitude waveform diagram of the average gray value of each row of pixels obtained by processing the fringe image in FIG. 1 .

Fig. 6 is a flow chart of the specific implementation process of the detection method.

FIG. 7 is a spectrum response diagram of the average gray value of each row of pixels obtained by processing the stripe image in FIG. 3 .

detailed description

The drawings are for illustrative purposes only, and should not be construed as limitations on this patent; in order to better illustrate this embodiment, some parts in the drawings will be omitted, enlarged or reduced, and do not represent the size of the actual product;

For those skilled in the art, it is understandable that some well-known structures and descriptions thereof may be omitted in the drawings. The imaging process of the bright and dark fringe pictures will be described below in conjunction with the accompanying drawings and embodiments.

Taking the current common optical camera communication (OCC) system as an example, the transmitter can generally use LED lamps driven by FSOOK modulation, and can be equipped with a lampshade to control the illumination area. Assume that the flicker frequency of the LED lamp is f _LED , and the flicker period is The LED lamp sends out a square wave signal with a duty ratio of 0.5, then the time-domain expression of the LED luminous intensity can be expressed by the following formula:

E(t)=E ₀ (t+kT ₀ )(1)

in:

${\mathrm{<span\; class="notranslate">\; E.</span>}}_{\mathrm{<span\; class="notranslate">\; 0</span>}}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; t</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; =</span>\left\{\begin{array}{cc}{\mathrm{<span\; class="notranslate">\; E.</span>}}_{\mathrm{<span\; class="notranslate">\; m</span>}}<span\; class="notranslate">\; ,</span>& \mathrm{<span\; class="notranslate">\; 0</span>}<span\; class="notranslate">\; \&le;</span>\mathrm{<span\; class="notranslate">\; t</span>}<span\; class="notranslate">\; <</span>\frac{{\mathrm{<span\; class="notranslate">\; T</span>}}_{\mathrm{<span\; class="notranslate">\; 0</span>}}}{\mathrm{<span\; class="notranslate">\; 2</span>}}\\ \mathrm{<span\; class="notranslate">\; 0</span>}<span\; class="notranslate">\; ,</span>& \frac{{\mathrm{<span\; class="notranslate">\; T</span>}}_{\mathrm{<span\; class="notranslate">\; 0</span>}}}{\mathrm{<span\; class="notranslate">\; 2</span>}}<span\; class="notranslate">\; \&le;</span>\mathrm{<span\; class="notranslate">\; t</span>}<span\; class="notranslate">\; <</span>{\mathrm{<span\; class="notranslate">\; T</span>}}_{\mathrm{<span\; class="notranslate">\; 0</span>}}\end{array}\right.<span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; 2</span>}<span\; class="notranslate">\; )</span>$

Where E _m is the peak light intensity. Alternatively, the receiver may employ a CIS with a rolling shutter mechanism. Under the shooting conditions of the actual scene, the light intensity distribution in the effective light source area is different. A high light area will be formed in the center, and the light intensity will gradually decrease when it extends outward. Figure 3 is an example diagram of using the CIS of a Huawei smartphone to take pictures of flashing LED lamps with a rolling shutter. It can be seen from the figure that the picture can be divided into two layers. The first layer is the image of the LED lamp (high light area) obtained by imaging, and the other layer is the image of the light and dark stripes formed by the rolling shutter. Under the ideal assumption that the light intensity distribution is uniform, only the light and dark fringe pictures formed by the rolling shutter will remain in the imaging picture in the effective light source area.

Assume that the number of rows of the imaging picture of the LED lamp is M, and each row has N pixels. According to the characteristics of the rolling shutter CIS, the exposure time T _e of the N pixels in each row is the same, and two adjacent rows will be separated by a fixed row interval time T _row during exposure. In rolling shutter exposure control, it is necessary to maintain the interval time T _row between exposures of adjacent rows. This is because each pixel in each row requires a certain readout time T _pix , and only when T _row >NT _pix is satisfied, can the optical signals of all pixels in each row be completely and correctly read out. Figure 4 shows a schematic diagram of the imaging process of CIS using a rolling shutter to take pictures of flickering LED lamps.

In Fig. 4, it is assumed that the start time of taking a new photograph is t ₀ , and the light energy loss coefficient of light in free space and optical system is K _op , and K _RC is the proportional constant of the CIS imaging circuit. For an OCC system consisting of one CIS and one LED lamp, the LED flashing frequency can be preset and the exposure time T _e of the CIS. For example, the flashing period T ₀ of the LED lamp can be set to an integer multiple of the row interval time T _row , that is, T ₀ = nT _row , where n is a positive integer and satisfies n>1, and the row scanning frequency is Then the i-th row output signal of the pixel array of the CIS imaging picture can be expressed as:

$\begin{array}{c}\mathrm{<span\; class="notranslate">\; V</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; =</span>{\mathrm{<span\; class="notranslate">\; V</span>}}_{\mathrm{<span\; class="notranslate">\; S</span>}}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; +</span>{\mathrm{<span\; class="notranslate">\; V</span>}}_{\mathrm{<span\; class="notranslate">\; N</span>}}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; )</span>\\ <span\; class="notranslate">\; =</span>{\mathrm{<span\; class="notranslate">\; K</span>}}_{\mathrm{<span\; class="notranslate">\; o</span>}\mathrm{<span\; class="notranslate">\; p</span>}}{\mathrm{<span\; class="notranslate">\; K</span>}}_{\mathrm{<span\; class="notranslate">\; R</span>}\mathrm{<span\; class="notranslate">\; C</span>}}{<span\; class="notranslate">\; \&Integral;</span>}_{{\mathrm{<span\; class="notranslate">\; t</span>}}_{\mathrm{<span\; class="notranslate">\; 0</span>}}<span\; class="notranslate">\; +</span><span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; -</span>\mathrm{<span\; class="notranslate">\; 1</span>}<span\; class="notranslate">\; )</span>{\mathrm{<span\; class="notranslate">\; T</span>}}_{\mathrm{<span\; class="notranslate">\; r</span>}\mathrm{<span\; class="notranslate">\; o</span>}\mathrm{<span\; class="notranslate">\; w</span>}}}^{{\mathrm{<span\; class="notranslate">\; t</span>}}_{\mathrm{<span\; class="notranslate">\; 0</span>}}<span\; class="notranslate">\; +</span><span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; -</span>\mathrm{<span\; class="notranslate">\; 1</span>}<span\; class="notranslate">\; )</span>{\mathrm{<span\; class="notranslate">\; T</span>}}_{\mathrm{<span\; class="notranslate">\; r</span>}\mathrm{<span\; class="notranslate">\; o</span>}\mathrm{<span\; class="notranslate">\; w</span>}}<span\; class="notranslate">\; +</span>{\mathrm{<span\; class="notranslate">\; T</span>}}_{\mathrm{<span\; class="notranslate">\; e</span>}}}\mathrm{<span\; class="notranslate">\; E.</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; t</span>}<span\; class="notranslate">\; )</span>\mathrm{<span\; class="notranslate">\; d</span>}\mathrm{<span\; class="notranslate">\; t</span>}<span\; class="notranslate">\; +</span>{\mathrm{<span\; class="notranslate">\; V</span>}}_{\mathrm{<span\; class="notranslate">\; N</span>}}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; )</span>\\ <span\; class="notranslate">\; =</span>{\mathrm{<span\; class="notranslate">\; K</span>}}_{\mathrm{<span\; class="notranslate">\; o</span>}\mathrm{<span\; class="notranslate">\; p</span>}}{\mathrm{<span\; class="notranslate">\; K</span>}}_{\mathrm{<span\; class="notranslate">\; R</span>}\mathrm{<span\; class="notranslate">\; C</span>}}{<span\; class="notranslate">\; \&Integral;</span>}_{{\mathrm{<span\; class="notranslate">\; t</span>}}_{\mathrm{<span\; class="notranslate">\; 0</span>}}<span\; class="notranslate">\; +</span><span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; -</span>\mathrm{<span\; class="notranslate">\; 1</span>}<span\; class="notranslate">\; )</span>{\mathrm{<span\; class="notranslate">\; T</span>}}_{\mathrm{<span\; class="notranslate">\; r</span>}\mathrm{<span\; class="notranslate">\; o</span>}\mathrm{<span\; class="notranslate">\; w</span>}}}^{{\mathrm{<span\; class="notranslate">\; t</span>}}_{\mathrm{<span\; class="notranslate">\; 0</span>}}<span\; class="notranslate">\; +</span><span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; -</span>\mathrm{<span\; class="notranslate">\; 1</span>}<span\; class="notranslate">\; )</span>{\mathrm{<span\; class="notranslate">\; T</span>}}_{\mathrm{<span\; class="notranslate">\; r</span>}\mathrm{<span\; class="notranslate">\; o</span>}\mathrm{<span\; class="notranslate">\; w</span>}}<span\; class="notranslate">\; +</span>{\mathrm{<span\; class="notranslate">\; T</span>}}_{\mathrm{<span\; class="notranslate">\; e</span>}}}{\mathrm{<span\; class="notranslate">\; E.</span>}}_{\mathrm{<span\; class="notranslate">\; 0</span>}}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; t</span>}<span\; class="notranslate">\; +</span>{\mathrm{<span\; class="notranslate">\; kT</span>}}_{\mathrm{<span\; class="notranslate">\; r</span>}\mathrm{<span\; class="notranslate">\; o</span>}\mathrm{<span\; class="notranslate">\; w</span>}}<span\; class="notranslate">\; )</span>\mathrm{<span\; class="notranslate">\; d</span>}\mathrm{<span\; class="notranslate">\; t</span>}<span\; class="notranslate">\; +</span>{\mathrm{<span\; class="notranslate">\; V</span>}}_{\mathrm{<span\; class="notranslate">\; N</span>}}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; )</span>\end{array}<span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; 3</span>}<span\; class="notranslate">\; )</span>$

Among them, the first term V _S (i) of the formula (3) represents the i-th line voltage (current) signal formed by the rolling shutter imaging, and the second term V _N (i) is a non-negative signal, Indicates the high light interference noise formed by the captured LED wick in the i-th row. Figure 5 shows the amplitude waveform diagram of the average gray value of each row of pixels in the fringe image shown in Figure 1. It can be seen from the figure that there is a large highlight interference value in the interval from 130 rows to 230 rows, and the V(i) signal Raised to a high place.

It can be seen from the above formula that when T _e =T ₀ =n·T _row , the signal amplitudes of V _S (i) are all the same, indicating that in this case no bright and dark fringe images will be generated in the effective light source area. When T _e ≠T ₀ , V _S (i)=V _S [i+k(n+1)], which means that the output signal of each row is a periodic waveform with a period of n. In the actual photographing process, the exposure time selected by CIS needs to match the flickering cycle of the LED light source, so that light and dark stripes conforming to statistical laws are formed in the effective light source area. According to the mathematical relationship between LED flicker frequency and light and dark stripes, this project will study the use of specific digital filters to realize the signal processing of fringes, so as to calculate the new detection algorithm of LED flicker frequency, and carry out simulation and actual measurement verification of the performance of the algorithm .

In the actual photographing process, the exposure time selected by the CMOS image sensor is longer than the flickering period of the LED light source, so the average pixel gray value of each row of the light and dark stripe image formed in the effective light source area is relative to the "number of rows" variable. A truncated periodic waveform. As shown in Figure 3, observing that there are 12 light and dark stripes in Figure 3 indicates that this picture captures a periodic waveform of 12 periods.

In order to more fully illustrate the beneficial effects of the present invention, a simulation analysis is performed below in conjunction with the specific example shown in FIG. 3 , to further illustrate the effectiveness and advancement of the present invention. Fig. 6 has provided the concrete implementation process of the present invention, briefly described as follows:

Step 1: Obtain the fringe image of the light source area. An example of the fringe image is shown in Figure 3, and then convert the light and dark fringe image into a grayscale image;

Step 2: Calculate the average gray value of all the pixel gray values in each row, the gray value is in the range of 0-255, and store it in an array to form an average gray value amplitude waveform. For example, the average gray value amplitude waveform obtained after processing the stripe picture in Figure 3 is shown in Figure 5;

Step 3: Perform M-point DFT transformation on the amplitude waveform of the average gray value, where M is the total number of rows of the stripe image, and obtain the spectral response value R(k) of the average gray value of each row of pixels. For example, by processing the stripe picture in Figure 3, the spectral response diagram of the average gray value of each row of pixels can be obtained as shown in Figure 7;

Step 4: Detect the K value corresponding to the peak value (maximum value) in R(k) except the low-frequency component, and set it as K _N , then K _N is the number of stripes in the stripe picture. Divide the number of pixels by K _N with the total number of rows M to obtain the number of pixel rows of the light and dark stripe width, that is, the periodic value of the variation law of the pixel gray value of each row of the stripe picture, set Row_N=M/K _N ;

Step 5: Obtain the blinking frequency f LED of the _LED = f _row /ROW_N, where f _row is the row scanning frequency of the CMOS image sensor.

Apparently, the above-mentioned embodiments of the present invention are only examples for clearly illustrating the present invention, rather than limiting the implementation of the present invention. For those of ordinary skill in the art, other changes or changes in different forms can be made on the basis of the above description. It is not necessary and impossible to exhaustively list all the implementation manners here. All modifications, equivalent replacements and improvements made within the spirit and principles of the present invention shall be included within the protection scope of the claims of the present invention.

Claims (1)

1. A method for detecting LED flicker frequency based on CMOS image sensor, is characterized in that, comprises the following steps: S1. Obtain the fringe image of the light source area, and convert the fringe image into a grayscale image; S2. Calculate the average gray value for all the pixels in each line of the stripe picture, assuming that the gray light and dark stripe picture has M rows in total, and store the M average gray values in the array I _Average (n), n=0, 1, ... , M-1; S3, carry out M point DFT operation to I _Average (n), seek the frequency response value R (k) of I _Average (n); in $R\left(k\right)=\mathrm{D.}fT\&lsqb;I_{Average}\left(\mathrm{no}\right)\&rsqb;=\underset{\mathrm{no}=0}{\overset{m-1}{\&Sigma;}}{I}_{Average}\left(\mathrm{no}\right)W_m^{\mathrm{no}k},0\&le;k\&le;m-1,$ is the Fourier coefficient; S4, detect the K value corresponding to the peak value except the low-frequency component in R(k), set it as K _N , K _N is the number of stripes in the stripe picture; then divide the total number of rows M by K _N to obtain the pixels of the width of the light and dark stripes The number of rows, i.e. the periodic value of the change law of the pixel gray value of each row of the stripe picture, is set as Row_N=M/K _N ; S5. Obtain the blinking frequency fLED of the _LED =f _row /ROW_N; wherein f _row is the row scanning frequency of the CMOS image sensor.