KR102123166B1 - A color modulation based visual light communication method and apparatus - Google Patents

Abstract

According to the present invention, disclosed are a color modulation-based visible light communication apparatus and a method thereof. According to the present invention, provided is the color modulation-based visible light communication apparatus comprising: an input unit for receiving an original frame; a color modulation unit for generating a color modulated frame for hiding data by color modulating the original frame based on a MacAdam model; and a display unit for displaying the color modulated frame at a predetermined refresh rate. A receiving device disposed adjacent to the display unit captures the color modulated frame through a camera, and decodes the hidden data in the captured frame.

Description

A color modulation based visual light communication method and apparatus

The present invention relates to a color modulation-based visible light communication method and apparatus, and more particularly, to a method and apparatus for allowing a display to simultaneously perform an image reproduction function and an information transmission function.

In the field of information hiding, techniques for inserting additional information into an encrypted image and techniques for protecting information by intentionally distorting the image (eg, watermarking) have been continuously studied.

CSC (Camera-Screen Communication) is a short-range optical information concealment technology that uses an LED or OLED display and a digital camera.

This technique alternately changes the state of the screen at a high refresh rate and sends a bit stream of the message. The optical receiver reconstructs the data by detecting minute changes in light emitted from the screen.

CSC must meet two major challenges at the same time to achieve a stable realization. That is, it should not interfere with a user who transmits a bit stream with high transmission accuracy and views an image. .

1 is a view showing the basic concept of the CSC.

As shown in FIG. 1, a complementary frame as shown in FIG. 1 represents a pair of sequential frames in a video stream that can be used to modulate data '1' or '0'.

가 주어질 때, 은닉 비트 '1'은

를 2개의 상보적 프레임

으로 분해하고, 비트 '0' 원본 프레임의 복제

함에 의해 삽입될 수 있다.Original frame

When is given, the hidden bit '1' is

2 complementary frames

Decompose into, and duplicate the original frame with bit '0'

Can be inserted.

및 캐리어 이미지를 통해 전송되는 2D 데이터 블록의 예를 도시한 것이다. Figure 2 (a) is a gray image

And an example of a 2D data block transmitted through a carrier image.

가 클수록 전송 정확도는 높아지지만 변조된 프레임

이 원본 프레임

대신 교대로 디스플레이될 때 눈에 띄는 시각적 깜박임이 커져 시청자에게 미치는 영향이 커진다. As shown in Figure 2, the modulation size

The greater the transmission accuracy, the higher the modulated frame.

This original frame

Instead, noticeable visual flickering when displayed alternately increases the impact on the viewer.

Korea Registered Patent 10-1794933

In order to solve the above-mentioned problems of the prior art, the present invention is to propose a color modulation-based visible light communication method and apparatus that can increase transmission accuracy while maintaining visual quality that does not interfere with user viewing in CSC.

In order to achieve the above object, according to an embodiment of the present invention, a color modulation-based visible light communication device, the input unit for receiving an original frame; A color modulator for color-modulating the original frame based on the MacAdam model to generate a color-modulated frame for data hiding; It includes a display unit for displaying the color modulation frame at a predetermined refresh rate (refresh rate), the receiving device disposed adjacent to the display unit captures the color modulation frame through the camera, and decodes the hidden data in the captured frame A color modulation based visible light communication device is provided.

The MacAdam model includes a plurality of MacAdam ellipses for a predetermined color, and the color modulator may generate a color modulation vector for the input color of the original frame through vector weighting with each of the plurality of MacAdam ellipses. .

The weight for the vector weighted sum may be calculated through the center of each of the plurality of MacAdam ellipses and the Euclidean distance of the input color.

The original frame is divided into a predetermined number of blocks, and the color modulator may generate a color modulated frame in block units.

The color modulator generates a pair of complementary frames by color-modulating one original frame,

The capture rate of the receiving device is n times the refresh rate (n is a natural number of 2 or more), and the receiving device extracts the pair of complementary frames from a plurality of capture frames for one original frame, and extracts the Data can be decoded by analyzing the complementary frames of the pair.

The receiving device may extract the pair of complementary frames using the color average value of the plurality of capture frames.

The receiving device may perform the data decoding through a network learned through a neural network.

According to another aspect of the present invention, a color modulation-based visible light communication device, a camera capturing a color modulation frame displayed at a predetermined refresh rate by a transmitting device, wherein the color modulation frame is a pair of colors generated by color modulation of one original frame Including complementary frames-; A complementary frame extraction unit extracting a complementary frame from the captured frame; And a data decoding unit decoding the hidden data using the extracted complementary frame.

According to another aspect of the present invention, a color modulation based visible light communication method comprising: receiving an input of receiving an original frame; Generating a color modulation frame for data hiding by color-modulating the original frame based on the MacAdam model; Displaying the color modulation frame at a predetermined refresh rate; Capturing the color modulation frame through a camera; And decoding the hidden data in the captured frame.

According to the present invention, the color-based image modulation technique has an advantage in that the user perceived image quality is superior to the conventional brightness modulation technique, thereby greatly reducing rejection and ensuring high transmission efficiency.

1 is a view showing the basic concept of the CSC.
2 illustrates an example of a 2D data block transmitted through a gray image and a carrier image.
3 is a view showing the configuration of a color modulation-based visible light communication system according to an embodiment of the present invention.
4 is a diagram showing the configuration of a transmission device according to the present embodiment.
5 shows the MacAdam ellipse with the outline of the elliptic chromaticity diagram.
FIG. 6 shows the MacAdam ellipse converted to a Lab color space.
7 shows an example of color modulation for any input color.
8 is a diagram showing the configuration of a receiving device according to the present embodiment.
FIG. 9 shows alternate display of the original frame and a pair of color-modulated complementary frames at a predetermined refresh rate.
10 shows an example of an inter-block crosstalk and weighted voting method occurring in the captured frame.

The present invention can be applied to various changes and can have various embodiments, and specific embodiments will be illustrated in the drawings and described in detail.

However, this is not intended to limit the present invention to specific embodiments, and should be understood to include all modifications, equivalents, and substitutes included in the spirit and scope of the present invention.

The present invention is based on the fact that the human visual system is less sensitive to changes in color than changes in light brightness. In fact, cone cells that sense color are less sensitive than rods that sense brightness.

Accordingly, the present invention expresses transmission data only by color modulation while maintaining the same brightness as the original image in the CSC technique. To this end, according to this embodiment, a MacAdam ellipse is used for color modulation.

Hereinafter, a visible light communication method according to the present embodiment will be described in detail with reference to the drawings.

3 is a view showing the configuration of a color modulation-based visible light communication system according to an embodiment of the present invention.

As shown in FIG. 3, the system according to the present embodiment is a color modulation of the original frame to transmit the color modulation frame for the concealed data (bit) and the transmission device 300 and capture the modulated frame to the color modulation frame It may include a receiving device 302 for decoding the included hidden data.

The transmitting device 300 and the receiving device 302 may each include a processor and memory.

The processor may include a central processing unit (CPU) capable of executing computer programs or other virtual machines.

The memory may include non-volatile storage devices such as fixed hard drives or removable storage devices. The removable storage device may include a compact flash unit, a USB memory stick, and the like. The memory may also include volatile memory, such as various random access memories.

Program instructions executable by the processor are stored in the memory.

4 is a diagram showing the configuration of a transmission device according to the present embodiment.

As illustrated in FIG. 4, the transmitting device 300 includes an input unit 400 that receives an original frame, and a color modulator for color-modulating the original frame based on a MacAdam model to generate a color modulation frame for data concealment ( 402) and a display unit 404 for reproducing the color modulation frame at a predetermined refresh rate.

According to a preferred embodiment of the present invention, the color modulator 402 generates a complementary frame in which the color of the original frame is modulated based on the MacAdam model, considering that the human vision is less sensitive to color changes. At this time, the color modulator 402 generates a pair of complementary frames whose colors are modulated with respect to one original frame.

At this time, the original frame is divided into a plurality of blocks, and the color modulator 402 generates a color modulated frame including hidden data in block units.

Since the MacAdam ellipse has only 25 ellipses for some of the representative colors, if an input color that does not belong to the MacAdam model exists in the original frame, the color modulation vector is estimated by weighting the 25 vectors of the MacAdam ellipse.

Here, the weight for vector weighting is calculated through the Euclidean distance of the center and input color of each of the plurality of MacAdam ellipses.

5 shows the MacAdam ellipse with the outline of the elliptic chromaticity diagram.

The outline of the MacAdam ellipse represents just a noticeable difference (JND) that can discern a change in chromaticity. This means that the color of each elliptic region cannot be distinguished from the center color of the ellipse by the average human eye.

According to a preferred embodiment of the present invention, the color modulator 402 modulates data by converting a MacAdam ellipse to an L _ab color space, as shown in FIG. 6.

According to the present embodiment, the MacAdam ellipse in the L _ab color space represents a boundary in which a human does not recognize the color difference with respect to the center color, and thus results of color modulation of the color of both ends in the long axis direction having the maximum distance within the ellipse. Use values.

Through such color modulation, it is possible to improve the transmission accuracy while maintaining a high cognitive image quality by using the maximum color distance that is invisible to humans.

Since the MacAdam ellipse has only 25 ellipses for some representative colors, if there is an input color that does not belong to the MacAdam model, the color modulation vector is estimated by weighting the 25 vectors of the MacAdam ellipse.

The following equation is for predicting the direction and size of color modulation for the input color.

는 임의의 색상 포인트 x에서의 색상 변조 벡터이다.

와

는 각각

번째 MacAdam 타원의 가중치와 벡터를 나타낸다.here,

Is a color modulation vector at any color point x.

Wow

Each

Represents the weight and vector of the second MacAdam ellipse.

The weight of each ellipse is calculated based on the Euclidean distance between the input color and the center of the MacAdam ellipse. It is expressed as follows.

는 입력 색상 포인트

에서

번째 타원의 중심까지의 유클리드 거리를 나타낸다.here,

Input color point

in

Denotes the Euclidean distance to the center of the ellipse.

7 shows an example of color modulation for any input color.

표시는 입력 색상의 변조된 색상 위치를 나타낸다.The black circle at the bottom left of FIG. 7 indicates the location of the input color,

The mark indicates the modulated color position of the input color.

As shown in FIG. 7 and Equation 2, the distance between the input color and the center of the MacAdam ellipse indicates the similarity with the new ellipse. The closer the distance, the more similar the color, and the greater the weight and orientation of the input color.

의 변조 색상과 위치

의 다른 변조 색상은 본래 입력 색상 대신 높은 재생률(refresh rate)로 디스플레이에 교대로 표시된다.location

Modulation color and position

The different modulation colors of are alternately displayed on the display at a high refresh rate instead of the original input color.

블록으로 분할된 원본 프레임을 블록 단위로 색상 변조하여 은닉 비트 '1'을 전달한다.On the other hand, when actually implementing the color modulation according to this embodiment for 2D data block embedding,

The original frame divided into blocks is color-modulated in block units to transmit the hidden bit '1'.

To pass bit '0', the block is duplicated. In order to mitigate inter-block crosstalk that may occur at the boundary between the '0' block and the '1' block, a spatial smoothing method is additionally applied by resizing the color modulation vector.

In this embodiment, it is considered that the transmission device 300 has a higher refresh rate than the conventional case. MacAdam ellipses are modeled at normal refresh rates, allowing for greater color modulation with higher refresh rates.

That is, a larger color modulation than the original JNCD of the MacAdam model may be allowed, and more chrominance differences are allowed to improve data reconstruction accuracy at the receiver.

Through various experiments, it was confirmed that the maximum size of the color modulation vector is preferably set to 5 times ( 5δ ) of the original MacAdam model at the center of the spatial block. Its length gradually decreases as it moves to the boundary ( δ ).

8 is a diagram showing the configuration of a receiving device according to the present embodiment.

As shown in FIG. 8, the reception device 302 according to the present embodiment may include a camera 800, a complementary frame extraction unit 802 and a data decoding unit 804.

The camera 800 captures a color modulation frame that the display unit 404 reproduces at a predetermined refresh rate at a predetermined capture rate.

According to the present embodiment, the capture rate may be n times the refresh rate of the display unit 404, and accordingly, a plurality of capture frames may be obtained for one original frame.

For example, when the capture rate is twice the refresh rate, considering that a pair of complementary frames are generated for one original frame, the receiving device 302 generates four capture frames for one original frame Order.

The complementary frame extracting unit 802 extracts a pair of complementary frames using the RGB average values of the four capture frames.

과 색상 변조된 한 쌍의 상보적 프레임

및

을

의 재생률로 교대로 표시하는 것을 도시한 것이다. 9 is the original frame

And color modulated pair of complementary frames

And

of

It is shown that the alternating rate is displayed alternately.

9C illustrates a case in which the display unit 404 and the camera 800 are ideally synchronized.

Ideally, every frame captured without wasting resources is used to decode the hidden bits.

9D is a case where synchronization is not performed.

The rolling shutter effect generates frames captured at different moments within a single frame, making it difficult to extract hidden data through sequential comparison of complementary frames.

가 화면 재생률

의 두 배인 경우를 도시한 것이다. Case 3 of FIG. 9E shows the camera capture speed

There screen refresh rate

It shows the case of twice.

Some of the captured frames (denoted by'X' in FIG. 9) are not useful for decoding due to the rolling shutter effect, but the remaining frames (denoted by'O' in FIG. 9) can successfully capture the entire original frame. .

Case 4 of Figure 9f shows the situation due to the unstable camera capture speed. Even in this case, it is difficult to extract a complementary frame.

에 대해 4개의 캡쳐된 프레임이 존재한다. 여기서, 'O'로 표시된 두 개의 프레임

은 상보적 프레임이며, 두 개의 혼합 프레임은 디스플레이의 프레임 전환에만 사용된다. Referring back to FIG. 9E, when the capture speed is twice the refresh rate, one original frame

There are 4 captured frames for. Here, two frames marked with'O'

Is a complementary frame, and the two mixed frames are used only for frame switching of the display.

According to this embodiment, an RGB average value of each captured frame is used to extract two mixed frames (X) of four captured frames for one original frame.

The two mixed frames have RGB values close to the average of the four captured frames, and two other frames having RGB values far from the four averages are extracted as complementary frames (O).

The data decoding unit 804 extracts hidden data from the extracted complementary frame.

According to the present embodiment, the data decoding unit 804 performs data decoding based on a neural network.

For learning of the data decoding unit 804, the extracted complementary frames and their Euclidean distances are input to the neural network, and the neural network learns the relationship between the color modulated hidden bits captured by the camera.

According to this embodiment, neural network based decoding is performed on a pixel basis, and the decoding results of each block are combined for data reconstruction using a weighted voting method.

Since different colors have different color modulation vector sizes, the colors of the complementary frames captured by the camera have different distances depending on the input color, and a linear classifier or linear support vector machine (SVM) is applied to the decoding of various image blocks. There is something difficult to do.

In addition, the display has a playable color range unique to each device, so that all modulated colors may not be reproduced properly in the display unit. As a result, it may be difficult to capture the same color as the modulated color in the transmitting device 300.

Therefore, color distortion that may occur during the display of the color modulation frame must be considered in the decoding process.

Accordingly, according to the present embodiment, a neural network based decoding method using a feed forward neural network to extract hidden data is proposed.

와

의 픽셀 RGB값

와

및 이들의 유클리드 거리가 신경망으로 입력된다. 신경망은 카메라가 캡쳐한 색상과 은닉 비트 간의 관계를 학습한다.Complementary frame captured

Wow

Pixel RGB value of

Wow

And their Euclidean distances are input to the neural network. The neural network learns the relationship between the color captured by the camera and the hidden bit.

That is, a modulated color distance between complementary frames is generated for the input of the RGB values captured by the camera.

The neural network according to this embodiment may be used to determine whether a pair of pixels in a complementary frame are the same using a color distance.

Neural network based decoding is performed on a pixel basis, and the decoding results of each block are combined for data reconstruction using a weighted voting method described below.

According to this embodiment, data is encoded in units of blocks. Also, because of the color dependence of JNCD in the MacAdam model, the modulation size depends on the input color. Therefore, blocking artifacts may occur in the color modulation frame. For the same reason, as in FIG. 10A, inter-block crosstalk is observed in the frame captured by the camera 800 of the receiving device 302.

To reduce blocking, spatial smoothing is applied to the size of the color modulation.

According to this embodiment, in the case of a uniform color block in the camera capture frame, it was confirmed that the center color and the border color are different because the border portion is affected by the color of the neighboring block.

In order to alleviate such inter-block interference, a weighted voting method is proposed.

Pixel-by-pixel demodulation results are combined with weighted voting results. A positive '1' is assigned to a pixel whose demodulation result is '1', and a negative'-1' is assigned to a pixel whose result is '0'. The sum of the numbers allocated within the block provides the decoding result of the block.

That is, if the sum is positive, the hidden bit of the block is '1', otherwise the hidden bit is '0'.

는 2D 가우스 함수이며 블록의 중앙에 위치한다. 그러면 블록의 디코딩 결과는 다음과 같이 주어진다. D denotes a matrix having pixel-wise demodulation results ('1'and'-1'),

Is a 2D Gaussian function and is located in the center of the block. Then, the decoding result of the block is given as follows.

은 블록의 디코딩 결과이고,

는 단위 함수이고,

와

는 블록 내의 행과 열 인덱스이다.here,

Is the decoding result of the block,

Is a unit function,

Wow

Is the row and column index within the block.

10B shows an example of a weighted voting method.

As shown in the gray image of the pixel-by-pixel decoding result, the pixels in the border portion are affected by the color of the neighboring block and exhibit different results from the pixel in the center of the block. Such a weighted voting decoding method can reduce crosstalk between blocks that reduces transmission accuracy.

The above-described embodiments of the present invention have been disclosed for purposes of illustration, and those skilled in the art having various knowledge of the present invention will be able to make various modifications, changes, and additions within the spirit and scope of the present invention. It should be regarded as belonging to the following claims.

Claims (9)

A color modulation based visible light communication device,
An input unit that receives an original frame;
A color modulator for color-modulating the original frame based on the MacAdam model to generate a color-modulated frame for data hiding;
It includes a display unit for displaying the color modulation frame at a predetermined refresh rate (refresh rate),
The receiving device disposed adjacent to the display unit captures the color modulation frame through a camera, decodes the hidden data in the captured frame,
The MacAdam model includes a plurality of MacAdam ellipses for a given color,
The color modulator generates a color modulation vector for the input color of the original frame through vector weighting with each of the plurality of MacAdam ellipses,
The weight for the vector weighting is a color modulation-based visible light communication device calculated through the center of each of the plurality of MacAdam ellipses and the Euclidean distance of the input color. delete delete According to claim 1,
The original frame is divided into a predetermined number of blocks,
The color modulation unit is a color modulation-based visible light communication device for generating a color modulation frame in block units. According to claim 1,
The color modulator generates a pair of complementary frames by color-modulating one original frame,
The capture speed of the receiving device is n times the refresh rate (n is a natural number of 2 or more),
The receiving device extracts the pair of complementary frames from a plurality of captured frames for one original frame, analyzes the extracted pair of complementary frames, and decodes data based on color modulation. The method of claim 5,
The reception device extracts the pair of complementary frames using the color average value of the plurality of capture frames. According to claim 1,
The reception device is a color modulation-based visible light communication device that performs the data decoding through a network learned through a neural network. delete A color modulation based visible light communication method,
An input receiving step of receiving an original frame;
Generating a color modulation frame for data concealment by color-modulating the original frame based on the MacAdam model;
Displaying the color modulation frame at a predetermined refresh rate;
Capturing the color modulation frame through a camera; And
Decoding the concealed data in the captured frame,
The MacAdam model includes a plurality of MacAdam ellipses for a given color,
The color modulator generates a color modulation vector for the input color of the original frame through vector weighting with each of the plurality of MacAdam ellipses,
The weight for the vector weighting is a color modulation based visible light communication method calculated through the center of each of the plurality of MacAdam ellipses and the Euclidean distance of the input color.

Images