KR102146665B1 - The method and apparatus for predicting led color using neural network model in visual-mimo environment - Google Patents

Abstract

As an embodiment of the present invention, an LED color prediction apparatus and method thereof using a neural network model in a VISUAL-MIMO environment may be provided. In the LED color prediction apparatus using a neural network model in a VISUAL-MIMO environment according to an embodiment of the present invention, an LED array identification unit identifying an LED array area from an image acquired in a VISUAL-MIMO environment, and an identified LED array. Based on the LED area detection unit that detects the LED area in the area, the image size adjustment unit that adjusts the size of the image for the detected LED area, the neural network learning unit that applies a neural network model to the adjusted size image and trains it, and the result of learning. Thus, a color predictor for predicting the color of the LED area may be included.

Description

LED color prediction device using neural network model in VISUAL-MIMO environment and its method {THE METHOD AND APPARATUS FOR PREDICTING LED COLOR USING NEURAL NETWORK MODEL IN VISUAL-MIMO ENVIRONMENT}

The present invention relates to an LED color prediction apparatus and method using a neural network model in a VISUAL-MIMO environment, and more particularly, to improve reception performance by using a neural network model (ex. BNN, etc.) in optical communication using LEDs. And it relates to an apparatus and a method to predict the LED color.

The Visual-MIMO communication system is a communication system that enables users to transmit and receive data using visible light so that users do not feel uncomfortable, and multiple inputs in consideration of data quality and data transmission capacity. And it means a visible light communication system to which the MIMO (Multi Input Multi Output) technology to enable multiple outputs is applied. In other words, MIMO technology mainly uses an LED array in the transmitting device to support multiple inputs of the transmitting device and multiple outputs of the receiving device, and the receiving device uses an image sensor capable of multiple inputs to increase data transmission capacity. Or, it is possible to obtain a spatial diversity effect, thereby increasing the efficiency of a visual-beauty (MIMO) communication system.

In color prediction, there have been some techniques of searching for an optimal color representation through machine learning learning from the prior art. However, such a conventional technique is not used to improve the color reception performance of the receiving end in the technical field such as Visual MIMO communication, but simply searches for the optimal color for color reproduction such as a manuscript in a color printer, color monitor, etc. It is only a technique used to express the color required in the water dyeing process.

In light of the trend that the development of related technologies such as autonomous driving is accelerating more and more due to the development of optical communication technology, the necessity of developing a technology for accurately predicting (judging) the color received at the receiving end has emerged strongly.

1. Japanese Laid-Open Patent Publication No. 2002-10095

The present invention relates to an LED color prediction apparatus and method using a neural network model in a VISUAL-MIMO environment. In optical communication using LEDs, a neural network model (ex. BNN, etc.) is used to improve reception performance and LED color. It is intended to provide an apparatus and method for predicting more accurately.

As an embodiment of the present invention, an LED color prediction apparatus and method thereof using a neural network model in a VISUAL-MIMO environment may be provided.

In the LED color prediction apparatus using a neural network model in a VISUAL-MIMO environment according to an embodiment of the present invention, an LED array identification unit identifying an LED array area from an image acquired in a VISUAL-MIMO environment, and an identified LED array. Based on the LED area detection unit that detects the LED area in the area, the image size adjustment unit that adjusts the size of the image for the detected LED area, the neural network learning unit that applies a neural network model to the adjusted size image and trains it, and the result of learning. Thus, a color predictor for predicting the color of the LED area may be included.

In the LED area detection unit, the inner circle area of the image corresponding to each LED included in the LED array area is identified, the outer circle area separated by a predetermined distance from the identified inner circle area is identified, and the identified outer circle area is identified. A figure of a predetermined shape is formed so as to surround it, and an image inside the formed figure of a predetermined shape can be detected as an LED area.

In the image size adjustment unit, the size of the image for the detected LED area may be adjusted to N x N (N is a natural number).

The neural network learning unit applies a neural network model to all pixels of an image adjusted to N x N (N is a natural number) to train, and the neural network model may be a boosting neural network model.

The LED color prediction method using a neural network model in a VISUAL-MIMO environment according to an embodiment of the present invention includes the steps of identifying an LED array area from an image acquired in a VISUAL-MIMO environment, and the LED in the identified LED array area. Detecting an area, adjusting the size of the image for the detected LED area, applying a neural network model to an image of the adjusted size and learning, and predicting the color of the LED area based on the result of the training. It may include steps.

In the step of detecting the LED region in the identified LED array region according to an embodiment of the present invention, the step of identifying an inner circle region of an image corresponding to each LED included in the LED array region, from the identified inner circle region The step of identifying an outer circle area spaced apart by a predetermined distance and forming a shape of a predetermined shape to surround the identified outer circle area are included, and an image inside the formed shape of the predetermined shape is detected as an LED area. I can.

In addition, in the step of adjusting the size of the image for the detected LED area, an operation of adjusting the size of the image for the detected LED area to N×N (N is a natural number) may be performed.

In addition, the step of training by applying the neural network model to the image of the adjusted size includes the step of applying and training the neural network model to all pixels of the image adjusted to N x N (N is a natural number), and the neural network model It may be a boosting neural network model.

Meanwhile, as an embodiment of the present invention, a computer-readable recording medium in which a program for executing the above-described method on a computer is recorded may be provided.

When using the apparatus and method according to an embodiment of the present invention, it is possible to improve the SER than the multiple linear regression (MLR) algorithm. In addition, as the SER is improved, robust symbol determination is possible even with a change in the size of a constellation.

In addition, according to an embodiment of the present invention, a constellation diagram may be arranged to have several different radii for one target color. Accordingly, since a larger number of symbols can be arranged than before, data of one symbol increases, and thus the data transmission speed may increase.

1 is a flowchart illustrating a method of predicting LED color using a neural network model in a VISUAL-MIMO environment according to an embodiment of the present invention.
2 is a flowchart illustrating a process of detecting an LED area in an LED array area identified according to an embodiment of the present invention.
3 is an example of a process of detecting an LED area in an LED array area identified according to an embodiment of the present invention.
4 and 5 illustrate a learning process using a boosting neural network model according to an embodiment of the present invention.
6 is a block diagram illustrating an LED color prediction apparatus using a neural network model in a VISUAL-MIMO environment according to an embodiment of the present invention.
7 shows improved SER performance compared to the Multiple Linear Regression (MLR) algorithm.
8 shows a constellation diagram having different radii for the same target color according to an embodiment of the present invention.

Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings so that those of ordinary skill in the art can easily implement the present invention. However, the present invention may be implemented in various different forms and is not limited to the embodiments described herein. In the drawings, parts irrelevant to the description are omitted in order to clearly describe the present invention, and similar reference numerals are assigned to similar parts throughout the specification.

The terms used in the present specification will be briefly described, and the present invention will be described in detail.

The terms used in the present invention have been selected from general terms that are currently widely used while considering functions in the present invention, but this may vary according to the intention or precedent of a technician working in the field, the emergence of new technologies, and the like. In addition, in certain cases, there are terms arbitrarily selected by the applicant, and in this case, the meaning of the terms will be described in detail in the description of the corresponding invention. Therefore, the terms used in the present invention should be defined based on the meaning of the term and the overall contents of the present invention, not a simple name of the term.

When a part of the specification is said to "include" a certain component, it means that other components may be further included rather than excluding other components unless otherwise stated. In addition, terms such as "... unit" and "module" described in the specification mean units that process at least one function or operation, which may be implemented as hardware or software, or as a combination of hardware and software. . In addition, when a part is "connected" to another part throughout the specification, this includes not only a case in which it is "directly connected", but also a case in which it is connected "with another element in the middle."

Hereinafter, the present invention will be described in detail with reference to the accompanying drawings.

1 is a flowchart showing a method of predicting LED color using a neural network model in a VISUAL-MIMO environment according to an embodiment of the present invention, and FIG. 2 is a flowchart illustrating an LED area in an LED array area identified according to an embodiment of the present invention. It is a flow chart showing the detection process, and FIG. 3 shows an example of a process of detecting an LED area in the identified LED array area according to an embodiment of the present invention.

The LED color prediction method using a neural network model in a VISUAL-MIMO environment according to an embodiment of the present invention includes the step of identifying an LED array area from an image acquired in a VISUAL-MIMO environment (S100), and the identified LED array. Detecting the LED area in the area (S200), adjusting the size of the image for the detected LED area (S300), applying a neural network model to the image of the adjusted size (S400) and training It may include predicting the color of the LED area based on the result (S500).

In the step of detecting the LED area in the identified LED array area (S200) according to an embodiment of the present invention, the step of identifying the inner original area of the image corresponding to each LED included in the LED array area (S210), Identifying the outer circle region spaced apart by a predetermined distance from the identified inner circle region (S220) and forming a shape of a predetermined shape to surround the identified outer circle region (S230), The image inside the shape of the shape can be detected as an LED area.

Referring to FIG. 3, for example, a light image output from a sign or a traffic light including an LED array provided on a road side may be acquired (received) from a vehicle or the like through an image collecting device such as a camera. An LED array area may be identified from the acquired image. The LED array area can be identified using general object detection and tracking techniques. Further, the LED array region may be automatically or manually selected as a region of interest (ROI).

According to an embodiment of the present invention, the detection of the LED area in the identified LED array area (ex. the dotted box portion of FIG. 3) identifies the inner circle area of the image corresponding to each LED included in the LED array area ( ex.(green circle in Fig. 3(a)), identify the outer circle area separated by a predetermined distance from the identified inner circle area (ex. the light blue circle in Fig. 3(b)), and the identified outer circle After forming a shape of a predetermined shape to surround the area (ex. the red box in Fig. 3(b)), an image inside the shape of the predetermined shape (ex. the image of Fig. 3(c)) is the LED area Can be detected as

For example, after estimating the diameter of the inner circle region for each LED, the first color (ex. green) is displayed, and the outer circle region separated by 5 pixels from the inner circle is identified and a second color (ex. sky blue). By forming a shape (eg, a polygon such as a rectangle) to surround the identified outer circle area, an image inside the formed shape may be obtained as an LED area. Depending on the intensity of the LED and the arrangement of the LED array, the distance value between the inner circle and the outer circle may be selected (set) differently from the above-described example.

In addition, according to an embodiment of the present invention, in the step of adjusting the size of the image for the detected LED area (S300), the operation of adjusting the size of the image for the detected LED area to N x N (N is a natural number) This can be done.

In the case of a boosting neural network model, a fixed input size may be required. In consideration of the computational complexity and computational speed, the size of the LED image can be preferably adjusted to 10 x 10. Afterwards, 100 RGB pixels can be used as inputs in the BNN model. Of course, the above-described numerical values are for illustrative purposes only, and may be larger or smaller in consideration of the accuracy and speed of prediction. Also, the size of the image can be adjusted to M x N (M and N are different natural numbers, respectively).

In the step of training by applying a neural network model to an image of an adjusted size according to an embodiment of the present invention (S400), learning by applying a neural network model to all pixels of the image adjusted to N x N (N is a natural number) The step of performing is included, and the neural network model may be a boosting neural network model.

4 and 5 illustrate a learning process using a boosting neural network model according to an embodiment of the present invention.

Boosting is one of the techniques for generating a plurality of classifiers by manipulating initial sample data, and refers to a method of generating a strong classifier by collecting several weak classifiers. In the case of boosting, several sampling data are extracted from the entire data, and the sample weight of the next training data is sequentially adjusted based on the result of the previous training classifier, while the training proceeds. The biggest feature of this boosting is that the weak classifier of the next step is affected by the weak classifier of the previous step. In other words, the next step is performed in a direction in which the previous classification pattern can be identified and better matched, and the process of updating the weight of each classifier is performed. Eventually, a strong classifier is created through several weak classifiers and different weights created by being sequentially combined and affected by each other.

In order to improve the reception performance of VLC (Visible Light Communication), all input pixels of the received LED image can be trained using a neural network model. In a method and apparatus according to an embodiment of the present invention, a received color is more accurately predicted using an ensemble method. The ensemble method means a combination of a number of weak models to form a strong model as described above. Each input pixel can be individually learned and delivered to a weak learner, and more accurate color prediction can be made according to a sequentially combined learning method. Learning starts with the same weight assigned to the raw data, but high weights are given to objects that are incorrectly classified by the predictor variables through modeling in the learning process, and low weights are given to objects that are correctly classified. Therefore, misclassified objects can be better classified.

The overall architecture of the Boosting Neural Network (BNN) model according to an embodiment of the present invention is divided into three parts as follows.

1) Select coefficient of Leaky ReLU activation function

2) Transition learning into weak learners to apply all input pixels to the NN model

3) Train a fully connected NN model by supplying weak learners

Referring to FIGS. 4 and 5 in detail, 1) the coefficient of the Leaky ReLU activation function is determined by an iterative procedure based on the training and validation data set. In this iteration, the coefficient value of Leaky ReLU can be increased by 0.01 from -0.1 to 0.1, for example.

2) Adjust each input pixel according to the output color of each LED image. We use mini-batch learning using back propagation algorithm. At this stage, only the training data set is used and no validation data set is considered. This step is to improve the performance of the '3rd step'. This step may be referred to as a boosting step.

3) A fully connected neural network is learned by inputting the output of'Step 2'. In this step, mini-batch learning using backpropagation algorithm is applied. The training session is then terminated and the validation data set is used to avoid overlearning problems. At this stage, the trained model needs to be considered because it is a low value that may not work properly on a hidden data set, and training RMSE (Root-Mean-Square-Error) is minimized. To prevent this, a hidden data set that monitors the performance of a model trained on an existing hidden data set is used as a validation data set. The validity data set is called a hidden data set because the weights are not updated using the data set.

The formula for the BNN learning algorithm is as follows:

와

는 i 번째 훈련 표본에 대한 각각의 뉴론(ex. 히든 레이어와 출력 레이어)의 Leaky ReLU 활성화 함수의 2 단계 출력이다.

는 입력 레이어로부터 히든 레이어 뉴론으로의 가중치이며,

는 히든 레이어로부터 출력 레이어 뉴론으로의 가중치를 나탄낸다. 식 (3)에서,

는 i 번째 훈련 표본에 대한 히든 레이어의 n 번째 뉴론의 입력이며,

는 입력 레이어로부터 히든 레이어 뉴론으로의 가중치를 나타낸다. Here,'i','m','n','h', and'o' denote a training sample number, an input intensity number, a third-stage hidden layer neuron number, a hidden layer, and an output layer, respectively. And '2' and '3' represent steps 2 and 3, respectively. The training sample number is the LED image number collected from the video for training, and the input pixel number is 1-100. 'C' represents R, G, B color channels. In equations (1) and (2)

Wow

Is the 2-step output of the Leaky ReLU activation function of each neuron (ex. hidden layer and output layer) for the ith training sample.

Is the weight from the input layer to the hidden layer neuron,

Represents the weight from the hidden layer to the output layer neurons. In equation (3),

Is the input of the nth neuron of the hidden layer for the ith training sample,

Represents the weight from the input layer to the hidden layer neurons.

는 3단계의 i 번째 훈련 샘플을 위한 n 번째 뉴론의 Leaky ReLU 활성화 함수의 출력이다. 식 (5)에서

는 i 번째 학습 표본에 대한 출력 레이어 뉴론의 입력이며, n 번째 히든 레이어 뉴론에서 출력 레이어 뉴론 3 단의 가중치이다. (6)에서,

는 3단계의 i 번째 훈련 샘플에 대한 출력 레이어 뉴론의 Leaky ReLU 활성화 함수의 출력이다. 이것은 BNN 모델의 최종 출력이다. 3 색(RGB) 채널의 밝기 강도를 추정할 필요가 있으므로, 3개의 BNN 모델을 사용한다. In equation (4),

Is the output of the Leaky ReLU activation function of the nth neuron for the ith training sample in step 3. In equation (5)

Is the input of the output layer neuron to the i-th training sample, and is the weight of the third stage of the output layer neuron in the n-th hidden layer neuron. In (6),

Is the output of the Leaky ReLU activation function of the output layer neuron for the ith training sample in step 3. This is the final output of the BNN model. Since it is necessary to estimate the brightness intensity of the three-color (RGB) channel, three BNN models are used.

,

,

이다.Therefore, all parameters indicated in equations (1) through (6) are considered for each color channel, and'C' is used for each parameter in all equations. The learnable parameters are

,

,

to be.

를 나타내며,

는 i 번째 표적의 목표 값이다. 식 (8)에서

,

,

,

,

는 각각

의 갱신 된 가중치, 이전 가중치, 제 2 단계의 학습율, 총 훈련 샘플 수 및

에 관한

의 편미분을 나타낸다. 식 (9)의

,

,

는

의 갱신 된 가중치, 이전 가중치 및

에 대한

의 편미분을 각각 나타낸다. 2 단계 학습을 위해

의 값은

로 설정될 수 있다. 이 단계에서는 각 반복에서 가중치를 업데이트하기 위해 총 훈련 샘플 수를 사용하므로 해당 단계의 학습을 전체 일괄 학습이라고 한다.Equation (7) is the cost function for two-step training according to the back propagation algorithm

Represents,

Is the target value of the i-th target. In equation (8)

,

,

,

,

Are each

Updated weights, previous weights, learning rate of the second step, total number of training samples, and

To about

Represents the partial derivative of Of equation (9)

,

,

Is

Updated weights, previous weights and

for

The partial derivatives of are shown respectively. For step 2 learning

Is the value of

Can be set to Since this step uses the total number of training samples to update the weights at each iteration, the training in that step is called full batch training.

를 보여준다. 식 (11)에서

,

,

,

는 각각

의 갱신 된 가중치,

의 이전 가중치,

의 세 번째 단계의 학습 속도 및

의 편미분이다. 식 (12)에서,

,

,

는

의 업데이트 된 가중치,

의 이전 가중치 및

에 대한

의

에 대한 편미분이다. 3 단계 훈련의 경우,

의 값을

로 설정될 수 있다.

와

는 각각 (11)과 (12)에서 사용 된 훈련 샘플의 하한과 상한이다. 이 하한과 상한의 차이는 4이다. 이는 두 번째 단계 훈련과 같이 이 단계에서 가중치를 업데이트하기 위해 각 애포크(epoch)에서 전체 교육 샘플을 사용하지 않는다는 것을 의미한다. 트레이닝 RMSE(Root Mean Square Error) 가 크게 감소하지 않기 때문에 두 번째 단계에서는 미니 배치 훈련을 사용하지 않는다. 따라서, 훈련은 오랜 시간이 걸리지는 않는다.Equation (10) is the cost function for three-stage training according to the back propagation algorithm

Show In equation (11)

,

,

,

Are each

Updated weight of,

Previous weights of,

The learning rate of the third stage of the and

Is the partial derivative of In equation (12),

,

,

Is

Updated weights of,

The previous weight of and

for

of

Is the partial derivative of For the three-level training,

Value of

Can be set to

Wow

Are the lower and upper bounds of the training samples used in (11) and (12), respectively. The difference between this lower limit and upper limit is 4. This means that we are not using the entire training sample at each epoch to update the weights at this stage, like the second stage training. Since the training root mean square error (RMSE) is not significantly reduced, mini-batch training is not used in the second step. Therefore, training does not take long.

6 is a block diagram illustrating an LED color prediction apparatus using a neural network model in a VISUAL-MIMO environment according to an embodiment of the present invention.

In the LED color prediction apparatus 1000 using a neural network model in a VISUAL-MIMO environment according to an embodiment of the present invention, an LED array identification unit 100 for identifying an LED array area from an image acquired in a VISUAL-MIMO environment. ), an LED area detection unit 200 that detects an LED area in the identified LED array area, an image size adjustment unit 300 that adjusts the size of an image for the detected LED area, and a neural network model is applied to an image of the adjusted size. A neural network learning unit 400 for learning by doing so and a color prediction unit 500 for predicting a color of an LED area based on a result of the learning may be included.

In the LED area detection unit 200, the inner circle area of the image corresponding to each LED included in the LED array area is identified, the outer circle area separated by a predetermined distance from the identified inner circle area is identified, and the identified outer circle area A shape of a predetermined shape is formed so as to surround the original area, and an image inside the shape of the predetermined shape may be detected as an LED area.

In the image size adjustment unit 300, the size of the image for the detected LED area may be adjusted to N x N (N is a natural number).

The neural network training unit 400 applies a neural network model to all pixels of an image adjusted to N x N (N is a natural number) to train, and the neural network model may be a boosting neural network model.

7 shows improved SER performance compared to the Multiple Linear Regression (MLR) algorithm, and FIG. 8 shows constellation diagrams having different radii for the same target color according to an embodiment of the present invention.

Referring to FIG. 7, a method and apparatus according to an embodiment of the present invention may improve SER than a multiple linear regression (MLR) algorithm. In addition, as shown in FIG. 8, robust symbol determination is possible even with a change in the size of the constellation diagram as the SER is improved.

FIG. 8 shows different sizes of constellation diagrams of the same color that can be applied using the BNN algorithm. As shown in FIG. 8, when constellation diagrams having several different radii can be arranged for one target color, In addition, since a larger number of symbols can be arranged than before, data that one symbol has (representable) increases, so that the data transmission speed can be increased compared to the prior art.

In relation to the apparatus according to an embodiment of the present invention, the contents of the above-described method may be applied. Accordingly, description of the same contents as those of the above-described method in relation to the apparatus is omitted.

Meanwhile, the above-described method can be written as a program that can be executed on a computer, and can be implemented in a general-purpose digital computer that operates the program using a computer-readable medium. Further, the structure of the data used in the above-described method may be recorded on a computer-readable medium through various means. A recording medium for recording executable computer programs or codes for performing various methods of the present invention should not be understood as including temporary objects such as carrier waves or signals. The computer-readable medium may include a storage medium such as a magnetic storage medium (eg, ROM, floppy disk, hard disk, etc.), and an optical reading medium (eg, CD-ROM, DVD, etc.).

The above description of the present invention is for illustrative purposes only, and those of ordinary skill in the art to which the present invention pertains will be able to understand that it can be easily modified into other specific forms without changing the technical spirit or essential features of the present invention. will be. Therefore, it should be understood that the embodiments described above are illustrative and non-limiting in all respects. For example, each component described as a single type may be implemented in a distributed manner, and similarly, components described as being distributed may also be implemented in a combined form.

The scope of the present invention is indicated by the claims to be described later rather than the detailed description, and all changes or modified forms derived from the meaning and scope of the claims and their equivalent concepts should be interpreted as being included in the scope of the present invention. do.

1000: LED color prediction device using a neural network model in a VISUAL-MIMO environment
100: LED Array identification unit
200: LED area detection unit
300: image resizing unit
400: neural network learning unit
500: color prediction unit

Claims (4)

As an LED color prediction device using a neural network model in a VISUAL-MIMO environment,
An LED array identification unit for identifying an LED array area from the image acquired in the VISUAL-MIMO environment;
An LED area detection unit detecting an LED area in the identified LED array area;
An image size adjusting unit for adjusting the size of the image for the detected LED area;
A neural network learning unit for training the image of the adjusted size by applying a neural network model; And
Includes a color predictor for predicting the color of the LED area based on the result of the learning,
In the LED area detection unit, an inner circle area of an image corresponding to each LED included in the LED array area is identified, an outer circle area separated by a predetermined distance from the identified inner circle area is identified, and the identified An LED color prediction apparatus using a neural network model in a VISUAL-MIMO environment, characterized in that a shape of a predetermined shape is formed to surround an outer circle area, and an image inside the shape of the predetermined shape is detected as an LED area.
delete The method of claim 1,
The image size adjustment unit, the LED color prediction apparatus using a neural network model in a VISUAL-MIMO environment, characterized in that the size of the image for the detected LED area is adjusted to N x N (N is a natural number).
The method of claim 3,
In the neural network learning unit, the neural network model is applied to all pixels of the image adjusted to N x N (N is a natural number) and trained, and the neural network model is a boosting neural network model. LED color prediction device using neural network model in VISUAL-MIMO environment.

Images