KR20190051619A - Image data processing apparatus using ensamble and fine tunning and controlling method thereof - Google Patents

Abstract

Disclosed are an image data processing apparatus and a control method thereof. The control method comprises the steps of: receiving image data; calculating an average value of output values outputted from a plurality of CNNs based on the received image data; calculating a loss value of an average model including the CNNs included in the average value calculation and calculating an auxiliary loss value for each CNN included in the average value calculation; calculating a total loss value based on the calculated loss value of the average model and the calculated auxiliary loss value; and performing fine-tuning on the average value based on the calculated total loss value.

Description

TECHNICAL FIELD [0001] The present invention relates to an image data processing apparatus and an image data processing method using an ensemble and a fine adjustment,

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to an image data processing apparatus and a control method, and more particularly, to an apparatus and method for processing image data using an ensemble and fine adjustment that ensemble various network models.

The development of intelligent traffic surveillance system has recently become an important issue. One of the most important functions of the intelligent traffic surveillance system is to analyze and extract from the recognized image data. In particular, the technique of classifying and localizing vehicles or pedestrians is a basic exaggeration for traffic surveillance analysis.

Conventional technologies for classifying and localizing vehicles capture images at regular intervals by a number of traffic cameras to obtain an actual traffic detection environment. Then, the obtained images are analyzed to classify and localize the vehicles. However, the accuracy of the analysis results of existing technologies is not satisfactory.

In order to increase the accuracy of image recognition, there is a method of using a plurality of image recognition network models. However, although the image recognition performance can be improved by averaging the output values of a plurality of network models, the synergy effect between the networks is not maximized, which is not efficient. Accordingly, there is a need for an efficient image recognition apparatus by using a plurality of network models to improve image recognition performance and maximize synergy effects.

SUMMARY OF THE INVENTION The present invention has been made to solve the above problems, and it is an object of the present invention to provide an image data processing apparatus and a control method for improving image recognition performance and maximizing synergy among a plurality of network models.

According to an embodiment of the present invention, there is provided a method of controlling an image data processing apparatus, the method including receiving image data, calculating an average value of output values output from a plurality of CNNs based on the input image data, A loss value calculating step of calculating a loss value of an average model including a plurality of CNNs included in the average value calculation and an auxiliary loss value for each of a plurality of CNNs included in the average value calculation, Calculating a total loss value based on the calculated loss value and the calculated auxiliary loss value; and finely adjusting the average value based on the calculated total loss value.

The loss value of the average model is obtained by normalizing the input image data and calculating a loss value of an average value of output values output from each of the plurality of CNNs included in the average value calculation based on the normalized image data and the learned weight parameter Lt; / RTI >

Meanwhile, in the step of calculating the average value, CNNs arbitrarily selected from all the plurality of CNNs may be excluded and the average value may be calculated.

The step of calculating the average value may repeat the process of arbitrarily selecting and excluding CNN and calculating the average value.

Also, the step of calculating the average value may uniformly select CNNs to be excluded with a predetermined probability among the plurality of CNNs.

According to an aspect of the present invention, there is provided an image data processing apparatus including an input unit for inputting image data, a plurality of CNNs for performing learning based on the input image data, Wherein the control unit calculates a loss value of an average model including a plurality of CNNs included in the average value calculation and an auxiliary loss value for each of a plurality of CNNs included in the average value calculation, Calculates a total loss value based on the calculated loss value of the average model and the calculated auxiliary loss value, and finely adjusts the average value based on the calculated total loss value.

The loss value of the average model is obtained by normalizing the input image data and calculating a loss value of an average value of output values output from each of the plurality of CNNs included in the average value calculation based on the normalized image data and the learned weight parameter Lt; / RTI >

Meanwhile, the control unit may exclude arbitrarily selected CNNs from all the plurality of CNNs and calculate the average value.

The control unit may repeat the process of arbitrarily selecting and excluding CNN and calculating the average value.

In addition, the controller can uniformly select CNNs to be excluded with a predetermined probability among the plurality of CNNs.

As described above, according to various embodiments of the present disclosure, an image data processing apparatus and a control method can improve image recognition performance and maximize synergy among a plurality of network models.

1 is a block diagram of an image data processing apparatus according to an embodiment of the present disclosure.
2 is a diagram illustrating a fine adjustment for network convergence according to one embodiment of the present disclosure;
3 is a diagram illustrating a network convergence model according to an embodiment of the present disclosure;
4 is a view for explaining a region-based image data processing process according to an embodiment of the present disclosure.
5 is a diagram illustrating a multi-network convergence model for an area-based data processing apparatus according to an embodiment of the present disclosure;
6 is a diagram showing the results of the precision and the recall of the network fusion model.
7 is a flowchart of an image data processing apparatus according to an embodiment of the present disclosure.

Various embodiments will now be described in detail with reference to the accompanying drawings. The embodiments described herein can be variously modified. Specific embodiments are described in the drawings and may be described in detail in the detailed description. It should be understood, however, that the specific embodiments disclosed in the accompanying drawings are intended only to facilitate understanding of various embodiments. Accordingly, it is to be understood that the technical idea is not limited by the specific embodiments disclosed in the accompanying drawings, but includes all equivalents or alternatives falling within the spirit and scope of the invention.

 Terms including ordinals, such as first, second, etc., may be used to describe various elements, but such elements are not limited to the above terms. The above terms are used only for the purpose of distinguishing one component from another.

In this specification, the terms " comprises " or " having ", and the like, are intended to specify the presence of stated features, integers, steps, operations, elements, parts, or combinations thereof, But do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, or combinations thereof. It is to be understood that when an element is referred to as being "connected" or "connected" to another element, it may be directly connected or connected to the other element, . On the other hand, when an element is referred to as being "directly connected" or "directly connected" to another element, it should be understood that there are no other elements in between.

In the meantime, " module " or " part " for components used in the present specification performs at least one function or operation. Also, " module " or " part " may perform functions or operations by hardware, software, or a combination of hardware and software. Also, a plurality of " modules " or a plurality of " parts ", other than a " module " or " part " which is to be performed in a specific hardware or performed in at least one control section, may be integrated into at least one module. The singular expressions include plural expressions unless the context clearly dictates otherwise.

In addition, in the description of the present invention, when it is judged that the detailed description of known functions or constructions related thereto may unnecessarily obscure the gist of the present invention, the detailed description thereof will be abbreviated or omitted.

1 is a block diagram of an image data processing apparatus according to an embodiment of the present disclosure.

Referring to FIG. 1, an apparatus 100 for processing image data includes an input unit 110, a plurality of CNNs (Covolutional Neural Networks) 120, and a controller 130. The input unit 110 receives image data. For example, the image data may be an image including a vehicle. For example, the input unit 110 may include a communication module connected to the communication interface, an input terminal connected to the input interface, and the like. That is, when the input unit 110 is implemented as a communication module, the image data processing apparatus 100 can receive image data using a wire / wireless communication method. Alternatively, when the input unit 110 is implemented as an input terminal, the image data processing apparatus 100 may receive image data from the internal and external storage devices.

The plurality of CNNs 120 may include at least two CNNs 120-1, 120-2, and 120-3. Each of the plurality of CNNs 120 receives the input image data and performs learning. A plurality of CNNs 120 can improve recognition performance of image data through a fine adjustment process and an ensemble process in a learning process. The fine adjustment process is a process of correcting an error to increase the recognition rate of an image, and the ensemble process is a process of averaging output values of a plurality of CNNs.

The control unit 130 calculates an average value of the output values of the firstarily learned output from the plurality of CNNs 120. [ Then, the control unit 130 calculates the loss value of the average model including the plurality of CNNs included in the average value calculation and the auxiliary loss value for each of the plurality of CNNs included in the average value calculation. The video data processing apparatus 100 can exclude the output values of some CNNs from all the CNNs in the ensemble process. That is, the control unit 130 can calculate an average value obtained by averaging the output values of all the CNNs 120 in the average value calculation, and can calculate the average value of at least one of the first to the n-th CNNs 120-1, 120-2, CNN may be excluded and the average value of the output values of the remaining CNNs may be calculated. The control unit 130 can arbitrarily select the CNN to be excluded. The control unit 130 may repeat the average value calculation process of the CNN output value, and may select and discard different CNNs each time it is repeated. Of course, the CNN used for the average value calculation in the repeated average calculation process may include the same CNN. The control unit 130 can uniformly select CNNs excluded from the average value calculation with a predetermined probability. The concrete ensemble process will be described later.

The controller 130 calculates the total loss value based on the calculated loss value of the average model and the calculated auxiliary loss value, and finely adjusts the average value based on the calculated total loss value (Joint fine tuning). For example, the loss value of the average model may be a loss value for an average value of output values output from each of the plurality of CNNs included in the average value calculation based on the normalized image data and the learned weight parameter, have. A detailed fine tuning procedure will be described later.

2 is a diagram illustrating a fine adjustment for network convergence according to one embodiment of the present disclosure;

Referring to FIG. 2, a plurality of CNNs and loss values are shown. As an example, CNN for classification tasks can be based on pre activation ResNet with 18 layers applying the shortest distance connection. The image data for learning may be an image having a size of 224 x 224 x 3. Since the size of the image data for learning varies, the controller 130 may resize the image data for each learning and randomly cut the image data to 224 × 224 × 3. The RGB pixels of each image data can be normalized as shown in the following equation.

---------- (1)

---------- (One)

Here, x (i, j) may be a vector having RGB pixel values at (i, j) coordinates. Photometric distortion and color augmentation can be applied to achieve good results. Each CNN uses ReLU (Rectified Linear Unit) as an activation function and batch normalization can be used for each layer. The number of outputs of the network may be equal to the number of categories of the data set, and may be 11, for example. A soft max layer may be applied to the final output of each CNN.

는 다음과 같이 정의될 수 있다.The image data processing apparatus may perform a fine tuning process for ensuring some CNNs of all the CNNs. The image data processing device calculates the total loss value based on the loss value of the average model and the calculated auxiliary loss value. Total loss

Can be defined as follows.

---------- (2)

---------- (2)

는 평균 모델의 손실 함수이다. 그리고,

는 i번째 CNN의 보조(auxiliary) 손실이다. 손실값은 학습 단계에서만 사용될 수 있다. o는 다음과 같이 정의된다.Where N is the number of CNN models,

Is the loss function of the mean model. And,

Is the auxiliary loss of the ith CNN. The loss value can only be used in the learning phase. o is defined as follows.

---------- (3)

---------- (3)

는 w_i의 학습된 가중치 파라미터를 가지는 i번째 CNN의 출력이고,

는 식(1)에 의해 정규화된 입력 데이터를 나타낸다. 각 네트워크는

를 최소화하기 위해 학습 비율

의 아주 작은 값을 사용하여 미세 조정될 수 있다. 예를 들어,

= 1e^-5일 수 있다. 가중치 파라미터의 값은 초기에 임의의 값일 수 있다. 그리고, 가중치 파라미터는 학습에 따라 가변되고, 역전파 과정을 통해 학습이 종료될 때까지 계속적으로 수정될 수 있다.here,

Is the output of the i-th CNN with the learned weight parameter of w _i ,

Represents the input data normalized by equation (1). Each network

To minimize the learning rate

Can be fine-tuned using a very small value of < RTI ID = 0.0 > E.g,

= 1e ^-5 . The value of the weight parameter may initially be any value. Then, the weight parameter is varied according to the learning, and can be continuously corrected until the learning is ended through the back propagation process.

)이 산출될 수 있다. 총 손실값은 평균 모델의 손실값과 산출된 보조 손실값에 기초하여 산출될 수 있다. 영상 데이터 처리 장치는 총 손실값을 최소화하도록 손실값을 산출하고 산출된 결과에 기초하여 복수의 CNN을 미세 조정할 수 있다.That is, referring to FIG. 2, first to nth CNNs are shown. Then, the auxiliary loss value of each CNN can be calculated. Then, the loss value (the average value) of the average value of the CNN (average model)

) Can be calculated. The total loss value can be calculated based on the loss value of the average model and the calculated auxiliary loss value. The image data processing device may calculate the loss value so as to minimize the total loss value and fine adjust the plurality of CNNs based on the calculated result.

3 is a diagram illustrating a network convergence model according to an embodiment of the present disclosure;

3 (a) and 3 (b), an ensemble process is shown. The ensemble process is the process of selecting a few CNNs of all CNNs and calculating an average value. The image data processing apparatus can uniformly and randomly select the CNN used for calculating the average value. Conversely, the video data processing apparatus can select CNN uniformly and randomly and exclude it from the average value calculation process.

The image data processing apparatus can select the CNN to be used for calculating the average value with a probability of 1-P _d . Where P _d represents the drop rate. In one embodiment, P _d may be 0.1.

Then, the video data processing apparatus can repeat the ensemble process. Since the video data processing apparatus randomly selects CNN, the same CNN can be selected and excluded in the average value calculation in the first and second processes.

For example, referring to FIG. 3 (a), first, second, fourth, and fifth CNNs are selected for average value calculation in the first ensemble process, and third, sixth, and seventh CNNs can be excluded. Referring to FIG. 3 (b), the first, second, fourth, sixth, and seventh CNNs are selected for the average value calculation in the second ensemble process, and the third and fifth CNNs can be excluded.

The selected CNNs may be used to calculate an average value for prediction at the learning stage. Selecting some CNNs and calculating the average value can help to make good predictions in the learning stage. That is, the ensemble process shows better results than the prediction using all models in the learning stage, as shown in Table 1.

Method Ensemble JF In this embodiment Cohen Kappa 0.9567 0.9576 0.9578 Accuracy 0.9721 0.9727 0.9728 Mean Precision 0.9293 0.9314 0.9307 Mean Recall 0.9018 0.8985 0.8991

For more accurate prediction in the test phase, the image data processing apparatus can perform a multi-crop test. For example, a video data processing apparatus may be implemented by transforming a fully connected layer of a network into a fully convolutional form and averaging the values at multiple scales. Then, the image data processing apparatus can average the soft max output values of the learned model through the ensemble process after the fine adjustment process.

4 is a view for explaining a region-based image data processing process according to an embodiment of the present disclosure.

Referring to FIG. 4, there is shown a structure of a region-based convolution sensor including a Convolutional Feature Extraction (CFE) layer and a RoF Feature Localization (RFL) layer.

The region-based convolution sensor may be a region-based CNN combining a position-sensitive score map and position-sensitive RoI pooling into a region-based fully convolutional network (R-FCN). The area-based convolutional detector may include a CFE layer and a RoI layer. The CFE layer can extract CNN features from the backborn architecture. And, the RFL layer can localize the bounding box with RoI features and NMS (Non-Maximum Suppression).

In one embodiment, the image data processing apparatus can be tested with several thousand test images and about 30,000 test images including 11 object classes. For example, eleven object classes may be used in conjunction with articulated trucks, bicycles, buses, cars, motorcycles, non-motorized vechiles, pedestrians, A pickup truck, a single unit truck, a work van, and a background.

In one embodiment, the image data processing apparatus can learn a classifier with an object function composed of a provided learning data, a sum of cross-entropy loss and box regression loss .

---------- (4)

---------- (4)

c * is a RoI ground truth label with a background label when c * is 0 and I (c *) is a RoI ground truth label with c * = 0 except when c * has a background label (c * = 0) It is the same indicator as 1.

는 클래스 출력(Oc*)이 주어졌을 때 분류를 위한 크로스 엔트로피 손실고,

는 t_b=(t(x), t(y), t(w), t(h))를 가지는 바운딩 박스 회귀 손실이다. 그리고,

는 균형 웨이트(balance weight)로서 1일 수 있다.

Is the cross entropy loss for classification when the class output (Oc *) is given,

Is the bounding box regression loss with t _b = (t (x), t (y), t (w), t (h)). And,

May be one as a balance weight.

FIG. 5 is a diagram illustrating a multi-network convergence model for a region-based data processing apparatus according to an embodiment of the present disclosure, FIG. 6 is a diagram showing the results of precision and detection of a network convergence model, to be.

Referring to FIG. 5, a plurality of CFE layers and a plurality of RFL layers are illustrated. After each R-FCN model of the video data processing apparatus is learned, it can be ensemble into one localization model as shown in Fig. Each model can receive input image data as input data and generate an initial bounding box for the image data. The bounding box of each of these models can eventually be merged into my NMS. As shown in Table 2, the ensemble model using the four learned R-FCNs of this disclosure showed a result of 79.24% mAP of the entire class, showing high accuracy compared with the existing technology.

[Table 2]

Referring to FIG. 6, a detailed mAP for each class of the final ensemble model is shown.

Various embodiments of the image data processing apparatus have been described so far. A flowchart of the video data processing apparatus control method will be described below.

7 is a flowchart of an image data processing apparatus according to an embodiment of the present disclosure.

The data processing apparatus receives the image data (S710). In one embodiment, the image data may be input to a data processing apparatus through a communication module connected to the communication interface, an input terminal connected to the input interface, or the like.

The data processing apparatus calculates an average value of output values output from a plurality of CNNs based on the input image data (S720). The data processing apparatus can exclude arbitrarily selected CNNs from all the plurality of CNNs and calculate an average value. Then, the data processing apparatus can repeat the process of arbitrarily selecting and excluding CNN and calculating an average value. The data processing apparatus can uniformly select CNNs to be excluded with a predetermined probability among the plurality of CNNs. Therefore, when the average value calculation is repeated, the CNN selected for the calculation of the average value of the previous process may be selected or excluded from the calculation of the average value of the next process. However, since the data processing apparatus makes the CNN selection uniform, the CNN selected as a whole can be made uniform if the average value calculation is repeated a number of times.

The data processing apparatus calculates the loss value of the average model including a plurality of CNNs included in the average value calculation and the auxiliary loss value for each of the plurality of CNNs included in the average value calculation (S730). For example, the input image data can be normalized. The loss value of the average model may be an average value of output values output from each of the plurality of CNNs included in the average value calculation based on the normalized image data and the learned weight parameter.

The data processing apparatus calculates the total loss value based on the calculated loss value of the average model and the calculated auxiliary loss value (S740). Then, the data processing apparatus finely adjusts the average value based on the total loss value (S750).

The control method of the image data processing apparatus according to the above-described various embodiments may be provided as a computer program product. The computer program product may include a software program itself or a non-transitory computer readable medium in which the software program is stored.

A non-transitory readable medium is a medium that stores data for a short period of time, such as a register, cache, memory, etc., but semi-permanently stores data and is readable by the apparatus. In particular, the various applications or programs described above may be stored on non-volatile readable media such as CD, DVD, hard disk, Blu-ray disk, USB, memory card, ROM,

While the present invention has been particularly shown and described with reference to exemplary embodiments thereof, it is to be understood that the invention is not limited to the disclosed exemplary embodiments, but, on the contrary, It will be understood by those skilled in the art that various changes in form and detail may be made therein without departing from the spirit and scope of the present invention.

100: image data processing device 110: input part
120: CNN 130:

Claims (10)

Receiving image data;
Calculating an average value of output values output from a plurality of CNNs based on the input image data;
A loss value calculation step of calculating a loss value of an average model including a plurality of CNNs included in the average value calculation and an auxiliary loss value for each of a plurality of CNNs included in the average value calculation;
Calculating a total loss value based on the calculated loss value of the average model and the calculated auxiliary loss value; And
And finely adjusting the average value based on the calculated total loss value. The method according to claim 1,
The loss value of the average model is calculated by:
Which is a loss value of an average value of output values output from each of the plurality of CNNs included in the average value calculation based on the normalized image data and the learned weight parameter, Control method. The method according to claim 1,
The step of calculating the average value includes:
And excluding the CNNs arbitrarily selected from all of the plurality of CNNs, and calculating the average value. The method of claim 3,
The step of calculating the average value includes:
CNN is arbitrarily selected and excluded, and the average value is calculated. 5. The method of claim 4,
The step of calculating the average value includes:
And selecting CNNs to be excluded with a predetermined probability among all of the plurality of CNNs uniformly. An input unit for receiving image data;
A plurality of CNNs for performing learning based on the input image data; And
And a controller for calculating an average value of output values output from the plurality of CNNs,
Wherein,
Calculating a loss value of an average model including a plurality of CNNs included in the average value calculation, and an auxiliary loss value for each of a plurality of CNNs included in the average value calculation, calculating a loss value of the calculated average model, Calculates a total loss value based on the loss value, and fine-adjusts the average value based on the calculated total loss value. The method according to claim 6,
The loss value of the average model is calculated by:
And normalizing the input image data and calculating a loss value for an average value of output values output from each of the plurality of CNNs included in the average value calculation based on the normalized image data and the learned weight parameter. The method according to claim 6,
Wherein,
And eliminates CNNs arbitrarily selected from all the plurality of CNNs and calculates the average value. 9. The method of claim 8,
Wherein,
CNN is arbitrarily selected and excluded, and the average value is calculated. 10. The method of claim 9,
Wherein,
And to uniformly select CNNs to be excluded with a predetermined probability among all of the plurality of CNNs.

Images