KR102426491B1 - Image data processing apparatus using ensamble and fine tunning and controlling method thereof - Google Patents

Abstract

An image data processing apparatus and a control method are disclosed. The control method of the image data processing apparatus includes the steps of receiving image data, calculating an average value of output values output from a plurality of CNNs based on the received image data, and an average model including a plurality of CNNs included in the average value calculation. A loss value calculation step of calculating an auxiliary loss value for each of the plurality of CNNs included in the loss value and average value calculation, calculating a total loss value based on the calculated average model loss value and the calculated auxiliary loss value, and calculation and fine-tuning the average value based on the total loss value obtained.

Description

Image data processing apparatus and control method using ensemble and fine adjustment

The present disclosure relates to an image data processing apparatus and a control method, and more particularly, to an image data processing apparatus and a control method using an ensemble and fine adjustment for ensemble of several network models.

The development of an intelligent traffic monitoring system has recently emerged as an important issue. One of the most important functions of an intelligent traffic monitoring system is useful analysis and extraction from recognized image data. In particular, techniques to classify and localize vehicles or pedestrians are fundamental exaggerations for traffic monitoring analysis.

Existing technologies for vehicle classification and localization obtain a real traffic detection environment by taking images at regular intervals by numerous traffic cameras. Then, it analyzes the numerous images obtained to classify and localize the vehicle. However, the accuracy of the analysis results of existing techniques is not satisfactory.

In order to increase the accuracy of image recognition, there is a method of using a plurality of image recognition network models. However, although the image recognition performance can be improved by averaging the output values of a plurality of network models, the synergistic effect between the networks is not maximized, which is not efficient. Therefore, there is a need for an efficient image recognition apparatus by using a plurality of network models to improve image recognition performance and maximize synergy.

SUMMARY The present disclosure provides an image data processing apparatus and control method for improving image recognition performance and maximizing synergy between a plurality of network models.

According to an embodiment of the present disclosure for achieving the above object, a control method of an image data processing apparatus includes receiving image data, and an average value of output values output from a plurality of CNNs based on the received image data. Calculating, a loss value calculating step of calculating a loss value of an average model including a plurality of CNNs included in the average value calculation and an auxiliary loss value for each of a plurality of CNNs included in the average value calculation, the calculated average model calculating a total loss value based on the loss value of , and the calculated auxiliary loss value, and finely adjusting the average value based on the calculated total loss value.

And, the loss value of the average model normalizes the input image data, and based on the normalized image data and the learned weight parameter, the loss with respect to the average value of the output values output from each of the plurality of CNNs included in the average value calculation can be a value.

Meanwhile, the calculating of the average value may include calculating the average value by excluding an arbitrarily selected CNN from among all the plurality of CNNs.

In addition, the calculating of the average value may include arbitrarily selecting and excluding CNNs and repeating the process of calculating the average value.

In addition, the calculating of the average value may include uniformly selecting a CNN to be excluded with a preset probability among all the plurality of CNNs.

According to an embodiment of the present disclosure for achieving the above object, an image data processing apparatus includes an input unit for receiving image data, a plurality of CNNs that perform learning based on the received image data, and the plurality of CNNs. A control unit for calculating an average value of output values, wherein the control unit includes a loss value of an average model including a plurality of CNNs included in the average value calculation and an auxiliary loss value for each of the plurality of CNNs included in the average value calculation and calculates a total loss value based on the calculated loss value of the average model and the calculated auxiliary loss value, and finely adjusts the average value based on the calculated total loss value.

And, the loss value of the average model normalizes the input image data, and based on the normalized image data and the learned weight parameter, the loss with respect to the average value of the output values output from each of the plurality of CNNs included in the average value calculation can be a value.

Meanwhile, the control unit may calculate the average value by excluding an arbitrarily selected CNN from among all the plurality of CNNs.

In addition, the control unit may arbitrarily select and exclude CNNs and repeat the process of calculating the average value.

In addition, the control unit may uniformly select a CNN to be excluded with a preset probability among all of the plurality of CNNs.

As described above, according to various embodiments of the present disclosure, the image data processing apparatus and control method may improve image recognition performance and maximize synergy between a plurality of network models.

1 is a block diagram of an image data processing apparatus according to an embodiment of the present disclosure.
2 is a diagram for explaining fine adjustment for network convergence according to an embodiment of the present disclosure.
3 is a diagram illustrating a network convergence model according to an embodiment of the present disclosure.
4 is a view for explaining a region-based image data processing process according to an embodiment of the present disclosure.
5 is a diagram illustrating a multi-network convergence model for a region-based data processing apparatus according to an embodiment of the present disclosure.
FIG. 6 is a diagram showing results for precision and recall of a network convergence model.
7 is a flowchart of an image data processing apparatus according to an embodiment of the present disclosure.

Hereinafter, various embodiments will be described in more detail with reference to the accompanying drawings. The embodiments described herein may be variously modified. Certain embodiments may be depicted in the drawings and described in detail in the detailed description. However, the specific embodiments disclosed in the accompanying drawings are only provided to facilitate understanding of the various embodiments. Therefore, the technical spirit is not limited by the specific embodiments disclosed in the accompanying drawings, and it should be understood to include all equivalents or substitutes included in the spirit and scope of the invention.

Terms including an ordinal number, such as first, second, etc., may be used to describe various elements, but these elements are not limited by the above-described terms. The above terminology is used only for the purpose of distinguishing one component from another component.

In the present specification, terms such as "comprises" or "have" are intended to designate that the features, numbers, steps, operations, components, parts, or combinations thereof described in the specification exist, but one or more other features It is to be understood that this does not preclude the possibility of the presence or addition of numbers, steps, operations, components, parts, or combinations thereof. When a component is referred to as being “connected” or “connected” to another component, it is understood that the other component may be directly connected or connected to the other component, but other components may exist in between. it should be On the other hand, when it is said that a certain element is "directly connected" or "directly connected" to another element, it should be understood that the other element does not exist in the middle.

Meanwhile, as used herein, a “module” or “unit” for a component performs at least one function or operation. In addition, a “module” or “unit” may perform a function or operation by hardware, software, or a combination of hardware and software. In addition, a plurality of “modules” or a plurality of “units” other than a “module” or “unit” to be performed in specific hardware or performed by at least one control unit may be integrated into at least one module. The singular expression includes the plural expression unless the context clearly dictates otherwise.

In addition, in describing the present invention, if it is determined that a detailed description of a related known function or configuration may unnecessarily obscure the gist of the present invention, the detailed description thereof will be abbreviated or omitted.

1 is a block diagram of an image data processing apparatus according to an embodiment of the present disclosure.

Referring to FIG. 1 , the image data processing apparatus 100 includes an input unit 110 , a plurality of Covolutional Neural Networks (CNNs) 120 , and a control unit 130 . The input unit 110 receives image data. For example, the image data may be an image including a vehicle. For example, the input unit 110 may include a communication module connected to the communication interface, an input terminal connected to the input interface, and the like. That is, when the input unit 110 is implemented as a communication module, the image data processing apparatus 100 may receive image data using a wired/wireless communication method. Alternatively, when the input unit 110 is implemented as an input terminal, the image data processing apparatus 100 may receive image data from an internal or external storage device.

The plurality of CNNs 120 may include at least two or more CNNs 120-1, 120-2, and 120-3. Each of the plurality of CNNs 120 performs learning by receiving input image data. The plurality of CNNs 120 may improve recognition performance of image data through a fine-tuning process and an ensemble process in the learning process. The fine-tuning process is a process of correcting errors in order to increase the recognition rate of an image, and the ensemble process means a process of averaging the output values of a plurality of CNNs.

The control unit 130 calculates an average value of the output values output primarily from the plurality of CNNs 120 . Then, the control unit 130 calculates a loss value of the average model including a plurality of CNNs included in the average value calculation and an auxiliary loss value for each of the plurality of CNNs included in the average value calculation. The image data processing apparatus 100 may exclude output values of some CNNs from among all CNNs in the ensemble process. That is, the control unit 130 may calculate an average value obtained by averaging the output values of all CNNs 120 to calculate the average value, and at least one of the first to n-th CNNs 120-1, 120-2, 120-3 It is also possible to calculate an average value by excluding the CNN and averaging the output values of the remaining CNNs. The controller 130 may arbitrarily select an excluded CNN. Then, the controller 130 may repeat the process of calculating the average value of the CNN output values, and each time it is repeated, a different CNN may be arbitrarily selected and excluded. Of course, the CNN used to calculate the average value in the repeated average value calculation process may include the same CNN. The controller 130 may uniformly select CNNs excluded from calculating the average value with a preset probability. A detailed ensemble process will be described later.

The controller 130 calculates a total loss value based on the calculated loss value of the average model and the calculated auxiliary loss value, and fine-tunes the average value based on the calculated total loss value (joint fine tuning). For example, the loss value of the average model normalizes the input image data, and based on the normalized image data and the learned weight parameter, the loss value for the average value of the output values output from each of a plurality of CNNs included in the average value calculation Can be have. A detailed fine-tuning process will be described later.

2 is a diagram for explaining fine adjustment for network convergence according to an embodiment of the present disclosure.

Referring to FIG. 2 , a plurality of CNNs and loss values are shown. As an embodiment, the CNN for the classification task may be based on a pre-activation ResNet with 18 layers that applies the shortest distance connection. The image data for learning may be an image having a size of 224×224×3. Since the size of the image data for learning varies, the controller 130 may resize the image data for each learning and randomly cut it to 224×224×3. RGB pixels of each image data may be normalized as shown in the following equation.

---------- (1)

---------- (One)

Here, x(i, j) may be a vector having RGB pixel values at (i, j) coordinates. Photometric distortion and color augmentation can be applied to get good results. Each CNN uses a Rectified Linear Unit (ReLU) as an activation function, and batch normalization can be used for each layer. The number of outputs of the network may be the same as the number of categories of the dataset, and may be 11, for example. A softmax layer can be applied to the final output of each CNN.

는 다음과 같이 정의될 수 있다.The image data processing device may perform a fine-tuning process to ensemble some CNNs among all CNNs. The image data processing apparatus calculates a total loss value based on the loss value of the average model and the calculated auxiliary loss value. total loss

can be defined as follows.

---------- (2)

---------- (2)

는 평균 모델의 손실 함수이다. 그리고,

는 i번째 CNN의 보조(auxiliary) 손실이다. 손실값은 학습 단계에서만 사용될 수 있다. o는 다음과 같이 정의된다.where N is the number of CNN models,

is the loss function of the average model. and,

is the auxiliary loss of the i-th CNN. Loss values can only be used in the learning phase. o is defined as

---------- (3)

---------- (3)

는 w_i의 학습된 가중치 파라미터를 가지는 i번째 CNN의 출력이고,

는 식(1)에 의해 정규화된 입력 데이터를 나타낸다. 각 네트워크는

를 최소화하기 위해 학습 비율

의 아주 작은 값을 사용하여 미세 조정될 수 있다. 예를 들어,

= 1e^-5일 수 있다. 가중치 파라미터의 값은 초기에 임의의 값일 수 있다. 그리고, 가중치 파라미터는 학습에 따라 가변되고, 역전파 과정을 통해 학습이 종료될 때까지 계속적으로 수정될 수 있다.here,

is the output of the i-th CNN with the learned weight parameters of w _i ,

denotes the input data normalized by Equation (1). each network

learning rate to minimize

can be fine-tuned using very small values of for example,

= 1e ^-5 . The value of the weight parameter may initially be any value. In addition, the weight parameter may vary according to learning and may be continuously modified through the backpropagation process until learning is terminated.

)이 산출될 수 있다. 총 손실값은 평균 모델의 손실값과 산출된 보조 손실값에 기초하여 산출될 수 있다. 영상 데이터 처리 장치는 총 손실값을 최소화하도록 손실값을 산출하고 산출된 결과에 기초하여 복수의 CNN을 미세 조정할 수 있다.That is, referring to FIG. 2 , first to nth CNNs are illustrated. Then, an auxiliary loss value of each CNN may be calculated. And, the loss value for the average value of the CNN (average model) used to calculate the average value (

) can be calculated. The total loss value may be calculated based on the loss value of the average model and the calculated auxiliary loss value. The image data processing apparatus may calculate a loss value to minimize the total loss value and fine-tune the plurality of CNNs based on the calculated result.

3 is a diagram illustrating a network convergence model according to an embodiment of the present disclosure.

Referring to Figs. 3(a) and 3(b), the ensemble process is illustrated. The ensemble process is a process of selecting some CNNs among all CNNs and calculating the average value. The image data processing apparatus may uniformly and randomly select the CNN used for calculating the average value. Conversely, the image data processing apparatus can select CNNs uniformly and randomly and exclude them from the average value calculation process.

The image data processing apparatus may select a CNN to be used for calculating the average value with a probability of 1-P _d . Here, P _d represents a drop rate. As an embodiment, P _d may be 0.1.

Then, the image data processing apparatus may repeat the ensemble process. Since the image data processing apparatus randomly selects a CNN, the same CNN may be selected or excluded for average value calculation in the first and second processes.

For example, referring to FIG. 3A , in the first ensemble process, the first, second, fourth, and fifth CNNs may be selected for calculating the average value, and the third, sixth, and seventh CNNs may be excluded. And, referring to FIG. 3(b) , in the second ensemble process, the first, second, fourth, sixth, and seventh CNNs may be selected for calculating the average value, and the third and fifth CNNs may be excluded.

The selected CNNs can be used to calculate an average value for prediction in the learning phase. The process of selecting some CNNs and calculating the average value can help to make good predictions in the learning phase. That is, as shown in Table 1, the ensemble process shows better results than when prediction is performed using all models in the learning stage.

Method Ensemble JF this example Cohen Kappa 0.9567 0.9576 0.9578 Accuracy 0.9721 0.9727 0.9728 Mean Precision 0.9293 0.9314 0.9307 Mean Recall 0.9018 0.8985 0.8991

For more accurate prediction in the test stage, the image data processing apparatus may perform a multi-crop test. For example, the image data processing apparatus may be implemented by transforming a fully connected layer of a network into a fully convolutional form, and may average values on multiple scales. In addition, after the fine adjustment process, the image data processing apparatus may average the softmax output values of the models learned through the ensemble process described above.

4 is a view for explaining a region-based image data processing process according to an embodiment of the present disclosure.

Referring to FIG. 4 , a structure of a region-based convolutional detector including a Convolutional Feature Extraction (CFE) layer and a RoI Feature Localization (RFL) layer is illustrated.

The region-based convolutional detector may be a region-based CNN that combines a position-sensitive score map and position-sensitive RoI pooling into a region-based fully convolutional network (R-FCN). A region-based convolutional detector may include a CFE layer and a RoI layer. The CFE layer can extract CNN features from the backborn architecture. In addition, the RFL layer may localize a bounding box having a RoI feature and a Non-Maximum Suppression (NMS).

As an embodiment, the image data processing apparatus may be trained with hundreds of thousands of training images and tested with about 30,000 test images including 11 object classes. For example, the 11 object classes are articulated truck, bicycle, bus, car, motorcycle, non-motorized vechile, pedestrian, It may be a pickup truck, a single unit truck, a work van, or a background.

As an embodiment, the image data processing apparatus may learn a classifier using the provided training data and an object function composed of a sum of cross-entropy loss and box regression loss. .

---------- (4)

---------- (4)

c* is the RoI ground truth label having a background label when c* is 0, and I(c*) is the case where c* has a background label (c*=0) except The same indicator as 1.

는 클래스 출력(Oc*)이 주어졌을 때 분류를 위한 크로스 엔트로피 손실고,

는 t_b=(t(x), t(y), t(w), t(h))를 가지는 바운딩 박스 회귀 손실이다. 그리고,

는 균형 웨이트(balance weight)로서 1일 수 있다.

is the cross-entropy loss for classification given the class output (Oc*),

is the bounding box regression loss with t _b =(t(x), t(y), t(w), t(h)). and,

may be 1 as a balance weight.

5 is a diagram illustrating a multi-network convergence model for a region-based data processing apparatus according to an embodiment of the present disclosure, and FIG. 6 is a diagram illustrating the results of precision and detection of the network convergence model. to be.

Referring to FIG. 5 , a plurality of CFE layers and a plurality of RFL layers are illustrated. After each R-FCN model of the image data processing apparatus is trained, it may be ensembled into one localization model as shown in FIG. 5 . Each model may receive image data as input data and generate an initial bounding box for the image data. The bounding box of each of these models can be finally merged into my NMS. As shown in Table 2, the ensemble model using the four learned R-FCNs of the present disclosure showed a result of 79.24% mAP of the entire class, showing high accuracy compared to the existing technique.

[Table 2]

Referring to FIG. 6 , detailed mAPs for each class of the final ensemble model are shown.

So far, various embodiments of the image data processing apparatus have been described. Hereinafter, a flowchart of a method for controlling an image data processing apparatus will be described.

7 is a flowchart of an image data processing apparatus according to an embodiment of the present disclosure.

The data processing apparatus receives image data (S710). As an embodiment, the image data may be input to the data processing apparatus through a communication module connected to the communication interface, an input terminal connected to the input interface, or the like.

The data processing apparatus calculates an average value of output values output from a plurality of CNNs based on the received image data (S720). The data processing apparatus may exclude an arbitrarily selected CNN from among all the plurality of CNNs and calculate an average value. In addition, the data processing apparatus may repeat the process of arbitrarily selecting and excluding CNNs and calculating an average value. The data processing apparatus may uniformly select a CNN to be excluded with a preset probability among all of the plurality of CNNs. Therefore, when the average value calculation is repeated, the CNN selected for the average value calculation of the previous process may be selected for the average value calculation of the next process, or it may be excluded. However, since the data processing device makes the CNN selection uniform, the overall selected CNN may be uniform if the number of average calculations is repeated.

The data processing apparatus calculates a loss value of an average model including a plurality of CNNs included in the average value calculation and an auxiliary loss value for each of the plurality of CNNs included in the average value calculation ( S730 ). For example, input image data may be normalized. The loss value of the average model may be an average value of output values output from each of the plurality of CNNs included in the average value calculation based on the normalized image data and the learned weight parameter.

The data processing apparatus calculates a total loss value based on the calculated loss value of the average model and the calculated auxiliary loss value (S740). Then, the data processing apparatus finely adjusts the average value based on the total loss value (S750).

The method for controlling the image data processing apparatus according to the above-described various embodiments may be provided as a computer program product. The computer program product may include the S/W program itself or a non-transitory computer readable medium in which the S/W program is stored.

The non-transitory readable medium refers to a medium that stores data semi-permanently, rather than a medium that stores data for a short moment, such as a register, cache, memory, etc., and can be read by a device. Specifically, the above-described various applications or programs may be provided by being stored in a non-transitory readable medium such as a CD, DVD, hard disk, Blu-ray disk, USB, memory card, ROM, and the like.

In addition, although preferred embodiments of the present invention have been illustrated and described above, the present invention is not limited to the specific embodiments described above, and the technical field to which the present invention pertains without departing from the gist of the present invention as claimed in the claims Various modifications are possible by those of ordinary skill in the art, and these modifications should not be individually understood from the technical spirit or prospect of the present invention.

100: image data processing device 110: input unit
120: CNN 130: control unit

Claims (10)

receiving image data;
calculating an average value of output values output from a plurality of CNNs based on the received image data;
a loss value calculation step of calculating a loss value of an average model including a plurality of CNNs included in the average value calculation and an auxiliary loss value for each of the plurality of CNNs included in the average value calculation;
calculating a total loss value by summing the calculated loss value of the average model and the calculated auxiliary loss value; and
Including; fine-tuning the average value based on the calculated total loss value;
The loss value of the average model is,
Normalizing the input image data, and a loss value with respect to the average value of the output values output from each of the plurality of CNNs included in the average value calculation based on the normalized image data and the learned weight parameter, control of the image data processing apparatus Way. delete According to claim 1,
Calculating the average value comprises:
A method of controlling an image data processing apparatus, excluding a CNN arbitrarily selected from among a plurality of CNNs and calculating the average value. 4. The method of claim 3,
Calculating the average value comprises:
A method of controlling an image data processing apparatus that randomly selects and excludes a CNN and repeats the process of calculating the average value. 5. The method of claim 4,
Calculating the average value comprises:
A method of controlling an image data processing apparatus for uniformly selecting a CNN to be excluded with a preset probability among all of the plurality of CNNs. an input unit for receiving image data;
a plurality of CNNs for learning based on the received image data; and
Including; a control unit for calculating an average value of the output values output from the plurality of CNNs;
The control unit is
A loss value of an average model including a plurality of CNNs included in the average value calculation and an auxiliary loss value for each of a plurality of CNNs included in the average value calculation are calculated, and the calculated average model loss value and the calculated auxiliary value are calculated. Calculate a total loss value by summing the loss values, and finely adjust the average value based on the calculated total loss value,
The loss value of the average model is,
The image data processing apparatus which normalizes the input image data and is a loss value with respect to an average value of output values output from each of the plurality of CNNs included in the average value calculation based on the normalized image data and the learned weight parameter. delete 7. The method of claim 6,
The control unit is
An image data processing apparatus for calculating the average value by excluding an arbitrarily selected CNN from all of the plurality of CNNs. 9. The method of claim 8,
The control unit is
An image data processing apparatus that arbitrarily selects and excludes a CNN and repeats the process of calculating the average value. 10. The method of claim 9,
The control unit is
An image data processing apparatus for uniformly selecting a CNN to be excluded with a preset probability among all of the plurality of CNNs.

Images