KR20190080388A - Photo Horizon Correction Method based on convolutional neural network and residual network structure - Google Patents

Abstract

The present invention relates to a method of leveling a tilted image by measuring and using the inclination of the image when the tilted image is inputted. The method includes: a step (a) of generating a learning data set for learning an angle measurement network in accordance with a residual network structure including first and second pulling layers and an angle prediction part; a step (b) of learning an optimal parameter of the angle measurement network; and a step (c) of rotating the image and cropping an empty pixel area.

Description

[0001] The present invention relates to an image horizontal correction method and a residual network structure using a CNN,

In the present invention, when a given input is an image that is not a horizontal image (when the image includes a roll rotation that is based on the view direction axis of the camera), the degree of tilt of the image is measured, And a method for standing upright in a horizontal direction.

The horizontal correction of the image plays an important role in the aesthetic evaluation of the photograph. For example, a photograph of a camera leaning like the left image of FIG. 1 ((a) in FIG. 1) does not seem to be aesthetically satisfactory in terms of sharpness and color. On the other hand, the right image of FIG. 1 (FIG. 1 (b)) is aesthetically good because the image is horizontally aligned.

Image stabilization technology is one of the most needed techniques for real users to take pictures with digital camera or smart phone. Conventional image horizontal correction techniques are based on the line information existing in the input image, but they do not work well for portrait images without clear line information.

The present invention can operate robustly against the photographs of various scenes taken by the actual users in daily life by correcting the horizontal position of the image by using the deep neural network technology. This robust image level correction technique is a technique that can be applied to various applications.

Much research has been conducted on the horizontal correction of images.

(Non-Patent Document 1) proposed a method of analyzing the line information in the image and setting up the image directly using the line information. However, since it is based on the existing line information, there is a limitation in that it does not work well for photographs that contain many artifacts or photographs that do not have sharp lines (for example, people or natural objects).

(Non-Patent Document 2) performed an angular measurement with respect to three difficult steps (angles between ± 30 °, ± 40 ° and ± 360 °) using deep neural network technology. This method has a great significance in applying the deep neural network technology, which has recently achieved excellent performance, to the image horizontal correction technique. However, by using a simple structure learning network, the horizontal correction result image has a large error of about 3 ° to 20 ° Practicality is lost because it is visible.

 Hyunjoon Lee, Eli Shechtman, Jue Wang, and Seungyong Lee. Automatic upright adjustment of photographs. In Computer Vision and Pattern Recognition (CVPR), 2012 IEEE Conference on, pages 877-884. IEEE, 2012.  Philipp Fischer, Alexey Dosovitskiy, and Thomas Brox. Image orientation estimation with convolutional networks. In German < / RTI > Conference on Pattern Recognition, pages 368-378. Springer, 2015.  Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.  Giorgos Tolias and Yannis Avrithis. Speeded-up, relaxed spatial matching. In Computer Vision (ICCV), 2011 IEEE International Conference on, pages 1653-1660. IEEE, 2011.

The present invention relates to an image horizontal correction method using a CNN that can obtain a resultant image that is horizontally corrected by measuring the degree of tilt of an image based on CNN when the input is a slanted image, And to provide a residual network structure used in the method.

In order to achieve the above object, a residual network structure according to the present invention comprises: a first pooling layer having a 1x1 feature map as an output at the end of a network structure; A second pooling layer having a 2x2 feature map as an output; And an angle predicting unit for combining the feature vector obtained from the first pooling layer and the feature vector obtained from the second pooling layer and using the feature vector as a final result prediction.

In the residual network structure, the angle predicting unit may uniformize the sizes of the two feature vectors, calculate an average value of the two feature vectors having the same size, and use the result as a final result prediction.

According to another aspect of the present invention, there is provided an image horizontal correction method using CNN, comprising: generating a learning data set for learning an angular measurement network according to the residual network structure; (b) learning the optimal parameters of the angular measurement network with different learning settings; And (c) measuring an inclined angle of the input image using the angular measurement network, rotating the image in a direction opposite to the measured inclined angle, and cropping the empty pixel region.

In the image horizontal correction method using the CNN, in step (a), a learning data set is generated by assigning a rotated image and a rotated angle at that time to a plurality of general images set up as labels.

In the image horizontal correction method using CNN, symmetric padding is performed by 1/2 of the horizontal and vertical lengths before rotating the image for generating the learning data set.

In the image horizontal correction method using the CNN, the angular measurement network may be initialized as an ImageNet image classification data set prior to the step (a), and then weight parameters of a predetermined number of layers located in the second half of the angle measurement network may be initialized .

In order to achieve the above object, a computer-readable recording medium according to the present invention records a program for causing a computer to execute an image horizontal correction method using the CNN.

In order to accomplish the above object, a computer program according to the present invention is stored in a medium for executing an image horizontal correction method using the CNN in a computer.

According to the present invention, in measuring the degree of tilting of an image for horizontal image correction, it is possible to detect a tilted image such as a line segment in which there is no predefined characteristic or information for determining tilting of the image It is possible to improve the accuracy even in the case of an image existing only in a specific region.

FIG. 1 is a view showing an example of an image in which a horizontal image matches a horizontal image.
FIG. 2 is a diagram showing the overall structure of an image horizontal correction method using CNN.
3 is a diagram illustrating a process of performing an image horizontal correction method.
FIG. 4 is a diagram showing a general CNN structure (a) having two layers and a residual network structure (b) having a skip connection structure added to two layers.
FIG. 5 is a diagram showing a general CNN structure (a) and a residual network structure (b) composed of 34 layers.
FIG. 6 illustrates a residual network structure for inclined angle measurement according to the present invention.
7 is a view illustrating a process of creating a rotated image using symmetric padding.
FIG. 8 is a view showing a result of rotating an arbitrary image at seven arbitrary angles.
9 is a diagram illustrating a process of determining whether a main line exists in an image.
FIG. 10 is a diagram illustrating a result of image horizontal correction using CNN according to the present invention.
FIG. 11 is a diagram showing a comparison between the result of the method of the present invention and Adobe Lightroom, which is one of existing commercial technologies.

Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings.

<Outline of Algorithm>

에 대한 각도 측정 네트워크를

라 정의하면, 네트워크의 학습은 크게 세 단계로 구성된다.The entire process of the image horizontal correction technique proposed in the present invention is shown in FIG. Tilted image

Angle measurement network for

The network learning consists of three stages.

의 구조를 구성한다. 네트워크 구조는 비특허문헌 3에서 제안한 레지듀얼 네트워크(residual network)를 기반으로 하여 두 갈래로 나뉘어진 풀링(pooling) 레이어를 추가하여 새롭게 구성한다. First network to learn

. The network structure is newly constructed by adding a two-divided pooling layer based on the residual network proposed in Non-Patent Document 3. [

을 학습시키기 위해 학습 데이터셋을 생성하는 것이 두 번째 단계인데, 바로 세워져 있다고 판단한 대규모 영상들을 임의의 각도로 회전시킨 후 회전된 영상과 그 때 회전시킨 각도를 레이블로 하여 학습 데이터셋을 구성한다. Then,

The second step is to generate a learning data set to train the learning data set. The learning data set is composed of the rotated image and the rotated angle as a label after rotating the large images determined to be erected at an arbitrary angle.

의 최적 파라미터

를 학습한다. 입력 영상이 네트워크의 입력으로 들어왔을 때 수행되는 과정은 도 3에서 표시된 과정을 거치게 된다.Finally, when the network and learning data sets are ready,

Optimal parameter of

. The process performed when the input image arrives at the input of the network goes through the process shown in FIG.

<Network structure>

The network structure used in the present invention is based on a residual network which has an excellent effect on object recognition.

Conventional CNN (Convolutional Neural Network) structures are learning using backpropagation technique. As the CNN structure becomes deeper, the gradient value of the next layer decreases exponentially according to the gradient of the previous layer Can occur. Therefore, the gradient near the output layer has a value, but at the input layer, the value becomes close to zero, and the problem is that the learning does not proceed any more (gradient vanishing problem).

The residual network is a structure that solves this gradient vanishing problem and enables efficient learning of the deep network structure. In this network, the sum of the output value (F (x) in FIG. 4) and the input value (identity in FIG. 4) obtained by passing an input value through several layers using a skip connection structure is calculated .

로 표현할 수 있다. 도 4의 (b)는 레지듀얼 네트워크(residual network) 구조인데, 2개의 레이어를 건너 뛴 skip connection을 활용한 것이 특징이다. 여기서

는

로 표현할 수 있다. 이러한 구성은 그라디언트를 계산하는데 있어 그 값이 너무 작아지지 않게 해주기 때문에 기존 CNN에서 발생되던 gradient vanishing 문제를 해결할 수 있다. 따라서 보다 많은 레이어의 모델을 잘 학습시킬 수 있고 적은 수의 레이어를 갖는 모델로도 높은 성능을 얻을 수 있다. 4 (a) shows a conventional network structure. In this structure, the result mapped after passing through two layers is

. 4B is a residual network structure, which is characterized in that a skip connection is used which skips two layers. here

The

. This configuration can solve the gradient vanishing problem that occurred in existing CNN because it does not become too small in calculating the gradient. Therefore, it is possible to learn the model of more layers well and obtain high performance even with a model with a small number of layers.

In the present invention, a residual network composed of 34 layers as shown in FIG. 5 (b) is used. Generally, since the residual network is used for object recognition in the input image, information of the whole input image is used at once, and a structure for the residual network is used for one aver- age pooling average pooling layer.

의 구조를 보여준다. 이를 위해 사용한 네트워크 구조는 기존의 레지듀얼 네트워크(residual network) 구조를 기반으로 한다. FIG. 6 is a flowchart illustrating a method of measuring a tilt angle of an input image according to an exemplary embodiment of the present invention.

. The network structure used for this is based on the existing residual network structure.

However, in the existing residual network structure, the last pooling layer serves to make the final feature map 1x1. In the present invention, this layer is replaced with two pooling layers We make maps 1x1 and 2x2, respectively, and combine them into final result prediction. This allows multi-scale features of the input image to be considered.

The two pooling layers have feature maps of 1x1 and 2x2, respectively. The 1x1 feature map extracts the features of the whole image, and the 2x2 feature map shows that each cell divides the image into four equal parts And extracts features of each region. Since the feature vectors obtained from two pooling layers are different in size, they are unified into a 256-size through a linear layer, and then the average of the two vectors is taken. Finally, the final tilt angle of the image is predicted through the linear layer. Loss (absolute difference) is used as a loss function to measure the prediction accuracy.

When the multi-scale features extracted from the two pooling layers are used, the information obtained from the 1x1 feature map represents the structural information of the whole image inclination, and each cell of the 2x2 feature map represents the image The feature information of different regions is expressed, and the positional relationship between the different regions is analyzed through the fully connected layer to help measure the image tilting.

For example, in the case of a human face image, the inclination of the entire outline of the face may be the feature information on the tilt of the image, but the information about the relationship between the eyes, or the relative positional relation between the nose and mouth, It can be a clue to the tilt measurement. In the case of considering multi-scale features as described above, accurate image tilting can be measured by learning the positional relationship between image regions through a large amount of data even in the case of various images in which there is no predefined characteristic in order to judge inclination as a line segment have.

In addition, since the feature map made of 2x2 independently determines the information of each area of the upper, lower, right and left sides of the image, even if information capable of determining the tilting of the image exists only in the specific area, Since it is possible to simultaneously learn features of various sizes in an image, it is possible to enhance the tilt angle measurement performance more stably.

<Network Learning>

을 학습하기 위해 우선 학습 데이터셋을 생성한다. 학습 데이터셋은 이미 바로 세워져 있는 영상들을 임의의 각도로 회전시킨 뒤 이때의 각도를 레이블로 사용하는 방식으로 생성한다. Image Tilt Measurement Network

The learning data set is generated first. The training data set is rotated by an arbitrary angle and then the angle is used as a label.

<Create Dataset>

을 학습하기 위한 영상 데이터셋을 만드는 과정은 도 7과 같다. 기울어지지 않고 바로 세워져 있는 영상에 대해 -20° ~ +20° 사이의 각도 7개를 무작위로 선택하여 그 각도만큼 회전시킨다. 이 때, 영상을 회전시키면 빈 픽셀 영역이 발생하게 되는데 이 부분을 채우기 위해 영상을 회전 시키기 전에 가로, 세로 길이의 1/2만큼 대칭적 패딩(symmetric padding)을 한다. 그 후 영상을 특정 각도만큼 회전시키고, 다시 원래 크기만큼 크롭하면 빈 픽셀 영역이 없는 사각형 꼴의 회전된 영상을 얻을 수 있다. Image Tilt Measurement Network

The process of creating the image data set for learning the image data is shown in FIG. For images that are not tilted, select seven random angles between -20 ° and + 20 ° and rotate them by that angle. In this case, when the image is rotated, an empty pixel area is generated. To fill the area, symmetric padding is performed by 1/2 of the horizontal and vertical length before the image is rotated. Thereafter, the image is rotated by a certain angle, and then the original image is cropped to obtain a rotated rectangular image without an empty pixel region.

The final training data set consists of the rotated images obtained using symmetric padding and the rotated angles at that time are assigned to labels. An example of a data set obtained by rotating at any 7 angles to an image is shown in FIG. The original image, which is not rotated, is also included in the learning data set with a label of 0.

The original image data set for generating the tilted image data set is the World Cities Dataset introduced in Non-Patent Document 4. It consists of 2.2 million images collected using a geographic query from 40 major cities from the Flickr website. In the present invention, 21,994 are part of the learning data set and 1,000 are the verification data sets Respectively. Although the images of World Cities Dataset include a small number of tilted images, most of them are assumed to have no significant influence on the overall trend of learning because they are built right up. Verification datasets were directly visualized and skewed (865).

In the present invention, since it is aimed at correcting skewing which operates robustly even for images lacking information to help judge skewing such as line segments, in addition to images (buildings, trees, etc.) in which main lines exist in the learning data set, And the ratio of each group was adjusted so that the non-existent images (portrait, natural scenery, etc.) were sufficiently included. For this purpose, when the center of the cluster with the longest sum of the lengths of the segments is detected as the main segment after clustering the segments found in the image, if the detected main line is shorter than 1/3 of the height and width of the image It is assumed that no main line exists in this image (see FIG. 9). The final learning data set consisted of 2: 1 ratio images with main lines and no main lines.

<Learning the image tilt measurement network>

The learning network used a pre-learned residual network model for the ImageNet image classification data set. The ImageNet classification problem is learned by the network so that it can be robustly categorized against the object's tilt. Therefore, it is contrary to the purpose of this algorithm that it is necessary to distinguish fine tilting information. In order to compensate for this, we initialized the weight parameters of the remaining layers (nine residual blocks) of the network, and then learned again with the learning data set described above. In the early layers of the network, low-level features of the learned image are extracted from the ImageNet image classification data set, and structural information for measuring the tilt in the latter layers and the relative positional relationship between the regions are newly learned.

<Results and Quantitative Evaluation>

은 입력 영상의 기울어진 각도를 측정하기 위해 사용된다. 이렇게 얻은 각도와 반대 방향으로 입력 영상

를 회전시키면 기울어짐이 보정된 결과 영상을 생성할 수 있다. 도 10은 다양한 입력 영상을 본 발명의 프레임워크에 적용시킨 결과이다. 본 방법의 결과는 영상 내 직선이 많인 경우뿐만 아니라 주요 선이 존재하지 않는 복잡한 장면에 대해서도 잘 동작하는 것을 확인할 수 있다. 또한 배경이 어두운 영상에 대해서도 강인하게 동작한다. The present invention proposes an image horizontal correction method using CNN. Network created through learning process

Is used to measure the tilted angle of the input image. In this case,

It is possible to generate an image having a corrected skew. 10 is a result of applying various input images to the framework of the present invention. The results of this method show that the proposed method works well not only for a large number of straight lines but also for complicated scenes where no main line exists. Also, it operates robustly against dark background images.

의 정확도를 정량적으로 평가하고, 영상의 기울어진 정도에 따라 측정 정확도가 어떻게 변하는지 평가하기 위해 865장의 다양한 영상으로 이루어진 검증 데이터셋을 다섯 가지 각도(3°, 5°, 10°, 15°, 20°)에 대해 (+, -) 방향으로 회전한 후, 각 기울어진 각도에 대해 정확도를 측정하였다. 결과는 표 1에서 보여진다. Tilt Measurement Network

(3 °, 5 °, 10 °, 15 °, and 90 °) were used to quantitatively evaluate the accuracy of the image and to evaluate how the measurement accuracy varies according to the degree of tilt of the image. 20 °), and then the accuracy was measured for each tilted angle. The results are shown in Table 1.

The error in Table 1 is the average of the difference between the angle measured by this method and the ground truth angle. As a result of the experiment, it can be confirmed that all five cases have high accuracy to show an error of about 1 °.

<Performance comparison according to network structure>

의 구조는 영상의 다중 스케일 특징을 동시에 고려하기 때문에 단일 스케일 특징만을 고려한 네트워크 구조에 비해 높은 성능을 얻을 수 있다. 이를 확인하기 위해 단일 스케일 특징만을 고려하는 네트워크를 동일한 조건에서 학습한 후 다중 스케일 네트워크와의 성능 비교를 수행하였다. 단일 스케일 네트워크 구조는 도 6에서 보여지는 다중 스케일 네트워크 구조에서 2개의 pooling 레이어를 1개의 pooling 레이어로 대체하여 512x1x1의 특징맵을 생성한 후 선형 레이어를 통해 512 크기의 특징을 거쳐 최종 기울어짐 각도를 측정하도록 구성하였다. 이를 본 알고리즘과 동일한 데이터셋으로 동일한 epoch 만큼 학습하고, 결과를 비교하였다. 전체 검증 데이터셋에 대해 단일 스케일, 다중 스케일 네트워크 각각을 이용하여 기울어짐 각도를 측정한 뒤 GT(ground truth) 각도와의 평균 오차를 측정, 학습 epoch가 진행됨에 따라 두 네트워크의 성능 변화를 비교하였다. The inclination measurement network proposed by the present invention

Since the multi-scale feature of the image is considered at the same time, the high performance can be obtained compared with the network structure considering only the single-scale feature. In order to verify this, a network considering only a single - scale feature is learned under the same conditions and performance comparison with a multi - scale network is performed. In the multi-scale network structure shown in FIG. 6, a single-scale network structure is created by replacing two pooling layers with one pooling layer to generate a 512x1x1 feature map, and then, through a linear layer, a final tilt angle Respectively. We learned the same data sets and learned the same epoch and compared the results. We measured the tilt angles using a single-scale and multi-scale networks for the entire validation dataset, measured the mean error with the GT (ground truth) angle, and compared the performance changes of the two networks as the learning epoch progressed .

The comparison results are shown in [ Table 2 ] .

As a result of comparison, single-scale network showed higher accuracy in early epoch, because multi-scale network has more kinds of features and parameters to learn than single-scale network. However, as the learning progresses, the parameter learning of the multi-scale network is sufficiently performed, and as a result, higher accuracy is obtained.

<Comparison of results according to dataset configuration>

를 학습했을 때 검증 데이터셋에 대한 기울임 측정 정확도가 어떻게 차이 나는지 비교한다. 이 실험에 사용한 네트워크 구조는 앞에서 설명한 다중 스케일 네트워크이다.Deep learning-based image analysis algorithms can vary depending on how the data set used in network learning is constructed. In this chapter, the learning data set consists of three different cases,

We compare how the accuracy of the tilt measurement for the verification dataset differs. The network structure used in this experiment is the multi-scale network described above.

를 학습하였다. 이렇게 학습된 모델은 선분 등의 구조적 특징을 갖고 있는 영상들에 대해서는 잘 동작할 것으로 예상할 수 있다. The first is a case where a learning data set is composed of images (buildings, trees, etc.) in which main lines exist. In the learning data set configuration, a data set having only main lines existing using the main line existence determination algorithm in FIG. 9,

Respectively. The learned model can be expected to work well for images with structural features such as line segments.

를 학습하였는데, 이 모델은 선이 존재하는 영상들에 대해서는 잘 작동하지 않을 것으로 예상할 수 있다. The second is a case where a dataset is composed only of images (no person-centered or natural scenery background images) in which no main line exists. In this case, it is also possible to generate a learning data set composed only of images in which there are no main lines by using the main line existence discrimination algorithm,

, Which can be expected to work well for images with lines.

를 학습하였다.Finally, the image with the main line and the image without the main line are each composed of 2: 1,

Respectively.

As shown in Table 3, the original data set including both the main line image and the non-existent image was found to be the most accurate. This means that the accuracy of the entire validation data set is enhanced when the tilt is measured by using additional information from various object features that can be obtained from a general image without a line segment, rather than using only the line information obtained from the image do. In addition, the accuracy of the validation dataset with segmentation is higher than that of the dataset with no segmentation in both networks with learning data set with or without major segmentation, Means that it is advantageous to measure the tilt of the image rather than the tilt information of a general subject.

<Comparison with existing commercial technology>

Finally, we compared the results of the proposed method with the conventional commercial photography technique. The existing commercial technology compared to this method is a feature built into Adobe Lightroom CC 2015.

The comparison result is shown in FIG. The first example image is easy to detect because many buildings are located in the image. As a result, it can be confirmed that the inclination correction is performed well both in the method of the present invention and in the result of the performance of the existing technique. On the other hand, it is unclear whether the second example image should be rotated around the horizontal line of the background or rotated to the main object. This method is similar to GT image by correcting based on the horizontal line of the background, but the existing technique is not good. In the case of the remaining example images, there is no main line in the image or the image is difficult to detect. In the case of the conventional technique, the method does not work properly. However, the method performs the skew correction well so that the person or animal Can be confirmed.

Meanwhile, the above-described embodiments of the present invention can be recorded in a medium used in a general-purpose computer including a personal computer. The medium may be a magnetic recording medium (e.g., ROM, floppy disk, hard disk, etc.), an optical reading medium (e.g. CD ROM, And the like.

The present invention has been described with reference to the preferred embodiments. It will be understood by those skilled in the art that various changes in form and details may be made therein without departing from the spirit and scope of the invention as defined by the appended claims. The disclosed embodiments should, therefore, be considered in an illustrative rather than a restrictive sense. The scope of the present invention is defined by the appended claims rather than by the foregoing description, and all differences within the scope of equivalents thereof should be construed as being included in the present invention.

Claims (8)

At the end of the network structure,
A first pooling layer having a 1x1 feature map as an output;
A second pooling layer having a 2x2 feature map as an output; And
And an angle predicting unit for combining the feature vector obtained from the first pooling layer and the feature vector obtained from the second pooling layer and using the combined feature vector as a final result prediction. The apparatus of claim 1, wherein the angle estimator
Wherein the average of the two feature vectors having the same size is calculated and used as a final result prediction. (a) an angle measurement network according to the residual network structure of (1)

Generating a learning data set for learning the training data set;
(b) the angular measurement network

Optimal parameter of

; And
(c) for the input image,

And rotating the image in the opposite direction by the measured tilted angle and cropping the blank pixel region. 4. The method of claim 3, wherein step (a)
And a learning data set is generated by assigning a rotated image and a rotated angle at that time to a plurality of general images set up immediately. 5. The method as claimed in claim 4, wherein symmetric padding is performed by 1/2 of the horizontal and vertical lengths before rotating the image for generating the learning data set. 4. The method of claim 3, wherein before step (a)
The angle measurement network

To an ImageNet image classification data set,

Wherein the weighting parameters of a predetermined number of layers located in the second half of the image are initialized. A computer-readable recording medium having recorded thereon a program for causing a computer to execute an image horizontal correction method using the CNN according to any one of claims 3 to 6. A computer program stored in a medium for executing in a computer an image horizontal correction method using the CNN of any one of claims 3 to 6.

Images