CN106650737A - Image automatic cropping method - Google Patents

Abstract

The invention relates to an image automatic cropping method. The method includes: extracting an aesthetic response map and a gradient energy map of an image to be cropped; intensively extracting candidate cropping images from the image to be cropped; screening candidate cropping images based on the aesthetic response map; and estimating the selected candidate based on the aesthetic response map and gradient energy map The composition score of the cropped image is determined, and the candidate cropped image with the highest score is determined as the cropped image. This scheme uses the aesthetic response map to explore the aesthetically affected area of the image, and uses the aesthetic response map to determine the aesthetically preserved part, thereby retaining the high aesthetic quality of the cropped image to the greatest extent. At the same time, this scheme also uses the gradient energy map to analyze the gradient distribution rules , and evaluate the composition score of the cropped image based on the aesthetic response map and the gradient energy map. The embodiment of the present invention makes up for the defect of image composition expression, and solves the technical problem of how to improve the robustness and precision of automatic image cropping.

Description

Image automatic cropping method

technical field

The invention relates to the technical fields of pattern recognition, machine learning and computer vision, in particular to an image automatic cropping method.

Background technique

With the rapid development of computer technology and digital media technology, people's needs and expectations for computer vision, artificial intelligence, machine perception and other fields are also increasing. Image automatic cropping, as a very important and common task in automatic image editing, has also received more and more attention and development. Image automatic cropping technology is to remove redundant areas and emphasize areas of interest, thereby improving the overall composition and aesthetic quality of the image. An effective and automatic image cropping method can not only liberate human beings from tedious work, but also provide some professional image editing suggestions to some non-professionals.

Since image cropping is a very subjective task, it is difficult for existing rules to consider all influencing factors. Traditional automatic image cropping usually uses a saliency map to identify the main region or region of interest in the image, and at the same time calculates the energy function minimization or learns a classifier to find the cropped region by formulating some rules. However, these formulated rules are not comprehensive enough for the subjective task of image cropping, and the accuracy is difficult to meet user needs.

In view of this, the present invention is proposed.

Contents of the invention

In order to solve the above-mentioned problems in the prior art, that is, to solve the technical problem of how to improve the robustness and accuracy of automatic image cropping, an automatic image cropping method is provided.

In order to achieve the above object, the following technical solutions are provided:

An image automatic cropping method, the method comprising:

Extract the aesthetic response map and gradient energy map of the image to be cropped;

densely extracting candidate cropped images for the image to be cropped;

Screening the candidate cropped images based on the aesthetic response map;

Based on the aesthetic response map and the gradient energy map, estimate composition scores of the screened candidate cropped images, and determine the candidate cropped image with the highest score as the cropped image.

Further, the extracting the aesthetic response map and the gradient energy map of the image to be cropped specifically includes:

Using a deep convolutional neural network and a category response mapping method, and using the following formula to extract the aesthetic response map of the image to be cropped:

Wherein, the M (x, y) represents the aesthetic response value at the spatial position (x, y); the K represents the total channel number of the feature map of the last layer of convolutional layer of the deep convolutional neural network; The k represents the kth channel; the f _k (x, y) represents the eigenvalue of the kth channel at the spatial position (x, y); the w _k represents the kth channel The weight of the channel's feature map pooling result to the high aesthetic category;

Perform smoothing processing on the image to be cropped, and calculate the gradient value of each pixel, so as to obtain the gradient energy map.

Further, the deep convolutional neural network is obtained through training in the following manner:

A convolutional layer is set at the bottom layer of the deep convolutional neural network structure;

After the last convolutional layer of the deep convolutional neural network structure, each feature map is pooled into a point by a global average pooling method;

Connect the same number of fully-connected layers and loss functions as aesthetic quality classification classes.

Further, the screening of the candidate cropped images based on the aesthetic response map specifically includes:

Calculate the beauty preservation score of the candidate cropped image by the following formula:

Wherein, the S _a (C) represents the aesthetic preservation score of the candidate cropped image; the C represents the candidate cropped image; the (i, j) represents the position of the pixel; the I represents the original image ; The A _{(i, j)} represents the aesthetic response value at the (i, j) position;

Sorting all candidate cropped images according to the aesthetic preservation scores from large to small;

Select the portion of candidate crops with the highest score.

Further, the estimation of composition scores of the screened candidate cropped images based on the aesthetic response map and the gradient energy map, and determining the candidate cropped image with the highest score as the cropped image specifically includes:

Establishing a composition model based on the aesthetic response map and the gradient energy map;

Estimate composition scores of the screened candidate cropped images by using the composition model, and determine the candidate cropped image with the highest score as the cropped image.

Further, the composition model is obtained in the following manner:

establishing a training image set based on the aesthetic response map and the gradient energy map;

Annotate the aesthetic quality category of the training images;

Train a deep convolutional neural network using labeled training images;

For the labeled training image, using the trained deep convolutional neural network to extract the spatial pyramid features of the aesthetic response map and the gradient energy map;

Stitch together the extracted spatial pyramid features;

Use the classifier to train, automatically learn the composition rules, and obtain the composition model.

An embodiment of the present invention provides an image automatic cropping method. The method includes: extracting an aesthetic response map and a gradient energy map of an image to be cropped; intensively extracting candidate cropping images from the image to be cropped; screening candidate cropping images based on the aesthetic response map; and estimating the selected candidate based on the aesthetic response map and gradient energy map The composition score of the cropped image is determined, and the candidate cropped image with the highest score is determined as the cropped image. This scheme uses the aesthetic response map to explore the aesthetically affected area of the image, and uses the aesthetic response map to determine the aesthetically preserved part, thereby retaining the high aesthetic quality of the cropped image to the greatest extent. At the same time, this scheme also uses the gradient energy map to analyze the gradient distribution rules , and evaluate the composition score of the cropped image based on the aesthetic response map and the gradient energy map. The embodiment of the present invention makes up for the defect of image composition expression, and solves the technical problem of how to improve the robustness and precision of automatic image cropping. The embodiments of the present invention can be applied to many fields related to automatic cropping of images, including image editing, photography, and image repositioning.

Description of drawings

Fig. 1 is a schematic flow chart of an image automatic cropping method according to an embodiment of the present invention;

Fig. 2 is a schematic structural diagram of a deep convolutional neural network according to an embodiment of the present invention;

Fig. 3a is a schematic diagram of an image to be cropped according to an embodiment of the present invention;

Fig. 3b is a schematic diagram of a cropped image according to an embodiment of the present invention.

detailed description

The technical problems solved by the embodiments of the present invention, the technical solutions adopted and the technical effects achieved are clearly and completely described below in conjunction with the accompanying drawings and specific embodiments. Apparently, the described embodiments are only some of the embodiments of the present application, not all of them. Based on the embodiments in the present application, all other equivalent or obviously modified embodiments obtained by persons of ordinary skill in the art without creative efforts fall within the protection scope of the present invention. Embodiments of the invention can be embodied in many different ways as defined and covered by the claims.

Deep learning has achieved rapid development and good results in various fields. The embodiment of the present invention considers the use of deep learning to automatically learn important areas of influence on image cropping, so as to automatically and comprehensively learn rules, so as to preserve high aesthetic areas as much as possible during cropping.

To this end, an embodiment of the present invention provides an automatic image cropping method. FIG. 1 exemplarily shows the flow of an image automatic cropping method. As shown in Figure 1, the method may include:

S100: Extracting an aesthetic response map and a gradient energy map of an image to be cropped.

Specifically, this step may include:

S101: Using the deep convolutional neural network and the category response mapping method, and using the following formula to extract the aesthetic response map of the image to be cropped:

Among them, M(x, y) represents the aesthetic response value at the spatial position (x, y); K represents the total number of channels of the feature map f of the last convolutional layer of the trained deep convolutional neural network; k represents the kth channel; f _k (x, y) represents the feature value of the kth channel at the spatial position (x, y); w _k represents the result of the feature map pooling of the kth channel to high aesthetics category weights.

The above steps can train the deep convolutional neural network according to actual needs when extracting the aesthetic response map. The training of deep convolutional neural networks can be done in the following ways:

Step 1: Set the convolutional layer at the bottom of the deep convolutional neural network structure.

Step 2: After the last convolutional layer of the deep convolutional neural network structure, each feature map is pooled into a point by the global average pooling method.

Step 3: Connect a fully connected layer and loss function with the same number of categories as aesthetic quality classification.

Fig. 2 exemplarily shows a deep convolutional neural network structure.

Through steps 1-3, a deep convolutional neural network model under the aesthetic quality classification task can be trained. Then, using the deep convolutional neural network trained for the aesthetic quality classification task and the category response mapping method; and then using the above formula, the aesthetic response map M of the image to be cropped under the high aesthetic category can be calculated.

S102: Perform smoothing processing on the image to be cropped, and calculate the gradient value of each pixel, so as to obtain a gradient energy map.

S110: Densely extract candidate cropped images from the image to be cropped.

Here, sliding windows of all sizes smaller than the size of the image can be used to intensively extract candidate cropping windows from the image to be cropped, and extract candidate cropping images through the candidate cropping windows.

S120: Screen candidate cropped images based on the aesthetic response map.

Specifically, this step may include:

S121: Calculate the aesthetic preservation score of the candidate cropped image by the following formula:

Among them, S _a (C) represents the aesthetic preservation score of the candidate cropped image; C represents the candidate cropped image; (i, _j ) represents the position of the pixel; I represents the original image; The aesthetic response value at .

Through this step, an aesthetic preservation model can be constructed. Filter the candidate cropping windows through the aesthetic preservation model to select the candidate windows with higher aesthetic preservation scores.

S122: Sort all candidate cropped images in descending order of aesthetic preservation scores.

S123: Select a part of candidate cropped images with the highest score.

For example: in practical applications, it may be set to retain candidate cropping images in the first 10,000 candidate cropping windows.

S130: Based on the aesthetic response map and the gradient energy map, estimate composition scores of the screened candidate cropped images, and determine the candidate cropped image with the highest score as the cropped image.

Specifically, this step may be implemented through steps S131 to S133.

S131: Establish a composition model based on the aesthetic response map and the gradient energy map.

In this step, the composition model can be trained according to the actual situation when the composition model is established. In the process of training the composition model, the training data can use images with better composition as positive samples, and images with composition defects as negative samples.

Compositional models can be trained in the following ways:

Step a: Build a training image set based on the aesthetic response map and the gradient energy map.

Step b: Annotate the aesthetic quality category of the training images.

Step c: Train a deep convolutional neural network using the annotated training images.

For the training process of this step, reference may be made to the above steps 1 to 3, which will not be repeated here.

Step d: For the labeled training images, use the trained deep convolutional neural network to extract the spatial pyramid features of the aesthetic response map and the gradient energy map.

Step e: stitch together the extracted spatial pyramid features.

Step f: using a classifier for training, automatically learning composition rules, and obtaining a composition model.

Wherein, the classifier may adopt a support vector machine classifier, for example.

S132: Use the composition model to estimate the composition scores of the screened candidate cropped images, and determine the candidate cropped image with the highest score as the cropped image.

Fig. 3a exemplarily shows an image to be cropped; Fig. 3b exemplarily shows a cropped image.

The present invention will be better described below with a preferred embodiment.

Step A: Send the image data set marked with aesthetic quality categories into the deep convolutional neural network for aesthetic quality category model training.

Step B: Input the image data set marked with the composition category into the trained deep convolutional neural network, extract the feature map of the last convolutional layer, and calculate the aesthetic response map, and calculate the aesthetic gradient map at the same time, and then use the support vector machine A classifier trains a composition model.

Step C: extract the aesthetic response map and the gradient energy map from the test image.

The extraction method in this step can refer to the method in the training stage.

Step D: Densely collect candidate cropping windows of the image to be tested.

For example, on a 1000×1000 image to be tested, a sliding window with an interval of 30 pixels is used for acquisition or extraction.

Step E: Use the aesthetic preservation model to screen candidate cropping windows.

In this step, the beauty preservation model is used to calculate the aesthetic preservation scores of the densely collected candidate cropping windows, and a part of the candidate cropping windows with the highest aesthetic classification is selected, for example, 10,000 candidate cropping windows are screened out.

Step F: Evaluate the screened candidate cropping windows using the composition model.

In this step, the composition model trained in the training phase is collected to evaluate the composition scores of the screened candidate cropping windows, and the one with the highest score is used as the final cropping window to obtain a cropped image.

To sum up, the method provided by the embodiment of the present invention makes good use of the aesthetic response map and the gradient energy map to preserve the aesthetic quality and the composition rules of the image to the greatest extent, and obtain a more robust and precise automatic cropping of the image Performance, which further illustrates the effectiveness of aesthetic response maps and gradient energy maps for automatic image cropping.

Although the above embodiments describe the methods provided by the embodiments of the present invention according to the above order, those skilled in the art can understand that in order to achieve the effect of this embodiment, they can also be executed in different orders such as parallel or reverse order, These simple changes are all within the protection scope of the present invention.

The above is only a specific implementation mode in the present invention, but the scope of protection of the present invention is not limited thereto. Anyone familiar with the technology can understand the conceivable transformation or replacement within the technical scope disclosed in the present invention. All should be covered within the scope of the present invention, therefore, the protection scope of the present invention should be based on the protection scope of the claims.

Claims (6)

1. a kind of automatic image cutting out method, it is characterised in that methods described includes：

The aesthetic feeling response diagram and gradient energy figure of cutting image is treated in extraction；

The intensive extraction candidate's cutting image of cutting image is treated to described；

Based on the aesthetic feeling response diagram, candidate's cutting image is screened；

The composition fraction of the candidate's cutting image filtered out based on the aesthetic feeling response diagram and the gradient energy figure, estimation, and Candidate's cutting image of highest scoring is defined as into cutting image.

2. method according to claim 1, it is characterised in that the aesthetic feeling response diagram and gradient of cutting image is treated in the extraction Energy diagram, specifically includes：

Using depth convolutional neural networks and classification response mapping method, and cutting image is treated using described in equation below extraction The aesthetic feeling response diagram：

$M(x,y)=\underset{k=1}{\overset{K}{\mathrm{\Σ}}}{w}_k{f}_k(x,y)$

Wherein, the M (x, y) represents the aesthetic feeling response at locus (x, y) place；The K represents depth convolutional Neural net The overall channel number of the characteristic pattern of last layer of convolutional layer of network；The k represents k-th passage；The f_k(x, y) represents described Characteristic value of k-th passage at locus (x, the y) place；The w_kAfter representing the characteristic pattern pond of k-th passage Result to high aesthetic feeling classification weights；

Treat that cutting image is smoothed to described, and calculate the Grad of each pixel, so as to obtain the gradient energy Spirogram.

3. method according to claim 2, it is characterised in that the depth convolutional neural networks are trained in the following manner Obtain：

In the bottom of the depth convolutional neural networks structure, convolutional layer is set；

By the method in global average pond after last convolutional layer of the depth convolutional neural networks structure, will be every One characteristic pattern pond turns to a point；

Connection and the full articulamentum of aesthetic qualities class categories number identical and loss function.

4. method according to claim 1, it is characterised in that described based on the aesthetic feeling response diagram, screens the candidate Cutting image, specifically includes：

The aesthetic feeling retention score of candidate's cutting image is calculated by equation below：

$S_a\left(C\right)=\frac{{\mathrm{\Σ}}_{(i,j)\∈C}{A}_{(i,j)}}{{\mathrm{\Σ}}_{(i,j)\∈I}{A}_{(i,j)}}$

Wherein, the S_a(C) the aesthetic feeling retention score of candidate's cutting image is represented；The C represents candidate's cutting Image；(i, j) represents the position of pixel；The I represents original image；The A_(i,j)Represent the U.S. at (i, j) position Sense response；

All candidate's cutting images are ranked up from big to small according to the aesthetic feeling retention score；

Choose a part of candidate's cutting image of highest scoring.

5. method according to claim 1, it is characterised in that described based on the aesthetic feeling response diagram and the gradient energy Figure, the composition fraction of candidate's cutting image that estimation is filtered out, and candidate's cutting image of highest scoring is defined as into cutting figure Picture, specifically includes：

Composition model is set up based on the aesthetic feeling response diagram and the gradient energy figure；

Using the composition model estimate described in the composition fraction of candidate's cutting image that filters out, and by the highest scoring Candidate's cutting image is defined as the cutting image.

6. method according to claim 5, it is characterised in that the composition model is obtained in the following manner：

Training image collection is set up based on the aesthetic feeling response diagram and the gradient energy figure；

The mark of aesthetic qualities classification is carried out to training image；

Using the training image training depth convolutional neural networks of mark；

For the training image for having marked, using the depth convolutional neural networks for training, the aesthetic feeling response diagram is extracted With the spatial pyramid feature of the gradient energy figure；

By the spatial pyramid merging features for extracting together；

It is trained using grader, automatically study composition rule, obtains composition model.