CN106650737B - Image automatic cropping method - Google Patents

Abstract

The invention relates to an automatic image cropping method. The method comprises the following steps: extracting an aesthetic response graph and a gradient energy graph of an image to be cut; intensively extracting candidate clipping images from the image to be clipped; screening candidate cutting images based on the aesthetic feeling response graph; and estimating the composition score of the screened candidate trimming images based on the aesthetic feeling response graph and the gradient energy graph, and determining the candidate trimming image with the highest score as the trimming image. The scheme utilizes the aesthetic feeling response graph to explore the aesthetic feeling influence area of the picture, utilizes the aesthetic feeling response graph to determine the aesthetic feeling retaining part, thereby retaining the high aesthetic feeling quality of the cut image to the maximum extent, and simultaneously utilizes the gradient energy graph to analyze the gradient distribution rule and evaluates the composition score of the cut image based on the aesthetic feeling response graph and the gradient energy graph. The embodiment of the invention makes up the defect of image composition expression and solves the technical problem of how to improve the robustness and precision of automatic image cutting.

Description

Image automatic cropping method

technical field

The invention relates to the technical fields of pattern recognition, machine learning and computer vision, in particular to an image automatic cropping method.

Background technique

With the rapid development of computer technology and digital media technology, people's demands and expectations for computer vision, artificial intelligence, machine perception and other fields are also getting higher and higher. As a very important and common task in automatic image editing, automatic image cropping has also received more and more attention and development. The automatic image cropping technology hopes to remove redundant areas and emphasize areas of interest, thereby improving the overall composition and aesthetic quality of the image. An effective and automatic image cropping method can not only free humans from tedious work, but also provide some professional image editing suggestions to some non-professionals.

Since image cropping is a very subjective task, it is difficult for existing rules to take into account all influencing factors. Traditional automatic cropping regions of images usually use saliency maps to identify the main regions or regions of interest in the image, and at the same time, some rules are formulated to minimize energy functions or learn classifiers to find cropped regions. However, these rules are not comprehensive enough for the subjective task of image cropping, and the accuracy is difficult to meet user requirements.

In view of this, the present invention is proposed.

SUMMARY OF THE INVENTION

In order to solve the above problems in the prior art, that is, to solve the technical problem of how to improve the robustness and accuracy of automatic image cropping, an automatic image cropping method is provided.

In order to achieve the above purpose, the following technical solutions are provided:

An image automatic cropping method, the method includes:

Extract the aesthetic response map and gradient energy map of the image to be cropped;

Densely extract candidate cropped images from the to-be-cropped images;

based on the aesthetic response map, screening the candidate cropped images;

Based on the aesthetic response map and the gradient energy map, the composition scores of the filtered candidate cropped images are estimated, and the candidate cropped image with the highest score is determined as the cropped image.

Further, the extraction of the aesthetic response map and the gradient energy map of the image to be cropped specifically includes:

Using a deep convolutional neural network and a class response mapping method, the following formula is used to extract the aesthetic response map of the to-be-cropped image:

Wherein, the M(x, y) represents the aesthetic response value at the spatial position (x, y); the K represents the total number of channels of the feature map of the last convolutional layer of the deep convolutional neural network; The k represents the k th channel; the f _k (x, y) represents the eigenvalue of the k th channel at the spatial position (x, y); the w _k represents the k th channel The result of pooling the feature map of the channel to the weight of the high aesthetic category;

The to-be-cropped image is smoothed, and the gradient value of each pixel is calculated to obtain the gradient energy map.

Further, the deep convolutional neural network is obtained by training in the following manner:

A convolutional layer is arranged at the bottom layer of the deep convolutional neural network structure;

After the last convolutional layer of the deep convolutional neural network structure, each feature map is pooled into a point by the method of global average pooling;

Connect the same number of fully-connected layers and loss functions as the aesthetic quality classification classes.

Further, the screening of the candidate cropped images based on the aesthetic response map specifically includes:

The aesthetic preservation score of the candidate cropped image is calculated by the following formula:

Wherein, the S _a (C) represents the aesthetic preservation score of the candidate cropped image; the C represents the candidate cropped image; the (i, j) represents the position of the pixel; the I represents the original image ; The A _{(i, j)} represents the aesthetic response value at the (i, j) position;

Sort all candidate cropped images according to the beauty preservation score from large to small;

Pick a subset of candidate cropped images with the highest scores.

Further, the composition score of the screened candidate cropped images is estimated based on the aesthetic response map and the gradient energy map, and the candidate cropped image with the highest score is determined as the cropped image, specifically including:

establishing a composition model based on the aesthetic response map and the gradient energy map;

The composition score of the screened candidate cropped images is estimated by using the composition model, and the cropped image candidate with the highest score is determined as the cropped image.

Further, the composition model is obtained by:

establishing a training image set based on the aesthetic response map and the gradient energy map;

Label the training images with aesthetic quality categories;

Train a deep convolutional neural network with annotated training images;

For the marked training image, use the trained deep convolutional neural network to extract the spatial pyramid feature of the aesthetic response map and the gradient energy map;

stitch together the extracted spatial pyramid features;

The classifier is used for training, and the composition rules are automatically learned to obtain the composition model.

Embodiments of the present invention provide an automatic image cropping method. The method includes: extracting an aesthetic response map and a gradient energy map of an image to be cropped; intensively extracting candidate crop images from the to-be-cropped image; screening candidate crop images based on the aesthetic response map; estimating the selected candidate images based on the aesthetic response map and the gradient energy map The composition score of the cropped image, and the candidate cropped image with the highest score is determined as the cropped image. This scheme uses the aesthetic response map to explore the aesthetic influence area of the picture, and uses the aesthetic response map to determine the aesthetic retention part, thereby retaining the high aesthetic quality of the cropped image to the greatest extent. At the same time, this scheme also uses the gradient energy map to analyze the gradient distribution rules. , and the composition score of the crop is evaluated based on the aesthetic response map and the gradient energy map. The embodiments of the present invention make up for the defects of image composition expression, and solve the technical problem of how to improve the robustness and accuracy of automatic image cropping. The embodiments of the present invention can be applied to many fields involving automatic image cropping, including image editing, photography, and image relocation.

Description of drawings

1 is a schematic flowchart of an image automatic cropping method according to an embodiment of the present invention;

2 is a schematic structural diagram of a deep convolutional neural network according to an embodiment of the present invention;

3a is a schematic diagram of an image to be cropped according to an embodiment of the present invention;

FIG. 3b is a schematic diagram of a cropped image according to an embodiment of the present invention.

Detailed ways

The technical problems solved by the embodiments of the present invention, the technical solutions adopted, and the technical effects achieved will be described clearly and completely below with reference to the accompanying drawings and specific embodiments. Obviously, the described embodiments are only a part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other equivalent or obviously modified embodiments obtained by those of ordinary skill in the art without creative efforts fall within the protection scope of the present invention. Embodiments of the invention can be embodied in a number of different ways as defined and covered by the claims.

Deep learning has achieved rapid development and good results in various fields. The embodiment of the present invention considers using deep learning to automatically learn an important influence area for image cropping, so as to automatically and comprehensively learn the rules, so that the high aesthetic area is preserved as much as possible during cropping.

To this end, an embodiment of the present invention provides an automatic image cropping method. FIG. 1 exemplarily shows the flow of an image automatic cropping method. As shown in Figure 1, the method may include:

S100: Extract the aesthetic response map and the gradient energy map of the image to be cropped.

Specifically, this step may include:

S101: Using a deep convolutional neural network and a class response mapping method, and using the following formula to extract the aesthetic response map of the image to be cropped:

Among them, M(x,y) represents the aesthetic response value at the spatial position (x,y); K represents the total number of channels of the feature map f of the last convolutional layer of the trained deep convolutional neural network; k represents the k-th channel; f _k (x, y) represents the feature value of the k-th channel at the spatial position (x, y); w _k represents the result of pooling the feature map of the k-th channel to high aesthetics category weights.

The above steps can train a deep convolutional neural network according to actual needs when extracting the aesthetic response map. Training of deep convolutional neural networks can be done in the following ways:

Step 1: Set up convolutional layers at the bottom layer of the deep convolutional neural network structure.

Step 2: After the last convolutional layer of the deep convolutional neural network structure, each feature map is pooled into a point by the method of global average pooling.

Step 3: Connect a fully connected layer and loss function with the same number of categories as the aesthetic quality classification.

Figure 2 exemplarily shows a deep convolutional neural network structure.

Through steps 1-3, a deep convolutional neural network model under the task of aesthetic quality classification can be trained. Then, using the deep convolutional neural network trained for the aesthetic quality classification task and the category response mapping method; and then using the above formula, the aesthetic response map M of the image to be cropped under the high aesthetic category can be calculated.

S102: Smooth the to-be-cropped image, and calculate the gradient value of each pixel to obtain a gradient energy map.

S110: Densely extract candidate cropped images from the to-be-cropped images.

Here, sliding windows of all sizes smaller than the image size can be used, and candidate cropping windows are intensively extracted from the to-be-cropped image, and candidate cropping images are extracted through the candidate cropping windows.

S120: Screen candidate cropped images based on the aesthetic response map.

Specifically, this step may include:

S121: Calculate the beauty preservation score of the candidate cropped image by the following formula:

Among them, S _a (C) represents the aesthetic preservation score of the candidate cropped image; C represents the candidate cropped image; (i, j) represents the position of the pixel; I represents the original image; A _{(i, j)} represents at (i, j) Aesthetic response value at .

Through this step, an aesthetic preservation model can be constructed. The candidate cropping window is filtered through the aesthetics preservation model to select the candidate window with higher aesthetics preservation score.

S122: Sort all candidate cropped images in descending order according to their beauty preservation scores.

S123: Select a part of candidate cropped images with the highest score.

For example, in practical applications, the candidate cropped images in the first 10,000 cropped candidate windows can be set to be retained.

S130: Based on the aesthetic response map and the gradient energy map, estimate the composition score of the selected candidate cropped images, and determine the candidate cropped image with the highest score as the cropped image.

Specifically, this step can be implemented through steps S131 to S133.

S131: Establish a composition model based on the aesthetic response map and the gradient energy map.

In this step, the composition model can be trained according to the actual situation when the composition model is established. In the process of training the composition model, the training data can use images with good composition as positive samples, and images with composition defects as negative samples.

The composition model can be trained in the following ways:

Step a: Build a training image set based on the aesthetic response map and the gradient energy map.

Step b: Label the training images with aesthetic quality categories.

Step c: Train a deep convolutional neural network with annotated training images.

For the training process of this step, reference may be made to the above-mentioned steps 1 to 3, which will not be repeated here.

Step d: For the labeled training images, use the trained deep convolutional neural network to extract the spatial pyramid features of the aesthetic response map and the gradient energy map.

Step e: stitch together the extracted spatial pyramid features.

Step f: Use the classifier for training, automatically learn composition rules, and obtain a composition model.

The classifier can be, for example, a support vector machine classifier.

S132: Use the composition model to estimate the composition score of the selected candidate cropped images, and determine the candidate cropped image with the highest score as the cropped image.

Fig. 3a exemplarily shows the image to be cropped; Fig. 3b exemplarily shows the cropped image.

Hereinafter, the present invention will be better described with a preferred embodiment.

Step A: Send the image dataset marked with the aesthetic quality category into the deep convolutional neural network for training the aesthetic quality category model.

Step B: Input the image dataset marked with the composition category into the trained deep convolutional neural network, extract the feature map of the last convolutional layer, calculate the aesthetic response map, and calculate the aesthetic gradient map at the same time, and then use the support vector machine. The classifier trains the composition model.

Step C: Extract the aesthetic response map and the gradient energy map from the test image.

For the extraction method in this step, refer to the method in the training phase.

Step D: Densely collect candidate cropping windows of the image to be tested.

For example, on a 1000×1000 image to be tested, a sliding window with an interval of 30 pixels is used to collect or extract.

Step E: Screen candidate cropping windows using an aesthetic preservation model.

In this step, the beauty retention model is used to calculate the beauty retention scores of the densely collected candidate cropping windows, and a part of the candidate cropping windows with the highest aesthetics classification is selected, for example, 10,000 candidate cropping windows are screened out.

Step F: Use the composition model to evaluate the screened candidate cropping windows.

In this step, the composition model trained in the training stage is collected to evaluate the composition scores of the selected candidate cropping windows, and the cropped image is obtained by taking the one with the highest score as the final cropping window.

To sum up, the method provided by the embodiment of the present invention makes good use of the aesthetic response map and the gradient energy map to maximize the preservation of the aesthetic quality and the composition rules of the image, so as to obtain a more robust and accurate automatic cropping of the image. performance, which further illustrates the effectiveness of aesthetic response maps and gradient energy maps for automatic image cropping.

Although the method provided by the embodiment of the present invention is described in the above-mentioned order in the above-mentioned embodiment, those skilled in the art can understand that, in order to achieve the effect of this embodiment, it can also be executed in different orders such as parallel or reversed order, etc. These simple variations are within the scope of the present invention.

The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited to this, any person familiar with the technology can understand the transformation or replacement that comes to mind within the technical scope disclosed by the present invention, All should be included within the scope of the present invention, therefore, the protection scope of the present invention should be subject to the protection scope of the claims.

Claims (5)

1. an image automatic cropping method, is characterized in that, described method comprises: Extract the aesthetic response map and gradient energy map of the image to be cropped; Densely extract candidate cropped images from the to-be-cropped images; based on the aesthetic response map, screening the candidate cropped images; Based on the aesthetic response map and the gradient energy map, the composition score of the screened candidate cropped images is estimated, and the candidate cropped image with the highest score is determined as the cropped image; The extraction of the aesthetic response map and the gradient energy map of the to-be-cropped image specifically includes: Using a deep convolutional neural network and a class response mapping method, the following formula is used to extract the aesthetic response map of the to-be-cropped image:

Wherein, the M(x, y) represents the aesthetic response value at the spatial position (x, y); the K represents the total number of channels of the feature map of the last convolutional layer of the deep convolutional neural network; The k represents the k th channel; the f _k (x, y) represents the eigenvalue of the k th channel at the spatial position (x, y); the w _k represents the k th channel The result of pooling the feature map of the channel to the weight of the high aesthetic category; The to-be-cropped image is smoothed, and the gradient value of each pixel is calculated to obtain the gradient energy map. 2. The method according to claim 1, wherein the deep convolutional neural network is obtained by training in the following manner: A convolutional layer is arranged at the bottom layer of the deep convolutional neural network structure; After the last convolutional layer of the deep convolutional neural network structure, each feature map is pooled into a point by the method of global average pooling; Connect the same number of fully-connected layers and loss functions as the aesthetic quality classification classes. 3. The method according to claim 1, wherein the screening of the candidate cropped images based on the aesthetic response map, specifically comprises: The aesthetic preservation score of the candidate cropped image is calculated by the following formula:

Wherein, the S _a (C) represents the aesthetic preservation score of the candidate cropped image; the C represents the candidate cropped image; the (i, j) represents the position of the pixel; the I represents the original image ; The A _{(i, j)} represents the aesthetic response value at the (i, j) position; Sort all candidate cropped images according to the beauty preservation score from large to small; Pick a subset of candidate cropped images with the highest scores. 4. The method according to claim 1, wherein, based on the aesthetic response map and the gradient energy map, the composition score of the screened candidate cropped images is estimated, and the candidate cropped image with the highest score is determined For cropping the image, including: establishing a composition model based on the aesthetic response map and the gradient energy map; The composition score of the screened candidate cropped images is estimated by using the composition model, and the cropped image candidate with the highest score is determined as the cropped image. 5. The method according to claim 4, wherein the composition model is obtained by: establishing a training image set based on the aesthetic response map and the gradient energy map; Label the training images with aesthetic quality categories; Train a deep convolutional neural network with annotated training images; For the marked training image, use the trained deep convolutional neural network to extract the spatial pyramid feature of the aesthetic response map and the gradient energy map; stitch together the extracted spatial pyramid features; The classifier is used for training, and the composition rules are automatically learned to obtain the composition model.

Images