CN113808014A - Image scaling method and device based on dynamic energy adjustment - Google Patents
Image scaling method and device based on dynamic energy adjustment Download PDFInfo
- Publication number
- CN113808014A CN113808014A CN202110868971.8A CN202110868971A CN113808014A CN 113808014 A CN113808014 A CN 113808014A CN 202110868971 A CN202110868971 A CN 202110868971A CN 113808014 A CN113808014 A CN 113808014A
- Authority
- CN
- China
- Prior art keywords
- energy
- image
- map
- cutting line
- preset
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000000034 method Methods 0.000 title claims abstract description 48
- 230000004927 fusion Effects 0.000 claims abstract description 71
- 238000010586 diagram Methods 0.000 claims abstract description 30
- 230000000007 visual effect Effects 0.000 claims abstract description 21
- 230000002093 peripheral effect Effects 0.000 claims description 20
- 230000004913 activation Effects 0.000 claims description 10
- 238000004364 calculation method Methods 0.000 claims description 9
- 238000004590 computer program Methods 0.000 claims description 7
- 238000010606 normalization Methods 0.000 claims description 5
- 230000007704 transition Effects 0.000 claims description 5
- 230000007246 mechanism Effects 0.000 claims description 4
- 230000001105 regulatory effect Effects 0.000 claims 1
- 238000009966 trimming Methods 0.000 description 18
- 230000008859 change Effects 0.000 description 5
- 230000008901 benefit Effects 0.000 description 2
- 238000000605 extraction Methods 0.000 description 2
- 238000013507 mapping Methods 0.000 description 2
- 239000011159 matrix material Substances 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 238000005070 sampling Methods 0.000 description 2
- 230000006870 function Effects 0.000 description 1
- 230000000630 rising effect Effects 0.000 description 1
- 238000000926 separation method Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T3/00—Geometric image transformations in the plane of the image
- G06T3/40—Scaling of whole images or parts thereof, e.g. expanding or contracting
- G06T3/4023—Scaling of whole images or parts thereof, e.g. expanding or contracting based on decimating pixels or lines of pixels; based on inserting pixels or lines of pixels
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/29—Graphical models, e.g. Bayesian networks
- G06F18/295—Markov models or related models, e.g. semi-Markov models; Markov random fields; Networks embedding Markov models
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20016—Hierarchical, coarse-to-fine, multiscale or multiresolution image processing; Pyramid transform
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Data Mining & Analysis (AREA)
- Artificial Intelligence (AREA)
- Life Sciences & Earth Sciences (AREA)
- Bioinformatics & Cheminformatics (AREA)
- Bioinformatics & Computational Biology (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Evolutionary Biology (AREA)
- Evolutionary Computation (AREA)
- General Engineering & Computer Science (AREA)
- Image Processing (AREA)
Abstract
The invention relates to an image zooming method, an image zooming device, a storage medium and an electronic device based on dynamic energy adjustment, wherein the method comprises the following steps: the method comprises the steps of calculating a gradient energy map and a visual salient energy map of a first image by obtaining the first image to be zoomed, fusing the gradient energy map and the visual salient energy map to obtain a first fused energy map, obtaining a cutting line by using a dynamic programming method, adjusting energy values of left and right preset pixel points of a vertical cutting line by line, adjusting energy values of upper and lower preset pixel points of a horizontal cutting line by line, deleting the vertical cutting line and the horizontal cutting line from the first fused energy map to obtain an adjusted second fused energy map, re-determining the cutting line according to the second fused energy map to cut until the size of the second image meets a preset target size, and outputting the second image. By dynamically adjusting the fusion energy diagram, a more appropriate cutting line is selected, and the precision of image zooming is improved.
Description
Technical Field
The present invention relates to the field of image scaling, and in particular, to an image scaling method and apparatus based on dynamic energy adjustment, a storage medium, and an electronic device.
Background
When the existing scheme zooms an image, a line cropping (team clipping) method is generally adopted to zoom the image. The line cropping method is a content-aware-based image scaling method, and image scaling is achieved by removing or inserting a cropping line with the least accumulated energy. After each cutting by using the line cutting method, the pixels at the two sides of the cutting line become new neighbors, so that the energy of the pixels at the two sides of the cutting line is increased, and the image energy after the cutting line is cut is also increased. For example, the energy of the original image is E, and the energy of the clipping line is EsThe image energy after cutting off the cutting line is EoAt this time, EoGreater than E minus Es。
However, the conventional line clipping method only focuses on processing the original energy map, ignores the increase of image energy caused by clipping, and further gradually accumulates the increase of image energy with the increase of the clipping times, so that the subsequently determined clipping line position is more and more deviated from the actual position, and further the clipping line set is easy to cause the problem, and finally the zoomed image is distorted.
Disclosure of Invention
In view of the foregoing, it is an object of the present invention to provide an image scaling method, apparatus, medium, and electronic device based on dynamic energy adjustment, which have the advantage of reducing image scaling distortion.
According to a first aspect of embodiments of the present application, there is provided an image scaling method based on dynamic energy adjustment, including the following steps:
acquiring a first image to be zoomed;
calculating a gradient energy map and a visually significant energy map of the first image;
fusing the gradient energy map and the visual saliency energy map to obtain a first fused energy map;
in the first fusion energy diagram, cutting lines are obtained by using a dynamic programming method, wherein the cutting lines comprise vertical cutting lines and horizontal cutting lines;
in the first fusion energy map, adjusting energy values of left and right preset pixels of the vertical cutting line by line, adjusting energy values of upper and lower preset pixels of the horizontal cutting line by line, and deleting the vertical cutting line and the horizontal cutting line from the first fusion energy map to obtain an adjusted second fusion energy map;
deleting the pixel points corresponding to the vertical cutting lines and the pixel points corresponding to the horizontal cutting lines in the first image to obtain a zoomed second image;
and when the size of the second image does not meet the preset target size, re-determining the cutting line according to the second fusion energy map for cutting until the size of the second image meets the preset target size, and outputting the second image.
According to a second aspect of embodiments of the present application, there is provided an image scaling apparatus based on dynamic energy adjustment, including:
the first acquisition module is used for acquiring a first image to be zoomed;
a calculation module for calculating a gradient energy map and a visually significant energy map of the first image;
the fusion module is used for fusing the gradient energy map and the visual saliency energy map to obtain a first fusion energy map;
the second acquisition module is used for acquiring cutting lines in the first fusion energy map by using a dynamic programming method, wherein the cutting lines comprise vertical cutting lines and horizontal cutting lines;
the first deleting module is used for adjusting the energy values of left and right preset pixel points of the vertical cutting line by line, adjusting the energy values of upper and lower preset pixel points of the horizontal cutting line by line in the first fused energy map, and deleting the vertical cutting line and the horizontal cutting line from the first fused energy map to obtain an adjusted second fused energy map;
the second deleting module is used for deleting the pixel points corresponding to the vertical cutting lines and the pixel points corresponding to the horizontal cutting lines in the first image to obtain a zoomed second image;
and the output module is used for re-determining the cutting line for cutting according to the second fusion energy map when the size of the second image does not meet the preset target size until the size of the second image meets the preset target size, and outputting the second image.
According to a third aspect of embodiments of the present application, there is provided an electronic apparatus, including: a processor and a memory; wherein the memory stores a computer program adapted to be loaded by the processor and to perform the image scaling method based on dynamic energy adjustment as defined in any of the above.
According to a fourth aspect of embodiments of the present application, there is provided a computer-readable storage medium, on which a computer program is stored, which, when being executed by a processor, implements the image scaling method based on dynamic energy adjustment as recited in any one of the above.
In the embodiment of the application, a gradient energy diagram and a visual saliency energy diagram of a first image are calculated by obtaining the first image to be zoomed, the gradient energy diagram and the visual saliency energy diagram are fused to obtain a first fusion energy diagram, a cutting line is obtained by using a dynamic programming method in the first fusion energy diagram, the cutting line comprises a vertical cutting line and a horizontal cutting line, the energy values of left and right preset pixel points of the vertical cutting line are adjusted line by line in the first fusion energy diagram, the energy values of upper and lower preset pixel points of the horizontal cutting line are adjusted line by line, the vertical cutting line and the horizontal cutting line are deleted from the first fusion energy diagram, an adjusted second fusion energy diagram is obtained, and pixel points corresponding to the vertical cutting line and pixel points corresponding to the horizontal cutting line are deleted in the first image, and obtaining a zoomed second image, re-determining the cutting line for cutting according to the second fusion energy graph when the size of the second image does not meet the preset target size, and outputting the second image until the size of the second image meets the preset target size. By dynamically adjusting the fusion energy diagram, the energy change caused by each cutting can be displayed, so that a more appropriate cutting line is selected, and the image zooming precision is improved.
It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the application.
For a better understanding and practice, the invention is described in detail below with reference to the accompanying drawings.
Drawings
FIG. 1 is a schematic flow chart of an image scaling method based on dynamic energy adjustment according to the present invention;
FIG. 2 is a flowchart illustrating the calculation of the visual saliency map at S20 in the image scaling method based on dynamic energy adjustment according to the present invention;
FIG. 3 is a schematic flow chart of S22 in the image scaling method based on dynamic energy adjustment according to the present invention;
FIG. 4 is a schematic flow chart of S50 in the image scaling method based on dynamic energy adjustment according to the present invention;
FIG. 5 is a schematic flow chart of S52 in the image scaling method based on dynamic energy adjustment according to the present invention;
FIG. 6 is a block diagram of an image scaling apparatus based on dynamic energy adjustment according to the present invention;
FIG. 7 is a block diagram of a first deleting module 65 of the image scaling apparatus based on dynamic energy adjustment according to the present invention;
FIG. 8 is a block diagram of the image scaling device obtaining unit 654 according to the present invention;
FIG. 9 is a block diagram of the computing module 62 for computing the visual saliency map based on the dynamic energy adjustment of the present invention;
fig. 10 is a block diagram of the image scaling device extraction unit 624 based on dynamic energy adjustment according to the present invention.
Detailed Description
In order to make the objects, technical solutions and advantages of the present application more clear, embodiments of the present application will be described in further detail below with reference to the accompanying drawings.
It should be understood that the embodiments described are only a few embodiments of the present application, and not all embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present application.
The terminology used in the embodiments of the present application is for the purpose of describing particular embodiments only and is not intended to be limiting of the embodiments of the present application. As used in the examples of this application and the appended claims, the singular forms "a", "an", and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It should also be understood that the term "and/or" as used herein refers to and encompasses any and all possible combinations of one or more of the associated listed items.
When the following description refers to the accompanying drawings, like numbers in different drawings represent the same or similar elements unless otherwise indicated. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present application. Rather, they are merely examples of apparatus and methods consistent with certain aspects of the application, as detailed in the appended claims. In the description of the present application, it is to be understood that the terms "first," "second," "third," and the like are used solely to distinguish one from another and are not necessarily used to describe a particular order or sequence, nor are they to be construed as indicating or implying relative importance. The specific meaning of the above terms in the present application can be understood by those of ordinary skill in the art as appropriate.
Further, in the description of the present application, "a plurality" means two or more unless otherwise specified. "and/or" describes the association relationship of the associated objects, meaning that there may be three relationships, e.g., a and/or B, which may mean: a exists alone, A and B exist simultaneously, and B exists alone. The character "/" generally indicates that the former and latter associated objects are in an "or" relationship.
Referring to fig. 1, an embodiment of the invention provides an image scaling method based on dynamic energy adjustment, including the following steps:
s10, acquiring a first image to be zoomed.
It can be understood that, when the image is zoomed, a preset scaling and a scaling direction which need to be zoomed are also obtained, and then the target size can be determined according to the scaling and the scaling direction. The scaling represents a ratio of a size of the target image after the scaling is completed to a size of the first image to be scaled, where the scaling directions include a horizontal scaling direction, a vertical scaling direction, and a horizontal and vertical simultaneous scaling direction, for example, a horizontal direction is set to 0.6, which represents that the original image is scaled to 60% of the original image size along an x-axis horizontal direction. Optionally, the scaling and the manner of the zooming direction may be obtained by inputting corresponding instruction parameters in the image zooming interface by the user.
And S20, calculating a gradient energy map and a visual salient energy map of the first image.
In this embodiment of the present application, if the first image to be scaled is represented as I, the size is n × m, n is the number of pixels per column in the longitudinal direction of the first image I, and m is the number of pixels per row in the transverse direction of the first image I, the gradient energy E of each pixel in the first image to be scaled is:
according to the formula, the gradient of each pixel gray value in the x axis and the gradient of each pixel gray value in the y axis are extracted according to a Sobel operator, and then the absolute values of the gradients of each pixel gray value in the x axis and the y axis are added to obtain the gradient energy value of each pixel. Wherein, the larger the gradient energy E of a pixel is, the more important the pixel is, and vice versa, the smaller the gradient energy E of the pixel is.
And S30, fusing the gradient energy map and the visual salient energy map to obtain a first fused energy map.
In the embodiment of the application, the gradient energy value of each pixel point in the gradient energy map and the significant energy value of each pixel point at the corresponding position of the visual significant energy map are summed to obtain a first fusion energy map.
S40, in the first fusion energy diagram, obtaining cutting lines by using a dynamic programming method, wherein the cutting lines comprise vertical cutting lines and horizontal cutting lines;
in the embodiment of the present application, in the first fusion energy map, the vertical clipping line is defined as follows:
where x is the mapping of the pixel matrix n to m, indicating its position in each row.
Similarly, the horizontal trim line is defined as follows:
where y is the mapping of the pixel matrix m to n, indicating its position in each column.
Line cropping corresponds to removing a row or column from an image, and if vertical crop lines are removed from the image, the vertical crop lines are determined by the order from top to bottom in the image. In two adjacent lines of the vertical cutting line, three modes are provided for selecting pixel points: right top, left top and right top. Setting the energy sum of each vertical trimming line as E (S), the optimal vertical trimming line is the minimum value of E (S). The vertical cutting line is obtained through a dynamic programming method, the energy of each pixel in the image I is set as e (I, j), and the accumulated energy value M (I, j) is defined as follows:
M(i,j)＝e(i,j)+min(M(i-1,j-1),M(i-1,j),M(i-1,j+1))
from the second row to the last row of the image I, the accumulated energy value of the pixels of each row is the energy of the pixel plus the energy of the least energetic pixel of the three pixels in the previous row, and then all the pixels in the vertical clipping line are identified in a backtracking manner. Similarly, for longitudinal zooming, the horizontal trimming line needs to be removed, the original image can be transversely zoomed after being rotated by 90 degrees, and the vertical trimming line is removed according to the method.
S50, in the first fusion energy map, adjusting energy values of left and right preset pixels of the vertical cutting line by line, adjusting energy values of upper and lower preset pixels of the horizontal cutting line by line, and deleting the vertical cutting line and the horizontal cutting line from the first fusion energy map to obtain an adjusted second fusion energy map.
Since the overall energy in the fused energy map increases after each crop. Therefore, in the embodiment of the application, it is assumed that after each cutting, the left and right preset pixels of the vertical cutting line and the upper and lower preset pixels of the horizontal cutting line obtain an energy added value, and the adjusted second fusion energy map is obtained by adjusting the energy value of the preset pixels through the energy added value, so that the change of an energy structure in the whole fusion energy map can be simulated, a more appropriate cutting line is selected, and the precision of image scaling is improved.
S60, deleting pixel points corresponding to the vertical cutting lines and pixel points corresponding to the horizontal cutting lines in the first image to obtain a zoomed second image;
in the embodiment of the application, in the first image, pixel points on a vertical cutting line are deleted, pixels on the right side of the vertical cutting line are all moved to the left by 1 position, pixel points on a horizontal cutting line are deleted, and pixels on the upper side of the horizontal cutting line are all moved to the downward by 1 position, so that a zoomed second image is obtained.
And S70, when the size of the second image does not meet the preset target size, re-determining the cutting line according to the second fusion energy map for cutting until the size of the second image meets the preset target size, and outputting the second image.
In the embodiment of the application, the number of times of cutting is calculated according to the preset target size needing to be zoomed, the cutting line is determined again according to the second fusion energy graph to be cut, the energy values of the left and right preset pixel points of the vertical cutting line are adjusted line by line, the energy values of the upper and lower preset pixel points of the horizontal cutting line are adjusted line by line, and then the appointed number of times of cutting is circularly performed until the size of the second image meets the preset target size, so that the final target zoomed image is obtained. When the cutting line is determined again for the second fusion energy diagram, normalization processing is carried out on the second fusion energy diagram, and the phenomenon that after multiple times of energy adjustment, the overall energy of the fusion energy diagram rises, so that subsequent adjustment is relatively weak is avoided.
By applying the embodiment of the invention, a gradient energy map and a visual salient energy map of a first image to be zoomed are calculated by obtaining the first image to be zoomed, the gradient energy map and the visual salient energy map are fused to obtain a first fused energy map, in the first fused energy map, a dynamic programming method is used for obtaining a cutting line, the cutting line comprises a vertical cutting line and a horizontal cutting line, in the first fused energy map, energy values of left and right preset pixel points of the vertical cutting line are adjusted line by line, energy values of upper and lower preset pixel points of the horizontal cutting line are adjusted line by line, the vertical cutting line and the horizontal cutting line are deleted from the first fused energy map, an adjusted second fused energy map is obtained, in the first image, pixel points corresponding to the vertical cutting line and pixel points corresponding to the horizontal cutting line are deleted, and obtaining a zoomed second image, re-determining the cutting line for cutting according to the second fusion energy graph when the size of the second image does not meet the preset target size, and outputting the second image until the size of the second image meets the preset target size. By dynamically adjusting the fusion energy diagram, the energy change caused by each cutting can be displayed, so that a more appropriate cutting line is selected, and the image zooming precision is improved.
In an alternative embodiment, referring to fig. 2, the step S20 of calculating the visually significant energy map includes steps S21-S24, which are as follows:
and S21, carrying out Gaussian blur and downsampling processing on the first image to obtain a Gaussian pyramid image.
In the embodiment of the application, the first image is subjected to gaussian pyramid operation, that is, the first image is subjected to gaussian blurring processing and then down-sampled, so that images of 9 scales in total including the original image scale are obtained. Wherein, the ratio of the area of the image under the scale 0 to the area of the original image is 1:1, and from the scale 0 to the scale 8, the ratio of the area of the down-sampled image to the area of the original image is [1: 1; 1: 2; 1: 4; 1: 8; 1: 16; 1: 32; 1: 64; 1: 128; 1:256], an image of one image at 9 scales, that is, 9 gaussian pyramid images were obtained.
And S22, extracting the gray characteristic, the color channel characteristic and the direction characteristic of the Gaussian pyramid image to obtain a characteristic diagram.
In the embodiment of the application, bottom layer features of 9 pairs of gaussian pyramid images are extracted to obtain a feature map, wherein the bottom layer features include a gray level feature, a color channel feature and a direction feature.
S23, converting the feature graph into a full-connection graph, taking nodes in the full-connection graph as states of a Markov chain, taking edge weights in the full-connection graph as transition probabilities of the states in the Markov chain, and obtaining an activation graph corresponding to the full-connection graph;
in the embodiment of the present application, each feature graph is converted into a directed full-connected graph, and the connection weight is defined as follows:
w((i，j)，(p，q))＝d((i，j)，(p，q))*F(i-p，j-q)
m denotes the characteristic diagram, d ((i, j), (p, q)) denotes a difference in pixel value between two points (i, j) and (p, q), F (i-p, j-q) denotes a difference in distance between two points (i, j) and (p, q), δ is a constant, and w ((i, j), (p, q)) is a weight between two points (i, j) and (p, q), thereby obtaining a full-connected diagram. Then for each of the n nodes, there are n ^2-1 edges pointing from this node to the other nodes, so there are n ^2-1 edge weights, which n ^2-1 edge weights result in a sum, each edge weight is now multiplied by a coefficient, making this sum 1. And making the nodes and the states in the Markov chain equivalent, and making the edge weights and the transition probabilities equivalent, namely, taking the nodes in the full-connection graph as the states of the Markov chain, taking the edge weights as the transition probabilities of the states in the Markov chain, and obtaining an activation graph from each feature graph.
And S24, normalizing and summing the activation graphs to obtain the visual saliency energy graph.
In the embodiment of the application, the activation maps are normalized, activation maps of the same type are added to obtain a gray level saliency map, a color channel saliency map and a direction saliency map, and the visual saliency energy map is obtained by summing image energy ratios of the gray level saliency map, the color channel saliency map and the direction saliency map as weights.
In an alternative embodiment, referring to fig. 3, the step S22 includes steps S221 to S222, which are as follows:
s221, extracting the gray characteristic, the color channel characteristic and the direction characteristic of the Gaussian pyramid image to obtain a gray characteristic pyramid image, a color channel characteristic pyramid image and a direction characteristic pyramid image;
in the embodiment of the present application, the grayscale feature I is (R + G + B)/3, R, G, and B are three component values of red, green, and blue of the gaussian pyramid image, respectively, the color channel feature includes R, B, G, Y four channel features, R ═ R- (G + B)/2, G ═ G- (R + B)/2, B ═ B- (R + G)/2, Y ═ R + G)/2 | -R-G |/2-B, the directional feature is obtained by using a Gabor pyramid for the grayscale feature I, represented by O (σ, θ), σ ∈ {0,1,2,3,4,5,6,7,8} represents an image with a scale σ, θ ∈ {0,45,90,135}, and the unit is degree, wherein the Gabor pyramid is a linear filter for edge extraction, is suitable for texture expression and separation.
S222, acquiring a gray characteristic image corresponding to the gray characteristic pyramid image, a color channel characteristic image corresponding to the color channel characteristic pyramid image and a direction characteristic image corresponding to the direction characteristic pyramid image under the combination of a central space scale and a peripheral space scale according to a central peripheral difference mechanism;
in the embodiment of the present application, after gaussian blurring and downsampling are performed on the first image, gaussian pyramid images at 9 scales are obtained. From the 9 dimensions {0,1,2,3,4,5,6,7,8}, a central spatial dimension and a peripheral spatial dimension are selected, the central spatial dimension c belongs to {2,3,4}, and the peripheral spatial dimension s belongs to {5,6,7,8}, so that 6 combinations of (c, s) { (2,5), (2,6), (3,6), (3,7), (4,7), (4,8) } are obtained. And acquiring a gray characteristic image corresponding to the gray characteristic pyramid image, a color channel characteristic image corresponding to the color channel characteristic pyramid image and a direction characteristic image corresponding to the direction characteristic pyramid image under the combination of the central space scale and the surrounding space scale according to a central peripheral difference mechanism (space scale). Wherein, the calculation formula is:
I(c,s)＝|I(c)ΘI(s)|
RG(c,s)＝|(R(c)-G(c))Θ(G(s)-R(s))|
BY(c,s)＝|(B(c)-Y(c))Θ(Y(s)-B(s))|
O(c,s,θ)＝|O(c,θ)ΘO(s,θ)|
wherein Θ represents a subtraction of the feature map at the central spatial scale and the feature map at the peripheral spatial scale, I (c, s) represents a grayscale feature map corresponding to the grayscale feature pyramid image at a combination (c, s) of the central spatial scale and the peripheral spatial scale, RG (c, s) represents a red-green channel feature map corresponding to the color channel feature pyramid image at a combination (c, s) of the central spatial scale and the peripheral spatial scale, BY (c, s) represents a blue-yellow channel feature map corresponding to the color channel feature pyramid image at a combination (c, s) of the central spatial scale and the peripheral spatial scale, and O (c, s, θ) represents an orientation feature map corresponding to the orientation feature pyramid image at a combination (c, s) of the central spatial scale and the peripheral spatial scale.
In an optional embodiment, after the step S30, the method further includes step S32, which is as follows:
s32, normalizing the energy values of all the pixel points on the first fusion energy graph; the normalization processing mode is as follows:
and (4) the converted energy value of the pixel point is equal to (the energy value before the pixel point is converted-the minimum energy value of the pixel point on the first fusion energy map)/(the maximum energy value of the pixel point on the fusion energy map-the minimum energy value of the pixel point on the first fusion energy map).
In the embodiment of the present application, in order to avoid the overall energy of the fused energy map rising after multiple energy adjustments, and make subsequent adjustments relatively weak, the energy of the pixel needs to be kept within a certain range. Therefore, the energy values of all the pixel points on the first fusion energy diagram are normalized, so that the energy value range is changed into a range from 0 to 1.
In an alternative embodiment, referring to fig. 4, the step S50 further includes steps S51-S53, which are as follows:
s51, acquiring an average energy value in the first fusion energy map, energy values of left and right preset pixels of the vertical cutting line, distances from the left and right preset pixels to the vertical cutting line, energy values of upper and lower preset pixels of the horizontal cutting line and distances from the upper and lower preset pixels to the horizontal cutting line;
s52, obtaining an energy adjusting value corresponding to the preset pixel point according to the average energy value in the first fusion energy map, the distance between the preset pixel point and the cutting line and the energy value of the preset pixel point.
In this embodiment, the magnitude of the energy adjustment value is related to the average energy value, the energy value of the preset pixel point, and the distance from the preset pixel point to the trimming line. Specifically, the smaller the difference between the energy value of the preset pixel point and the average energy value is, the higher the significance of the preset pixel point is, and the larger the corresponding energy adjustment value is. Similarly, the larger the distance from the preset pixel point to the cutting line is, the smaller the corresponding energy adjusting value is.
And S53, adding the energy values of the preset pixel points to the corresponding energy adjusting values.
In the embodiment of the application, the energy values of the preset pixel points are added with the corresponding energy adjusting values, so that the energy change caused by each cutting line in an energy diagram is equivalent, the more suitable cutting line is convenient to select, and the distortion of image scaling is reduced.
In an alternative embodiment, referring to fig. 5, step S52 further includes steps S521 to S523, which are as follows:
and S521, obtaining an energy adjusting item of the preset pixel point according to the average energy value of the first fusion energy map and the energy value of the preset pixel point.
In this embodiment of the present application, in the first fusion energy map, the energy value of each preset pixel point i is viAnd a represents an average energy value of the first fused energy map, in order to protect the original structure of the fused energy map by adjusting energy, the concentrated clipping lines become more dispersed, resulting in some clipping lines extending to a significant region. In order to avoid excessive cutting of the significant region, the sigmoid function is adopted to adjust the energy, so that the energy adjusting item of the preset pixel point is obtained. Wherein, the calculation formula is:
qirepresenting said energy regulation term, riDenotes an energy regulation parameter, w and M are constants, qiThe upper limit of (1) avoids the overlarge difference between the energy values of some points in the non-significant area due to multiple adjustments, so that the energy adjustment of the non-significant area is more uniform, and the original point of the image is protectedThe original structure is not destroyed.
S522, obtaining a distance adjusting item of the preset pixel point according to the distance between the preset pixel point and the cutting line.
The larger the energy adjustment range, the better the protection of the image structure. In the embodiment of the present application, in order to make the range of each energy adjustment as large as possible, a distance adjustment term d (n) of the preset pixel point is obtained according to a distance n between the preset pixel point and the trimming line, wherein d (n) (2/3)n-1And the distance adjusting item can prevent the difference of the distance adjusting items between the pixels with different distances from the cutting line from being too small, thereby avoiding the generation of saw-toothed shapes after the image is cut. Meanwhile, the distance adjusting item is not rapidly reduced along with the increase of the distance, so that the point far away from the cutting line can be adjusted, and the energy adjusting range is enlarged.
S523, obtaining an energy adjusting value corresponding to the preset pixel point according to the product of the energy adjusting item and the distance adjusting item;
in the embodiment of the present application, an energy adjustment value corresponding to each preset pixel point is obtained according to a product of an energy adjustment item of each preset pixel point and a distance adjustment item corresponding to each preset pixel point. Wherein, the calculation formula is:
ΔEi＝qi*d(n)
ΔEiand expressing the energy adjusting value corresponding to the preset pixel point i.
Referring to fig. 6, an embodiment of the present invention provides an image scaling apparatus 6 based on dynamic energy adjustment, including:
a first obtaining module 61, configured to obtain a first image to be zoomed;
a calculation module 62 for calculating a gradient energy map and a visually significant energy map of the first image;
a fusion module 63, configured to fuse the gradient energy map and the visually significant energy map to obtain a first fusion energy map;
a second obtaining module 64, configured to obtain a trimming line in the first fusion energy map by using a dynamic programming method, where the trimming line includes a vertical trimming line and a horizontal trimming line;
a first deleting module 65, configured to adjust energy values of left and right preset pixels of the vertical trimming line row by row in the first fused energy map, adjust energy values of upper and lower preset pixels of the horizontal trimming line row by row, and delete the vertical trimming line and the horizontal trimming line from the first fused energy map, so as to obtain an adjusted second fused energy map;
a second deleting module 66, configured to delete, in the first image, a pixel point corresponding to the vertical trimming line and a pixel point corresponding to the horizontal trimming line, so as to obtain a zoomed second image;
and the output module 67 is configured to, when the size of the second image does not meet a preset target size, re-determine the clipping line according to the second fusion energy map for clipping until the size of the second image meets the preset target size, and output the second image.
Optionally, referring to fig. 7, the first deleting module 65 includes:
an obtaining unit 652, configured to obtain an average energy value in the first fusion energy map, energy values of left and right preset pixels of the vertical clipping line, distances from the left and right preset pixels to the vertical clipping line, energy values of upper and lower preset pixels of the horizontal clipping line, and distances from the upper and lower preset pixels to the horizontal clipping line;
an obtaining unit 654, configured to obtain an energy adjustment value corresponding to the preset pixel point according to the average energy value in the first fusion energy map, the distance between the preset pixel point and the trimming line, and the energy value of the preset pixel point;
and an adding unit 656, configured to add the energy values of the preset pixel points to the corresponding energy adjustment values.
Optionally, referring to fig. 8, the obtaining unit 654 includes:
a first obtaining unit 6542, configured to obtain an energy adjustment item of the preset pixel point according to the average energy value of the first fusion energy map and the energy value of the preset pixel point;
a second obtaining unit 6544, configured to obtain a distance adjusting item of the preset pixel point according to the distance between the preset pixel point and the trimming line;
a third obtaining unit 6546, configured to obtain an energy adjustment value corresponding to the preset pixel point according to a product of the energy adjustment term and the distance adjustment term.
Optionally, after the fusion module 63, the method includes:
the normalization unit 632 is configured to normalize the energy values of all the pixels on the first fusion energy map.
Optionally, referring to fig. 9, the calculating module 62 calculates a visually significant energy map, including:
a gaussian blurring and down-sampling unit 622, configured to perform gaussian blurring and down-sampling processing on the first image to obtain a gaussian pyramid image;
an extracting unit 624, configured to extract a grayscale feature, a color channel feature, and a direction feature of the gaussian pyramid image, so as to obtain a feature map;
an activation unit 626, configured to convert the feature graph into a full-connected graph, use a node in the full-connected graph as a state of a markov chain, use an edge weight in the full-connected graph as a transition probability of the state in the markov chain, and obtain an activation graph corresponding to the full-connected graph;
a normalization summing unit 628, configured to normalize and sum the activation maps to obtain the visually significant energy map.
Optionally, referring to fig. 10, the extracting unit 624 includes:
a first extracting unit 6242, configured to extract the grayscale feature, the color channel feature, and the direction feature of the gaussian pyramid image, so as to obtain a grayscale feature pyramid image, a color channel feature pyramid image, and a direction feature pyramid image;
a feature map obtaining unit 6244, configured to obtain, according to a central peripheral difference mechanism, a grayscale feature map corresponding to the grayscale feature pyramid image, a color channel feature map corresponding to the color channel feature pyramid image, and a direction feature map corresponding to the direction feature pyramid image under a combination of a central spatial scale and a peripheral spatial scale.
By applying the embodiment of the invention, a gradient energy map and a visual salient energy map of a first image to be zoomed are calculated by obtaining the first image to be zoomed, the gradient energy map and the visual salient energy map are fused to obtain a first fused energy map, in the first fused energy map, a dynamic programming method is used for obtaining a cutting line, the cutting line comprises a vertical cutting line and a horizontal cutting line, in the first fused energy map, energy values of left and right preset pixel points of the vertical cutting line are adjusted line by line, energy values of upper and lower preset pixel points of the horizontal cutting line are adjusted line by line, the vertical cutting line and the horizontal cutting line are deleted from the first fused energy map, an adjusted second fused energy map is obtained, in the first image, pixel points corresponding to the vertical cutting line and pixel points corresponding to the horizontal cutting line are deleted, and obtaining a zoomed second image, re-determining the cutting line for cutting according to the second fusion energy graph when the size of the second image does not meet the preset target size, and outputting the second image until the size of the second image meets the preset target size. By dynamically adjusting the fusion energy diagram, the energy change caused by each cutting can be displayed, so that a more appropriate cutting line is selected, and the image zooming precision is improved.
The present application further provides an electronic device, comprising: a processor and a memory; wherein the memory stores a computer program adapted to be loaded by the processor and to perform the method steps of the above embodiments.
The present application further provides a computer-readable storage medium, on which a computer program is stored, which computer program, when being executed by a processor, carries out the method steps of the above-mentioned embodiments.
The above-mentioned embodiments only express several embodiments of the present invention, and the description thereof is more specific and detailed, but not construed as limiting the scope of the invention. It should be noted that, to those skilled in the art, changes and modifications may be made without departing from the spirit of the present invention, and it is intended that the present invention encompass such changes and modifications.
Claims (10)
1. An image scaling method based on dynamic energy adjustment, comprising:
acquiring a first image to be zoomed;
calculating a gradient energy map and a visually significant energy map of the first image;
fusing the gradient energy map and the visual saliency energy map to obtain a first fused energy map;
in the first fusion energy diagram, cutting lines are obtained by using a dynamic programming method, wherein the cutting lines comprise vertical cutting lines and horizontal cutting lines;
in the first fusion energy map, adjusting energy values of left and right preset pixels of the vertical cutting line by line, adjusting energy values of upper and lower preset pixels of the horizontal cutting line by line, and deleting the vertical cutting line and the horizontal cutting line from the first fusion energy map to obtain an adjusted second fusion energy map;
deleting the pixel points corresponding to the vertical cutting lines and the pixel points corresponding to the horizontal cutting lines in the first image to obtain a zoomed second image;
and when the size of the second image does not meet the preset target size, re-determining the cutting line according to the second fusion energy map for cutting until the size of the second image meets the preset target size, and outputting the second image.
2. The image scaling method based on dynamic energy adjustment according to claim 1, wherein the step of adjusting the energy values of the left and right preset pixels of the vertical clipping line row by row and the energy values of the upper and lower preset pixels of the horizontal clipping line column by column in the first fusion energy map comprises:
acquiring an average energy value in the first fusion energy map, energy values of left and right preset pixels of the vertical cutting line, distances from the left and right preset pixels to the vertical cutting line, energy values of upper and lower preset pixels of the horizontal cutting line and distances from the upper and lower preset pixels to the horizontal cutting line;
obtaining an energy adjusting value corresponding to the preset pixel point according to the average energy value in the first fusion energy map, the distance between the preset pixel point and the cutting line and the energy value of the preset pixel point;
and adding the energy values of the preset pixel points to the corresponding energy adjusting values.
3. The image scaling method based on dynamic energy adjustment according to claim 2, wherein the step of obtaining the energy adjustment value corresponding to the preset pixel point according to the average energy value in the first fused energy map, the distance between the preset pixel point and the clipping line, and the energy value of the preset pixel point comprises:
obtaining an energy adjusting item of the preset pixel point according to the average energy value of the first fusion energy graph and the energy value of the preset pixel point;
obtaining a distance adjusting item of the preset pixel point according to the distance between the preset pixel point and the cutting line;
obtaining an energy adjusting value corresponding to the preset pixel point according to the product of the energy adjusting term and the distance adjusting term;
wherein, the calculation formula is:
ΔEi＝qi*d(n)
ΔEirepresenting the energy regulating value q corresponding to the preset pixel point iiRepresenting said energy adjustment term, d (n) representing said distance adjustment term, riRepresenting the energy regulation parameter, viAnd representing the energy value of the preset pixel point i, wherein a represents the average energy value of the first fusion energy graph, and w and M are constants.
4. The method for image scaling based on dynamic energy adjustment according to claim 1, wherein after fusing the gradient energy map and the visually significant energy map to obtain a first fused energy map, the method further comprises:
normalizing the energy values of all the pixel points on the first fusion energy diagram; the normalization processing mode is as follows:
and (4) the converted energy value of the pixel point is equal to (the energy value before the pixel point is converted-the minimum energy value of the pixel point on the first fusion energy map)/(the maximum energy value of the pixel point on the fusion energy map-the minimum energy value of the pixel point on the first fusion energy map).
5. The method of claim 1, wherein computing the visually significant energy map comprises:
performing Gaussian blur and downsampling processing on the first image to obtain a Gaussian pyramid image;
extracting gray features, color channel features and direction features of the Gaussian pyramid image to obtain a feature map;
converting the feature graph into a full-connection graph, taking nodes in the full-connection graph as states of a Markov chain, taking edge weights in the full-connection graph as transition probabilities of the states in the Markov chain, and obtaining an activation graph corresponding to the full-connection graph;
and normalizing and summing the activation maps to obtain the visual saliency energy map.
6. The image scaling method based on dynamic energy adjustment according to claim 5, wherein the step of extracting the gray feature, the color channel feature and the direction feature of the Gaussian pyramid image to obtain the feature map comprises:
extracting the gray characteristic, the color channel characteristic and the direction characteristic of the Gaussian pyramid image to obtain a gray characteristic pyramid image, a color channel characteristic pyramid image and a direction characteristic pyramid image;
acquiring a gray characteristic image corresponding to the gray characteristic pyramid image, a color channel characteristic image corresponding to the color channel characteristic pyramid image and a direction characteristic image corresponding to the direction characteristic pyramid image under the combination of a central space scale and a peripheral space scale according to a central peripheral difference mechanism;
wherein, the calculation formula is:
I(c,s)＝|I(c)ΘI(s)|
RG(c,s)＝|(R(c)-G(c))Θ(G(s)-R(s))|
BY(c,s)＝|(B(c)-Y(c))Θ(Y(s)-B(s))|
O(c,s,θ)＝|O(c,θ)ΘO(s,θ)|
wherein the central spatial scale c is set to {2,3,4}, the peripheral spatial scale s is set to {5,6,7,8}, Θ represents a subtraction of the feature map at the central spatial scale and the feature map at the peripheral spatial scale, I ═ R + G + B)/3, R, G and B are the red, green and blue three component values of the gaussian pyramid image, respectively, R ═ R- (G + B)/2, G ═ G- (R + B)/2, B ═ B- (R + G)/2, Y ═ R + G)/2 |/G |/2-B, θ is the direction angle, θ is set to {0,45,90,135}, the unit is degree, I (c, s) represents the gray level feature map corresponding to the gray level feature map at the combination of the central spatial scale and the peripheral spatial scale (c, s), RG (c, s) represents the red-green channel feature map corresponding to the color channel feature pyramid image under the combination (c, s) of the central and peripheral spatial scales, BY (c, s) represents the blue-yellow channel feature map corresponding to the color channel feature pyramid image under the combination (c, s) of the central and peripheral spatial scales, and O (c, s, θ) represents the directional feature map corresponding to the directional feature pyramid image under the combination (c, s) of the central and peripheral spatial scales.
7. The method according to claim 1, wherein the step of fusing the gradient energy map and the visually significant energy map to obtain a first fused energy map comprises:
and summing the gradient energy value of each pixel point in the gradient energy map and the significant energy value of each pixel point at the corresponding position of the visual significant energy map to obtain a first fusion energy map.
8. An image scaling apparatus based on dynamic energy adjustment, comprising:
the first acquisition module is used for acquiring a first image to be zoomed;
a calculation module for calculating a gradient energy map and a visually significant energy map of the first image;
the fusion module is used for fusing the gradient energy map and the visual saliency energy map to obtain a first fusion energy map;
the second acquisition module is used for acquiring cutting lines in the first fusion energy map by using a dynamic programming method, wherein the cutting lines comprise vertical cutting lines and horizontal cutting lines;
the first deleting module is used for adjusting the energy values of left and right preset pixel points of the vertical cutting line by line, adjusting the energy values of upper and lower preset pixel points of the horizontal cutting line by line in the first fused energy map, and deleting the vertical cutting line and the horizontal cutting line from the first fused energy map to obtain an adjusted second fused energy map;
the second deleting module is used for deleting the pixel points corresponding to the vertical cutting lines and the pixel points corresponding to the horizontal cutting lines in the first image to obtain a zoomed second image;
and the output module is used for re-determining the cutting line for cutting according to the second fusion energy map when the size of the second image does not meet the preset target size until the size of the second image meets the preset target size, and outputting the second image.
9. An electronic device, comprising: a processor and a memory; wherein the memory stores a computer program adapted to be loaded by the processor and to perform the image scaling method based on dynamic energy adjustment of any of claims 1 to 7.
10. A computer-readable storage medium, on which a computer program is stored, which, when being executed by a processor, implements the method for image scaling based on dynamic energy adjustment according to any one of claims 1 to 7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110868971.8A CN113808014B (en) | 2021-07-30 | 2021-07-30 | Image scaling method and device based on dynamic energy adjustment |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110868971.8A CN113808014B (en) | 2021-07-30 | 2021-07-30 | Image scaling method and device based on dynamic energy adjustment |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN113808014A true CN113808014A (en) | 2021-12-17 |
| CN113808014B CN113808014B (en) | 2024-04-05 |
Family
ID=78942533
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110868971.8A Active CN113808014B (en) | 2021-07-30 | 2021-07-30 | Image scaling method and device based on dynamic energy adjustment |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113808014B (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN117197127A (en) * | 2023-11-02 | 2023-12-08 | 广东乾威精密连接器有限公司 | Automobile terminal detection method based on machine vision |
Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105488758A (en) * | 2015-11-30 | 2016-04-13 | 河北工业大学 | Image scaling method based on content awareness |
| CN106530232A (en) * | 2016-11-10 | 2017-03-22 | 河北工业大学 | Image zooming method |
-
2021
- 2021-07-30 CN CN202110868971.8A patent/CN113808014B/en active Active
Patent Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105488758A (en) * | 2015-11-30 | 2016-04-13 | 河北工业大学 | Image scaling method based on content awareness |
| CN106530232A (en) * | 2016-11-10 | 2017-03-22 | 河北工业大学 | Image zooming method |
Non-Patent Citations (2)
| Title |
|---|
| 张娟;: "动态取样机制耦合卡通成分估算的图像修复算法", 微电子学与计算机, no. 01, pages 21 - 25 * |
| 胡琳;李岩;: "基于显著图的遥感图像多分辨区域生长分割算法", 湖南理工学院学报(自然科学版), no. 02, pages 30 - 33 * |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN117197127A (en) * | 2023-11-02 | 2023-12-08 | 广东乾威精密连接器有限公司 | Automobile terminal detection method based on machine vision |
| CN117197127B (en) * | 2023-11-02 | 2024-02-20 | 广东乾威精密连接器有限公司 | Automobile terminal detection method based on machine vision |
Also Published As
| Publication number | Publication date |
|---|---|
| CN113808014B (en) | 2024-04-05 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN109101914B (en) | Multi-scale-based pedestrian detection method and device | |
| CN110163198B (en) | Table identification reconstruction method and device and storage medium | |
| CN108304820B (en) | Face detection method and device and terminal equipment | |
| US8538074B2 (en) | Image processing apparatus and method, image capturing apparatus, and program | |
| CN112330574B (en) | Portrait restoration method and device, electronic equipment and computer storage medium | |
| EP3686794A1 (en) | Learning method and learning device for cnn using 1xk or kx1 convolution to be used for hardware optimization, and testing method and testing device using the same | |
| CN112381061B (en) | Facial expression recognition method and system | |
| CN106650615A (en) | Image processing method and terminal | |
| CN111597932A (en) | Road crack image identification method, device and system based on convolutional neural network | |
| CN112862871A (en) | Image fusion method and device | |
| CN113095470A (en) | Neural network training method, image processing method and device, and storage medium | |
| CN111612004A (en) | Image clipping method and device based on semantic content | |
| JP4639754B2 (en) | Image processing device | |
| CN111640116A (en) | Aerial photography graph building segmentation method and device based on deep convolutional residual error network | |
| CN113808014A (en) | Image scaling method and device based on dynamic energy adjustment | |
| Oliveira et al. | A novel Genetic Algorithms and SURF-Based approach for image retargeting | |
| CN114926734A (en) | Solid waste detection device and method based on feature aggregation and attention fusion | |
| CN111209908A (en) | Method and device for updating label box, storage medium and computer equipment | |
| CN114119373A (en) | Image cropping method and device and electronic equipment | |
| CN113077477B (en) | Image vectorization method and device and terminal equipment | |
| CN113744142A (en) | Image restoration method, electronic device and storage medium | |
| CN112037130B (en) | Self-adaptive image stitching fusion method and device, electronic equipment and storage medium | |
| CN113065404A (en) | Method and system for detecting train ticket content based on equal-width character segments | |
| CN111860208A (en) | Remote sensing image ground object classification method, system, device and medium based on super pixels | |
| CN115423697A (en) | Image restoration method, terminal and computer storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |