CN109493275B - Slot cutting redirection method fusing saliency map and depth map - Google Patents
Slot cutting redirection method fusing saliency map and depth map Download PDFInfo
- Publication number
- CN109493275B CN109493275B CN201811364228.3A CN201811364228A CN109493275B CN 109493275 B CN109493275 B CN 109493275B CN 201811364228 A CN201811364228 A CN 201811364228A CN 109493275 B CN109493275 B CN 109493275B
- Authority
- CN
- China
- Prior art keywords
- image
- map
- images
- seam
- cutting
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000000034 method Methods 0.000 title claims abstract description 49
- 238000012545 processing Methods 0.000 claims abstract description 6
- 230000004927 fusion Effects 0.000 claims description 7
- 239000011159 matrix material Substances 0.000 claims description 6
- 239000011541 reaction mixture Substances 0.000 claims description 4
- 230000011218 segmentation Effects 0.000 claims description 4
- 238000004364 calculation method Methods 0.000 claims description 2
- 238000004891 communication Methods 0.000 claims description 2
- 230000001186 cumulative effect Effects 0.000 claims description 2
- 230000003247 decreasing effect Effects 0.000 claims description 2
- 238000006386 neutralization reaction Methods 0.000 claims description 2
- 229920001577 copolymer Polymers 0.000 claims 2
- 230000015572 biosynthetic process Effects 0.000 claims 1
- 150000001875 compounds Chemical class 0.000 claims 1
- 238000003786 synthesis reaction Methods 0.000 claims 1
- 230000006870 function Effects 0.000 description 6
- 230000000694 effects Effects 0.000 description 4
- 230000008447 perception Effects 0.000 description 3
- 238000010586 diagram Methods 0.000 description 2
- 230000000007 visual effect Effects 0.000 description 2
- 230000009286 beneficial effect Effects 0.000 description 1
- 238000007796 conventional method Methods 0.000 description 1
- 230000007547 defect Effects 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 230000018109 developmental process Effects 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 239000000203 mixture Substances 0.000 description 1
Images
Classifications
-
- G06T3/04—
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T3/00—Geometric image transformation in the plane of the image
- G06T3/40—Scaling the whole image or part thereof
- G06T3/4084—Transform-based scaling, e.g. FFT domain scaling
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/30—Determination of transform parameters for the alignment of images, i.e. image registration
Landscapes
- Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Image Analysis (AREA)
- Image Processing (AREA)
Abstract
The invention relates to the technical field of image processing and multimedia, in particular to a method for redirecting a seam cut by fusing a saliency map and a depth map. The method comprises the following steps: acquiring an image saliency map by using a GBVS algorithm; combining the image gradient map with the proposed image depth map obtained by the SIFT matching method to construct a more accurate importance map; and acquiring cutting seams at lower energy positions according to the energy distribution of the importance map, recording the position and the motion process of each seam, and processing the original image to obtain a final redirection result. The invention considers the salient image and the depth image of the image at the same time, can reserve the salient part of the image to the maximum extent, and improves the distortion and distortion problems in the original slit cutting method.
Description
Technical Field
The invention relates to the technical field of image processing and multimedia, in particular to a method for reorienting a seam cut by fusing a saliency map and a depth map.
Background
With the rapid development of internet technology, explosively increasing network multimedia data, and the rapid increase in the number and types of display devices of different sizes in the modern society, the direct matching between various image information and screens of electronic devices is a problem that needs to be solved urgently. An important aspect in the field of image processing is the display of the same image on different devices having different sizes and resolutions, i.e. image redirection problems.
The traditional image redirection method has many methods, but the effect is poor. The uniform scaling method uses simple non-uniform scaling and bicubic interpolation to remap the original image, and is only suitable for occasions with consistent redirection proportion; the deformation method considers the important part of the image, the zooming effect is ideal, and the distortion is easy to occur; the method of the deformation cropping is to select the window with the optimal target size from the original image through the manual cropping mode, although the image is not deformed, a large amount of important information is lost during the redirection, and therefore the method cannot meet the actual requirements of people.
In order to make up for the shortcomings of the conventional methods, image redirection based on content perception attracts a lot of attention in the image and vision fields, and in general, such methods first use an algorithm to obtain an importance map of an original image, and then redirect the original image according to the importance map. The significance map is often a gradient map of the image, and the reorientation is usually performed based on a method such as suture cutting. The method of slot cutting generally uses a gradient map as an importance map, and achieves the aim of reorientation by adding or removing eight-connected slots with minimum energy, and obtains better reorientation results. However, how to make the redirected image information more accurate is a more general and pending problem.
Disclosure of Invention
Aiming at the defects, the invention provides the method for redirecting the seam cutting by fusing the saliency map and the depth map, which can reserve important information and boundary information in the image to the maximum extent and reduce distortion and deformation generated in the process of redirecting the image.
The invention is realized by adopting the following technical scheme:
a method for redirecting a seam cut by fusing a saliency map and a depth map comprises the following steps:
1) Acquiring an image saliency map by using a GBVS algorithm;
2) The method comprises the steps of obtaining an image depth map through SIFT matching, and constructing a more accurate fusion map by combining an image gradient map;
3) And 3) acquiring cutting seams with lower energy according to the energy distribution of the fusion graph obtained in the step 2), recording the position and the motion process of each seam, and processing the original image to obtain a final redirection result.
Preferably, in step (1), the image to be processed is enhanced to display important information in the image and weaken the edge region by using the GBVS algorithm.
In the step (1), the GBVS algorithm uses a Markov chain to perform significance calculation, obtains the significance state of the feature map through the stable state of the Markov chain, and superposes the obtained feature significance maps of multiple types to obtain a final significance result.
Preferably, in step (2), for the image gradient map, a gradient energy function is formed by selecting a sum of absolute values of gradients of the image in the transverse x direction and the longitudinal y direction.
In the step (2), for the image depth map, dividing the common database RGBD into a color map library A and a depth map library B corresponding to the color map library A, and extracting an input image
And HOG characteristics of all images in the gallery A, classifying all images according to the extracted characteristics by utilizing a K-nearest neighbor algorithm, and obtaining the HOG characteristics in the color gallery A
An and image
Similar images belonging to the same class
Meanwhile, the input image and the obtained color image are subjected to superpixel segmentation, and an image matching function of an SIFT method is utilized to obtain the color image
A super pixel region most similar to the input image
Wherein, in the process,
and
is a natural number greater than 1.
The image matching process of the SIFT method is as follows:
2-1）
in the formula (1), the reaction mixture is,
representing the spatial four neighborhoods of a pixel,
is a target map
Wherein an image corresponds to the original image
Middle pixel point
Deviation in (1) to obtain
Neutralization of
The most similar characteristic region is represented as
，
The feature descriptors are the minimum difference sum between the descriptors is the best matching result;
in the formula
And 
The value depends on the number of similar images in color gallery A, when there are fewer images in color gallery A that are similar to the original image, i.e. the value is determined by the number of similar images in color gallery A
Taking more feature areas when smaller, i.e.
And (4) a region.
2-2) definition of
And
in a proportional relationship of
By adjusting the ratio
Controlling similar image and characteristic selection to obtain and input image
The most similar image; if it is not
Then get
Image matching, wherein all images are matched with the input image; otherwise, if K is less than or equal to 0.8, the
Matching the similar area with the input image after the superpixel segmentation to obtainTo the most similar region coordinates;
in the process of the step 2-2), when the same position of different images is obtained, selecting matching with lower energy, and when the position which is not obtained exists, selecting the position with the lowest energy in all the images;
2-3) corresponding the color image area coordinate obtained in the step 2-2) with the corresponding depth image, and extracting sub-areas of all images in the depth map library B
The composition of each subarea is optimized to obtain the image
The depth image of (a), wherein,n 3 is a natural number greater than 1.
Preferably, the step (3) comprises the following steps:
3-1) for
Input image of (2) 
In a stitch-cutting algorithm, a vertically oriented stitch typically defines a path connecting the top to the bottom of the image, i.e. a stitch
In the formula (2), the reaction mixture is,
is composed of
To
Is mapped to a single one of the images,
for cutting seams
Coordinates, subscripts, of a certain pixel
Represents
Axial direction of when
In the process, an eight-communicated seam can be obtained;
which represents the width of the image or images,
representing the height of the image. Similarly, a horizontal seam is defined as
In the same way as above, the first and second,
is composed of
To
Is mapped to a single one of the images,
for cutting seams
The coordinates of a certain pixel in (a);
3-2) calculating the gradient of the fusion map in the transverse x direction and the longitudinal y direction to obtain an energy function
The sum of the energies on one of the vertical cutting seams is
Thus, the optimal trim line is the one with the least amount of energy among all the eight-communication trim lines, i.e., the trim line
3-3) setting up a matrixMStoring each point on the vertical seam cut line
Of the cumulative minimum energy value, matrixNStoring the energy value in the current energy function, and using a dynamic programming method to determine the optimal cutting line, i.e. traversing from the 2 nd line to the last line of the image to obtain the minimum value in the last line
MThe minimum value in the last row in the matrix is the energy minimum value in the optimal suture line; continuously selecting the minimum value in the neighborhood of the pixel point 8 with the minimum energy value to carry out reverse pushing to obtain a final redirection result; similarly, traversing the minimum value obtained from the 2 nd column to the last column of the image to obtain a horizontal cutting line;
3-4) according to the requirement of the reorientation size, repeating the steps 3-1) to 3-3) by continuously increasing or decreasing the obtained horizontal and vertical cutting lines (namely cutting seams) to achieve the reorientation purpose.
The invention has the beneficial effects that: the method applies the saliency map and the depth map of the image to the energy function of the seam cutting algorithm, improves the energy function in the reorientation algorithm, reorients the image, can better reserve the important part of the image, improves the reorientation quality of the image, and improves the distortion and distortion problems in the current content perception reorientation method.
Drawings
The invention will be further described with reference to the accompanying drawings:
FIG. 1 is a schematic flow diagram of the process of the present invention;
FIG. 2 is a visual comparison of the present invention as applied to the image redirection problem, with other methods.
Detailed Description
Referring to the attached figure 1, the method acquires an image saliency map according to an original image by using an SIFT matching method, and acquires an image depth map by using the SIFT matching method; forming a gradient energy function by selecting the sum of absolute values of gradients of the image in the transverse x direction and the longitudinal y direction to obtain a gradient map; combining the image depth map and the image gradient map to construct a more accurate fusion map; and finally obtaining a final zoom map through an SC algorithm.
Fig. 2 shows that, in order to compare the visual effects of the algorithm of the present invention with those of other algorithms, the deformed and missing regions of each image are circled in red, as can be seen from fig. 2 (b), the WARP algorithm causes local deformation of the image, in fig. 2 (b), the legs of the girls at the far left and the far right are deformed, and the CR algorithm is to crop the image, as can be seen from fig. 2 (c), in the algorithm, nearly half of the region is cropped and removed, and most of the edge information of the original image is lost; the SC algorithm is not continuous when selecting a cutting seam, and as can be seen from fig. 2 (d), the deformed region is more obvious; the SM algorithm improves SC by using graph cut, and as can be seen from fig. 2 (e), after the algorithm is redirected, the information of the image partial area is lost; as can be seen from fig. 2 (f) (g), the SNS and SV algorithms are not significantly deformed, but the image subject region is somewhat reduced; the AA algorithm has no deformation and distortion in the main body area, but the non-important area has larger change; finally, fig. 2 (i) is an effect diagram of the algorithm of the present invention, when the image is reoriented, no obvious deformation and distortion occur, and no important area and non-important area in the image are lost, so that a better reorientation result is obtained.
In conclusion, compared with other existing algorithms, the algorithm has a remarkable effect in image redirection, greatly improves the problems of distortion and distortion in the current content perception redirection method, and is worthy of popularization and application.
Claims (7)
1. A method for reorienting a seam cutting by fusing a saliency map and a depth map is characterized by comprising the following steps:
1) Acquiring an image saliency map by using a GBVS algorithm;
2) Acquiring an image depth map through SIFT matching, and constructing a fusion map by combining the image saliency map, the image depth map and the image gradient map;
3) Acquiring cutting seams at lower energy positions according to the energy distribution of the fusion graph obtained in the step 2), recording the position and the motion process of each seam, and processing the original image to obtain a final redirection result;
the step (3) comprises the following steps:
3-1) tom×nInput image of (2)
In a seam cutting algorithm, the vertically oriented seam defines a path connecting the top to the bottom of the image, i.e.
In the formula (2), the reaction mixture is,
is a copolymer of (1),m]to the extent that the ratio of [1 ],n]is mapped to a single one of the images,
for cutting seams
Coordinates, subscripts, of a certain pixelxRepresentxAxial direction, when δ =1, one eight-connected slot can be obtained; mWhich represents the width of the image or images,nrepresenting the height of the image; similarly, a horizontal seam is defined as
In the same way as above, the first and second,
is a copolymer of (1),n]the molecular weight distribution of the compounds of formula (1),m]is mapped to a single one of the images,
for cutting seams
The coordinates of a certain pixel in (a);
3-2) calculating the gradient of the fusion map in the transverse x direction and the longitudinal y direction to obtain an energy function
The sum of the energies on one of the vertical cutting seams is
Thus, the optimal trim line is the one with the least amount of energy among all the eight-communication trim lines, i.e., the trim line
3-3) arranging a matrixMStoring each point on the vertical seam cut line
Of the cumulative minimum energy value, matrixNStoring the energy value in the current energy function, and using a dynamic programming method to determine the optimal cutting line, i.e. traversing from the 2 nd line to the last line of the image to obtain the minimum value in the last line
MThe minimum value in the last row in the matrix is the energy minimum value in the optimal suture line; continuously selecting the minimum value in the neighborhood of the pixel point 8 with the minimum energy value to carry out reverse pushing to obtain a final redirection result; similarly, traversing the minimum value obtained from the 2 nd column to the last column of the image to obtain a horizontal cutting line;
3-4) according to the requirement of the reorientation size, continuously increasing or decreasing the obtained horizontal and vertical cutting lines, namely cutting seams, and repeating the steps 3-1) -3) to achieve the reorientation purpose.
2. The method of seam-cutting redirection fusing a saliency map and a depth map of claim 1, characterized in that: in step (1), the image to be processed is enhanced to display important information in the image and weaken the edge area by using the GBVS algorithm.
3. The method of seam-cut redirection fusing a saliency map and a depth map of claim 2, wherein: in the step (1), the GBVS algorithm uses a Markov chain to perform significance calculation, obtains the significance state of the feature map through the stable state of the Markov chain, and superposes the obtained feature significance maps of multiple types to obtain a final significance result.
4. The method of seam-cut reorientation fusing a saliency map and a depth map according to claim 1, characterized in that: in the step (2), for the image gradient map, a gradient energy function is formed by selecting the sum of absolute values of gradients of the image in the transverse x direction and the longitudinal y direction.
5. The method of seam-cut reorientation fusing a saliency map and a depth map according to claim 1, characterized in that: in the step (2), for the image depth map, dividing the common database RGBD into a color map library A and a depth map library B corresponding to the color map library A, and extracting an input image 
And HOG characteristics of all images in the gallery A, classifying all images according to the extracted characteristics by utilizing a K-nearest neighbor algorithm, and obtaining the HOG characteristics in the color gallery A
Personal information and image
Similar images belonging to the same class
Meanwhile, the input image and the obtained color image are subjected to superpixel segmentation, and an image matching function of an SIFT method is utilized to obtain the color image
A super pixel region most similar to the input image
Wherein, in the step (A),
and
is a natural number greater than 1.
6. The method for seam-cutting reorientation fusing a significant map and a depth map according to claim 5, characterized in that the image matching process of SIFT method is as follows:
2-1）
in the formula (1), the reaction mixture is,
representing four neighborhoods of the space of one pixel,
is a target map
Wherein an image corresponds to the original image
Middle pixel point
Deviation in (1) to obtain
Neutralization of
The most similar characteristic region is represented as
，
The minimum difference sum between the descriptors is a best matching result;
in the formula
And
the value depends on the number of similar images in color gallery A, when there are fewer images in color gallery A that are similar to the original image, i.e. the value is determined by the number of similar images in color gallery A
Taking more feature areas when smaller, i.e.
An area;
2-2) definition of
And
in a proportional relationship of 
By adjusting the ratio
Controlling similar images and feature selection to obtain and input images
The most similar image; if it is not
Then get
Image matching, wherein all images are matched with the input image; otherwise, if K is less than or equal to 0.8, the
Matching the similar area with the input image after the super-pixel segmentation to obtain the most similar area coordinate;
2-3) corresponding the color image area coordinate obtained in the step 2-2) with the corresponding depth image, and extracting sub-areas of all images in the depth map library B
The synthesis of each sub-region is optimized to obtain the image
The depth image of (a), wherein,n 3 is a natural number greater than 1.
7. The method of seam-cutting reorientation fusing a saliency map and a depth map according to claim 6, characterized in that during step 2-2), when the same position of different images is taken, a lower energy match is selected, and when there are not taken positions, the position with the lowest energy is selected in all images.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201811364228.3A CN109493275B (en) | 2018-11-16 | 2018-11-16 | Slot cutting redirection method fusing saliency map and depth map |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201811364228.3A CN109493275B (en) | 2018-11-16 | 2018-11-16 | Slot cutting redirection method fusing saliency map and depth map |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN109493275A CN109493275A (en) | 2019-03-19 |
| CN109493275B true CN109493275B (en) | 2023-01-24 |
Family
ID=65695976
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201811364228.3A Active CN109493275B (en) | 2018-11-16 | 2018-11-16 | Slot cutting redirection method fusing saliency map and depth map |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN109493275B (en) |
Families Citing this family (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111028152B (en) * | 2019-12-02 | 2023-05-05 | 哈尔滨工程大学 | Super-resolution reconstruction method of sonar image based on terrain matching |
| CN112184558B (en) * | 2020-11-09 | 2024-03-08 | 辽宁工程技术大学 | RGB-D image irregular scaling method based on saliency detection |
Family Cites Families (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104992403B (en) * | 2015-07-07 | 2017-05-03 | 方玉明 | Hybrid operator image redirection method based on visual similarity measurement |
| CN107330885B (en) * | 2017-07-07 | 2020-10-02 | 广西大学 | Multi-operator image redirection method for keeping aspect ratio of important content area |
-
2018
- 2018-11-16 CN CN201811364228.3A patent/CN109493275B/en active Active
Also Published As
| Publication number | Publication date |
|---|---|
| CN109493275A (en) | 2019-03-19 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| Ji et al. | Deep view morphing | |
| US8213711B2 (en) | Method and graphical user interface for modifying depth maps | |
| US8249394B2 (en) | Method and system for shift-map image editing | |
| US20150131924A1 (en) | Creation of Rectangular Images from Input Images | |
| US20050206643A1 (en) | Image display method and iamge display device | |
| CN102365651A (en) | Method and apparatus for modifying an image by using a saliency map based on color frequency | |
| US20150077639A1 (en) | Color video processing system and method, and corresponding computer program | |
| CN109493275B (en) | Slot cutting redirection method fusing saliency map and depth map | |
| Kiess et al. | A survey on content-aware image and video retargeting | |
| US8577166B1 (en) | Optimizing web site images using a focal point | |
| Kopf et al. | Algorithms for video retargeting | |
| US20160364626A1 (en) | Image processing apparatus, image processing method, image processing system, and non-transitory computer readable medium | |
| US10089954B2 (en) | Method for combined transformation of the scale and aspect ratio of a picture | |
| Kim et al. | A structure-aware axis-aligned grid deformation approach for robust image retargeting | |
| CN109600667B (en) | Video redirection method based on grid and frame grouping | |
| CN112365518A (en) | Image splicing method based on optimal suture line self-selection area gradual-in and gradual-out algorithm | |
| Shi et al. | Misalignment-eliminated warping image stitching method with grid-based motion statistics matching | |
| Xiong et al. | Mask-based image blending and its applications on mobile devices | |
| JP5614835B2 (en) | Image layout setting method and apparatus | |
| JP2018010359A (en) | Information processor, information processing method, and program | |
| CN112203023A (en) | Billion-pixel video generation method, device, equipment and medium | |
| CN112184558B (en) | RGB-D image irregular scaling method based on saliency detection | |
| CN112669355B (en) | Method and system for splicing and fusing focusing stack data based on RGB-D super pixel segmentation | |
| Ran et al. | Fast seam carving using Gaussian pyramid | |
| JP5192940B2 (en) | Image conversion apparatus, image conversion program, and image conversion method |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |