CN112184558B

CN112184558B - RGB-D image irregular scaling method based on saliency detection

Info

Publication number: CN112184558B
Application number: CN202011241661.5A
Authority: CN
Inventors: 陈曦涛; 訾玲玲
Original assignee: Liaoning Technical University
Current assignee: Liaoning Technical University
Priority date: 2020-11-09
Filing date: 2020-11-09
Publication date: 2024-03-08
Anticipated expiration: 2040-11-09
Also published as: CN112184558A

Abstract

The invention discloses an RGB-D image irregular scaling method based on saliency detection, which comprises the following steps: generating a saliency map using a saliency method; weighting and fusing the saliency map, the image to be zoomed and the gradient map of the depth image to generate an importance map for seam cutting; scaling the image to be scaled into an image using a sea-clamping algorithm; establishing an elliptic standard equation by taking the center of an image as an origin, wherein the horizontal direction of a rectangular coordinate system is taken as an x axis, and the vertical direction is taken as a y axis; dividing the image into 4 parts according to the first quadrant, the second quadrant, the third quadrant and the fourth quadrant; and sequentially completing the scaling of the second, third and fourth quadrants. The invention introduces depth image information, obtains better image scaling effect, ensures significant interest and content of the image, retains important content of background information, completes irregular shape scaling of the image, obtains considerable effect on irregular scaling of the image, can perform irregular scaling of any elliptic shape, has more real and beautiful scaling effect and has certain theoretical and application values.

Description

RGB-D image irregular scaling method based on saliency detection

Technical Field

The invention belongs to the technical field of image processing, and particularly relates to an RGB-D image irregular scaling method based on saliency detection.

Background

The traditional image scaling technology, such as nearest neighbor interpolation, bilinear interpolation and bicubic interpolation method, can well finish the image magnification and reduction on the image, but extrusion or stretching of a significant target can occur when the aspect ratio of the image is changed, especially the important information of the image can generate distortion, and the visual effect is seriously affected. The clipping method corresponding to the clipping method directly intercepts the most interested area in the image, solves the problem of image distortion, but inevitably omits background information, and leads to the loss of the whole content of the image. The root cause of the defects of the traditional image scaling technology is that only the output size after image scaling is considered in the process of processing the image scaling, only the geometric limitation is met, no much thought is made on the image content information, and the image scaling is mechanically carried out according to the target size, so that the obvious target distortion occurs, and the background content is necessarily lost.

In order to solve the shortcomings of the conventional image scaling method, in 2007, an image scaling method (sea-caving algorithm) based on content perception was proposed by avaian and Shamir, which uses a gradient map as an energy map of an image, thereby discriminating importance of the image, a region with high energy is regarded as a significant region, and a region with low energy is regarded as a background region. The central idea is to dynamically program an 8-way joint with the minimum energy in an energy diagram, and complete image scaling by adding or removing the joint, so that the region with high importance can keep the original characteristics, and the region with low importance absorbs distortion and deformation in the scaling process. Compared with the traditional image scaling technology, the Seam-Cavring algorithm senses the importance of the image content, can maintain the image important content without distortion even when the aspect ratios of the image scaling are inconsistent, and can continuously and rapidly finish the image scaling of different sizes. However, for some images with complex backgrounds, the team-Cavring can cause image background deformity, and the background information can be lost, so that the method has certain limitation. Therefore, the image scaling mainly solves the problem of keeping the complete background content under the condition of ensuring that important content of the image is not distorted, so that the scaled image is not distorted.

With the development of society, the requirements of the image are not only satisfied with the rectangular shape of the image, but users more hope to obtain the image with a non-rectangular shape to express a creative in specific situations. Shaoyu Qi et al propose CASAIR (Content and Shape-Aware Image Retargeting) algorithm that can redirect images into non-rectangular shapes from the perspective of the content of the image and the shape of the image. Inspired by the team-caving algorithm, the casai algorithm adopts the idea of carving based on joint segments, and continuously deletes the joint segments with low energy values from the image so as to simultaneously realize two targets of content perception identification and image shape change. Since the image boundary requirements are non-rectangular, the number of pixels to be deleted from each row (or each column) of the image is no longer constant, but depends on the target shape, which naturally results in the casai algorithm deleting a seam segment instead of one seam during execution, where a seam segment is a seam that can start and end in any two rows (or columns) of the image, several such seams making up the seam segment described above. The CASAIR algorithm has been practically applied, and a smart camera projector system has been developed using this method. The CASAIR algorithm can perform irregular scaling of an image on the basis of maintaining the content of the image, but has problems in that the CASAIR algorithm has a strict limitation on the size of the scaled image, the size of the irregular scaled image can be fixed, flexibility is lacking, and important contents can be easily removed in the process of removing a Seam section, for example, a pixel point with a higher energy value may be included in a low-energy Seam section, and the removal of a similar Seam section may result in the content deletion of the image.

Seam image is a content-aware based image scaling algorithm that achieves image scaling by removing or inserting seams that minimize accumulated energy. The idea is to dynamically plan a pixel line with minimum energy in the vertical or horizontal direction at a time, called a seam, and to implement image reduction by deleting the seam, and to implement image magnification by inserting the seam. Compared with the traditional image scaling technology, the method has the characteristic that the important content of the image can be well maintained unchanged for the image with inconsistent length-width ratio scaling in a certain scaling range. This solution has the following drawbacks:

1. the problems that the background of the image is malformed and the background content is lost are caused by facing some images with complex backgrounds.

The team-caving algorithm removes the entire Seam from the low end to the top end or from the leftmost to the rightmost of the image, and the resulting image is still rectangular regardless of the aspect ratio of the image, so that irregular shape scaling cannot be accomplished using only the team-caving algorithm.

3. The processing process of the energy map is single and simple, and the significant objects in the image are not well protected.

In addition, CASAIR (Content and Shape-Aware Image Retargeting) is an irregular image scaling method for content and shape perception, which is based on the idea of the team-Cavring algorithm, and proposes a joint segment concept, and continuously deletes low-cost joint segments to achieve both content and shape scaling. Wherein the selection of the seam segments is determined by combining features from the image content and the target shape. For a complete representation of the type of seam section, the concept of bhv-convex shape was introduced and bhv-convex shape (bhv stands for boundary, horizontal and vertical) was demonstrated, and the seam section could be removed only if the condition was met, and the CASAIR algorithm could reposition the image to the target shape. This solution has the following drawbacks:

the CASAIR algorithm has strict limitation on the size of the scaled image, and can fix the size of the irregular scaled image, and the CASAIR algorithm lacks flexibility.

2. During the process of removing the seam section, important contents can be easily removed, for example, the low-energy seam section may contain pixel points with higher energy values.

3. The significant object region in the image is not well protected, and the significant object region is easy to deform due to the removal of the joint section in irregular scaling.

Disclosure of Invention

Based on the defects of the prior art, the technical problem solved by the invention is to provide an RGB-D image irregular scaling method based on significance detection, solve the problems of image background deformity and background content loss in a Seam clamping algorithm, and solve the problems that the CASAIR algorithm is fixed to irregular shape and size and high-energy pixels are possibly removed.

In order to solve the technical problems, the invention provides an RGB-D image irregular scaling method based on significance detection, which comprises the following steps:

step 1: generating saliency map I using a saliency method _sal ；

Step 2: weighted fusion significance map I _sal Image I to be zoomed _in Depth image I _d Generating an importance map for seam cropping;

step 3: image I to be scaled using the Seam-clamping algorithm _in Scaling to image I _ab ；

Step 4: in image I _ab Establishing a rectangular coordinate system with the horizontal direction as the x axis and the vertical direction as the y axis by taking the center as an origin, and constructing an elliptic standard equation;

step 5: image I _ab Dividing into 4 parts according to the first quadrant, the second quadrant, the third quadrant and the fourth quadrant;

step 6: and sequentially completing the scaling of the second, third and fourth quadrants.

Optionally, in step 4, an elliptic standard equation is constructed as follows:

wherein a is an elliptic long half shaft and b is a short half shaft.

Optionally, in step 6, the ellipse is divided into a plurality of small rectangles, each rectangle is called a element, each element is a part of the ellipse, the original image of the first quadrant is cut out to a part corresponding to the element, the element is subjected to a sea-clamping algorithm, the content sensing operation is performed, and the number of the found seams is m, which is represented by the following formula:

wherein a, b are a long half shaft and a short half shaft which are input by a user respectively, and x is the abscissa of the ellipse in the quadrant;

obtaining the next element with the step length as delta x, obtaining the joint from the next element, wherein the joint set of the elements is { s } _i All pixels included in one clipping line s are:

and splicing all the elements to obtain the irregular shape scaling of the first quadrant.

From the above, the RGB-D image irregular scaling method based on saliency detection has the following beneficial effects:

the invention introduces depth image information, obtains better image scaling effect, ensures significant interest and content of the image and retains important content of background information; the invention completes the irregular shape scaling of the image, obtains considerable effect on the irregular scaling of the image, can perform the irregular scaling of any elliptic shape, has more real and beautiful scaling effect and has certain theoretical and application values.

The foregoing description is only an overview of the present invention, and is intended to be implemented in accordance with the teachings of the present invention, as well as to provide further clarity and understanding of the above and other objects, features and advantages of the present invention, as described in the following detailed description of the preferred embodiments, taken in conjunction with the accompanying drawings.

Drawings

In order to more clearly illustrate the technical solution of the embodiments of the present invention, the drawings of the embodiments will be briefly described below.

Fig. 1 is a flowchart of the RGB-D image irregular scaling method based on saliency detection of the present invention.

Detailed Description

The following detailed description of the invention, taken in conjunction with the accompanying drawings, illustrates the principles of the invention by way of example and by way of a further explanation of the principles of the invention, and its features and advantages will be apparent from the detailed description.

The RGB-D image irregular scaling method based on saliency detection firstly applies a saliency detection algorithm to detect a saliency target area of an image, and is used as a basis for protecting the saliency area of subsequent image scaling, and secondly, a gradient map of an RGB image and a depth image is obtained, so that the image can be redirected into a non-rectangular shape.

If given depth image I _d The gradient energy function is defined as equation (1):

compared with the traditional visible RGB image, the depth image has better edge information, and the foreground and the background of the image are easy to distinguish. However, because the depth image has good edge information, the depth value in the depth image is considered to be very sensitive to the visual angle, namely the depth value is changed strongly in the edge area or kept unchanged in the background area, and the characteristic value proposed by the traditional gradient characteristic-based extraction algorithm is also changed strongly in the edge area of the image, so that the depth image I can be obtained _d As an aid to the gradient map of image I. In order to reduce the complexity of feature fusion, a gradient map of an RGB image and a depth image and a saliency map I are adopted _sal Fusion is carried out in a serial manner, and the energy function of the fused importance map is defined as formula (2):

E(I)＝αe(I)+βI _sal +(1-α-β)e(I _d ) (2)

wherein alpha and beta are weight parameters of the image, the value range is [0,1], and alpha+beta is less than or equal to 1.

In the invention, the elliptic long half axis a and the shorter half axis b are required to be input by a user at will, and the invention performs irregular shape scaling according to any elliptic shape input by the user. Establishing a coordinate system in the input RGB image according to the nature of the ellipse and establishing an ellipse standard equation, such as formula (3):

finding the partial integral formula of the elliptical area in the first quadrant in the coordinate system is as follows (4):

inspired from the area formula of the ellipse: we can use the "infinitesimal method" to divide the ellipse into several small rectangles, each rectangle is called a "element", each element is a part of the ellipse, we can cut the original image of the first quadrant into the part corresponding to the "element", and perform the sea-clamping algorithm on the "element" to perform the content sensing operation, thus, the number of the seams to be found is m, as in formula (5):

wherein a and b are respectively a long half shaft and a short half shaft which are input by a user. x is the abscissa of the ellipse in the quadrant.

Obtaining the next element with the step length delta x, obtaining the joint from the next element, wherein the joint set of the element is { s } _i All pixels included in one clipping line s are formula (6):

finally, all the 'elements' are spliced, so that irregular scaling of the first quadrant can be obtained, and the operations are repeated on the image areas of the other three quadrants, and finally, irregular scaling of the elliptical shape is completed.

The RGB-D image irregular scaling method based on saliency detection mainly comprises the following steps:

the RGB-D image irregular scaling method based on saliency detection solves the problems of image background deformity, background content loss and the like in a sea-care algorithm, and flexibly uses the sea-care algorithm to solve the problem that the sea-care algorithm cannot perform irregular scaling. The method solves the problem that the existing algorithm cannot protect the saliency target. The problem of the CASAIR algorithm is solved that the irregular shape is fixed in size and high energy pixels may be removed.

While the invention has been described with respect to the preferred embodiments, it will be understood that the invention is not limited thereto, but is capable of modification and variation without departing from the spirit of the invention, as will be apparent to those skilled in the art.

Claims

1. An RGB-D image irregular scaling method based on saliency detection is characterized by comprising the following steps:

step 1: generating saliency map I using a saliency method _sal ；

step 6: sequentially completing the scaling of the second, third and fourth quadrants;

in step 6, the ellipse is divided into a plurality of small rectangles, each rectangle is called a unit, each unit is a part of the ellipse, the original image of the first quadrant is cut out to a part corresponding to the unit, the unit is subjected to a team-surviving algorithm, the content sensing operation is carried out, the number of the found joints is m, and the following formula is adopted:

2. The RGB-D image irregular scaling method based on saliency detection of claim 1, wherein in step 4, an elliptic standard equation is constructed as follows:

wherein a is an elliptic long half shaft and b is a short half shaft.