TWI410896B - Compressibility-aware media retargeting with structure preserving - Google Patents

Abstract

The present invention provides a method for retargeting an image, comprising determining a total block structure energy of an input image content. A compressibility rate of the input image content is determined based on the total block structure energy. An optimal scaling factor of the input image content is obtained. The input image content is warped by using a new coordinate matrices and uniformly scaling the input image content to a target image resolution.

Description

Compressed sensing multimedia picture adjustment method with structure preservation

The invention relates to an image processing method for adjusting image/video data, in particular to a method for adjusting a compressed sensing multimedia picture with structure preservation.

Image resizing is a standard tool in many image processing applications. In practice, the image is evenly adjusted to a target size. At present, the growing image size adjustment method is interested in finding changes in image size while maintaining the integrity of important features, which can be detected by top-down or bottom-up methods. The top-down approach utilizes a tool such as a face detector to detect key areas in the image, while the bottom-up approach relies on a visual saliency approach to create a visual saliency map of the image. Once the visual saliency map is constructed, the screen crop can be utilized to display the most important areas of the image.

In recent years, content sensing video/video picture adjustment methods have become more and more important due to the increasing demand for display images on various display devices of different resolutions or aspect ratios. Some algorithms have proposed adapting image or video content to different display settings. Most of the existing methods fall into three categories: one applies a picture cropping or cutting strategy to remove fewer important areas (Reference 1: S. Avidan and A. Shamir, “Seam carving for content-aware Image resizing,"ACM Transactions on Graphics(SIGGRAPH), vol. 26, no. 3,2007; Reference 2:M. Rubinstein, A. Shamir, and S. Avidan, "Improved seam carving for video retargeting," ACM On Graphics (SIGGRAPH), pp. 1-9, 2008); the second is to split the image into foreground and back layers and scale each layer independently (Ref. 3: V. Setlur, S. Takagi, R. Raskar, M. Gleicher, and B. Gooch, "Automatic image retargeting," ACM SIGGRAPH Sketches, vol. 154, p. 4, 2004; Reference 4: C. Tao, J. Jia, and H. Sun, "Active window oriented Dynamic video retargeting, "Workshop on Dynamical Vision ICCV, 2007"; and its three methods of adapting the image according to the regional image saliency (Ref. 5: L. Wolf, M. Guttmann, and D. Cohen -Or, "Non-homogeneous content-driven video-retargeting," Internation Al Conference on Computer Vision, pp. 1-6, 2007; Reference 6: Y.-S. Wang, C.-L. Tai, O. Sorkine, and T.-Y. Lee, “Optimized scale-and- Stretch for image resizing, "ACM Transactions on Graphics (SIGGRAPH Asia), pp. 1-8, 2008).

For the method based on picture cropping using some algorithms, it uses the method of picture cropping to remove less important areas from the image, which may discard a large amount of image information, and if important features are Parts that are located farther in the image/video sometimes fail. In order to deal with the problems caused by the cropping method, Avidan and Shamir (Ref. 1) propose an interesting idea of incrementally removing or inserting regions, called seam carving. However, the simple extended image slitting method is applied to the adjustment of the video screen to cause swaying artifacts. Therefore, an improved image slitting algorithm is proposed to reduce artifacts by calculating the transmitted energy to find the lowest cost (Ref. 2). Although in some cases the problem of artifacts or distortion of the content structure cannot be avoided, the image slitting method is still an effective technique for image/video image adjustment.

Without the method of cropping or cutting the image content, the image segmentation method can provide another way to separate different important image regions. Currently, Wolf et al. (Ref. 5) propose a method based on picture distortion, which combines saliency measurement, face detector and motion estimation to automatically detect important areas for video image adjustment. . They developed a grid map of image adjustments to solve a large sparse linear system. Based on a similar concept, Wolf et al. (Ref. 6) propose a method that allows important regions to be uniformly scaled and homogenous regions to be distorted. This method gives more freedom to utilize the homogenous image area. From the results of references 5 and 6, it is limited to the overall shape of the highlighted object that is preferably stored in the adjusted image/video. Different methods can also be used to adjust images with different content. On the other hand, it may be necessary to combine many methods to properly adjust an image size.

In the human visual system, the human eye is sensitive to the shape of certain objects, such as circles or straight lines. Based on image saliency measurement, most of the previous work limited the image adjustment method to maintain the continuity of the structure on a bottom-up strategy, which may not completely preserve the overall image structure. For example, in the image slitting method (Refs. 1 and 2), it removes/inserts an eight-joint seam to adjust the image size.

Most of the prior art techniques adjust the image size based on the energy assessed and minimize the deformation of adjacent pixels (reference documents 1, 2, 5 or 6). According to these bottom-up methods, the deformation will accumulate to produce unintended image structure deformation or significant artifacts. In the method based on picture deformation, most of the deformation is caused by a non-uniform picture deformation on an easily injured object. Furthermore, based on the basic concept of the image adjustment algorithm proposed by Wolf et al. (Ref. 5), the formula for adjusting the image size is selected to solve a constrained linear system. Most of the previous work focused on spatial image/video resizing. Some other work is applied to video size adjustments along the time domain, and less important frame refurbishment results in shorter image sequences. Recently, some researchers have dealt with image resizing problems as jigsaw puzzles, that is, reorganizing patches or pixels. These methods can be used to edit images through human interaction. In the foregoing methods, each method has a common problem in most of the aforementioned work, that is, the problem of structural distortion.

In view of the foregoing shortcomings, the present invention provides a method of improving the size adjustment of multimedia images to overcome most of the problems encountered in the foregoing work.

In order to overcome the conventional technical problems, the present invention provides a technique for energy evaluation and multimedia compression rate evaluation from top to bottom block structure to achieve the purpose of image content structure preservation.

It is an object of the present invention to provide a method of preserving an overall image structure.

In contrast to previous work, the target structure can be protected and the overall approach is utilized to reduce distortion and balance content perception and multimedia compression.

A further object of the present invention is to provide a compressed sensing multimedia picture adjustment method with structure preservation, so as to achieve better image content and target structure preservation without excessive compression and elongation.

In order to achieve the above object, the present invention provides a method for image adjustment, comprising: determining a whole block structure energy of an input image content by a central processing unit or a processing unit; and performing central processing based on the overall block structure energy Or processing unit to determine the compression ratio of the input image content; to determine an optimal scaling factor of the input image content by the central processing unit or processing unit; and to deform by using a new coordinate matrix by the central processing unit or processing unit Input image content and evenly scale the input image content to output a target image resolution.

The image adjustment method of the present invention further includes solving the least squares solution of a sparse linear system by a central processing unit or a processing unit before deforming the input image content.

According to another aspect of the present invention, a method for video adjustment includes: determining, by a central processing unit or a processing unit, an overall block structure energy and a compression ratio of one input video content of each frame; The processor or the processing unit determines an optimal scaling factor of the input video content; and the central processing unit or the processing unit deforms the input video content by using a new coordinate matrix and uniformly scales the input video content to output a target video resolution degree.

In addition, the method for video adjustment of the present invention further includes solving a least square solution of a sparse linear system including a constraint condition and smoothing the optimal scaling factor before deforming the input video content.

The invention will be described in detail below in conjunction with its preferred embodiments and the accompanying drawings. It should be understood that all of the preferred embodiments of the invention are intended to be illustrative only and not limiting. Therefore, the invention may be applied to other embodiments in addition to the preferred embodiments. The invention is not limited to any embodiment, but should be determined by the scope of the appended claims and their equivalents.

In order to overcome the shortcomings of the prior art, the present invention proposes an algorithm for automatically adjusting the image size, which can considerably preserve the significant structure in the image. Instead of reducing the distortion of adjacent pixels or grids, the present invention defines a block structure energy that evenly distributes the local structure energy on the pixels within the bounding box of the detected structure segment. Based on this energy, the proposed algorithm forces the deformation of each block area to be as uniform as possible. Based on the image content, the present invention further proposes to evaluate the compression ratio in each direction. The compression ratio and total entropy help determine the optimal scaling factor for adjusting the image to the best resolution for the same aspect ratio as the target image size. Compared with the experimental results of the image adjustment method described above, the results show that the image size adjustment using the algorithm of the present invention has a superior structure preservation effect.

The present invention proposes a top-down strategy to preserve the image structure by utilizing a block saliency map to accommodate the size of the architectural target. From the image gradient size and direction distribution, the compression ratio of each image can be evaluated. The image adjustment method can be optimized based on the compression ratio evaluation of the X and Y directions. Therefore, the adjusted image frame meets the basic requirements: maintaining significant content and preserving the overall structure. In the present invention, different strategies can be used to automatically adjust the image size so that significant image content and structure can be optimally preserved.

In the image/video size adjustment method, the image content can be adapted to different types of display devices, and the aspect ratio can be changed while the important portion of the content is maintained as much as possible. In the previous study of the present invention, focusing on the preservation of the image centerline structure (Ref. 7: S.-F. Wang and S.-H. Lai, "Fast structure-preserving image retargeting" Proceedings of the 2009 IEEE International Conference) On Acoustics, Speech and Signal Processing, pp. 1049-1052, 2009), which is insufficient for different kinds of object structures. In the present invention, image size adjustment will be handled from a new perspective, that is, the image aspect ratio is adjusted. Instead of considering the true target size, first determine the optimal width and length to best fit the original image to the target aspect ratio and then scale it evenly to the target image size. The algorithm proposed by the present invention can be summarized in the first figure.

Please refer to the first figure, which shows a flow chart of image size adjustment according to the present invention. The image size adjustment described above can be performed in an image/video processing device or a display device 10, as shown in the second figure. The second image shows a block diagram of the image/video processing device 10 . The image/video processing device 10 includes an image/video adjustment module 11 , a central processing unit or processing unit 12 , a display 13 , an operating system 15 , An input unit 16, an output unit 17, a memory unit 19, a frame memory 20, a video memory 21, and the like. The above-described block diagrams are for illustrative purposes only and are not intended to limit the scope of the invention. For example, different types of video/video processing devices or display devices include digital televisions, computers, notebook computers, mobile phones, Apple mobile phones (iPhones), Apple music players (iPods), e-books, and digital photo frames. Wait. The input image 5 is input to the image/video adjustment module 11, and the image/video adjustment module 11 serves as an image processing unit for performing pre-processing of the input image signal output from a display device. Since the video/video size adjustment module is not a main feature of the present invention, a detailed description thereof will be omitted. The memory unit 19 includes a read only memory, a random access memory and a non-volatile flash memory. The frame memory 20 stores the frame image processed by the image/video adjustment module 11. The video memory 21 stores the video image processed by the video/video adjustment module 11. The input unit 16 is for reading data and programs from an external device, and the output unit 17 is for outputting data from the processing unit 12.

In a first step 100, the overall block structure energy of the content structure in an input image is determined/calculated by a central processing unit or a processing unit, such as processing unit 12 in the second figure. The input image 5 can be processed by the image processing unit (image/video adjustment module 11), and then calculated by the central processing unit or the processing unit to calculate the overall block structure energy of the image content structure. Many possible important images can be processed by block structure energy. In an embodiment, based on the method of picture deformation based, a block structure energy can be introduced for structural preservation. In order to evaluate the importance of an image, the present invention utilizes intensity gradients, block structure energy, and content saliency (figure) to achieve image size adjustment, and to protect object structures and to minimize distortion by a holistic approach. In other words, the present invention proposes a holistic approach to each structural member; that is, the top-down method, without the need to connect pixels (image slitting), adjacent pixels (Wolf et al., 2007) or adjacent Grid (the method proposed by Wang et al. in 2008), the distortion produced by these methods will accumulate in the local smoothing. The holistic approach forces the bounding box to scale evenly so that the content structure is safe within the frame. The relative importance can be determined by the energy of the proposed block structure.

In order to capture the outline of a color image, reference is made to Document 9 (JVD Weijer, T. Gevers, and AWM Smeulders, "Robust photometric invariant features from the color tensor," IEEE Trans. Image Processing, vol. 15, p. 2006, 2004. The color tensor mentioned in ) can be applied to detect protruding edges. The captured contour can be connected to other object contours and spread over the overall image. Therefore, the edges of many pieces can be decomposed by simple cutting at the corners, which can be detected by the Harris corner detector. It is worth noting that structural segments containing insufficient pixels are processed as noise and then removed.

The present invention provides a simpler and more efficient way to protect captured structural segments; that is, block structure energy. The basic concept of structure preservation is that all pixels within the energy of the block structure should be stretched or compressed as evenly as possible. In order to maintain a significant structure, each structural segment should be treated as a single unit. For the straight line example, the pixels on the line segment should have the same slope after adjustment. Therefore, all pixels on the line segment are limited to have the same slope. There are many significant structural segments of different shapes that are protected after adjustment. As shown in the third figure, the captured (content) structure segment 200 includes lines and curves and forces it to be uniformly deformed in each direction. In the third figure, the block structure energy constitutes a jelly-like space and protects the structural segments from severe distortion. For example, the capture structure segment 200 can be compressed along the x-axis, y-axis, and (x and y) axis directions to become structural segments 201, 202, and 203, respectively. It is to be noted that the straight portions of the structural segments 201, 202 and 203 are still straight, and the curved portions of the structural segments 201, 202 and 203 are also appropriately curved without artifacts.

Next, the elasticity of the size adjustment of each structural segment in the x and y directions can be defined. Where Bi represents the set of pixels in the boundary block, which delimits and protects the structure i . For the adjustment of the x direction, the energy values of the block structure of all pixels in the Bi block are the same, which is defined as:

Where ei represents the set index of the edge pixels of the structure i , Indicates the partial differentiation of an input image to x, that is, the intensity gradient. It is worth noting that the pixel Pj may be covered by more than one block, so the total combined energy in the x direction can be defined as:

Where Esal represents a saliency map, which can be provided by Itti et al. (Ref. 8: L. Itti, C. Koch, and E. Niebur, "A model of saliency-based visual attention for rapid scene analysis," IEEE Trans Pattern Anal. Mach. Intell., 1998). It is worth noting that ∣ Gx ∣, Esal and Normalized and ranged from 0 to 1. The total block structure energy along the y direction can also be defined in a similar manner:

According to Equations 2 and 3, the importance of the input image can be evaluated by intensity gradient, block structure energy and saliency patterns. The total block structure energy can be selected or determined by intensity gradient, block structure energy, and saliency patterns, such as the maximum value therein. The combined energy maps in different axial directions (eg, energy maps for horizontal or vertical adjustment) are defined separately to provide better flexibility to facilitate adjustment (adjustment in the x and y directions). It is worth noting that the energy of the block structure in different directions is different according to the directionality of each structure. In other words, if some structures are easily damaged in one direction, the structure of the block will be reinforced in that direction.

Next, referring to the first figure, in a second step 101, it is performed by a central processing unit or processing unit to determine or evaluate the compression ratio of the input image content structure in each direction. Excessive compression/elongation can distort the content structure, so distributing compression/extension along the x and y directions is important to minimize distortion. From the above, the importance of the input image can be evaluated by intensity gradient, block structure energy, and saliency map. In other words, the optimal compression ratio can be evaluated based on the energy map of the image content. In one embodiment, the present invention utilizes adjustments in two directions, such as using a homogenous image area to narrow and extend in the x and y directions, which can be referenced to file 6. This method is called using the change width and length to match the target object aspect ratio. Images with different content can have different resilience to adjust, and the compression ratio in each direction can be evaluated. The compression ratio is defined as follows:

among them Indicates the intensity gradient of the pixel (x, y), versus The average of the maximum intensity gradient values along the x and y directions. The above two values versus It is the critical value of the magnitude of the important intensity gradient, which also provides the bottom line of the compression ratios r _x and r _y . The compression ratios r _x and r _y are evaluated by Equation 4. Please refer to the fourth figure, which shows an example of the compression ratio. In the fourth figure, an input image 300, such as a car, displays intensity gradient sizes 301 and 302 along the x and y directions, respectively, and the compressed region 303 can be selected or determined. The compressed area 303 may be a relatively unimportant area. Since this embodiment uses the maximum operator, this measurement is a fundamental safety line.

Next, referring to the first figure, in a third step 102, it is performed by a central processing unit or processing unit to determine the optimal scaling factor for the input image content structure in each direction. In the present invention, only the aspect ratio of the source image to the target image can be considered. Image resizing is defined as adjusting the source (input) image of size Wo × Ho to the target (adjusted) image of size Wr × Hr . Wo represents the width of the source image, and Ho represents the height of the source image. Wr represents the width of the image being adjusted, and Hr represents the height of the image being adjusted. In an embodiment, the optimal scaling factor between the source image and the target image may only consider its aspect ratio. The adjustment factors of the x and y directions are set to S _x and S _{y respectively} , so the linear relationship between the image resolution between the input image and the adjusted image can be defined as:

Depending on the image content, the optimal S _x and S _y may be different, even if the target aspect ratio ( W _r / H _r ) remains unchanged.

Please refer to the fifth figure, which shows the adjustment factor (S _x , S _y ) coordinate system, where the x-direction adjustment factor S _x represents the x-axis coordinate and the y-direction adjustment factor S _y represents the y-axis coordinate. The adjustment factor is located in the x direction (S _x , 0), which is located at the right and left of the point (1, 0), indicating the coordinate elongation and compression, respectively. Similarly, the adjustment factor is located along the y-direction (0, S _y ), which is located at the top and bottom of the point (0, 1), respectively, indicating coordinate elongation and compression. In the fifth figure, it shows the best adjustment factor resolution ( ) must meet the following requirements: aspect ratio limit (ie, Equation 5), compression ratio and close to the original size, that is, point (1,1). In other words, in order to reduce the distortion of the adjusted image, the first consideration is to make the adjustment factor (S _x , S _y ) close to the original size point (1, 1). The second consideration is that the adjustment factor (S _x , S _y ) is close to the safety line defined by evaluating the compression ratio, for example (1- r _x , 1+ r _y ).

As shown in the fifth figure, combined with the above two considerations and aspect ratio limitations, the problem can be formulated as:

Where β = W _r ‧ H _o / W _o ‧ H _r is a constant. Therefore, the closed-form solution of the above formula, that is, the optimal adjustment factor ( ), can be easily obtained as follows:

It should be noted that ω can be defined or a constant. Depending on the content of the image, ω can be automatically determined by structural complexity assessment. If the structure is clearer, its weight on the compression ratio is greater, and vice versa. Structural complexity can be defined as a function (combination) of image gradient size and direction.

ω= c ‧exp(- H _all )

H _all =( H _{∥ g ∥} + H _g ) (8)

Where H _{∥ g ∥} is the entropy of the image gradient size, which represents the complexity of the gradient variable. H _g represents the entropy of the directional gradient, which is related to the density of the content structure. c is a constant used to rescale ω to a range between 0 and c . Notably H _all a positive number, the complexity related to the gradient magnitude and orientation of the image. Thus, H _all the larger, the smaller is ω. The structural complexity ω can be used as a degree of freedom in image size adjustment.

Referring to the first figure, in step 103, it is performed by a central processing unit or processing unit to solve the least squares solution of the sparse linear system. As described above, the elongation/reduction relationship between the x and y directions is defined as S _x =β‧S _y by the resolution of the source image and the target image. Corresponding to the optimal adjustment factor for different image content versus It can be determined by Equation 7. If β is less than 1, the image should be compressed along the x-axis and elongated along the y-axis. If β is greater than 1, the image is converted to be compressed along the y-axis and elongated along the x-axis. In the present invention, for example, it is required that a given size image W _o ×H _{o is} adjusted to a size of S _x W _o ×S _y H _o , which needs to re-establish the pixel p=( under three types of restriction conditions) The new coordinates (Xp, Yp) of x, y). For example, the above restrictions may be geometric constraints. For example, take the calculation of Xp. In the first constraint, each pixel is assumed to be at a fixed distance from its left and right sides: X _{x, y} -X _{x-1, y} =1 and X _{x+1, y} -X _{x, y} =1. The second constraint is to map each pixel to a position similar to one of its upper and lower edges: X _{x , y} - X _{x , y + 1} =0. The third constraint condition is the size of the image deformed image to the size of the adjusted image: X _{1 , y} =1 and . In one embodiment, for content-aware image adjustment, an important pixel should be deformed to a distance close to one of its neighboring pixels, while a less important pixel can be mixed or split with its neighboring pixels. Therefore, the first constraint condition should be based on the corresponding structural energy value. (that is, Equation 2) is weighted. Because the reduction and elongation of the pixels in the energy of each block structure should be as uniform as possible, if , the maximum important value of 1 can be assigned to the second constraint (smooth term), where For all pixel collections, covered by structural blocks. The equations in the linear system are as follows:

Weighting Represents the importance between the upper and lower pixels, which controls the smoothness of the pixels X _{x , y} . All of the equations in Equation 9 constitute an overdetermined constrained sparse linear system in which the first three equations represent neighboring constraints and the fourth equations indicate boundary constraints. The best new coordinates of the pixels can be obtained by reducing the sum of the squared errors of the above equations, which is equivalent to finding the least squares solution (x) of the sparse linear system, ie . Similarly, the coordinate variables Y _{x , y of the} pixel (x, y) can be obtained from the least squares solution of the corresponding sparse linear system:

Weighting Represents the importance between the left and right pixels, which controls the smoothness of the pixels Y _{x , y} . Similarly, all of the equations in Equation 10 constitute an overdetermined constrained sparse linear system in which the first three equations represent neighboring constraints and the fourth equations indicate boundary constraints. In other words, according to Equations 9 and 10, the least squares solution of the above sparse linear system can be obtained. Based on the solution of the above-described sparse linear system, the degree of image deformation can be obtained.

Referring to the first figure, in step 104, it is executed by a central processing unit or processing unit to deform the input image with a new coordinate matrix (X and Y) and uniformly scale the input image to a target image resolution. Based on Equations 9 and 10, the adjusted image can be generated by deforming the original image based on the new coordinates (X and Y) of all pixels. It is worth noting that the target size (S _x W _o ×S _y H _o ) may not be equal to the true target image resolution (W _r ×H _r ). Although the image resolution is adjusted to (S _x W _o ×S _y H _o ) instead of (W _r ×H _r ) after the deformation process, it can be uniformly scaled to (W _r ×H _r due to the same aspect ratio. ). Examples of compression ratio and adjustment factor evaluation can be shown in the third, fourth and fifth figures. Based on the different energy maps in the x and y directions, it can be known whether the evaluation result is suitable for adjustment in a particular direction.

For example, S _x = 0.62 and S _y = 1.24, the compression ratio is high in both directions, so the compression in the x direction may be larger, and the elongation in the y direction may be smaller. To visualize the final operation on the overall image, the distance between adjacent pixels in the x and y directions along the X and Y matrices is measured separately. Corresponding compression/elongation distribution versus It is colored by a jet color strip. It can be observed that the structure can be well preserved by the energy of the block structure.

Another subject of the invention relates to a video retargeting. Regarding video adjustment, the summarized algorithm is shown in the sixth figure. Video adjustment is an existing method of video conversion to meet the size of an arbitrary display. Please refer to the sixth figure, which shows the method flow of the video adjustment according to the present invention. Similarly, video adjustments can be implemented in an image/video processing device or display, such as the device in the second figure. In a first step 500, it is performed by a central processing unit or processing unit to determine/calculate the total block structure energy and compression ratio for each frame. Based on medium video cameras becoming more affordable, users can easily create their own multimedia content. In order to conform to different types of video/video processing devices or display devices, such as digital TVs, computers, notebook computers, mobile phones, Apple mobile phones (iPhone), Apple music players (iPod), e-books and digital photo frames Etc., depending on its content, the frame should be adaptively adjusted. However, individually adjusting each frame may result in flicker artifacts. In order to meet the needs of video adjustment, some corrections may be made and extended to the above-mentioned image adjustment technology. First, the most significant area that attracts the viewer's attention in a video clip is the moving target (object). Based on this consideration, the significance (outburst) measurements in Equations 2 and 3 can be expressed as: by normalizing the moving energy Em:

Where E _m represents the size of the moving motion field calculated from the adjacent image frame. Based on efficiency considerations, the motion vector can be evaluated only on the detection corner. As with camera movement, it approximates the average of the motion vectors, while real object movement can be evaluated by removing the camera movement. It is worth noting that E _m is also normalized to a range between 0 and 1.

Next, referring to the sixth diagram, in a second step 501, it is performed by a central processing unit or processing unit to determine the optimal scaling factor for each frame. Based on the total energy assessed, the optimal adjustment factor for each frame in the video ( ) can be evaluated. According to the above, the above optimal scaling factor ( ) can be easily determined by Equation 7.

Next, in a third step 502, it is performed by a central processing unit or processing unit to resolve a sparse linear system with additional constraints and a smoothing scaling factor versus . In order to maintain time homology, the best scaling factor It can be smoothed over time by using a moving average method. corresponding Can be limited by aspect ratio ( Smoothing And decided. Similar to the temporal smoothing in Reference 5, the smoothing of the mapping between adjacent frames is forced by a linear system comprising the following constraints (i.e., Equation 12) to Equations 9 and 10. The above restriction equation is as follows:

Similarly, according to Equations 9 and 10 and additional constraints (ie, Equation 12), the least squares solution of a sparse linear system can be easily determined.

Referring to the sixth diagram, in step 503, it is executed by the central processing unit or processing unit to deform the input image using the coordinate matrix (X and Y) and uniformly scale the input image to output the target video resolution. Based on equations 9 and 10 and additional constraints, the adjusted image can be generated by deforming the original image based on the new coordinates of all pixels. Since the deformation matrix X and Y are smooth and similar between adjacent frames, the deformation matrix can be evaluated by the initial guess obtained from the previous frame, and the corresponding linear system can utilize the symmetric LQ method. Solved efficiently (Ref. 10: CC Paige and MA Saunders, "Solution of sparse indefinite systems of linear equations," SIAM J. Numer. Anal., vol. 12, pp. 617-629, 1975).

For more comparison results, please refer to the demo video http://cv.cs.nthu.edu.tw/research/CAMR-SP/index.htm .

Referring to the sixth diagram, in step 504, it is executed to maintain the above-described evaluation coordinate matrices X and Y as the initial solution for the next frame. Then, in step 505, the program reverts to step 500 to begin image adjustment for the next frame. Next, repeat the above steps.

The present invention has been described above by way of a preferred example, and is not intended to limit the spirit of the invention. Modifications and similar configurations made within the spirit and scope of the invention are intended to be included within the scope of the appended claims.

5. . . Input image

10. . . Image/video processing device or display device

11. . . Image/video adjustment module

12. . . Central processing unit or processing unit

13. . . monitor

15. . . working system

16. . . Input unit

17. . . Output unit

19. . . Memory unit

20. . . Frame memory

twenty one. . . Video memory

100. . . Calculate the energy of the overall block structure

101. . . Estimated compression ratio

102. . . Determine the best scale factor

103. . . Solving the least squares solution of sparse linear systems

104. . . Deforming the input image by X and Y coordinate matrix and uniformly scaling the input image to become the target image resolution

200, 201, 202, 203. . . Structural fragment

300. . . Input image

301. . . Intensity gradient size distribution in the X direction

302. . . Intensity gradient size distribution in the Y direction

303. . . Compressed area

500. . . Calculate the total block structure energy and compression ratio of each frame

501. . . Determine the best scaling factor for each frame

502. . . Solve sparse linear systems with additional constraints and smoothing scale factors

503. . . Deform the input image using the coordinate matrices X and Y and evenly scale the input image to output the target video resolution

504. . . Maintain evaluation of the coordinate matrix X and Y as the initial solution for the next frame

505. . . step

The above elements, as well as other features and advantages of the present invention, will become more apparent after reading the contents of the embodiments and the drawings thereof:

The first figure shows a flow chart of image size adjustment in accordance with the present invention.

The second figure shows a block diagram of an image/video processing device.

The third graph shows the compression of the content structure along the x and y directions.

The fourth figure shows an example of a compression ratio.

The fifth graph shows a schematic diagram of the aspect ratio limitation of the preferred scale factor in accordance with the present invention.

The sixth diagram shows a flow chart of video adjustment in accordance with the present invention.

100. . . Calculate the energy of the overall block structure

101. . . Estimated compression ratio

102. . . Determine the best scale factor

103. . . Solving the least squares solution of sparse linear systems

104. . . Deforming the input image by X and Y coordinate matrix and uniformly scaling the input image to become the target image resolution

Claims (8)

A method for image adjustment includes: determining, by a processing unit, an overall block structure energy of an input image content; and determining, by the processing unit, a compression ratio of the input image content based on the overall block structure energy; The compression ratio ( r _x and r _y ) of the input image content can be obtained by the following equation: Where G(x,y) is equal to , which represents the intensity gradient of the pixel (x, y), versus Means an average of the maximum intensity gradient values along the x and y directions; the processing unit determines an optimum scaling factor for the input image content; and the processing unit deforms the input image by utilizing a new coordinate matrix The content and uniformly scales the input image content to output a target image resolution. The method for image adjustment according to claim 1, further comprising solving the least square solution of a sparse linear system by the processing unit before deforming the input image content. The method of image adjustment according to claim 1, wherein the overall block junction The energy of the structure can be determined by the intensity gradient, the block structure energy, and the saliency map. The method of image adjustment according to claim 1, wherein the optimal scaling factor ( , ) can be obtained by the following equation: Where β is equal to W _r . H _o / W _o . H _r , ( S _x , S _y ) represents an adjustment factor. A method for video adjustment includes: determining, by a processing unit, an overall block structure energy and a compression ratio of one of the input video content, wherein the compression ratio ( r _x and r _{y ) of} the input video content ) can be obtained by the following equation: Where G(x,y) is equal to , which represents the intensity gradient of the pixel (x, y), versus Means an average of the maximum intensity gradient values along the x and y directions; the processing unit determines an optimum scaling factor for the input video content; and the processing unit deforms the input video by utilizing a new coordinate matrix The content and the input video content are evenly scaled to output a target video resolution. The method of video adjustment according to claim 5, further comprising: before the deforming the input video content, solving a least square solution of a sparse linear system including a constraint condition by using the processing unit, and smoothing the optimal scaling factor, Where the restrictions include Where X and Y represent the new coordinate matrix, and x and y represent an original calibration matrix. The method of video adjustment according to claim 5, wherein the overall block structure energy can be determined by an intensity gradient, a block structure energy, and a saliency map. The method of image adjustment according to claim 5, wherein the optimal scaling factor ( , ) can be obtained by the following equation: Where β is equal to W _r . H _o / W _o . H _r , ( S _x , S _y ) represents an adjustment factor.