TWI384876B - Method for upscaling images and videos and associated image processing device - Google Patents

Description

Image and video enlargement method and related image processing device

The present invention relates to an image and video upscaling method, and more particularly to an enlarged output image when a low-resolution input image is enlarged and displayed on a high-resolution video device. It still has anti-aliasing, sharp edges, rich detail and strong contrast.

In recent years, various digital camera devices (such as digital cameras and video cameras) have been continuously updated. Not only are image quality getting higher and higher, the product volume is getting smaller and smaller, and the market price is becoming cheaper. Therefore, these digital camera devices are becoming cheaper. It has become increasingly popular and has become an indispensable tool in people's daily life and work. Take the mobile phone with camera function, which is quite popular on the market, as an example. In addition to the CCD or CMOS camera unit, the mobile phone has a small LCD screen for displaying images. The captured image is for the user to browse. Generally, when a user uses a mobile phone to take a scene, the captured image is stored in a memory card inserted therein, and when the user clicks on the image on the mobile phone, the mobile phone will be self-made from the memory card. After reading the image to be browsed and re-encoding, the reduced image of the image is displayed on the liquid crystal display screen, so that the user can reduce, enlarge, drag or adjust the image on the liquid crystal display screen. Control actions such as page size. Therefore, regardless of whether the image stored on the aforementioned digital camera device is several million bytes (eg, 1.2 Mega byte), or hundreds of thousands of bytes (eg, 120K byte), the images are all Stored in a storage device (such as a memory card, hard disk or flash drive) set on the computer, the digital camera device reads the image from the storage device each time the user clicks on the image to be browsed. After re-encoding, the original image is reduced to a tens of thousands of bytes (such as: 75K byte) or thousands of bytes (such as: 7.5K byte), and the reduced image is displayed in a small size. The LCD screen is displayed on the screen for the user to browse, and the user can perform operations such as zooming out, zooming in, dragging, or resizing the thumbnail according to actual needs.

Since the image resolution and the capture speed of these digital camera devices have become higher and higher, various digital camera devices have been widely used in various fields of expertise, such as criminal, biological, medical and astronomical sciences. Etc., to preserve many important documents, such as: critical clues in criminal cases, evidence of evidence or crime scenes; new discoveries or experimental results in biological sciences; medical X-rays or computed tomography images, etc. For medical personnel to judge the symptoms of the disease, therefore, how to save important documents in the laboratory or other research fields for professional researchers, and archive them in digital image format, as important data for review or comparison in future experiments or research And when the digital image is enlarged, the image distortion can be effectively reduced, and the digital image with high resolution and favorable recognition can be provided for interpreting and judging the relevant features presented in the digital image, that is, becoming the professional A topic that people in the field attach great importance to.

According to image processing companies and designers, image and video amplification technology has been researched for many years, from the initial linear amplification technology to the later edge-based amplification technology. Theories and application methods have been continuously developed. The commonly used linear amplification techniques include Bilinear Interpolation, Bicubic Interpolation and Lanzcos, while edge-based amplification. The most representative of the technology is NEDI (New Edge-Directed Interpolation). However, these amplification techniques still have many shortcomings, in which the aforementioned linear amplification technique is prone to edge jagged edge effects, loss of detail and edge blurring ( Blurr) and the like, and the aforementioned edge-based magnification technique performs interpolation according to the gradient direction along the edge of the image, although some problems such as edge aliasing and blurring can be solved, but the edges are based on edges. The amplification technique has a large dependence on the accuracy of the edge direction, which is prone to edge The direction is not accurate, which causes interpolation errors. In particular, in areas with rich details and cluttered edges, the problem of interpolation errors is particularly obvious and serious. In addition, these edge-based amplification techniques maintain their results. The accuracy of the image often requires the use of huge calculations, which is easy to cause inefficiency in image processing. Finally, since the currently used amplification techniques generally use a weighted summation of neighboring pixels (pixels) to perform interpolation calculation, the original image has a low-pass filtering effect, so after the image is enlarged Inevitably, it will cause the loss of sharpness and detail information (high frequency part) on the original image. In view of this, in order to restore the quality of the original image, the image processing industry tends to perform some enhancement and recovery processing on the image after the image is enlarged, but these processes generate edge overshoot and ringing. New flaws such as the ringing effect.

Therefore, how to provide a new image enlargement technology, so that the lower resolution image and video are enlarged and displayed on a higher resolution video device, still can display a clear and advantageous digital image for identifying relevant features. For example, it has become an important issue that every image processing company and designer is trying to study and urgently need to solve.

In view of the above, it is an object of the present invention to provide an image and video enlargement method and related image processing apparatus to improve the disadvantages of the aforementioned conventional image enlargement technique.

The invention discloses an image and video enlargement method, which is applied to an image processing device, and after a digital image is input into the image processing device, the image processing device can be according to a predetermined enlargement ratio. Outputting an enlarged digital image, the method comprising: providing a preprocessing module and a composite amplification module, wherein the preprocessing module performs a high pass filtering process on the input image to extract a high frequency of the input image a portion for performing high frequency compensation on the enlarged image, performing an image decomposition process on the input image, extracting a gradient of the input image by using a gradient operation element, and fixing a threshold according to one of presets, The input image is decomposed into a flat area and an edge area, and the two areas are marked on the input image, and the composite amplification module respectively amplifies the original input image, the flat area, the edge area, and the high frequency part, For the original input image and the flat region, a simple interpolation operation, such as a bicubic interpolation, is performed for the edge region and the high frequency portion. At the same time, complex interpolation operation and simple interpolation operation are used at the same time. The complex interpolation operation uses the directional interpolation operation on the edges of the edge region and the high-frequency portion to perform amplification processing, and the result of the interpolation operation in the direction is performed. Confidence processing, that is, the more the edge direction is clear, the higher the credibility of the direction interpolation operation result is. Otherwise, the credibility of the direction interpolation operation result is lower, and then the direction is interpolated according to the calculated credibility. The result of the operation and the result of the simple interpolation operation are weighted sum. Finally, the original input image, the flat region, the edge region, and the high-frequency portion are amplified, and the operation is small. In the case of low complexity and high speed, the output image with the characteristics of anti-aliasing, sharp edges, rich details and strong contrast is output according to the predetermined magnification ratio.

The present invention further discloses an image processing apparatus for outputting an enlarged digital image according to an amplification ratio after a digital image is input to the image processing apparatus. The image processing apparatus includes: a preprocessing module configured to perform a high pass filtering process on the input image to extract a high frequency portion of the input image, the high frequency portion being used for the input image Amplifying the result to perform high frequency compensation, and the preprocessing module performs an image decomposition process on the input image, wherein the image decomposition process extracts an image gradient of the input image by using a gradient operation element, and according to the preset a fixed threshold, the input image is decomposed into a flat area and an edge area, and the two areas are marked on the input image; a composite amplification module that uses a simple input image and the flat area Interpolation operation, performing amplification processing, using a complex interpolation operation and the simple interpolation operation for the edge region and the high frequency portion, respectively performing amplification processing, and then performing reliability processing on the result of the complex interpolation operation, wherein The more clear the edge direction, the higher the credibility of the result of the complex interpolation operation. Otherwise, the credibility of the result of the complex interpolation operation is lower, and finally, according to the credibility And performing a weighted summation of the result of the complex interpolation operation and the result of the simple interpolation operation; and a fusion processing unit for merging the original input image, the flat region, the edge region, and the amplification result of the high frequency portion into the Output image.

In order to enable the review committee to have a better understanding and understanding of the purpose, processing program and its effects of the present invention, the examples are illustrated and accompanied by the illustrations, which are described in detail as follows:

The present invention relates to an image and video enlargement method. Referring to FIG. 1, the method is applied to an image processing apparatus 1 after a digital image is input to the image processing apparatus 1. The image processing apparatus 1 can output an enlarged digital image according to a predetermined magnification ratio, so that a lower resolution input image and video are enlarged and displayed on a higher resolution video device. Digital images that are clear and useful for identifying relevant features can still be presented. In a preferred embodiment of the present invention, as shown in FIG. 1, the method includes providing a preprocess module 10 and a composite up-scaling module 30, see section 2. As shown in the figure, the pre-processing module 10 performs a high-pass filtering process 11 on an input image, and extracts a high-frequency portion of the input image to utilize the high-frequency portion of the input image during subsequent processing. The amplification result of the image is subjected to high frequency compensation, and an image decomposition processing 12 is performed on the input image, and the input image is decomposed into a flat region and an edge region by using a gradient operation unit, and the composite amplification module 30 is paired with the original The input image, the flat region, the edge region, and the high-frequency portion are respectively subjected to amplification processing, wherein simple interpolation operation is used for the original input image and the flat region, and complex interpolation operation is used for the edge region and the high-frequency portion, and finally, The fusion result of the original input image, the flat area, the edge area, and the high frequency portion is subjected to a blending process 33, that is, an output image can be generated according to a predetermined magnification ratio.

Since the conventional interpolation amplification technique currently used has the characteristics of low-pass filtering, after the input image is enlarged, the amplified output image necessarily loses the high-frequency information in the input image. In this embodiment, the reference is repeated. As shown in FIG. 1 , in order to prevent excessive loss of high-frequency information, the pre-processing module 10 performs preparatory work for subsequent processing, and performs high-pass filtering processing on the input image first. Referring to FIG. 2, Extracting high-frequency components in the input image. During the amplification process of the input image, the high-frequency portion and the edge region are enlarged using the same interpolation algorithm, when the original input image, flat region, edge region, and high frequency After the amplification is partially completed, the fusion 33 is performed to compensate for the high frequency portion of the input image lost during the amplification process.

In addition, since the human eye is often particularly sensitive to the edge regions with strong gradients in the image, in this embodiment, in order to reduce the amount of computation of the subsequent interpolation amplification processing, to simplify and accelerate the overall computational performance, only the image The edge region that is particularly sensitive to the human eye uses complex interpolation operations (such as directional interpolation) for high-precision amplification, and simple interpolation operations for flat regions in the image, such as bicubic interpolation (bicubic interpolation), the amplification process, this expedient method, in the end can still obtain a clear magnified image without affecting the visual effect. In addition, due to the more chaotic edge regions in the image (such as grass, etc.), the edge direction is not easy to accurately judge, and the human eye is not too interested in the amplification processing of the disordered edge regions, so, In this embodiment, simple interpolation operations are also used for the chaotic edge regions to simplify the complexity and computational complexity of the interpolation amplification process. In order to achieve the foregoing object, referring to FIG. 3, when performing the image decomposition processing 12 on an input image, the embodiment first performs an image decomposition 121 on the input image to decompose the flat region and the edge region. The method uses a gradient operation element to extract a gradient of the high-frequency component Grd ( x ). In this embodiment, a sobel operator is used to extract a gradient of the high-frequency component Grd ( x ), and then, according to the human eye. sensitivity to changes in gradient, using the fixed threshold ThreshD 16, (1) a decomposition of the following formula flat area (Plane) and the edge region (edge), and the high-frequency component element (pixel) labeled Label (x ):

; 嗣, refer to Figure 3, and then perform a de-scrambling process 122 for the more chaotic edge elements in the edge region, and remove the cluttered edge primitives. The method is to extract one of the edge primitives first. a predetermined range of neighborhoods M × N , and then counting the number Nedge of the inner edge primitives in the neighborhood, and judging that the number of edge primitives in the neighborhood exceeds a predetermined number according to the following formula (2) (too much or When there are too few), the edge element is deleted:

If ( Nedge > ThrH or Nedge < ThrL ) remove the edge ;......................(2)

, where ThrL = min( M , N ), ThrH = 0.8. M. N. Finally, in order to enlarge the edge region, as shown in Fig. 3, this embodiment performs a morphological dilation 123 on the periphery of the edge primitive, and in this embodiment, a cross structure element is used. The morphological expansion process 123 is performed on the edge primitives to enlarge the edge regions.

In this embodiment, as shown in FIGS. 1 and 3, the composite amplification module 30 performs amplification processing on the original input image, the flat region, the edge region, and the high frequency portion, respectively, for the edge region and the high frequency. In part, the first composite interpolation amplification module 31 is used for interpolation amplification and gain processing. For the original input image and the flat region, the second composite interpolation amplification module 32 is used for interpolation and amplification processing, and the first composite interpolation amplification module 31 is used. The interpolation amplification algorithm used in the second composite interpolation amplification module 32 can be the same, and in the actual operation, the interpolation and amplification operations in the first composite interpolation amplification module 31 and the second composite interpolation amplification module 32 are performed. Simultaneously, the overall calculation amount can be saved. Finally, the image outputted by the first composite interpolation amplification module 31 and the second composite interpolation amplification module 32 is subjected to a fusion process 33, and the entire calculation process is performed. The following formula (3) can produce a high-resolution output image according to a predetermined magnification ratio:

HR ( x )= LR ( x )* Hp ( x )* CUp ( x ). Gain + LR ( x )* CUp ( x ) ................(3)

Where HR ( x ) is a high-resolution output image, LR ( x ) is a low-resolution input image, and Hp ( x ) is a high-pass filter function used by the pre-processing module 10, CUp ( x ) is the The interpolation amplification function used in the first composite interpolation amplification module 31 and the second composite interpolation amplification module 32, Gain is a constant gain factor used in the first composite interpolation amplification module 31.

Generally speaking, the composite interpolation amplification modules 31 and 32 mainly perform the amplification function of the input image, and the interpolation amplification technique currently used can only enlarge the input image to an even multiple, and if it is to be enlarged to any multiple, it is combined. Use the down sample technique. Now, taking the magnification twice as an example, the functions of the composite interpolation amplification modules 31 and 32 will be described. Referring to FIG. 4, a high-resolution image obtained by interpolation and magnification is shown. It is a primitive that is directly copied from a low-resolution input image. Other primitives are primitives that are interpolated from black primitives. In the interpolation and amplification operation of the present invention, the calculation is first marked as gray. Finally, the white primitive is recalculated, and the present invention uses different interpolation and amplification operations for the edge region and the flat region which are decomposed from the original input image, wherein a smart interpolation is used for the edge region. For example, direction interpolation operation, for flat areas, simple interpolation is used, such as: bicubic interpolation operation.

In this embodiment, the first composite interpolation amplification module 31 respectively performs interpolation and amplification processing on the edge region and the high frequency portion of the output of the preprocessing module 10, and the second composite interpolation amplification module 32 respectively processes the preprocessing module. 10 output original input image and flat area are subjected to interpolation and amplification processing, and the first composite interpolation amplification module 31 and the second composite interpolation amplification module 32 can use the same interpolation amplification algorithm, as shown in FIG. 5, the only difference The point is that the amplification result of the first composite interpolation amplification module 31 is to be subjected to gain processing. Because the pre-processing module 10 has marked the edge region and the flat region on the original input image, according to the markers, the first composite interpolation amplification module 31 and the second composite interpolation amplification module 32 are targeted to the image. The primitive performs the edge region judging process 51. If it is marked as an edge region, the complex interpolation operation 52 is employed, and if it is marked as a flat region, a simple interpolation operation (for example, a bicubic interpolation operation) 53 is employed, wherein the complex interpolation is performed. The operation structure of the operation 52, as shown in FIG. 6, is for the current primitive in the edge region, and simultaneously performs a directional interpolation operation 521 (a directional interpolation) and a simple interpolation operation 53 (for example, a bicubic interpolation operation), and then, The result of the direction interpolation operation 521 is performed to perform a confidence and a weight calculation 522. The principle of the reliability calculation is that the edge direction is more clear, and the reliability of the direction interpolation operation 521 is higher. Otherwise, The lower the reliability; finally, the credibility obtained by calculating 522 based on the credibility and weight, and then the result of the direction interpolation operation 521 and the result of the simple interpolation operation 53 A weighted sum process 523 is processed and output.

In this embodiment, the direction interpolation operation 521 first estimates the edge direction according to the gradient value of the current primitive in the edge region, and then acquires other primitives along the edge direction thereof to perform interpolation and amplification operations. Taking the gray point x shown in Fig. 7 as an example, how to use the direction interpolation operation 521 to rotate the interpolation template of the gray primitive x by 45 degrees, and obtain the operation process of the white primitive shown in Fig. 7:

First, referring to Figure 7, using the 12 primitives P ₀ ~ P _{11 of} the gray primitive x neighborhood, interpolate the DPx according to the following formula (4), since the interpolation calculation is from six The direction is judged, and the arrangement of the primitives P ₀ ~ P ₁₁ is closer to a circle, so it has higher precision:

, where a _{i is} the weighting coefficient of the neighboring primitives P ₀ ~ P ₁₁ , and the gradient values in six directions are calculated according to the following formula (5):

Dir ₀ =| P ₀ - P ₃ |, Dir ₁ =| P ₁ - P ₂ |, Dir ₂ =| P ₄ - P ₇ |, Dir ₃ =| P ₅ - P ₆ |, Dir ₄ =| P ₈ - P ₁₁ |, Dir ₅ =| P ₉ - P ₁₀ | ...................(5)

If some of the neighborhood primitives P ₀ ~ P _{11 in} equation (5) have not yet been calculated, a simple interpolation operation 53 (eg, a double-cubic interpolation operation) may be used to estimate the substitute value. Then, using the gradient values, the weight a _{i of} each primitive is calculated according to the following formula (6):

After calculating the weighting coefficients a _i , the weighted sum processing 523 can be performed on the result of the direction interpolation operation 521 and the result of the simple interpolation operation 53 according to the formula (4) to calculate the complexity. The result of interpolation operation 52 is DPx and is output.

In addition, in order to improve the robustness of the direction interpolation operation 521, the present invention prevents an edge region that is not noticeable in the edge direction from generating an erroneous ghost (Ghost) point, as shown in FIG. 2, for the first The amplification result of the simple interpolation operation and the direction interpolation operation in the composite interpolation amplification module 31 and the second composite interpolation amplification module 32 is performed by the following formula (7), and the fusion processing 33 is performed to generate a high resolution according to a predetermined amplification ratio. Output image:

HPx = (1- fMix ). SPx + fMix. DPx ..............................(7)

, where HPx is the primitive value of the final output image, SPx is the amplification result of the simple interpolation operation for the input image primitive, DPx is the amplification result of the direction interpolation operation for the input image, and fMix adopts the following formula (8) The fusion coefficient obtained:

In addition, since the interpolation amplification function used in the foregoing interpolation and amplification operation has low-pass filtering characteristics, the edge of the enlarged image may be blurred, and therefore, the present invention improves the sharpness of the edge of the enlarged image. It is necessary to perform a sharpening process 34 on the image, as shown in FIG. 2, and in order to avoid an overshoot phenomenon at the edge during the sharpening process, the present invention particularly uses the following formula (9). A nonlinear high-pass filter, such as a finite pulse response (Finite Impulse Response, FIR) high-pass filter:

SHP ( x )= Medain ( LocMαx ( x ). LocMin ( x ), Fir ( x )) .....................(9)

, where SHP ( x ) is the result of sharpening, Medain () function is the median operation, LocMax ( x ) and LocMin ( x ) are the maximum and minimum values in the neighborhood of the current primitive, Fir ( x ) is A nonlinear high-pass characteristic FIR filter. In this way, the result of the high-pass filtering is limited between the maximum and minimum values in the local neighborhood, that is, the edge overshoot phenomenon during the sharpening process can be effectively prevented.

Thus, using the method of the present invention to amplify a low-resolution input image to display on a high-resolution video device, not only can the amplification operation be smaller than other edge-based methods, but also in complexity. In the case of low and high speed, the amplified output image still has anti-aliasing, sharp edges, rich details and strong contrast, which can effectively reduce image distortion and improve the image when zooming in on the digital image. Resolution and facilitates identification of digital images of related features.

When the aforementioned image and video amplification method is implemented in software, the execution can be accelerated by the assistance of the hardware. Specifically, the pre-processing module 10 and the composite amplifying module 30 of FIG. 1 are implemented by software code, and the code can be executed by a hardware graphics processing unit (GPU). . If the image processor supports general parallel computing, that is, a General Purpose GPU (GPGPU), the code can be executed at a faster speed to improve the image and video amplification efficiency. For example, if an image processor supporting a CUDA technology (Compute Unified Device Architecture, parallel computing technology developed by NVIDIA Corporation) is used, the foregoing items executed by the preprocessing module 10 and the composite amplification module 30 can be used. Operations, such as high-pass filter processing 11, simple interpolation operation 53, complex interpolation operation 52, fusion processing 33, sharpening processing 34, etc., are written as CUDA kernel function (kernel function) code, and support CUDA technology When performing these CUDA kernel functions, the image processor can generate multiple threads to execute the same CUDA kernel function at the same time. This parallel operation can greatly improve the efficiency.

The aforementioned image and video enlargement method can be applied to the field of post processing, for example, to a video playback software, as shown in FIG. 8A. 8A is a diagram showing a video playback software architecture in which a video player 80 includes a video decoder 81, a preprocessing module 10, a composite amplification module 30, and a video renderer. ) 82. The video decoder 81 can obtain image data from various video sources to decode the same; the pre-processing module 10 and the composite amplification module 30 amplify the decoded image data; and the video renderer 82 uses image rendering (rendering) Technology, such as Microsoft's Direct3D technology, draws and displays the magnified image data.

In general, when the architecture of FIG. 8A is placed in a computer, the video decoder 81 stores the decoded image data in the system memory of the computer, and the pre-processing module 10 and the composite amplification module 30 The decoded image data stored in the system memory is enlarged. The pre-processing module 10 and the composite amplification module 30 can also be integrated into the video decoder 81. However, if the computer has an image processor that supports CUDA technology, and the pre-processing module 10 and the composite amplification module 30 are implemented by the code of the CUDA kernel function, the pre-processing module 10 and the composite amplification module 30 are implemented. Integrated into the video renderer 82, as shown in FIG. 8B, the effect of hardware acceleration can be further improved. This is because, when the video decoder 81 passes the decoded image data to the video renderer 82, the video renderer 82 establishes a Direct3D surface (here, using Direct3D technology as an example) to display the Image data, and these surface images are stored in display memory. On the other hand, the CUDA technology supports the Direct3D interoperability mechanism, so that the pre-processing module 10 and the composite amplification module 30 can directly process the Direct3D surface. Therefore, the pre-processing module 10 and the composite amplification module are The 30 is integrated into the video renderer 82 to directly zoom in on the Direct3D surface stored in the display memory, which helps to improve efficiency and avoid wasting time on data transfer between the system memory and the display memory.

Figure 9 is a block diagram of one embodiment of a video playback software architecture of Figure 8B, wherein video player 90 includes video decoder 91 and video renderer 92, and video renderer 92 includes amplification module 921 and fusion. Mixer and presenter 922. The architecture of Figure 9 is implemented by an image processor that supports CUDA technology, and the amplification module 921 is implemented by the code of the CUDA kernel function. After the video renderer 92 receives the decoded image data from the video decoder 91, the Direct3D surface is created in the display memory, and the zoom module 921 extracts the image texture from the Direct3D surface in the display memory (ie, Block 9210), which is divided into a luminance component (ie, a Y component) and a chrominance component (ie, a U/V component) to perform different amplification processes: the luminance component sequentially passes through a denoising point (ie, 9211) ), de-blocking (block 9212), edge interpolation (block 9213), anti-aliasing (block 9214), edge sharpening (block 9215), etc. Processing; the chrominance component is subjected to bilinear interpolation (block 9216). Finally, the amplification module 921 sets the image texture to be displayed according to the enlarged luminance component and the chrominance component (ie, block 9217), and sends it to the fuser and the renderer 922 for drawing display. For the foregoing blocks 9211~9216, an appropriate CUDA kernel function can be separately written to implement, so that when the image processor executes these CUDA kernel functions, it can be executed in parallel to greatly increase the amplification module 921. Operational efficiency.

The above description of the present invention is intended to be illustrative of the preferred embodiments of the invention. It will be apparent to those skilled in the art that many variations are possible in light of the above embodiments without departing from the spirit and scope of the invention.

1. . . Image processing device

10. . . Preprocessing module

11. . . High-pass filtering

12. . . Image decomposition processing

121. . . Image decomposition

122. . . Go to messy processing

123. . . Morphological expansion

30. . . Composite amplifier module

31. . . First composite interpolation amplification module

32. . . Second composite interpolation amplification module

33. . . Fusion processing

34. . . Sharpening

51. . . Judgment processing of edge regions

52. . . Complex interpolation

53. . . Simple interpolation

521. . . Direction interpolation

522. . . Credibility and weight calculation

523. . . Weighted summation

80,90. . . Video player

81,91. . . Video decoder

82,92. . . Video renderer

921. . . Amplification module

922. . . Fusion and renderer

9210. . . Extract image texture

9211. . . Denoising

9212. . . Deblocking

9213. . . Edge interpolation

9214. . . Anti-aliasing

9215. . . Edge sharpening

9216. . . Bilinear interpolation

9217. . . Set image texture

1 is a schematic structural view of an image processing apparatus of the present invention;

2 is a schematic diagram showing a detailed structure of an image processing apparatus of the present invention;

Figure 3 is a detailed schematic diagram of the image decomposition processing of the present invention;

Figure 4 is a schematic diagram of a high resolution image obtained by interpolation amplification in a preferred embodiment of the present invention;

FIG. 5 is a schematic structural diagram of a first composite interpolation amplification module and a second composite interpolation amplification module in the embodiment;

Figure 6 is a detailed schematic diagram of a complex interpolation amplification operation in the embodiment;

Figure 7 is a schematic diagram of a high resolution image obtained by direction interpolation in the embodiment;

Figure 8A is a schematic diagram of the architecture of a video playback software;

8B is a schematic diagram of integrating the pre-processing module and the composite amplification module of FIG. 8A into a video renderer; and

Figure 9 is a block diagram of one embodiment of a video playback software architecture of Figure 8B.

10. . . Preprocessing module

11. . . High-pass filtering

12. . . Image decomposition processing

30. . . Composite amplifier module

31. . . First composite interpolation amplification module

32. . . Second composite interpolation amplification module

33. . . Fusion processing

34. . . Sharpening

Claims (22)

An image and video enlargement method is applied to an image processing apparatus, and after a digital image is input to the image processing apparatus, the image processing apparatus can output an enlarged digit according to an amplification ratio. Image, the method includes: providing a pre-processing module for performing a high-pass filtering process on the input image to extract a high-frequency portion of the input image, the high-frequency portion being used for the input image The amplification result performs high frequency compensation, and the preprocessing module performs an image decomposition process on the input image, and the image decomposition process extracts an image gradient of the input image by using a gradient operation element, and according to the pre Setting a fixed threshold, decomposing the input image into a flat area and an edge area, and marking the two areas on the input image; providing a composite amplification module for using the original input image and the flat area a simple interpolation operation for performing amplification processing; for the edge region and the high frequency portion, a complex interpolation operation and the simple interpolation operation are respectively used for amplification processing; and then, the complexity is The result of the value operation is subjected to credibility processing, wherein the more the edge direction is, the higher the credibility of the result of the complex interpolation operation is. Otherwise, the credibility of the result of the complex interpolation operation is lower; finally, according to the Credibility, weighting and summing the result of the complex interpolation operation and the result of the simple interpolation operation; and performing a fusion process to fuse the original input image, the flat region, the edge region, and the amplification result of the high frequency portion Become the output image. The method of claim 1, wherein the composite amplification module comprises a first composite interpolation amplification module and a second composite interpolation amplification module, wherein the first composite interpolation amplification module is an input image Interpolation amplification and gain processing are performed on the edge region and the high frequency portion, and the second composite interpolation amplification module performs interpolation and amplification processing on the original input image and the flat region, wherein the first and second composite interpolation amplification modules are used. The interpolation operation is the same. The fusion processing combines the images output by the first composite interpolation amplification module and the second composite interpolation amplification module. The interpolation amplification processing and the fusion processing are performed according to the following formula, with a predetermined amplification. Proportional, produces a high-resolution output image: HR ( x )= LR ( x )* Hp ( x )* CUp ( x ). Gain + LR ( x )* CUp ( x ) where HR ( x ) is a high-resolution output image, LR ( x ) is a low-resolution input image, and Hp ( x ) is used by the pre-processing module The high-pass filter function, CUp ( x ), is an interpolation amplification function used by the first and second composite interpolation amplification modules, and Gain is a constant gain factor used by the first composite interpolation amplification module. The method of claim 2, wherein the image decomposition processing uses the gradient operation element to extract an image gradient Grd ( x ) of the input image, and then, according to the sensitivity of the human eye to the gradient change Using the fixed threshold ThreshD , the flat and edge regions are resolved according to the following formula, and the input image is labeled Labele1 ( x ): The method of claim 3, wherein the complex interpolation operation uses a one-way interpolation operation to amplify the edges of the edge region and the high-frequency portion. The method of claim 4, wherein the direction interpolation operation estimates the edge direction according to the gradient value of a certain edge primitive, and then acquires the primitive in the neighborhood along the edge direction to perform the direction. Interpolation operation. The method of claim 5, wherein the direction interpolation operation first estimates the edge direction according to the gradient value of the current primitive x in the edge region, and then acquires the current primitive along the edge direction thereof. The other 12 primitives P ₀ ~ P _{11 of the} x neighborhood are interpolated to calculate DPx according to the following formula: Where a _{i is} the weighting coefficient of the neighboring primitives p _i (where i =0 to 11), and the gradient values in the six directions are calculated according to the following formula: Dir ₀ =| P ₀ - P ₃ |, Dir ₁ =| P ₁ - P ₂ |, Dir ₂ =| P ₄ - P ₇ |, Dir ₃ =| P ₅ - P ₆ |, Dir ₄ =| P ₈ - P ₁₁ |, Dir ₅ =| P ₉ - P ₁₀ | Then, using the gradient values, the weighting coefficients a _{i of the} respective primitives p _i are calculated according to the following formula: After calculating the weighting coefficients a _i , the result of the direction interpolation operation and the result of the simple interpolation operation are subjected to weighted summation processing, and the result DPx of the complex interpolation operation is calculated and output. The method of claim 6, wherein the amplification result of the simple interpolation operation and the direction interpolation operation in the first composite interpolation amplification module and the second composite interpolation amplification module is performed by using the following formula: To produce the high-resolution output image at a predetermined magnification: HPx = (1- fMix ). SPx + fMix. DPx where HPx is the primitive value of the final output image, SPx is the result of a simple interpolation amplification operation for the primitive of the input image, and DPx is the result of the direction interpolation operation for the primitive of the input image, fMix is The fusion factor obtained using the following formula: The method of claim 7, wherein the image decomposition process further performs a de-scrambling process on the edge primitives in the edge region to remove cluttered edge primitives. The method of claim 8, wherein the de-scrambling process first extracts a predetermined range of neighborhood M × N of an edge primitive, and then counts the number Nedge of the edge primitives in the neighborhood, and According to the following formula, when it is determined that the number of edge primitives in the neighborhood is not within a predetermined range, the edge primitive is deleted: if ( Nedge > ThrH or Nedge < ThrL ), the edge primitive is deleted; wherein ThrL = Min( M , N ), ThrH =0.8. M. N. The method of claim 9, wherein the pre-processing module performs a morphological expansion process on the edge region to complete the de-noising process to enlarge the edge region. The method of claim 10, wherein the morphological expansion process uses a cross-structure element to extend the edge regions to the edge elements. The method of claim 11, further comprising: using a finite pulse wave response filter of nonlinear high-pass filtering, performing a sharpening process on the output image to limit the result of the high-pass filtering to a local neighborhood Between the maximum and minimum values. The method of claim 12, wherein the finite pulse response filter is sharpened by the following formula: SHP ( x ) = Medain ( LocMax ( x ). LocMin ( x ), Fir ( x ) Where SHP ( x ) is the result of sharpening, Medain () function is the median operation, LocMax ( x ) and LocMin ( x ) are the maximum and minimum values in the neighborhood of the current primitive, Fir ( x ) is the Finite pulse response filter. The method of claim 1, wherein the simple interpolation operation is a bicubic interpolation operation. The method of claim 1, wherein the gradient operand is a sobel operand. The method of claim 1, wherein the pre-processing module and the composite amplifying module are implemented by a code, and the code is executed by an image processor (GPU), wherein The image processor supports parallel operations to improve the efficiency of execution of the code. The method of claim 16, wherein the image processor supports CUDA technology. The method of claim 17, wherein the pre-processing module and the composite amplifying module are integrated in a video renderer, wherein the video renderer establishes a Direct3D surface in a display memory, and The pre-processing module and the composite amplification module extract the input image from the Direct3D surface. An image processing apparatus for outputting an enlarged digital image according to an amplification ratio after a digital image is input to the image processing apparatus, the image processing apparatus comprising: a preprocessing module, Performing a high-pass filtering process on the input image to extract a high-frequency portion of the input image, the high-frequency portion is for performing high-frequency compensation on the amplified result of the input image, and the pre-processing module The input image performs an image decomposition process for extracting an image gradient of the input image by using a gradient operation element, and decomposing the input image into a flat region according to a preset fixed threshold. And an edge region, and marking the two regions on the input image; a composite amplification module that performs a magnification process on the original input image and the flat region using a simple interpolation operation; for the edge region and the height The frequency part is respectively subjected to a magnification process using a complex interpolation operation and the simple interpolation operation; then, the result of the complex interpolation operation is subjected to credibility processing, wherein the edge direction It is clear that the higher the credibility of the result of the complex interpolation operation, otherwise, the lower the credibility of the result of the complex interpolation operation; finally, according to the credibility, the result of the complex interpolation operation and the simple interpolation operation The result is weighted and summed; and a fusion processing unit is used to fuse the original input image, the flat region, the edge region, and the amplification result of the high frequency portion into the output image. The image processing device of claim 19, wherein the simple interpolation operation is a bicubic interpolation operation. The image processing device of claim 19, wherein the complex interpolation operation is a direction interpolation operation. The image processing device of claim 19, wherein the gradient operation unit is a sobel operation element.