TW201032592A - Method for upscaling images and videos and associated image processing device - Google Patents

Abstract

A method for upscaling images and videos and an associated image processing device are provided. The method provides a preprocess module for extracting a high-frequency portion of an image inputted into the device and decomposing the image into plane and edge regions. The method also provides a composite upscaling module for executing the upscaling processes on the image and the high-frequency portion thereof respectively, wherein the upscaling process of the plane regions of the image and the high-frequency portion is based on a simple interpolation while the edge regions of the image and the high-frequency portion is based on both a complex interpolation and the simple interpolation. The upscaling results of the image and the high-frequency portion are then processed by a fusion process, so as to output an image having sharp but not blocky edges, rich details and strong contrast.

Description

201032592 VI. Description of the Invention: [Technical Field] The present invention relates to an image and video (vide〇) upscaling method, and more particularly to a low-resolution input image that is enlarged and displayed at a high level. On a resolution video device, the magnified output image still has the features of anti-aliasing, sharp edges, rich detail, and strong contrast. 〇[Prior Art] In recent years, 'various ages: camera installations such as digital cameras and photo-taking, etc., continue to innovate, not only the image quality is getting higher and higher, the product volume is getting smaller and smaller, and its market price is getting cheaper. These digital camera devices have become increasingly popular and become an indispensable tool for people's daily life and work. For example, in the market, there are mobile phones with cameras in the market. In addition to CCD or CM〇s camera repair, these mobile phones are used to shoot _ outside's still-small size LCD display screen. • Used to display the captured image for the user to browse. In general, when the viewer's mobile phone is used to view the scene, the captured image is stored in a memory card inserted in it. When the user clicks on the image on the mobile phone, the mobile phone will be self-made. The image of the image to be read is read in the memory card, and after re-encoding, the thumbnail image of the image is displayed on the liquid crystal display screen. Thus, the image can be reduced and enlarged on the liquid crystal display screen. Drag or adjust the page size and other manipulations. Therefore, regardless of whether the above-mentioned digital camera device stores a few million pours (such as U Mega byte) or hundreds of thousands of bytes (eg, 120 Kbyte), the images are stored in 4 In 201032592 - in the storage of money (such as memory card, hard drive or flash drive), when the user selects the job _ image, the digital camera reads the image from the storage and re-encodes , the original image is reduced to tens of thousands of groups (such as: 75 Kbyte) or thousands of bytes (such as: 7.5 Kbyte) After the inclination, the beginning of the state _ in the small size Wei Jing display on the screen The user is paralyzed and allows the user to perform zooming, dragging, or page resizing operations on the zoomed out image as needed. From the position of the county, the image resolution and the spine speed have become higher, so that the money simple image devices have been disregarded in various professional fields such as criminal, biological, medical and astronomical sciences, etc. Many important facts 'such as critical clues in criminal cases, evidences of evidence or crime scenes; new discoveries or experimental results in biological sciences; medical X-ray films or computed tomography images for medical personnel to judge symptoms The information 'how' is how to save important documents for professional researchers in the laboratory or other research areas (4) and to store them in digital image format for review or comparison of important data in future experiments or studies, and in the logarithmic position. When the image is magnified, it can effectively reduce the image distortion, provide high resolution and help to distinguish the 'Like digital image' for (4) riding the touch image (10) to develop relevant features', which is highly valued by professionals in this field. a subject. According to 'image processing operators and designers for image and video (video) amplification technology, has been carried out for many years, whether from the initial linear amplification technology 'to later edge-based amplification technology, new Theories and application methods have been continuously developed. Among them, the commonly used linear amplification techniques include Bilinear Interpolation, Bicubic Interpolation and Lannzc〇s. 201032592 The most representative of the edge-enlarged zoom technology is NEDI (New Edge-Directed Interpolation). However, there are still many shortcomings in these amplification techniques. The aforementioned linear amplification technique is prone to edge jagged effects and details. Problems such as loss and edge blur, and the aforementioned edge-based magnification technique performs interpolation according to the gradient direction along the edge of the image, although the part can be solved.

Edge aliasing and blurring, etc., however, these edge-based magnification techniques have a large dependence on the accuracy of the edge direction, and the valley is prone to interpolation errors due to inaccurate edge directions, especially Yes, in areas with rich details and cluttered edges in the image, the problem of interpolation errors is particularly obvious and serious; in addition, these edge-based amplification techniques often require the use of magic operations for the quasi-mixing of the results of the county. It is easy to cause problems in image processing efficiency. Finally, since the amplification techniques currently used are generally based on the neighborhood (the weighted sum of the nei moving b〇rh〇〇(j) primitives (pixel), (10) the stored value meter #, the low-pass filtering of the shed image Μ 'Therefore, the image is placed in the field, which will cause the loss of the sharpness and detail information (Gao Qingfen) that should be on the original image. There is a product in this image processing industry to restore the shed image. f, often after the image is enlarged, 'the image is enhanced and restored, but the processing ΐ ϋ ϋ 缘 〇 〇 〇 〇 〇 〇 〇 〇 ring ring ring ring ring ring ring ring ring ring ring ring ring ring ring ring ring ring ring ring ring ring ring ring ring ring 如何 如何 如何 如何 如何 如何 如何 如何A new _ image amplification technology, so that the resolution == frequency is amplified and displayed in the higher resolution video gift / η I is now clear and the __ feature of the digital image, that is, become "image processing and design The human thorn is working hard and = 6 201032592 An important subject to be solved. [Invention] In view of the above, an object of the present invention is to provide an image and video enlargement method and related image processing apparatus to improve The aforementioned conventional figure • the disadvantage of zooming in. * The invention discloses A method for magnifying an image and a video, which is applied to an image processing apparatus, and after the image-in-digital image is input to the image processing apparatus, the image processing apparatus can output according to a predetermined magnification ratio An enlarged digital image includes a preprocessing module and a composite amplification module, wherein the preprocessing module performs a high pass filtering process on the input image to extract a high frequency portion of the input image And performing high frequency compensation on the enlarged image, performing an image decomposition process on the input image, extracting a gradient of the input image by using a gradient operation element, and fixing a threshold according to one of presets The input image is decomposed into a flat area and an edge area, and the two areas are marked on the input image, and the composite amplification module respectively amplifies the original input image image, the flat area, the edge area, and the high frequency part, For the original input image and the flat region, a simple broadcast value operation is used, such as: bicubic interpolation, and the amplification process is performed for the edge region and the high frequency portion. At the same time, complex interpolation operation and simple interpolation operation are used. 'Complex interpolation operation is to use the directional interpolation operation on the edges of the edge region and the high-frequency part to perform amplification processing, and the result of the interpolation operation in the direction can be performed. Confidence processing (confidence process), that is, the more the edge direction is clear, the more the entanglement of the direction interpolation operation result is, otherwise, the credibility of the direction interpolation operation result is more than 7201032592 ^ then according to the credibility obtained by 7 The result of the interpolation operation and the weighted yarn (weighted _) of the single interpolation operation, and finally the amplification result of the original input image, the flat region, the edge region and the high frequency portion, = line fusion, that is, the operation can be performed Smaller amount, lower complexity ^ In the case of a money of 5, according to the predetermined amplification, the output - has an output image with anti-aliasing, sharp edges, rich details and strong contrast. • The present invention further discloses a “image processing device” which, after inputting the digital image into the image processing device, can output a large digital image according to the magnification ratio. The image processing apparatus includes: a pre-processing module for performing a high-pass filtering process on the input image to extract a high-frequency portion of the input image, the high-frequency portion being used for the input image Amplifying the result to perform high frequency compensation 'the preprocessing module and performing an image decomposition process on the input image, the image decomposition process extracting an image gradient of the input image by using a gradient operation element, and according to the preset a fixed threshold, the input image is decomposed into a flat area and an edge area' and the area is marked on the input image, a composite amplification module that uses one for the original input image and the flat region Simple interpolation operation, performing amplification processing, using a complex interpolation operation and the simple-interpolation operation 'for amplification processing' for the edge region and the high-frequency portion respectively, and then performing reliability processing on the result of the complex interpolation operation The more clear the edge direction, the higher the credibility of the result of the complex broadcast value operation. Otherwise, the credibility of the result of the complex interpolation operation is lower. And weighting the result of the complex interpolation operation and the result of the simple interpolation operation according to the reliability degree; and a fusion processing unit for amplifying the original input image, the flat region, the edge region, and the high frequency portion As a result, the fusion becomes the output image. 201032592 In order to enable the reviewing committee to have a better understanding and understanding of the purpose, processing program and function of the present invention, the embodiments are described in detail with reference to the accompanying drawings. An image and video enlargement method, as shown in FIG. 1 ', the method is applied to an image processing apparatus 1 after a digital image is input to the image processing apparatus 1, the image Processing device i ❹ outputting an enlarged digital image at a predetermined magnification ratio so that a lower resolution input image and video are amplified and displayed on a higher resolution video device It can present digital images that are clear and useful for identifying relevant features. In a preferred embodiment of the present invention, as shown in FIG. 1, the method includes providing a pre-processing module, a repr〇cess module, and a composite amplification module (c〇mp〇site up seaiing module). 30 'Refer to Figure 2, where the pre-processing module 1 pair = input image execution - high-pass filtering process u, extract the © frequency portion of the input image' for use in subsequent processing The high-frequency part of the amplified image of the input image is subjected to a high-frequency surface, and an image decomposition processing is performed on the input image. The gradient operator is decomposed into a flat region and a miscellaneous _, Lai New The module 30 performs amplification processing on the original input image, the flat region, the edge region, and the high-frequency portion respectively. [In the original input image and the flat region, simple interpolation operation is used.] For the edge region and the high-frequency portion, the use is complicated. The interpolation operation, finally, the original input image, the flat region, the edge region, and the high-frequency portion of the amplification result '--fusion processing 33, that is, the predetermined amplification ratio, 9 201032592 produces an output image.

Since 'the traditional interpolation amplification technique currently used has the characteristics of low-pass filtering', after the input image is enlarged, the amplified output image will inevitably lose the high-frequency information in the input image. In this embodiment, As shown in FIG. 1 , in order to prevent excessive loss of high-frequency information, the pre-processing module 1 performs preparatory work for subsequent processing, and performs high-pass filtering processing on the input image first. Referring to FIG. 2, To extract high-frequency components in the input image, during the amplification process of the input image, the high-frequency portion and the edge region are enlarged using the same interpolation algorithm, when the original input image, the flat region, the edge region, and the height are After the frequency portion is amplified, the fusion processing (fUsion) 33 is performed to compensate the high frequency portion of the input image lost during the amplification process. In addition, since the human eye is often particularly sensitive to the edge regions with strong gradients in the image, in this embodiment, in order to reduce the amount of computation of the subsequent interpolation amplification processing, to simplify and accelerate the overall computational performance, only the image The edge area that is particularly sensitive to the human eye uses complex interpolation operations (eg, directional interpolation) to perform high-precision amplification processing. For flat areas in the image, simple interpolation operations such as bicubic interpolation are used. (bicubic interpolation) 'Amplify the processing, this is an expedient method, and finally can obtain a clear magnified image without affecting the visual effect. In addition, due to the more chaotic edge regions in the image (such as grass, etc.), the edge direction is not easy to accurately judge, and the human eye is not too interested in the amplification processing of the disordered edge regions, so, In this embodiment, simple interpolation operations are also used for the chaotic edge regions to simplify the complexity and computational complexity of the interpolation amplification process. 201032592 In order to achieve the foregoing object, as shown in FIG. 3, when performing the image decomposition processing 12 on an input image, the embodiment first performs an image decomposition 121 on the input image to decompose the flat region and the edge. The region, which uses the gradient operator to extract the gradient of the high frequency component, in this embodiment uses the sobel operator to extract the gradient of the high frequency component - and then, according to the sensitivity of the human eye to the gradient change Using the fixed 'threshold', the flat area (edge) and the edge area are decomposed according to the following formula (1), and the primitive (pixei) of the south frequency component is marked ❹ Label{x)

LabeKx)4Edge,^Grd^Thr^D) ... [Plane, if{Grd{x)<ThreshD) ...................... v ^ ^ ; 嗣, refer to Figure 3, and then perform a de-scrambling process 122 for the more cluttered edge primitives in the edge region to remove the cluttered edge primitives. The method is to extract one of the edge primitives first. The neighborhood of the predetermined range ΜΧΛΤ 'Recount the number of edge elements in the neighborhood

Nedge, and according to the following formula (2), when it is determined that the number of edge primitives in the neighborhood exceeds a predetermined number (too much or too little), the edge primitive is deleted: if {Nedge > ThrH or Nedge < ThrL) remove the edge ...................... (2) , where 77wi = min(M,A〇,7%r//= 〇.8 M." Finally, in order to enlarge the edge region, as shown in Fig. 3, this embodiment performs a morphological dilation 123 on the periphery of the edge primitive, in this embodiment 11 201032592 The morphological expansion process 123 is performed on the edge primitives by using a cross structure element to enlarge the edge regions. In this embodiment, as shown in FIGS. 1 and 3, the composite amplification module 30 The original input image, the flat region, the edge region, and the high frequency portion are respectively subjected to amplification processing, wherein the first composite interpolation amplification module 31 is used for the edge region and the high frequency portion, and interpolation amplification and gain processing are performed, The original input image and the flat region are interpolated and amplified using the second composite interpolation amplification module 32. The interpolation amplification method used in the first composite interpolation amplification module and the first composite interpolation amplification module 32 may be the same, and in the actual operation, the first composite interpolation amplification module 31 and the second composite The interpolation and amplification operations in the interpolation and amplification module 32 can be simultaneously performed to save the overall calculation amount. Finally, the images output by the first composite interpolation and amplification module 31 and the second composite interpolation and amplification module 32 are merged. (iUsion) processing 33, the entire calculation process according to the following formula (3), that is, a high-resolution output image can be generated according to a predetermined amplification ratio: β HR(x) = LR(x) * Hp(x) * CUp(x) Gain + LR(x) * CUp(x) ( 3) , where ww is a high-resolution output image and is a low-resolution input image' is a Qualcomm consideration used by the pre-processing module 1Q The wave function, o/pw, is an interpolation amplification function used in the first composite interpolation amplification module 3丨 and the second composite interpolation amplification module 32. The constant is a constant benefit factor used in the first composite interpolation amplification module 31. In general, the composite interpolation amplification modules 31 and 32 are mainly implemented. The zoom function of the input image 'The interpolation amplification technique currently used can only enlarge the image into the even multiple of 12 201032592. If you want to enlarge it to any multiple, you should use the down sample technique. Now zoom in two For example, the functions of the composite interpolation amplification modules 31 and 32 are described. Referring to FIG. 4, a high-resolution image obtained by interpolation and amplification is shown, wherein the symbol marked as black is directly from The primitives copied from the low-resolution input image, and the other primitives are the primitives based on the interpolation of the black primitives. In the interpolation and amplification operation of the present invention, the grayscale is first calculated. The primitive, and finally, the white primitive is recalculated, and the present invention uses different interpolation and amplification operations for the edge region and the flat region derived from the original input image, wherein a complex interpolation operation is used for the edge region. For example, direction interpolation operation, for flat areas, simple interpolation (such as: double-cube interpolation). In this embodiment, the first composite interpolation amplification module 31 respectively performs interpolation and amplification processing on the edge region and the high frequency portion of the output of the preprocessing module 10, and the second composite interpolation amplification module 32 respectively processes the preprocessing module. The first input image and the flat area of the output are subjected to interpolation and amplification processing, and the first complex interpolation amplification module 31 and the second composite interpolation amplification module 32 can use the same interpolation amplification algorithm, as shown in FIG. The only difference is that the amplification result of the first composite interpolation amplification module 31 is to be subjected to gain processing. Because the pre-processing module 1〇 has marked the edge region and the flat region on the original input image, according to the markers, the first composite interpolation amplification module 31 and the second composite interpolation amplification module 32 are targeted to the image. The primitive element performs the edge region judgment processing 51. If it is marked as the edge region, the complex interpolation operation 52 is employed, and if it is marked as a flat region, a simple interpolation operation (for example, a double-cubic interpolation operation) 53 is employed, wherein the complex Interpolation operation 13 201032592 Calculate the operation structure of 52, as shown in Fig. 6, for the current picture in the edge region, simultaneously perform direction interpolation operation 521 (directi〇nal interpolation) and simple interpolation operation 53 (eg: double cubic interpolation Then, for the result of the direction interpolation operation 521, the reliability (confid·) and the weight calculation are performed. The principle of the reliability calculation is that the edge direction is more clear, and the result of the direction interpolation operation 521 is credible. Degrees are more than •, otherwise, the lower the degree of volatility; finally, the credibility obtained by calculating 522 according to the credibility and weights Single ❹ interpolation result 53 'of a weighted sum (weighted sum) process 523, and to output. In this embodiment, the 'direction interpolation operation 521 first estimates the edge direction according to the gradient value of the current primitive in the edge region, and then acquires other primitives along the edge direction thereof to perform interpolation and amplification operations. Taking the gray point x of the 7th riding as an example, how to use the direction interpolation operation 52 to rotate the interpolation model of the gray primitive X to obtain the operation process of the white primitive shown in FIG. 7: ❹ First, the complex Referring to Figure 7, the 12 primitives in the neighborhood of the gray primitive X are used, and the interpolation calculation is performed according to the following formula (4). Since the interpolation calculation is performed from six directions Judging, and the arrangement of the primitives η~Λ is closer to a circle, so it has higher precision:

Dpxh ρ〆..............................(4) where α is the neighboring primitive p&quot The weighting factor is calculated according to the following (5) 201032592 formula (5), and the gradient values in six directions are calculated: ^o-IPo-^l » D/r, =|/>_/>2| &gt ;〇/>2 =|^4_p71 D,r^Ps-P6\ » D/r4=|P8_pM| , Dir5=\P9-PX0\ , if there are some neighborhood primitives in equation (5). ~& has not been calculated, you can use the simple interpolation operation % (such as: double cubic interpolation operation) to estimate the replacement

value. Then, using these gradient values, the weight α of each primitive is calculated according to the following formula (6): gamma ==min(Z)i>0,...£), wide 5) (6) lf (Dirnm ~ Dir0 ) a0=a3= 0.5, other at=〇=Dirt) a'= a2 = Q,5, other ai=Q .(^min == Dir2) a4=a7= 0.5, other a,. = 〇=Dir3) A5 = a6 = 0.5, age a: = 〇.... If 伽-=Dir4) ag=au = 0.5, other a. = 〇.if (Dirmn = Dir5) a9 = a10 = 0.5, other = 〇〇 , 俟 calculate the weighting coefficients ", after which, according to the formula (4), then the result of the direction interpolation operation 521 and the simple interpolation operation magic knot. If 'weighted sum processing 523, The result of the complex interpolation operation 52 is calculated and output. Further, 'the present invention is for improving the robustness of the direction interpolation operation 521' to prevent an edge region which is not noticeable in the edge direction, and generates an erroneous ghost (Ghost) point, as shown in Fig. 2, for the first The amplification result of the simple interpolation operation and the direction interpolation operation in the composite interpolation amplification module 31 and the second composite interpolation amplification module 32 is performed by the following formula (7), 15 201032592 to perform a fusion process 33 to generate a predetermined amplification ratio. High-level output image: # HPx = (1 - βίΐχ). SPx + /Mix - DPx ......................... (7 )

Where ^/Λ is the primitive value of the final output image, milk is the enlarged result of the simple interpolation operation of the input image primitive, and is the enlarged result of the direction interpolation operation for the input image. The fusion coefficient obtained by the following formula (8): A

Dir /6 /«0 if (D/rmin '2 > Dirmean) fMix = 0.25: else if (Dir* . 4 > Dir lion)/Mix = 〇·5; ...... (8) else /Mix = h In addition, since the interpolation amplification function used in the aforementioned interpolation amplification operation has low-pass filtering characteristics, the edge of the enlarged image may be blurred. Therefore, the present invention aims to improve the enlarged image. The sharpness of the edge 乃 requires a sharpening of the image 34, as shown in Fig. 2, and in order to avoid overshooting at the edges during sharpening, the present invention particularly uses the following A nonlinear high-pass filter as shown in equation (9), such as a finite pulse response (Finite Impulse Response, FIR) high-pass filter: SHP(x) = MedaiiKLocMax(x).L〇cMin(x), Fir (x)) ....... ( 9 ^ , where the result of 颂〇〇疋 sharpening, the ubi-cafe () function is taking the median operation, and is the maximum and minimum values in the neighborhood of the current primitive, 201032592 It is a FIR filter with nonlinear high-pass characteristics. Thus, the result of high-pass filtering is kept between the local __maximum and minimum values, which is effective. Prevent edge overshoot during sharpening. So 'Let's invent the Green's low-resolution input image to be displayed on a high-resolution video device, not only can it be enlarged The edge of the wire is small, and in the case of low complexity and speed 'Nhigh', the enlarged image is made of anti-woven, sharp edges, rich details and strong contrast, in the logarithmic map. When performing ❹ amplification, it can effectively reduce image distortion, and provide high-resolution digital images that are useful for recognizing related features. When the above-mentioned image and video amplification method is implemented in software, it can be assisted by hardware. To speed up the execution. Specifically, the pre-processing module 10 and the composite amplification module 3 of the figure are implemented by software code, and the code can be implemented by a hardware image processor (graphics pr〇) The cessing unit (GPU) is executed. If the image processor supports general parallel computing, that is, the General Purpose GPU (GPGPU), the code can be executed at a faster speed to enhance the image and Video placement For example, if an image processor supporting CUDA technology (Compute Unified Device Architecture 'parallel computing technology developed by NVIDIA Corporation) is used, the pre-processing module 10 and the composite amplification module 3 can be executed. The foregoing operations, such as high-pass filter processing n, simple interpolation operation 53, complex interpolation operation 52, fusion processing 33, sharpening processing 34, etc., are written as code of the CUDA kernel function, and Image processors that support CUDA technology can generate multiple CUDA kernel functions to simultaneously execute the same 17 201032592 CUDA kernel function. This type of parallel operation can greatly improve efficiency. The aforementioned image and video enlargement method can be applied to the field of post processing, for example, in video playback software, as shown in Fig. 8A. Figure 8A shows the architecture of a video playing software, wherein the 'video player 80' includes a video de-embryo (vide〇decoder) 81, a pre-processing module 1〇, a composite amplification module 3〇, and a video. Video renderer 82. The video decoder 81 can obtain image data from various video sources to decode the same; the pre-processing module 1 and the complex 10 amplification module 3 amplify the decoded image data; the video renderer 82 utilizes Image rendering technology, such as Microsoft's Direct3D technology, displays the magnified image data. In general, when the architecture of Figure 8A is placed in a computer, the video decoder 81 stores the decoded image data in the system memory of the computer, while the pre-processing module 10 and the composite amplification module 3 The decoded image data stored in the system memory is enlarged. The pre-processing module 1A and the composite amplifying module 30 can also be integrated in the video decoder 81. However, if the computer has an image processor that supports CUDA technology, and the pre-processing module 1 and the composite amplification module 30 are implemented by the code of the CUDA kernel function, then the pre-processing module 10 and the composite amplification mode are implemented. The group 3 is integrated into the video dying 82, as shown in Fig. 8B, which further enhances the effect of hardware acceleration. This is because, when the video decoder 81 passes the decoded image data to the video renderer 82, the video renderer 82 establishes a Direct3D surface (here, using Direct3D technology as an example) to display the Image data' and these surfaces are stored in display memory. On the other hand, the CUDA technology supports the Direct3D interoperability 201032592 mechanism, so that the pre-processing module 10 and the composite amplification module 30 can directly process the Direct3D surface. Therefore, the pre-processing module 1〇 and the composite amplification module 30 are Integrated into the video renderer 82 to make it possible to directly zoom in on the Direct3D surface stored in the display memory, which helps to improve efficiency and avoid wasting time on data transfer between the system memory and the display memory. Figure 9 is a block diagram of one embodiment of a video playback software architecture of Figure 8B, wherein 'video player 90 includes video decoder 91 视频 and video > narrator 92' and video annihilator 92 contains The amplification module 921 and the mixer and presenter 922. The architecture of Fig. 9 is implemented by an image processor that supports CUDA technology, and the amplification module 921 is implemented by the code of the CUDA kernel function. After the video eliminator 92 receives the decoded image data from the video decoder 91, the Direct3D surface is stored in the display memory, and the amplification module 921 extracts the image texture from the Direct3D surface in the § 己 体 (texmre) (ie, block 9210), which is divided into a luminance component (ie, a gamma component) and a chrominance component (ie, a U/V component) to perform different amplification processes: the luminance component is sequentially denoised (de-noising) 'ie block 9211), de-blocking (block 9212), edge interpolation (block 9213), anti-aliasing (block 9214), edge sharpening ( Block 9215) is processed; the chrominance component is subjected to a bilinear interpolation (block 9216). Finally, the enlargement module 921 sets the image texture to be displayed (i.e., block 9217) according to the enlarged luminance component and the chrominance component, and sends it to the fuser and renderer 922 for drawing display. For the foregoing blocks 9211~9216, an appropriate CUDA kernel function can be implemented by 201032592 respectively, so that the fine processor can execute these CUDA kernel scales in parallel, so as to greatly enhance the amplification module 921. Operational efficiency. The above description is difficult to implement in detail (4) and is not intended to limit the scope of the present invention. Those skilled in the art will be able to do so without departing from the spirit and scope of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS Fig. 1 is a schematic view showing the structure of an image processing apparatus of the present invention; Fig. 2 is a detailed diagram of an image processing apparatus of the present invention, and Fig. 3 is an image decomposition of the present invention. The detailed structure of the process is illustrated in Fig. 4, which is a schematic diagram of a high resolution image obtained by interpolation in a preferred embodiment of the present invention; Fig. 5 is a first composite interpolation in this embodiment. Schematic diagram of the amplification module and the second composite interpolation amplification module; Fig. 6 is a detailed schematic diagram of the complex interpolation and amplification operation in the embodiment; Fig. 7 is an analysis of the direction interpolation operation in the embodiment Figure 8A is a schematic diagram of a video playback software architecture; Figure 8B is a schematic diagram of integrating the pre-processing module and composite amplification module of Figure 8A into a video renderer; and 201032592 Figure 9 A block diagram of one embodiment of a video playback software architecture of Figure 8B. [Explanation of main component symbols] Image processing device................1 Preprocessing module................10 • Qualcomm Filter processing................11 Image decomposition processing................12 ® Image decomposition 121 Unscrambled processing 122 Form Learning expansion processing 123 compound amplification module ................30 first composite interpolation amplification module................31 Two composite interpolation amplification module ................32 fusion processing................33

A. Sharpening processing................34 Judgment processing of edge areas................51 Complex interpolation operation... .............52 Simple interpolation operation................................. ...53 Direction Interpolation Operation 521 Credibility and Weight Calculation 522 Weighted Summation Processing 523 21 201032592 Video Player................80, 90 Video Decoder... .............81,91 Video Renderer................82, 92 Amplifier Module......... .......921 . Cage and renderer ................922 . Extract image texture............... .9210 Denoising ................9211 © Deblocking................9212 Edge Interpolation Operation... ..........9213 Anti-aliasing................9214 Edge sharpening................9215 Bilinear interpolation operation................9216 Setting image texture................9217 ❹ 22

Claims (1)

201032592 VII. Patent application scope: 1. An image and video enlargement method is applied to an image processing device, and the image processing device can be based on a digital image after being input into the image processing device. Enlarging the ratio, outputting an enlarged number = image, the method comprising: providing a preprocessing module for performing a high pass filtering on the input image to extract a high frequency portion of the input image, The high frequency is used to perform high frequency compensation on the amplified result of the input image, and the preprocessing module performs an image decomposition process on the input image, and the image decomposition processing uses a gradient operation element Extracting an image gradient of the input image, and decomposing the input image into a flat area and an edge area according to one of preset presets, and marking the two areas on the input image; providing a composite amplification module And performing amplifying processing on the original input image and the flat region using a simple interpolation operation; using a complex interpolation operation for the edge region and the high frequency portion respectively Simple interpolation operation is performed to perform amplification processing; then, the result of the complex interpolation operation is subjected to credibility processing, wherein the edge direction is more clear 'the higher the credibility of the result of the complex interpolation operation', otherwise, the complex interpolation operation The lower the credibility of the result is; finally, according to the credibility, the result of the complex interpolation operation and the result of the simple interpolation operation are weighted and summed; and a fusion process is performed to flatten the original input image The enlarged results of the region, the edge region, and the frequency portion are merged into the output image. 2. The method of claim 1, wherein the composite amplifier module 23 201032592 includes a first composite interpolation amplification module and a second composite interpolation amplification module, wherein the first composite interpolation amplification module The group performs interpolation amplification and gain processing on the edge region and the high frequency portion of the input image, and the second composite interpolation amplification module performs interpolation and amplification processing on the original input image and the flat region, wherein the first and second composites The interpolation algorithm used in the interpolation amplification module is the same. The fusion processing system fuses the images output by the first complex interpolation amplification module and the second composite interpolation amplification module, and the interpolation amplification processing and the fusion processing are Produce a high-resolution output image at a predetermined magnification ratio according to the following convention: HR(x) = LR(x) * Hp{x) * CUp(x) · Gain + LR{x) * CUp{ x) where (8) the output image of the Minnan resolution is a low-resolution input image. You W is the high-pass filter function used by the pre-processing module, and ct/pW is the first and second composite interpolation amplification modes. The interpolation amplification function used by the group is Amplifying the first composite interpolation constant gain factor module used. 3. The method of claim 2, wherein the image decomposition process uses the gradient operator to extract an image gradient of the input image, and then, according to the sensitivity of the human eye to the gradient change, Using the fixed threshold of 7VWD, the flat and edge regions are resolved according to the following formula' and the input image is marked: Label{x) = ' ^Grd(x) > ThreshD 〇 [flat, if 4. For patent application The method according to Item 3, wherein the complex interpolation operation performs a magnification process on the edge region and the high-frequency portion of the primitive using a one-direction interpolation operation. The method of claim 4, wherein the direction interpolation operation estimates the edge direction according to the gradient value of a certain edge primitive, and then acquires the primitive in the neighborhood along the edge direction. , perform direction interpolation. 6. The method of claim 5, wherein the direction interpolation* operation first estimates the edge direction based on the gradient value of the current primitive in the edge region, and then obtains other directions along the edge direction thereof. The primitive is interpolated according to the following formula: ODPx = y'alPi ^λ,=1 i=〇9 i=0 where 4 is the weighting coefficient of the neighboring primitives, calculated according to the following formula The gradient values in six directions: Dir0=iP0-P3\ , Dir}=\Pl-P2\ , Dir2^P4~p7\ , Diri=\P5-P6\ , /)//-4=1/3⁄4 -/iil , D/r5 I Then, using the gradient values, calculate the weighting factor A of each primitive A according to the following formula: = min(Dir0,...£>irs) If (gamma *min = gamma.) = α3 = 0.5, other = 〇 if (gamma - = gamma)% = α2 = 0.5, other α, = 0 < (10) ' ==Λ>2) α4 = seven = 〇.5, other % = 〇 if (10) = Λ > 3) α5 = = 0.5, other & = Ο ^(Dirnm = Dir,) a%=au= 0.5 , a, = 0. If gamma_ = gamma·5) a9 = 丨〇 = 0.5, other a, = 0 After calculating the number of lions added, the result of the interpolation operation and the result of the simple interpolation operation The weighted summation process is performed, and the result of the complex interpolation operation is calculated and output. 7. The method of claim 6, wherein the amplification result of the simple interpolation 25 201032592 value calculation and the direction interpolation operation is used for the first composite interpolation amplification module and the second composite interpolation amplification module. The following formula 'fuses the fusion to produce the high-resolution output image at a predetermined magnification: HPx = (1 - β4ίχ). SPX + βίϊχ. DPx where ///3⁄4 is the primitive of the final output image The value, milk is the result of a simple interpolation and enlargement operation for the primitive of the input image, and is the result of the interpolation operation for the direction of the primitive of the input image, and you are using the fusion coefficient obtained by the following formula: ^rmean ~ / 6 mine (gamma-.2 > U record = 0.25; • else if {Dir^ · 4 > Dirmean) βίχχ = 〇5: else βίιχ = \. 8·如申s奋 patent scope 7th The method of the item, wherein the image decomposition process further performs a de-scrambling process on the edge primitives in the edge region to remove cluttered edge primitives. 9. The method of claim 8, wherein the de-scrambling process first extracts a predetermined range of neighborhood w~ of a certain edge primitive, and then counts the number of edge primitives in the neighborhood. And according to the following formula, when it is judged that the number of edge primitives in the neighborhood is not within the predetermined range, the edge primitive is deleted: if (iVetfee > 73⁄4 or < 77yi:) 'delete the edge primitive Wherein ThrL = min(M,N), ThrH =di-MN 〇10. The method of claim 9, wherein the pre-treatment 10,000 group will complete the un-disordered processing The area is carried out - with the expansion of the 'state' to expand the marginal area. 11. The method of claim 10, wherein the morphological & 26 201032592 bulging process uses a cross-structure element to expand the edge region around the edge primitive. 12. The method of claim 11, further comprising using a nonlinear high-pass filtered finite pulse wave response filter to sharpen the output image to limit the result of the high pass filtering to local neighbors Between the maximum and minimum values within the domain. 13. The method of claim 12, wherein the finite pulse response filter is sharpened by the following formula: SHP(x) = Medain{LocMctx{x).LocMin{x), Fir( x)) where is the result of sharpening, the function takes the median operation, and is the maximum and minimum values in the neighborhood of the current primitive, which is the finite impulse response filter. M. The method of claim 1, wherein the simple interpolation operation is a bicubic interpolation operation. 15. The method of claim 1, wherein the gradient operator is a sobel operand. The method of claim 1, wherein the pre-processing module and the composite amplifying module are implemented by a code, and the code is executed by an image processor (GPU). The image processor supports parallel operations to improve the execution efficiency of the code. 17. The method of claim 4, wherein the image processor supports CUDA technology. 18. The method of claim 17, wherein the pre-processing module and the composite amplification module are integrated in a video renderer, wherein the video renderer establishes a display memory in a display memory. ^ to 31) Table 27 201032592 face 'and the thief Lai _ Lai New Butterfly to the Direct3D surface to extract the input image. 9. An image processing apparatus for outputting an enlarged digital image according to an amplification ratio after the digital image is torn into the image processing apparatus. The image processing apparatus comprises: a preprocessing module 'Used input image execution-high-pass filtering process to extract the high-frequency part of the input image, the high-frequency part is used for the input image thief A wire break, the pre-processing module And performing an image decomposition process on the input image, the image decomposition process extracting an image gradient of the input image by using a gradient operation element and decomposing the input image into a preset threshold according to a predetermined threshold a flat area and a side field, and marking the two areas on the input image; a composite amplifying module that performs amplifying processing on the original input image and the flat area system by a simple interpolation operation; The region and the high frequency portion respectively perform amplification processing using a complex interpolation operation and the simple interpolation operation; and then, the result of the complex interpolation operation is subjected to reliability processing, wherein the edge direction is more It is clear that the higher the credibility of the result of the complex interpolation operation, otherwise the lower the credibility of the result of the complex broadcast value operation; finally, according to the credibility of the complex interpolation operation and the simplicity, The result of the value operation is off-weighted and summed; and the -fusion processing unit' is used to fuse the original input image, the flat region, the edge region, and the enlarged result of the sorghum portion into the output image. The image processing device of claim 19, wherein the simple interpolation operation is a bicubic interpolation operation. 21. The image processing device of claim 19, wherein the complex interpolation operation is a one-way interpolation operation. 22. The image processing apparatus of claim 19, wherein the gradient operation unit is a sobel operand. 29