AU2009251141A1 - Image resampling with adaptive sharpening - Google Patents

Abstract

-50 IMAGE RESAMPLING WITH ADAPTIVE SHARPENING A method (Fig. 6) for processing an image comprising input samples located on a 5 regularly spaced grid to produce an output image comprising output samples. The method estimates an isophote orientation (6010) based on a location of an output sample and locations of input samples close to the location of the output sample, and selects (6040), based on the estimated isophote orientation, a directional sharpening filter (6050,6060) having a corresponding sharpening direction. The output of the selected 10 sharpening filter (6050) includes directional frequencies beyond the Nyquist limit determined by a sampling rate of the input samples in a direction of the selected sharpening filter. The method applies the selected sharpening filter to input samples located about the location of the output sample to produce a plurality of sharpened samples (6070) and combines (6080, 7080) the sharpened samples (6080) to produce an 15 output sample at the output sample location based on the location and the isophote orientation. 2453925_1 930061_specilodge start Determine local isophote 6010 6000 orientation Determine reliability of the isophote 6020 orientation 6030 6040 6050 Is yes sophote yes Select vertical orientation predominantly sharpening filter reliable? orizontal no no Select horizontal sharpening filter Calculate default Calculate sharpened 6070 output sample samples using using default selected filter filter Combine sharpened 6080 samples to produce one anti-aliased sample Fig. 6 9Ar.;AO1' 1 930061 fias lodae

Description

S&F Ref: 930061 AUSTRALIA PATENTS ACT 1990 COMPLETE SPECIFICATION FOR A STANDARD PATENT Name and Address Canon Kabushiki Kaisha, of 30-2, Shimomaruko 3 of Applicant: chome, Ohta-ku, Tokyo, 146, Japan Actual Inventor(s): Alan Valev Tonisson Address for Service: Spruson & Ferguson St Martins Tower Level 35 31 Market Street Sydney NSW 2000 (CCN 3710000177) Invention Title: Image resampling with adaptive sharpening The following statement is a full description of this invention, including the best method of performing it known to me/us: 5845c(2458360_1) IMAGE RESAMPLING WITH ADAPTIVE SHARPENING TECHNICAL FIELD The present invention relates to digital image processing and, in particular, to a method and apparatus for resampling a digital image to produce a resampled image having increased sharpness and reduced aliasing. 5 BACKGROUND When sampling a continuous signal with a finite sampling rate, a misrepresentation of the signal may occur. This misrepresentation may result in features of the continuous signal not existing in the sampled signal, or the sampled signal may contain a feature not in the continuous signal. Misrepresentations caused by sampling 10 are called aliasingg". Aliasing may be prevented by passing the continuous signal through a low-pass filter prior to sampling. This low-pass filter removes frequencies higher than (2s)', where s is the spacing between samples. Unfortunately, low-pass filtering blurs the image. Slightly under-sampling images may make sampled images appear sharper. 15 A signal which approximates the original, continuous signal can be produced by convolving image samples with a kernel. An image can be resampled by calculating values of the signal at new sample points using an interpolating kernel. Commonly used interpolating methods include bilinear and bicubic interpolation. Bilinear and bicubic interpolation are equivalent to separable convolution with tent kernel and cubic 20 convolution kernels respectively. Aliasing causes visible artefacts in the scaled images when images are resampled at a higher resolution. These are particularly visible along edges where they are sometimes referred to as "jaggies". 2453925_1 930061_speci_lodge -2 Many methods have been proposed to improve the sharpness and reduce the aliasing artefacts in resampled images. A number of these are based on modifying the interpolation process in the presence of edges. For example, the sharpness of an interpolating kernel may be varied in edge regions or a kernel may be selected that is 5 stretched or otherwise orientated to match the orientation of the edge. The least expensive of these methods involve selecting one of several kernels, but this often leads to visible artefacts in output images at the boundaries between regions where different kernels have been applied. More sophisticated methods involve modifying the characteristics of the kernel to 10 match the characteristics of the data. Unfortunately, the more sophisticated methods are often iterative and typically too complex to be implemented in hardware for consumer electronics, for example with a video display device, such as a TV. A number of methods are based on specific geometric relationships between input and output images and only provide scaling by a fixed factor in each scaling 15 direction. For example, many known methods only provide scaling by a factor of two (2) in each sampling axis and must be combined with other methods to achieve scaling by arbitrary scaling factors. Typical peeking circuits can be used to improve the sharpness of an edge by increasing the gradient. However, peeking circuits can create undershoots and/or 20 overshoots at the ends of the edge. Undershoots and overshoots cause aliasing in the output image. Therefore, there remains a need for an efficient image scaling method which reduces the jaggie artefacts when increasing the resolution of the image while retaining 2453925_1 930061_speci_lodge -3 or improving sharpness of the image. This method should be able to be adapted to a range of rescaling rates. SUMMARY In accordance with aspect of the present disclosure, there is provided a method 5 for processing an image comprising input samples located on a regularly spaced grid to produce an output image comprising output samples, the method comprising: estimating an isophote orientation based on a location of an output sample and locations of input samples close to the location of the output sample; selecting, based on the estimated isophote orientation, a directional sharpening 10 filter having a corresponding sharpening direction, where the output of the selected sharpening filter includes directional frequencies beyond the Nyquist limit determined by a sampling rate of the input samples in a direction of the selected sharpening filter; applying the selected sharpening filter to input samples located about the location of the output sample to produce a plurality of sharpened samples; 15 combining the sharpened samples to produce an output sample at the output sample location based on the location and the isophote orientation. Other aspects are also disclosed. BRIEF DESCRIPTION OF THE DRAWINGS At least one embodiment of the present invention will now be described with 20 reference to the following drawings, in which: Fig. 1 illustrates how a thin line is represented using samples in a digital image; Figs. 2A to 2D illustrate the difference between an original line image and an image of the line reconstructed from aliased samples using a cubic interpolating filter; Fig. 3 shows bicubic interpolation of input samples to generate output samples; 2453925_1 930061_specijlodge -4 Fig. 4 is a diagram illustrating how large angles between gradient vectors can be used to indicate isophote curvature; Fig. 5 illustrates the periodic nature of the aliasing artefacts along a resampled isophote when samples along the isophote are reconstructed using an interpolating filter; 5 Fig. 6 is a flowchart describing a method of scaling images to produce scaled images with enhanced detail; Fig. 7 is a schematic block diagram representation of a system for scaling images to produce scaled images with enhanced detail; Fig. 8 shows a schematic block diagram representation of FIR interpolating; 10 Fig. 9 shows an example of the 2D sample areas involved when calculating an interpolated value with reduced aliasing; Fig. 10 illustrates how auxiliary point samples are calculated using a sharpening filter; Fig. 11 shows how an adaptive filter can be used to sharpen an edge; 15 Figs. 12A to 12D illustrate the difference between an original line image, an image of the line reconstructed from aliased samples using a cubic interpolating filter and intermediate samples produced by an adaptive sharpening filter; Fig. 13 shows how an estimate of an original value of an isophote may be determined at a specified output point from two interpolated auxiliary samples; 20 Fig. 14 shows how an estimate of an original value of an isophote may be determined with primary (fundamental) and secondary aliasing harmonics removed at a specified output point from four interpolated auxiliary samples; Fig. 15 shows how an estimate of the original value of an isophote may be determined from interpolated points on the isophote; 2453925_1 930061_specilodge -5 Fig. 16 illustrates a blending factor to smooth the transition between oblique edges and approximately vertical or horizontal lines; Fig. 17A illustrates a conventional FIR interpolating filter kernel and its frequency response; 5 Fig. 17A illustrates a wideband FIR interpolating filter kernel and its frequency response; Fig. 18 shows the steps involved in combining sharpened samples to produce an anti-aliased sample; and Figs. 19A and 19B form a schematic block diagram of a general purpose 10 computer system upon which the arrangements described can be practiced. DETAILED DESCRIPTION INCLUDING BEST MODE Figs. 19A and 19B collectively form a schematic block diagram of a general purpose computer system 1900, upon which the various arrangements to be described can be practiced. 15 As seen in Fig. 19A, the computer system 1900 is formed by a computer module 1901, input devices such as a keyboard 1902, a mouse pointer device 1903, a scanner 1926, a camera 1927, both of which can form inputs for digital images, and a microphone 1980, and output devices including a printer 1915, a display device 1914 and loudspeakers 1917. An external Modulator-Demodulator (Modem) transceiver 20 device 1916 may be used by the computer module 1901 for communicating to and from a communications network 1920 via a connection 1921. The network 1920 may be a wide-area network (WAN), such as the Internet or a private WAN. Where the connection 1921 is a telephone line, the modem 1916 may be a traditional "dial-up" modem. Alternatively, where the connection 1921 is a high capacity (eg: cable) 2453925_1 930061_specitodge -6 connection, the modem 1916 may be a broadband modem. A wireless modem may also be used for wireless connection to the network 1920. The computer module 1901 typically includes at least one processor unit 1905, and a memory unit 1906 for example formed from semiconductor random access 5 memory (RAM) and semiconductor read only memory (ROM). The module 1901 also includes an number of input/output (1/0) interfaces including an audio-video interface 1907 that couples to the video display 1914, loudspeakers 1917 and microphone 1980, an I/O interface 1913 for the keyboard 1902, mouse 1903, scanner 1926, camera 1927 and optionally a joystick (not illustrated), and an 10 interface 1908 for the external modem 1916 and printer 1915. In some implementations, the modem 1916 may be incorporated within the computer module 1901, for example within the interface 1908. The computer module 1901 also has a local network interface 1911 which, via a connection 1923, permits coupling of the computer system 1900 to a local computer network 1922, known as a Local Area Network (LAN). As also 15 illustrated, the local network 1922 may also couple to the wide network 1920 via a connection 1924, which would typically include a so-called "firewall" device or device of similar functionality. The interface 1911 may be formed by an Ethernet circuit card, a BluetoothTM wireless arrangement or an IEEE 802.11 wireless arrangement. The interfaces 1908 and 1913 may afford either or both of serial and parallel 20 connectivity, the former typically being implemented according to the Universal Serial Bus (USB) standards and having corresponding USB connectors (not illustrated). Storage devices 1909 are provided and typically include a hard disk drive (HDD) 1910. Other storage devices such as a floppy disk drive and a magnetic tape drive (not illustrated) may also be used. An optical disk drive 1912 is typically provided to act as a 2453925_1 930061_speci_lodge -7 non-volatile source of data. Portable memory devices, such optical disks (eg: CD-ROM, DVD), USB-RAM, and floppy disks for example may then be used as appropriate sources of data to the system 1900. The components 1905 to 1913 of the computer module 1901 typically 5 communicate via an interconnected bus 1904 and in a manner which results in a conventional mode of operation of the computer system 1900 known to those in the relevant art. Examples of computers on which the described arrangements can be practised include IBM-PC's and compatibles, Sun Sparcstations, Apple MacTM or alike computer systems evolved therefrom. 10 The method of resampling a digital image may be implemented using the computer system 1900 wherein the processes of Figs. I to 18, to be described, may be implemented as one or more software application programs 1933 executable within the computer system 1900. In particular, the steps of the methods of resampling are effected by instructions 1931 in the software 1933 that are carried out within the computer system 15 1900. The software instructions 1931 may be formed as one or more code modules, each for performing one or more particular tasks. The software may also be divided into two separate parts, in which a first part and the corresponding code modules performs the resampling methods and a second part and the corresponding code modules manage a user interface between the first part and the user. 20 The software 1933 is generally loaded into the computer system 1900 from a computer readable medium, and is then typically stored in the HDD 1910, as illustrated in Fig. 19A, or the memory 1906, after which the software 1933 can be executed by the computer system 1900. In some instances, the application programs 1933 may be supplied to the user encoded on one or more CD-ROM 1925 and read via the 2453925_1 930061_specijlodge -8 corresponding drive 1912 prior to storage in the memory 1910 or 1906. Alternatively the software 1933 may be read by the computer system 1900 from the networks 1920 or 1922 or loaded into the computer system 1900 from other computer readable media. Computer readable storage media refers to any storage medium that participates in 5 providing instructions and/or data to the computer system 1900 for execution and/or processing. Examples of such storage media include floppy disks, magnetic tape, CD ROM, a hard disk drive, a ROM or integrated circuit, USB memory, a magneto-optical disk, or a computer readable card such as a PCMCIA card and the like, whether or not such devices are internal or external of the computer module 1901. Examples of 10 computer readable transmission media that may also participate in the provision of software, application programs, instructions and/or data to the computer module 1901 include radio or infra-red transmission channels as well as a network connection to another computer or networked device, and the Internet or Intranets including e-mail transmissions and information recorded on Websites and the like. 15 The second part of the application programs 1933 and the corresponding code modules mentioned above may be executed to implement one or more graphical user interfaces (GUIs) to be rendered or otherwise represented upon the display 1914. Through manipulation of typically the keyboard 1902 and the mouse 1903, a user of the computer system 1900 and the application may manipulate the interface in a functionally 20 adaptable manner to provide controlling commands and/or input to the applications associated with the GUI(s). Other forms of functionally adaptable user interfaces may also be implemented, such as an audio interface utilizing speech prompts output via the loudspeakers 1917 and user voice commands input via the microphone 1980. 2453925_1 930061 speci-lodge -9 Fig. 19B is a detailed schematic block diagram of the processor 1905 and a "memory" 1934. The memory 1934 represents a logical aggregation of all the memory devices (including the HDD 1910 and semiconductor memory 1906) that can be accessed by the computer module 1901 in Fig. 19A. 5 When the computer module 1901 is initially powered up, a power-on self-test (POST) program 1950 executes. The POST program 1950 is typically stored in a ROM 1949 of the semiconductor memory 1906. A program permanently stored in a hardware device such as the ROM 1949 is sometimes referred to as firmware. The POST program 1950 examines hardware within the computer module 1901 to ensure 10 proper functioning, and typically checks the processor 1905, the memory (1909, 1906), and a basic input-output systems software (BIOS) module 1951, also typically stored in the ROM 1949, for correct operation. Once the POST program 1950 has run successfully, the BIOS 1951 activates the hard disk drive 1910. Activation of the hard disk drive 1910 causes a bootstrap loader program 1952 that is resident on the hard disk 15 drive 1910 to execute via the processor 1905. This loads an operating system 1953 into the RAM memory 1906 upon which the operating system 1953 commences operation. The operating system 1953 is a system level application, executable by the processor 1905, to fulfil various high level functions, including processor management, memory management, device management, storage management, software application 20 interface, and generic user interface. The operating system 1953 manages the memory (1909, 1906) in order to ensure that each process or application running on the computer module 1901 has sufficient memory in which to execute without colliding with memory allocated to another process. Furthermore, the different types of memory available in the system 1900 must be used 2453925_1 930061_speci_lodge -10 properly so that each process can run effectively. Accordingly, the aggregated memory 1934 is not intended to illustrate how particular segments of memory are allocated (unless otherwise stated), but rather to provide a general view of the memory accessible by the computer system 1900 and how such is used. 5 The processor 1905 includes a number of functional modules including a control unit 1939, an arithmetic logic unit (ALU) 1940, and a local or internal memory 1948, sometimes called a cache memory. The cache memory 1948 typically includes a number of storage registers 1944 - 1946 in a register section. One or more internal buses 1941 functionally interconnect these functional modules. The processor 1905 typically also 10 has one or more interfaces 1942 for communicating with external devices via the system bus 1904, using a connection 1918. The application program 1933 includes a sequence of instructions 1931 that may include conditional branch and loop instructions. The program 1933 may also include data 1932 which is used in execution of the program 1933. The instructions 1931 and 15 the data 1932 are stored in memory locations 1928-1930 and 1935-1937 respectively. Depending upon the relative size of the instructions 1931 and the memory locations 1928-1930, a particular instruction may be stored in a single memory location as depicted by the instruction shown in the memory location 1930. Alternately, an instruction may be segmented into a number of parts each of which is stored in a separate 20 memory location, as depicted by the instruction segments shown in the memory locations 1928-1929. In general, the processor 1905 is given a set of instructions which are executed therein. The processor 1905 then waits for a subsequent input, to which it reacts to by executing another set of instructions. Each input may be provided from one or more of a 2453925_1 930061_specijlodge -11 number of sources, including data generated by one or more of the input devices 1902, 1903, data received from an external source across one of the networks 1920, 1922, data retrieved from one of the storage devices 1906, 1909 or data retrieved from a storage medium 1925 inserted into the corresponding reader 1912. The 5 execution of a set of the instructions may in some cases result in output of data. Execution may also involve storing data or variables to the memory 1934. The disclosed digital image resampling processing arrangements use input variables 1954, that are stored in the memory 1934 in corresponding memory locations 1955-1958. The processing arrangements produce output variables 1961, that 10 are stored in the memory 1934 in corresponding memory locations 1962-1965. Intermediate variables may be stored in memory locations 1959, 1960, 1966 and 1967. The register section 1944-1946, the arithmetic logic unit (ALU) 1940, and the control unit 1939 of the processor 1905 work together to perform sequences of micro operations needed to perform "fetch, decode, and execute" cycles for every instruction in 15 the instruction set making up the program 1933. Each fetch, decode, and execute cycle comprises: (a) a fetch operation, which fetches or reads an instruction 1931 from a memory location 1928; (b) a decode operation in which the control unit 1939 determines which 20 instruction has been fetched; and (c) an execute operation in which the control unit 1939 and/or the ALU 1940 execute the instruction. 2453925_1 930061 speciJodge -12 Thereafter, a further fetch, decode, and execute cycle for the next instruction may be executed. Similarly, a store cycle may be performed by which the control unit 1939 stores or writes a value to a memory location 1932. Each step or sub-process in the processes of Figs. 1 to 18 is associated with one 5 or more segments of the program 1933, and is performed by the register section 1944 1947, the ALU 1940, and the control unit 1939 in the processor 1905 working together to perform the fetch, decode, and execute cycles for every instruction in the instruction set for the noted segments of the program 1933. The method of image resampling and processing may alternatively be 10 implemented in dedicated hardware such as one or more integrated circuits performing the functions or sub functions of resampling. Such dedicated hardware may include graphic processors, digital signal processors, or one or more microprocessors and associated memories. Such further may be configured as one or more embedded devices with a specific product, such as a digital camera. 15 Fig. I shows an input image 1010 depicting dark edges 1030 of lines and curves on a light background 1020. An "isophote" is hereafter defined as a line where intensity in the image is constant. A section 1040 of the input image 1010 is enlarged 1050 to show several rows 1060, 1070, 1080 and 1090 of samples 1110 and 1120 near an edge 1100, being one of the edges 1030. Samples 1120 on the edges 1110 and 1030 have low 20 intensity. Samples 1110 not on the edges have a high intensity. Fig. I shows intensity plots 1130, 1160, 1170 and 1180 on for each of the rows 1060, 1070, 1080 and 1090 respectively. Intensity plot 1130 shows an ideal intensity curve 1131 and the spot intensities, 1140 and 1150, at the samples 1110 and 1120 respectively. 2453925_1 930061_specilodge - 13 Interpolation is required to resample a signal so as to increase the resolution of the image 1010. Intensity plots 1130, 1160, 1170 and 1180 in Fig. I are reproduced as intensity plots 2010, 2040, 2070 and 2100 respectively in Figs. 2A to 2D. The reproduced intensity plots 2010, 2040, 2070 and 2100 show the ideal intensities 2020, 5 2050, 2080 and 2110 and cubic interpolated intensities 2030, 2060, 2090 and 2120. The cubic interpolation 2030 was calculated using the intensities of samples 2011, 2012, 2013 and 2014 around the dark edge 1110. The cubic interpolated intensities 2030, 2060, 2090 and 2120 may result in overshoots. An overshoot is shown between the intensity of samples 2011 and 2012 for the intensity plot 2010. 10 Fig. 3 shows a cubic interpolation. An input signal 3000 is part of the input image 1010. The input signal 3000 is interpolated to generate an output signal 3020 which forms part of the output (resampled) image. A set of input samples 3001, 3002, 3003 and 3004 from the input signal 3000 are combined using a weighted sum to generate a single output sample 3030 of the output signal 3020. In Fig. 3, there are 5 15 input samples in the input signal 3000. The output signal 3020 has 13 output samples. The resolution has been increased. The weights 3011, 3012, 3013 and 3014, hereafter referred to as "kernel coefficients", are applied to the input samples 3001, 3002, 3003 and 3004 respectively. The kernel coefficients are determined from a cubic kernel 3010 centered at the position of the output sample 3030. The cubic kernel 3010 is evaluated at 20 points 3011, 3012, 3013 and 3014 corresponding to the relative positions of each input sample 3001, 3002, 3003 and 3004 respectively. Since different kernel coefficients are applied to produce different output samples, depending on the relative offset (or phase) between the input samples and the output sample to be produced, such a kernel is 2453925_1 930061_specijlodge -14 referred to as a polyphase kernel. The interpolation described in Fig. 3 may be referred to as convolution with a polyphase kernel. A standard cubic kernel is described by Equation (1): (a+2)x -(a+3)|x1 2 +1 , 0!IxI<I C(x,a)= axI 3 -5alx 2 +8ajxl-4a , 1s;jx<2, (1) 0 , 2 !x| where x is the offset from the center point of the cubic kernel, which ranges from 5 negative two (-2) to positive two (+2) inclusive, and a is a constant which is typically selected to be negative one half (-0.5). The cubic interpolation can be performed in two or more dimensions by separable application to each dimension. A modified cubic kernel may be defined and used to interpolate the intensity at the point, x. The modified cubic kernel implements a low-pass filter which has a lower 10 cut-off frequency than a standard cubic kernel. The modified cubic kernel is used to remove aliasing and noise when calculating the intensity gradient. The modified cubic kernel is described by Equation (2): 3 2 x xII (a +±2) - _(a±+3)H-+1 , 0 -Ixl<2 2 2 K~x~)= a -5a 8ajj-4a ,2 Ix1l<4,(2 222 0 , 45|x| where x is the offset from the center point of the modified cubic kernel and a is a constant which is typically selected to be negative one half (-0.5). 15 A derivative kernel can be used to determine the rate of change of the intensity. Equation (3) is the derivative of the modified cubic kernel described in Equation (2): 2453925_1 930061_speci lodge - 15 2 3(a+2) ] -2(a+3)i , 0:5x|<2 1 x~x2 K'(x, a) =-sign(x)- 3a- -10a +8a , 2 1 jx|< 4, (3) 4 221 0 , 4 |x| where x is the offset from the center point of the derivative of the modified cubic kernel and a is a constant which is typically selected to be negative one half (-0.5). Equation (2) and Equation (3) can be used together to calculate partial derivatives of the intensity at the output resolution. 5 A continuous image may be treated as a continuous, real valued function I(x, y) of two variables. The function I(x, y) may be referred to as an intensity function. Such an intensity function is assumed to be piecewise differentiable. In any optical system, a real world image has some degree of blurring. This blurring is equivalent to smoothing the image intensity. 10 The intensity gradient vector, G, always points in the direction of the maximum increase in intensity. The maximum increase in intensity is always perpendicular to lines of constant intensity, such as an isophote 5010 as shown in Fig. 5. In Fig. 5, a gradient vector 5030 is also shown for the point 5022. The gradient vector 5030 is defined by Equation (4): G = (g.,gy,=,) (4) ax 0y 15 where g, is the horizontal gradient vector component and gy is the vertical gradient vector component. The gradient vector is perpendicular to the isophotes, and thus the gradient vector indicates the isophote orientation. 2453925_1 930061.speciJodge - 16 Isophote curvature can be measured by calculating the difference between the isophote orientations at two points on the isophote. Fig. 4 illustrates how the angle between gradient vectors can be used to indicate isophote curvature. Fig. 4 shows an object 4010 with three sample points 4021, 4022 and 4023. Gradient vectors 4031, 4032 5 and 4033 are calculated at each sample point 4021, 4022 and 4023 respectively. A first angle 4041 between isophotes can be calculated from the first two gradient vectors 4031 and 4032. A second angle 4042 between isophotes can be calculated from the second two gradient vectors 4032 and 4033. The first angle 4041 between isophotes is small suggesting the first two sample points 4021 and 4022 are on the same isophote. The 10 second angle 4042 between isophotes is large, indicating the second two sample points 4022 and 4023 are not on the same isophote. The second angle 4042 between isophotes can be used to predict undesirable features such as rapidly changing orientations (large curvature) or a vertex, such as the vertex 4050, occurs between the second two samples 4022 and 4023. 15 An estimate of isophote curvature may be used to determine if an isophote has non-negligible curvature. The measure of isophote curvature, Ca,b, can be estimated using Equation (5) when the magnitude of the gradient vector at two points, a and b, are approximately equal and their sum is non-zero: Cab = Gsinf| JIG+ G b (5) K2. JIG all + JIG bl where Ga and Gb are two gradient vectors, 6, is the angle between the two gradient 20 vectors and ||Gi| is the magnitude of the gradient vector G = (g, gy) which is determined as ||G| = jg + gY 2 . Where the estimate of isophote curvature is low, the reliability of the isophote orientation calculation may be considered high. Conversely, 2453925_1 930061_speci-lodge - 17 where the estimate of isophote curvature is high, the reliability of the isophote orientation calculation may be considered low. Other methods of estimating the curvature of the isophotes may be used. Examples of such other methods include traditional curvature formulae from differential geometry, or based on measuring the 5 variation in the gradient square tensor over the neighbourhood of the output sample, to name but a few. When an image containing an orientated high frequency feature (e.g. such as an edge of the triangular object 4010) is sampled, aliasing appears as a repeated stepping pattern along the feature. Interpolating such an aliased image to reconstruct a higher 10 resolution image produces a periodic variation of intensity in the reconstructed image along isophotes of the ideal image, where the fundamental harmonic of the aliasing is the most visible. It can therefore be assumed that the error due to aliasing along an isophote is a sinusoidal function of the vertical or horizontal position of the interpolated sample. Fig. 5 shows a sampled image 5000 of a thin edge with pixels 5041, 5042, 5043, 15 5044, 5045, 5046 and 5047 representing input samples taken close to the edge. The intensity of the sampled image 5000, varies periodically about the actual intensity of the isophote 5010. The intensity plot 5070 shows the intensity profile 5071 along the isophote 5010. Two points 5021 and 5022 on the isophote 5010 are shown as the points 5081 and 5082 in the intensity plot 5070. As shown in Fig. 5, the points 5081 and 5082 20 have equal intensity and are separated by a distance along the isophote 5010 equal to an aliasing period 5090. The aliasing period vector, D = (d, d,), is a period vector of the aliasing period 5090. The aliasing period vector has the component d, 5050 in the vertical direction and the component d, 5060 in the horizontal direction. The aliasing period 5090 is the period of intensity variation along the isophote 5010, where the 2453925_1 930061_speciJodge - 18 intensity variation results from aliasing. By determining the aliasing period vector, D, the variation in intensity 5071 can be reduced to estimate the ideal intensity 5100. Isophotes with orientations deviating from horizontal by less than 45 degrees (i.e. when abs(g,) > abs(g,)), (abs = absolute value) and are hereafter referred to as 5 "predominantly horizontal", noting that the gradient vector is perpendicular to the isophote. For predominantly horizontal isophotes, the aliasing period vector has a vertical component equal to the vertical sample spacing and a horizontal component determined by the isophote orientation. Equation (6) can be used to determine the aliasing period vector for predominantly horizontal isophotes. D = (d., d,)= ~- ,y1 ,(6) ((6) 10 where g, is the horizontal gradient vector component, gy is the vertical gradient vector component, dx is the horizontal aliasing period vector component and d, is the vertical aliasing period vector component. Isophotes with orientations deviating from vertical by less than 45 degrees (i.e. when abs(g) > abs(gy)), and are hereafter referred to as "predominantly vertical", noting 15 that the gradient vector is perpendicular to the isophote. For predominantly vertical isophotes, the aliasing period vector has a horizontal component equal to the horizontal sample spacing and a vertical component determined by the isophote orientation. Equation (7) can be used to determine the aliasing period vector for predominantly vertical isophotes. D=(d,,d,)= 1,- (7) g2 2453925 1 930061_speci_lodge -19 where gx is the horizontal gradient vector component, gy is the vertical gradient vector component, d, is the horizontal aliasing period vector component and d, is the vertical aliasing period vector component. Isophote orientations where gx equals gy correspond to an angle of 45 degrees 5 from vertical or horizontal. In this case, the aliasing period vector produced by Equation (6) is (-1, 1) and the aliasing period produced by Equation (7) is (1, -1). The aliasing period vectors (-1, 1) and (1, -1) are equivalent because the aliasing period is the same. For isophote orientations where gx equals -gy, both Equation (6) and Equation (7) produce an aliasing period vector of (1, 1). Equation (6) and Equation (7) are valid for 10 any isophote orientation except exactly horizontal or exactly vertical. The magnitude of the aliasing period vector approaches infinity for exactly horizontal and exactly vertical isophote orientations. The system of the present disclosure improves the quality of the interpolated image near edges compared to techniques such as bicubic interpolation. Adaptive 15 sharpening of edges is performed to produce a better match to the original, ideal image. Fig. 6 illustrates an adaptive edge sharpening method 6000, indicating the steps involved in producing an output sample of an image with improved sharpness. The adaptive edge sharpening method 6000 can be implemented in software as the application 1933 executable in the computer 1901 by the processor 1905 and operating upon digital image 20 data stored in the HDD 1910 or sourced from the camera 1927, scanner 1926 or the networks 1920, 1922, for example. The method 6000 calculates output samples for an output image with higher resolution than the input image; the output image having sharper edges than an image obtained without using the adaptive edge sharpening method 6000. 2453925_1 930061_speci_lodge - 20 Isophotes that pass through the position of the output sample in the output image are hereafter referred to as a "local isophote". The adaptive edge sharpening method 6000 begins by the processor 1905 estimating the isophote orientation of a local isophote in a local isophote orientation process 6010. In the process 6010, determinations such as 5 those described above with respect to Fig. 5 and gradient vectors are performed, as will be described further. The adaptive edge sharpening method 6000 continues with an isophote orientation reliability estimation determination process 6020. In the isophote orientation process 6020, the processor 1905 estimates the reliability of the isophote orientation 10 determined in the local isophote orientation process 6010. Low reliability of the isophote orientation occurs when there are isophotes of very different orientations near the position of the output sample. Additionally, low reliability occurs when the image is constant near the output sample position (as the local isophotes have no well-defined orientation), when the isophotes have high curvature, or when the image has a complex 15 texture near the position of the output sample (causing rapid variation in isophote direction). Low reliability can be assessed by comparing the estimate against a predetermined threshold. Other methods may be employed to estimate the reliability or consistency of the isophote orientations. The adaptive sharpening of edges method 6000 continues to a selection based on 20 reliable isophote orientation process 6030. The selection based on reliable isophote orientation process 6030 uses the reliability of the isophote orientation estimated by the isophote orientation reliability determination process 6020 to determine if the estimate is unreliable and, as such, which process is used to generate the output sample at the output resolution. A default interpolated sample process 6090 is selected by the processor 1905 2453925_1 930061_speci-lodge -21 when the isophote orientation has low reliability. The default interpolated sample process 6090 uses input samples to generate a default output sample, at the required output sample location. Step 6090 may be performed using simple interpolation. Step 6040 checks whether the isophote orientation has high reliability thus permitting 5 selection of a directional sharpening filter. The sharpening directions are preferably one of horizontal or vertical, and such provides for selecting one of a vertical sharpening filter in step 6050, or a horizontal sharpening filter in step 6060. Step 6040 thus represents a filter selection process. If, as tested in step 6040, the isophote is predominantly horizontal, a vertical sharpening filter, having a vertical sharpening 10 direction, is selected in step 6050. If the isophote is predominantly vertical, a horizontal sharpening filter, having a horizontal sharpening direction, is selected in step 6060. The selected filter is used by a sharpening filter process 6070 to generate sharpened samples. The selected filter produces an output having directional frequencies beyond the Nyquist limit determined by a sampling rate of the input samples in a direction of the selected 15 sharpening filter. In one implementation, a bicubic interpolating filter may be used to calculate the default sample. Other interpolation means may be used to calculate the default output sample, examples of which include linear interpolation, and Lanczos filters, to name but a few. 20 In another implementation, the default output sample is calculated in step 6090 as a combination of at least one vertically sharpened sample and at least one horizontally sharpened sample using the same vertical and horizontal sharpening filters as used to calculate the sharpened samples. This has the advantage of reducing the visible change between default output samples and anti-aliased output samples. Since the output of the 2453925_1 930061_specilodge -22 vertical or horizontal sharpening filters may not coincide with the output sample location, interpolation of the output of the sharpening filters may be required to produce a point at the output sample location. The process of producing sharpened samples is described in more detail below with reference to Fig. 10, Fig. 11 and Fig. 12. 5 The isophote orientation determined from the local isophote orientation process 6010 is used by the processor 1905 in the checking step 6040, which determines filter selection, to select either the vertical sharpening filter process 6050 if the isophote orientation is predominantly horizontal, or a horizontal sharpening filter process 6060 if the isophote orientation is predominantly vertical. The sharpening filter process 6070 10 uses the selected sharpening filter to generate sharpened samples, near the output sampling point, from the input samples. Details of the sharpening filter are described below with reference to Fig. 10, Fig. 11 and Fig. 12. A combine sharpened samples process 6080 then operates to combine the sharpened samples to produce one anti-aliased sample from the sharpened samples. The 15 steps involved in producing an anti-aliased sample are illustrated in Fig. 18. Sharpened samples calculated in the sharpening step 6070 are interpolated by the processor 1905 in a step 18010 as seen in Fig. 18 to produce auxiliary samples, which may be stored in the memory 1906. The auxiliary samples are combined together by the processor 1905 in step 18020 to produce an anti-aliased sample for outputting. Examples of two 20 alternative methods of calculating auxiliary samples are described in more detail below with reference to Fig. 9 and Fig. 15. In one implementation, the anti-aliased sample is output as the output sample. The output sample may be stored in the memory 1906, 1910, displayed on the display device 1914 or printed by the printer 1915, for example. 2453925_1 930061_speciJodge -23 Fig. 7 schematically illustrates an apparatus 7000 which can perform the method illustrated in Fig. 6. The output samples 7110 are calculated from a stream of input samples 7010. An output image is formed from output samples 7110. The output image has an increased resolution and sharper edges than the input image and may be stored, 5 displayed, printed or further processed as required. Input samples 7010 from an image 1010 are input to each of a vertical sharpening filter block 7020, a horizontal sharpening filter block 7030, a local isophote orientation estimation block 7040, a default interpolating filter block 7060 and a block 7070 to determine the isophote orientation reliability. In another implementation, orientation 10 estimation and orientation reliability are calculated together by the same block. The vertical sharpening filter block 7020 incorporates a vertical sharpening filter that increases the resolution in the vertical sampling direction through interpolation. Vertically sharpened interpolated samples 7021 are generated by the vertical sharpening filter block 7020 from the input samples 7010. The vertical sharpening filter block 7020 15 performs the sharpening step 6070 depicted in Fig. 6 in the case where the vertical filter is selected. The horizontal sharpening filter block 7030 incorporates a horizontal sharpening filter that increases the resolution in the horizontal sampling direction through interpolation. Horizontally sharpened interpolated samples 7031 are generated by the 20 horizontal sharpening filter block 7030 from the input samples 7010. The horizontal sharpening filter block 7030 implements the horizontal sharpening step 6070 depicted in Fig. 6 in the case where the horizontal filter is selected. The local isophote orientation block 7040 performs the local isophote orientation process 6010. The local isophote orientation block 7040 estimates the local isophote 2453925_1 930061_speci_lodge -24 orientation based on the input samples 7010. The local isophote orientation block 7040 outputs an isophote orientation signal 7041. The sharpening filter selection block 7050 selects either horizontally sharpened samples or vertically sharpened samples based on an isophote orientation signal output 5 7041 of the local isophote orientation block 7040. The sharpening filter selection block 7050 implements the selection step 6040 in Fig. 6. The sharpening filter selection block 7050 uses the isophote orientation signal 7041 to output the sharpened samples 7051, either from the vertically sharpened interpolated samples 7021 or the horizontally sharpened interpolated samples 7031. 10 The default interpolating filter block 7060 calculates default interpolated samples using a default interpolating filter. The default interpolating filter block 7060 generates default interpolated samples 7061 at output sample locations, from the input samples 7010. The determine reliability of the isophote orientation block 7070 performs the 15 determine reliability of the isophote orientation process 6020. The determine reliability of the isophote orientation block 7070 uses the input samples 7010 to output a reliability of the isophote orientation signal 7071. A combine block 7080 performs the combine sharpened samples process 6080. The combine block 7080 outputs anti-aliased samples 7090, representing samples at the 20 output sample locations. A blend block 7100 produces the output samples 7110 by blending the default interpolated samples 7061 and anti-aliased samples 7090 together. The blending operation is dependant on the reliability of the isophote orientation signal 7071. In one implementation, if the reliability is low (eg. below a predetermined limit), the blend 2453925_1 930061_specilodge -25 7100 results in the default interpolated sample 7061 being selected as the output sample 7110. If the reliability is high, the anti-aliased sample 7090 is selected as the output sample 7110. In another implementation which makes used of the blending function, if the reliability is low the output sample 7110 is predominantly the default interpolated 5 sample 7061 as a consequence of the blend, and if the reliability is high the output sample 7110 is predominantly the anti-aliased sample 7090 as a consequence of the blend. If the reliability is intermediate, an even blend of the two is used. This results in a more gradual, smoother transition which reduces visible artefacts at the point of switching. In another arrangement, the blending between the default output sample 7061 10 and the anti-aliased output sample 7090 also depends on the estimated isophote orientation 7041 the calculation of the blend amount is described below with reference to Fig. 16. Fig. 8 shows an example of a one dimensional finite impulse response (FIR) filter apparatus 8000. Input samples 8010 are delayed by delay blocks 8050, 8060, 8070 and 15 8080. An input clock 8020 causes each of a number of delay blocks 8050, 8060, 8070 and 8080 to pass delayed input samples 8010, 8051, 8061 and 8071 respectively to form delayed output samples 8051, 8061, 8071 and 8081 respectively. The delay blocks 8050, 8060, 8070 and 8080 are cascaded so the preceding delay blocks 8050, 8060 and 8070 output samples 8051, 8061 and 8071 respectively are the input samples for the next 20 delay blocks 8060, 8070 and 8080 respectively. Each delay block 8050, 8060, 8070 and 8080 outputs samples 8051, 8061, 8071 and 8081 respectively and are also each connected to a multiplier 8090, 8100, 8110 and 8120 respectively. The example shown in Fig. 8 is representative of a four tap FIR filter. More taps or fewer taps may be used depending on the desired trade-off between accuracy and cost. 2453925_1 930061_specilodge -26 A kernel coefficient generator 8040 outputs kernel coefficients 8041, 8042, 8043 and 8044 in synchronism with an output clock 8030. The output clock 8030 maybe the same or a multiple of the input clock 8020. The output clock 8030 is synchronised with the input clock 8020. The kernel coefficients 8041, 8042, 8043 and 8044 are each 5 connected to a corresponding multiplier 8090, 8100, 8110 and 8120. Each multiplier 8090, 8100, 8110 and 8120 produces a weighted sample output 8091, 8101, 8111 and 8121 respectively from the kernel coefficients 8041, 8042, 8043 and 8044 respectively and delayed input samples 8051, 8061, 8071 and 8081 respectively. The weighted sample outputs 8091, 8101, 8111 and 8121 are summed 10 8130 to produce the output sample 8140. In one implementation, the kernel coefficient generator 8040 stores several sets of kernel coefficients 8041, 8042, 8043 and 8044. Each set of kernel coefficient 8041, 8042, 8043 and 8044 represent a different FIR interpolating filter. A select signal 8032, is used to specify the required set of kernel coefficients 8041, 8042, 8043 and 8044. The 15 select signal 8032 may alternatively may be implemented using the clock 8030. In this implementation, a single FIR interpolating filter apparatus can be used to perform one of several different FIR interpolating filters operations. By dynamically generating the coefficients or retrieving the coefficients from a table based on the offset (phase), polyphase kernels can be implemented using the kernel coefficient generator 8040. 20 The delays 8050, 8060, 8070, 8080, in Fig. 8 represent single sample delays. When processing a stream of video in raster scan order, such a filter can be used to implement a horizontal one dimensional filter. A similar arrangement can be used to implement a vertical one dimensional filter by replacing the single sample delays 8050, 8060, 8070; 8080 with delays of one raster line of video. Other arrangements for 2453925_1 930061_speciJodge -27 implementing horizontal and vertical filters may be employed in alternative implementations. In one implementation, a suitably configured finite impulse response (FIR) interpolating filter, such as bicubic, is used as the default interpolating filter 7060. 5 Similarly, in an alternative implementation, FIR filters, are used for the vertical sharpening filter 7020 and the horizontal sharpening filter 7030. An example of how to design a wideband filter that passes frequencies beyond the Nyquist limit, that may be used as a sharpening filter according to a specific implementation is explained below with reference to Fig. 17A and Fig.17B. 10 In one implementation, the isophote orientation, represented by the gradient (gx, gy) 5030, is estimated using the modified cubic kernel defined in Equation (2) and its derivative kernel defined in Equation (3). The horizontal gradient component, gx, is estimated for the isophote 5010 by applying the stretched cubic kernel K(x, -0.5) in the vertical direction in accordance with Equation (2) and the derivative kernel K'(x, -0.5) in 15 the horizontal direction in accordance with Equation (3). The vertical gradient component, gy, is estimated for the isophote 5010 by applying the stretched cubic kernel K(x, -0.5) in the horizontal direction in accordance with Equation (2) and the derivative kernel K'(x, -0.5) in the vertical direction in accordance with Equation (3). In other implementations, different techniques and kernels may also be used to 20 estimate an isophote orientation (gx, gy) 5030. Whilst a range of methods may be used to determine the gradient, these only generate results at the same resolution as the input. Such orientation results can be interpolated, using methods such as bilinear or bicubic interpolation, to produce the isophote orientation signal 7041 at the output resolution. 2453925_1 930061_speci-lodge - 28 In one implementation, the output of the blend block 7100 depends on both the isophote orientation signal 7041 and the reliability 7071. As the local isophote orientation approaches horizontal or vertical then the length of the aliasing period vector, D, as defined in Equations (7) and Equation (6) becomes infinite. Thus, the size of the 5 region of input samples 7010 required to generate the output sample 7090 increases as the orientation approaches vertical or horizontal. To avoid this problem, an upper limit on the required number of samples 7010 can be applied by controlling a threshold for classifying orientations as approximately vertical or approximately horizontal 16010 16040. 10 The output of the combine block 7080 or the combine process 6080 may be inaccurate when the data to be interpolated contains; curved edges, complex structures or textures. For example, if an isophote 5010 is highly curved, the isophote orientation accuracy is limited because the rate of change in orientation is large which significantly increases errors in calculations that depend on the isophote orientation estimation. When 15 complex structures and curved edge cases are encountered, it is advantageous to revert to a standard, non-adaptive interpolation method such as bicubic filtering. One example of this type of implementation is to use the curvature blend amount, aCa.b, described later in Equation (15), to determine if the isophote curvature is too large (aca,b equals zero). In this case the default interpolating filter block 7060 is used to 20 generate the output sample. In some implementations, the vertical sharpening filter 7020 and horizontal sharpening filter 7030 may be adaptive sharpening filters. Such filters are non-linear filters whose output includes frequencies beyond the Nyquist limits defined by the vertical and horizontal spacing between the input samples. By including high frequency 2453925_1 930061_specijlodge - 29 detail present in aliased frequencies, a sharper edge is produced; however, aliasing may also be increased. An anti-aliasing filter is used to reduce aliasing in the increased resolution output of the adaptive sharpening filters. Fig. 9 illustrates how the anti-aliased samples are produced from the sharpened 5 samples according to one implementation. Fig. 9 shows input samples 9000 in a neighbourhood of an output sample location 9060. The samples 9000 are seen to form a regularly spaced grid, for example pixel locations in an imaging device. The estimation of an isophote orientation along a local isophote 9030 and the corresponding aliasing period vector, D, at an output sample position 9060 are determined from the input 10 samples 9000 contained in a neighbourhood of the output sample position 9060. The two components of the aliasing period vector, are used to determine locations of auxiliary points, P 1 9010 and P 2 9020, on the local isophote 9030 which passes through the output position 9060. The isophote illustrated in Fig. 9 is predominantly horizontal, and thus the sharpening direction is vertical. As such, the 15 auxiliary points are located so that the vertical distance or separation 9040 between the output point 9060 and each of the auxiliary points 9010 and 9020 is one quarter of the vertical spacing between the input samples 9000. The sharpened samples are interpolated at each of the auxiliary points, P, 9010 and P 2 9020, based on support windows 9070 and 9080 respectively using a bicubic kernel. The support windows 9070 20 and 9080 are centered at the auxiliary points P; 9010 and P2 9020 respectively. The output sample 9060 is determined by the average of the sharpened samples interpolated at the auxiliary points P, 9010 and P 2 9020. The specific methods described herein are operative when the auxiliary points do not coincide with the input samples. However, in some instances, one of the auxiliary points may coincide with input samples. In one 2453925_1 930061_specilodge -30 implementation used to determine the auxiliary points at least one of the auxiliary points does not coincide with any input sample as the auxiliary points are spaced half a pixel apart in either the vertical or horizontal direction. In a preferred implementation the auxiliary points are a quarter of a pixel either side of output pixel, making for half a pixel 5 spacing. Whilst one auxiliary point may be on the grid, another auxiliary point associated with the same output point may not. To facilitate the anti-aliasing calculation illustrated in Fig. 9, in one implementation, aliasing reduction is performed. The horizontal and vertical sharpening filters 7030 and 7020, each produce two parallel streams of output (the second stream 10 being illustrated in phantom in Fig. 7). The two parallel streams 7031 of horizontally sharpened samples are produced, one at a horizontal resolution equal to the horizontal output resolution and the other at a vertical resolution equal to the vertical resolution of the input. The streams 7031 are horizontally offset from the columns of output sample positions at a distance of plus and minus one quarter of the horizontal spacing between 15 the input samples. The offset is preferably determined using filter coefficients having a different phase. Similarly, the two parallel streams 7021 of vertically sharpened samples are produced, one at a vertical resolution equal to the vertical output resolution and the other at a horizontal resolution equal to the horizontal resolution of the input. The streams 7021 are vertically offset from the rows of output sample positions at a distance 20 of plus and minus one quarter of the horizontal spacing between the input samples. Fig. 10 illustrates how interpolated vertically sharpened samples corresponding to arbitrary spatial locations may be produced according to one implementation. Input samples 10000 are considered to be contained in the support window (essentially equivalent to those shown in Fig. 9). Samples 10010 are output from the vertical 2453925_1 930061_speci-lodge -31 sharpening filter. A vertically sharpened sample 10030 is produced by applying a one dimensional vertical sharpening filter to a set of four (adjacent) samples 10010 bounded by a moving window 10020. An auxiliary sample 10050 is produced by applying a cubic filter to a set of four vertically sharpened samples 10030 bounded by a moving window 5 10040. Similar methods may be used to produce interpolated horizontally sharpened samples by applying a one dimensional horizontal sharpening filter and a vertical interpolating filter. The order of application of the one dimensional sharpening and one dimensional interpolating filters may be interchanged as may be required in certain implementations. 10 Many known sharpening methods operate by boosting high frequencies present in the input signal but do not introduce any new frequencies into the output that are not present in the input. This has the effect of making the image appear sharper but it does not improve the actual level of detail present in the image. An advantage of the presently disclosed arrangements is that frequencies beyond the Nyquist limit defined by 15 the input sample spacing are present in the output of the sharpening filters. This is done by introducing extra aliasing across edge signals. The undesirable aliasing artifacts introduced will be cancelled during the combine processes 6080, 7080. The extra aliasing is introduced by either applying a wideband filter as described with reference to Fig. 17 below in a direction across the edge, or by applying a non-linear sharpening filter 20 as illustrated in Fig. 11 across the edge. To represent such high frequencies, a higher sampling rate, i.e. a higher resolution than the input resolution must be used. Therefore, the output from the sharpening filters is produced at a higher resolution than that of the input. The output from such a filter is not useful on its own because of the significant amount of aliasing present, but when multiple outputs from such a sharpening filter are 2453925_1 930061.speciJodge - 32 combined 6080, 7080, the aliasing can be reduced or removed, and improved detail can be achieved. An advantage of the present adaptive sharpening method is that information present in the input signal in the form of aliased frequencies can be restored to produce an output image having increased detail. 5 Fig. 17A and Fig. 17B illustrate how a conventional FIR interpolating filter kernel with a cutoff frequency equal to the Nyquist limit may be modified to produce a wideband filter with a cuttoff frequency equal to twice the Nyquist limit. Fig. 17A shows a conventional FIR interpolating kernel 17010 and the magnitude of its spectrum 17020. Fig. 17B shows a modified kernel 17020 that is the same general shape, but 10 formed with half of the width and twice the amplitude as that of Fig. 17A. Such a filter will have a spectrum 17030 that is the same shape except it is twice the width as the spectrum 17020. In Fig. 17B, the one dimensional filter in the sharpening direction has impulse response equal to zero if the absolute value of x is greater than one half, and equal to two minus the absolute value of four times x, if the absolute value of x is less 15 than one half. In this fashion, cubic or linear interpolation kernels may be modified and used to produce small polyphase kernels that are efficient for hardware implementation of the sharpening filters of Figs. 17A and 17B. Fig. 11 illustrates an example of a one-dimensional adaptive (non-linear) sharpening filter according to one implementation, that may be used to perform the 20 sharpening step 6070 to produce sharpened samples 7031 or 7021. Fig. 11 particularly shows a plot of intensity (i.e. magnitude of samples), in the vertical direction versus sample position in the horizontal direction. Input samples 11011, 11012 and 11013 from an original ideal signal 11040 are processed to generate intermediate samples 11021, 11022, 11023, 11024, 11025 and 11026. The intermediate samples 11021, 11022, 2453925_1 930061_specilodge - 33 11023, 11024, 11025 and 11026 can be further interpolated using a conventional interpolation filter, such as a cubic filter, to produce a sharpened signal 11060 that approximates the original ideal signal 11040. The sharpened signal may be sampled at output resolution to produce output samples 11031, 11032 and, 11033. 5 The one-dimensional adaptive (non-linear) sharpening filter operates on a window of three input samples 11011, 11012 and 11013 to produce two intermediate samples 11023 and 11024. The intermediate samples are offset from the input samples by a distance equal to a quarter 11083 of the input sample spacing 11081. The distance 11082 between any two adjacent intermediate samples is half of the spacing between the 10 input samples, so the intermediate samples form a signal with twice the resolution of the input samples. The values of the intermediate samples are s - 3 and s + 6 where s is the value of the center input sample 11012 of the window and 5 11073 is calculated according to Equation (8) b.909S~ , sign(6)= sign(S,) (8) .. =2(60 +8 (8) 0 , sign(S 0 )!=sign(S,) 15 where 6o 11071, is the difference in intensity between the previous input sample 11011 and the current input sample 11, 6i 11072 is the difference in intensity between the current (center) input sample 11012 and the following input sample 11013, and b is a sharpness parameter which is used to control the degree of sharpening. If b is chosen to 20 be greater than 1.0, e.g. b = 3.0, the intermediate samples will form a signal with steeper edge transitions than the signal formed by the input samples without introducing undesirable overshoots This is described below with reference to Fig. 12. 2453925_1 930061.specijlodge -34 The discrete non-linear one dimensional sharpening filter illustrated in Fig. 11, has the desirable property of being equivariant to affine transformations of the intensity, and the local average of the intensity of signal formed by the intermediate samples is the same as that of the input samples because the average of the two output samples 11023 5 and 11024 is equal to the value of the center sample 11012 of the input window. These properties ensure that the non-linear discrete filter does not distort the appearance of the input signal too much. When the non-linear discrete filter is combined with a further linear interpolating filter, e.g. cubic to produce output samples at a given output resolution, the equivariance with respect to affine transforms of the input is preserved 10 because linear interpolating filters also have the desired equivariance property. Since this non-linear filter increases the slope of edges without adding overshoots, frequencies beyond the Nyquist limit that are aliased in the input signal or do not occur in the input signal are consequently introduced into the output of the non-linear filter. The intensity plots 2010, 2040, 2070 and 2100 in Figs. 2A - 2D are reproduced 15 in Figs. 12A - 12D as intensity plots 12010, 12050, 12080 and 12110 respectively. The reproduced intensity plots 12010, 12050, 12080 and 12110 show plots of the ideal intensity 12040, 12070, 12100 and 12130 respectively and the cubic interpolated intensity 12030, 12060, 12090 and 12120 respectively. The intensity at the intermediate samples 12015, 12016, 12017, 12018, 12019, 12020, 12021 and 12022 are also shown. 20 In most cases, the intermediate samples match the ideal signal more closely than the cubic interpolated signal. The intermediate samples 12015, 12016, 12017, 12018, 12019, 12020, 12021 and 12022 can be interpolated to produce a better approximation to the ideal intensity 12040 than the cubic interpolated intensity 12030. 2453925_1 930061_speci_lodge - 35 In one implementation, the intermediate samples 11021, 11022, 11023, 11024, 11025 and 11026 are output directly from the vertical sharpening filter block 7020 and the horizontal sharpening filter block 7030 as vertically sharpened interpolated samples 7021 and horizontally sharpened interpolated samples 7031 respectively. In this case, 5 the combine block 7080 performs further vertical and horizontal interpolation of the sharpened samples to produce samples at auxiliary points 9010, 9020. In another implementation, the sharpening filters 7020, 7030 perform additional interpolation to produce samples at the output resolution at either the correct horizontal or vertical positions respectively and the combine block 7080 only performs one dimensional 10 interpolation to produce samples at auxiliary points 9010, 9020. In this case the sharpening filters produce multiple streams of output at appropriate offsets from the output sample locations. Other arrangements of filters are possible. In another implementation, the vertical sharpening filter 7020 and the horizontal sharpening filter 7030 are wider frequency response filters. A typical interpolating filter 15 is a low-pass filter that has a cut-off frequency equal to the Nyquist limit. The wider frequency response filters have a cut-off frequency that is higher than a typical interpolating filter. In an exemplary implementation, a cut-off frequency equal to twice the Nyquist limit is used. The wider frequency response filters retain higher frequencies which produces a sharper edge. However, aliasing may be present in the output of these 20 wider frequency response filters. Such aliasing is removed or reduced by the combine step 6080 as further described above with reference to Fig. 9, or as described below with reference to Fig. 14 or Fig. 15. Fig. 13 shows the intensity profile 13010 of samples that would be produced using a conventional interpolating filter along the isophote 9030. This figure illustrates a 2453925_1 930061_speci_lodge - 36 method to remove aliasing by creating an output sample 13040 along the isophote 9030 passing through the output sample location 9060, approximating the value of the original signal at this location. The original isophote value may be estimated by averaging two interpolated auxiliary samples 13031 and 13032, located at points P 9010 and P 2 9020. 5 These points are determined from the period vector components d, and d, as shown in Equation (9) and Equation (10) where P = (xo, yo) is the output sample position 13040 and D = (dx, dy) is the aliasing period vector measured at the output sample position, P, according to Equation (6) or Equation (7). The auxiliary points are separated by a distance 13050, equal to half the aliasing period. Since the points 13031 and 13032 are 10 180 degrees out of phase with respect to each other, variation due to the primary (or fundamental) frequency of the aliasing is cancelled out to produce an output sample 13040 close to the ideal value 13010. The approach shown in Fig. 13 reduces intensity variations 13010 along the isophote 9030. D P, =P--, (9) 4 D P2 P+-. (10) 4 In one implementation, the two auxiliary samples Sz 13031 and S 2 13032 are 15 determined using a bicubic or a bilinear interpolation method at the auxiliary points P, and P 2 respectively. In this implementation, the input to the bicubic or bilinear interpolation stage is formed by the sharpened samples, for example 11021 to 11026. Averaging the two points acts like a notch filter that removes the fundamental harmonic of the aliasing error. 20 In an alternative implementation, the two auxiliary samples S 1 13031 and S 2 13032 are determined using a cubic filter with a wider frequency response. If the 2453925_1 930061_specilodge - 37 isophote 9030 is predominantly horizontal, the cubic filter is applied vertically and if the isophote 9030 is predominantly vertical, the cubic filter is applied horizontally. In another implementation, a wider notch anti-aliasing filter is used. The wider notch anti-aliasing filter produces a better estimate of the output sample than the two 5 auxiliary point anti-aliasing filter when there are small errors in the isophote orientation estimate. The wider notch filter is determined by calculating interpolated values at three specific locations and calculating a weighted average of the three interpolated values. It is referred to as a wider notch filter because its frequency response has a wider region where the response is close to zero than the aforementioned two auxiliary point anti 10 aliasing filter. Three auxiliary points, P1, P 2 and P 3 , on the local isophote are generated after determining the aliasing period vector components d, and d. These auxiliary points are separated or spaced apart by a distance equal to the aliasing period. P; and P 2 are located half the aliasing period either side of the output sample position and P 3 is located at the output sample location. Auxiliary samples S 1 , S 2 and S3 are interpolated at 15 the three auxiliary points P 1 , P 2 and P 3 . The output sample is obtained from a weighted average of the auxiliary samples S;, S 2 and S 3 . The weighted average is such that the auxiliary samples at P 1 and P 2 are assigned weights of 0.25 each and P 3 is assigned a weight of 0.5. In another implementation, a primary (fundamental) and secondary aliasing 20 harmonic removal anti-aliasing filter is used. Desirably, four auxiliary points are used to perform anti-aliasing. The second harmonic of aliasing may become more visible when (i.e. there was a large amount of aliasing) large up-scaling ratios are used (i.e. greater than 3 times), or when isophote orientations are close to horizontal or close to vertical. 2453925_1 930061_specilodge -38 Fig. 14 illustrates an approach for creating an output sample 14040 approximation of an original value 14020, referred to as the secondary harmonic anti aliasing filter. The output sample 14040 has the primary and second aliasing harmonic removed. Four auxiliary points P, 14031, P 2 14032, P 3 14033 and P 4 13034 on the 5 intensity profile 14010 are used. Each auxiliary point 14031, 14032, 14033 and 14034 is separated by a quarter 14060 of the aliasing period, offset by an eighth 14050 of the aliasing period from the output sample 14040. The auxiliary points 14031, 14032, 14033 and 14034 are averaged to produce an output sample 14040 close to the ideal value 14020. This removes intensity variation 14010, due to aliasing in the sampled 10 input image. In another implementation, an estimated original isophote equation is used for anti-aliasing. Firstly, an equation for the original isophote is generated. The output sample is then calculated at the output sampling point using the original isophote equation. Unlike the described two auxiliary point anti-aliasing filter, the wider notch 15 anti-aliasing filter and the secondary harmonic anti-aliasing filter, this method does not require auxiliary points 180 degrees or 90 degrees out of phase with each other. This implementation achieves better results than other anti-aliasing embodiments for approximately vertical and approximately horizontal isophote orientations. However, this filter still breaks down at exactly vertical and horizontal isophote orientations. 20 Fig. 15 shows an image 15010 with samples 15020, and a desired output sample 15050 at an output location. Four interpolated isophote points 15041, 15042, 15043 and 15044 are found on the isophote 15030. An approximation, p, of the original isophote at the desired output sample 15050 may be determined from two or more isophote points 15041, 15042, 15043 and 15044. Fig. 15 shows four isophote points 15041, 15042, 2453925_1 930061_speci_lodge - 39 15043 and 15044 with interpolated sample values, p,, for i equal to 1, 2, 3 and 4 and positional offsets y, 15061, 15062, 15063 and 15064. The interpolated isophote points 15041, 15042, 15043 and 15044 are interpolated using a sharpening filter. Equation (11) defines an equation for an isophote assuming a sinusoidal aliasing 5 error. p 1 =p+Mf+n,, (11) where p is an approximation of the original isophote at the desired output sample, M is the amplitude of the sinusoidal variation due to aliasing along the isophote 15030, f = sin(2ny) is the sinusoidal variation, y, is the positional offset associated with each isophote point, p,, and n, is an error or noise term associated with each corresponding 10 isophote point. The error terms, ni, represents the difference between the interpolated samples, P,, and the assumed model,fi. Minimising the error in Equation (11) using a least squares fitting of the interpolated points 15041, 15042, 15043, and 15044 produces a best match equation approximating the original isophote at the desired output sample 15050 as defined in 15 Equation (12): 4 4 44 E WifE2 W P~J EwfEA P 4 4 2(12) where P, is the interpolated output sample, ] = sin(2ny ) is the sinusoidal variation, y, is the positional offset associated with each isophote point, Pi, and wi is a weighting factor for the noise term ni, where the sum of the weighting factors is equal to one. Fig. 16 illustrates how the blending operation includes blending the default 20 interpolated sample 7061 and the sharpened sample 7090 together, depending on the 2453925_1 930061.specilodge - 40 angle of the isophote orientation 7041. Segments 16010-16040 correspond to angles that are approximately horizontal or approximately vertical. In these segments, the default interpolated sample 7061 is used to produce the output sample 7110 (the magnitude along these segments is one) and the sharpened sample 7090 is not used. This has the 5 effect of containing the area of input samples required to produce an output sample to be localised to the output sample location 9060. Segments 16050-16080 correspond to angles that are not approximately horizontal or approximately vertical. In these segments, the default interpolated sample 7061 is not used to produce the output sample 7110 (the magnitude along these segments is zero) and the sharpened sample 7090 is 10 output as the output sample 7110. Segments 16110-16180 correspond to angles that are near horizontal or near vertical. In these segments, the output sample 7110 is a blend of the default interpolated sample 7061, proportional to the amplitude of the point on the segment corresponding to the angle of the isophote orientation 7041, and the sharpened sample 7090. 15 In one implementation, the reliability 7071 and the orientation blend factor as illustrated in Fig. 16 are both used to determine the blend between the output 7080 from the combine step 6080. One way to determine the blend is to determine the blend amount as calculated based on the orientation angle and subtract it from 1.0 to determine the proportion of the anti-aliased sample 7090 to be used and multiply the proportion by 20 the amount as determined based on the reliability 7071. Other means of determining the proportions based on both the reliability and the orientation may be used. For example, the relative proportion of the anti-aliased sample 7090 may be determined based on the reliability and based on the orientation, and a modified proportion based on the 2453925_1 930061_specilodge -41 minimum of the two relative proportions may be used as the combined proportion that depends on both the reliability and the orientation. INDUSTRIAL APPLICABILITY It is apparent from the above that the arrangements described are applicable to 5 the computer and data processing industries and particularly for the processing of image data for upsampling or scaling. Such finds particular application to computerized apparatus such as display devices including video and TV displays, for example. The foregoing describes only some embodiments of the present invention, and modifications and/or changes can be made thereto without departing from the scope and 10 spirit of the invention, the embodiments being illustrative and not restrictive. For example, it is possible to use the described methods for filtering without resampling. This might be useful for example for post-processing an image resampled at an increased resolution to remove jaggie artefacts. In this case the aliasing period is a function of both the isophote orientation and prior increase in resolution factor. If the 15 prior increase in resolution factor is not known in advance, an additional step may be required to estimate the aliasing period along the isophote orientation at each point in the image. An isophote orientation specific periodic variation in intensity may be observed along lines corresponding to isophotes in an ideal image and a filter method may be applied that specifically suppresses variation at this frequency along the isophote. 20 The term "sample" used above means one component of one pixel. If the image data or video has more than one pixel component (eg. Red, Green and Blue), the algorithm and method is applied independently for all components. Cubic and bicubic interpolation have been used above due to their widely acknowledged quality when used with natural images. In some cases, other methods 2453925_1 930061_specijiodge -42 such as bilinear interpolation or implementations such as Catmull Rom or Lanczos may be preferable and may be used with suitable variation in window sizes. In general any interpolation method may be used within the context of the described methods. Any FIR filter may be modified to allow a substantial response to frequencies beyond the Nyquist 5 limit. In the context of this specification, the word comprising means including principally but not necessarily solely or having or including, and not consisting only of. Variations of the word comprising, such as comprise and comprises have correspondingly varied meanings. 10 2453925_1 930061_specijlodge

Claims (25)

1. A method for processing an image comprising input samples located on a regularly spaced grid to produce an output image comprising output samples, said 5 method comprising: estimating an isophote orientation based on a location of an output sample and locations of input samples close to the location of the output sample; selecting, based on the estimated isophote orientation, a directional sharpening filter having a corresponding sharpening direction, where the output of said selected 10 sharpening filter includes directional frequencies beyond the Nyquist limit determined by a sampling rate of the input samples in a direction of the selected sharpening filter; applying the selected sharpening filter to input samples located about the location of the output sample to produce a plurality of sharpened samples; combining said sharpened samples to produce an output sample at the output 15 sample location based on the location and the isophote orientation.

2. The method of claim I wherein the directional sharpening filter is one of: (i) a horizontal sharpening filter having an output which includes horizontal frequencies beyond the Nyquist limit determined by a horizontal sampling rate; and 20 (ii) a vertical sharpening filter having an output which includes vertical frequencies beyond the Nyquist limit determined by a vertical sampling rate.

3. The method of claim 1, wherein said combining comprises reducing aliasing artifacts in the image formed by the output samples. 25 2453925_1 930061_specijlodge - 44

4. The method of claim 1, further comprising: estimating a second isophote orientation based on the location of a second output sample and input samples located about the second output sample; estimating a reliability of the estimated second isophote orientation; 5 calculating a default output sample where the second isophote orientation is estimated as unreliable; outputting said default output sample at said second output sample location when said reliability estimate has a low value. 10

5. The method of claim 3, further comprising: estimating a reliability of said isophote orientation; and calculating, where the reliability is estimated as unreliable, a default output sample based on said output sample location; wherein said combining comprises: 15 reducing aliasing by processing said sharpened samples to produce an anti-aliased sample; determining said output sample according to a weighted combination of said anti-aliased sample and said default output sample, wherein the weighting is based on said reliability. 20

6. The method of claim 4, wherein said default sample is calculated by combining the outputs of both a horizontal sharpening filter and a vertical sharpening filter applied to input samples near the output sample location. 2453925_1 930061_speci_lodge - 45

7. The method of claim 5, wherein said default sample is calculated by combining the outputs of both a horizontal sharpening filter and a vertical sharpening filter applied to input samples near the output sample location. 5

8. The method of claim 3, wherein said aliasing reduction removes the fundamental harmonic of periodic aliasing artifacts in a direction corresponding to said isophote orientation.

9. The method of claim 5, wherein said aliasing reduction comprises: 10 determining a plurality of auxiliary points based on said isophote orientation and the location of said output sample; applying the selected sharpening filter to produce said sharpened samples at each of said auxiliary points; calculating said anti-aliased sample based on a weighted average of said 15 sharpened samples.

10. The method of claim 9, wherein the number of the auxiliary points, and the corresponding number of sharpened samples, used to calculate said anti-aliased sample is two, and the two auxiliary points are spaced apart from each other in said sharpening 20 direction a distance equal to one half of the spacing between the input samples in the sharpening direction, and the spacing of the auxiliary points perpendicular to the sharpening direction depends on said isophote orientation. 2453925_1 930061_specijlodge - 46

11. The method of claim 10, wherein each of said auxiliary points is located so that the separation in the sharpening direction between each auxiliary point and the output sample location is one quarter of the spacing between the input samples in the sharpening direction. 5

12. The method of claim 9, wherein the number of auxiliary points, and the corresponding number of sharpened samples, used to calculate said anti-aliased sample is three, and the three auxiliary points are spaced apart in the sharpening direction at distance equal to one half of the spacing in the sharpening direction between the input 10 samples, and where the spacing of the auxiliary points in a direction perpendicular to said sharpening direction depends on said orientation and where said weights for combining the corresponding three sharpened samples are 0.25, 0.5 and 0.25 in which a center sharpened sample of the three is assigned the weight of 0.5. 15

13. The method of claim 9, wherein the number of auxiliary points, and the corresponding number of sharpened samples, used to calculate said anti-aliased sample is four, and the four auxiliary points are spaced apart in the sharpening direction at distance equal to one quarter of the spacing in the sharpening direction between the input samples, and where the spacing of the four auxiliary points in the direction perpendicular 20 to the sharpening direction depends on said orientation and further, the corresponding four sharpened samples being combined using equal weights of 0.25.

14. The method of claim 13, wherein said four auxiliary points are located so that the separation in the sharpening direction between each of the center two auxiliary points 2453925_1 930061_specijlodge - 47 and the output sample location is one eighth of the spacing between the input samples in the sharpening direction.

15. The method of claim 1, wherein said sharpening filter comprises a separable filter 5 comprising a one dimensional horizontal filter and a one dimensional vertical filter, and the separable one dimensional filter in the sharpening direction is a one dimensional sharpening filter, and the separable one dimensional filter perpendicular to the sharpening direction is an interpolating filter. 10

16. The method of claim 15, wherein said one dimensional sharpening filter is a polyphase finite impulse response filter having non-zero response to frequencies beyond the Nyquist limit.

17. The method of claim 15 wherein the one dimensional filter in the sharpening 15 direction is a non-linear filter that is equivariant with respect to affine transformations.

18. The method of claim 15, wherein the one dimensional filter in the sharpening direction has impulse response equal to zero if the absolute value of x is greater than one half, and equal to two minus the absolute value of four times x, if the absolute value of x 20 is less than one half.

19. The method of claim 17 wherein said non-linear filter comprises a non-linear sharpening filter and an interpolating filter, wherein said non-linear filter produces an 2453925_1 930061_specijlodge - 48 output at a higher resolution that the corresponding input and wherein said non-linear filter is equivariant with respect to affine transformations.

20. The method of claim 1, wherein said combining comprises: 5 calculating a default value at said output sample location using a non-adaptive filter; calculating an anti-aliased value, based on the estimated orientation; combining said default value with said anti-aliased value to produce said output sample. 10

21. The method of claim 20 wherein said combining is based on said orientation.

22. The method of claim 3 wherein said reducing aliasing artifacts includes fitting said sharpened samples to an aliasing model using a least squares fit and determining 15 said output sample value based on a parameter of said aliasing model.

23. A method of image processing substantially as described herein with reference to any one of the embodiments as that embodiment is illustrated in the drawings. 20

24. Computerized apparatus adapted to perform the method of any one of claims I to 23. 2453925_1 930061_speci-lodge - 49

25. A computer readable storage medium having a program recorded thereon, the program being executable by computerized apparatus to perform the method of any one of claims I to 23. 5 Dated this L- 3 C day of December 2009 CANON KABUSHIKI KAISHA Patent Attorneys for the Applicant Spruson&Ferguson 2453925_1 930061_specijlodge