Classifications

• • G—PHYSICS
  • G06—COMPUTING; CALCULATING OR COUNTING
  • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
  • G06T9/00—Image coding
  • G06T9/007—Transform coding, e.g. discrete cosine transform

Definitions

• the present invention relates to image processing, and more particularly to image processing using tiles and reducing tile edge artifacts.
• Modern computers deal with memory and processing demands of scanning and manipulating images in several ways. For a large image, only a part or block of the image is read from disk storage and transferred into active memory. These blocks of the image undergoing active processing are called tiles. When a tile has been processed, the tile is written back to disk storage, and the next tile is read from the disk. By processing an image in tiles, the entire image need not be stored in active memory at one time. Many modern computers, such as the Intel MMX system, operate on multiple data paths in parallel. To utilize parallel processing, different parts of the image must be processable in parallel.
• the MPEG algorithms used in digital video processing are examples of tiled image processing. The tiles permit full use of parallel processing and minimal active memory, but such processing may produce edge artifacts between the tiles which are seen after the tiles are reassembled and are commonly known to MPEG and JPEG images.
• Figure 1 illustrates the processing of an image 102 to produce a resultant image 104.
• the image Prior to applying the processing algorithm 106, the image is sectioned into two-dimensional blocks 108, and the processing step 106 is applied individually to each block, such as block 110, to produce resultant blocks, such as block 112.
• the processed blocks 114 are then reassembled like a jigsaw puzzle to form the resultant image 104.
• Similar processing can be applied to a one-dimensional string of data, such as that from a processed signal, instead of a two-dimensional ordering of data.
• the sectioned one-dimensional blocks are called frames.
• the string is sectioned into frames, the processing applied to each frame, then the processed frames recombined to form a resultant string of processed signal data.
• the input signal can be composed of discrete samples as in digital audio and image, or the signal can be a continuum such as that found in analog audio and film images or analog video. Block processing methods apply to both analog and digital signals.
• the processing step 106 of Figure 1 can take many forms. For example, it may include transforming, compressing, and storing the data, as is common in JPEG and MPEG image compressions.
• a class of processings is illustrated in Figure 2, wherein a data frame 202 undergoes a transformation step 204 to form a related frame 206 of numbers in a different transform space.
• the input data frame 202 oscillates slowly as seen by the numbers inside the individual spatial elements.
• the transformation step 204 is a frequency transform, so the related frame 206 is in frequency transform space. A higher number is found at the frequency element 208 corresponding to the frequency of the oscillation.
• the frame 208 in transform space may undergo additional processing 210 such as a compression process step which compresses or limits high frequencies. Compression is but one example of an additional process which the data may undergo.
• the frame 206 is processed by transformation step 212, the inverse of transformation step 204, to output the processed frame 214.
• Block 302 is a single block from a segmented image, such as block 110 of image 102 shown in Figure 1. Based on the specific data shown, block 302 has a horizontal low frequency oscillation and a high frequency vertical oscillation. Transformation step 304 converts the numbers in block 302 into a frequency transform space 306, with specific frequency element 308 high due to the vertical high frequency oscillation, element 310 high due to horizontal low frequency oscillation, and element 312 high due to the "DC", or average.
• processing step 314 is applied in the transform space to block 306 to produce block 316.
• the processing step 314 is a low pass filter which erases to zero in block 316 all frequency terms exceeding a frequency denoted by dotted line 318, such as high frequency element 308.
• transformation step 320 the inverse of transformation step 304, is applied to block 316 to produce block 322.
• block 322 contains numbers similar to block 302 except that rapid changes with space, such as the high frequency oscillations, have been removed.
• Input signal 410 will be processed using the prior art method described above.
• Input signal 410 is represented in this case by a continuum. It is understood that input signal 410 could also be represented by a string of numbers as was used in Figure 2.
• the difference between numbers and a continuum is the difference between numbers stored on a compact disk versus voltages fed to a speaker, or the difference between pixels scanned digitally versus light patterns received by the eye.
• the translations between discrete numbers and a continuum are very well understood in the art, such that processes may be though of and explained interchangeably in either form.
• Input signal 410 in Figure 4a is divided into four, discrete, nonoverlapping frames as shown in Figure 4b. These frames are selected to cover all of the signal without overlapping any portion of the signal.
• step 412 the change from a low value to a higher value
• step 416 occurs between frames 3 and 4, and thus appears to be "lost”.
• each frame is then processed by a low pass filter to soften any edges. It is seen that step 412 has received the desired softening from the low pass filter to produce step 420. Note that the "lost" step 416 has received no softening because it occurred at the transition between frames 3 and 4, and so appears in no frame.
• step 412 in Figure 4a appears as softened edge 424 as desired, but the step 416 that just happened to occur on the boundary between frame slices still appears as a hard step 426.
• edge tolerant processing such as a discrete cosine transform (DCT)
• DCT discrete cosine transform
• the DCT assembles a wave with only cosine terms that have zero slope at transitions between frames, if any of the waves is removed by, for example, applying a low pass filter, the resultant function will not change slope between frames.
• Other transforms are not so edge tolerant.
• DFT discrete Fourier transform
• a discrete Fourier transform has many desirable properties, but is not in common use for block processing because of its behavior at edges.
• the present invention segments the image or signal being processed into overlapping blocks, separately processes each block, and reassembles the blocks following the processing.
• the interstitial topology can be on a diagonal grid.
• the blocks are windowed such that when they are reassembled by adding them together, the block windows mutually add to unity. Further, the windowing may be applied after processing, or divided before and after the processing step to facilitate edge sensitive processes. In addition, some attributes, such as the "DC" bias, can be separated from the block before processing, and windowed separately before recombining.
• Figure 1 illustrates a prior art method of processing an image using blocks
• Figure 2 illustrates a prior art method of processing a one-dimensional signal block in transform space
• Figure 3 illustrates a prior art method of processing a two-dimensional image block in transform space
• Figures 4a-4d illustrate prior art problems which arise during processing of non- overlapping one-dimensional blocks
• FIGS 5a-5d illustrate prior art problems which arise during processing of non-overlapping one-dimensional blocks with an edge sensitive process
• Figures 6a-6d illustrate the present invention applied to one-dimensional blocks
• Figure 7 illustrates the present invention applied to two-dimensional image blocks
• Figures 8a-8c illustrate a specific embodiment of the present invention processing two-dimensional blocks
• Figures 9a-9e illustrate the present invention applied to one-dimensional blocks with an edge sensitive process
• Figure 10 illustrates the present invention applied to two-dimensional image blocks with an edge sensitive process
• Figures 1 la-1 lc illustrate a specific case of applying the present invention to two- dimensional blocks compatible with an edge sensitive process.
• a continuous graph 602 is divided into multiple frames. As in the prior art processing, the graph is sectioned into frames 1 to 4 of Figure 6a; however, additional frames 1.5, 2.5, and 3.5 are selected such that they overlap a portion of the signal included in frames 1 to 4 of Figure 6a. All of these frames are individually displayed in Figure 6b.
• edge 604 is represented in frame 2 as edge 606 as in the prior art.
• edge 608 that was "lost" in the prior art method because of its position between frames 3 and 4 is now represented in one of the overlapping frames, frame 3.5, as edge 610.
• a low pass filter is applied to every frame in Figure 6b to form the frames in Figure 6c.
• edges 606 and 610 are softened to produce filtered edges 612 and 614.
• overlapping frames of Figure 6c need to be reassembled into a complete graph. Because frames are overlapping, frames can not be simply concatenated like a jigsaw puzzle as was done in the prior art described in Figures 4 and 5. Rather, frames must be blended together with a gradual transition between edges taking into account the degree of overlap between blocks. This blending is accomplished by applying a blending window to each frame and then combining the windowed frames. "Applying a blending window" in this context means multiplying each element of a frame by the corresponding element of the window to produce a resulting element of the windowed frame.
• a frame element can be a digital sample, such as a pixel, or a brief segment of time in an analog signal.
• “Combining the windowed frames” means to first register the frames so that corresponding elements align, and then to add the values in each of the corresponding elements together to form the processed signal. If one is working in the logarithmic domain, the actual computer operation needed to form an effective multiplication or addition is different than if one is working in a linear domain, and hence the terms “apply” and “combine” encompass these operations and are used to prevent restriction to a specific domain.
• blending windows are selected such that multiple copies will mutually add to unity when those copies of the window are laid over each signal block or frame. This insures that, wherever two or more signals overlap, the ultimate processed signal will have a value that is intermediate, or a weighted average, of the different overlapping signals.
• blending windows are indicated as the dotted curves in Figures 6c and 6d.
• time 640 and time 642 in Figure 6c and time 644 in Figure 6d are the same time registered to different frames. At time 640, assume that the value for frame 612 is 2.80 and the corresponding blending window is 0.25 (25% of unity).
• windowed signal 648 (shown by the dotted curve) having a value of 0J0.
• the value for frame 646 is 3.00 and the corresponding blending window is 0J5 (75% of unity).
• a windowed signal 620 with a value of 2.25.
• the value 2.95 is intermediate between the two frame signal values of 2.80 and 3.00. It is in fact an average of 25% of 2.80 and 75% of 3.00, where 25% plus 75% is unity.
• the resulting processed signal would of course be ten times larger, and if desired, the entire processed signal could be divided by ten, or left amplified.
• a gain of unity is used illustratively for simplicity, it should be understood that the wording of "unity" or “intermediate” further anticipates allowing the entire signal to be multiplied by a gain constant either before processing, within the window, post processing, or at any time, without departing from the
• blending windows that possess the property of adding to unity.
• One such window is the triangle response such as that shown at 620 and 622 for frame 2.5 and frame 3 in Figure 6c. These windows are also portrayed as they are designed to overlap in Figure 6d.
• the triangle 622 starts with 0% gain at the left edge 626 of frame 3. This corresponds to the position at which the overlapping triangle 620 has 100%) gain, and so they mutually add to 100% unity.
• the triangle 622 rises linearly to 100% at center position 630 in frame 3. At this position the overlapping triangles on each side have 0% gain, and so again they add to 100%).
• triangle 622 is 50%, and the overlapping triangle 620 is 50%, so both add again to 100%.
• the pedestal of 1/2 causes this function to vary from 0.0 at the left and right edges to 1.0 at the center position.
• a windowed curve 632 is obtained. After windowing each frame of Figure 6c, and positioning and adding the windowed frames, the continuous function 634 of Figure 6d is obtained which is free of the artifacts seen in the prior art of Figure 4. In particular, both the rising edge 636 as well as the falling edge 638 have been smoothed by the method of the present invention.
• Figure 7 is analogous to Figure 1 except for the division of the original image into intermediate overlapping blocks, such as block 702. Also needed to practice this version of the invention
• -8- is the introduction of the blending window 704 to multiply by the unwindowed blocks, such as block 706, to produce windowed blocks, such as block 708. After windowing, the windowed blocks are aligned and added to produce the assembled output image 710.
• Figure 8 gives a more detailed and specific example of applying the present invention in two dimensions.
• Figure 8a shows a nonoverlapping set of blocks such as block 802, which may be considered analogous to block 712 of Figure 7.
• Each of these blocks in this specific example contains an array of 8 by 8 pixels.
• the centers of these first blocks are indicated by the numerals 1 in Figure 8b.
• Interstitial with this first set of blocks is a second set of overlapping blocks of the same size, such as block 804, analogous to block 702 of Figure 7.
• This second set of overlapping blocks are offset from the first set of nonoverlapping blocks in this specific example by 4 pixels both horizontally and vertically thereby producing centers indicated by the numerals 2 in Figure 8b.
• the centers of the two sets of blocks lie on a diagonal grid. Together these two sets of blocks cause each pixel of an underlying image to lie in two blocks, one from the first set, and one from the second set.
• Figure 8c shows a blending window useful for the diagonal grid arrangement of
• FIG 8. This blending is analogous to blending window 704 of Figure 7.
• the blending window will have 8 by 8 numerical values, such as shown in Figure 8c. Because in this example each pixel in the underlying image is covered by exactly two blocks, the blending
• -9- window must contain gain value such that when the window is offset to the next tier (in the same way that dotted block 810 is offset relative to block 812), the sum of window values will mutually add to unity.
• element 814 of window block 812 contains 1/8, which will overlay the equivalent of element 816 of window block 810, which contains 7/8. The sum of 1/8 and 7/8 is 1.
• the invention may be further extended to work with an edge sensitive transform.
• Any transform is useful to the extent it can segregate a wanted aspect of data from unwanted aspects, such as noise.
• the DCT mentioned earlier is commonly used because it divides an image into various frequencies, with lower or stronger frequencies assumed to be more useful. However, the DCT is poor at distinguishing diagonal angles, hence the speckle around diagonal edges in images using the DCT algorithm.
• the DFT is capable of distinguishing both frequency and angle, but as has been mentioned, is strongly sensitive to edge boundary conditions.
• the present invention will allow the use of edge sensitive transforms, such as the DFT, without boundary problems as illustrated below.
• Figure 9 uses the nomenclature of Figures 5 and 6, and it will be useful to view Figure 9 in this context.
• a continuous signal 902 is received in Figure 9a, and as explained before, is divided into overlapping frames in Figure 9b.
• Typical frames include frame 910 which is flat with a DC bias, or average value, of 1.0; frame 912 which includes a step and a DC bias of 2.0; and frame 914 which is again flat but with a higher DC bias of 3.0.
• Figure 9b also portrays a blending window 916, normally the same for all frames, as illustrated.
• Blending window 916 is shaped differently than the triangle 620 of Figure 6c.
• window 916 portrays a curve wherein each point is the square root of a triangle window. As will be seen, this window will be applied twice, and so its double application, or mathematical square, will result in the triangle window such as 620 of Figure 6c that will sum to unity when the frames are reassembled.
• the window applied before the process is called the preprocess blending window, and the
• frame basis or block basis for the two-dimensional case
• frame basis data are the DC, or average value, and the slope of the curve. In this example, only the DC value will be removed.
• frame 910 the DC value of 1.0, term 920, is separated out, leaving a flat frame of zero value. This flat frame is then multiplied by blending window 916, resulting in the flat frame 926 in Figure 9c.
• frame 912 has subtracted from it the DC term 922 of 2.0, leaving a frame that begins at -1.0 and ends at +1.0. This frame with DC removed is multiplied by window 916 to result in frame 928 of Figure 9c.
• frame 914 has a higher DC bias term 924 of 3.0, but after removal gives a flat frame 930 identical to flat frame 926.
• each frame of Figure 9b has the DC term removed, and the residue is then multiplied by window 916 to result in the continuous curve frames shown in Figure 9c, presented below each frame's associated removed DC term.
• each of the continuous line frames of Figure 9c is processed to yield the dotted line frames shown in Figure 9c.
• frame 928 may be transformed to frequency space, altered in frequency space, and transformed back to spatial domain, as was illustrated in Figure 2. If such processing involves a low pass filter, as was illustrated in Figure 2, this processing will yield low pass filtered frame 932 illustrated by a dotted line in Figure 9c.
• Each of the dotted line processed frames of Figure 9c is next multiplied by the blending window 916 to produce the dotted line curves of Figure 9d, such as curve 940.
• frame 912 of Figure 9b has been low pass filtered and multiplied by a triangle window as previously described in conjunction with Figure 6.
• the critical difference in the process illustrated in Figure 9 is that the triangle window has been split into two "half triangle" windows 916 that when multiplied together, or applied twice, form the triangle window.
• first half triangle window was applied to frame 912 before the low pass filter process to remove edge artifacts going into the processing
• the second half triangle window was applied to the processed signal 932 to remove artifacts of the processing.
• the first window can be the 2/3 power of a triangle and the second window a 1/3 power of that triangle, rather than both being the 1/2 power, or square root. This power imbalance would put more emphasis on removing edge artifacts going into the processing at the expense of less emphasis on removing edge artifacts of the processing.
• the product curve also does not need to be a triangle, as will be demonstrated in a specific example to be described in relation to Figure 11.
• each of the DC terms is also multiplied by both the pre and post processing windows, in this case that means multiplying by window 916 twice, or more simply, multiplying by the square of window 916 which is a triangle.
• DC term 922 2.0 ( Figure 9c) is multiplied by a triangle window to yield curve 942 ( Figure 9d) which peaks at 2.0.
• Curve 942 is registered with curve 950 in Figure 9e as curve 952.
• DC term 920 ( Figure 9c) is multiplied by a triangle to give curve 944 in Figure 9d and registered curve 954 in Figure 9e.
• the larger DC term 924 naturally results in taller triangle curve 946 and registered triangle curve 956.
• Figure 9 has application to signals such as audio processing.
• Figure 10 presents the same concept in two dimensions for applications such as image processing. The concept can also be used for higher dimensions, only the names
• Figure 10 uses the nomenclature of Figure 7, and it will be useful to view Figure 10 in this context.
• an image 1010 is separated into overlapping blocks 1012, such as the specific block 1014.
• the bases are extracted to give a set of basis values 1016, such as specific basis value 1018 from block 1014, and a set of blocks 1020 with the bases removed, such as specific block 1022 containing the residue of block 1014 after removing the basis.
• Each of the blocks of the set 1020 is multiplied by a blending window 1030 to yield a set of windowed blocks 1032.
• block 1022 is multiplied by window 1030 to give the specific windowed block 1034. Multiplication in this sense means to multiply each pixel in block 1022 by the corresponding pixel in window 1030 to yield the corresponding pixel in the windowed block 1034.
• process 1040 is applied to each of the windowed blocks 1032 to yield the set of processed windowed blocks 1042.
• process 1040 may take a Fourier transform of block 1034, remove the high frequencies by zeroing terms in transform space, and then take the inverse transform to yield low pass filtered block 1044.
• each of the processed blocks 1042 is multiplied by window 1050 to yield the set of completed blocks 1052.
• windows 1030 and 1050 are selected such that when multiplied together, windows 1030 and 1050 form a window which, when a copy is positioned in register with each of the blocks in the set 1052, will sum to an identical value, typically unity, at all pixels.
• specific processed block 1044 is multiplied by window 1050 to yield the specific finished block 1054.
• the set of basis values 1016 is each multiplied by both windows 1060 and 1062 to yield windowed set 1064.
• Multiplication in this sense means that the basis values are first reconstructed into a two-dimensional block. In the case where the only basis value is the DC term, every pixel in the reconstructed block is the same DC term. In the case where the basis values include a slope or higher order curve, the pixels in the reconstructed block will track that slope or higher order curve, using formulae well known in regression
• each pixel in the reconstructed block derived from, for example, the specific basis values 1018 is scalarly multiplied by the corresponding pixel in window 1060 and again multiplied by the corresponding pixel in window 1062 to yield window blocks such as the specific windowed block 1066.
• Window 1060 is a copy of window 1030, and window 1062 is a copy of window
• Figure 11 illustrates a numerically specific implementation of the procedure just presented in Figure 10.
• each pixel in a two-dimensional image will be covered by 8 blocks. This high positional redundancy will almost completely remove even subtle edge effects at the expense of computational speed. This would be used in a case where image quality matters over speed, such as, for example, in an image beauty enhancement or grain management program.
• Figure 11a illustrates a single 16x16 pixel block 1102 centered at point 1104. This center point is shown by the solid dot 1106 in Figure l ib. Another single 16x16 pixel block 1110 is centered at point 1112, similarly shown in Figure l ib by the open dot 1114. Figure l ib shows the center points of all blocks in this specific implementation. The centers are seen to lie on a diagonal grid, and each pixel in the image will be overlaid by 8 blocks in differing positions.
• the blocks could be 8x8 pixels with the upper right pixel missing so the blocks are not squares. These nonsquare 8x8 blocks could lie on a square grid with 7 pixel spacing between block centers. In such a case, the edge pixels would be overlaid twice, the corner pixels would be overlaid three times, and most pixels not overlaid at all, for an average 63/49 overlay redundancy. As seen by this later example, the other previously described specific examples of 2x and 8x redundancy, the specific examples using a diagonal pattern of block centers, or indeed the specific examples with square blocks should not be construed as limits.
• Figure l ie illustrates a window operable with the block topology of Figures 11a and
• Block 1120 is illustrated partially drawn around center point 1122.
• the numeric values in each pixel are given by formula 1124 as follows:
• this example uses a cosine blending window.

Landscapes

• Physics & Mathematics (AREA)
• General Physics & Mathematics (AREA)
• Engineering & Computer Science (AREA)
• Discrete Mathematics (AREA)
• Multimedia (AREA)
• Theoretical Computer Science (AREA)
• Image Processing (AREA)
• Compression Or Coding Systems Of Tv Signals (AREA)

Applications Claiming Priority (3)

Publications (1)

Family

ID=22127225

Family Applications (1)

Country Status (4)

Families Citing this family (5)

Family Cites Families (6)

• 1999
  • 1999-02-10 AU AU26677/99A patent/AU2667799A/en not_active Abandoned
  • 1999-02-10 EP EP99906861A patent/EP1057142A1/de not_active Withdrawn
  • 1999-02-10 WO PCT/US1999/002828 patent/WO1999042954A1/en not_active Application Discontinuation
  • 1999-02-23 TW TW88102665A patent/TW410316B/zh not_active IP Right Cessation

Non-Patent Citations (1)

Also Published As

Similar Documents

Legal Events