WO2008004903A1

WO2008004903A1 - Method for digital inpainting without priority queue

Info

Publication number: WO2008004903A1
Application number: PCT/RU2006/000347
Authority: WO
Inventors: Alexander Alexeevich Kibkalo
Original assignee: Intel Corporation
Priority date: 2006-07-03
Filing date: 2006-07-03
Publication date: 2008-01-10

Abstract

Digital inpainting of an area of a source image may be accomplished by setting a status of each pixel of the area to unknown; setting a status of each pixel of the source image outside of the area to known; computing a priority value for each unknown pixel using a distance transform process; sorting the unknown pixels in order of priority value; and for each unknown pixel in order of priority, restoring a pixel value using, at least in part, neighboring known pixel values, and changing the unknown pixel's status to known. Computation of the priority value each of the unknown pixels includes applying a Fast Marching Method (FMM) computation as a distance transform function.

Description

METHOD FOR DIGITAL INP AINTING WITHOUT PRIORITY QUEUE

BACKGROUND

1. FIELD

The present invention relates generally to modification, enhancement, and restoration of digital images and, more specifically, to filling damaged or unknown regions in an image.

2. DESCRIPTION

Modification of some parts of digital images is a common task in many professional and personal digital image processing applications. Restoration or repairing of damaged parts of images, removing some undesirable features from the images, and similar tasks, are typically solved manually by skilled artists or automatically using known digital inpainting techniques. The goal of digital inpainting is to restore the value of an unknown pixel of an image (e.g., a pixel of a damaged image part, a pixel of an object to remove from an image, etc.) using values of known pixels in the neighborhood of the unknown pixel. Image restoration is done pixel-wise starting at the border of the unknown image area. After a pixel is restored it becomes a known pixel and the unknown image area is now smaller. The process is continued until all unknown pixels are restored.

The general approach to digital inpainting assumes that priorities are assigned to all unknown pixels, and the pixels are restored in descending order of their priorities. At all stages of the digital inpainting process, pixels belonging to the boundary of the unknown image area have higher priorities and are processed before internal pixels of the unknown image area.

To assign the priorities and thus order the pixel processing, known techniques explicitly model the border of the inpainted area. For example, a one-pixel thick boundary is used in M. Oliveira, B. Bowen, R. McKenna, and Y.-S. Chang, "Fast Digital Image Inpainting," In Proc. VHP 2001, pp. 261-266, [CITY]: [PUB], 2001; a narrow band of pixels is used in A. Telea, "An Image Inpainting Technique Based on Fast Marching Method," Journal of Graphics Tools, vol. 9, no. 1, Association of Computing Machinery (ACM) Press, 2004; and a fill front is used in A. Criminisi, P. Perez, and K. Toyama, "Region Filling and Object removal by Exemplar-Based Image Inpainting," Institute of Electrical and Electronics Engineers (IEEE) Trans, on Image Processing, Vol. 13, No. 9, pp.1200-1212, September 2004.

In the prior art, different variations of a priority queue or heap are used to control the pixel processing in the order of the priorities. A priority queue is a set of first-in, first-out (FIFO) queues containing elements with the same priority. When an element with priority p is added to the priority queue, the element is placed at the tail of the FIFO queue for priority p. If an element is to be removed from the priority queue, the head element of the highest priority FIFO queue is removed.

A typical known inpainting scheme is as follows. First, a priority is computed for each border pixel of the unknown area and these pixels are added to the priority queue. The unknown pixel with the highest priority is then removed from the priority queue and its value is restored using the values of neighboring known pixels. The restored pixel is marked as a known pixel, and the neighboring internal unknown pixels become border pixels. A priority is computed for these new border pixels and they are added to the priority queue. The steps after the first one above are repeated until all pixels in the unknown area become known.

Different inpainting algorithms are known in the art. Each performs explicit modeling of the border of the unknown image area and uses the priority queue to select unknown pixels that are ready for restoration. Pixel priority computation is tightly integrated with the border modeling in algorithms such as disclosed in M. Bertalmio, A.L. Bertozzi, and G. Sapiro, "Navier- Stokes, Fluid Dynamics, and Image and Video Inpainting," Proc. ICCV 2001, pp. 1335-1362, 2001, and A. Telea, "An Image Inpainting Technique Based On The Fast Marching Method," Journal of Graphics Tools, Vol. 9, No. 1, ACM Press, 2004. One popular method of pixel priority computation is the Fast Marching Method as disclosed in J.A. Sethian, "Level Set Methods and Fast Marching Methods," Second Edition, Cambridge University Press, Cambridge, UK, 1999. It computes the distance between an unknown pixel and the border of the unknown area along the normal vector to the border line and uses the negative distance as a pixel priority.

High quality digital inpainting algorithms often use continuous functions to represent pixel priority. The use of a priority queue with priority values represented as floating point numbers results in a very computationally expensive algorithm. Explicit modeling of the border of a two-dimensional area is also slow. Both of these techniques process individual values and are difficult to optimize using single instruction multiple data (SIMD) instructions of modern computer systems. Hence, current digital inpainting techniques are too slow, and advances in computational efficiency of digital inpainting are desired. BRIEF DESCRIPTION OF THE DRAWINGS

The features and advantages of the present invention will become apparent from the following detailed description of the present invention in which:

Figure 1 is a diagram of a digital inpainting system according to an embodiment of the present invention; Figure 2 is a flow diagram illustrating digital inpainting without a priority queue or border modeling in accordance with an embodiment of the present invention; Figure 3 is a diagram of a sample source image. Figure 4 is a diagram of the sample source image as damaged. Figure 5 is a diagram of a sample inpainting mask. Figure 6 is a diagram of a sample restored image according to an embodiment of the present invention; and

Figure 7 is a diagram of a sample inpainting mask for parallel computation by four parallel processes according to an embodiment of the present invention.

DETAILED DESCRIPTION An embodiment of the present invention comprises a digital inpainting system having a pixel priority computation process. Embodiments of the present invention eliminate traditionally used algorithms for explicit border modeling of the unknown image area and the priority queue. These algorithms are replaced by a much more efficient computation of pixel priorities as a distance transform and by unknown pixel sorting. A new process for Fast Marching Method (FMM) computation as a distance transform function is disclosed. The resulting digital inpainting speed may be increased by at least an order of magnitude in comparison with known techniques, with the same or better inpainting quality.

Embodiments of the present invention make the inpainting process more flexible. It allows for complete separation of the pixel priority computation and the pixel value restoration processes. These two parts of the inpainting process can now be performed independently using selected algorithms in either software or hardware implementations. Decomposition of the inpainting process to simple steps facilitates low-level optimization through the use of SIMD instructions and by parallel implementation of the digital inpainting process. The increase of inpainting speed allows embodiments of the present invention to be used in real-time movie processing and other applications.

Reference in the specification to "one embodiment" or "an embodiment" of the present invention means that a particular feature, structure or characteristic described in connection with the embodiment is included in at least one embodiment of the present invention. Thus, the appearances of the phrase "in one embodiment" appearing in various places throughout the specification are not necessarily all referring to the same embodiment.

Figure 1 is a diagram of a digital inpainting system according to an embodiment of the present invention. A source image 100 includes at least one unknown area 102. The source image is a two dimensional digital image having a plurality of pixels. A first dimension of the image array may be designated by N, and a second dimension may be designated by M. The size of the source image is therefore N x M pixels. The size of a pixel is implementation dependent (e.g., 8 bits, 16 bits, 24 bits, etc.). In embodiments of the present invention, each pixel may store additional information such as known/unknown status and priority. The source image may be stored in a memory as part of or readable by any digital processing system. The source image may have been previously captured by a digital camera, by a film camera and subsequently digitally scanned, or may be an image digitally created by a digital processing system. The unknown area 102 refers to a plurality of pixels that cause the source image to be damaged in some way (perhaps aesthetically), or otherwise denote pixels that need to be modified to improve the appearance of the source image. The unknown area is called a mask and denoted Ω. The mask may be determined manually by user interaction with an image processing application program or automatically using a mask detection algorithm embodied in the image processing application program. There may be one or more unknown areas per source image, each unknown area being processed individually in turn. The unknown areas are typically small as compared to the overall size of the source image.

Figure 3 is an example of a source image without damage. Figure 4 is an example of the source image as damaged (i.e., the words 'fast digital inpainting' have been written into the source image by "whiting out" some of the pixels). In this example, the unknown areas (i.e., the mask) include the words "fast digital inpainting."

Turning back to Figure 1, according to embodiments of the present invention, digital inpainting system 104 may be used to process the pixels of the source image to remove the unknown areas and produce restored image 110. Digital inpainting system

104 may be implemented in software, hardware, or any combination of the two, in any digital processing system. Digital inpainting system comprises at least two components.

Pixel priority computation module 106 computes the priority for all pixels of the unknown area of the source image. Pixel restoration module 108 restores the pixels of the unknown area according to the computed priorities to produce the restored image.

Generally, digital inpainting system 104 processes the source image to produce the restored image as follows. First, in an embodiment of the present invention, a pixel priority may be computed for each pixel of the unknown area using a distance transform approach (raster-antiraster scanning). A new process for computation of the Fast Marching Method (FMM) using a distance transform technique may be used to compute the pixel priorities. The FMM distance may be computed for each pixel of the unknown area and neighboring pixels. Next, all unknown pixels may be sorted based on their priority value. While sorting is being performed, each pixel in the set of unknown pixels may be associated with one of a plurality of sorted subsets for parallel processing during pixel value restoration processing. The values of the unknown pixels may then be restored in descending order (in one embodiment) of their priorities and in parallel by sets. In one embodiment, pre-computed values of image and priority gradients may be used to accelerate these calculations. Embodiments of the present invention do not require the use of a priority queue and do not perform explicit modeling of the unknown area border (in marked contrast to the prior art, which emphasizes the important role of the border modeling).

Figure 2 is a flow diagram illustrating digital inpainting without a priority queue according to an embodiment of the present invention. At block 200, for each of the pixels of the unknown area of the source image, a status may be set to unknown. At block 202, for each pixel of the source image outside the unknown area, a status may be set to known. At block 204, a priority for each unknown pixel may be computed using the distance transform technique disclosed herein. The pixel status and priority values may be stored along with the color values of each pixel of the image in a memory according to well known techniques. In one embodiment, block 204 may be performed by pixel priority computation module 106 of Figure 1. At block 206, the unknown pixels may be sorted in descending order of priority. In one embodiment, each of the unknown pixels may be associated into a selected one of a plurality of sets for subsequent parallel processing. At block 208, for each unknown pixel in order of priority, the pixel value may be restored based, at least in part, on the neighboring known pixels. In one embodiment, the unknown pixels may be processed in order of descending priority. In one embodiment, the restoration of the pixel values may be performed on the sets in parallel. The status of each unknown pixel may be changed to known after each unknown pixel is processed at block 208. In one embodiment, block 208 may be performed by pixel restoration module 108 of Figure 1. In one embodiment, blocks 200, 202, and 206 may be performed by logic implemented in digital inpainting system 104. In another embodiment, implementation of blocks 200- 208 may be combined into one or more logical components within digital inpainting system 104. In this way all unknown pixels may be modified to become known pixels and the source image is thus restored.

The distance transform technique is a popular approach in image processing, as disclosed in G. Borgefors, "Distance Transformations in Digital Images," Comp. Vision, Graphics and Image Proc, 34:3, pp. 344-371, 1986. It allows for calculating functions with local dependency using raster-antiraster scanning. The FMM distance for neighboring pixels is often used in pixel value restoration algorithms. However, the distance transform technique has never before been applied to FMM distance computation for purposes of determining pixel priority.

In the prior art the FMM or other distance f(Iij ) for a pixel of image I is calculated from four pairs of neighboring pixels

) , (Ij₊I j₁IjJ-I) , (Ij₊IjJj₁J₊I) , (Ii-IJ Ji, j+\) when pixel Ijj becomes the border pixel of decreasing area Ω and is added to the priority queue. In contrast, FMM computation using the distance transform paradigm according to an embodiment of the present invention is as follows. Let I be the image with N rows and M columns (i.e., N x M pixels), mask area Ω cz / contains unknown pixels to restore, the Euclidean k-neighborhood of Ω is O_k(Ω) = {x e I\ Ω : 3y e Ω,\ x -y \≤ k}, md the border SΩ = O_\ (Ω) . The FMM distance f(x) for mask area Ω is the solution of the equation

x e Ω and T( x) = 0, x e dΩ . In the digital inpainting process, /fxj is calculated for all pixels x e O)t f Ω^ . It is positive for x e Ω , zero for χ e 9Ω , and negative for xe Ofc (Ω) \ dΩ . In embodiments of the present invention, the FMM distance for all pixels is calculated by performing the algorithm of Table I (where Ifj is the pixel in i-th row and j-th column of the image). This process does not require explicit modeling of 9Ω or special structures for it. Only a mask image for area Ω is used. The neighborhood O^ (Ω) can be built from mask area Ω by a morphology dilate operation or modeled implicitly using the fact that f(x) ≤k -1 for all x e O^ (Ω) . Dilation of the binary image I with the neighborhood O of the point is the operation that transforms the set (area) Ω c / to its neighborhood O(Ω) = (J O(x) . Efficient algorithms of dilation are widely used xeΩ in the computer vision domain. Table I presents pseudocode for computation of the FMM distance in the unknown area and its neighborhood as performed by block 204 of Figure 2. The symbol "I" represents the image containing the mask Ω ₅ the symbol "F" represents the image of priority values (FMM distance) and N and M are the dimensions of the image. The result of the procedure comprises a priority for each pixel in the unknown area. Table I

FastMarching (I, F ) { Il I is the input mask image // F is the output Fast Marching distance image for i=l to N forj=l to M calculate (I, F,i, j -1,-1 j for j=M to 1 update (I, F, Uj -U) for i=N-l to l for j=l to M update ( I, F, i,j, 1,-1) for j=M to 1 update ( I, F, i,j, 1,1) }

calculate (I, F, i, j, k,l) { if Ii, e a

if I_{i+k j}, Ii _j+ι e O_k(Ω)

pdate if I_f

else i

}

The FMM distance computation code implementing the pseudocode of Table I may be vectorized using SIMD instructions of modern computer systems (e.g., MMX, SSE, SSE2 instructions).

Figure 5 is a diagram of a sample inpainting mask generated according to an embodiment of the present invention. The mask is the set of unknown area pixels. The percentage of unknown area pixels to the overall source image is generally small (perhaps 5-15% of the source image), so performance of the pixel sorting step at block 206 as part of the entire digital inpainting process is small (perhaps 5-10% of overall execution time).

While sorting, all unknown pixels may be easily split into a plurality of sorted subsets with a fixed distance between each other. Each subset may be inpainted in parallel, and then pixels connecting subsets may be inpainted sequentially. This facilitates an implementation of parallel processing for digital inpainting on dual-core and multi-core computer systems. Figure 7 illustrates one of the possible ways of splitting the pixels. The top left quadrant, the top right quadrant, the bottom left quadrant, and the bottom right quadrant of the inpainting mask may be processed in parallel. Then, the parts of the inpainting mask forming the boundaries between the quadrants may be processed sequentially. In one embodiment, these boundaries parts may be 3 to 7 pixels in width. In other embodiments, the number of sets in the inpainting mask may vary (e.g., four, eight, 16, 32, and so on). Embodiments of the present invention allow for the complete splitting of the two parts of the inpainting process into separate modules: a pixel priority computation module and a pixel value restoration module, as shown in Figure 1. Thus, different algorithms may be selected for the pixel priority computation and pixel inpainting processes. In one embodiment of the present invention, the FMM distance is implemented for pixel priority and one of several different algorithms of computation of the unknown pixel value for restoration may be used (e.g., the Telea algorithm disclosed in A. Telea, "An Image Inpainting Technique Based on Fast Marching Method," Journal of Graphics Tools, vol. 9, no. 1, Association of Computing Machinery (ACM) Press, 2004, and the algorithm based on Navier-Stokes equation disclosed in M. Bertalmio, A.L. Bertozzi, and G. Sapiro, "Navier-Stokes, Fluid Dynamics, and Image and Video Inpainting," Proc. ICCV 2001, pp. 1335-1362, 2001). These algorithms do not restore image texture. But the use of the FMM distance transform technique as described herein allows for the exclusion of the priority queue and explicit border modeling from these algorithms for iterative image restoration or image texture restoration.

Figure 6 is a diagram of a sample restored image according to an embodiment of the present invention. All unknown areas have been processed and restored, resulting in the restored image.

Embodiments of the present invention eliminate the two most computationally intensive parts of the prior art techniques: 1) use of the priority queue; and 2) explicit modeling of the border of the unknown image area. This results in an acceleration of the speed of inpainting by more than an order of magnitude. Embodiments of the present invention include the method of computation of the Fast Marching Method distance using the distance transform paradigm instead of calculating it while explicit border modeling as in the prior art. Use of the distance transform approach facilitates optimization using SIMD instructions and parallel implementation of the inpainting algorithm for dual-core and multi-core computer systems.

Embodiments of the present invention facilitate inpainting of the same pattern in successive frames of a movie. Unknown pixel priorities can be calculated once and than can be used to inpaint all frames of a film. The separation of processing of the pixel priority computation and the pixel value restoration allows for on-the-fly elimination of the same image pattern from frames of live movies.

Although the following operations described herein may be described as a sequential process, some of the operations may in fact be performed in parallel or concurrently. In addition, in some embodiments the order of the operations may be rearranged. The techniques described herein are not limited to any particular hardware or software configuration; they may find applicability in any computing or processing environment. The techniques may be implemented in hardware, software, or a combination of the two. The techniques may be implemented in programs executing on programmable machines such as mobile or stationary computers, personal digital assistants, set top boxes, cellular telephones and pagers, and other electronic devices, that each include a processor, a storage medium readable by the processor (including volatile and non-volatile memory and/or storage elements), at least one input device, and one or more output devices. Program code is applied to the data entered using the input device to perform the functions described and to generate output information. The output information may be applied to one or more output devices. One of ordinary skill in the art may appreciate that the invention can be practiced with various computer system configurations, including multiprocessor systems, minicomputers, mainframe computers, and the like. The invention can also be practiced in distributed computing environments where tasks may be performed by remote processing devices that are linked through a communications network.

Each program may be implemented in a high level procedural or object oriented programming language to communicate with a processing system. However, programs may be implemented in assembly or machine language, if desired. In any case, the language may be compiled or interpreted.

Program instructions may be used to cause a general-purpose or special-purpose processing system that is programmed with the instructions to perform the operations described herein. Alternatively, the operations may be performed by specific hardware components that contain hardwired logic for performing the operations, or by any combination of programmed computer components and custom hardware components. The methods described herein may be provided as a computer program product that may include a machine accessible medium having stored thereon instructions that may be used to program a processing system or other electronic device to perform the methods. The term "machine accessible medium" used herein shall include any medium that is capable of storing or encoding a sequence of instructions for execution by a machine and that cause the machine to perform any one of the methods described herein. The term "machine accessible medium" shall accordingly include, but not be limited to, solid-state memories, optical and magnetic disks, and a carrier wave that encodes a data signal. Furthermore, it is common in the art to speak of software, in one form or another (e.g., program, procedure, process, application, module, logic, and so on) as taking an action or causing a result. Such expressions are merely a shorthand way of stating the execution of the software by a processing system cause the processor to perform an action of produce a result.

Claims

CLAIMSWhat is claimed is:

1. A method of digital inpainting of an area of a source image comprising: setting a status of each pixel of the area to unknown; setting a status of each pixel of the source image outside of the area to known; computing a priority value for each unknown pixel using a distance transform process; sorting the unknown pixels in order of priority value; and for each unknown pixel in order of priority, restoring a pixel value using, at least in part, neighboring known pixel values, and changing the unknown pixel's status to mown.

2. The method of claim 1, further comprising associating each unknown pixel with a selected one of a plurality of sets after sorting, and performing the restoring of the pixel values and the changing of the pixel status of unknown pixels of the sets in parallel.

3. The method of claim 1, wherein computing the priority value for each unknown pixel using a distance transform process eliminates modeling of border pixels of the area.

4. The method of claim 1, wherein computing the priority value for each unknown pixel using a distance transform process eliminates using a priority queue to control priority of unknown pixels of the area.

5. The method of claim 1, wherein computing the priority value for each unknown pixel using a distance transform process comprises applying a Fast Marching Method (FMM) computation as a distance transform function.

6. An article comprising: a tangible machine accessible medium containing instructions, which when executed, result in digital inpainting of an area of a source image by setting a status of each pixel of the area to unknown; setting a status of each pixel of the source image outside of the area to known; computing a priority value for each unknown pixel using a distance transform process; sorting the unknown pixels in order of priority value; and for each unknown pixel in order of priority, restoring a pixel value using, at least in part, neighboring known pixel values, and changing the unknown pixel's status status to known.

7. The article of claim 6, further comprising instructions to associate each unknown pixel with a selected one of a plurality of sets after sorting, and to perform the restoring of the pixel values and the changing of the pixel status of unknown pixels of the sets in parallel.

8. The article of claim 6, wherein instructions to compute the priority value for each unknown pixel using a distance transform process eliminates modeling of border pixels of the area.

9. The article of claim 6, wherein instructions to compute the priority value for each unknown pixel using a distance transform process eliminates using a priority queue to control priority of unknown pixel processing.

10. The article of claim 6, wherein instructions to compute the priority value for each unknown pixel using a distance transform process comprise instructions to apply a

Fast Marching Method (FMM) computation as a distance transform function.

11. A digital inpainting system for processing a source image having a damaged area into a restored image comprising: logic adapted to set a status of each pixel of the area to unknown, and to set a status of each pixel of the source image outside of the area to known; a pixel priority computation module to compute a priority value for each unknown pixel using a distance transform process; logic adapted to sort the unknown pixels in order of priority value; and a pixel restoration module to process each unknown pixel in order of priority, restoring a pixel value using, at least in part, neighboring known pixel values, and changing the unknown pixel's status to known.

12. The system of claim 11, further comprising logic to associate each unknown pixel with a selected one of a plurality of sets after sorting, the pixel restoration module performing the restoring of the pixel values and the changing of the pixel status of unknown pixels of the sets in parallel.

13. The system of claim 11, wherein the pixel priority computation module is adapted to compute the priority value for each unknown pixel using distance transform process and eliminate modeling of border pixels of the area.

14. The system of claim 11, wherein the pixel priority computation module is adapted to compute the priority value for each unknown pixel using a distance transform process and eliminate using a priority queue to control priority of unknown pixel processing.

15. The system of claim 11, wherein the pixel priority computation module is adapted to compute the priority value for each unknown pixel using a distance transform process by applying a Fast Marching Method (FMM) computation as a distance transform function.

16. A computer-implemented method of computing priority values of pixels of an area of a digital image comprising: setting a status of each pixel of the area to unknown; setting a status of each pixel of the digital image outside of the area to known; computing a priority value for each unknown pixel by applying a Fast Marching Method (FMM) computation as a distance transform function.

17. The computer-implemented method of claim 16, further comprising storing the priority value with the unknown pixel.

18. The computer-implemented method of claim 16, further comprising sorting the unknown pixels in order of priority value.

19. The computer-implemented method of claim 18, further comprising associating each unknown pixel with a selected one of a plurality of sets after sorting.

20. An article comprising: a tangible machine accessible medium containing instructions, which when executed, result in computing priority values of pixels of an area of a digital image by setting a status of each pixel of the area to unknown; setting a status of each pixel of the digital image outside of the area to known; computing a priority value for each unknown pixel by applying a Fast Marching Method (FMM) computation as a distance transform function.