KR20180017737A - Laplacian patch-based image synthesis method and apparatus therefor - Google Patents

Abstract

A Laplacian patch-based image synthesis method and apparatus are disclosed. An image synthesis method according to an aspect of the present invention includes a step of generating a Laplacian pyramid for an input image; a step of approximating a second order differential property at each level using the generated Laplacian pyramid through difference of Gaussian; and a step of synthesizing the image of a target region in an input image using the second order differential property at each approximated level.

Description

[0001] Laplacian patch-based image synthesis method and apparatus [0002]

The present invention relates to a Laplacian patch-based image synthesis technique, and more particularly, to a Laplacian-based image synthesis technique using a Laplacian pyramid to approximate a second-order differential property at each scale through DoG (Difference of Gaussian) And more particularly, to a patch-based image synthesis method and apparatus.

When people are dealing with digital photos, they are often faced with situations such as unwanted objects in the picture, areas that are obscured, or image corruption due to transmission errors. In this case, people want to repair the damaged areas in a plausible way. To solve this problem, the inpainting algorithm, which is an image that fills in the damaged area, has received a lot of attention in the past few years. Although the image-in-painting algorithm is primarily used for beginners, no algorithms are available to produce robust results under any circumstances.

The existing in-painting algorithm used an image gradient that is the first derivative of the image intensity to detect the image structure and copy (or fill) in from the boundary. The first order differential measures the directional change of the intensity around the edge. And image pyramids that represent image representations based on multi-scale signals are widely used to improve structural consistency when trying to complete a missing region in patch-based synthesis. At each level of the recent image pyramid, the image gradient is used to improve the edge structure of the synthesized image, as well as the consistent image composition. Recently, the approach of combining a gradient with an image pyramid has improved the detail and structural consistency of in-painting, but since the gradient operation is directional, it requires computation twice per direction and requires a very heavy operation to solve the Poisson equation. . In addition, artifacts arise when filling non-integrable first derivative regions.

The Laplacian operation, which is the divergence of the image gradient, has the advantage that it is rotation-free and has no directionality. Also, the Laplacian values are well aligned with the edges and are suitable for representing the image structure near the edge. The Laplacian pyramid decomposes the basic structure and detail structure of an image into frequency components. These data structures are used in various applications such as image blending / fusion, enhancement, and dinoing.

To explain the inpainting algorithm more in detail, the inpainting algorithm can be largely divided into diffusion-based and example-based methods.

The diffusion method uses a smoothness constraint that enhances the local connection structure by extending the geometry of the adjacent region and filling the empty region. Most diffusion methods work by solving partial differential equations locally, but there is also a local optimization method that minimizes the variation of the entire filled area. Diffusion inpainting seems to be efficient in reconstructing lines, curves, and small holes, but blurring artifacts can occur when filling large holes.

An example-based approach has been proposed as an extension of the diffusion method. In this method, first a certain target patch (a patch in a blank area) is to be filled in advance with respect to an arbitrary blank area (target area), and a candidate patch In an image area outside the target area. Finally, the candidate patch is synthesized to the target patch position. Since this approach has been proposed, a global optimization method and various modification algorithms have been proposed. For example, there is a method of propagating the structure based on user input, a method of determining the filling order by using the tensor-based data term, and the method of determining the filling order by using the tensor-based data term. The candidate patches expressed in the nearest neighbor field are searched after the local glowing algorithm. Also, a method of greatly reducing the amount of computation using a random search method has been proposed, and this method is widely used for searching for corresponding points. Priority reliability belief propagation has also been proposed for patch synthesis through discrete global optimization.

An existing patch-based painting algorithm that calculates similarity in image pyramids, another in-painting algorithm, produces a visually pleasing result, but some results have an incoherent structure due to lack of spatial consistency. Recently, it has been tried to improve the structural consistency by utilizing the image gradients at each level, but some methods have inherent limitations of the image gradients, and because the image gradients introduce horizontal and vertical intensity changes, the Poisson equation must be solved for integration . In addition, these partial differential equation based solutions often generate artifacts for gradient fields that are not integrable.

Another in-painting algorithm is to improve the patch-based search algorithm by using a mask that considers the background and foreground. This greatly reduces the artifacts generated by the existing patch synthesis algorithm. Patch-based synthesis and the Laplacian pyramid have been extensively used in the last decade, but only a few studies have sought to combine these two areas.

In this way, the existing base image synthesis studies have not yet studied intensively the usability of the Laplacian pyramid.

Embodiments of the present invention provide a method and apparatus for image synthesis based on a Laplacian patch that can approximate a second-order differential property at each scale using a Laplacian pyramid through a Difference of Gaussian (DoG) and synthesize an image using the Laplacian pyramid do.

According to an aspect of the present invention, there is provided an image synthesis method comprising: generating a Gaussian pyramid and a Laplacian pyramid for an input image; Synthesizing a target region in the input image using the Gaussian pyramid and the Laplacian pyramid; And performing the image synthesis by sequentially repeating the image synthesis for each level of the Gaussian pyramid and the Laplacian pyramid.

The generating step may generate the Laplacian pyramid using the upsampling Gaussian pyramid upsampled with the Gaussian pyramid and the Gaussian pyramid.

Wherein image compositing of the target region comprises searching for source patches for each of the target patches of the target region using the Gaussian pyramid and the Laplacian pyramid, wherein the source patches are patches outside the target region, By updating the pixels of each of the target patches using the retrieved source patches, the target region can be image-synthesized.

The image compositing of the target region may retrieve nearest-neighbor source patches for each of the target patches using the Gaussian pyramid and the Laplacian pyramid.

The image compositing of the target region may update pixels of each of the target patches based on a weight based on the distance of each of the target patches at the boundary of the retrieved source patches and the target region.

The image compositing of the target region may update pixels of each of the target patches based on the similarity of the retrieved source patches and the target patches and the confidence weights at the target pixels of each of the target patches.

The step of performing the image composition of the target area may include repeating the step of filling the target area with the image synthesis result of the target area at the previous level and synthesizing the image.

According to another aspect of the present invention, there is provided an image synthesis method comprising: generating a Laplacian pyramid for an input image; Approximating the second order differential properties at each level using the generated Laplacian pyramid through Difference of Gaussian; And synthesizing an image of a target region in the input image using the second derivative at each approximated level.

The generating may generate the Laplacian pyramid using a Gaussian pyramid for the input image and an upsampling Gaussian pyramid upsampled to the Laplacian pyramid.

Wherein image compositing of the target region comprises: applying source patches for each of the target patches of the target region using a Gaussian pyramid for the input image and the Laplacian pyramid, wherein the source patches are patches outside the target region; , And image combining the target region by updating pixels of each of the target patches using the searched source patches.

The image compositing of the target region may retrieve nearest-neighbor source patches for each of the target patches using the Gaussian pyramid and the Laplacian pyramid.

The image compositing of the target region may update pixels of each of the target patches based on a weight based on the distance of each of the target patches at the boundary of the retrieved source patches and the target region.

The image compositing of the target region may update pixels of each of the target patches based on the similarity of the retrieved source patches and the target patches and the confidence weights at the target pixels of each of the target patches.

The image compositing of the target area may perform the image compositing of the target area by filling the target area with the image synthesis result of the target area at the previous level and sequentially repeating the image combining result at each level.

According to an aspect of the present invention, there is provided an image synthesizing apparatus including: a generating unit generating a Laplacian pyramid for an input image; An approximation unit for approximating the second derivative at each level through a Gaussian difference using the generated Laplacian pyramid; And a synthesis unit for synthesizing an image of a target region in the input image using the second derivative at each approximated level.

According to the embodiments of the present invention, it is possible to avoid the problem of synthesizing the first-order differential compound image by calculating the difference of Gaussian (DoG), approximating the second derivative, and performing image synthesis using the second derivative.

Also, according to the embodiments of the present invention, by combining an image using a second differential patch, edge structures in different directions can be detected well as compared with a first derivative having a large edge direction dependence.

In addition, according to embodiments of the present invention, an image compositing algorithm for filling a blank portion in an image can be applied to various applications.

For example, many people, automobiles, etc. are photographed on a goggle road image, which is related to privacy invasion. After detecting an object and emptying the sensed area, Filling can solve the problem of privacy invasion.

It can also be used to fix damaged areas if the image is damaged by wrinkling or dirt, or you can fill in the initial sketch with image comple- tion when you design new parts that you do not like.

As described above, according to the present invention, an image including a region that has not been shot well for some reason, an image including an area remaining as a blank area in the background as a result of removing a foreground object, You can naturally fill an empty area within an image with images you include.

In addition, according to the embodiments of the present invention, since it is possible to reduce the time required for image compilation while maintaining thecompletion quality as compared with the conventional image complementation, users can easily and quickly use the image.

FIG. 1 illustrates an example of an image synthesizing apparatus according to an embodiment of the present invention. Referring to FIG.
FIG. 2 is a block diagram illustrating an internal configuration of a computing device including an image synthesizing apparatus according to the present invention.
Fig. 3 shows an example for explaining Gaussian and Gaussian difference (DoG).
4 is a conceptual diagram of an edge-recognition corresponding point search in the Laplacian pyramid of the present invention.
FIG. 5 is a conceptual diagram of a Laplacian patch-based image synthesis method according to an embodiment of the present invention.
6 illustrates a Laplacian based image modification algorithm in accordance with an embodiment of the present invention.
7 is a flowchart illustrating an operation of an image synthesizing method according to an embodiment of the present invention.
FIG. 8 is a block diagram illustrating a configuration of an image synthesizing apparatus according to an embodiment of the present invention.

Hereinafter, embodiments according to the present invention will be described in detail with reference to the accompanying drawings. However, the present invention is not limited to or limited by the embodiments. In addition, the same reference numerals shown in the drawings denote the same members.

FIG. 1 illustrates an example of an image synthesizing apparatus according to an embodiment of the present invention. Referring to FIG.

As shown in FIG. 1, the image synthesizing apparatus according to the present invention can be mounted or installed inside the computing device 100, and can fill a vacant certain area naturally in an input image in which a certain area is empty .

Here, the input image must be filled with an image including a region that has not been photographed for some reason, an image including an area that remains as a blank area in the background as a result of removing the foreground object, an image including a damaged area for unknown reasons, You can include all sorts of images that have regions.

The computing device 100 shown in FIG. 1 may include any device capable of installing or mounting an image compositing algorithm according to the present invention, for example, a smart phone capable of installing software such as Photoshop ), A computer, a notebook, a personal digital assistant (PDA), a tablet PC, and the like.

Such a computing device may communicate with other electronic devices and / or servers over a network using a wireless or wired communication scheme.

The communication method is not limited, and may include a communication method using a communication network (for example, a mobile communication network, a wired Internet, a wireless Internet, a broadcasting network) that the network may include, as well as a short-range wireless communication between the devices. For example, the network may be a personal area network (LAN), a local area network (LAN), a campus area network (CAN), a metropolitan area network (MAN), a wide area network (WAN), a broadband network Lt; RTI ID = 0.0 > and / or < / RTI > The network may also include, but is not limited to, any one or more of a network topology including a bus network, a star network, a ring network, a mesh network, a star bus network, a tree or a hierarchical network, .

FIG. 2 is a block diagram illustrating an internal configuration of a computing device including an image synthesizing apparatus according to the present invention.

2, the computing device 100 may include a memory 210, a processor 220, a communication module 230, and an input / output interface 240.

The memory 210 may be a computer-readable recording medium and may include a permanent mass storage device such as a random access memory (RAM), a read only memory (ROM), and a disk drive. Also, the operating system and at least one program code may be stored in the memory. Such software components may be loaded from a computer readable recording medium separate from the memory using a drive mechanism. Such a computer-readable recording medium may include a computer-readable recording medium such as a floppy drive, a disk, a tape, a DVD / CD-ROM drive, and a memory card. In other embodiments, the software components may be loaded into memory via a communication module other than a computer-readable recording medium. For example, at least one program may be loaded into a memory based on a file distribution system that distributes installation files of developers or applications, for example, a program installed by files provided by a server over a network.

The processor 220 may be configured to process instructions of a computer program by performing basic arithmetic, logic, and input / output operations. The command may be provided to the processor by a memory or communication module. For example, the processor may be configured to execute instructions received in accordance with program code stored in a recording device, such as a memory. Here, the processor 220 may be constituent means for performing the image composition method of the present invention.

The communication module 230 may provide a function for communicating with other electronic devices or servers via the network.

The input / output interface 240 may be a means for interfacing with the input / output device 250. For example, the input device may include a device such as a keyboard or a mouse, and the output device may include a device such as a display for displaying a communication session of the application. As another example, the input / output interface may be a means for interfacing with a device having integrated functions for input and output, such as a touch screen.

Further, in other embodiments, a computing device may include more components than the components of FIG. However, there is no need to clearly illustrate most prior art components. For example, the computing device may be implemented to include at least some of the input / output devices described above or may further include other components such as speakers, microphones, various sensors, and the like.

The present invention can be installed and executed in such a computing device, and a method and an apparatus according to embodiments of the present invention performed in a computing device will be described below with reference to FIGS. 3 to 8. FIG.

The present invention utilizes a laplacian pyramid to improve structural consistency in image synthesis. The closest neighbor search (or search) in the Laplacian pyramid is unaffected by the rotation of the propagated structure information, so it can improve the accuracy of the correspondence search compared to existing methods. Further, since the present invention uses an upsampling Gaussian image and a Laplacian image, robust results can be obtained with respect to changes in noise and parameters. In the description of the present invention, an edge-aware (or edge-end) image synthesis technique is described.

1. Laplacian Coherent Spaces

Gaussian, Gradient, and Laplacian: Image pyramids provide multi-resolution representation of images and are used in many applications. The parameters are generated in two steps: filtering and sampling.

은 라플라시안 커널

로 필터링된 이미지와 동일하다. 필터링에 의한 신호를 다운샘플링함으로써, 이미지 피라미드를 형성한다.As shown in Fig. 3, the Gaussian function (a), the Gaussian gradient (b) and the gradient divergence (c) are usually used for filtering. Since convolutional filtering is a linear operation, the Laplacian of the Gaussian filtered image

The Laplacian kernel

Is the same as the filtered image. By downsampling the signal by filtering, an image pyramid is formed.

와 라플라시안

은 이미지에서 에지 구조를 검출하는데 주로 사용된다. 그래디언트는 x와 y의 일차 편미분으로부터 계산된다. 즉,

이 된다. 적어도 두 개의 연산자는 이미지에서 로컬 구조의 변화를 검출하기 위해 필요하다. 도 3에 도시된 바와 같이, 그래디언트는 각 방향에 대한 다중 연산자를 요구하는 방향성 에지 검출기이다, 따라서, 그래디언트 크기(magnitude)는 에지를 검출하는 용도로 사용된다. In particular,

And Laplacian

Is mainly used to detect the edge structure in the image. The gradient is computed from the first-order partial derivatives of x and y. In other words,

. At least two operators are needed to detect changes in the local structure in the image. As shown in Figure 3, the gradient is a directional edge detector that requires multiple operators for each direction, so the gradient magnitude is used to detect the edge.

는 이미지에서 함수의 회전에 불변하는 등방성 에지 검출기이다. Unlike the gradient, the Laplacian operator

Is an isotropic edge detector that is invariant to the rotation of the function in the image.

The Laplacian pyramid stores the bandpassed structure information of each frequency band as shown in FIG. 4A. Pyramids are widely used for various edge-aware image processing.

Computing the Laplacian: Since convolution with a large weighting function requires a large amount of computation, the Laplacian of Gaussian (LoG) of Gaussian, as shown in Figures 3D to 3F, Can be simply approximated by a Difference of Gaussian (DoG). The image scale of level (l + 1) is reduced to half scale of level 1 in Gaussian pyramid, and DoG can be approximated with LoG with high accuracy. Figures 3c and 3f compare the similarity of LoG and DoG. Because the backward calculation of a gradient requires solving the Poisson equation, it is computationally intensive, and unwanted artifacts can be caused by unintegrated gradient fields. Conversely, the forward and backward computation of Laplacian using DoG is very efficient for subtraction and summation operations, and the Laplacian pyramid has excellent potential for patch-based image synthesis. For this reason, the present invention does not require the additional computational expense to be computed in existing patch-based synthesis methods, such as solving the Poisson equation.

2. Patch-based synthesis of a laplacian pyramid (Laplacian pyramid)

The choice of the pyramid kernel operator is critical to the information that leads to the image. The Gaussian operator is effective in determining the basic structure at each level of frequency. This operator is often used in many image optimization algorithms to achieve spatial consistency in searching correspondences and aggregating similarities proposed in various existing algorithms. Contrary to Gaussian, the Gradient and Laplacian operators can find the edge structure of each level of an image. The pyramid element of the derivative is preserved in the local domain in the spatial domain of the gradient or Laplacian. Differential image pyramids are decomposed into edge localization at each level of frequency.

Recently, in conventional methods, there has been introduced a corresponding point search (or search) for checking colors and gradients in an image pyramid. The first differential inspection is very beneficial in detecting the correspondence points of structural consistency in image comple- tion. The present invention differs from the conventional method in two points, namely, the corresponding point and the rotation invariance.

Building a Laplacian Pyramid: The input to the example-based image completion is the color image I and the mask image M. Here, the mask image M separates the source region S and the target region T from each other. The purpose of the example-based image compilation is to fill the target area T of the image I with the contents of the source area S. In order to use a Laplacian pyramid for painting in an image, the present invention first constructs a Gaussian pyramid G for an image I as: < EMI ID = 1.0 >

[Equation 1]

Here, G _i means the i th Gaussian pyramid image (or the i th scale of the Gaussian pyramid G), the number of images (or the total number of scales) of the Gaussian pyramid is n + 1 and the down sample (blurred ) &Lt; / RTI &gt; image (or a filtered scale). The finest level in the Gaussian pyramid is G ₀ , the same as the original image I. The i-th Gaussian scale G _{i +1} can be obtained by performing Gaussian filtering and partial sampling on the previous scale G _i .

Then, the Laplacian pyramid L is calculated as the difference (DoG) of the Gaussian pyramid, and can be expressed as Equation (2) below.

&Quot; (2) &quot;

Where upsample () is the upsampling operator. To briefly describe the algorithm of the present invention, an upsampled Gaussian pyramid U is defined. The i-th Laplacian image L _i is a detail structure existing between G _i and G _{i +1} . Since the number of Gaussian levels in a Gaussian pyramid is limited, the coplanar level of the Laplacian pyramid L _n is equal to the coarsest level image G _n of the Gaussian pyramid.

The DoG-based Laplacian pyramid of the present invention comprises two images, a Gaussian pyramid and a Laplacian pyramid, as shown in Figures 4a and 4b. The frequency of the fundamental structure is decomposed into the intensity of each level in the Gaussian pyramid, and the frequency of the edge structure is localized to Laplacian at each level of the frequency of the Laplacian pyramid. FIG. 4 is a conceptual diagram of an edge-recognition corresponding point search in the Laplacian pyramid of the present invention, which shows a comparison between a gradient-based search (d) and a conventional Gaussian based search (e). The present invention produced a Gaussian pyramid using a 5 x 5 blur kernel.

Aggregated correspondence: The reason we chose two pyramids is the aggregated correspondence of base structure and edge structure.

4C shows an example of the patch color distance from the upper left corner of the circular function of the scale of U ₂ and L ₂ to the target patch of all the source patches, The structure information and the detail structure information are aggregated. Since the random search algorithm attempts to minimize the color distance of the patch gradually as shown in Equation (3) to be described later, the aggregated edges of the base structure and the detail structure are important clues .

In contrast, the gradient size (or gradient distance function) shown in FIG. 4d and the Gaussian scale (or Gaussian distance function) shown in FIG. 4e have significantly higher values for only a few patches. As a result, most patch structures are ignored by the weights when calculating the weight sum of colors in the vote process, as shown in Equation (4). These two image pyramids with gradients and Gaussian can be used as an energy function of the patch distance.

Based on this observation, the main strategy of finding a correspondence in the present invention is to not only minimize the distance of the low-frequency base structure but also to preserve the distance of the high-frequency detail structure.

The energy function in the present invention is similar to the repetitive Expectation-Maximization (EM) algorithm, but there is a big difference in that it uses two image pyramids of upsampling Gaussian and Laplacian.

In the expectation step, source patches are searched for all target patches as shown in the middle diagram of Fig. 4C. The present invention uses a random correspondence point search algorithm to approximate a nearest neighbor field (NNF) of target patches using a distance function, and can be expressed as Equation (3) below.

&Quot; (3) &quot;

Here, i means the current level, p and q mean the pixel positions in the target area T and the source area S, U _{i , p} and L _{i , p} denote the pixel positions p D represents a sum of squared distances (SSD) between CIELAB colors of two patches, and? And? Represent the ratio of the low frequency fundamental scale U to the high frequency detail scale L , And? +? = 1 can be determined.

와

는 이들이 방향 연산자이기 때문에 오직 수평 방향과 수직 방향의 구조적 변화를 감지할 수 있다. 그래디언트 기반 방식과는 달리, 본 발명은 회전 불변의 성질을 갖고 있는 라플라시안 연산자를 사용하기 때문에 회전에 불변하는 중요한 이점을 가진다. 라플라시안은 어떤 방향의 에지 근방 구조적 변화에서 똑같이 동작하며 디테일 구조를 감지하는데 강인하다. 라플라시안의 등방성 성질은 로컬(또는 지역적) 구조의 대응점을 사용하는데 다중 연산자인 그래디언트 사용을 피할 수 있으며 이는 연산적 효율을 기대할 수 있다.Rotational invariance: As shown in Figure 4d, the image gradient

Wow

Can only detect structural changes in the horizontal and vertical directions because they are directional operators. Unlike the gradient-based approach, the present invention has significant advantages that are invariant to rotation because it uses a Laplacian operator with rotational invariant properties. Laplacian is robust enough to detect the detail structure, behaving the same in structural changes near the edge in any direction. The isotropic properties of Laplacian can avoid the use of multiple operators, which is a local (or regional) structure, to avoid the use of a gradient, which can be expected to be computationally efficient.

Combined Vote: The present invention updates the upsampling Gaussian image and the Laplacian image by blending nearest source patches to maximize the similarity of the target patches and source patches after finding the corresponding points.

To accelerate the convergence of the algorithm, the present invention performs weighted blending of the scales. Patches with similar color distances and patches close to the completion boundary are given higher weights.

의 미세 스케일(finer scale)은 상기 수학식 2를 이용하여 레벨 i에서 업샘플링 가우시안

와 라플라시안

를 더하여 구해질 수 있다. 결과적으로, 재구성된 가우시안

는 채워진 가우시안을 업샘플링한

로부터 구해질 수 있다. 그러나, 레벨 i-1에서 라플라시안

의 미세 스케일은 이러한 방식으로 재구성될 수 없다. 라플라시안 스케일

은 고주파 디테일을 보유하고 있기 때문에 레벨 i에서 획득된 검색 패치

의 최근접 이웃 필드 대응점을 이용하여 레벨 i-1에서 소스 영역의 정보로 라플라시안 스케일

의 타겟 영역을 채울 수 있다.Laplacian Structure Reconstruction: When the EM optimization at level i converges, the filled result L _i , U _i should be conveyed to the next level i-1. That is, the compaction at the i-th level can be utilized as an initial complement of the next level i-1. As shown in Figs. 5 and 6, the Gaussian

The finer scale of &lt; RTI ID = 0.0 &gt;

And Laplacian

Can be obtained. As a result, the reconstructed Gaussian

Upsamples the filled Gaussian

. &Lt; / RTI &gt; However, at level i-1,

Can not be reconstructed in this way. Laplacian scale

Because it has high frequency detail, the search patch acquired at level i

Using the closest neighbor field correspondence point of the Laplacian scale

Lt; / RTI &gt;

In this way, the present invention can complete the basic structure and detail structure in two image pyramids from the coarse level to the finest level.

.

를 계산한다. 여기서, σ는 검출하고자 하는 유사도의 감도(sensitivity)를 결정할 수 있다. 유사도 이외에, 본 발명은 타겟 픽셀 q에서 신뢰 가중치 Λ(q)를 계산할 수 있으며, 이는 채워진 경계에 더 가까운 타겟 포인트에 더 높은 신뢰 값을 할당함으로써, 경계 오류를 피할 수 있다. 이런 보팅 프로세스는 타겟 영역 내의 이미지가 이미지 내 타겟 영역 바깥에 위치되는 것을 가정으로 한다. 본 발명은 픽셀 q에서의 유사도와 신뢰의 두 메트릭을 가중치

로 결합한다. 모든 타겟 픽셀 q에 대해, 본 발명은 가중치

를 사용하여 최근접 이웃 필드

로부터

의 중첩 컬러의 가중치 평균 c_q를 계산하며, 가중치 평균은 아래 <수학식 4>와 같이 계산될 수 있다.Vote: Describing the color voting step, the present invention first calculates the similarity between the target patch of the pixel q at level 1 and the source patch of the corresponding pixel p

. Here,? Can determine the sensitivity of the similarity to be detected. In addition to the similarity, the present invention can calculate the confidence weight A (q) at the target pixel q, which can avoid boundary errors by assigning a higher confidence value to the target point closer to the filled boundary. This voting process assumes that the image in the target area is located outside the target area in the image. The present invention applies both metrics of similarity and confidence at pixel q to weight

Lt; / RTI &gt; For all target pixels q,

To the nearest neighbors field

from

Calculating a weighted average color of the superposition of _q c, and the weighted average may be calculated as shown in the following <Equation 4>.

&Quot; (4) &quot;

Here, Q may mean overlapping patches at the target pixel q.

As described above, the Laplacian patch-based image synthesis method according to the embodiment of the present invention calculates DoG (difference of Gaussian) using a Gaussian pyramid and a Laplacian pyramid, approximates a second derivative using DoG, It is possible to avoid the problem of synthesizing the first-order differential compound image by performing image synthesis through the differential of the second order.

Such a method according to the present invention can be applied to an image including an image including a region that has not been shot well for some reason, an image including an area remaining as a blank area in the background as a result of removing a foreground object, You can naturally fill in the empty area within.

The method according to the present invention will now be described in more detail with reference to FIG.

7 is a flowchart illustrating an operation of an image synthesizing method according to an embodiment of the present invention.

Referring to FIG. 7, an image combining method according to an embodiment of the present invention generates a Laplacian pyramid for an input image when an input image is received (S710).

Here, the Laplacian pyramid can be generated using the upsampling Gaussian pyramid upsampled with the Gaussian pyramid and the Gaussian pyramid for the input image, and can be found from the above-described equation (2).

Specifically, the present invention produces an upsampling Gaussian pyramid and a Laplacian pyramid, wherein the upsampling god cyan pyramid and the Laplacian pyramid are used.

When the Laplacian pyramid is generated in step S710, the generated second Laplacian pyramid is approximated through a Difference of Gaussian (S720).

Then, the target area in the input image is image-synthesized using the second-order differential properties approximated at each level (S730).

Here, in step S730, the source patches for each of the target patches of the target area are retrieved using the Gaussian pyramid and the Laplacian pyramid for the input image, and the pixels of each of the target patches are updated using the retrieved source patches, Can be synthesized.

At this time, in step S730, the closest neighbor source patches for each of the target patches are retrieved using the Gaussian pyramid and the Laplacian pyramid, and the nearest neighbor source patches thus searched are blended with the target patches, Can be updated.

Further, step S730 may update the pixels of each of the target patches based on the weighting of the distance of each of the target patches at the boundaries of the detected source patches and the target area, and further, The pixel of each of the target patches may be updated based on the similarity and the confidence weight at the target pixel of each of the target patches.

Specifically, the image synthesis result of the target area at the previous level is filled in the target area, and the image synthesis is performed for each level, that is, from the coarse level to the finest ( finest) level, thereby performing image synthesis of the target area. In other words, an inpainting process is repeatedly performed to synthesize an image of a target area for each level, and the inpainting process updates the pixels of the target patch using the retrieved source patches By performing the voting process, the result of the previous level is used at the next level. Painting, which is an image of the present invention at one level, results in an image filled with upsampling Gaussian and an image filled with Laplacian, and then using the two images at the next level.

Further, a method according to another embodiment of the present invention generates a Gaussian pyramid and a Laplacian pyramid for an input image, and uses a Gaussian pyramid and a Laplacian pyramid in parallel to image-synthesize a target area in an input image, The synthesis of the target region can be performed by sequentially repeating the synthesis process for each level of the Gaussian pyramid and the Laplacian pyramid.

Of course, the contents of generating the Laplacian pyramid, the contents of retrieving nearest neighbor source patches, the contents of synthesizing images considering weight, the contents of synthesizing images using similarity and reliability are the same as those of FIG. 7 The description thereof will be omitted.

Such a technique according to the present invention is an image-related software or application, for example, a technology capable of appropriately filling an empty space of an image in Photoshop, which is a region that is not shot well for some reason, It is a technology that can fill the damaged area as naturally as possible because of an unknown area.

FIG. 8 is a block diagram of an image synthesizing apparatus according to an embodiment of the present invention. The arrangement of the apparatus for performing the contents of FIGS. 1 to 7 described above corresponds to the processor of the computing apparatus of FIG. 2 .

Referring to FIG. 8, an apparatus 800 according to an embodiment of the present invention includes a generating unit 810, an approximating unit 820, and a composing unit 830.

The generating unit 810 generates a laplacian pyramid for the input image when the input image is received.

Here, the generating unit 810 may generate the Laplacian pyramid using the upsampling Gaussian pyramid upsampled with the Gaussian pyramid and the Gaussian pyramid for the input image.

The approximation unit 820 uses the generated Laplacian pyramids to approximate the second order differential properties at each level through a Difference of Gaussian.

The combining unit 830 images the target area in the input image using the second-order differential property approximated at each level.

At this time, the combining unit 830 searches the source patches for each of the target patches of the target area using the Gaussian pyramid and the Laplacian pyramid for the input image, and updates the pixels of each of the target patches using the retrieved source patches The image of the target area can be synthesized.

At this time, the combining unit 830 searches for nearest neighboring source patches for each of the target patches using the Gaussian pyramid and the Laplacian pyramid, blends the nearest neighbor source patches thus searched to the target patches, Each pixel can be updated.

Further, the compositing unit 830 can update the pixels of each of the target patches based on the weight based on the distance of each of the target patches at the boundaries of the retrieved source patches and the target area, and further, And update the pixel of each of the target patches based on the similarity of the target patches and the confidence weights at the target pixels of each of the target patches.

The synthesizing unit 830 sequentially repeats the image synthesis for each level of the pyramid. Specifically, the synthesizing unit 830 fills the target area with the image synthesis result of the target area at the previous level, ) Level to the finest level by sequentially repeating the image synthesis of the target region and the target region.

Of course, it will be apparent to those skilled in the art that the apparatus according to one embodiment of the present invention as well as all of the elements described in Figs.

Further, the apparatus according to the present invention is not limited to being constituted only by the configuration means shown in Fig. 8, and can be configured by functional means that can be performed by the processor of the computing apparatus. That is, the configuration means of Fig. 8 is a functional configuration means of the processor, and the function of the configuration means of Fig. 8 can be performed by at least one processor.

The system or apparatus described above may be implemented as a hardware component, a software component, and / or a combination of hardware components and software components. For example, the systems, devices, and components described in the embodiments may be implemented in various forms such as, for example, a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field programmable array ), A programmable logic unit (PLU), a microprocessor, or any other device capable of executing and responding to instructions. The processing device may execute an operating system (OS) and one or more software applications running on the operating system. The processing device may also access, store, manipulate, process, and generate data in response to execution of the software. For ease of understanding, the processing apparatus may be described as being used singly, but those skilled in the art will recognize that the processing apparatus may have a plurality of processing elements and / As shown in FIG. For example, the processing unit may comprise a plurality of processors or one processor and one controller. Other processing configurations are also possible, such as a parallel processor.

The software may include a computer program, code, instructions, or a combination of one or more of the foregoing, and may be configured to configure the processing device to operate as desired or to process it collectively or collectively Device can be commanded. Software and / or data may be stored on any type of machine, component, physical device, virtual equipment, computer storage medium, or device, such as a computer readable medium, , Or may be permanently or temporarily embodied in a transmitted signal wave. The software may be distributed over a networked computer system and stored or executed in a distributed manner. The software and data may be stored in one or more computer readable recording media.

The methods according to embodiments may be implemented in the form of program instructions that may be executed through various computer means and recorded in a computer readable medium. The computer readable medium may include program instructions, data files, data structures, etc., alone or in combination. The program instructions recorded in the medium may be those specially designed and constructed for the embodiments or may be available to those skilled in the art of computer software. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks and magnetic tape, optical media such as CD-ROMs and DVDs, magnetic media such as floptical disks, Magneto-optical media, and hardware devices specifically configured to store and execute program instructions such as ROM, RAM, flash memory, and the like. Examples of program instructions include machine language code such as those produced by a compiler, as well as high-level language code that can be executed by a computer using an interpreter or the like. The hardware devices described above may be configured to operate as one or more software modules to perform the operations of the embodiments, and vice versa.

While the present invention has been particularly shown and described with reference to exemplary embodiments thereof, it is to be understood that the invention is not limited to the disclosed exemplary embodiments. For example, it is to be understood that the techniques described may be performed in a different order than the described methods, and / or that components of the described systems, structures, devices, circuits, Lt; / RTI &gt; or equivalents, even if it is replaced or replaced.

Therefore, other implementations, other embodiments, and equivalents to the claims are also within the scope of the following claims.

Claims (21)

Generating a Gaussian pyramid and a Laplacian pyramid for the input image;
Synthesizing a target region in the input image using the Gaussian pyramid and the Laplacian pyramid; And
Performing the image synthesis of the target region by sequentially repeating the image synthesis for each level of the Gaussian pyramid and the Laplacian pyramid
&Lt; / RTI &gt;
The method according to claim 1,
The generating step
Wherein the Laplacian pyramid is generated using the upsampling Gaussian pyramid up-sampled with the Gaussian pyramid and the Gaussian pyramid.
The method according to claim 1,
The step of image-compositing the target region
Using the Gaussian pyramid and the Laplacian pyramid to search for source patches for each of the target patches of the target region, wherein the source patches are patches outside the target region, And the image of the target area is synthesized by updating the pixels of each of the patches.
The method of claim 3,
The step of image-compositing the target region
And searching for nearest-neighbor source patches for each of the target patches using the Gaussian pyramid and the Laplacian pyramid.
The method of claim 3,
The step of image-compositing the target region
And updates the pixels of each of the target patches based on a weight based on the distance of each of the target patches at a boundary between the searched source patches and the target area.
The method of claim 3,
The step of image-compositing the target region
And updating the pixels of each of the target patches based on the similarities of the target patches and the searched source patches and the confidence weights of the target pixels of each of the target patches.
The method according to claim 1,
The step of performing image composition of the target area
Filling the target area with the image synthesis result of the target area at the previous level, and repeating the image synthesis.
Generating a laplacian pyramid for the input image;
Approximating the second order differential properties at each level using the generated Laplacian pyramid through Difference of Gaussian; And
Synthesizing an image of a target region in the input image using a second derivative property at the approximated level,
&Lt; / RTI &gt;
9. The method of claim 8,
The generating step
Wherein the Laplacian pyramid is generated using a Gaussian pyramid for the input image and an upsampling Gaussian pyramid for upsampling the Laplacian pyramid.
9. The method of claim 8,
The step of image-compositing the target region
Searching source patches for each of the target patches of the target region using the Gaussian pyramid for the input image and the Laplacian pyramid, wherein the source patches are patches outside the target region, Wherein the target regions are image-synthesized by updating pixels of each of the target patches.
11. The method of claim 10,
The step of image-compositing the target region
And searching for nearest-neighbor source patches for each of the target patches using the Gaussian pyramid and the Laplacian pyramid.
11. The method of claim 10,
The step of image-compositing the target region
And updates the pixels of each of the target patches based on a weight based on the distance of each of the target patches at a boundary between the searched source patches and the target area.
11. The method of claim 10,
The step of image-compositing the target region
And updating the pixels of each of the target patches based on the similarities of the target patches and the searched source patches and the confidence weights of the target pixels of each of the target patches.
9. The method of claim 8,
The step of image-compositing the target region
Wherein the image synthesis of the target area is performed by filling the target area with the image synthesis result of the target area at the previous level and sequentially repeating the result at each level.
A generator for generating a laplacian pyramid for an input image;
An approximation unit for approximating the second derivative at each level through a Gaussian difference using the generated Laplacian pyramid; And
A synthesis section for synthesizing an image of a target area in the input image using a second derivative property at each approximated level,
Lt; / RTI &gt;
16. The method of claim 15,
The generating unit
Wherein the Laplacian pyramid is generated using a Gaussian pyramid for the input image and an upsampling Gaussian pyramid for upsampling the Laplacian pyramid.
16. The method of claim 15,
The combining unit
Searching source patches for each of the target patches of the target region using the Gaussian pyramid for the input image and the Laplacian pyramid, wherein the source patches are patches outside the target region, Wherein the target pixels are image-synthesized by updating the pixels of each of the target patches.
18. The method of claim 17,
The combining unit
And searches the nearest-neighbor source patches for each of the target patches using the Gaussian pyramid and the Laplacian pyramid.
18. The method of claim 17,
The combining unit
And updates the pixels of each of the target patches based on a weight based on the distance of each of the target patches at a boundary between the searched source patches and the target area.
18. The method of claim 17,
The combining unit
And updates the pixels of each of the target patches based on the similarities of the target patches and the searched source patches and the confidence weights of the target pixels of each of the target patches.
16. The method of claim 15,
The combining unit
Wherein the image compositing unit performs the image compositing of the target area by filling the target area with the image synthesis result of the target area at the previous level and sequentially repeating the image synthesis by the level.

Images