KR101906142B1 - Image processing apparatus and method - Google Patents

Abstract

An image processing apparatus is provided. The render unit may render an input 3D model to generate a color image and a guide image. The guide image may include a depth map and a normal map corresponding to the input 3D model. The filter unit may filter the color image using a guided image filtering technique. The parameter determination unit may determine a filter parameter used for the guided image filtering.

Description

[0001] IMAGE PROCESSING APPARATUS AND METHOD [0002]

The present invention relates to an image processing apparatus and method, and more particularly, to an image processing apparatus and method for performing noise removal processing on an image rendered through a pass tracing technique.

The path tracing algorithm accurately simulates the physical characteristics of the light so that the rendered image can be accepted as a realistic image.

This path tracing algorithm requires a lot of computation time, especially in order to reflect the natural light condition. Thus, typically, only passable number of passes per pixel is used to generate pass-traced images.

However, when only a limited number of passes are used, the image becomes very noisy and noise filtering is required.

According to an aspect of the present invention, there is provided an image processing apparatus including a render unit for rendering an input 3D model to generate a color image and a guide image; A parameter determination unit for determining a filter parameter to be used for filtering using the guide image; And a filter unit for filtering the color image using the filter parameter.

According to an exemplary embodiment, the filter unit may filter the color image using a guided image filtering technique using the filter parameters.

In this case, the filter parameter may include at least one of a filter size and a weight used in the guided image filtering technique.

According to one embodiment, the guide image may include a depth map matching the color image and a normal map indicating normal information of the depth map.

According to one embodiment, the render unit may include: a first color image generation unit for generating a first color image in which the 3D model is rendered in consideration of direct light among the illumination environments considered in the rendering; And a second color image generation unit for generating a second color image by rendering the 3D model in consideration of indirect light in the illumination environment, wherein the color image includes the first color image and the second color image .

In this case, the parameter determination unit may separately determine a first filter parameter applied to the first color image and a second filter parameter applied to the second color image.

According to an embodiment of the present invention, the image processing apparatus may further include: a first filtered image obtained by filtering the first color image by the filter unit using the first filter parameter; And an image generation unit that generates a result image by rendering the 3D model by synthesizing a second filtered image obtained by filtering the second color image.

Meanwhile, according to one embodiment, the parameter determination unit may analyze at least one characteristic of the 3D model, and may compare the at least one characteristic with a pre-calculated perceptual metric to determine the filter parameter.

In this case, the at least one characteristic may include at least one of a geometric complexity of the 3D model, a material property, and a smoothness level of an illumination environment considered in the rendering.

According to another aspect of the present invention, there is provided a method of generating a first color image in which a 3D model is rendered in consideration of direct light among illumination environments considered in rendering, and generating a second color image in which the 3D model is rendered, A rendering unit to generate; A parameter determination unit for determining a first filter parameter to be applied to the first color image and a second filter parameter to be applied to the second color image; And generating a first filtered image by filtering the first color image using the first filter parameter and filtering the second color image using the second filter parameter to generate a second filtered image A filter portion; And an image generating unit for generating a rendered image of the 3D model using the first filtered image and the second filtered image.

According to an embodiment, the filter unit may filter the first color image and the second color image using a guided image filtering technique.

In this case, the first filter parameter and the second filter parameter may include at least one of a filter size and a weight used in the guided image filtering technique.

According to another aspect of the present invention, there is provided an image processing apparatus comprising: a rendering step of rendering an input 3D model by a render unit of an image processing apparatus to generate a color image and a guide image; A parameter determination step of the image processing apparatus determining a filter parameter to be applied to the filtering using the guide image; And filtering the color image using a filter parameter of the filter unit of the image processing apparatus.

According to one embodiment, the filtering step may include filtering the color image with a guided image filtering technique using the filter parameter, and the filter parameter may include at least one of a filter size and a weight used in the guided image filtering technique .

According to one embodiment, the guide image may include a depth map matching the color image and a normal map indicating normal information of the depth map.

According to an embodiment of the present invention, the rendering step may include: a first color image generation step of generating a first color image in which the 3D model is rendered in consideration of direct light among the illumination environments considered in the rendering; And a second color image generation step of generating a second color image by rendering the 3D model in consideration of indirect light in the illumination environment, wherein the color image includes the first color image and the second color image can do.

In this case, the parameter determination step may separately determine a first filter parameter applied to the first color image and a second filter parameter applied to the second color image.

In addition, the image processing method may further include an image generating unit of the image processing apparatus, the image processing apparatus including: an image generating unit that generates a first filtered image by filtering the first color image using the first filter parameter, And generating an image by rendering the 3D model by synthesizing a second filtered image obtained by filtering the second color image by the filter unit.

According to one embodiment, the parameter determination step comprises: analyzing at least one characteristic for the 3D model; And comparing the at least one property with a pre-computed perceptual metric to determine the filter parameter.

1 is a block diagram of an image processing apparatus according to an embodiment.
2 is a detailed block diagram of a render unit according to one embodiment.
3 is a conceptual diagram for explaining a guided image filtering technique according to an embodiment.
FIG. 4 is a conceptual diagram for explaining a process of filtering and applying a different filter parameter to direct light and indirect light according to an embodiment and generating a resultant image.
5 is a conceptual diagram for explaining a process of determining a filter parameter using characteristics of a 3D model according to an embodiment.
6 is a flowchart illustrating an image processing method according to an exemplary embodiment.

In the following, some embodiments will be described in detail with reference to the accompanying drawings. However, it is not limited or limited by these embodiments. Like reference symbols in the drawings denote like elements.

1 is a block diagram of an image processing apparatus 100 according to an embodiment.

According to one embodiment, the image processing apparatus 100 includes a render unit 110, a parameter determination unit 120, and a filter unit 130.

When a 3D model to be rendered is inputted, the render unit 110 generates the color image by rendering the 3D model.

According to one embodiment, the render unit 110 renders the 3D model and generates a normal map including a depth map and normal information of the depth map.

According to an exemplary embodiment, the filter unit 130 filters the color image using a guided image filtering (GIF) technique.

The process of filtering the color image using the guided image filtering technique according to an embodiment can be understood by the following equation.

Where I is a Guide Image or Guidance Image, p is a filtered color image, and q is a filtered result image. And i and j represent pixel indices.

According to one embodiment, at least one of the depth map and the normal map is used in the calculation of the weight Wij applied to the pixel pj. Illustratively, a normal map is used.

Referring to the normal map, the Wij is determined to perform much blurring in a region where the normal is constant, and the Wij is determined to be less blurred in the region where the normal is not constant. If the norm is not constant, it means that there is a high probability that the region is an edge portion.

By determining the weight applied to the GIF filtering by referring to the normal map, it is possible to prevent the edge portion from being well preserved and unnecessarily blurring the image. Here, the determination of the weight can be understood as an example of a filter parameter determined by the parameter determination unit 120. [

This process will be described in more detail below with reference to FIG. 2 to FIG.

On the other hand, other embodiments may adaptively adjust other examples of filter parameters. For example, it is also possible to adjust the filtering block size adaptively. This will be described later in more detail with reference to FIGS.

According to one embodiment, the render unit 110 may generate a first color image in which a 3D model is rendered in consideration of direct light among light environments considered in rendering. This process can be performed by performing path tracing by sampling only light directly in the rendering process.

In addition, the render unit 110 may generate a second color image in which the 3D model is rendered in consideration of indirect light in the illumination environment.

The generation of the first color image and the second color image may be performed in consideration of illumination obtained by sampling only the direct light and passing through the path tracing, It is also possible to perform the ISing, but it is also possible to estimate the tracing result by direct samples of the light and the tracing result by the indirect light samples in the result of sampling and rendering both direct light and indirect light It is possible.

Therefore, although the various methods are not repeatedly described below, any embodiment that separately generates the image rendering result reflecting the direct light and the indirect rendering light is also possible.

According to an embodiment, the parameter determination unit may determine the first filter parameter to be applied to the first color image and the second filter parameter to be applied to the second color image differently. In general, direct and indirect light can cause different noise characteristics in rendering results.

In previous researches such as "McCool 1999; Bauszat et al 2011, Dammertz et al 2010; Segovia et al 2006", adaptation of the blur kernel size (filter size) I was not enemy. For example, a kernel (filter block) of the same size and shape is used for both direct light and indirect light, and phenomena such as loss of edges due to unnecessary blurring have been found. Even direct light has not been considered in the filtering process.

According to an embodiment, a first filter parameter that is adapted to the characteristics of direct light is applied to a first color image in which a 3D model is rendered in consideration of direct light by separately handling direct light and indirect light having different noise characteristics, Adaptive filtering is performed to apply second filter parameters matching the characteristics of the indirect light to the second color image in which the 3D model is rendered.

According to one embodiment, the image generator 140 combines the first filtered image and the second filtered image, which are separately filtered results, to generate a noise-canceled result image. It can be confirmed that the noise is effectively removed while the edge is well preserved in the resulting image.

2 is a detailed block diagram of the render unit 110 according to one embodiment.

As described above, the render unit 110 may include a map generation unit 210 that generates the depth map by rendering the 3D model and generating the geometry information.

The map generating unit 210 may generate the normal map from the 3D model directly or from the depth map.

The generated depth map and / or normal map may be stored in the buffer 240.

According to one embodiment, the normal map can be used to adaptively determine the weight applied to the GIF filtering process, as described above with reference to FIG.

The first color image generating unit 220 may generate a first color image in which a 3D model is rendered in consideration of direct light among illumination environments to be considered in rendering the 3D model.

The second color image generation unit 230 may generate a second color image of the 3D environment by considering the indirect light.

As described with reference to FIG. 1, the image generation process of the first color image generation unit 220 and the second color image generation unit 230 includes the image rendering result reflecting the direct light and the image rendering result reflecting the indirect light Any other embodiment is possible.

Therefore, not only the first color image and the second color image are calculated by the pass tracing process, but also the result of tracing both the direct light and the indirect light by the samples of the direct light and the indirect light It is also possible to distinguish tracing results by samples and to estimate them.

3 is a conceptual diagram for explaining a guided image filtering technique according to an embodiment.

The render unit 110 may generate the guide image 310 and the noisy color image 301 using the 3D model.

According to one embodiment, the guide image 310 may include the depth map 311 and / or the normal map 312 described above with reference to FIGS. Since the direct geometry information is calculated using the 3D model, the depth map 311 and the normal map 312 may be much less noise than the color image 301. [

According to one embodiment, the parameter determination unit 120 uses the depth map 311 and / or the normal map 312 as a guide image, and determines a weight used for the filtering unit 130 to perform GIF filtering As shown in FIG.

For example, with reference to FIG. 1, as described above, Wij can be determined so as to perform much blurring in an area corresponding to a flat and regular surface. And, since the probability that an area where the normal is not constant includes an edge is large, Wij can be determined so as to perform less blurring.

In the exemplary result image 302 generated according to this process, it is confirmed that the edge portion is well preserved and the noise is effectively removed.

For reference, the guided image filtering technique itself is described in "HE, K., SUN, J., and TANG, X. 2010. Guided image filtering. (In Proceedings of the 11th European Conference on Computer Vision: Part I, 10, 1 ?? 14) ", so that detailed description is omitted.

FIG. 4 is a conceptual diagram for explaining a process of filtering and applying a different filter parameter to direct light and indirect light according to an embodiment and generating a resultant image.

As described above, according to one embodiment, the render unit 110 generates the first color image 410 in which the 3D model is rendered in consideration of the direct light among the illumination environments considered in the rendering, The second color image 420 may be generated by rendering the 3D model in consideration of the indirect light in the illumination environment.

Then, when the parameter determination unit 120 adaptively determines different filter parameters for the first color image 410 and the second color image 420, the filter unit 130 performs filtering using the filter parameters.

A second filtered image 421 that is a result of filtering the first filtered image 411 and the second color image 420 as a result of filtering the first color image 410 is generated.

Then, the image generating unit 140 generates a result image 430 by combining the first filtered image 411 and the second filtered image 421.

5 is a conceptual diagram for explaining a process of determining a filter parameter using characteristics of a 3D model according to an embodiment.

According to one embodiment, the parameter determination unit 120 analyzes at least one characteristic of the 3D model, compares the at least one characteristic with a pre-computed Perceptual metric, You can decide.

Illustratively, the at least one property includes at least one of a complexity of geometry, a material feature, and a smoothness level of the illumination environment to be considered in rendering, of the input 3D model associated with the reference image 510. [ level of Light) may be included

Illustratively, in image 520, when GIF filtering is applied to each pixel, the maximum allowable filter size is indicated in color. Since the filter size should be small in the geometric complexity or irregularity of the material properties, it is displayed in the blue color system. In the part where the geometric complexity is small, it is displayed in the red color system. However, since this color is for the sake of understanding, embodiments can not be construed to be limited by this exemplary image 520. [

Therefore, the parameter determination unit 120 determines the filter size to be small in order to filter a pixel having a high probability of being an edge portion because of high geometric complexity, and conversely, in order to filter a pixel having a small geometrical complexity and a high probability of a flatness, The size can be increased.

With this adaptive filtering, the edges can be well preserved and the noise can be effectively removed.

6 is a flowchart illustrating an image processing method according to an exemplary embodiment.

According to one embodiment, in step 610, a rendering process is performed for the input 3D model.

In this process, the render unit 110 may generate a guide image for the 3D model. According to one embodiment, the guide image may include the depth map 311 and / or the normal map 312 described above with reference to FIG. 1 and FIG.

Then, the parameter determination unit 120 adaptively calculates the pixel weights to be applied to the GIF filtering using the depth map and / or the normal map (step 620), and determines the filter parameters accordingly (step 630).

In step 640, the render unit 110 generates a first color image in which the 3D model is rendered in consideration of direct light among the illumination environments considered in the rendering (step 640) The second color image may be generated by rendering the 3D model in consideration of the light (step 650).

Then, in step 630, the parameter determination unit 120 determines a first filter parameter and a second filter parameter to be applied to the first color image and the second color image. Also, in step 630, the parameter determination unit 120 may adjust the filter size or the like by comparing at least one characteristic information of the 3D model with the perceptual metric.

Then, in steps 660 and 670, the filter unit 130 filters the first color image and the second color image.

In operation 680, the first filtered image and the second filtered image are synthesized to generate a resultant image. The above process is as described above with reference to Figs.

The apparatus described above may be implemented as a hardware component, a software component, and / or a combination of hardware components and software components. For example, the apparatus and components described in the embodiments may be implemented within a computer system, such as, for example, a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field programmable array (FPA) A programmable logic unit (PLU), a microprocessor, or any other device capable of executing and responding to instructions. The processing device may execute an operating system (OS) and one or more software applications running on the operating system. The processing device may also access, store, manipulate, process, and generate data in response to execution of the software. For ease of understanding, the processing apparatus may be described as being used singly, but those skilled in the art will recognize that the processing apparatus may have a plurality of processing elements and / As shown in FIG. For example, the processing unit may comprise a plurality of processors or one processor and one controller. Other processing configurations are also possible, such as a parallel processor.

The software may include a computer program, code, instructions, or a combination of one or more of the foregoing, and may be configured to configure the processing device to operate as desired or to process it collectively or collectively Device can be commanded. The software and / or data may be in the form of any type of machine, component, physical device, virtual equipment, computer storage media, or device , Or may be permanently or temporarily embodied in a transmitted signal wave. The software may be distributed over a networked computer system and stored or executed in a distributed manner. The software and data may be stored on one or more computer readable recording media.

The method according to an embodiment may be implemented in the form of a program command that can be executed through various computer means and recorded in a computer-readable medium. The computer-readable medium may include program instructions, data files, data structures, and the like, alone or in combination. The program instructions to be recorded on the medium may be those specially designed and configured for the embodiments or may be available to those skilled in the art of computer software. Examples of computer-readable media include magnetic media such as hard disks, floppy disks and magnetic tape; optical media such as CD-ROMs and DVDs; magnetic media such as floppy disks; Magneto-optical media, and hardware devices specifically configured to store and execute program instructions such as ROM, RAM, flash memory, and the like. Examples of program instructions include machine language code such as those produced by a compiler, as well as high-level language code that can be executed by a computer using an interpreter or the like. The hardware devices described above may be configured to operate as one or more software modules to perform the operations of the embodiments, and vice versa.

While the present invention has been particularly shown and described with reference to exemplary embodiments thereof, it is to be understood that the invention is not limited to the disclosed exemplary embodiments. For example, it is to be understood that the techniques described may be performed in a different order than the described methods, and / or that components of the described systems, structures, devices, circuits, Lt; / RTI > or equivalents, even if it is replaced or replaced.

Therefore, other implementations, other embodiments, and equivalents to the claims are also within the scope of the following claims.

Claims (20)

A renderer for rendering an input 3D model to generate a color image and a guide image;
A parameter determination unit for determining a filter parameter to be used for filtering using the guide image; And
A filter unit for filtering the color image using the filter parameter,
Lt; / RTI >
Wherein the parameter determination unit analyzes at least one characteristic for the 3D model and compares the at least one characteristic with a pre-calculated perceptual metric to determine the filter parameter. The method according to claim 1,
Wherein the filter unit filters the color image using a guided image filtering technique using the filter parameters. 3. The method of claim 2,
Wherein the filter parameter comprises at least one of a filter size and a weight used in the guided image filtering technique. The method according to claim 1,
Wherein the guide image includes a depth map matched with the color image and a normal map indicating normal information of the depth map. The method according to claim 1,
Wherein,
A first color image generation unit for generating a first color image in which the 3D model is rendered in consideration of direct light among the illumination environments considered in the rendering; And
A second color image generation unit for generating a second color image in which the 3D model is rendered in consideration of indirect light in the illumination environment,
Lt; / RTI >
Wherein the color image includes the first color image and the second color image. 6. The method of claim 5,
Wherein the parameter determination unit separately determines a first filter parameter applied to the first color image and a second filter parameter applied to the second color image. The method according to claim 6,
A second filtered image obtained by filtering the first color image by the filter unit using the first filter parameter and a second filtered image obtained by filtering the second color image by using the second filter parameter; And generating an image by rendering the 3D model,
Further comprising: delete The method according to claim 1,
Wherein the at least one property comprises at least one of a geometric complexity of the 3D model, a material property, and a smoothness level of the illumination environment considered in rendering. A renderer for generating a first color image in which a 3D model is rendered in consideration of direct light among illumination environments considered in rendering and generating a second color image in which the 3D model is rendered in consideration of indirect light among the illumination environments;
A parameter determination unit for determining a first filter parameter to be applied to the first color image and a second filter parameter to be applied to the second color image; And
A filter for generating a first filtered image by filtering the first color image using the first filter parameter and generating a second filtered image by filtering the second color image using the second filter parameter, part; And
And an image generation unit for generating a resultant image of the 3D model using the first filtered image and the second filtered image,
Lt; / RTI >
Wherein the parameter determination unit analyzes at least one characteristic for the 3D model and compares the at least one characteristic with a pre-calculated perceptual metric to determine the filter parameter. 11. The method of claim 10,
Wherein the filter unit filters the first color image and the second color image using a guided image filtering technique. 12. The method of claim 11,
Wherein the first filter parameter and the second filter parameter comprise at least one of a filter size and a weight used in the guided image filtering technique. A rendering step of rendering an input 3D model by a render unit of an image processing apparatus to generate a color image and a guide image;
A parameter determination step of the image processing apparatus determining a filter parameter to be applied to the filtering using the guide image; And
Wherein the filtering unit of the image processing apparatus filters the color image using the filter parameter
Lt; / RTI >
Wherein the parameter determination step comprises:
Analyzing at least one characteristic for the 3D model; And
Comparing the at least one property with a pre-computed perceptual metric to determine the filter parameter
And an image processing method. 14. The method of claim 13,
Wherein the filtering includes filtering the color image using a guided image filtering technique using the filter parameter, and the filter parameter includes at least one of a filter size and a weight used in the guided image filtering technique. 14. The method of claim 13,
Wherein the guide image includes a depth map matched with the color image and a normal map indicating normal information of the depth map. 14. The method of claim 13,
Wherein the rendering step comprises:
A first color image generation step of generating a first color image by rendering the 3D model in consideration of direct light among the illumination environments considered in the rendering; And
A second color image generation step of generating a second color image in which the 3D model is rendered in consideration of indirect light in the illumination environment
Lt; / RTI >
Wherein the color image includes the first color image and the second color image. 17. The method of claim 16,
Wherein the parameter determination step separately determines a first filter parameter applied to the first color image and a second filter parameter applied to the second color image. 18. The method of claim 17,
An image generation unit of the image processing apparatus generates a first filtered image in which the filter unit filters the first color image using the first filter parameter and a second filtered image in which the filter unit uses the second color parameter, An image generating step of generating a result image by rendering the 3D model by synthesizing a second filtered image obtained by filtering the image,
Further comprising the steps of: delete A computer-readable recording medium including a program for performing the image processing method according to any one of claims 13 to 18.

Images