KR101191319B1 - Apparatus and method for painterly rendering based on objective motion information - Google Patents

Abstract

PURPOSE: A pictorial rendering device based on motion information of an object and a method thereof are provided to classify several objects in a rendering target image, apply a brush stroke according to an object attribute, and reflect extracted motion information of the object to rendering. CONSTITUTION: A motion extracting unit(110) extracts motion information of each pixel from input images. An area separating unit(120) separates pixels of a selected target image into object areas. An object classifying unit(130) determines the type of an object included in each object area. A rendering unit(140) applies a brush stroke with a preset attribute according to the type of the object. The rendering unit generates a result image corresponding to the target image. [Reference numerals] (110) Motion extracting unit; (120) Area separating unit; (130) Object classifying unit; (140) Rendering unit

Description

Apparatus and method for painterly rendering based on objective motion information}

The present invention relates to a pictorial rendering apparatus and method based on motion information of an object, and more particularly, to an apparatus and method for expressing an image photographed with a camera in a pictorial feeling as if drawn by hand.

Non-realistic rendering refers to the technique of rendering things in various ways that are not realistic. Non-realistic rendering can generally be classified into painterly rendering, mosaic rendering, cartoon rendering, etc., according to expression techniques.

Among them, pictorial rendering refers to a technique for creating an image painted by a painter using two-dimensional or three-dimensional model data such as a photograph. Painterly rendering is composed of basic units called brush strokes, unlike photographs composed of pixels.

Therefore, in pictorial rendering, instead of expressing pixel-by-pixel information one by one, the painter renders a scene using a brush stroke of a certain thickness. In addition, the stroke position, size, direction, color, etc. are determined based on local information such as color, edge, and gradient of the input image to express the shape of the object in the scene.

This conventional pictorial rendering technique has two limitations. First, the properties of brush strokes in all areas are determined in the same way. Although A Hertzmann, "Painterly Rendering with Curved Brush Strokes of Multiple Sizes", In Proc. Of SIGGRAPH 1998, pp. 453-460 used strokes of various sizes, but the same algorithm was used to generate all strokes. The result generated by such a rendering algorithm does not reflect various expression methods for each object used in actual painting, thereby reducing the realism.

Second, it is difficult to reflect the expression of movement. Artists control the direction, thickness, length, and color of the brushes and express the movements of objects through various expression techniques. However, the existing pictorial rendering research was unable to extract these movements using a single image as input.

H. Lee, C.H Lee, and K. Yoon, "Motion Based Painterly Rendering" Computer Graphics Forum, 28, 4, pp. 1207-1216,2009. And Korean Unexamined Patent Publication No. 2010-0122381, which is a study considering motion, apply one of the motion expression methods to pictorial rendering by rendering a stroke along the direction of movement in the part where the motion is input as a fixed view video. However, various representation techniques except the direction of stroke are not considered.

Therefore, there is a need for a pictorial rendering apparatus and method based on motion information of an object capable of realizing realistic painting by overcoming the limitations of conventional pictorial rendering by various expression techniques.

An object of the present invention is to provide a pictorial rendering apparatus and method based on motion information of an object that can apply various brush strokes according to the properties of an object and express the motion of the object.

Another technical problem to be solved by the present invention is to apply various brush strokes according to the properties of an object, and to read it with a computer that records a program for executing a pictorial rendering method based on motion information of an object that can express the motion of the object on a computer. To provide a recording medium that can be.

According to an aspect of the present invention, there is provided a pictorial rendering apparatus based on motion information of an object, the motion extractor extracting motion information of each pixel from a plurality of input images input at a fixed time point; An area dividing unit dividing a plurality of pixels constituting the selected target image among the plurality of input images into a plurality of object regions; An object classification unit determining a type of an object included in each object area based on an object classification database storing coordinate information, color information, and motion information of the object in an image corresponding to a preset object type; And a rendering unit generating a resultant image corresponding to the target image by applying a brush stroke having a predetermined property to each object region according to the type of the object included in the object region.

According to an aspect of the present invention, there is provided a pictorial rendering method based on motion information of an object, including: a motion extraction step of extracting motion information of each pixel from a plurality of input images input at a fixed time point; A region dividing step of dividing a plurality of pixels constituting a selected target image among the plurality of input images into a plurality of object regions; An object classification step of determining a type of an object included in each object area based on an object classification database storing coordinate information, color information, and motion information of the object in an image corresponding to a preset object type; And a rendering step of generating a resultant image corresponding to the target image by applying a brush stroke having a property set in advance according to the type of the object included in the object region to each object region.

According to the pictorial rendering apparatus and method based on the motion information of the object according to the present invention, various objects included in the rendered target image are classified according to the type of the object, and various brush strokes are applied according to the properties of the object. It is possible to perform realistic pictorial rendering by extracting motion information of an object from an input image and reflecting it in a rendering.

1 is a block diagram showing the configuration of a preferred embodiment of a pictorial rendering apparatus based on motion information of an object according to the present invention;
2 is a view for explaining a pictorial rendering process by the pictorial rendering apparatus according to the present invention;
3 is a diagram illustrating a direction of motion information obtained from a single input image and an average motion vector obtained from a plurality of input images;
4 is a diagram illustrating a result of dividing a target image by a minshift division method;
5 is a diagram illustrating an embodiment of classification of objects constituting a landscape painting;
6 is a view of normalizing and analyzing distribution of region information;
7 is a diagram illustrating an example of creating a layer for each object area;
8 is a diagram showing an outline of an algorithm of a rendering attribute database;
9 is a view showing a brush texture applied to each layer,
10 illustrates brush orientation along an interpolated gradient of the technique of the ETF,
11 is a view showing a visualization image of the mask map and the contour map of the leaf background area,
12 is a view illustrating a process of converting a brush texture used for expressing a brush texture in the rendering unit;
13 is a flowchart illustrating a process of performing a preferred embodiment of the pictorial rendering method according to the present invention;
14 is a diagram illustrating an example of rendering an input image using the pictorial rendering apparatus according to the present invention;
15 is an image comparing an application result of an algorithm for differentiating rendering according to an object type in a pictorial rendering apparatus according to the present invention, and
16 is a result comparison image according to the presence or absence of color control using motion information.

Hereinafter, exemplary embodiments of a pictorial rendering apparatus and method based on motion information of an object according to the present invention will be described in detail with reference to the accompanying drawings.

1 is a block diagram showing the configuration of a preferred embodiment of a pictorial rendering apparatus based on motion information of an object according to the present invention.

Referring to FIG. 1, the pictorial rendering apparatus according to the present invention includes a motion extractor 110, an area divider 120, an object classifier 130, and a renderer 140.

2 is a view for explaining a pictorial rendering process by the pictorial rendering apparatus according to the present invention.

Referring to FIG. 2, the motion extractor 110 extracts motion information of each pixel from a plurality of input images input at a fixed time point to extract motion information. Here, the plurality of input images correspond to a plurality of frames of the moving image photographed at the fixed viewpoint.

In a preferred embodiment of the pictorial rendering apparatus according to the present invention, the motion extractor 110 may use an optical flow technique for extracting motion information.

Optical flow techniques have been used mainly in motion estimation and video compression, and have been used in many non-realistic rendering studies that require motion information. In this case, the optical flow refers to a form of movement of an object, a face, and an outline that can be seen by the eye caused by the relative movement between the camera and the scene.

The motion extractor 110 calculates motion information from a plurality of input images consecutive in time by an optical flow technique according to Equation 1 below. According to Equation 1, a total of n-1 motion information can be obtained from n input images.

로 정의한다.

는 모두 이차원 배열로서 인접한 두 입력 영상 사이에 나타나는 각 픽셀의 x,y 축에 대한 움직임 크기를 나타낸다.At this time, the motion information between the i th and i + 1 th input images in the n input images

.

Are all two-dimensional arrays and represent the magnitude of the motion of the x, y axis of each pixel appearing between two adjacent input images.

의 집합으로부터 입력 영상의 평균 움직임 벡터

을 구하여 잡음의 영향을 최소화한다.On the other hand, since the optical flow technique estimates the motion of the input image unit, noise may occur when referring to a small number of input images. Therefore, the motion information through the following equation (2)

Average motion vector of the input image from the set of

To minimize the effects of noise.

의 방향과 크기는 이후 객체 영역 분류 및 렌더링 과정에 사용된다. Here, avg () is a function to find the average. Average motion vector calculated by Equation 2

The direction and size of are then used in the object area classification and rendering process.

3 is a diagram illustrating a direction of motion information obtained from a single input image and an average motion vector obtained from a plurality of input images.

Referring to FIG. 3, the direction of the average motion vector is represented through the h (hue: color) channel of the HSV color model, and the s (saturation) and v (value) channels are represented as images using fixed values. It can be seen that the direction of the average motion vector obtained from the plurality of input images is more consistent and less noise than the direction of the motion information obtained from the single input image.

The region dividing unit 120 divides a plurality of pixels constituting the selected target image among the plurality of input images into a plurality of object regions.

In order to combine the characteristics of paintings with different expression methods depending on objects, the process of reconstructing the target image composed of pixel units into object units, that is, region division, is required.

In a preferred embodiment of the pictorial rendering apparatus according to the present invention, the region divider 120 may use a mean-shift segmentation method for region segmentation.

The minshift segmentation method is a method of finding a local density maximum point in each pixel constituting a target image and clustering a group having similar maximum points into an area. In the minshift segmentation method, the segmentation results are shown as areas of similar colors, and since clustering is used, the resultant image may be obtained even if the histogram is irregular. Also, since it does not require much user input and calculates the areas to be divided by using the sample data of the pixels of the input image, excessive division caused by setting the starting point of the area diffusion to a low level like the watershed algorithm over segmentation is less problematic.

4 is a diagram illustrating a result of dividing a target image by a minshift division method.

Referring to FIG. 4, it can be seen that the target image is divided into an object area corresponding to a cactus, a sky and a cloud.

The object classifying unit 130 includes an object included in each object area partitioned from the target image based on an object classification database storing coordinate information, color information, and motion information of the object in the image corresponding to a preset object type. Determine the type of.

5 is a diagram illustrating an embodiment of classification of objects constituting the landscape painting.

Referring to FIG. 5, a process of determining, by the object classifier 130, the type of an object included in an object area will be described in detail.

Painters draw paintings. In general, artists bring a variety of expression methods to life, depending on the characteristics of the object, such as the characteristics of the object and the distance to the observer. The object classification unit 130 classifies the object area obtained through the minshift division method for each object according to the coordinates, colors, and movements of the object. The classified objects are then rendered by the rendering unit 140 in a unique expression method according to their characteristics.

When the object classification unit 130 classifies the objects included in the object area, an object classification database in which properties according to the types of objects are stored in advance is used. The object classification database stores object type data collected in advance from one or more sample images for object classification.

Generally, the sky is at the top of the image, and the water and the ground are at the bottom. In addition, water, clouds, leaves, etc. are moved by the wind or gravity, but the ground and artificial structures, etc. always maintain the same shape, except in special cases. In the case of an object that is easily shaken by the wind, such as a leaf or a blade of grass, the edge of the region often overlaps with other objects in the scene. In this case, since the segmentation of the target image is not clear, the region may be defined separately.

For objects that are not located at a certain distance, such as sky, cloud land, or water, and are distributed over a wide range from near to far, the distance should be expressed by gradually adjusting the contrast of colors according to the distance even if the object does not have a clear boundary. do.

Therefore, by analyzing the information such as the color, coordinates, and movement of each object area obtained through the minshift segmentation algorithm from the sample image, the characteristics of each object area are quantified and stored in the object classification database to be included in the object area of the target image. Can be used to classify objects.

For this purpose, representative objects frequently appearing in paintings are selected and classified according to distance (perspective) and movement in the target image. In particular, in the preferred embodiment of the pictorial rendering apparatus according to the present invention, as a representative object frequently appearing in landscape painting will be described taking the sky and clouds, water, land, buildings and roads, tree trunks, leaves as shown in FIG.

The color, coordinates, and motion information of each object region obtained through the minshift segmentation method are all in pixels. In order to understand the characteristics of the object area, information that is organized in pixels must be integrated into area-specific information. In Table 1 below, the average of pixel information constituting each object area in 1163 object areas of three sample images is calculated and summarized in part by information of each object area.

Classification
Normalized coordinate information
Color information
Motion information
X-coordinate average Y coordinate average B channel average G channel average R channel average X direction average Y direction average Leaves / blades 0.511365 0.439843 53.4272 58.778 58.3604 0.00043 0.03215 Tree trunk 0.31168 0.580718 37.9 33.4333 39.9778 -0.04241 0.01186 water 0.541315 0.810529 120.787 110.293 110.907 -0.07789 0.02485 Sky / cloud 0.528963 0.100331 176.196 161.174 159.674 0.07870 0.02985 land 0.748359 0.711835 105.842 98.6316 105.421 -0.04705 0.01400

Some of the information included in Table 1 reflects the characteristics of each classification, while others do not. Information with a large interval between the averages of the object regions and small variances of each region represents the difference between the classifications and is suitable for use as a criterion for classification. On the contrary, the averages of the regions have similar values to each other, and the information having a large variance in each region is information having a low contribution to the object classification of the region.

6 is a view of normalizing and analyzing distribution of region information.

Referring to FIG. 6, the information of each object region is normalized to 0 to 1 by analyzing the distribution of region information, and the average and distribution range of the information for each object classification are visualized. The center dot of the gray bar represents the mean of each classification, and the gray bar represents the range of μ (parent mean) ± 3σ (standard deviation). 6 (a) shows a small difference between the averages of the regions, and is relatively difficult to use as an index for classifying the regions due to a relatively large standard deviation. On the other hand, Figure 6 (b) is suitable for the indicator of the area classification because the difference between the average of the areas of the sky, land, water, etc. and the relatively small standard deviation.

The object classification unit 130 determines the object type based on the object classification database constructed as described above. In classifying object regions, principal component analysis may be applied to more easily analyze each object region.

Principal component analysis is a technique of multivariate analysis. It is a mathematical procedure that converts variables that can be correlated into independent variables called principal components.

The first principal component found by principal component analysis is an arbitrary axis where the variance of the data is maximum, and the principal component found thereafter is an arbitrary axis that is orthogonal to all the principal components found before and the next largest variance. The new data space found through principal component analysis is in principle the same dimension as the input data. The principal components found represent one axis each, and the origin is the mean of the data. The eigen calue and the eigen vector are obtained through the principal component analysis. The eigenvalue is the size of the eigenvector in the existing data space and the eigenvector is the new coordinate system. This can be projected to the eigenspace through the following equation (3).

Where X ^T is the data coordinate in the existing space, Y ^T is the coordinate in the eigenspace, and ∑ is the eigenvector matrix.

In addition, the object classification unit 130 may use Bayesian classification to determine the type of the object included in the object region after performing the principal component analysis.

Bayesian taxonomy is a method of statistically predicting the classification of a new pattern based on the Bayesian theorem for each classification. When predicting a new pattern classification, the posterior probability of each classification is calculated according to Equation 4 below to determine that the largest posterior probability value is obtained.

Where g _i (x) is the probability that the new pattern x is a classification w _i and w _i Is calculated as the product of the probability p (x | w _i ) and the frequency P (w _i ). p (x | w _i ) is calculated by Equation 5 assuming that w _{i follows} N (μ _i ∑ _i ).

Where x and μ _i are represented as d-dimensional vectors, where ∑ is the d * d covariance matrix, and | ∑ | and ∑ ^{- 1} are the determinants and inverses of the covariance, respectively.

The rendering unit 140 generates a result image corresponding to the target image by applying a brush stroke having a property set in advance according to the object type corresponding to the object region for each object region.

In this case, the rendering unit 140 creates a layer for each object region classification, generates a brush stroke, determines a property of the brush stroke according to the purpose of the layer, and renders the texture by expressing the texture.

7 is a diagram illustrating an example of generating a layer for each object region.

Referring to FIG. 7, (a) of FIG. 7 is a rendering target image for representing the foreground of a leaf. White portions of the images (b) to (h) of FIG. 7 mean an object area. 7 (h) to FIG. 7 (h), which are the object regions including the leaf foreground, are combined to generate a single object region of the leaf foreground. 7 (h) is used for three layers representing the foreground of the leaves.

The renderer 140 may generate three layers according to the object of the object for each object region. The rendering unit 140 uses a layer set as shown in Equation 6 below to generate a rendering result image.

Where n is the number of object areas in the target image, and each object area has layers L _{k , u} for sketching, layers L _{k , d} for describing the shape of the object, and layers L _{k , m} for describing the motion of the object. This is used. Each layer L _k is defined as a tuple of Equation 7 below.

In this case, Λ _k is an area occupied by L _k in the target image, and l _k is a kind of object included in the object area. Π _p has the sketch representation Π _u , the form representation Π _d , and the motility representation Π _m for the purpose of layer representation. When Λ is the entire area of the image to be rendered, the entire area Λ is equal to the union of the respective object areas, as shown in Equation 8 below.

The process of calculating the region of Λ _k l _k is as follows. When the object area is C _{i , j} = {l _i , l _j : i∈ [1, n], j∈ [1, n]}, the area Λ _k of l _k is an object as shown in Equation 9 below. It can be expressed by the sum of the regions Λ _{i, j} of the regions C _{i , j} .

Painters usually draw landscapes in the order of sketches, forms, and movements. It also paints colors in chronological order from distant objects. Accordingly, the rendering unit 140 renders in the order of the sketch representation, the form representation, and the movement expression according to the expression purpose Π _p of the layer as the primary rendering reference. In addition, l _k is used to represent the type and distance of the object as the secondary rendering criteria, and the order is sky, land, water, building / route background, tree trunk, leaves, building / route view, tree trunk view, and leaf foreground. Can be.

Hereinafter, an embodiment in which the rendering unit 140 changes and applies the property of the brush stroke according to the layer type will be described in detail.

When drawing, the first sketch you draw represents the approximate form and color of the scene you are drawing. Therefore, the rendering unit 140 generates a brush stroke in all grids by generating a grid having a uniform size in the target image to render the sketch layer. According to the characteristics of the sketch, the largest brush texture is used among all the layers, and the color extracted from the blurred target image is used as the color of the brush stroke.

After drawing the sketch, express the form of things and finally express the sense of movement. Various forms of expression are used according to the characteristics of objects and shapes of movements according to the shapes and characteristics of movements.

When a similar feature is found in the object to be expressed or the purpose of the expression is the same, a similar method is used. The form representation layer aims to describe the form of each object. Therefore, all shape expression layers commonly use interpolated gradient information so that brush strokes follow the contour direction of an object, and limit the length of the stroke by using an edge map so as not to invade the contour.

Sky, land, and water, which play a large role in compositional composition in landscape painting, are commonly depicted in form of distance. In general, the sky is near the top of the image, the land and water below is closer to the observation point. Therefore, in the shape expression layer of the object area that includes these objects, the distance is determined according to the y-coordinate, and as the distance increases, a larger kernel blurs the brightness value and uses a color that shows less contrast with the surroundings.

Also, artificial structures such as tree trunks and leaves, buildings and roads are divided into foreground and background according to the distance. In the target image, the background is simplified in characteristics of the object compared to the foreground. Therefore, use the same brush texture but use a relatively large grid to provide more detailed shape representation in the foreground. In addition, in order to express the characteristics of each object, it is rendered using different brush textures according to the type of object.

The expression method is applied differently to the movement expression layer according to the motion information of the object. The artists maximize the effect of the movement expression by expressing the form of the object and then selectively painting only the part to emphasize the movement. Also, make sure the direction of the brush stroke matches the direction of movement so that the direction of movement is well represented, and use various colors to emphasize the movement more.

To simulate this representation, the rendering unit 140 generates brush strokes only for lattice cells whose average motion vectors are larger than or equal to a threshold value in the movement expression layer so that the brush stroke directions follow the directions of the average motion vectors. In addition, the larger the average motion vector in color determination, the more noise is added to the H channel to emphasize the movement in various colors.

8 is a diagram illustrating an outline of an algorithm of a rendering attribute database.

Referring to FIG. 8, the rendering unit 140 may perform rendering using a rendering property database in which a method of determining a generation position, a size, a color, and the like of a brush stroke applied differently for each layer and a texture used are stored in advance. .

9 is a diagram illustrating a brush texture applied to each layer.

Referring to FIG. 9, the pictorial rendering apparatus according to the present invention selects and applies a brush stroke for each sketch, shape, and movement expression according to an object type.

Hereinafter, a method of generating a brush stroke when the rendering unit 140 performs rendering will be described in detail.

The renderer 140 uses a brush generation algorithm based on a grid. The size of the grid used for each layer is consulted by the rendering properties database and is determined by considering the effect of the specified brush texture. In the preferred embodiment of the painterly rendering device according to the present invention, the grid size used for each layer is shown in Table 2 below.

Object Classification / Expression Model Sketch shape Movement sky 20 8 10 land 20 5 10 Leaves background 20 10 10 Leaves foreground 20 10 8 Tree foreground 20 2 10 etc 20 3 10

In Table 2, the unit of the grid size is pixels.

The conditions for creating a brush depend on the purpose of the layer's presentation. The sketch or shape representation layer traverses the generated grid and creates a brush when the current position is included in the object area. On the other hand, the movement expression layer expresses the movement by selecting only the part where the movement is observed by generating a brush only when the average movement size in the grid is larger than a preset value.

Hereinafter, a method of determining an attribute of a brush when rendering in the rendering unit 140 will be described in detail.

The attributes of the brush determined when the rendering unit 140 performs rendering are position, direction, length, and color. The properties of all brushes are determined by applying various algorithms to meet the presentation intent of each layer.

The position of the brush stroke is determined at the same time as the creation, and the process of determining the remaining properties is largely divided into two. The former is used in the sketch expression layer and the shape expression layer, and the latter is used in the motion expression layer by referring to the gradient of the image to be rendered and the direction of the average motion vector.

Determining the direction of a brush stroke in the painterly rendering is the same as the direction of the surrounding contour is one of the important factors. Many studies have used gradients to determine the direction of brush strokes and used various gradient interpolation algorithms to make the brush strokes show more consistent orientation in the resulting image.

The rendering unit 140 determines the brush stroke direction of the sketch expression layer and the shape expression layer by using an interpolated gradient map of the target image using an edge tangent flow (ETF) technique, and uses the direction of an average motion vector. Determine the brush stroke direction of the movement expression layer. In this case, the ETF may be applied to the direction of the average motion vector in order to maintain the consistency of the direction of the brush stroke for the motion expression layer.

10 is a diagram illustrating brush stroke directions along an interpolated gradient of the technique of the ETF. Referring to FIG. 10, it can be seen that the consistency of the brush stroke direction is greatly improved compared to the direction of the original gradient.

The renderer 140 prevents the brush strokes from invading the contour to maintain structural consistency. In this case, for the sketch expression layer having a relatively low importance of the shape description, the length of the brush stroke is limited by considering only the region boundary instead of the outline. The region boundary generates and uses region information of a layer in a mask map form. For the shape expression layer and the motion expression layer, the contour and the area boundary are considered together so that the brush stroke does not invade the boundary of the object. Canny edge detection may be used to obtain a contour map of the target image.

11 is a diagram illustrating a visualization image of a mask map and an outline map of a leaf background region. Referring to FIG. 11, the white portion represents the leaf background region and the gray line represents the extracted contour.

Since the sketch expression layer aims to roughly describe the shape, the rendering unit 140 determines the color of the representative brush stroke of each object area in the image to which Gaussian blur is applied instead of the target image.

In the shape expression layer, color determination methods are classified according to types of objects. When expressing sky, land, water, etc. in natural paintings, the contrast of colors in the far place is relatively lower than in the near place. Therefore, the rendering unit 140 adjusts the brightness value according to the distance in the form representation layer of the sky, the ground, and the water. The perspective relationship of the distance is estimated from the y coordinate. The greater the y value, the farther away the sky is from the observation position. On the other hand, in the case of land and water, the larger the y value, the closer the observation position is.

The rendering unit 140 obtains the minimum / maximum y value of each object region for each layer and reduces the difference between the brightness values of the brush strokes applied to the object region located far away using the y value through Gaussian blur. The background and foreground layers except for sky, land, and water determine the brush stroke color from the original color of the target image.

The color of the movement expression layer expresses the movement more dramatically by applying random noise to the hue. That is, as described above, the larger the average motion vector of the brush stroke generating position is, the stronger the noise is applied to express the effect.

Hereinafter, a method of determining the texture representation of the brush stroke when rendering in the rendering unit 140 will be described in detail.

Painterly rendering aims to produce a result of painterly feeling. The texture representation of brush strokes is a very important factor in achieving the purpose of this painterly rendering. In a preferred embodiment of the pictorial rendering apparatus according to the present invention, the rendering unit 140 may express the texture of brush strokes through texture mapping.

In this case, two black and white images are used for each brush for texture mapping of the brush. One is the alpha mask for alpha blending, and the other is the texture for brush texture.

Brush textures exist as two-dimensional images. Therefore, rendering a brush requires a conversion process that changes the shape of the brush texture to match the position and shape of the area where the brush is applied. The conversion process consists of scaling, rotation, and color application.

FIG. 12 is a diagram illustrating a process of converting a brush texture used by the renderer 140 to express a texture of a brush stroke.

Referring to FIG. 12, an attribute value of a brush is used for the transformation. First, the original texture is scaled according to the set length of the brush stroke. The scaled brush texture is rotated to match the direction of the brush stroke, and finally the specified color is applied. This converted texture is blended with the previously applied brush by the alpha value of the alpha mask to complete rendering of the brush.

13 is a flowchart illustrating a process of performing a preferred embodiment of the pictorial rendering method according to the present invention.

Referring to FIG. 13, the motion extractor 110 extracts motion information of each pixel from a plurality of input images input at a fixed time point (S1100), and the area divider 120 selects a target image selected from a plurality of input images. A plurality of pixels constituting a plurality of pixels are divided into a plurality of object regions (S1200), and the object classification unit 130 stores coordinate information, color information, and motion information of the object in an image corresponding to a preset object type. The type of the object included in each object area is determined based on the object classification database (S1300). The renderer 140 generates a result image corresponding to the target image by applying a brush stroke having a property set in advance to each object region according to the type of the object included in the object region (S1500).

An experiment was conducted to evaluate the performance of the pictorial rendering apparatus according to the present invention. The program used for the experiment was implemented based on MFC using Microsoft's Visual Stidio 2005 on a Windows-based PC. Also, OpenCV 1.0 library was used for 2D image processing. The system used for the experiment was a standard PC with an Intel core2Duo E6550 CPU (2.33MHz) and 4GB of RAM. It took about 30 seconds in the scene analysis stage and about 20 seconds in the rendering stage based on 38 frames of 640 * 480 size video.

14 is a diagram illustrating an example of rendering an input image using the pictorial rendering apparatus according to the present invention.

Referring to FIG. 14, FIG. 14A illustrates an input image to be rendered, and FIG. 14B illustrates an image of motion information. 14C to 14M show a rendering result of a layer according to the type of each object and the purpose of expression. The objects shown in the target image were classified into seven kinds of objects, and based on the motion information, a motion expression layer for the foreground of water and leaves was created and rendered.

14 (n) shows a final result image of the rendering. The resulting image was derived by stacking the rendered layers according to the rendering order of each layer. You can see tree trunks, leaves, sky, water and ground rendered with different brush textures. In addition, in the case of water in which the movement mainly occurs in the target image, it can be seen that the brush stroke color is different from the brush stroke color in the shape description according to the rendering method of the motion expression layer.

15 is an image comparing an application result of an algorithm for differentiating rendering according to an object type in a pictorial rendering apparatus according to the present invention.

Referring to FIG. 15, by expressing objects in various textures according to the object type of each region, the sky, the leaves, the buildings, and the water were rendered using the defined rendering methods and textures, respectively. In addition, in the case of water where motion is well represented, the conventional method preserves and renders the shadow outline of the bridge, but the pictorial rendering apparatus according to the present invention shows the motion better through rendering considering the motion of the input image instead of the outline.

16 is a result comparison image according to the presence or absence of color control using motion information.

Referring to FIG. 16, it can be seen that the movement is emphasized by adjusting the brush stroke color of the motion expression layer using the motion information in the pictorial rendering apparatus according to the present invention.

The present invention can also be embodied as computer-readable codes on a computer-readable recording medium. A computer-readable recording medium includes all kinds of recording apparatuses in which data that can be read by a computer system is stored.

Examples of the computer-readable recording medium include a ROM, a RAM, a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, and the like, and may be implemented in the form of a carrier wave (for example, transmission via the Internet) . The computer readable recording medium can also be distributed over network coupled computer systems so that the computer readable code is stored and executed in a distributed fashion.

Although the preferred embodiments of the present invention have been shown and described above, the present invention is not limited to the above-described specific preferred embodiments, and the present invention belongs to the present invention without departing from the gist of the present invention as claimed in the claims. Various modifications can be made by those skilled in the art, and such changes are within the scope of the claims.

110-motion extractor
120-region divider
130-object classifier
140-renderer

Claims (17)

A motion extractor configured to extract motion information of each pixel from a plurality of input images input at a fixed viewpoint;
An area dividing unit dividing a plurality of pixels constituting the selected target image among the plurality of input images into a plurality of object regions;
An object classification unit determining a type of an object included in each object area based on an object classification database storing coordinate information, color information, and motion information of the object in an image corresponding to a preset object type; And
And a rendering unit generating a resultant image corresponding to the target image by applying a brush stroke having a predetermined property to each object region according to the type of the object included in the object region. A pictorial rendering device based on motion information of an object. The method of claim 1,
The motion extractor extracts motion information corresponding to the target image by using an optical flow technique that minimizes the influence of noise through an average motion vector and measures the size of motion of pixels constituting the plurality of input images. A pictorial rendering device based on motion information of an object. 3. The method according to claim 1 or 2,
The region dividing unit divides the target image into a plurality of object regions by a mean shift dividing method using color average values of the plurality of pixels constituting the target image. Rendering device. 3. The method according to claim 1 or 2,
The object classifying unit pictorial rendering apparatus based on the motion information of the object, characterized in that for determining the type of the object by the principal component analysis and Bayesian classification method based on the motion information and the color and coordinate information obtained from the respective object area. . 3. The method according to claim 1 or 2,
The rendering unit generates a sketch expression layer, a shape expression layer, and a motion expression layer corresponding to each object area, and based on a rendering property database in which positions, directions, lengths, and colors of a brush are stored in advance as attributes of the brush stroke. And pictorial rendering apparatus based on motion information of an object, wherein the rendering is performed on each layer. 6. The method of claim 5,
The rendering unit determines a direction of a brush stroke applied to the sketch expression layer and the shape expression layer by using a gradient map obtained from the target image by an edge tangent flow (ETF) technique, and inputs the plurality of inputs. A pictorial rendering apparatus based on motion information of an object, characterized in that the direction of the brush stroke applied to the motion expression layer is determined based on the direction of the average motion vector of the image. 6. The method of claim 5,
The rendering unit generates the boundary of each object area in the form of a mask map to determine the length of the brush stroke applied to the sketch expression layer, and the contour and the respective contours extracted from the target image using the Canny contour extraction technique. And a brush stroke applied to the shape expression layer and the movement expression layer based on a boundary of an object region. 6. The method of claim 5,
The renderer determines a color of a brush stroke applied to the sketch expression layer based on color information of each pixel of the image to which Gaussian blur is applied to the target image, and applies random noise to a hue channel of the target image. A pictorial rendering apparatus based on motion information of an object, characterized by determining a color of a brush stroke applied to the movement expression layer based on color information of each pixel of an image. A motion extraction step of extracting motion information of each pixel from a plurality of input images input at a fixed viewpoint;
A division region extraction step of dividing a plurality of pixels constituting a selected target image among the plurality of input images into a plurality of object regions;
An object classification step of determining an object type of each object area on the basis of an object classification database storing coordinate information, color information, and motion information of the object in an image corresponding to a preset object type; And
And a rendering step of generating a resultant image corresponding to the target image by applying a brush stroke having a property set in advance according to the object type corresponding to the object region to each object region. Painterly rendering method based on motion information of an object. The method of claim 9,
The extracting of motions may include extracting motion information corresponding to the target image by using an optical flow technique for minimizing the influence of noise through an average motion vector and measuring the magnitude of motion of pixels constituting the plurality of input images. A pictorial rendering method based on motion information of an object. 11. The method according to claim 9 or 10,
The segmentation step may be performed by splitting the target image into a plurality of object regions by a mean shift splitting method using color average values of a plurality of pixels constituting the target image. Painterly rendering method. 11. The method according to claim 9 or 10,
In the object classification step, the kind of the object is determined by principal component analysis and Bayesian classification based on the motion information and the color and coordinate information obtained from the respective object areas. Way. 11. The method according to claim 9 or 10,
The rendering step generates a sketch expression layer, a shape expression layer, and a motion expression layer corresponding to each object area, and renders a rendering property database in which the position, direction, length, and color of the brush are stored in advance as properties of the brush stroke. A pictorial rendering method based on motion information of an object, characterized in that the rendering is performed for each of the layers as a basis. The method of claim 13,
The rendering may include determining a direction of a brush stroke applied to the sketch expression layer and the shape expression layer using a gradient map obtained from the target image by an edge tangent flow (ETF) technique. The method of the pictorial rendering based on the motion information of the object, characterized in that for determining the direction of the brush stroke applied to the motion expression layer based on the direction of the average motion vector of the input image. The method of claim 13,
In the rendering step, the boundary of each object area is generated in the form of a mask map to determine the length of the brush stroke applied to the sketch expression layer, and the contour and the respective contours extracted from the target image using the Canny contour extraction technique. And a length of a brush stroke applied to the shape expression layer and the motion expression layer based on a boundary of an object area of the pictorial rendering method. The method of claim 13,
The rendering step determines a color of a brush stroke applied to the sketch expression layer based on the color information of each pixel of the image to which Gaussian blur is applied to the target image, and applies random noise to a hue channel of the target image. A pictorial rendering method based on motion information of an object, characterized by determining the color of a brush stroke applied to the motion expression layer based on color information of each pixel of an applied image. A computer-readable recording medium having recorded thereon a program for executing the method of claim 9 or 10 on a computer.

Images