JP2005341569A - Method for generating stylized image of scene including object - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a method for generating a stylized image of a scene including an object. <P>SOLUTION: A set of n input images are acquired of the scene with a camera. Each one of the n input images is illuminated by one of a set of n light sources mounted on a body of the camera at different positions from a center of projection of a lens of the camera. Ambient lighting can be used to illuminate one image. Features in the set of n input images are detected. The features include depth edges, intensity edges, and texture edges to determine qualitative depth relationships between the depth edges, the intensity edges and the texture edges. The set of n input images are then combined in an output image to enhance the detected features according to the qualitative relationships. <P>COPYRIGHT: (C)2006,JPO&NCIPI

Description

  The present invention relates generally to rendering non-photorealistic images, and more particularly to generating non-photorealistic images from images acquired from natural scenes and objects.

  An object of the present invention is to generate a stylized image from images acquired from real world scenes and objects. Stylized images can deepen the viewer's understanding of the shape of the object being depicted. Non-photorealistic rendering (NPR) techniques are intended to outline the shape of an object, highlight moving parts to show motion, and reduce visual clutter such as shadows and texture details. Stylized images are useful for rendering low contrast, geometrically complex scenes such as the body of a patient undergoing an examination, such as mechanical parts, plants or endoscopes.

  If a rich 3D geometric model of the scene is available, rendering a subset of contours that depends on the simple scene viewpoint is a relatively well-understood task (Saito et al., “Comprehensible Rendering of 3 -D Shapes "(Proceedings of SIGGRAPH '90, 1990), and" Real-Time Non-photorealistic Rendering "by Markosian et al. (Proceedings of SIGGRAPH '97, pp. 415-420, 1997)). However, it is difficult, if not nearly impossible, to extend this approach to real-world scenes, such as flowered plants, by first generating a 3D model of the plant.

  It is desirable to bypass the acquisition of 3D scene geometry. Instead, the goal is to generate a stylized image of the real world scene directly from the image acquired by the camera.

  In the prior art, most of the available techniques process a single image to produce a stylized image. Usually, morphological operations such as image segmentation, edge detection and color assignment are applied. Some techniques are intended for stylized depiction (DeCarlo et al. “Stylization and Abstraction of Photographs” (Proc. Siggraph 02, ACM Press, 2002), and Hertzmann “Painterly Rendering with Curved Brush Strokes”. of Multiple Sizes "(see ACM SIGGRAPH, pp. 453-460, 1998), some of which improve legibility.

  Interactive techniques for stylized rendering such as rotoscoping are also known. However, it is desirable to automate the process of generating stylized images without the need for careful manual manipulation.

  In aerial photography, shadows in the scene are identified by thresholding a single luminance image assuming a flat ground and a uniform albedo to infer terrain and building height (Huertas et al., “ "Detecting buildings in aerial images" (Computer Vision, Graphics and Image Processing 41, 2, pp. 131-152, 1998), "Methods for exploiting the relationship between buildings and their shadows in aerial imagery" (IEEE Transactions on Systems) , Man and Cybernetics 19, 6, pp. 1564-1575, 1989), and Lin et al., “Building detection and description from a single intensity image” (Computer Vision and Image Understanding: CVIU 72, 2, pp. 101-121, 1998)).

  Some techniques use shadow extraction techniques to improve the capture of shadows, thereby determining shadow mattes ("Shadow matting and compositing" by Chuang et al. (ACM Trans. Graph. 22, 3, pp. 494-500, 2003) or removing all shadows to improve scene segmentation (Toyama et al. “Wallflower: Principles and Practice of Background Maintenance” (ICCV, pp. 255-261, 1999)). Using intrinsic images, other techniques remove shadows without detecting shadows clearly (Deriving intrinsic images from image sequences by Weiss (Proceedings of ICCV, vol. 2, pp. 68). -75, 2001)).

  Stereo techniques using passive and active illumination are usually designed to determine depth values or surface orientations rather than detecting depth edges. Depth edges or discontinuities cause problems with traditional stereo techniques due to partial occlusion that disrupts the matching process (Geiger et al., “Occlusions and binocular stereo” (European Conference on Computer Vision, pp. 425-433). , 1992)).

  Some techniques attempt to model discontinuities and occlusions directly (Intille et al. “Disparity-space images and large occlusion stereo” (ECCV (2), pp. 179-186, 1994), Birchfield et al. “Depth discontinuities by pixel-to-pixel stereo” (International Journal of Computer Vision 35, 3, pp. 269-293, 1999) and Scharstein et al. “A taxonomy and evaluation of dense two-frame stereo correspondence algorithms (International Journal of Computer Vision, vol. 47 (1), pp. 7-42, 2002)).

  Active lighting methods that produce better results than usual are used for depth extraction, shape from shadows, shadow-time stereo, and photometric stereo. Unfortunately, these methods are unstable around depth discontinuities (Sato et al. “Stability issues in recovering illumination distribution from brightness in shadows” (IEEE Conf. On CVPR, pp. 400-407, 2001). ).

  One technique performs logical operations on detected luminance edges in images acquired under a wide variety of illuminations and preserves shape boundaries (Shirai et al., “Extraction of the line drawing of 3-dimensional”). objects by sequential illumination from several directions "(Pattern Recognition, 4, 4, pp. 345-351, 1972)). This technique is also limited to uniform albedo scenes.

  When photometric stereo techniques are used, luminance statistics can be analyzed to detect areas of high curvature or folds in the occluded contours (Huggins et al., “Finding Folds: On the Appearance and Identification of Occlusion "(IEEE Conf. On Computer Vision and Pattern Recognition, IEEE Computer Society, vol. 2, pp 718-725, 2001)). This technique detects the area near the occluded contour, but not the contour itself. This technique assumes that the surface portion is locally smooth. Thus, this technique fails for flat foreground objects such as a leaf or piece of paper, or for edge-independent edges such as cube corners.

  Techniques for determining shapes from shadows or dark areas construct a continuous representation, such as a projection, from a moving light source. Continuous depth estimation is possible by projection (Raconstruct et al., “Reconstruction of three-dimensional surfaces from two-dimensional binary images” (IEEE Transactions on Robotics and Automation, vol. 5 (5), pp. 701-710). , 1989), “Space occupancy using multiple shadowimages” by Langer et al. (International Conference on Intelligent Robots and Systems, pp. 390-396, 1995), and “On 3-D surface reconstruction using shape from shadows” by Daum et al. (CVPR , pp. 461-468, 1998)). This technique requires accurate detection of shadow start and end. This makes it difficult to estimate the continuous height.

  An overview of shadow-based shape analysis methods is provided by Yang “Shape from Darkness Under Error” (PhD thesis, Columbia University, 1996), Kriegman et al. “What shadows reveal about object structure” (Journal of the Optical Society of America, pp. 1804-1813, 2001) and “Shadow Carving” by Savarese et al. (Proc. Of the Int. Conf. On Computer Vision, 2001).

  A limitation common to known active illumination is that the light source needs to surround the object in order to obtain effective shading and change in the image from the estimated or known 3D light position. This requires a fixed lighting rig that limits the application of these techniques to studio or industrial environments.

  Because depth edges often violate the smoothness assumption inherent in many techniques, it is desirable to extract depth edges from an image in a manner complementary to known methods for determining depth and 3D directional shape.

  If the location of depth discontinuities can be reliably detected and supplied as input, the performance of many 3D surface reconstruction processes can be significantly enhanced.

  It is desirable to detect depth edges without solving the correspondence problem or analyzing pixel brightness statistics using moving light. In addition, complex real-world scenes with rapidly changing surface normals (eg, potted plants), or scenes with high depth complexity or small changes in brightness (eg, humans undergoing car engines or screening) It is desirable that an NPR image can be generated from the bone).

  The present invention provides a system and method for non-photorealistic rendering (NPR) that accurately captures and conveys real-world scene and object shape features without reconstructing the 3D model.

  The present invention uses a camera having a plurality of light sources, eg, flash units that are positioned around the lens of the camera body and cast shadows from different angles along depth discontinuities in the scene.

  Next, the projection-geometric relationship between the camera and the flash unit is used to detect depth discontinuities and distinguish them from luminance edges due to material discontinuities in the scene. .

  The present invention uses the detected edge features to generate stylized still and moving images, such as videos. The detected features can be emphasized and unnecessary details can be suppressed. In addition, features from a plurality of images can be combined. The resulting image clearly conveys the 3D structure of real world scenes and objects.

  The system according to the invention can be configured as a single package of the same size and shape as a conventional digital camera. Thus, the system according to the present invention is portable, easy to use and can be manufactured at low cost compared to prior art NPR systems and methods.

  The present invention provides a simple but effective way to convey shapes by rendering stylized images and stylized video of real world scenes. The present invention extracts geometric features from multiple images of a scene using the epipolar relationship between the light source and the cast shadow. By taking advantage of image space discontinuities rather than based on 3D scene reconstruction, the method of the present invention can strongly capture and render the underlying scene primitives in different ways.

  The present invention provides a basic camera configuration, associated feature extraction and rendering process. Since depth edges are fundamental primitives in every image of a real world scene, the present invention describes how depth edge information can be used in applications beyond non-photorealistic rendering.

  The present invention uses a still camera or video camera having a plurality of flash units arranged around the lens of the camera body. Instead of estimating the full 3D coordinates of points in the scene and then finding depth discontinuities, the method according to the present invention replaces the general 3D problem of depth edge recovery with the problem of luminance step edge detection. To reduce.

  Thus, imaging geometry of imaging geometry can be used to render stylized images of real world scenes and objects simply and inexpensively. The camera solution of the present invention can be a useful tool for professional artists as well as general photographers.

NPR Camera FIG. 1A shows a digital camera 100 for generating non-photorealistic (NPR) images according to the present invention. The camera 100 includes a plurality of light sources, for example, flash units 101 to 104 and a single lens 105. The flash units 101 to 104 are distributed around the projection center (COP) 106 of the lens 105. In order to obtain the best results, the flash unit is placed as close as possible to the COP. Therefore, it makes sense to attach the flash unit to the front face 107 of the camera body.

  In one embodiment described below, the flash unit is positioned at different distances from the projection center of the lens.

  In the optimal configuration shown in FIG. 1B, the flash units 101-104 are operated in left / right and top / bottom pairs. As a result, the pixels in the images 110 to 114 are imaged for at least four lighting conditions, that is, a shadow on the lower side, a shadow on the upper side, a shadow on the left side, and a shadow on the right side. Can be done. Thus, adjacent pixels are covered with shadows in at least one image and not covered with shadows in at least one other image. Moreover, even if there is a shadow, one image can be collected with light from the surroundings so that there is little.

  This configuration also makes the epipolar traversal efficient. In the case of a left-right pair, the crossing can be approximated along a horizontal scan line. In the case of an upper and lower pair, the crossing is along the vertical direction. FIG. 1C shows a configuration with three flash units. FIG. 8 shows another configuration.

  The flash unit can be activated by an optically coupled LED. The LEDs are turned on one by one in a certain order by the microcontroller 120 and the corresponding flash unit is activated. The flash duration is about 4 milliseconds. One image is collected each time one flash unit illuminates the scene.

  The camera resolution is about 4 megapixels, although lower or higher resolutions can be realized.

  As with all new digital cameras, the camera of the present invention includes a microprocessor 120 and a memory 130. The microprocessor 120 is designed to perform the operations described herein, specifically the method 200 for generating stylized images. Please refer to FIG.

  A memory 130 is used to store the acquired image and other intermediate images in a linearized form. The memory also stores data used by the methods and procedures described herein. The stylized image 201 that is output can also be stored in memory for later downloading via the port 140 to an external device for viewing.

  It should be noted that the camera 100 can continuously capture a plurality of images at a high speed to generate a frame sequence or video. These can also be stylized as described herein. Note also that images 110-114 can be collected by using other techniques consistent with the present invention for later processing in a stand-alone processor, such as a desktop system or portable computing device. Should.

  It should also be noted that the camera may be in the form of a video camera. In this case, the camera collects a plurality of image sequences (videos) so that each sequence corresponds to a different lighting condition by the flash unit and the output is a stylized video.

Camera Operation FIG. 2 shows the inventive method 200 for generating a stylized image 201 of a scene. Stylization, as described herein, emphasizes any non-photorealistic image with specific details such as edges, reduces and abstracts other details such as textures and monotonous backgrounds Or it means that it is simplified by other methods.

  Initially, a set of images 110-114 is collected 210 by the camera 100 of FIG. 1 or by some other camera and flash unit similar to the configuration shown in the figure. Each image is collected under different controlled lighting conditions. In the illustrated configuration, one image is taken with ambient light, ie without a flash, one image is taken with the upper flash unit, and one image is taken with the lower flash unit. One image is taken using the left flash unit and one image is taken using the right flash unit. At a minimum, two flash units and three images are required, but better results can be obtained when using more, for example 4-8 images and flash units. Even better results can be obtained if the flash units are arranged at different distances from the projection center of the lens.

  A set of images is processed by the microprocessor 120, silhouette edges are detected 220, and texture regions are identified 230. A stylized image 201 is then provided using the silhouette edges and texture regions of the image combination. For example, the width of the silhouette edge is proportional to the size of the depth discontinuity at that edge, and the texture region is deemphasized according to the color gradient.

  The present invention classifies each pixel in each image as either a silhouette edge pixel, a texture edge pixel, or a featureless pixel.

  In the present invention, the term silhouette edge pixel is used to refer to a pixel having a C0 depth discontinuity in the scene. These silhouette edge pixels include pixels corresponding to points on the surface of the object in the scene whose normal is perpendicular to the viewing direction. The invention also includes pixels that correspond to the edges of thin objects, such as the border of a piece of paper, and the edges of objects that do not depend on the viewpoint, such as cubes, both of which are depth discontinuous. Has a point. Silhouette edge pixels can also belong to the interior of an object due to self-occlusion.

  Texture edge pixels correspond to reflectance changes and material discontinuities in the scene. Texture regions are usually outlined by texture edges. However, the texture region may not be completely enclosed.

  Pixels in the featureless region of the input image correspond to regions in the scene that have a generally constant reflectivity and low curvature, such as a monotonous background wall. These pixels can also accommodate slight changes in appearance due to changes in illumination or viewing direction, including anisotropic reflections on “glossy” materials such as metals and plastics. .

Image Collection A set of images 110-114 is collected 210 from a scene using flash units 101-104 that are placed very close to the projection center (COP) 106 of the lens 105. Please refer to FIG. Due to the short baseline 150 between the COP and the flash unit, a narrow sliver of shadows representing depth discontinuities (edges) in the scene are located near each silhouette edge in the image. appear.

  By combining information about attached cast shadows from two or more images with different distinct lighting, the present invention can detect silhouette edge pixels.

  In this specification, attached cast shadows are defined in image space. This definition is quite different from the conventional shape-from-shadow definition in object space. In that case, a surface boundary in which the ray is tangential to a smooth object is considered to have an attached shadow, or “as is” shadow. Here, the attach cast shadow is generated when an object and a shadow cast on the background by the object are continuous in the image space.

  For edges that do not depend on most viewpoints, the shadowed area contains the shadow cast on the next depth layer. In the case of an edge that depends on the viewpoint, a small part of the shaded area includes its own shadow.

  In general, if the shadow is on the “opposite” side of the flash unit, that is, if the flash unit is on the right side of the lens, the shadow will appear on the left side of the depth discontinuity in the camera image and another flash unit will Cast a shadow on the place. See FIG. 5B.

Silhouette Edge Pixel Detection FIG. 3 shows a procedure 300 for detecting silhouette edge pixels that underlie the present technique. The concept is surprisingly simple. Although the image acquisition process of the present invention is closely related to photometric stereo, to the best of our knowledge, a computer vision system for detecting depth discontinuities as reflected by shadows in a scene The illuminance difference stereo has never been used. The photometric stereo allows the present invention to classify other edges according to the erasing process.

  The basic procedure 300 operates as follows.

A surrounding image I _ambient 301 of a certain scene is collected. However, I _ambient is an image taken under the light from the surroundings without using any flash unit.

At location P _k , a point light source, ie a flash unit, is used to collect n images I ′ _k (k = 1,..., N) 302 of the scene illuminated in a controlled manner. The images can be collected in any order. In the example of the present invention, n = 4.

A difference image I _k 303 is obtained by subtracting 310 the surrounding image from the illuminated image, ie I ′ _k −I _ambient .

From the difference image, a maximum image I _max 304 is generated 320 as I _max (x) = max _k (I _k (x)) (k = 1,..., N). That is, each pixel in the maximum image has the maximum luminance value among the corresponding pixels in the difference image.

For each difference image I _k , a radial image 305 is generated 330 for all pixels (x) by division as R _k (x) = I _k (x) / I _max (x). That is, each difference image is divided by the largest image.

Pixel e _k 341 in the radiation image is an epipole pixel, ie an epipole pixel is an image of the corresponding light source at P _k . Epipoles can be identified using conventional stereo photography techniques.

For each radiation image R _k 305, the image is traversed 340, comparing the pixel brightness and the pixel brightness of the epipole pixels. In the present invention, radial traversal means detecting a luminance edge along a radial line from an epipole pixel to a given pixel. This traversal detects a transition from the illuminated area to the shadow area and vice versa.

  A pixel y is identified in a step having a negative luminance transition. Those pixels y having a negative luminance transition are indicated as silhouette edge pixels 306. All pixels identified as silhouette pixels can be overlaid in the maximum image, and as a result, when all images are processed, the maximum image will show all silhouette contours.

  FIG. 4 is a stylized image 400 of a flower vase with silhouette edges enhanced according to the present invention.

  If a sufficiently large number of flash units n (at least two, but usually 4-8) are used, silhouette edge pixels can be detected in all orientations and sufficient depth differences. If the silhouette has a component perpendicular to the epipolar line, an edge pixel having a negative luminance transition from the illuminated area to the shaded area is detected.

  It should be understood that for very simple scenes, several silhouette edges can be detected from a single illuminated image. For example, in a scene having mainly vertical depth discontinuities, such as a pile fence, multiple edges can be detected by illumination from one side.

The procedure 300 is now described in further detail. The I _max image is an approximation of an image having a light source on the COP 106 of the camera. This approximation is generally accurate when the point light sources have the same size and the baseline between the light sources is significantly smaller than the depth of the scene. Thus, the I _max image has no pixels in the shadow.

In the area illuminated by the light source k, the ratio (I _k / I _max ) is approximately equal to 1 and in the shaded area is approximately equal to 0. Therefore, the location of the negative luminance transition indicates a shadow edge.

_Given a light source at P _k , the luminance at pixel x for point X of the 3D scene with diffuse reflection in the camera coordinate system is given by I _k (x).

When the point X of the 3D scene is illuminated by the light source P _k , the following equation holds.
I _k (x) = μ _k ρ _k L _k (x) · N (x), otherwise
I _k (x) = 0
Here, the value μ _k is the magnitude of light, ρ _k (x) is the reflectance at point X in the scene, and L _k (x) is the normalized light vector L _k (x) = P _k -X, N (x) is the surface normal, and they are all in the camera coordinate system.

Therefore, when a point X in the scene is illuminated by a point light source P _k , the ratio is expressed as follows:

For diffuse objects with ρ _k other than 0, this ratio R _k (x) is independent of the albedo ρ _k and is a function of local geometry only. Furthermore, if the light source is close to the camera COP 106 and X >> P (k), this ratio is approximately (μ _k / max _i (μ _k )), which is a set of omnidirectional Constant in the case of sex light sources. Note that R _k (x) is very small near the silhouette of a curved object with edges that depend on the viewpoint.

This is (L _k (x) · N (x) ˜ = 0), and the dot product of the light source on the opposite side is larger, that is, (L _i (x) · N (x)> L (x ) · N (x)). Therefore, the luminance of the pixel in the radiation image R _k (x) decreases rapidly even when the pixel is not in the shaded area. However, this is not a big problem, and as a result, only the shaded area becomes thick, and it does not lead to the inversion of the luminance profile along the epipolar line.

Therefore, the ratio R _k (x) is approximately close to 0 in the shaded area due to secondary scattering. The luminance profile along the epipolar line shows a sharp negative transition at the silhouette edge when traversing from a shadowless foreground to a shadowed background, and when traversing from a shadowed area on the background to an unshadowed area Shows a sharp positive transition.

  Any standard 1-D edge detector can be applied along the radial epipolar line from the epipole to a given pixel, both these transitions are detected, and the pixels with negative transitions are silhouette edge pixels As indicated.

  Since luminance transitions are detected, noise and secondary scattering can affect the accuracy of the location of detected silhouette edge pixels, which also always exists.

  Several conditions exist when a negative luminance transition at the silhouette edge cannot be detected in the radiation image R (x), or when other conditions cause false transitions. The silhouette can be missed due to isolated shadows, low background albedo, holes and pits, or when the silhouette is in shadowed areas. Some pixels may be misclassified as silhouette edge pixels due to specular reflection, self-occlusion, or silhouettes that depend on the viewpoint.

  There is a trade-off in selecting the baseline between the camera COP and the flash unit. When the image width for the shadow d is increased, the longer the baseline, the better. However, when the baseline is shortened, the separation of the shadow is avoided. Even better results can be obtained if the baseline is varied.

The width of a particular attached cast shadow in an image is d = f (z ₂ −z ₁ ) B / (z ₁ · z ₂ ). Where f is the focal length, B is the baseline in mm, and z ₁ and z ₂ are the depth in mm to the shadowed and shadowed edges.

Shadow separation occurs when the threshold width T of the object is smaller than (z ₂ −z ₁ ) × B / z ₂ . Thus, shorter baselines allow for smaller widths T without shadow separation. Since a higher resolution camera can be used to increase the effective focal length, the present invention provides a high resolution camera with a very small baseline and a long distance or “depth” to the object (z ₂ ). Used.

The non-uniformity of the light source also affects the result. Since (μ _k / max _i (μ _k )) is not a constant, it affects R _k (x). Conveniently, even when using a non-uniform light source, it does not introduce false edges in R _k because the lobe brightness varies smoothly across the field.

  Detection of negative luminance steps leading to shadow areas is a binary decision, and the method according to the invention can withstand noise. However, the light generated by the flash unit still needs to be superior to ambient light. The present invention is also based on the assumption that silhouettes are separated from a background having a finite depth, which means that the present invention requires a background. Another embodiment described below does not rely on the requirement that this background be necessary.

Reducing details in texture regions The present invention also provides a procedure for reducing details or complex portions in regions of an image that are not related to the silhouette of a scene, such as texture and lighting variations. . Given a silhouette edge pixel 306, the pixels belonging to the texture edge can be identified 230. Therefore, the texture region can be specified. These pixels are independent of the direction of illumination. Texture edges are those from the luminance edges in the maximum image I _max, minus the detected silhouette edges.

  Ideally, the present invention should identify a set of pixels corresponding to all texture regions, i.e., texture details in the original image, and de-emphasize the texture details in the output image 201. However, although the present invention can identify texture edges, it cannot reliably find all texture regions. This is because texture edges do not necessarily form closed contours around texture regions due to gaps after silhouette edge detection, or such regions disappear gradually and become dominant in the scene. This is because there is a possibility of becoming a different color.

  Thus, in the present invention, for example, performing “adjustable” abstractions that can give higher importance to geometric features and can de-emphasize texture features such as edges and regions Is desirable. In the present invention, it is also desirable to delete pixels that do not have a dominant color.

  In one coping method, the area associated with the texture edge is blurred. It is also possible to simply assign a weighted average value of the color of neighboring non-feature pixels to texture pixels and neighboring pixels. However, it only reduces the texture region boundary and does not remove the gradient. Furthermore, texture regions outlined by texture edges can have a width of several pixels and are therefore not deleted completely. Another approach uses distance field or diffusion type techniques.

  Instead of operating at the pixel level, the present invention uses an edge-based procedure.

  The edge-based procedure according to the invention is based on the observation that a high luminance gradient in texture pixels separates non-dominant color pixels from dominant color pixels. If the pixel is reconstructed from the gradient without using a high gradient in the texture pixel, the color pixel from the opposite side of the texture pixel is automatically filled with the non-dominant color pixel smoothly. There is no need to make a decision as to which luminance value should be used to fill a hole, as is necessary in conventional pixel-based systems, and no need to feather or blur.

Edge Enhancement The procedure according to the invention for detecting silhouette edge pixels also generates further useful information that can be used to enhance an image.

The shadow width d = f (z ₂ −z ₁ ) B / (z ₁ z ₂ ) is proportional to the depth difference (z ₂ −z ₁ ) at the depth discontinuity point. This information can be used to enhance the image.

At the silhouette edge, the present invention can determine which side of the silhouette belongs to the foreground and which side belongs to the background. The calculation is based on which side of the edge has higher brightness in the _Rk image at negative transitions along the epipolar line.

  This qualitative depth relationship can also be used to enhance edges. The present invention first generates a silhouette image in which the silhouette is white on a black background. The present invention performs convolution using a filter that is the gradient of the edge enhancement filter. The present invention uses a Gaussian filter minus an impulse function. According to the present invention, when the convolved image is integrated, a sharp transition is obtained at the silhouette edge.

Comparison with prior art Better lighting can be used to improve contrast and enhance objects against the background. Although the success of this approach is generally dependent on the skill of the photographer, the present invention provides a general solution based on obvious geometry. Further, since the light source is close to the COP of the camera, the camera 100 having a sufficient function by itself according to the present invention does not require an external light source, and the operation is simplified.

  A second simple option is to perform edge detection on the luminance image. However, an abrupt change in the image value does not necessarily mean an object boundary, and vice versa. For example, a complex scene can generate a large number of false luminance edges, while in a simple image few luminance edges are detected. Therefore, the image enhancement method by examining only the change of the image value often fails.

  Conventional NPR techniques that operate on a single image are based on images with very high image quality and good contrast so that luminance edge detection and color splitting are reliable.

  The approach according to the present invention may look like active lighting techniques such as traditional stereo, photometric stereo and Helmholtz stereo. However, depth discontinuities create problems with conventional stereo techniques. Stereo techniques often fail due to semi-shielding that disrupts the matching process.

  Photometric stereo estimates the geometry and albedo over a scene simultaneously. The main limitations of classical photometric stereo are that the light source must be far away from the camera COP and that the location of the light source must be known accurately. This requires a fixed lighting rig that is possible in a studio, industrial or laboratory environment, but not possible with a self-contained camera unit. Furthermore, the approach is based on detecting normals along a smooth surface and fails at depth discontinuities, shadowed areas and semi-obscured parts.

  The method according to the present invention is the opposite, and performs a binary decision for luminance variation at a certain scene depth discontinuity.

  In the case of Helmholtz stereo, shaded and semi-shielded areas correspond. The surface in the shadow in the left image cannot be seen in the right image and vice versa. The problem is that it is difficult to calculate a shaded area in a single image. The only reliable way to classify a pixel as being in the shadow area is to compare that pixel with yet another pixel when not in the shadow. This binary decision makes the technique of the present invention powerful. Also, in the present invention, most operations can be performed for each pixel without using any matching or correlation process. This allows the present invention to incorporate the entire method within the camera's microprocessor.

Description of Additional Embodiments The method of the present invention for generating a stylized image of a scene includes the following steps. First, a set of images is acquired from the scene under staggered light produced by a plurality of flash units positioned around the lens of the camera and light from the environment. These images are processed to automatically detect features such as depth edges, luminance edges, and texture edges to determine a qualitative depth relationship between the features. The output image is a combination of acquired images and enhances the detected features according to the detected features and qualitative relationships to provide a stylized image.

  In the present invention, the term depth edge is used to refer to an edge having a C0 depth discontinuity. A depth edge corresponds to a smooth surface, a thin object, i.e. a piece of paper boundary, or an edge object that does not depend on the viewpoint, e.g. an inside or outside occlusion contour or silhouette for a cube.

  The restored depth edge is signed. In the local neighborhood, the side with the lower depth value, for example the foreground, is considered positive and the other side, for example the background, is considered negative.

  Examples of texture edges are reflectance changes or material discontinuities. A texture edge usually outlines a texture region, but may not form a closed contour.

Depth Edge As shown in FIGS. 5A and 5B, the method of the present invention for detecting depth edges is the basis of the present technique and allows other edges to be classified by the removal process. The method of the present invention is based on two observations relating to the epipolar geometry of the shadow.

In FIG. 5A, the scene 501 includes an object 502. Due to the lighting used, the object casts a shadow 503 in the image I _k 504 taken by the camera 505. In the present invention, all orientation depth edges cause shadows in at least one image to ensure that the same shadowed point is illuminated in some other image.

  FIG. 5B shows an image of the object 502 superimposed on the epipolar rays emanating from e1, e2, and e3, along with corresponding shadows 521-523.

Image of the point light source in P _k is located at pixel e _k in the camera image, called an optical epipole. The image of light rays emanating from P _k is epipolar rays emanating from e _k . If P _k is behind the center of the camera and away from the image plane, the epipolar ray folds back at infinity.

  First, note that the shadow of a depth edge pixel is constrained to lie along an epipolar ray that passes through that pixel. Second, shadows are observed only when the background pixel is on the opposite side of the epipole with respect to the depth edge along the epipolar ray. Thus, in general, when two optical epipoles are on opposite sides of the edge, shadows are observed at the depth edge of one image and not the other image.

Positions P ₁ , P ₂ ,. . . The basic process for n light sources positioned at P _n is as follows.
Acquire image I ₀ with ambient light.
Image _{I k} by a light source in the P _{k. 0} (k = 1,..., N) is acquired.
The image I _k = I _{k, 0} −I ₀ is obtained.
For all pixels x, I _max (x) = max _k (I _k (x)) (k = 1,... N).
For each image k,
A ratio image R _k is generated. Here, R _k (x) = Ik (x) / I _max (x).
For each image R _k
Each epipolar ray is traversed from the epipole ek.
Find a pixel y with a step edge that has a negative transition.
Pixel y is indicated as a depth edge.

  A plurality of, for example 2-8, light sources arranged around the camera can be used to detect all orientation depth edges with a sufficient depth difference. In each image, if the dot product of the tangent and epipolar ray at the depth edge is non-zero, a step edge having a negative transition from the illuminated part to the shaded part is detected. If the depth edge is oriented along the epipolar ray, the step edge cannot be detected.

Note that the image I _k has the surrounding components removed, ie I _k = I _{k, 0} −I ₀ . Here, I ₀ is an image obtained with only ambient light and without turning on any of the n light sources. The original image is a maximum composite image I _max , and this maximum composite image I _max is an approximation of an image having a light source to the COP of the camera, and generally has shadows from any of the n light sources. Absent. This approximation is generally accurate when n light sources have the same size and the baseline is sufficiently smaller than the depth of the scene being imaged.

Consider an image of a 3D point X given in the camera coordinate system and imaged at pixel x. When point X is illuminated by a light source at P _k , the luminance I _k (x) is given by the following equation assuming uniform diffuse reflection (Lambertian).

Otherwise, I _k (x) is zero. The scalar μ _k is the magnitude of the light intensity, and ρ (x) is the reflectance at the point X. The value (^) L _k (x) is the normalized light vector (^) L _k (x) = P _k -X, N (x) is the surface normal, and they are all in the camera coordinate system Is in. Note that (^) L _k indicates that there is ^ on L _k .

Thus, if point X appears to be P _k , the ratio is

For diffuse objects with an albedo ρ (x) other than 0, it is clear that Rk (x) is independent of the albedo ρ (x) and is a function of local geometry only. Furthermore, if the light source-camera baseline | Pk | is small compared to the distance to that point, ie | X | << | P _k |, this ratio is approximately μ _k / max _i (μ _i ), Which is constant for a set of omnidirectional light sources.

The ratio value in (R _k = I _k / I _max ) is close to 1.0 in the area illuminated by the light source k and close to 0 in the shaded area. Generally, this value is non-zero due to mutual reflection. The intensity profile along the epipolar ray in the ratio image shows a sharp negative transition at the depth edge when traversing from a shadowless foreground to a shadowed background, and there is no shadow on the foreground from the shadowed background When crossing the region, it shows a sharp transition to positive.

  This reduces the problem of depth edge detection to the problem of luminance step edge detection. A single edge detector along the epipolar ray detects both positive and negative transitions, and in the present invention indicates negative transitions as depth edges. As mentioned above, noise and interreflection only affect the accuracy of the position, not the detection of the presence of depth edges. This is because the present invention detects separate transitions rather than continuous values.

In summary, there are basically three steps.
A ratio image in which the value of the shadowed region is close to 0 is generated.
Luminance edge detection is performed along the epipolar ray on each ratio image, and negative step edges are indicated as depth edges.
The edge maps from all n images are combined to obtain the final depth edge map.

Material Edge In addition to the depth edge, the present invention also considers lighting and material edges in the image. The illumination edge is the boundary between the illuminated area and the shaded area due to ambient light rather than the flash unit attached to the camera of the present invention. Individual images I _k have no illumination from the surroundings and no ambient lighting edges. In general, the material edge is independent of the illumination direction, and can be classified by the removal process. Material edge is obtained by subtracting the depth edges from the luminance edge of I _max. This edge classification scheme works well and involves a minimum number of parameters for adjustment. The parameters required in the present invention are only the ratio edge detection parameter and the Imax image luminance edge detection parameter for detecting the depth edge and the material edge, respectively.

Our technique for detecting depth edges is false negatives where negative transitions at depth edges cannot be detected in the ratio image R _k , or false positives where other conditions cause false transitions in R _k . Except for some conditions like, it is surprisingly reliable. Depth edges can also be missed due to isolated shadows, low background albedo, holes and depressions, or when depth edges are in shadowed areas. Due to specular reflection, self-occlusion, or depth edges that depend on the viewpoint, some pixels may be mislabeled as depth edge pixels.

The curved surface ratio R _k (x) is a very small and short depth edge of a curved object with edges that depend on the viewpoint. This is because the dot product ((^) Lk (x) · N (x)) is larger, and the dot product of the light source on the “opposite” side is larger. This is expressed as follows.

Therefore, R _k (x) decreases rapidly even when the pixel is not in a shaded area. However, this is not a big problem, only the slope at the negative transition of R _k is reduced.

Tradeoffs when selecting the baseline The longer the baseline distance between the camera and the flash, the better. The longer the baseline, the wider the detectable shadow cast in the image, but a shorter baseline is required to avoid separating the shadow from the associated depth edge.

The width of the adjacent shadow in the image is d = fB (z ₂ −z ₁ ) / (z ₁ z ₂ ). Here, f is the focal length, B is the base line, and z ₁ and z ₂ are the depth to the shadowed edge and the shadowed edge. Shadow isolation occurs when the width T of the object is smaller than (z ₂ −z ₁ ) B _f / z ₂ . Therefore, the shorter the base line B, the narrower the object having a smaller T, without causing shadow separation. Fortunately, the downsizing of the digital camera allows the present invention to select a small baseline while increasing the pixel resolution in proportion to f so that the product fB remains constant. This makes it possible to detect the depth of a narrow object.

Hierarchical Baseline As shown in FIG. 6, a hierarchical baseline can be used when the camera resolution is limited. Here, the set 610 of the first flash unit from COP106 is located at a first distance 611 acquires image F _S, larger second than the set 620 of the second flash unit is a first distance It disposed a distance 622 to acquire an image _{F L} with.

Therefore, in the present invention, it is possible to detect a small depth discontinuity using a longer baseline without causing shadow separation in a narrow object using a short baseline. In practice, two different baselines are sufficient. However, in this case, it must handle false edges due to shadow the separation of longer image F _L baseline. Shorter image F _S of the baseline sometimes miss small depth discontinuities.

There are four cases in which the depth edges are detected by combining information in a pair of images.
Image F _S has a narrow shadows undetectable width, image F _L has detectable shadows.
Image F _S has a narrow shadows detectable width, image F _L has a broad shadow width.
Image F _S has a detectable shadow image F _L has isolated shadow overlapping the shadow image F _S.
Image F _S has detectable shadows, the image F _L has a shadow isolated, the shadow image F _S and the image F _L do not overlap.

The method of the present invention takes a minimum composition of two images. In the first three cases this conveniently increases the effective width of adjacent shadows without any artifacts and can be handled by the basic process without modification. 4 th case, since the shadow-free region separating the two shadows in the minimum composite image appears to be incorrect shadow in F _L.

The solution of the present invention is as follows. A depth edge is obtained using the image F _S and the image F _L. Next, the epipolar ray is traversed. It appears depth edges in some point D ₁ of the within the image F _S, if not appear in the image F _L, crossing the epipolar rays in image F _L to the next depth edge is detected. If there is no depth edges corresponding to the image F _L, no depth edges corresponding to the image F _S, is indicated as erroneous edge of this edge.

Specular reflection Specular highlights that appear in pixels of one image but not in the other image can cause false transitions in the ratio image. Although there are methods for detecting specular reflection in a single image, it is difficult to reliably detect specular reflection in a texture region.

  The method of the invention is based on the fact that the specular spot shifts according to the deviation of the light source that causes specular reflection. In the present invention, three cases are considered as to how specular reflection spots at different light positions appear in each image. That is, for example, when a glossy spot remains sharp on a highly specular and moderately curved surface, when several glossy spots overlap, and, for example, a substantially specular front parallel ( fronto-parallel) where glossy spots completely overlap on the plane. In this last case, there is no false gradient in the ratio image.

  Although the specular reflection overlaps in the input image, the boundary around the specular reflection, that is, the luminance edge generally does not overlap. The main idea is to take advantage of the gradient variation at a given pixel (x, y) in n images. In the first two cases, if pixel (x, y) is a specular region, the gradient due to the specular boundary is high in only one or a few of the n images under separate lighting.

  The median value of the n gradient at that pixel removes the outlier. The method of the present invention is an adaptation of the unique image technique described by Weiss (see above). Here, the shadow of the outdoor scene is removed by noting that the shadow boundary is not stationary.

In the present invention, the image is reconstructed by using the median of the gradient of the input image as follows.
Luminance gradient G _k (x, y) = ∇I _k (x, y) is obtained.
Find the median G (x, y) = median _k (G _k (x, y)) of the gradient.
Construct an image I ′ that minimizes | ∇I′−G |.

In the present invention, before applying the integral, the image is filled into a square image that is closest to a power of 2, and then the resulting image is cropped to its original size. The present invention uses a similar gradient domain technique to simplify some rendering tasks as described below. The resulting inherent image brightness I (x, y) is used as a common feature for calculating the ratio image instead of the maximum composite image I _max (x, y). In this case, the ratio Ik (x, y) / I ′ (x, y) is larger than 1.0 in the specular reflection region. This is set to 1.0 so that negative transitions in the ratio image are not located in the specular reflection part.

Lack of Background So far, the present invention has assumed that the depth edges that cast shadows on the background are within a finite distance. However, this is not always the case. In this case, in the present invention, only the outermost depth edge, that is, the edge shared by the foreground and the distant background may be missed. This can be easily detected using foreground-background estimation techniques. The ratio of the image I ₀ / I _max is close to 1 in the background and close to 0 in the foreground.

Image Synthesis NPR based on 3D models is known, but not for photographs. In the absence of a complete 3D representation of the scene, the present invention develops a new rendering algorithm using the following 2D queues. The depth edge sign, foreground versus background at the edge, the relative depth difference based on shadow width, the color near the signed edge, and the smooth surface normal at the occlusion contour.

  The method of the present invention automates tasks for stylized rendering, such as image editing and rotoscoping, which originally require careful manual manipulation.

Rendering edges Connect edge edge pixels to a contour and then smooth the contour. Unlike the conventional method of selecting the next edge pixel based on the orientation similarity in the T-shaped joint, the present invention uses the information from the shadow to decompose the connected components.

In the negative transition along the epipolar ray in the signed depth edge ratio image R _k , the higher luminance side with respect to the edge is the foreground, and the lower luminance side corresponding to the shaded area is the background.

  As shown in FIG. 7, the qualitative depth relationship can be used to clearly indicate the foreground-background separation at each depth edge of the anatomical site. In the present invention, the mat emulates the over-under style used by the artist. The foreground side of the depth edge is white and the background side is black. Both sides of the depth edge are rendered by displacing the contour along the surface normal 701.

Light Direction As shown in FIG. 8, in the present invention, an arbitrary light direction 801 is conveyed in the output image 800 of the anatomical site by changing the depth edge widths 811 to 812 according to the direction of the depth edge with respect to the lighting direction. be able to. Since the direction of the edge in 3D is substantially the same as the direction of its projection in the image plane, the width can be proportional to the dot product of the normal of the image space and the desired light direction.

Color Change As shown in FIG. 9, the present invention can indicate the color of the original object by rendering the depth edges with the selected color. From the signed depth edge, select a foreground color along the normal at a fixed pixel distance without intersecting another depth or luminance edge. The foreground colored depth edge may be superimposed on the divided input image, for example, an input image acquired with ambient light.

Color assignment Since there is no 3D model of the scene, rendering a non-edge pixel requires a different method of processing the captured 2D image.

Normal Interpolation As shown in FIGS. 10A and 10B for smooth objects, depth edges correspond to occluded contours where the surface normal is perpendicular to the viewing direction. Therefore, the normal at the depth edge is located in the image plane, and the normal at the other pixels can be predicted. In the present invention, this sparse interpolation problem can be solved by solving a 2D Poisson differential equation. The method of the present invention is described in detail by Johnston “Lumo: illumination for cel animation” (Proceedings of the second international symposium on Non-photorealistic animation and rendering, ACM Press, PP. 45-ff, 2002). Automate manual methods for generating undermats. For the present invention, signed depth edges allow normal interpolation, while maintaining normal discontinuities at the depth edges to enhance the surface 1001 in a direction away from the lighting direction. .

Image attenuation Professional photographers sometimes use local light to enhance contrast at shape boundaries. In the present invention, this is imitated by an image attenuation map as follows. The depth edge exists in white on a black background. The present invention performs convolution using a filter that is the gradient of the edge enhancement filter. The filter of the present invention is obtained by subtracting an impulse function from a Gaussian function. When 2D integration is performed on the convolved image, a sharp transition is obtained at the depth edge.

A depiction of change Some stationary figures show activity, for example changing oil in the car, by brightening certain parts of the foreground motion. However, if the colors are similar and there is no texture, foreground detection using a luminance-based method, for example, the detection of hand movements in front of other skin colored parts, as shown in FIG. Have difficulty.

  The present invention captures two sets of separate multi-flash images, a handless “background” set 1101 and a foreground set 1102 with a hand in front of the face to capture the reference scene and the changed scene. . In the present invention, the inside of the region contributing to the new depth edge is emphasized without clearly detecting the foreground. The present invention determines a gradient field, in which unit-size gradients are assigned to pixels indicated as depth edges in a changed scene rather than a reference scene. The orientation matches the image space normal to the depth edge. The gradient at other pixels is zero. The image reconstructed from 2D integration is a pseudo depth map-least square error solution by solving the Poisson equation. The present invention thresholds this map at 1.0 to obtain the foreground mask 1103 and uses this foreground mask 1103 to brighten the corresponding portion of the output image 1104.

  Note that the width of the shadow along the epipolar ray is proportional to the ratio of the depth values on both sides of the edge. Thus, instead of a unit magnitude gradient, the present invention assigns a value proportional to the logarithm of the shadow width along the epipolar ray to obtain a higher quality pseudo depth map.

  Unfortunately, the inventors have discovered that positive transitions along the rays are not strong due to the use of non-point light sources and interreflection. In principle, the estimated shadow width can be used for adjustable abstraction to remove edges with small depth differences.

Abstraction One way to reduce visual clutter in an image and clarify the shape of an object is to simplify changes in details, such as texture and lighting, that are not related to scene shape boundaries or depth edges. Conventional image segmentation techniques impose severe restrictions on closed contours and assign a fixed color to each region. In addition, the division may miss a depth edge and the foreground and background may be integrated into a single color object.

  The method of the present invention reconstructs an image from gradients without using gradients in texture pixels. No determination is made as to which luminance value should be used to fill the hole, as is required in conventional pixel-based systems, and no feathering and blurring is required.

Mask image γ (x, y) = a, where
(X, y) is a texture edge pixel,
d (x, y) is a featureless pixel,
(X, y) is a depth edge pixel set to 1.

The factor d (x, y) is the ratio of the texture pixel distance field to the depth edge pixel distance field. This parameter a controls the degree of abstraction and the texture is suppressed for a = 0. The procedure is as follows.
Generate mask image γ (x, y) Find luminance gradient ∇I (x, y) Change masked gradient G (x, y) = ∇I (x, y) ^{γ (x, y)} Reconstruct the image I ′ to minimize | I′−G | so that the colors in the output image I ′ (x, y) substantially match the colors of the input image I (x, y) Normalize to

  Following image reconstruction, the Poisson equation is solved by a multigrid approach, such as in the specular attenuation technique described above.

Dynamic Scenes So far, the method of the present invention obtains geometric features by acquiring multiple images of a still scene. The present invention examines the lack of simultaneity of capturing scenes with moving objects or moving cameras. Again, there are many works for estimating motion in an image sequence, and sensible techniques use motion compensation techniques to correct introduced artifacts using results from static algorithms. Is to apply.

  However, finding optical flow and motion boundaries is a difficult problem, especially in areas without texture. As with stationary, the present invention bypasses the difficult problem of finding rich pixel-by-pixel motion representations and focusing directly on finding discontinuities in motion, ie, depth edges.

  In this case, the camera is a video camera that acquires a sequence of image sets (frames) using one flash for each frame. Next, the sequence of images is processed in time order as described above.

  The present invention finds a depth edge in each frame and connects the found edges in each frame to a complete depth edge.

The lack of simultaneity of depth edge acquisition in motion means that the original image, eg, the maximum composite I _max , is unreliable and the features are not aligned. This prevents the shadow area from being reliably found. The present invention makes a simplistic assumption that the motion of the scene is moderate, especially in the image space, the motion of the object features is smaller than the width of the object. A high speed camera can reduce the amount of motion between frames, but cannot assume the lack of simultaneity. With the novel illumination method of the present invention, it has been found that this can be solved relatively easily by observing a moving shadowed area. In order to simplify this explanation, the present invention considers only the flash units on the left and right sides of the lens to determine the vertical depth edge.

The present invention identifies three types of intensity gradients in such sequences.
Gradient with adjacent shadow edges due to depth edges,
Gradient due to texture edge of stationary part and gradient due to texture edge of moving part.

  Using the basic method for generating the image ratio described above, the present invention can remove a small amount of stationary texture edges.

  The present invention takes advantage of the fact that due to the left-to-right switch in lighting, the shadow edges disappear in alternating frames and the moving texture edges do not go away.

Given frame m-1 and frame m + 1, the process for determining the depth edge in frame m is as follows. Here, a = frame I _m−1 , b = frame I _m , and c = frame I _{m + 1} .
A shadow for storing the image I _ab is obtained. Here, I _ab = min (a, b, c).
An image _Iac without a shadow is obtained. Here, I _ac = min (a, c).
Determining the ratio image _{R m.} Here, R _m . _Iab / _Iac .
traverses along the epipolar ray from e _m, instructs a negative transition.

The ratio R _m is close to 0 when the pixels in frame m are between corresponding pixels in frames m−1 and m + 1. That is, the luminance of both these frames is high. In the case of a depth edge, whether it is stationary or moving, adjacent shadows appearing in frame m cause this condition, but not for texture step edges that are stationary or moving.

Edges and colors Depth edges in a given frame m are imperfect because they only span a limited orientation. In dynamic scenes, the combination of the depth edges of four consecutive frames may not be aligned enough to identify discontinuous contours. The present invention compares signed depth edges corresponding to the same flash unit, ie, m and m + n, and interpolates the displacement of the intermediate frame. In order to assign colors, the present invention takes the maximum of three consecutive frames, for example the previous frame, the current frame and the next frame.

  Note that the depth edge extraction method can also be used for spectra other than visible light that produce “shadows” in infrared, sonar, x-ray and radar imaging. Specifically, the camera can be equipped with an invisible infrared light source so that the resulting flash is not in the way.

  Indeed, a frequency-division multiplexing scheme can be used to generate a single shot multi-flash system. The flash fires four different colors at different wavelengths simultaneously. A Bayer mosaic pattern of filters on the camera imager decodes four separate wavelengths.

Depth Edge Applications Detecting depth discontinuities is fundamental to understanding images and can be used in many applications. The method is mainly based on the outermost silhouette of the object, but we have found that a full depth edge map can help with visual hull, segmentation, depth layer and aspect graph decomposition problems. I think you can. Aerial photography techniques can improve building detection by looking at multiple time difference images of shadows cast from a known direction of sunlight. In addition, effects such as depth of field during post-processing, composite apertures using camera arrays and screen matting for virtual sets using arbitrary backgrounds require high quality signed depth edges. .

  Stereo correspondence based on edges or areas can be improved by matching signed depth edges, constraining dynamic programming within depth edges, and modifying the correlation filter to handle partial occlusion. it can. Edge classification can also provide a reliability map to help classify colors and textures in low contrast images.

  Although the invention has been described by way of illustrating the preferred embodiments, it is to be understood that various other applications and modifications can be made within the spirit and scope of the invention. Therefore, the purpose of the appended claims is to cover all such modifications or changes as fall within the true spirit and scope of the invention.

1 is a block diagram of a non-photorealistic camera according to the present invention. FIG. 1B is a block diagram of the camera of FIG. 1A including a flash unit of another configuration. It is a block diagram of the camera of FIG. 1A provided with the flash unit of another structure. 3 is a flow diagram of a method for generating a stylized image according to the present invention. 2 is a flow diagram of a method for detecting silhouette edges. It is the image of the vase which laid the flower which emphasized the silhouette edge by this invention. It is a block diagram of the epipolar ray corresponding to the cast shadow. It is a block diagram of the epipolar ray corresponding to the cast shadow. FIG. 2 is a block diagram of a camera having flash units positioned at various distances around the camera lens. Output image using over-under style rendering. It is an output image having various edge widths. It is an output image having a de-emphasized texture. It is an output image of a smooth object. It is an output image of a smooth object. It is an output image of a changing object in the scene.

Claims (44)

A method for generating a stylized image of a scene including an object, comprising:
Each of the n input images is illuminated by one light source of a set of n light sources attached to the body of the camera at a different position from the projection center of the camera lens, and from the scene using the camera. obtaining a set of n input images;
A feature including a depth edge, a luminance edge, and a texture edge in the set of the n input images is detected to obtain a qualitative depth relationship between the depth edge, the luminance edge, and the texture edge. And
Combining the set of n input images into a single output image to produce a stylized image of the scene containing the object comprising enhancing the detected features according to the qualitative relationship Method. The method of claim 1, wherein the depth edge corresponds to a discontinuity in depth within the n input images. The depth edge is signed, and foreground pixels in the n input images have a positive value and background pixels have a negative value,
The texture edges outline the reflectance and material discontinuities in the scene,
The method of claim 1, wherein the luminance edge outlines a region in the scene having a substantially constant reflectivity. The method of claim 1, wherein the depth edge corresponds to a shadow region in the scene relative to image space coordinates. The method of claim 4, wherein the object and the shadow region in the scene are continuous in the image space. For each of the n input images, generating a ratio image from the pixel brightness in the shadow region substantially close to 0;
Performing luminance edge detection on each ratio image along an epipolar ray indicating a negative step edge at the depth edge to generate a depth edge map of each ratio image;
The method of claim 4, further comprising combining the edge maps from all n input images to obtain a final depth edge map. The method of claim 1, wherein the set of n light sources is attached to the camera body at different distances from a projection center of the camera lens to provide a plurality of baselines. The n input images include specular highlights;
The method of claim 1, further comprising removing the specular highlight from the n input images. Determining n gradient variations at each pixel in the n input images;
The method of claim 8, further comprising: determining a median value of the n gradients and removing outliers. The luminance gradient at each pixel (x, y) in the specific input image I _k is G _k (x, y) = ∇I _k (x, y),
The median value of the gradient is G (x, y) = median _k (G _k (x, y))
The method of claim 9, wherein the reconstructed image I ′ minimizes | ∇I′−G |. The method of claim 10, wherein the ratio I _k (x, y) / I ′ (x, y) is set to 1.0 to remove the specular highlight. Connecting the depth edge pixels to a contour;
The method of claim 1, further comprising smoothing the contour. The foreground side of the depth edge has the positive value, the background side of the depth edge has the negative value,
Rendering the foreground side as white;
The method of claim 3, further comprising rendering the background side as black. The method of claim 13, further comprising rendering the foreground side and the background side of the depth edge by displacing the contour along a surface normal. The method of claim 14, further comprising changing a width of the depth edge to convey an arbitrary light direction in the output image. The method of claim 15, wherein the width is proportional to a normal of the image space and a dot product in the arbitrary light direction. The method of claim 1, further comprising rendering the depth edge with a selected color. One input image is illuminated by ambient light,
Splitting the one input image illuminated by light from the surroundings;
18. The method of claim 17, further comprising overlaying colored depth edges on the segmented input image. Rendering the depth edge in white on a black background in the output image;
Convolving the edge with a filter that is a gradient of an edge enhancement filter to obtain a convolved image;
The method of claim 1, further comprising integrating the convolved image to obtain a sharp transition at the depth edge. The method of claim 19, wherein the filter is a Gaussian function minus an impulse function. The scene includes a background object and a foreground object;
Obtaining a first n + 1 set of input images from the scene having only the background object;
Obtaining a second n + 1 set of input images from the scene having the background object and the foreground object;
The method of claim 1, further comprising applying a mask to a portion of the second n + 1 set of input images corresponding to the foreground object to enhance the appearance of the foreground object in the output image. The method of claim 21, wherein a pixel luminance value in the mask is set to one. The texture edge defines a texture region in the n + 1 images,
The method of claim 1, further comprising masking the texture region according to a mask image to reduce clutter in the output image. The pixels in the mask image are γ (x, y) = a,
Setting the luminance value of the pixel (x, y) at the depth edge to 1,
Setting the luminance value of the texture edge pixel (x, y) to 0;
24. The method of claim 23, further comprising: setting a luminance value of a luminance edge of pixel d (x, y) to a ratio of a texture edge pixel distance field to a depth edge pixel distance field. Generating an initial mask image γ (x, y);
Obtaining a luminance gradient ∇I (x, y);
Changing the masked gradient G (x, y) = ∇I (x, y) ^{γ (x, y)} ;
Reconstructing the output image I ′ to minimize | I′−G |;
25. The method of claim 24, further comprising normalizing colors in the output image I '(x, y) to substantially match colors in the input image I (x, y). The camera is a video camera and the object moves in the scene;
Obtaining a sequence of a set of n + 1 input images from the scene using the video camera;
Detecting the feature in a sequence of the set of n input images;
The method of claim 1, further comprising: combining the sequence of the n + 1 input image sets into an output image sequence to enhance the detected features according to the qualitative relationship. A luminance gradient includes a gradient due to the depth edge, a gradient due to the texture edge of the stationary portion, and a gradient due to the texture edge of the moving portion;
Identifying a luminance gradient in the sequence of sets of images;
27. The method of claim 26, further comprising removing a texture edge having the stationary portion. 28. The method of claim 27, further comprising interpolating displacements of depth edges that move within the sequence of output images. 27. The method of claim 26, further comprising assigning pixel colors in a particular image of the output image sequence according to a maximum of three consecutive images. 30. The method of claim 29, wherein the three consecutive images are a specific image, a previous image, and a next image in the sequence. The method of claim 1, wherein the light source emits visible light. The method of claim 1, wherein the light source emits infrared light. The method of claim 1, wherein the light source emits a sonar signal. The method of claim 1, wherein the light source emits x-rays. The method of claim 1, wherein the light source emits a radar signal. The light source emits light of a plurality of wavelengths simultaneously,
The method of claim 1, further comprising decoding a plurality of frequencies using a plurality of filters at a sensor in the camera. The method of claim 36, wherein the filter has a Bayer mosaic pattern. The method of claim 1, wherein the edges include all edges in the n input images. 39. The method of claim 38, further comprising determining a visual hull from the edge. 40. The method of claim 38, further comprising segmenting the output image according to the edges. 40. The method of claim 38, further comprising decomposing a depth layer in the output image according to the edge. 40. The method of claim 38, further comprising decomposing an aspect graph of the output image according to the edges. The method of claim 1, wherein the input image is acquired by aerial photography. The method of claim 1, wherein the light source is a flash unit.

Images