JP2004133919A - Device and method for generating pseudo three-dimensional image, and program and recording medium therefor - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a pseudo three-dimensional image generating device which is small in size and can easily be used, and to provide a pseudo three-dimensional image generating method. <P>SOLUTION: Simulative depth information of a photographing object is generated from a plurality of images photographed with the existence/absence of illumination for the photographing object or by changing the intensity of the illumination. This pseudo three-dimensional image generating device is provided with an image recording means 13 for recording a plurality of images and an operating means 14 for calculating discrete depth data of the photographing object on the basis of a comparison operation of pixel values of corresponding pixels in the plurality of images and calculating a simulative depth value on the basis of the discrete depth data, and generates a pseudo-three-dimensional image. According to the present invention, a small and simple three-dimensional image image generating device can be generated. <P>COPYRIGHT: (C)2004,JPO

Description

The present invention relates to an image generating apparatus and method for generating a pseudo three-dimensional image from an image, and a program and a recording medium therefor.

Conventionally, various methods have been used to input a three-dimensional image of an object. The three-dimensional image input method can be roughly divided into two methods, a passive method and an active method. For example, a stereo method is a typical example of the passive method, and a light section method is a typical example of the active method.

The stereo method is a method in which two photographing devices are arranged at a certain interval, and the position of the photographing target is obtained from the difference between the projection positions of the same photographing target in both images, that is, the difference between corresponding points, based on the principle of triangulation. . The light cutting method is a method in which a slit light projecting device and a photographing device are arranged at a certain interval, and a position of the photographing object is obtained from the position where the slit light projected on the photographing object is projected in the image of the photographing device by the principle of triangulation. It is.

As a technique for irradiating a subject with illumination light to obtain a depth value, the subject is illuminated with illumination whose brightness changes at least depending on a position on the subject (for example, illumination whose brightness increases from left to right). Then, there is a method of detecting reflected light for each light intensity and obtaining a depth value using the principle of triangulation. However, in this technique, since the principle of triangulation is used for calculating the depth value, it is necessary to provide a certain distance between the camera and the lighting device, and it is difficult to reduce the size of the device (for example, see Patent Document 1). 1).

Further, as another technique, a so-called Time that irradiates an object with intensity-modulated illumination light whose light intensity changes with time and calculates the distance to the object from the intensity of the reflected light based on the light propagation time. There is a method of obtaining a depth value using the principle of of flight. However, this technique requires a special illumination device for projecting the intensity-modulated illumination light and a camera having a high-speed shutter for capturing changes in the intensity-modulated illumination light, which is suitable for reducing the cost of the device. (For example, see Patent Document 2).

Further, as a technique for obtaining a pseudo depth value of a subject, a gradation value of a transmission image used for creating a divided image is compared with a divided image for each discrete depth value of the subject obtained using an X-ray CT apparatus or the like. There is a method in which a depth value is pseudo-calculated so as to be smoothly connected to an adjacent divided image. However, this technique is based on the premise that an X-ray CT apparatus or the like has previously obtained a transmission image divided for each discrete depth value of the subject, and it is difficult to apply the technique to a special purpose such as medical treatment. (For example, see Patent Document 3).
JP 2000-329524 A JP-A-2000-121339 JP-A-2-079179

In recent years, it has become common to mount cameras on mobile terminals. The images acquired from the camera are only two-dimensional images. However, if three-dimensional images can be input, the range of use of the images is widened and the field of use of the images is increased. However, since the conventional method uses the principle of triangulation in both the passive method and the active method, a certain long distance is required between devices such as a photographing device and a slit light projecting device, so that miniaturization is difficult and not suitable for portable terminals. In general, the passive method has a short measurement time but low accuracy, so the quality of an image generated from the input depth value is low. The active method has high accuracy but a long measurement time, and is suitable for industrial use. . Furthermore, the other methods described above are also intended for limited applications, such as the need for special equipment.

The present invention has been made in view of the above problems, and an object of the present invention is to provide a pseudo three-dimensional image generation device and a three-dimensional image generation method which are small and can be easily used. In the present invention, when the true distance is unknown, a distance to be applied instead of the true distance is defined as pseudo.

The pseudo three-dimensional image generation device according to the present invention is a pseudo three-dimensional image generation device that generates a pseudo three-dimensional image of the shooting target from a plurality of images obtained by shooting the shooting target in a plurality of lighting conditions, Image recording means for recording a plurality of images, image calculation means for calculating discrete depth data of the corresponding pixels based on a comparison operation of pixel values of corresponding pixels in the plurality of images, and the discrete depth Depth value calculating means for calculating a pseudo depth value of the corresponding pixel based on the data.

３ The three-dimensional image generation device according to the present invention inputs a plurality of images under different lighting conditions to a photographing target. The depth value is calculated by comparing the plurality of images. As a comparison method, for example, “illuminated” and “non-illuminated” are set as the illumination conditions, and when the difference between the images under each condition is calculated, the greater the difference in luminance, the closer to the illumination. The depth value calculated at this time is discrete, and a pseudo three-dimensional image having a discrete depth can be generated. Also, by calculating using an appropriate continuous function that smoothly connects all depths so that the depth values do not become discrete, it is possible to generate an image that prevents the writing phenomenon and gives the viewer a high quality stereoscopic effect. It is possible.

{Other objects and features of the present invention will become apparent from the following detailed description with reference to the drawings.

よ う As described above, according to the present invention, the following effects are produced.

According to one embodiment of the present invention, the distance between the photographing device and the lighting means may be adjacent to each other, so that the size can be reduced as compared with a conventional device.

According to another aspect of the present invention, the resolution of an image used to calculate depth can be reduced to reduce the number of pixels on which calculations are performed, so that high-speed processing with a small calculation load can be performed.

According to another aspect of the present invention, high-speed processing with a small calculation load can be performed by using the result of the simple binarization processing as a pseudo distance.

According to another aspect of the present invention, a continuous pseudo depth value can be easily obtained without performing complicated measurement processing.

According to another aspect of the present invention, it is possible to reduce a sense of incongruity of an observer due to a sudden change in a depth value calculated in a pseudo manner.

Hereinafter, the best mode for carrying out the present invention will be described in detail.
[First Embodiment]
FIG. 1 is a block diagram showing a pseudo three-dimensional image generation device according to the first embodiment of the present invention. The pseudo three-dimensional image generating apparatus 10 shown in FIG. 1 is connected to a photographing apparatus 11 provided with or connected to an illuminating unit 12 for illuminating a photographing target. The pseudo three-dimensional image generation device 10 includes an image recording unit 13 that captures and holds image data of a photographing target, and calculates a pseudo depth value of the photographing target by comparing a plurality of images of the photographing target with different levels of illumination. And control means 15 for controlling each of the above means.

FIG. 2 is a flowchart for explaining a pseudo three-dimensional image generation method executed by the pseudo three-dimensional image generation device 10 shown in FIG. Hereinafter, the operation of the pseudo three-dimensional image generation device 10 will be described with reference to the flowchart of the pseudo three-dimensional image generation method illustrated in FIG.

生成 When generating a pseudo three-dimensional image of a certain photographing target, first, an illumination condition and the number k of photographed images are input (step S21). The lighting conditions and the number of shots may be determined in advance. Next, the object to be photographed is illuminated using the illuminating means 12 under the j-th illumination condition, and an image photographed by the photographing apparatus 11 is input (step S22). The illumination condition is determined by the intensity of the illumination, and illumination is performed in accordance with an illumination signal from the imaging device 11 (for example, a pulse synchronized with an imaging shutter). The j-th input image signal is stored as image data in the frame memory j of the image recording means 13 (step S23). The processing from step S22 to step S23 is repeated until the number of shots reaches k (steps S24 and S25).

(4) When the input of the image is completed, the photographed image data is transferred from the frame memories 131 to 133 to the image calculation means 141 of the calculation means 14 according to a command from the control means 15 (step S26). Based on the plurality of images having different illumination conditions, the image calculation unit 141 performs operations such as subtraction and division on the pixel values of the corresponding pixels, comparison with a predetermined threshold, and the like to generate discrete depth data (step S27). . The discrete depth data will be described later in detail.

{Circle around (4)} The image calculation means 141 also performs preprocessing for synthesizing color information between a plurality of images and calculating a depth value. Examples of the color information synthesizing process include a method of converting each image so as to maximize the contrast and extracting an image having the highest contrast, and a method of synthesizing them with appropriate weighting. Further, pre-processing for depth value calculation includes edge detection, object detection by color, and dilation / contraction processing after binarization. A combination of these processes may be selected by an instruction from the control unit 15.

Using the discrete depth data generated by the image calculation means 141, the depth value calculation means 142 calculates a pseudo depth value for each pixel (step S28).

In order to perform the above-described depth value calculation processing, it is only necessary to illuminate the imaging target, and the distance between the imaging device and the illumination unit may be adjacent. Therefore, the device can be made smaller than the conventional device, and can be mounted on a portable terminal.

FIG. 3 is a flowchart showing a process (step S27) of generating the discrete depth data of FIG. Note that the discrete depth data qualitatively represents depth, and represents only the depth order. Further, the depth value quantitatively represents the depth. In this embodiment, for ease of explanation, the depth value is set to two values before and after. It is assumed that two lighting conditions, “with lighting” and “without lighting”, are set, and the number of shots is two. At this time, the pixel value of the i-th pixel of the image data when “illuminated” is Ai, the pixel value of the i-th pixel of the image data when “no illumination” is Bi, and qualitatively indicates a discrete depth. Binary data is represented by Ci.

When calculating the depth value, first, the threshold value n and the image size s are determined (step S31). The threshold value n is a parameter for dividing each pixel of the image into two types, one near and far from the photographing apparatus 11, and can be arbitrarily determined. The method for determining the threshold value n will be described later. The image size s is a value obtained by multiplying the number of horizontal pixels x and the number of vertical pixels y of the input image data. Next, the difference between the pixel values Ai and Bi is obtained. If the difference is equal to or larger than the threshold value n, Ci is set to 1, otherwise 0 is set (steps S32 to S34). The processing of steps S32 to S34 is repeated for all the pixels of the image data (steps S35 to S36).

画素 A pixel having a value of binary discrete depth data Ci of 1 is determined to be near the imaging device, and the other pixels are determined to be far away. As described above, the binary data Ci is data that qualitatively represents the discrete depth of each pixel in comparison with the threshold value n. If the result of the simple binarization process is a pseudo depth value, a pseudo three-dimensional image can be generated by high-speed processing with a small calculation load.

多 Multi-level depth values can be obtained by a combination of binarization processing based on a plurality of images with different illumination light amounts or binarization processing using a plurality of different thresholds. In this case, the calculation may be performed between any other image and the reference image, using any one of the plurality of images as the reference image.

FIG. 4 is a flowchart showing a modification of the process of generating discrete depth data (step S27 in FIG. 2) described with reference to FIG. The discrete depth data generation process illustrated in the flowchart of FIG. 4 is different from the discrete depth data generation process illustrated in FIG. 3 in that the binary data Ci of the pixel i is obtained instead of the pixel data Ai and Bi. The difference is that the average values A′i and B′i of the several pixels in the vicinity are used. By using the average value, even when data of a plurality of images to be compared is shifted, the influence of the shift can be reduced. If the range of pixels for which the average value is calculated is appropriately determined, it is possible to support a moving image. Furthermore, since the S / N ratio is improved by taking the average value, the result of the comparison operation is also stabilized.
[Second embodiment]
A second embodiment of the present invention will be described. The pseudo three-dimensional image generation device and the pseudo three-dimensional image generation method of the second embodiment are the same as those of the first embodiment except for a discrete depth data generation process. Therefore, only the differences will be described.

FIG. 5 is a flowchart showing a process of generating discrete depth data according to the second embodiment of the present invention. Also in the present embodiment, the depth value is a depth of two values before and after. Two lighting conditions, “lighting on” and “lighting off,” are set, and the number of shots is two. At this time, the pixel value of the i-th pixel of the image data when “illuminated” is Ai, the pixel value of the i-th pixel of the image data when “no illumination” is Bi, and the discrete depth is qualitatively indicated. Binary data is represented by Ci.

When calculating the depth value, first, the threshold value m and the image size s are determined (step S51). The threshold value m can be arbitrarily determined by a parameter for dividing an image into binary values. The image size s is a value obtained by multiplying the number of horizontal pixels x and the number of vertical pixels y of the input image data. Next, if the rate of change of the pixel value Ai with respect to the image data Bi is equal to or greater than m, Ci is set to 1; otherwise, Ci is set to 0 (steps S52 to S54). The processing of steps S52 to S54 is repeated for all pixels of the image data (steps S55 to S56).

画素 A pixel whose binary data Ci has a value of 1 is determined to be near the imaging device, and the other pixels are determined to be far away.

Here, as shown in FIG. 5, in the process of generating discrete depth data in the second embodiment, division of pixel values Ai and Bi is used. This is because, when there is a plurality of imaging targets having different reflectances, if the division of the pixel values Ai and Bi is used as in the second embodiment, the difference in the reflectance does not affect the discrete depth data Ci.

と If the subtraction of the pixel values Ai and Bi is used as in the first embodiment, the difference in the reflectance may affect the discrete depth data Ci. Since the reflectance differs depending on the imaging target, an imaging target having a high reflectance is determined to be closer than the actual distance, and an imaging target having a low reflectance is determined to be farther than the actual distance. There are cases. However, in applications requiring very low range resolution, such as perspective separation, this effect is often negligible.

FIG. 6 is a diagram for explaining the effect of a difference in reflectance on discrete depth data Ci when there are a plurality of imaging targets with different reflectances. In FIG. 6, the imaging targets 621 and 622 are equidistant from the imaging device 61 and the illumination unit 63. Each of reflectivity r _a and r _b, different. When the photographing target 621 and the photographing target 622 are photographed by the photographing device 61, the images are displayed on the display device 64. When the amount of light reaching from the illumination means 63 to the imaging target 621 and 622 is X, the imaging object 621 is an object 641 of the luminance r _a × X, as the object 642 to be imaged 622 is luminance r _b × X, on the display device 64 Is displayed. Similarly, if the amount of light reaching from the illumination means 63 to the imaging target 621 and 622 is Y, imaging object 621 and 622, as the object luminance r _a × Y and r _b × Y respectively, are displayed on the display device 64 You. In this case, the luminance difference of the imaging target 621 is r _a × (X−Y), and the luminance difference of the imaging target 622 is r _b × (X−Y). Although the photographing targets 621 and 622 are at the same distance from the photographing device 61, the difference in luminance between them is different.

When the discrete depth data Ci is obtained by division as in the second embodiment, the luminance ratio of the imaging target 621 is X / Y and the luminance ratio of the imaging target 622 is X / Y. The difference in reflectance is canceled, and the luminance ratios of the imaging targets at the same distance become equal.

(4) Since the material does not reflect much illumination, the difference in luminance itself is small in a portion where the luminance is low in the image data, but correct discrete depth data can be obtained by using the luminance ratio.

Multilevel depth values can be obtained by binarization processing based on a plurality of images with different illumination light amounts or binarization processing using a plurality of different threshold values, as in the first embodiment. It is.

FIG. 7 is a flowchart showing a modification of the process of generating discrete depth data (step S27 in FIG. 2) according to the second embodiment described with reference to FIG. The discrete depth data generation process shown in the flowchart of FIG. 7 is different from the discrete depth data generation process shown in FIG. 5 in that the binary data Ci of the pixel i is obtained instead of the pixel data Ai and Bi. The difference is that the average values A′i and B′i of the several pixels in the vicinity are used. By using the average value, even when data of a plurality of images to be compared is shifted, the influence of the shift can be reduced. If the range of pixels for which the average value is calculated is appropriately determined, it is possible to support a moving image. Furthermore, since the S / N ratio is improved by taking the average value, the result of the comparison operation is also stabilized.

Here, with reference to FIG. 13, a method of determining the threshold used in the first embodiment and the second embodiment will be described. FIG. 13 is a graph showing the relationship between the distance of the shooting target and the illuminance. Curves 1310 and 1320 represent changes in the illuminance of the object to be photographed with distance due to illumination conditions having different intensities. The graph shows that the amount of light reaching the object from the illumination is inversely proportional to the square of the distance.

In the case of the illumination light amount 1310, if the light amount reaching the shooting target at the distance L is Y, the light amount reaching the shooting target at the distance 2L is Y / 4. Assuming that the threshold value n is as shown in the graph, a pixel whose illuminance is equal to or larger than the threshold value n is closer than the distance 1311. Similarly, when the photographing target has the illumination light amount 1320, it is closer than the distance 1321. Therefore, when the discrete depth data is obtained from the difference between the pixel values as in the first embodiment, the threshold value for the distance can be determined according to the illumination light amount. At this time, an appropriate value must be assumed as the reflectance of the imaging target. At this time, the effect of the amount of incident light from the external light on the object to be photographed (corresponding to the amount of light to the object without illumination in the case of the present embodiment) is offset by the difference, so it is not necessary to consider it. Absent. By adjusting the assumed reflectance based on the result of generating the discrete depth data, a more appropriate threshold can be determined.

On the other hand, when the discrete depth data is obtained by dividing the pixel value as in the second embodiment, the threshold for the distance is determined by the amount of incident light from the illumination of the object to be photographed and the amount of incident light from the external light (for example, indoor fluorescent lamp). It can be determined from the amount of light.

Regardless of the amount of external light incident on the object to be photographed, if the amount is constant, the minimum value of the illuminance in FIG. 13 is increased and only a fixed amount of offset is added. Therefore, a relative distance can be obtained by division. For example, since the amount of light from the illumination attenuates with the square of the distance, the ratio of the change obtained by dividing one pixel to the ratio of the change obtained by dividing another pixel is 1/4. The distance of another pixel is twice as long as the distance from the camera.

At this time, a threshold value for the distance can be determined by assuming a constant appropriate value as the amount of incident light from the external light to the imaging target. In this case, there is no need to consider the influence of the reflectance of the object to be photographed because the influence of the reflectance is offset by the division.

By adjusting the assumed amount of incident light based on the result of generating discrete depth data, a more appropriate threshold can be determined.
[Third Embodiment]
A third embodiment of the present invention will be described. The pseudo three-dimensional image generation device and the pseudo three-dimensional image generation method of the third embodiment are the same as those of the first embodiment, except for a discrete depth data generation process. Therefore, only the differences will be described.

FIG. 8 is a flowchart showing a process of generating discrete depth data according to the third embodiment of the present invention. In the present embodiment, the depth value is two depths before and after. It is assumed that two lighting conditions, “with lighting” and “without lighting”, are set, and the number of shots is two. At this time, the pixel value of the i-th pixel of the image data when “illuminated” is Ai, the i-th pixel value of the image data when “no illumination” is Bi, and the binary data indicating the discrete depth is Ci. Expressed by

When calculating the depth value, first, the threshold value n for the difference operation, the threshold value m for the division operation, the division value p, and the image size s are determined (step S81 in FIG. 8). The threshold value n for the difference operation, the threshold value m for the division operation, and the division value p can be arbitrarily determined by parameters for dividing an image into binary values. The image size s is a value obtained by multiplying the number of horizontal pixels x and the number of vertical pixels y of the input image data. Next, if the pixel value Ai is equal to or larger than the division value p, the discrete depth data generation process based on the threshold n described in the first embodiment, otherwise, the discrete depth data based on the threshold m described in the second embodiment The generation process is performed (steps S82 to S86 in FIG. 8).

画素 A pixel whose binary data Ci is 1 is determined to be close to the imaging device, and the other pixels are determined to be far away.

画素 For pixels having a luminance value equal to or greater than the division value p, high-speed processing with a small calculation load can be performed by using the result of the binarization processing based on the difference as a pseudo distance. In addition, since a portion having low luminance in the image data is a material that does not reflect much illumination, a pixel having a luminance value equal to or less than the division value p is properly separated from a near portion by using a result of binarization processing by division. it can.

Multilevel depth values can be obtained by binarization processing based on a plurality of images with different illumination light amounts or binarization processing using a plurality of different threshold values, as in the first embodiment. It is.

FIG. 9 is a flowchart showing a modification of the process of generating discrete depth data (step S27 in FIG. 2) according to the third embodiment described with reference to FIG. The discrete depth data generation process illustrated in the flowchart of FIG. 9 is different from the discrete depth data generation process illustrated in FIG. 8 in that the binary data Ci of the pixel i is obtained instead of the pixel data Ai and Bi. The difference is that the average values A′i and B′i of the several pixels in the vicinity are used. By using the average value, even when data of a plurality of images to be compared is shifted, the influence of the shift can be reduced. If the range of pixels for which the average value is calculated is appropriately determined, it is possible to support a moving image. Furthermore, since the S / N ratio is improved by taking the average value, the result of the comparison operation is also stabilized.

Further, as a modified example of the third embodiment, a plurality of division values may be used. For example, when two divided values of p1> p2 are determined, the difference operation at the difference threshold n1 when the pixel value is larger than p1, the difference operation at the difference threshold n2 when the pixel value is larger than p2, the pixel value Is smaller than or equal to p2, a process such as a difference calculation at a difference threshold value n3 is performed. As described above, the predetermined threshold value may be changed according to the pixel value. Further, the threshold value may be determined by performing an arithmetic operation as needed for each local region as in adaptive binarization.
[Fourth embodiment]
As described above, in the first to third embodiments, discrete depth data is generated (Step S27 in FIG. 2), and the data is used as it is as the depth of the pseudo three-dimensional image (Step in FIG. 2). S28). Specifically, a predetermined distance as a near view and a predetermined distance as a distant view are applied. In the fourth embodiment of the present invention, the generated discrete depth data is further processed in step 28 of FIG. 2 to generate a more realistic pseudo three-dimensional image.

FIG. 10 is a diagram for explaining a pseudo three-dimensional image generation method according to the fourth embodiment of the present invention. Hereinafter, the processing algorithm will be described with reference to FIG.

分割 The original image 101 is divided based on the discrete depth data 102 generated in any of the first to third embodiments. Since the discrete depth data Ci is binary data, the discrete depth data image in which Ci is represented as a two-dimensional image is represented by black and white binary values. The original image 101 is divided into two images based on the discrete depth data image. The divided images are called a front image 103 and a rear image 104, respectively. The depth data image 102 may be created by using the method described in the first to third embodiments.

Next, objects are extracted by focusing on the front image 103 and performing grouping. Since the black portion on the front image 103 is a region existing on another divided image (that is, the rear image 104), if a partial image (1031, 1032) obtained by connecting adjacent pixels on the remaining region is obtained, These are so-called objects corresponding to the respective imaging targets. In this case, the grouping is performed by connecting the pixels. However, a method may be adopted in which the area is divided based on the color information and each area is set as a divided partial image, that is, an object. Objects (1031, 1032) belonging to the near-side divided image (front image 103) such that the depth value of these two objects (1031, 1032) is continuously connected to the depth value of the adjacent divided image (back image 104). Recalculate the depth value of. Specifically, the depth value is replaced by an arbitrary function.

In FIG. 10, as an example of an arbitrary function, an image 105 in which the depth value of an object is replaced by a quadratic function that changes in the horizontal direction, a quadratic function in which the depth value of the object is fixed at the center and the peripheral portion changes in the horizontal direction And an image 107 in which the depth value of the object is replaced by a quadratic function that changes the depth value of the object along the line radially extending from the center of gravity while keeping the center constant.

To generate an image 105 in which the depth value of an object is replaced by a quadratic function that changes in the horizontal direction, a scan is performed for each horizontal line of the image to give a pseudo depth value. For example, when the line 1051 is scanned, the left end and the right end of the object on the line 1051 are detected, and the depth value connected to the rear image 104 is defined as min, and the depth value farthest from the rear image 104 on the front image 103 is defined as max. The object is replaced by a quadratic function 1052 that passes through min at the left end and right end and through max at the center. By giving a pseudo depth value by such simple processing, a real three-dimensional image can be created without performing measurement.

To generate an image 106 in which the depth value of the object is replaced with a quadratic function that changes the depth value of the object at the center and the peripheral portion in the horizontal direction, a scan is performed for each horizontal line of the image to give a pseudo depth value. For example, when the line 1061 is scanned, the left end and the right end of the object on the line 1061 are detected, the depth value connected to the rear image 104 is defined as min, and the depth value farthest from the rear image 104 on the front image 103 is defined as max. The function 1062 is a combination of a quadratic function that passes max at a position 15% away from the center to the left (right end) at the left end (right end) of the object and a constant max at 30% of the center section. . By giving a pseudo depth value in which the vicinity of the center of the object pops out with more emphasis, the observer can feel as if the object is closer.

To generate an image 107 in which the depth value of an object is fixed at the center and the periphery is replaced by a quadratic function that changes along a line extending radially from the center of gravity, the center of gravity of each object is calculated and the boundary between the center of gravity and the object is calculated. Gives a pseudo depth value for each line passing through. For example, when calculating the depth value of the line 1071, the intersection of the boundary of the object and the line 1071 is detected, and the depth value connected to the rear image 104 is min, and the depth value of the front image 103 that is farthest from the rear image 104 is max. , Min at the intersection, and a function 1072 obtained in the same manner as 1062 passing through max at a length of 30% centered on the center of gravity of the length between the two intersections.

By giving a pseudo depth value in which the vicinity of the center of the object pops out with more emphasis, the observer can feel as if the object is closer. Further, since the depth value is calculated along a line extending radially from the center of gravity of the object, the object can be evenly rounded regardless of the inclination of the boundary line, and a more natural three-dimensional effect can be given to the observer.
[Fifth Embodiment]
FIG. 11 is a diagram illustrating a pseudo three-dimensional image generation method according to the fifth embodiment of the present invention. The original depth value 1110 of each pixel of a certain divided image calculated by the pseudo three-dimensional image generation method according to the fourth embodiment is subjected to a smoothing process for averaging a 7 × 7 range centered on the pixel. Then, a smoothed depth value 1130 was obtained.

FIG. 12 shows the coefficients of the smoothing filter used in the smoothing processing of the present embodiment. Here, as the smoothing filter, the pixel values of all the pixels within the range of the filter size (7 × 7) centering on the pixel to be processed are multiplied by the reciprocal of the filter size (1/49) as a coefficient and added. A simple moving average filter was applied. The filter size can be arbitrarily changed, and another filter having a smoothing effect may be applied. Examples of other filters having a smoothing effect include a weighted moving average filter that weights each matrix in the filter as needed, and a Gaussian filter that weights according to a normal distribution.

As is clear from comparison between the enlarged view 1112 of the jaw portion 1111 of the original depth value 1110 and the enlarged view 1132 of the jaw portion 1131 of the smoothed depth value 1130, the smoothing process is performed to reduce the depth value. Rapid changes are mitigated.

場合 When a sudden change in the depth value occurs inside the object by the pseudo three-dimensional image generation methods according to the first to fourth embodiments, the observer may feel uncomfortable. Further, when a sudden change in the depth value occurs at the boundary of the object, the image may be separated from another adjacent divided image of the depth value, giving a sense of incompatibility.

However, by smoothing the depth value once calculated in this way, it is possible to prevent a viewer from feeling uncomfortable when a sudden change in the depth value exists in an object. Also, when a sharp change in the depth value occurs at the boundary of the object, the connection with the depth value of another adjacent divided image becomes loose, and the observer may feel uncomfortable in the connection with the other divided image. Can be prevented.

Note that it is apparent that a program for causing a computer to execute the procedure shown in the flowchart can be realized, and that the program can be recorded on a computer-readable recording medium.

In the first to fifth embodiments, a shooting target is extracted as an object based on discrete depth data, and a pseudo depth value is given to the extracted object. In another embodiment of the present invention, the extraction of the object is performed by the conventionally used edge detection and color, and the object extracted by the conventional object extraction method is given a pseudo depth value based on the discrete depth data. May be. Here, for edge detection, for example, there is an extraction method using a primary differential filter or a secondary differential filter. The object extraction by color includes, for example, an extraction method using a color space and an extraction method using saturation.

オ ブ ジ ェ ク ト Alternatively, an object extracted based on discrete depth data may be corrected by an object extracted by edge detection or color. An example of the process will be described with reference to FIG. 1410 is, for example, an outline of an object extracted by edge detection, and 1420 is a region extracted based on discrete depth data. The discrete depth data image may deviate from the actual one due to the influence of the reflectance of the imaging target or the movement of the imaging target. At this time, the correction is performed using the contour line 1410. Specifically, the contour of the region 1420 extracted based on the discrete depth data is corrected to the closest contour 1410. For example, the point 1431 is corrected to a point 1441, and the point 1432 is corrected to a point 1442.

In addition to the correction of the contour line, it is also possible to correct a region having a small area in a discrete depth data image or to fill a hole having a small area opened in the image. For example, when a part of a photographing target (for example, a part of a person's eyes or a button) does not appear as an object, the part can be added or conversely deleted when it appears. The area of the region to be deleted or filled may be specified based on a separate rule of thumb. As a correction similar to the above, when a gap between objects is small, the gaps may be filled to connect the objects. The width of the region for filling the gap may be specified based on a separate rule of thumb, similarly to the area for filling the hole.

In addition, in order to further reduce the calculation load, the resolution of the image may be reduced before generating the discrete depth data. By reducing the resolution, the amount of calculation can be reduced. Also, if the actually extracted object is displaced due to the influence of the reflectance or the movement of the shooting target, the resolution of the depth value becomes coarser than the texture, so the depth value This has the effect of blurring the outline and showing less deviation.

When the resolution is further reduced, if the resolution is reduced by taking an average value of peripheral pixels before the resolution is reduced, the effect of improving the S / N ratio is obtained as in the case of applying the smoothing filter. Can be obtained.

Here, signal S indicates image information captured with no noise and infinite gradation. The noise N indicates a noise amount including all noise factors such as optical or electrical noise and quantization noise caused by finite length quantization.

画素 Note that only the pixels required to reduce the resolution need to take the average value. For example, the number of pixels can be reduced to １ ／ by calculating the average value by one pixel in both the vertical and horizontal directions. The generation of discrete depth data using the division described in the second embodiment is sensitive to image noise. Also in this case, the S / N ratio can be improved by lowering the resolution.

Note that it is not necessary to perform all of the above processing, and the processing may be performed as needed based on the instruction of the control unit 15.

As described above, the invention made by the inventor has been specifically described based on the embodiment. However, the present invention is not limited to the embodiment, and can be variously modified without departing from the gist of the invention. Of course.

According to the present invention, it is possible to realize a smaller and easier pseudo three-dimensional image generation device as compared with a three-dimensional image generation device according to a conventional method.

FIG. 1 is a diagram illustrating a pseudo three-dimensional image generation device according to a first embodiment of the present invention. FIG. 4 is a diagram illustrating a flowchart of a process according to the first embodiment of the present invention. 5 is a flowchart illustrating a process of generating discrete depth data according to the first embodiment of the present invention. 4 is a flowchart when an average value is used for comparison in the example shown in FIG. 3. 9 is a flowchart illustrating a process of generating discrete depth data according to a second embodiment of the present invention. FIG. 5 is a diagram for explaining the influence of the reflectance of the imaging target on discrete depth data. 6 is a flowchart in a case where an average value is used for comparison in the example shown in FIG. 5. 11 is a flowchart illustrating a process of generating discrete depth data according to a third embodiment of the present invention. 9 is a flowchart when an average value is used for comparison in the example shown in FIG. 8. It is a figure for explaining a 4th embodiment of the present invention. FIG. 4 is a diagram for explaining processing by a smoothing filter according to the embodiment of the present invention. FIG. 3 is a diagram illustrating an example of a smoothing filter according to the embodiment of the present invention. 5 is a graph illustrating a concept of a threshold for generating discrete depth data according to an embodiment of the present invention. FIG. 5 is a diagram for describing correction of an object according to the embodiment of the present invention.

Explanation of reference numerals

DESCRIPTION OF SYMBOLS 11 ... Photographing device, 111 ... Solid-state image sensor, 112 ... A / D conversion means, 12 ... Illumination means, 13 ... Image recording means, 131 ... Frame memory 1, 132 ... Frame memory 2, 133 ... Frame memory k, 14 ... Calculation means, 141 ... image calculation means, 142 ... depth value calculation means, 15 ... control means, 61 ... photographing device, 621 ... photographing object, 622 ... photographing object, 63 ... lighting means, 64 ... display device, 641 ... image, 642: image, 101: original image, 102: depth value, 103: front image, 1031: partial image on front image, 1032: partial image on front image, 104: rear image, 105: depth value in horizontal direction 1051... Line, 1052... Function, 106... Depth value is a quadratic function that changes the peripheral portion in the horizontal direction while keeping the center constant. Image, 1061... Line, 1062... Function, 107... Image in which the center of the depth value is fixed and the peripheral portion is replaced by a quadratic function that changes along a line extending radially from the center of gravity of the partial image, 1071. , 1072... Function, 1110... Original depth value, 1111... Noticed part, 1112... Enlarged view, 1130... Smoothed depth value, 1313... Noticed part, 1132. 1311: distance that satisfies threshold n, 1321: distance that satisfies threshold n, 1410: contour by discrete depth data, 1420: contour by another object extraction method, 1431, 1441 ... corresponding points on the contour, 1432, 1442 ... corresponding point on contour

Claims (18)

A pseudo three-dimensional image generation device that generates a pseudo three-dimensional image of the shooting target from a plurality of images obtained by shooting the shooting target in a plurality of lighting states,
Image recording means for recording the plurality of images,
Image calculation means for calculating discrete depth data of the corresponding pixel based on a comparison operation of pixel values of corresponding pixels in the plurality of images,
A depth value calculating unit configured to calculate a pseudo depth value of the corresponding pixel based on the discrete depth data. The pseudo three-dimensional image generation device according to claim 1,
The image calculation means reduces the resolution of the plurality of images, and calculates discrete depth data of the corresponding pixel based on a comparison operation of pixel values of corresponding pixels in the plurality of reduced-resolution images. A pseudo three-dimensional image generation device, characterized in that: The pseudo three-dimensional image generation device according to claim 2,
A pseudo three-dimensional image generation apparatus, wherein the image calculation means smoothes the plurality of images and lowers resolution. The pseudo three-dimensional image generation device according to any one of claims 1 to 3,
The image calculation means assigns discrete depth data for each pixel by comparing a difference or a ratio of pixel values of corresponding pixels in the plurality of images with a threshold value for each pixel. Image generation device. The pseudo three-dimensional image generation device according to claim 1, wherein:
The pseudo three-dimensional image generation device according to claim 1, wherein the image calculation unit extracts the object to be photographed based on the calculated discrete depth data of the corresponding pixel. The pseudo three-dimensional image generation device according to claim 5,
The image calculating means assigns discrete depth data for each pixel by comparing a difference or a ratio of pixel values of corresponding pixels in the plurality of images with a predetermined threshold value corresponding to the pixel value, and A pseudo three-dimensional image generation apparatus, wherein, in any one of the images, pixels having the same discrete depth data and adjacent pixels are extracted as the object. The pseudo three-dimensional image generation device according to claim 5 or 6,
The pseudo-three-dimensional image generation device according to claim 1, wherein the depth value calculation unit applies a depth function to the object based on the discrete depth data. The pseudo three-dimensional image generation device according to claim 7,
The pseudo-three-dimensional image generation apparatus according to claim 3, wherein the depth value calculation unit performs a smoothing process of a depth value with respect to a position of a pixel in the entire image, a portion adjacent to the object, or in the object. A pseudo three-dimensional image generation method for generating a pseudo three-dimensional image of the shooting target from a plurality of images obtained by shooting the shooting target in a plurality of lighting conditions,
An image recording step of recording the plurality of images,
Based on a comparison operation of pixel values of corresponding pixels in the plurality of images, a discrete depth data calculation step of calculating discrete depth data of the corresponding pixels,
A depth value calculating step of calculating a pseudo depth value of the corresponding pixel based on the calculated discrete depth data. The pseudo three-dimensional image generation method according to claim 9,
The discrete depth data calculation step,
Reducing the resolution of the plurality of images;
Calculating the discrete depth data of the corresponding pixels based on a comparison operation of the pixel values of the corresponding pixels in the plurality of images with reduced resolutions. The pseudo three-dimensional image generation method according to claim 10, wherein
A pseudo three-dimensional image generation method, characterized in that in the step of reducing the resolution of the plurality of images, the plurality of images are smoothed and the resolution is reduced. The pseudo three-dimensional image generation method according to claim 9, wherein
In the discrete depth data calculation step, discrete depth data is assigned to each pixel by comparing a difference or a ratio of pixel values of corresponding pixels in the plurality of images with a threshold value for each pixel. A pseudo three-dimensional image generation method. The pseudo three-dimensional image generation method according to claim 9, wherein:
A pseudo three-dimensional image generation method, wherein in the discrete depth data calculation step, the object to be photographed is extracted based on the calculated discrete depth data of the corresponding pixel. The pseudo three-dimensional image generation method according to claim 13,
In the discrete depth data calculation step, by comparing the difference or ratio of the pixel values of the corresponding pixels in the plurality of images with a threshold value for each pixel, discrete depth data is assigned to each pixel, In any one of the first to third aspects, a pixel having the same discrete depth data and adjacent pixels is extracted as the object. The pseudo three-dimensional image generation method according to claim 13 or 14,
The pseudo three-dimensional image generating method according to claim 1, further comprising a depth function fitting step of fitting a depth function to the object based on the discrete depth data in the depth value calculating step. The pseudo three-dimensional image generation method according to claim 15,
Performing a process of smoothing a depth value with respect to a position of a pixel in the entire image, a portion adjacent to the object, or any of the inside of the object. A pseudo three-dimensional image generation program, characterized in that it is a program for causing a computer to execute the process in the pseudo three-dimensional image generation method according to any one of claims 9 to 16. A program for causing a computer to execute a process in the pseudo three-dimensional image generation method according to any one of claims 9 to 16,
A recording medium recording a pseudo three-dimensional image generation program, wherein the program is recorded on a recording medium readable by the computer.