JP5824896B2 - Image processing apparatus and method, and program - Google Patents

Description

  The present technology relates to an image processing apparatus, method, and program, and in particular, a stereo image composed of a left-eye image and a right-eye image capable of realizing a three-dimensional stereoscopic view from an input image composed of a two-dimensional image by a simple method. The present invention relates to an image processing apparatus and method capable of realizing image generation, and a program.

  There is generally a process (2D to 3D conversion process) for converting a 2D image into a 3D image.

  In the 2D to 3D conversion process, the depth value of the subject in the two-dimensional image is obtained in units of pixels, and each pixel of the two-dimensional image is shifted to the left and right by the number of pixels set based on the depth value. Processing for creating a three-dimensional image composed of an eye image and a left eye image is known.

  As a method of generating a three-dimensional image by shifting the above pixels, a method of shifting a target pixel in the left-right direction by a parallax amount generated from a depth image is generally used (Non-Patent Document 1).

2Dto3D content conversion technology (2010 Annual Conference of the Institute of Image Information and Television Engineers): Toshiba: Yuji Mita

  In principle, it has been confirmed that this pixel shift method operates correctly.

  However, a region where there is no pixel to be projected (hidden region) is generated in principle. Further, in order to fill this hidden surface area, for example, processing such as in-painting is required at a later stage, which increases the processing cost.

  The present technology has been made in view of such a situation, and in particular, a stereo image composed of a right-eye image and a left-eye image that can be three-dimensionally viewed from an input image composed of a two-dimensional image by a simple method. Can be generated appropriately.

An image processing apparatus according to an aspect of the present technology includes a depth image generation unit that generates a depth image from an input image signal including a two-dimensional image signal, a blur processing unit that blurs the depth image, and the blur processing unit. A shift amount calculation unit that calculates a shift amount that is set based on a depth value of each pixel for each pixel of the depth image that has been subjected to the blurring process, and a calculation that is performed by the shift amount calculation unit for each pixel in the input image signal A shift for generating a right-eye image or a left-eye image in a three-dimensional image signal by projecting a pixel at a position shifted to the right or left by a shift amount corresponding to each pixel of the depth image a Department seen including, the blurring processing unit includes an edge detection unit for detecting an edge from the depth image, a detected edge by the edge detecting section, a region in the vicinity thereof LPF to be subjected to LPF processing, the edge of the depth image is detected by the edge detection unit, and the edge detected by the LPF and the area in the vicinity thereof are determined according to the amount of change in the depth value of the corresponding depth image. Blur is processed by LPF processing with high strength .

According to an image processing method of one aspect of the present invention, a depth image is generated from an input image signal including a two-dimensional image signal in a depth image generation unit that generates a depth image from an input image signal including a two-dimensional image signal. A depth image generation step, a blur processing step for blurring the depth image in a blur processing unit that blurs the depth image, and each pixel of the depth image subjected to the blur processing by the blur processing unit. In the shift amount calculation unit that calculates the shift amount set based on the depth value, for each pixel of the depth image blurred by the blur processing unit, the shift amount set based on the depth value of each pixel A shift amount calculating step to calculate, and for each pixel in the input image signal, the shift amount calculation unit calculates the shift amount In the shift unit that generates the image for the right eye or the image for the left eye in the three-dimensional image signal by projecting the pixel at the position shifted to the right or left by the shift amount corresponding to each pixel of the row image, For each pixel in the input image signal, by projecting a pixel at a position shifted rightward or leftward by the shift amount corresponding to each pixel of the depth image calculated by the processing of the shift amount calculation step, look including a shift step of generating an image for the right eye or the left eye image in the three-dimensional image signal, processing of the blurring processing step, an edge detection step of detecting an edge from the depth image, the edge detection step An edge detected by the processing, and an LPF processing step of applying an LPF to a region in the vicinity thereof, and the edge of the depth image is the edge Is detected by the processing of the outgoing step, the edge detected by the LPF, the region in the vicinity thereof, to the blurring process by which the LPF processing in intensity in accordance with the change amount of the depth value of the corresponding depth image.

A program according to one aspect of the present invention includes a depth image generation unit that generates a depth image from an input image signal including a two-dimensional image signal, a blur processing unit that blurs the depth image, and a blur process performed by the blur processing unit. A shift amount calculation unit that calculates a shift amount that is set based on a depth value of each pixel for each pixel of the depth image that has been calculated, and a shift amount calculation unit that calculates each pixel in the input image signal A shift unit that generates a right-eye image or a left-eye image in a three-dimensional image signal by projecting a pixel at a position shifted rightward or leftward by a shift amount corresponding to each pixel of the depth image; only including, the blurring processing unit includes an edge detection unit for detecting an edge from the depth image, and edge detected by the edge detecting section, a region in the vicinity thereof LPF LPF to be processed, the edge of the depth image is detected by the edge detection unit, the edge detected by the LPF and the area in the vicinity thereof according to the amount of change in the depth value of the corresponding depth image Depth image generation step of generating a depth image from an input image signal consisting of a two-dimensional image signal in the depth image generation unit in the depth image generation unit in the computer that controls the image processing apparatus that performs the blur process by performing LPF processing with intensity, and the blur A blur processing step for blurring the depth image in the processing unit, and each pixel of the depth image subjected to the blur processing by the processing of the blur processing step in the shift amount calculation unit, based on the depth value of each pixel. A shift amount calculating step for calculating a set shift amount; and for each pixel in the input image signal in the shift unit. The right eye in the three-dimensional image signal is projected by projecting a pixel at a position shifted rightward or leftward by a shift amount corresponding to each pixel of the depth image calculated by the processing of the shift amount calculation step. A process including a shift unit that generates an image for use or an image for the left eye, and the process of the blur processing step is detected by an edge detection step of detecting an edge from the depth image and a process of the edge detection step. And an LPF processing step for applying an LPF to an area in the vicinity thereof, the edge of the depth image is detected by the processing in the edge detection step, the edge detected by the LPF, and an area in the vicinity thereof, Blur processing is performed by performing LPF processing with strength corresponding to the amount of change in the depth value of the corresponding depth image .

In one aspect of the present technology, a depth image is generated from an input image signal including a two-dimensional image signal, the depth image is subjected to a blurring process, and the depth value of each pixel is determined for each pixel of the blurred depth image. A shift amount set based on the image is calculated, and for each pixel in the input image signal, a pixel at a position shifted rightward or leftward by the shift amount corresponding to each pixel of the calculated depth image is projected. As a result, an image for the right eye or an image for the left eye in the three-dimensional image signal is generated , an edge is detected from the depth image, and the detected edge and an area in the vicinity thereof are LPF processed by LPF, The edge of the depth image is detected, and the edge detected by the LPF and the area in the vicinity thereof are subjected to LPF processing with a strength corresponding to the amount of change in the depth value of the corresponding depth image. In Ru is blurring.

  The image processing apparatus of the present technology may be an independent apparatus or a block that performs image processing.

  According to the present technology, a left-eye image and a right-eye image that can be three-dimensionally viewed from a two-dimensional input image can be generated by a simple method.

It is a block diagram which shows the structural example of 1st Embodiment of the image conversion apparatus to which the image processing apparatus which is this technique is applied. 3 is a flowchart illustrating image conversion processing by the image conversion apparatus in FIG. 1. It is a figure explaining the example of an input image. It is a figure explaining the example of the depth image with respect to the input image of FIG. It is a figure explaining the image when LPF is applied to the depth image of FIG. It is a flowchart explaining the shift method of a pixel. It is a figure explaining the image for left eyes, and the image for right eyes. It is a figure explaining the example which a hidden surface area | region generate | occur | produces. It is a figure explaining the example which a hidden surface area | region generate | occur | produces. It is a figure explaining the example which a flying area generate | occur | produces. It is a figure explaining the example which a flying area generate | occur | produces. It is a figure explaining the example which a hidden surface area | region and a flying field area | region do not generate | occur | produce. It is a figure explaining conditions when an enclave area does not occur. It is a figure which shows the structural example of 2nd Embodiment of the image conversion apparatus made to expand after reducing a depth image. It is a flowchart explaining the image conversion process by the image conversion apparatus of FIG. It is a figure which shows the structural example of 3rd Embodiment of the image converter which performed the edge detection process to the depth image, and applied the LPF to the detected edge and its vicinity. It is a flowchart explaining the image conversion process by the image conversion apparatus of FIG. And FIG. 11 is a diagram illustrating a configuration example of a general-purpose personal computer.

Hereinafter, modes for carrying out the present technology (hereinafter referred to as embodiments) will be described. The description will be given in the following order.
1. 1. First embodiment Second embodiment (an example where a reduction unit and an enlargement unit are used instead of LPF)
3. Third embodiment (an example in which an edge detection unit is provided and LPF processing is performed only on the edge and its neighboring area)

<1. First Embodiment>
[Configuration example of image conversion apparatus]
FIG. 1 illustrates a configuration example of an image conversion apparatus that generates a left-eye image and a right-eye image that enable three-dimensional stereoscopic viewing from an input image that includes a two-dimensional image.

  The image conversion apparatus 11 includes a depth image generation unit 21, an LPF 22, a shift amount calculation unit 23, a shift amount buffer 24, a right eye image shift unit 25-1, a left eye image shift unit 25-2, and a right eye image buffer. 26-1 and a left-eye image buffer 26-2. The depth image generation unit 21 obtains the distance to the subject in the input image, that is, the depth information from the input image composed of the two-dimensional image, in units of pixels. Then, the depth image generation unit 21 generates a depth image by setting a pixel value corresponding to the depth information in units of pixels, and supplies the depth image to the LPF 22. An LPF (Low Pass Filter) 22 performs an LPF process on the depth image, smoothes the entire image, adds a blurring process, and supplies the result to the shift amount correction unit 23. The shift amount calculation unit 23 calculates a shift amount corresponding to the depth value (parallax) for each pixel of the depth image subjected to the blurring process, and causes the shift amount buffer 24 to buffer the pixel image in association with the pixel position.

  The right-eye image shift unit 25-1 and the left-eye image shift unit 25-2 each read out the shift amount buffered in the shift amount buffer 24 for each pixel of the input image composed of a two-dimensional image. The right-eye image shift unit 25-1 and the left-eye image shift unit 25-2 respectively shift each pixel of the input image leftward or rightward by the number of pixels corresponding to the read shift amount. Projective transformation is performed with the pixels present at the specified position. The right-eye image shift unit 25-1 and the left-eye image shift unit 25-2 perform projective transformation on each pixel with pixels that are present on the left and right by the shift amount, whereby the right-eye image and the left-eye image shift unit 25-1, respectively. An ophthalmic image is generated. The right-eye image shift unit 25-1 and the left-eye image shift unit 25-2 convert the generated right-eye image and left-eye image into the right-eye image buffer 26-1 and the left-eye image 26, respectively. -2 to buffer. The right-eye image buffer 26-1 and the left-eye image 26-2 output the right-eye image and the left-eye image, respectively, after conversion by projection is completed for all pixels.

[Image Conversion Processing by Image Conversion Device in FIG. 1]
Next, image conversion processing by the image conversion apparatus 11 in FIG. 1 will be described with reference to the flowchart in FIG.

  In step S <b> 1, when an input image composed of a two-dimensional image is supplied, the depth image generation unit 21 obtains a distance from the imaging position of the subject in the image, that is, a depth value for each pixel in the input image. Then, the depth image generation unit 21 generates a depth image by setting a pixel value corresponding to the depth value for each pixel, and supplies the depth image to the LPF 22. As the depth value, for example, when an input image is captured, a value obtained in advance using the principle of triangulation by the TOF (Time of Flight) method or stereo imaging may be used. Alternatively, a value measured after the input image is captured may be used.

  For example, when the input image is an image as shown in the upper part of FIG. 3, the depth image is an image as shown in FIG. That is, the image shown in the upper part of FIG. 3 is an image in which an object having a height as shown in the side view of the lower part of FIG. In other words, as shown in the lower part of FIG. 3, when viewed from the side, an image obtained by capturing an object such as a stepped platform from the upper surface becomes the upper part of FIG. In the depth image of FIG. 4, the higher the depth value is, the higher the brightness, that is, the pixel value close to white, and the lower the depth value, the lower the brightness, that is, the pixel value close to black. Is set. As shown in the depth image of FIG. 4, the area corresponding to the upward-sloping hatched portion in FIG. 3 is set to white, indicating that it is the frontmost side. In addition, the region corresponding to the downward slanting line portion in FIG. 3 is set to gray, which indicates that it is the front side next to the right sloping line portion in FIG. Furthermore, the area corresponding to the lattice-shaped part in FIG. 3 is set to black, indicating that it is the farthest area.

  In step S <b> 2, the LPF 22 smoothes the supplied depth image, blurs the entire image, and supplies it to the shift amount calculation unit 23. That is, in the case of a depth image as shown in FIG. 4, the LPF 22 performs blurring processing to generate an image as shown in FIG. That is, in the image of FIG. 5, the entire image of FIG. 4 is subjected to the blurring process, but the blurring is particularly noticeable near the boundary where the pixel value changes greatly.

  In step S <b> 3, the shift amount calculation unit 23 sets an unprocessed pixel among the pixels of the depth image subjected to the blurring process as a target pixel.

  In step S4, the shift amount calculation unit 23 calculates the shift amount based on the depth value of the target pixel of the depth image subjected to the blurring process.

  By the way, the shift amount has conventionally been as shown in the right part of FIG. That is, for the pixel at the position Xp, when the right eye R observes the target position r in the real space D shown in the uppermost stage in the drawing as the line of sight RB and the left eye EL observes it. Is set as the line of sight LB. Then, when the right eye ER observes the target position r in the real space D, the pixel RI observed on the image I through the line of sight RB, and the right eye image when observing the right eye image R through the line of sight RB. Shift to a pixel position on the image R. Similarly, when the left eye EL observes the target position r in the real space D, the right eye when the pixel LI observed on the image I through the line of sight RL and the left eye image L through the line of sight LB are observed. The pixel position on the image R is shifted. By shifting in this way, a stereo image composed of a right-eye image and a left-eye image has been generated.

  On the other hand, the shift amount calculated by the shift amount calculation unit 23 is the shift amount d shown on the left side of FIG. That is, the right-eye image shift unit 25-1 and the left-eye image shift unit 25-2, which will be described later, perform the image I with respect to the target position r in the real space D, as shown in the left part of FIG. For the pixel at the position Xp, the right-eye image R and the left-eye image L are obtained by shifting a pixel shifted left and right by a predetermined shift amount d corresponding to the distance to the real space D, that is, the depth value. The right eye image and the left eye image are generated by projective transformation as pixels on the position Xp at. This shift amount d is a value set linearly corresponding to the depth value of each pixel. The method for setting the shift amount d will be described later in detail.

  In step S4, the shift amount calculation unit 23 causes the shift amount buffer 24 to buffer the calculated shift amount of the target pixel in association with the pixel position information.

  In step S5, the shift amount calculation unit 23 determines whether or not there is an unprocessed pixel in the blurred depth image. If there is an unprocessed pixel, the process returns to step S3. . That is, the processes in steps S3 to S5 are repeated until there is no unprocessed pixel and the shift amount is calculated for all pixels. If it is determined in step S5 that there is no unprocessed pixel and the shift amount is calculated for all pixels, the process proceeds to step S6.

  In step S6, the right-eye image shift unit 25-1 and the left-eye image shift unit 25-2 each set an unprocessed pixel among the pixels of the input image as a target pixel.

  In step S <b> 7, the right-eye image shift unit 25-1 and the left-eye image shift unit 25-2 read the shift amount of the target pixel from the shift amount buffer 24.

  In step S8, the right-eye image shift unit 25-1 and the left-eye image shift unit 25-2 right out of the pixels around the target pixel in the input image based on the read shift amounts of the target pixel, respectively. A pixel existing at a position corresponding to the shift amount only in the direction or the left direction is subjected to projective transformation as a target pixel, and is associated with the pixel position, and the image buffer for right eye 26-1 and the image buffer for left eye 26- 2 to buffer. That is, as described with reference to the left part of FIG. 6, the right-eye image shift unit 25-1 and the left-eye image shift unit 25-2 start from the target pixel in the input image based on the shift amount. Projection conversion (shift) is performed on a pixel existing at a position shifted by the shift amount in the right direction or the left direction to the target pixel.

  In step S9, the right-eye image shift unit 25-1 and the left-eye image shift unit 25-2 each determine whether or not there is an unprocessed pixel on the input image, and there is an unprocessed pixel. If so, the process returns to step S6. That is, the processes in steps S6 to S9 are performed until all the pixels are set as the target pixel by the right-eye image shift unit 25-1 and the left-eye image shift unit 25-2 and the shift process by the projective transformation ends. Is repeated. If it is determined in step S9 that there is no unprocessed pixel, the process proceeds to step S10.

  In step S10, the right eye image buffer 26-1 and the left eye image buffer 26-2 output the buffered right eye image and left eye image, respectively. In this case, for example, a right-eye image R and a left-eye image L as shown in FIG. 7 are output. In FIG. 7, the right-eye image is an image that shifts to the left as the depth value is smaller as a whole with respect to the input image, and conversely, the left-eye image is as a whole with respect to the input image. The smaller the depth value, the more the image is shifted to the right.

  In step S11, the depth image generation unit 21 determines whether or not there is an input image to be the next frame. If there is, the process returns to step S1, and the subsequent processes are repeated. On the other hand, if there is no input image to be the next frame in step S11, the process ends.

[Projection in right-eye image shift unit and left-eye image shift unit]
Here, the projection (shift) in the right-eye image shift unit 25-1 and the left-eye image shift unit 25-2 will be described.

  When the conventional shift method shown in the right part of FIG. 6 is adopted, in the case of the input image shown in the upper part of FIG. 3, for example, the left-eye image is generated as an image as shown in FIG. In FIG. 8, hidden surface areas BL1 and BL2 are generated in the step portion of the object. Here, the hidden surface region is a region where no pixel is projected, that is, not shifted, and there is no pixel.

  This occurs for the following reasons.

  The upper part of FIG. 9 shows a side view of an object having a steep step with a convex part at the center and a concave part in the other area. The middle part of FIG. 9 shows the upper part of FIG. The pixel arrangement on the horizontal side cross-section of the input image obtained by imaging the object consisting of the side view shown in FIG. 4 from the upper surface is shown, and in the lower stage, the image shown in the middle stage is a projective transformation process, i.e. A pixel array after the shift processing is shown. In the middle and lower parts of FIG. 9, each square is a pixel, the pixel in the hatched portion with the lower right indicates the pixel in the object area in front of the imaging position, and the pixel with the lower left oblique is the back side. The pixels in the object (background) area are shown, and the pixels painted in a grid pattern are pixels in the area where two pixels are projected by projection. As shown in FIG. 9, the convex region protruding from the virtually set display surface, that is, the pixel in the object region in the input image is shifted to the right by a predetermined shift amount and vice versa. The left eye image is formed by shifting the recesses, that is, the pixels in the background area in the input image to the left by a predetermined shift amount. As a result, a hidden surface area composed of pixels shown in white and having no projected pixels as shown by an elliptical area Z0 in the figure is generated in the vicinity of a nearby area where a level difference occurs from the viewpoint position. Note that the object areas in the input image are positions P2 to L1 in FIG. Furthermore, in the left-eye image, the object areas are positions L0 to L1, and the hidden surface areas are positions P1 to L0.

  Since the hidden surface area is generated in this way, when the conventional method is used, processing for generating and filling the generated image for the left eye and the image for filling the hidden surface area generated in the right eye image is performed. It is necessary to do. For this reason, in the conventional method, since it is necessary to perform the process of filling the hidden surface area in the subsequent stage, processing cost is increased.

  On the other hand, in the case of the projection (shift) method in the right-eye image shift unit 25-1 and the left-eye image shift unit 25-2 shown in the left part of FIG. In the case of the input image shown in the upper part of FIG. 3, the image for the left eye is generated as an image as shown in FIG. In FIG. 10, flying areas BL11 and BL12 are generated at the step portion of the object. Here, the enclave region is a region where adjacent pixels that should originally be positioned are replaced and projected, that is, shifted.

  This occurs for the following reasons.

  That is, the upper and middle stages in FIG. 11 are the same as those in FIG. 9, and the lower stage in FIG. 11 is generated by performing the processing shown in the left part of FIG. 6 on the pixels in the middle stage in FIG. The left-eye image is shown. In the case of FIG. 11, the convex portion protruding from the virtually set display surface in the image for the left eye, that is, the pixels of the input image are uniformly shifted in the right direction by a predetermined shift amount, On the contrary, the pixels of the input image are uniformly shifted leftward by a predetermined shift amount into the concave portion, that is, the region of the object (background) on the back side in the image for the left eye. As a result, an enclave area in which projected pixels are replaced as shown by an elliptical area Z1 in the figure is generated in the vicinity of an adjacent area where a step is generated in the distance from the viewpoint position. Further, the object areas in the input image are positions P12 to L1. On the other hand, the object areas in the left-eye image are positions L0 to L1, and the flying areas are positions P11 to P12 and positions P12 to L0.

  By the way, it is known that this flying region occurs in the vicinity of the boundary of the uneven portion where the input image is steep. That is, as shown in FIG. 12, when the boundary between the convex portion and the concave portion that is an object is gentle, the shift amount is set according to the depth value, thereby suppressing the occurrence of the flying region. That is, in FIG. 12, as shown in the uppermost stage of FIG. 12, the depth value gradually decreases from the position P31, and when the position of the display surface is exceeded at the position P32, the depth becomes the tip of the convex portion at the position L0. The value is minimized. Then, when the depth value gradually increases from the position P33 and becomes lower than the position of the display surface at the position L1, the depth value becomes maximum at the position P34.

  That is, when the depth value is set as shown in FIG. 12, all the pixels included in the convex region rather than the position virtually set as the display in the image for the left eye are all in the input image. Shifted to pixels in the object area. Similarly, in the image for the left eye, all the pixels included in the recessed area from the position virtually set as the display are shifted to the pixels in the object (background) area on the back side in the input image. Is done. As a result, an enclave area as shown in FIG. 11 does not occur.

  More specifically, as shown in FIG. 13, when the amount of change in the depth value Depth (parallax) (= slope) | α | ≦ 1, no flying area is generated. In other words, when the gradation H of the pixel value of the depth image set according to the depth value Depth (parallax) is 0 to 255 and the maximum amount of parallax (depth value) is 10, the amount of change When (= slope) | β | ≦ 25.5, no flying area is generated. In FIG. 12, the change amount α (= slope) of the depth value Depth is | α | ≦ 1.

  In the image conversion apparatus 11 of FIG. 1, the depth image is smoothed from the image shown in FIG. 4 to the image shown in FIG. 5 by the LPF 22, thereby causing a sharp change in the depth value. The area is converted into a slowly changing area. As a result, in the image conversion process in the image conversion apparatus 11 of FIG. Therefore, it is ideal that the smoothing intensity of the LPF 22 can be set to the above-described change amount (slope) | α | ≦ 1 when a region where the steepest change occurs is generated. Is.

  Through the above processing, a stereo image such as a left-eye image and a right-eye image capable of three-dimensional stereoscopic viewing of an input image made up of a two-dimensional image while suppressing the occurrence of hidden surface areas and flying areas by a simple method. Can be generated.

<2. Second Embodiment>
[Other configuration examples of image conversion apparatus]
In the above, an example in which the depth image is subjected to the blurring process by applying LPF 22 to suppress a sharp change in the depth value has been described. However, in a region where the depth value changes sharply, the change in the depth value is moderate. For example, after the depth image is once reduced and then enlarged, an area in which the depth value changes sharply so as to reduce the resolution is changed to an area in which the depth value changes gradually. You may do it.

  FIG. 14 shows a configuration example of the image conversion apparatus 11 in which the depth image is reduced and then enlarged to eliminate a region where the depth value changes sharply, and the change in the depth value is moderated. In the image conversion apparatus 11 of FIG. 14, configurations having the same functions as those of the image conversion apparatus of FIG. 1 are denoted by the same names and the same reference numerals, and description thereof will be omitted as appropriate. To do.

  That is, the image conversion apparatus 11 in FIG. 14 is different from the image conversion apparatus 11 in FIG. 1 in that a reduction unit 51 and an enlargement unit 52 are provided instead of the LPF 22. The reduction unit 51 reduces the depth image generated by the depth image generation unit 21 and supplies the reduced depth image to the enlargement unit 52. The enlargement unit 51 enlarges the reduced image, returns it to the size of the original depth image, and supplies it to the shift amount calculation unit 23. That is, the depth image is once reduced in resolution by the reduction unit 51 and then enlarged by the enlargement unit 52. Therefore, even if the resolution is restored to the original state, the resolution is once reduced and the resolution is reduced. The average change amount of the depth value between adjacent pixels becomes small, and the change amount of the depth value becomes gentle.

[Image Conversion Processing by Image Conversion Device in FIG. 14]
Next, image conversion processing will be described with reference to the flowchart of FIG. Note that the processing of steps S31, S34 to S42 in the flowchart of FIG. 15 is the same as the processing of steps S1, S3 to S11 described with reference to the flowchart of FIG.

  That is, when a depth image is generated in step S31, in step S32, the reduction unit 51 reduces the depth image supplied from the depth image generation unit 21 by thinning out the line image or the column unit. To the enlargement unit 52.

  In step S33, the enlargement unit 52 enlarges the reduced depth image supplied from the reduction unit 51 so as to return to the same size as the generated depth image by duplicating and inserting the reduced depth image in units of rows and columns. And supplied to the shift amount calculator 23. That is, in this case, the depth image that has been reduced by the thinning process and then enlarged by copying and inserting is restored to the original size, and the resolution is lower than the generated state. Will be blurred. As a result, since the amount of change is small for the region where the depth value changes sharply, even if the pixel is shifted by the processing shown in the left part of FIG. 6, the generation of the hidden surface region and the flying region is suppressed. Is possible.

  Since the depth image is reduced by the above processing and then processed after being enlarged, the depth image is blurred, so that the amount of change in the depth value in the region where the change in the depth value is steep is reduced. Therefore, the image for the right eye and the image for the left eye capable of three-dimensionally viewing the input image composed of the two-dimensional image while suppressing the generation of the hidden surface area and the flying area by a simple method. It becomes possible to convert into the stereo image which consists of.

<3. Third Embodiment>
[Another configuration example of the image conversion apparatus]
In the above, in order to suppress a sharp change in the depth value, the depth image is reduced by the reduction unit 51 and the enlargement unit 52 to reduce the resolution, and then is enlarged to the original size and subjected to the blurring process. However, since only the region where the depth value changes sharply becomes a problem, the blurring process may be performed only on the region where the depth value changes sharply.

  FIG. 16 shows a configuration example of the image conversion apparatus 11 in which an area where the depth value changes sharply in the depth image is detected, and LPF is applied only to that area as a blurring process. A region in the image where the depth value changes sharply can be considered as an edge portion. Therefore, in the image conversion apparatus 11 in FIG. 16, edge detection processing is performed on the depth image, and only the detected edge and a region in the vicinity thereof are blurred. In the image conversion apparatus 11 of FIG. 16, configurations having the same functions as those of the image conversion apparatus of FIG. 1 are denoted by the same names and the same reference numerals, and description thereof will be omitted as appropriate. To do.

  That is, the image conversion apparatus 11 in FIG. 16 differs from the image conversion apparatus 11 in FIG. 1 in that an edge detection unit 71 and an edge vicinity region LPF 72 are provided instead of the LPF 22. The edge detection unit 71 is, for example, a Laplacian filter, and detects a pixel that becomes an edge and supplies the detected pixel to the edge vicinity region LPF 72.

  The edge vicinity region LPF 72 is based on the information on the pixel serving as the edge supplied from the edge detection unit 71 in the depth image supplied from the depth image generation unit 21, and the pixels near the edge. Only LP is multiplied by LPF and output to the shift amount calculation unit 23. It should be noted that the setting of the edge region and the vicinity of the edge and the strength to apply the LPF are set so that the above-described depth value change amount (slope) | α | ≦ 1.

[Image Conversion Processing by Image Conversion Device in FIG. 16]
Next, image conversion processing will be described with reference to the flowchart of FIG. Note that the processing of steps S61, S64 to S62 in the flowchart of FIG. 15 is the same as the processing of steps S1, S3 to S11 described with reference to the flowchart of FIG.

  That is, when a depth image is generated in step S61, in step S62, the edge detection unit 71 applies a Laplacian filter to the depth image supplied from the depth image generation unit 21, for example. The edge is detected, and the edge detection image is supplied to the edge vicinity region LPF 72.

  In step S <b> 63, the edge vicinity region LPF 72 determines the edge region and the pixels corresponding to the pixels in the vicinity region of the edge detection image supplied from the depth image supplied from the depth image generation unit 21. By applying the LPF, only the region where the depth value changes abruptly is made a gradual change.

  By detecting the edge from the depth image by the above process and applying LPF only to the detected edge area and its neighboring area, the depth value change is likely to occur in the depth image even in the depth image. Since only the steep areas are blurred, the amount of change in the depth values in the areas where the change in the depth values is steep is reduced and gradually changes. It is possible to convert an input image made up of a two-dimensional image into a stereo image made up of a right-eye image and a left-eye image that can be three-dimensionally viewed while suppressing the generation of a region.

  By the way, the series of processes described above can be executed by hardware, but can also be executed by software. When a series of processing is executed by software, a program constituting the software may execute various functions by installing a computer incorporated in dedicated hardware or various programs. For example, it is installed from a recording medium in a general-purpose personal computer or the like.

  FIG. 18 shows a configuration example of a general-purpose personal computer. This personal computer incorporates a CPU (Central Processing Unit) 1001. An input / output interface 1005 is connected to the CPU 1001 via a bus 1004. A ROM (Read Only Memory) 1002 and a RAM (Random Access Memory) 1003 are connected to the bus 1004.

  An input / output interface 1005 includes an input unit 1006 including an input device such as a keyboard and a mouse for a user to input an operation command, an output unit 1007 for outputting a processing operation screen and an image of a processing result to a display device, a program, and various data. Are connected to a storage unit 1008 including a hard disk drive and the like, and a local area network (LAN) adapter and the like, and a communication unit 1009 that executes communication processing via a network represented by the Internet. Also, a magnetic disk (including a flexible disk), an optical disk (including a CD-ROM (Compact Disc-Read Only Memory), a DVD (Digital Versatile Disc)), a magneto-optical disk (including an MD (Mini Disc)), or a semiconductor A drive 1010 for reading / writing data from / to a removable medium 1011 such as a memory is connected.

  The CPU 1001 is read from a program stored in the ROM 1002 or a removable medium 1011 such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory, installed in the storage unit 1008, and loaded from the storage unit 1008 to the RAM 1003. Various processes are executed according to the program. The RAM 1003 also appropriately stores data necessary for the CPU 1001 to execute various processes.

  In this specification, the step of describing the program recorded on the recording medium is not limited to the processing performed in time series in the order described, but of course, it is not necessarily performed in time series. Or the process performed separately is included.

  DESCRIPTION OF SYMBOLS 11 Image converter, 21 Depth image generation part, 22 LPF, 23 Shift amount calculation part, 24 Shift amount buffer, 25-1 Image shift part for right eyes, 25-2 Image shift part for left eyes, 26-1 Right eye Image buffer, 26-2 left-eye buffer, 51 reduction unit, 52 enlargement unit, 71 edge detection unit, 72 edge vicinity region LPF

Claims (3)

A depth image generating unit that generates a depth image from an input image signal composed of a two-dimensional image signal;
A blur processing unit that blurs the depth image;
A shift amount calculation unit that calculates a shift amount set based on a depth value of each pixel for each pixel of the depth image subjected to the blur processing by the blur processing unit;
For each pixel in the input image signal, three-dimensional projection is performed by projecting a pixel at a position shifted rightward or leftward by a shift amount corresponding to each pixel of the depth image calculated by the shift amount calculation unit. look including a shift unit that generates an image for the right eye or the left eye image in the image signal,
The blur processing unit
An edge detection unit for detecting an edge from the depth image;
Including an edge detected by the edge detection unit, and an LPF for LPF processing the area in the vicinity thereof,
The edge of the depth image is detected by the edge detection unit, and the edge detected by the LPF and an area in the vicinity thereof are subjected to LPF processing with intensity corresponding to the amount of change in the depth value of the corresponding depth image. An image processing device that performs blurring processing . A depth image generating step for generating a depth image from the input image signal consisting of the two-dimensional image signal in a depth image generating unit for generating a depth image from the input image signal consisting of the two-dimensional image signal;
In a blur processing unit that blurs the depth image, a blur processing step that blurs the depth image;
The depth image blurred by the blur processing unit in the shift amount calculation unit that calculates the shift amount set based on the depth value of each pixel for each pixel of the depth image blurred by the blur processing unit. A shift amount calculating step for calculating a shift amount set based on the depth value of each pixel for each of the pixels;
For each pixel in the input image signal, three-dimensional projection is performed by projecting a pixel at a position shifted rightward or leftward by a shift amount corresponding to each pixel of the depth image calculated by the shift amount calculation unit. The shift amount corresponding to each pixel of the depth image calculated by the processing of the shift amount calculation step for each pixel in the input image signal in the shift unit that generates the right-eye image or the left-eye image in the image signal only the right direction or by projecting the position of the pixel shifted to the left, seen including a shift step of generating an image for the right eye or the left eye image in the three-dimensional image signal,
The blur processing step includes:
An edge detection step for detecting an edge from the depth image;
Including an edge detected by the processing of the edge detection step, and an LPF processing step of applying an LPF to a region in the vicinity thereof,
The edge of the depth image is detected by the processing of the edge detection step, and the edge detected by the LPF and the area in the vicinity thereof are subjected to LPF processing with intensity corresponding to the amount of change in the depth value of the corresponding depth image. An image processing method for performing blur processing . A depth image generating unit that generates a depth image from an input image signal composed of a two-dimensional image signal;
A blur processing unit that blurs the depth image;
A shift amount calculation unit that calculates a shift amount set based on a depth value of each pixel for each pixel of the depth image subjected to the blur processing by the blur processing unit;
For each pixel in the input image signal, by projecting a pixel at a position shifted to the right or left by the shift amount corresponding to each pixel of the depth image calculated by the shift amount calculation unit, three-dimensional look including a shift unit that generates an image for the right eye or the left eye image in the image signal,
The blur processing unit
An edge detection unit for detecting an edge from the depth image;
Including an edge detected by the edge detection unit, and an LPF for LPF processing the area in the vicinity thereof,
The edge of the depth image is detected by the edge detection unit, and the edge detected by the LPF and an area in the vicinity thereof are subjected to LPF processing with intensity corresponding to the amount of change in the depth value of the corresponding depth image. A depth image generating step of generating a depth image from an input image signal composed of a two-dimensional image signal in the depth image generation unit in a computer that controls the image processing apparatus that performs blurring in
A blur processing step for blur processing the depth image in the blur processing unit;
A shift amount calculation step of calculating a shift amount set based on a depth value of each pixel for each pixel of the depth image subjected to the blur processing by the processing of the blur processing step in the shift amount calculation unit;
For each pixel in the input image signal in the shift unit, a pixel at a position shifted rightward or leftward by a shift amount corresponding to each pixel of the depth image calculated by the processing of the shift amount calculation step. By performing projection, a process including a shift unit that generates an image for the right eye or an image for the left eye in the three-dimensional image signal is executed ,
The blur processing step includes:
An edge detection step for detecting an edge from the depth image;
Including an edge detected by the processing of the edge detection step, and an LPF processing step of applying an LPF to a region in the vicinity thereof,
The edge of the depth image is detected by the processing of the edge detection step, and the edge detected by the LPF and the area in the vicinity thereof are subjected to LPF processing with intensity corresponding to the amount of change in the depth value of the corresponding depth image. A program that blurs by letting

Images