JP2016131297A - Stereoscopic image generation device, stereoscopic image generation method and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To enable an image picked up from a subject to be efficiently output as a stereoscopic image.SOLUTION: A stereoscopic image generation device has generating means for converting an image achieved by imaging a subject on the basis of distance information concerning the subject to generate a stereoscopic image. In the generating means, the respective pixels of the image are arranged in the depth direction based on the distance information to generate data. In the data, a background face and a foreground face which have predetermined inclinations in the depth direction and are in parallel to each other are set so as to sandwich the image constituting the subject therebetween, and pixels which are located at the rear side of the background face in the depth direction are projected to the background face to generate the stereoscopic image.SELECTED DRAWING: Figure 1

Description

  The present invention relates to a stereoscopic image generation device, a stereoscopic image generation method, and a program.

  In recent years, 3D printers have begun to spread widely. A 3D printer is a device that forms a solid by stacking recording materials such as resin and gypsum powder. By using this, a device or system for outputting a three-dimensional relief of a photographed image, that is, a three-dimensional image, based on the photographed image and distance information indicating the depth has been provided.

  Such a stereoscopic image is used for viewing in a room or the like, instead of a conventional planar photograph. When decorating a photo, it is often placed in a photo stand and supported by a stand or wall and tilted at an angle. Similarly, if a stereoscopic image can be displayed obliquely, it can be easily viewed and can be prevented from falling down.

  In addition to the 3D printer, there are various methods such as a method of forming a solid by thermally deforming a thermally expandable sheet, embossing printing, embossing printing, laser engraving, etc. There is. Each of the output methods is characterized in that there is a restriction on the maximum height difference of the stereoscopic image to be output, and the more the recording material is stacked, the more cost is required.

  For example, when a typical photographed image in which a short person is in front and a tall person is in the back is three-dimensionalized, each person is given a depth within the range of the maximum height difference. Therefore, the difference in height is used not only for the three-dimensional effect of the person but also for the positional relationship of the person, so that the three-dimensional effect of each person is diminished.

  In the apparatus of Patent Document 1, when an input image is converted into a line drawing and a three-dimensional layer having a thickness corresponding to the distance information of the subject is formed on the medium, the upper part is thickened and the three-dimensional object is thinned toward the lower part. In this way, the three-dimensional effect is emphasized. Further, in the apparatus of Patent Document 2, in a stereoscopic image output method in which the maximum height difference is limited, the height of each pixel is divided by the maximum height difference, and the height of the pixel is replaced with the remainder, thereby generating a stereoscopic image. Is output to satisfy the maximum height difference constraint. In the apparatus of Patent Document 3, in a 3D printer using a powder material as a recording material, a frame surrounding the main body to be output is output simultaneously, thereby eliminating the need for a recording material outside the frame.

JP 2013-157597 A JP 2009-199230 A JP 2011-156783 A

  However, since Patent Document 1 uses an input image as a line drawing, there are cases where the three-dimensional shape of the subject cannot be expressed sufficiently. In addition, the original three-dimensional shape of the subject may not be expressed by being thinner toward the bottom. In the apparatus of Patent Document 2, the area in which the height in the optical axis direction of the photographed image changes gently is changed in a saw-tooth shape every time it is divisible by the maximum height difference. If it expresses, it will become a three-dimensional image with a great discomfort. In the apparatus of Patent Document 3, since the recording material having the difference in elevation between the input image and the distance information is laminated as it is in the z-axis direction, a large amount of recording material is required when the elevation difference is large. End up.

  Therefore, in the present invention, it is possible to efficiently output an image showing a subject as a stereoscopic image.

The present invention for solving the above problems is a stereoscopic image generating device,
Generation means for converting an image obtained by imaging a subject based on distance information about the subject to generate a stereoscopic image;
In the data generated by arranging each pixel of the image in the depth direction based on the distance information, a pixel constituting the subject is composed of a parallel background surface and a foreground surface having a predetermined inclination in the depth direction. Set to pinch,
The image processing apparatus includes a generating unit configured to generate the stereoscopic image by projecting, on the background surface, pixels positioned rearward in the depth direction from the background surface in the data.

  According to the present invention, an image showing a subject can be efficiently output as a stereoscopic image.

The figure which shows the structural example of the stereo image production | generation apparatus corresponding to Embodiment 1 of invention. The figure for demonstrating an example of the method of setting the conditions for the stereo image production | generation corresponding to Embodiment 1 of invention, and the preview screen at the time of stereo image imaging | photography. The figure for demonstrating the picked-up image and distance information corresponding to embodiment of invention. The flowchart for demonstrating an example of the production | generation process of the stereo image corresponding to Embodiment 1 of invention, and a production | generation process. The figure which shows an example of the preview screen at the time of the image photography corresponding to Embodiment 2 of invention. The figure which shows the structural example of the stereo image production | generation apparatus corresponding to Embodiment 3 of invention. The flowchart which shows an example of the production | generation process of the stereo image corresponding to Embodiment 3 of invention. The figure for demonstrating the production | generation process of the stereo image corresponding to Embodiment 3 of invention. The figure for demonstrating the appearance of the stereo image corresponding to Embodiment 4 of invention. The figure for demonstrating the production | generation process of the stereo image corresponding to Embodiment 4 of invention. The figure for demonstrating an example of the process of the background surface corresponding to Embodiment 4 of invention.

[Embodiment 1]
As a first embodiment of the invention, a stereoscopic image generating apparatus having a stereoscopic image data output function will be described. First, with reference to FIG. 1, the example of a structure of the stereo image production | generation apparatus corresponding to embodiment of invention is demonstrated. An apparatus corresponding to an embodiment of the invention is configured as a digital camera, for example. In addition, any information processing terminal such as a personal computer, a mobile phone, a smartphone, a PDA, a tablet terminal, a digital video camera, or an imaging device can be used. Note that the stereoscopic image generation processing corresponding to the embodiment of the present invention only needs to be able to prepare a captured image and distance information of the subject included in the image. The function and the distance measuring function may not be provided. In that case, it may be configured as a program on an output device such as a 3D printer or a computer that outputs a stereoscopic image based on an externally captured image and distance information input.

  In FIG. 1A, a digital camera 100 has the following configuration. In the digital camera 100, each block may be configured by hardware using a dedicated logic circuit or memory, except for physical devices such as an image sensor and a display element. Alternatively, the processing program stored in the memory may be configured by software by a computer such as a CPU executing the processing program.

  The control unit 101 controls each block connected by the internal bus 102. The FLASH memory controller 103 reads / writes nonvolatile data from / to the FLASH memory 104 based on a request from the control unit 101. The control unit 101 operates based on a program stored in the FLASH memory 104 and stores nonvolatile data such as a captured image in the FLASH memory 104.

  The DRAM controller 105 reads and writes data to and from the DRAM 106 based on requests from each block. The control unit 101 and each block use temporary data during operation stored in the DRAM 106. The operation unit 107 includes a shutter button or the like provided on the exterior of the digital camera 100, and transmits the user's button operation to the control unit 101. The camera signal processing unit 108 controls the camera 109 based on an instruction from the control unit 101 to photograph the subject, and stores the obtained image data in the DRAM 106.

  The camera 109 of the present embodiment can acquire a plurality of parallax images by one shooting (exposure). Various cameras are known as representative cameras such as a multi-lens camera such as a stereo camera. In this embodiment, a microlens array in which an imaging device divides an exit pupil of an imaging lens on a light receiving surface. (Hereinafter referred to as MLA). FIG. 1B schematically shows a state in which the image sensor 4 is viewed from the front and side of the camera 1. An MLA 141 is formed on the light receiving surface of the pixel group 143 included in the image pickup device 4, and each pixel constituting the pixel group 143 includes one microlens 142 and two photodiodes (see FIG. 1C). Photoelectric conversion region) 143a and 143b.

  FIG. 1C conceptually shows the exit pupil 144 of the imaging lens 2. The A image pupil 145a and the A pixel 143a, and the B image pupil 145b and the B pixel 143b are conjugated by the microlens 142, respectively. Have. Accordingly, each pixel of the image pickup device has a pupil division function. A light beam that has passed through the A image pupil 145a in the right half of the exit pupil 144 is transmitted to the A pixel 143a, and B in the left half of the exit pupil 144 is applied to the B pixel 143b. The light beam that has passed through the image pupil 145b enters. Therefore, an image composed of the A pixel group and an image composed of the B pixel group are parallax images. Then, by detecting the phase difference between a plurality of parallax images obtained by such an image sensor, distance information (depth information) of the subject can be obtained. Also, each parallax image or an added image obtained by adding the parallax images can be used as two-dimensional information of the subject.

  The camera signal processing unit 108 performs development processing such as A / D conversion for converting an electrical signal from the camera 109 into a digital signal, gamma processing, noise reduction processing, and edge enhancement processing. Simultaneously with the shooting, the camera 109 measures the defocus amount for each area of the subject, and stores it in the DRAM 106 together with the image data. Further, the image data is reduced for display and stored in a display frame memory provided in the DRAM 106.

  The camera 109 can include, for example, an optical system such as a focus lens, a diaphragm, and a shutter, a sensor, and an imaging control unit. The sensor converts the amount of light of the subject imaged in the optical system into an electrical signal by photoelectric conversion, and is configured as an image sensor such as a CMOS or CCD. The imaging control unit controls the focus and aperture of the optical system, the shooting sensitivity of the shutter and the image sensor, and the like.

  The distance map generation unit 111 generates distance information for each image area from the defocus amount for each area stored in the DRAM 106 and stores the distance information in the DRAM 106. The stereoscopic image generation unit 112 generates stereoscopic image data according to the control of the control unit 101 based on the image data stored in the DRAM 106 and the distance information, and stores the generated stereoscopic image data in the DRAM 106. The memory card controller 113 records the stereoscopic image data stored in the DRAM 106 on a memory card 150 connected to the digital camera 100 main body. The final recording (output) format of the stereoscopic image data is, for example, the STL format. By outputting the stereoscopic image data stored in the memory card with an external 3D printer or the like, the stereoscopic image data can be materialized.

  Based on an instruction from the control unit 101, the graphic drawing unit 114 draws information to be notified to the user as characters and figures in the display frame memory secured in the DRAM 106. An LCD (Liquid Crystal Display) controller 115 reads the content of the display frame memory stored in the DRAM 106 and displays it on the LCD touch panel 116. The touch panel controller 117 detects a user's touch operation on the LCD touch panel 116 and transmits the detected operation to the control unit 101.

  Next, an example of a method for setting a condition for generating a stereoscopic image and an example of a preview screen at the time of capturing a stereoscopic image will be described with reference to FIG. First, FIG. 2A shows an example of the input screen 200 displayed on the LCD touch panel 116 of the digital camera 100. By using the input screen 200, the user of the digital camera 100 can input the size and angle of the stereoscopic image to be output. Alternatively, the user (user) may input three-dimensional information including at least the depth, and the size and inclination of the input image may be set so that the user (user) fits in the region (cuboid).

  Specifically, the input screen 200 shows a base portion 201 that is a background surface of a stereoscopic image and a foreground surface 202 that is separated from the base portion 201 by a maximum height difference in the front direction. While viewing the input screen 200, the user can set the size of the stereoscopic image by using the input boxes for the image width w203 and the image height h204. In addition, the distance between the base portion 201 and the foreground surface 202, that is, the maximum height difference 205 in the depth direction of the stereoscopic image can be set. Furthermore, it is possible to set an installation angle θ206 when installing a stereoscopic image.

  When the user touches any input box, the control unit 101 displays a numeric keypad or the like on the LCD touch panel 116. The user can set each value by touching the LCD touch panel 116. In addition, since display of a numeric keypad and the touch operation with respect to it are common, illustration is abbreviate | omitted here. A cancel button 207 is a button for the user to cancel the value input on the screen. The decision button 208 is a button for the user to confirm the setting of the value input on the screen. When the enter button 208 is operated, a stereoscopic image is generated thereafter according to the size, angle, and depth values set on the input screen 200.

  Next, FIG. 2B shows an example of a preview screen during image shooting. What is displayed on the LCD touch panel 116 is a captured image 211 captured by the camera 109. The captured image 211 has an aspect ratio corresponding to a predetermined size and angle of the stereoscopic image input on the input screen 200 shown in FIG. That is, the aspect ratio of the captured image is w / (h · cos θ) with respect to the width w, height h, and angle θ of the stereoscopic image. Assuming that the width of the LCD touch panel 116 is Width and the height is Height, Width / Height <w / (h · cos θ) in the example of this figure, and the captured image 211 is assumed to be horizontally long. Therefore, the photographed image 211 is displayed in accordance with the width W of the LCD touch panel 116, and a portion where the photographed image 211 is not sufficient for the height Height of the LCD touch panel is displayed with black bands 212 and 213 on the top and bottom.

  Next, an example of a captured image and distance information will be described with reference to FIG. FIG. 3A is a schematic view of a situation where an image is being taken by the digital camera 100 as seen from the side. In this example, the subject 301 and the subject 302 are located at different distances d1 and d2 in the shooting direction of the digital camera 100. Although FIG. 3 shows a person as an example of a subject, the subject is not limited to a person. z is the optical axis of the camera, and y is the vertical axis. FIG. 3B shows an example of the image photographed in FIG. In the image 310, in addition to the subjects 301 and 302, the sun 303 is shown in the background. x indicates the horizontal direction, and y indicates the vertical axis.

  FIG. 3C shows an example of distance information corresponding to the image 310 obtained at the time of shooting shown in FIG. In the distance information 320, a dark color region is positioned on the near side and a thin region is positioned on the back side. A subject 301 close to the digital camera 100 is relatively dark, and then a subject 302 is slightly dark. Since the sun 303 is located at infinity, distance information cannot be obtained. FIG. 3D is a diagram showing the positions and sizes of the faces of the subjects 301 and 302 detected by the feature detection unit 601 as the subject of interest surrounded by frames 311 and 312 with respect to the image 310. FIG. 3D is referred to in a third embodiment to be described later.

  Next, with reference to FIG. 4, the flow of the stereoscopic image generation process will be described. FIG. 4A is a flowchart illustrating an example of a process in which the stereoscopic image generation unit 112 generates a stereoscopic image based on the control of the control unit 101. 4B to 4E show examples of stereoscopic image data generated temporarily or finally by the stereoscopic image generation unit 112 in each step of the generation process.

The process corresponding to the flowchart of FIG. 4 can be realized by executing a program (stored in a ROM or the like) corresponding to one or more processors functioning as the stereoscopic image generation unit 112, for example. First, in S401, based on the captured image and distance information stored in the DRAM 106, each pixel of the captured image is arranged in the depth direction (z direction) to generate data that is the source of the stereoscopic image. FIG. 4B is a view of a stereoscopic image 421 obtained by stereoscopicizing the subjects 301 and 302 included in the captured image 310 based on the distance information 320 from the side. A depth is added to each image region in the z-axis direction, an infinite region is the background 411, and the subjects 301 and 302 are three-dimensionalized with a depth.
Next, in S402, a foreground surface and a background surface are set for the stereoscopic image obtained in S401. FIG. 4C is a diagram illustrating a case where a foreground surface 412 and a background surface 413 are set for the stereoscopic image 421. The foreground surface 412 is provided to be inclined with respect to the background 411 by a previously input angle θ, and is disposed at a position in contact with the stereoscopic image 421 on the near side in the z-axis direction. Further, the background surface 413 is arranged in parallel to the foreground surface 412 with a predetermined maximum height difference H on the far side in the z-axis direction from the foreground surface 412. Thereby, the foreground surface 412 and the background surface 413 are set so as to sandwich the pixels constituting the subjects 301 and 302.

  In step S403, projection onto the background surface 413 is performed. In FIG. 4C described above, the area of the subject behind the background surface 413 in the depth direction (z direction) is projected onto the background surface 413 in the z-axis direction. The result is a stereoscopic image 422 having a background 414 projected on the background surface 413 in FIG.

  In step S404, the stereoscopic image is rotated. The stereoscopic image 422 is rotated by 90 degrees −θ so that the background 414 in FIG. FIG. 4E shows the stereoscopic image 422 after rotation. By providing an r-axis and a q-axis rotated by 90 degrees −θ with respect to the z-axis and the y-axis, and storing the data of the stereoscopic image 422 in the DRAM 106 with reference to these axes, a stereoscopic image generation process is performed. Finish.

  In the description of the present embodiment, a 3D printer is cited as a method for outputting a stereoscopic image as an entity, but the same effect can be obtained even when a stereoscopic image is output by another method described in the related art. . Further, in the description of the present embodiment, the distance map is generated from the defocus amount for each region obtained from the camera. However, a plurality of images obtained by using parallax images obtained by a two-lens imaging system or photographed at different focal lengths are used. You may produce | generate by another method, such as detecting the area | region for every focal distance from the image of 1 sheet.

  Further, in the description of the present embodiment, when the foreground plane is arranged, the foreground plane is in contact with the near side in the z-axis direction of the entire stereoscopic image. However, the subject of interest is detected from a plurality of subjects in the image. The subject of interest may be arranged so as to fit between the foreground surface and the background surface. In this case, since there may be a subject in the depth direction from the foreground surface, the subject in front of the foreground surface 412 is placed in the z-axis direction in the foreground surface 412 in the same manner as when the projection is performed on the background surface 413. Project to.

[Embodiment 2]
Next, a second embodiment of the invention will be described. This embodiment is different from the first embodiment in that a three-dimensional image is generated based on a preview image and a region is displayed as a three-dimensional image while the preview screen is displayed during image shooting.

  With reference to FIG. 5, a preview screen at the time of image capturing in the present embodiment will be described. FIG. 5 shows an example of a preview screen at the time of image shooting. The difference from FIG. 2 of the first embodiment is that the region projected onto the background surface and becomes a flat surface is identifiable. For example, as shown in FIG. 5, a zebra pattern (striped pattern) can be superimposed on an area that becomes a flat surface, so that an area that is displayed three-dimensionally and an area that is displayed flat can be identified. In FIG. 5, a zebra pattern is superimposed on portions 501 and 502 projected on the background of the subject in addition to the background of the image 311. Reference numeral 500 denotes a blinking pattern indicating a region where the stereoscopic image is in contact with the foreground surface. For example, white and transparent colors may be alternately displayed and blinked.

  In the present embodiment, the stereoscopic image generation unit 112 repeatedly performs a stereoscopic image generation process corresponding to the flowchart of FIG. 4A on the captured image based on the control of the control unit 101 while the preview screen is displayed. The control unit 101 extracts a region serving as a background surface from the stereoscopic image data stored in the DRAM 106, and the graphic drawing unit 114 draws a zebra pattern on the display frame stored in the DRAM 106. Further, an area in contact with the foreground surface is extracted, and a blinking pattern is drawn on the display frame stored in the DRAM 106. When the subject is projected on the foreground surface, the portion projected on the foreground surface may be displayed so as to be identifiable using a pattern different from the background surface.

  With such a preview display, the user can recognize which region of the subject is three-dimensionalized by photographing as it is.

[Embodiment 3]
Next, a third embodiment of the invention will be described as a modification of the above-described first and second embodiments. FIG. 6 is a diagram illustrating a configuration of a digital camera 600 as a stereoscopic image generating apparatus corresponding to the present embodiment. The basic configuration of the digital camera 600 is the same as that of the digital camera 100 of the first and second embodiments, and corresponding blocks are denoted by the same reference numerals. Hereinafter, a configuration unique to the digital camera 600 will be described, and description of the configuration common to FIG. 1 will be omitted.

  The feature detection unit 601 detects the face of a person as the subject of interest from the image data stored in the DRAM 106 and notifies the control unit 101 of the position and size. Based on the position and size, the control unit assigns a score of the subject of interest and treats the subject of interest with the highest score as the main subject. The stereoscopic image output unit 602 outputs the stereoscopic image data stored in the DRAM 106 to the outside. By connecting an external 3D printer or the like as an output destination, it is possible to obtain an output result that materializes stereoscopic image data. Alternatively, the stereoscopic image output unit 602 itself may function as a 3D printer. The LCD 603 is a liquid crystal display device and is used instead of the LCD touch panel 116 of FIG. In the present embodiment, the LCD touch panel 116 may not be a touch panel like the LCD touch panel 116 of FIG. 1, but the LCD touch panel 116 may be used as it is.

  Next, with reference to FIG. 7 and FIG. 8, the flow of the stereoscopic image generation process in the present embodiment will be described. FIG. 7 is a flowchart illustrating an example of processing in which the stereoscopic image generation unit 112 generates a stereoscopic image based on the control of the control unit 101. FIG. 8 shows an example of stereoscopic image data generated temporarily or finally by the stereoscopic image generation unit 112 in each step of the generation process. The processing corresponding to the flowchart of FIG. 7 can be realized by executing a program (stored in a ROM or the like) corresponding to one or more processors functioning as the stereoscopic image generation unit 112, for example.

  First, in step S <b> 701, the image is three-dimensionalized based on the captured image and distance information stored in the DRAM 106. FIG. 8A is a view of a stereoscopic image 811 obtained by stereoscopicizing the subjects 301 and 302 included in the captured image 310 based on the distance information 320 from the side. A depth is added in the z-axis direction for each image area, an infinite area is the background 801, and the subjects 301 and 302 are three-dimensionalized with a depth.

  In step S <b> 702, a background surface and a foreground surface are initially set for the stereoscopic image 811. First, as shown in FIG. 8B, the background surface 803 and the foreground surface 802 are provisionally set at predetermined initial setting positions. Here, the background surface 803 is disposed on the same plane as the infinitely distant background 801, and a foreground surface 802 parallel to the background surface 803 is disposed in front of the background surface 803 with a predetermined maximum height difference H therebetween. 8B correspond to the face frames of the subjects 301 and 302 shown in FIG. 3D detected by the feature detection unit 601.

  In subsequent S703, the background surface 803 and the foreground surface 802 are moved and rotated. The frames 311 and 312 are arranged based on the position where the face is detected and the distance information of the area, and are three-dimensionalized by providing a depth corresponding to the detected size in the z-axis direction. The background surface 803 and the foreground surface 802 are moved and kept at the maximum height difference H so that these frames are at least partially included in a space sandwiched between the background surface 803 and the foreground surface 802. Rotate. In FIG. 3D, the positions and sizes of the faces of the subjects 301 and 302 detected by the feature detection unit 601 are indicated by frames 311 and 312, and a case where a person's face is used as a feature region is shown. However, since the subject can include other than a person, a feature region corresponding to the subject can be set. For example, in the case of a plant, a frame can be set with a flower or fruit portion as a feature region. In addition, it is possible to set a feature region that the user is impressed according to the type of subject.

  FIG. 8C shows the result of movement and rotation of the background surface 803 and the foreground surface 802. Here, the space between the background surface 803 and the foreground surface 802 includes a frame 311 of the subject 301 that is the main subject and a frame 312 of the subject 302 that is another subject of interest. Here, the background surface 803 and the foreground surface 802 are rotated to an angle of θ with respect to the y-axis, but this rotation angle can be constrained in advance. For example, when the stereoscopic image data generated from the photographed image taken with the optical axis in the horizontal direction as the image used for the description is materialized and decorated as a photo frame, the range of the tilt angle is as follows: It is appropriate to set θ to about 0 to 30 degrees.

  The movement and rotation of the background surface and the foreground surface are performed with the highest priority given to placing the frame of the main subject in the space, and then performed so that the total score of the subject of interest within the space becomes the highest. If there is room for movement and rotation after satisfying them, the rotation angle is made as close to 0 degree as possible. If there is still room for movement, the size of the area occupied by the subject behind the background when shifted to the back is compared with the size of the area occupied by the subject before the foreground when shifted forward. In the former case, the area occupied by the subject in the back is larger than that in the front, so the background surface and the foreground surface are shifted to the back within the range of the room, and in the latter case, they are shifted to the front.

  In step S704, projection is performed on the background surface and the foreground surface. In FIG. 8C described above, the area of the subject behind the background surface 803 is projected onto the background surface 803 in the z-axis direction. In this example, there is no subject in front of the foreground surface 802, but when it exists, the region is projected onto the foreground surface 802 in the z-axis direction. The result is a stereoscopic image 812 having a background 804 projected onto the background surface 803 in FIG.

  In subsequent S705, the stereoscopic image is rotated. The stereoscopic image 812 is rotated by 90 degrees −θ so that the background 804 in FIG. FIG. 8E is a diagram illustrating the stereoscopic image 812 after rotation. By providing an r-axis and a q-axis rotated by 90 degrees −θ with respect to the z-axis and the y-axis, and storing the data of the stereoscopic image 812 in the DRAM 106 with reference to these axes, a stereoscopic image generation process is performed. Finish. The stereoscopic image 812 generated in this way can be materialized by a connected external 3D printer or the like by sequentially outputting the stereoscopic image output unit 602 in a slice form from the background 804.

  According to the digital camera 600 as a stereoscopic image output apparatus corresponding to the present embodiment, when a plurality of objects of interest shown in a captured image are three-dimensionally expressed, recording necessary for three-dimensional expression is performed due to the inclination of the base background surface. The agent can be reduced. In addition, even when a device for outputting a stereoscopic image has a maximum height difference restriction, a plurality of subjects of interest can be efficiently three-dimensionally represented within the restriction range.

  In the description of the present embodiment, for the sake of simplicity, the background surface and the foreground surface have been described as rotating with respect to the x axis. However, the rotation may be performed including the y axis. . With such a configuration, the effect of the present invention can be obtained even for images in which a plurality of persons are side by side as viewed from the camera at different distances. In addition, the face of a plurality of persons has been described as an example of a plurality of attention subjects, but the effect of the present invention can be obtained even if a plurality of portions of one attention subject are handled as a plurality of attention subjects, respectively. it can.

[Embodiment 4]
Next, a fourth embodiment of the invention will be described as a modification of the above-described first to third embodiments. In the present embodiment, the stereoscopic image generating unit 112 that generates a stereoscopic image in accordance with the control of the control unit 101 reduces the uncomfortable feeling when viewing the stereoscopic image from the normal direction of the inclined background surface. Is generated.

  With reference to FIG. 9, how the stereoscopic image is seen from the normal direction of the background surface will be described. A stereoscopic image 812 in FIG. 9A is a stereoscopic image generated by the stereoscopic image generation processing corresponding to the third embodiment. A stereoscopic image 912 in FIG. 9B is a stereoscopic image generated by a stereoscopic image generation process corresponding to the present embodiment.

  First, in FIG. 9A, when the stereoscopic image 412 is viewed from the viewpoint 901 in the optical axis direction at the time of shooting, the image 921 looks equivalent to the shot image. On the other hand, when the stereoscopic image 812 is viewed from the viewpoint 902 in the normal direction of the background surface 804, as shown in the image 922, a depth is added, and partial areas 961 and 962 of the three-dimensional objects 301 and 302 extend. It looks like. In addition, in the image 922, since the captured image projected on the background surface inclined to the angle θ is viewed from the normal direction, the length in the vertical direction, which was h in the image 921, is (h / cos θ).

  In contrast, in FIG. 9B, the depth of the subjects 301 and 302 is changed from the optical axis direction at the time of shooting to the normal direction of the background surface with respect to the stereoscopic image 912. Accordingly, when the stereoscopic image 912 is viewed from the viewpoint 902, it looks like an image indicated by 932. In the image 932, unlike the image 922 shown above, a part of the subjects 301 and 302 does not appear to extend. It should be noted that the areas 971 and 972 drawn in the back of the head are shadows drawn flat on the background surface by processing to be described later.

  On the other hand, when the stereoscopic image 912 is viewed from the viewpoint 901, it looks like an image 931. In the image 931, compared to the image 921 shown above, some areas 981 and 982 of the subject that are three-dimensionalized with a depth appear behind the necks of the subjects 301 and 302 as extended portions. However, since the visible range is small compared to the regions 961 and 962 in the image 922 in FIG. 9A, the visual discomfort is relatively small. In addition, since the regions 981 and 982 that appear to extend extend toward the lower direction (the ground direction) of the image, if the illumination that illuminates the stereoscopic image 912 is in the ceiling direction, these surfaces become shadows and are less noticeable. Discomfort is reduced. Or, for a three-dimensional part with depth, for a surface that is not shown in the captured image, draw and output like a shadow with a reduced brightness according to the proportion of the surface facing the ground. Also good.

  Further, the background surface 904 of the stereoscopic image 912 in FIG. 9B is reduced in length in the vertical direction to (cos θ + 1) / 2 times the background surface 804 of the stereoscopic image 812 shown in FIG. ing. As a result, the length of the background surface viewed from the viewpoint 902 is reduced to a length of h (1 + 1 / cos θ) / 2 in the vertical direction. On the other hand, the background surface viewed from the viewpoint 901 has a length in the vertical direction h (cos θ + 1) / 2, which appears to be shorter than the photographed image, but is uncomfortable as compared with the vertical extension of the image 922 described above. small.

  Next, a stereoscopic image generation process corresponding to the present embodiment will be described with reference to FIG. FIG. 10 illustrates an example of stereoscopic image data generated temporarily or finally by the stereoscopic image generation unit 112 in the stereoscopic image generation processing corresponding to the present embodiment. Since the stereoscopic image generation process of the present embodiment is the same as the process in FIG. 4 of the first embodiment and FIG. 7 of the third embodiment to the middle, only the differences will be illustrated and described.

  FIG. 10A shows a z-axis that does not appear in the captured image, that is, does not constitute a pixel of the captured image, with respect to the stereoscopic image 422 in FIG. 4E or the stereoscopic image 812 illustrated in FIG. Areas 1001 and 1002 with directional depths are indicated by hatching. In this embodiment, the regions 1001 and 1002 are removed, and the three-dimensionalization is performed by adding a depth in the r-axis direction orthogonal to the background surface 804 instead. FIG. 10B shows a stereoscopic image 1012 that is three-dimensionalized with a depth in the r-axis direction.

  Furthermore, in the present embodiment, the background surface 804 is reduced by (cos θ + 1) / 2 times in the q-axis direction. FIG. 10C shows a stereoscopic image 912 obtained by reducing the background surface. The vertical length of the background surface 904 of the stereoscopic image 912 is reduced to h (1 + 1 / cos θ) / 2. By storing the data of the stereoscopic image 912 based on the r-axis and the q-axis thus obtained in the DRAM 106, the stereoscopic image generation process is completed.

  Next, processing of the background surface of the stereoscopic image 1012 will be described with reference to FIG. In the present embodiment, processing for adding a depth in a direction different from the optical axis at the time of shooting and processing for reducing the background surface in the vertical direction are performed, so that an area where a captured image projected onto the background surface is insufficient is generated. FIG. 11 shows an area where the captured image projected onto the background surface is insufficient.

  FIG. 11A illustrates an area 1101 in which a subject in front of the background surface 804 exists when the background surface 804 is viewed from the z-axis direction with respect to the stereoscopic image 812 temporarily generated by the stereoscopic image generation unit 112. 1102 is shown. There are no captured images to be projected in the areas 1101 and 1102 of the background surface 804.

  FIG. 11B shows the background surface 904 of the stereoscopic image 912 of FIG. 10C finally generated by the stereoscopic image generation unit 112, where the areas 1101 and 1102 occupy the background surface 804 of FIG. Regions are shown as 1111 and 1112. Since the background surface 804 is contracted in the q-axis direction to form the background surface 904, these regions are also contracted in the q-axis direction. There are no captured images to be projected in the areas 1111 and 1112 of the background surface 904.

  FIG. 11C shows a three-dimensional portion of the stereoscopic image 1012 of FIG. 10B temporarily generated by the stereoscopic image generation unit 112 when the background surface 804 is viewed from the z-axis direction. Cross sections 1121 and 1122 cut along a surface 804 are shown. It is not necessary to project a captured image on these cross sections.

  FIG. 11D shows cross sections 1121 and 1122 obtained by cutting the three-dimensional portion with the background surface 904 in the background surface of the stereoscopic image 912 in FIG. 10C finally generated by the stereoscopic image generation unit 112. . The positions and shapes of the cross sections 1121 and 1122 are not changed from (c). Regions 1131 and 1132 shown in (d) are portions that do not overlap the cross sections 1121 and 1122 in the regions 1111 and 1112 of the background surface 904 shown in (b). These areas 1131 and 1132 are areas where a captured image projected onto the background surface is insufficient. For these areas, the stereoscopic image generating unit 112 interpolates based on peripheral pixel areas on the background surface. For example, the average value of the luminance and color of the peripheral pixel region is obtained, and the region is interpolated by drawing a shadow with reduced brightness with respect to the obtained pixel value. Thereby, when the stereoscopic image 512 is viewed from the normal direction of the background surface, the uncomfortable feeling is reduced even in the background surface region where the captured image is insufficient.

  With this configuration, it is possible to reduce a sense of incongruity when a stereoscopic image is viewed from the normal direction of the inclined background surface.

  Note that, in the present embodiment, a captured image in which no subject is present in front of the foreground surface has been described as an example. However, when these images are present, the foreground surface may be configured to change the length in the vertical direction. It may be configured not to change, or may be configured to switch between them. Since the foreground surface is more conspicuous than the background surface, there are cases where the sense of discomfort may be greater depending on the content of the image. However, there may be a case where the sense of incongruity does not increase by filling the shortage area with the average value of the luminance and color of the peripheral area for the defocus amount of the captured image projected on the foreground surface. Therefore, whether or not to change the length may be switched according to the captured image projected on the foreground surface.

  Further, in the present embodiment, for the sake of simplicity, the background and foreground planes have been described as rotating with respect to the x axis, but the rotation may be performed including the y axis. In such a configuration, not only the length of the background surface and the foreground surface in the vertical direction but also the length in the horizontal direction can be changed to reduce the sense of incongruity when viewed from the normal direction of the background surface. .

(Other examples)
The present invention supplies a program that realizes one or more functions of the above-described embodiments to a system or apparatus via a network or a storage medium, and one or more processors in a computer of the system or apparatus read and execute the program This process can be realized. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

  100: stereoscopic image generating apparatus, 101: control unit, 102: internal bus, 103: FLASH memory controller, 104: FLASH memory, 105: DRAM controller, 106: DRAM, 107: operation unit, 108: camera signal processing unit, 109 : Camera, 111: Distance information generation unit, 112: Stereo image generation unit, 113: Memory card controller, 114: Graphic drawing unit, 115: LCD controller, 116: LCD touch panel, 117: Touch panel controller, 150: Memory card, 601 : Feature detection unit, 602: Stereoscopic image output unit, 603: LCD

Claims (15)

Generation means for converting an image obtained by imaging a subject based on distance information about the subject to generate a stereoscopic image;
In the data generated by arranging each pixel of the image in the depth direction based on the distance information, a pixel constituting the subject is composed of a parallel background surface and a foreground surface having a predetermined inclination in the depth direction. Set to pinch,
A stereoscopic image generating apparatus, comprising: generating means for generating the stereoscopic image by projecting, on the background plane, pixels located behind the background plane in the depth direction in the data.   2. The stereoscopic image generation according to claim 1, wherein the generation unit generates the stereoscopic image by projecting, on the foreground plane, a pixel located forward in the depth direction with respect to the foreground plane in the data. apparatus.   When the image includes a plurality of subjects, the generation unit sets the background surface and the foreground surface so that pixels constituting at least one subject are sandwiched between the background surface and the foreground surface. The three-dimensional image generation device according to claim 1, wherein:   4. The stereoscopic image according to claim 1, wherein the generation unit sets the background surface and the foreground surface based on a feature region of the subject detected from the image. 5. Generator.   The feature region has a depth corresponding to the size of the region, and the generation unit includes at least a part of the depth of the feature region included in a space sandwiched between the background surface and the foreground surface. The stereoscopic image generating apparatus according to claim 4, wherein the background plane and the foreground plane are set.   6. The three-dimensional image according to claim 1, wherein the generation unit generates the stereoscopic image so that a depth region connecting a pixel constituting the subject and the background surface is orthogonal to the background surface. The stereoscopic image generating apparatus according to item 1.   The generating means includes a peripheral pixel projected on the background surface of a portion of the background surface that is in contact with the depth region and that does not overlap with a region where the pixel constituting the subject is projected on the background surface. The stereoscopic image generating apparatus according to claim 6, wherein interpolation is performed based on information.   The stereoscopic image generation apparatus according to claim 1, wherein the generation unit reduces the background surface according to the inclination. Imaging means for imaging a subject and generating the image;
Distance information generating means for generating the distance information;
Display means for displaying a preview of the image captured by the imaging means;
The display means displays, when the generating means converts the image into a stereoscopic image, an area that is stereoscopically displayed in the image and an area that is projected onto the background surface are distinguishable. Item 9. The stereoscopic image generation device according to any one of Items 1 to 8.   The stereoscopic image generation apparatus according to claim 1, further comprising an output unit that outputs the stereoscopic image.   The stereoscopic image generation apparatus according to any one of claims 1 to 10, wherein the predetermined inclination is determined based on an installation angle when the output result of the stereoscopic image is installed.   The stereoscopic image generating apparatus according to claim 1, wherein the inclination is an inclination with respect to a plane orthogonal to a depth direction of the stereoscopic image.   The stereoscopic image generating apparatus according to claim 1, wherein the distance between the background surface and the foreground surface is based on a maximum height difference when the stereoscopic image is output. A stereoscopic image generation method for generating a stereoscopic image by converting an image obtained by imaging a subject based on distance information about the subject,
In the data generated by arranging each pixel of the image in the depth direction based on the distance information, a pixel constituting the subject is composed of a parallel background surface and a foreground surface having a predetermined inclination in the depth direction. A step of setting to sandwich,
A method of generating the stereoscopic image by projecting, on the background surface, a pixel located rearward in the depth direction from the background surface in the data.   A program that causes a computer to function as the generation unit according to any one of claims 1 to 13.