JP5751472B2 - Photography equipment - Google Patents

Description

  The present invention relates to a photographing apparatus that enables photographing of a three-dimensional image.

  In recent years, portable devices with a photographing function such as a digital camera have overcome various poor photographing scenes by making full use of image processing. In recent years, there has been a 3D (three-dimensional) wave in the movie industry with a sense of realism, and the 3D display devices are also becoming popular following the television industry.

  In consumer photography equipment such as digital cameras, devices capable of 3D photography have been developed. There are a wide variety of proposals for methods of shooting and recording images including three-dimensional information and reproducing and observing them. In order to enable stereoscopic viewing, it is necessary to capture a right image for the right eye and a left image for the left eye, that is, two images with parallax corresponding to the viewpoints of the left and right eyes.

  For example, a technique for obtaining a stereoscopic image by two horizontal shootings using a 2D shooting lens is disclosed. In addition, an apparatus that obtains a 3D image by simultaneously using two imaging apparatuses, that is, an imaging apparatus that captures a right image and an imaging apparatus that captures a left image, has been commercialized.

Japanese Patent Laid-Open No. 2001-22014

  By the way, the right image and the left image have parallax, and there is a difference between the shooting ranges of the right image and the left image. Only a portion of a common shooting range at the center of the right image and the left image (hereinafter referred to as a common shooting portion) is displayed in a three-dimensional manner. The portions of the shooting range on both the left and right sides of the common shooting portion (hereinafter referred to as a single shooting portion) are included only in one of the right image and the left image, and the common shooting portion excluding the single shooting portion is used as a 3D image. .

  However, in particular, when the distance to the subject is short, most of the common shooting portion is occupied by the short-distance subject, and the left and right background portions of the short-distance subject are not included in the 3D image. In some cases, the perspective relationship between the subject and the subject cannot be expressed. For example, when the distance between the camera and the subject (photographer) is short as in self-portrait, the ratio of the background image decreases, and the image cannot be captured sufficiently, and an appropriate perspective cannot be obtained. In addition to the distance, the manner in which the background enters differs depending on the angle of view that determines the shooting range during shooting.

  For this reason, in order to obtain an appropriate perspective, it is necessary to consider the distance to the subject, and the photographing operation tends to be complicated, and it has been difficult to photograph the eyelids.

  In Patent Document 1, a stereo camera is disclosed in which an appropriate convergence angle is given to the right and left photographing optical axes, and a good stereoscopic image can be taken. However, the invention of Patent Document 1 has a problem that a complicated mechanism is required.

  The present invention provides a photographing apparatus capable of obtaining a 3D image with an appropriate perspective, with a simple configuration, without being aware of distance, angle of view, convergence angle, and parallax during framing. With the goal.

The imaging device of the present invention is a distance between an imaging unit that obtains a plurality of images with different parallaxes by one or more imagings, and a left and right background portion of the subject at the center of the plurality of images obtained by the imaging unit It is determined for each background part whether or not the change in the background is smaller than a predetermined threshold, and the background determination unit that determines that the background part determined to be small can be diverted, and can be diverted based on the determination result of the background determination unit And a combining unit that combines the determined background portion with the central image.

  According to the present invention, there is an effect that it is possible to acquire a 3D image with a simple configuration and being able to enjoy an appropriate perspective with easy shooting without being aware of various factors during framing.

1 is a block diagram illustrating a circuit configuration of a photographing apparatus according to an embodiment of the present invention. Explanatory drawing for demonstrating the 3D imaging | photography range obtained in 3D imaging | photography. Explanatory drawing for demonstrating the production | generation method of the general 3D image by a left-right image. Explanatory drawing for demonstrating the background image added to the right and left of a to-be-photographed object. Explanatory drawing which shows an example of the determination method of whether a background part is diverted to an image missing part. Explanatory drawing which shows an example of the determination method of whether a background part is diverted to an image missing part. FIG. 7 is a flowchart illustrating an operation flow when the imaging unit 2 includes two imaging elements 2a. The flowchart for determining whether a background part can be diverted to an image missing part. Explanatory drawing for demonstrating the flow of FIG. Explanatory drawing for demonstrating the flow of FIG. FIG. 11 is a flowchart showing an operation flow when the imaging unit 2 has only one image sensor 2a.

  Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.

  FIG. 1 is a block diagram showing a circuit configuration of a photographing apparatus according to an embodiment of the present invention.

  The photographing apparatus 1 has an imaging unit 2 configured by an imaging element 2a such as a CCD or a CMOS sensor. In the present embodiment, an example is shown in which an imaging unit for 3D imaging having a right imaging unit 2R and a left imaging unit 2L constituting a right and left optical system is employed as the imaging unit 2. Note that, as the imaging unit 2, a photographing lens for 2D photographing having only one optical system on the assumption that left and right images are obtained by photographing twice may be employed.

  An optical image OA of a subject from an optical system including a photographic lens 2b provided on the front surface of the photographic device 1 is formed on the imaging surface of each imaging device 2a constituting the left and right imaging units 2L and 2R. The imaging unit 2 is driven and controlled by the signal processing and control unit 11. The signal processing and control unit 11 outputs a drive signal for the image sensor 2 a to the image capturing unit 2 and captures an image signal obtained by photoelectrically converting the optical image by the image sensor 2 a.

  The signal processing and control unit 11 performs predetermined signal processing, such as color signal generation processing, matrix conversion processing, and other various digital processing, on the image signal obtained by photoelectric conversion of the image sensor 2a. The signal processing and control unit 11 can output image information, audio information, and the like compressed by performing an encoding process when recording an image signal, an audio signal, and the like.

  The photographing device 1 is also provided with an operation determination unit 3, an operation unit 4, a clock unit 5, and a touch panel 6. The clock unit 5 generates time information used by the signal processing and control unit 11. The operation determination unit 3 generates an operation signal based on a user operation on the operation unit 4 and outputs the operation signal to the signal processing and control unit 11. The operation unit 4 is configured by various switches (not shown) such as a shooting start / end button and shooting mode setting provided on the shooting device 1. The touch panel 6 generates an operation signal based on a user's touch operation and outputs the operation signal to the signal processing and control unit 11. The signal processing and control unit 11 controls each unit based on the operation signal.

  In addition, the photographing apparatus 1 is provided with a recording / playback control unit 11a and a display control unit 11e. The recording / playback control unit 11a can provide the recording unit 8 with the image information and audio information from the signal processing and control unit 11 for recording. The recording unit 8 is controlled by the recording / reproduction control unit 11a to perform recording and reproduction. As the recording unit 8, for example, a card interface can be adopted, and the recording unit 8 can record image information, audio information, and the like on a recording medium such as a memory card. The recording unit 8 can read out the image information and the audio information recorded on the recording medium and supply them to the recording / reproduction control unit 11 a of the signal processing and control unit 11. The signal processing and control unit 11 can decode the image information and audio information from the recording unit 8 reproduced by the recording / reproduction control unit 11a to obtain an image signal and audio signal.

  The display control unit 11e gives a captured image from the imaging unit 2 and a reproduction image from the recording / reproduction control unit 11a to the display unit 7 to display these images. In addition, the display control unit 11e can display a menu display for operating the photographing apparatus 1 on the display unit 7.

  The display unit 7 is provided on the back surface of the main body of the photographing apparatus 1 so that the photographer can visually recognize the display screen of the display unit 7 at the time of photographing. The display unit 7 is a display unit for 3D display having a left-eye display region 7L and a right-eye display region 7R for displaying a left image and a right image, which are parallax images, respectively.

  As the display unit 7, a display unit that employs a parallax division method such as a parallax barrier method or a lenticular method capable of 3D display can be used. In the parallax division method, 3D display is possible without using glasses for 3D display. Further, as the display unit 7, a display unit that enables 3D observation using 3D display glasses or the like may be employed, and further, a display unit for 2D display may be employed.

  When the right image and the left image that are 3D images recorded in the recording unit 8 are reproduced, the display control unit 11e displays the right image and the left image corresponding to the display area 7R for the right eye and the left eye corresponding to the display unit 7. Display control is performed so as to display in the display area 7L. Thereby, the captured 3D image can be stereoscopically displayed on the display unit 7, and the photographer can confirm the stereoscopic shooting effect on the display screen of the display unit 7. The display control unit 11e can perform 2D display by giving the display unit 7 the same image as the left and right images.

  The touch panel 6 generates an operation signal corresponding to the position pointed by the user with a finger. The touch panel 6 can be provided on the display screen of the display unit 7, for example. In this case, the user can instruct various command buttons displayed on the screen of the display unit 7 using the touch panel 6. The touch panel 6 supplies an operation signal based on a user operation to the signal processing and control unit 11.

  In the present embodiment, in order to easily obtain a 3D image with an appropriate perspective without being aware of framing at the time of shooting with a simple configuration, the signal processing and control unit 11 has a main subject determination. A unit 11b, a background determination unit 11c, a distance determination unit 11d, a common shooting range determination unit 11f, an auxiliary image generation unit 11g, and a synthesis unit 11h are provided.

  Each unit in the signal processing and control unit 11 may be configured by a software program executed by the CPU, or may be configured by a hardware circuit. Further, the signal processing and control unit 11 may be configured by a digital signal processor (DSP).

  Next, signal processing and camera control by the control unit 11 will be described with reference to FIGS. FIG. 2 is an explanatory diagram for explaining a 3D imaging range obtained in 3D imaging.

  The two imaging units 2R and 2L of the imaging device 1 are arranged apart from each other by the baseline length B in the left-right direction. In FIG. 2, in order to make each constituent element recognizable on the drawing, the size of the photographing apparatus 1 and the constituent elements therein, the base line length B, and the like are different in scale for each constituent element. is there. In addition, it is assumed that the imaging direction of the imaging unit is not changed and there is no convergence angle.

  When the angle of view of each imaging unit 2R, 2L is θ and the distance from the imaging device 1 to the subject S is L, the imaging range X1 of each imaging unit is a value calculated by the following equation (1). . In the following formula, * indicates multiplication.

X1 = 2 * (L * tan (θ / 2)) (1)
In addition, since the 3D imaging range ED is an overlap region where two images captured by the two imaging units 2R and 2L overlap each other, the 3D imaging range ED is a value calculated by the following equation (2).

ED = X1-B Formula (2)
For example, the angle of view θ that can be approximated to X1 = L * tan θ (e.g., equivalent to a lens of approximately 50 mm in 35 mm format), the base line length B is 5 cm, and the distance L is the distance when taking a selfie. For example, in the case of 50 cm (= l1), the 3D imaging range ED by the two imaging units 2R and 2L is as follows.

ED = X1-B = l1 * tan [theta] -B = 50-5 = 45 (3)
That is, the 3D imaging range ED is only 45 cm in the lateral direction. In this case, if a person has a shoulder width of 45 cm, his / her 3D shooting cannot be performed properly unless the face of the subject S is positioned at the approximate center of the screen. Furthermore, in this case, a person is arranged in substantially the entire left and right direction of the shooting range, there is no area where the background is arranged, and it is difficult to obtain an appropriate stereoscopic effect.

  On the other hand, when the distance L between the photographing device 1 and the subject S is L2 which is larger than l1, the 3D photographing range ED becomes wider as compared with the case where the distance l1 as shown in FIG. That is, as the distance L between the imaging device 1 and the subject S increases, the 3D imaging range ED also increases.

  FIG. 3 is an explanatory diagram for explaining a general 3D image generation method using left and right images.

  FIG. 3A shows an example in which an imaging unit for 2D imaging is adopted as the imaging unit 2, and a right image and a left image are obtained by performing imaging twice at different times while moving the imaging unit 2 in the horizontal direction. Is shown. FIG. 3B is a diagram for explaining an example of an image obtained by the two imaging operations shown in FIG. 3A, and the broken line indicates the imaging range 25.

  FIG. 3B shows a state in which a person 21 stands in a room 23 surrounded by a ceiling 23u, a floor 23d, left and right walls 23l and 23r, and a back wall 23o, and a frame 22 is displayed on the back wall 23o. Show. In FIG. 3A, for simplification of the drawing, the person 21 is represented by a square subject 21 and the frame 22 is represented by a round subject 22.

  As shown in FIG. 3A, the imaging unit 2 includes a photographic lens 2b and an imaging element 2a. From the imaging unit 2, a person 21 (hereinafter also referred to as a subject 21), a frame 22 (hereinafter also referred to as a subject 22). Are distances of l1 and l2, respectively.

  Now, in FIG. 3A, it is assumed that the imaging unit 2 images the subjects 21 and 22 at the left position L1. FIG. 3C1 shows a captured image (left image) in this case, and shows that an image 23L corresponding to the subject 21 and an image 24L corresponding to the subject 22 are captured. Next, it is assumed that the imaging unit 2 is moved to the right position R1 and a right image is captured with a baseline length x1. A captured image (right image) when the same subjects 21 and 22 are captured at the right position R1 is shown in FIG. In this case, an image 23R corresponding to the subject 21 and an image 24R corresponding to the subject 22 are captured.

  As shown in FIG. 3C1, the right image wall 23r of the left and right walls is not included in the imaging range 25L of the left image, and only a part of the left wall 23l is photographed. On the contrary, as shown in FIG. 3 (c2), in the imaging range 25R of the right image, the left wall 23l is not included in the left and right walls, and only a part of the right wall 23r is photographed. Yes.

  The left and right images shown in FIGS. 3 (c1) and 3 (c2) are overlaid so that the image portions of the long-distance subject 22 coincide. FIG. 3D shows a captured image in this case. The stereoscopic effect of the person 21 with respect to the frame 22 is obtained according to the parallax of the images 23R and 23L of the subject 21. Note that a subject at a short distance is more susceptible to parallax than a background at a long distance, and the background is far away and a wide range is reflected, so the influence of parallax is small.

  As shown in FIG. 3D, the left and right image shooting ranges 25L and 25R include a range (single shooting portion) that exists only in one image. Therefore, strictly speaking, the 3D image generated is only the common range (common shooting portion) 26 of the shooting ranges 25L and 25R, as shown in FIG.

  As is clear from the above description, the size of the common photographing portion changes according to the distance to the subject, the photographing angle of view, the convergence angle at the time of photographing, the parallax, and the like. Accordingly, when such a 3D image is displayed over the entire display screen of the display unit 7, a correct aspect ratio cannot be obtained. Also, if the aspect ratio is displayed correctly, an image with the top and bottom and the left and right of the screen missing is displayed. For example, when the image of FIG. 3E is displayed on the screen, the image of FIG. 3F may be displayed. FIG. 3F shows a state in which an image missing portion 27 where no image is displayed on the left and right of the screen is generated.

  In particular, when the distance between the photographing device 1 and the subject is small, as in the case of so-called self-photographing, where the user takes the photographing device 1 by himself, the portion of the background image on the left and right sides of the subject tends to be lost. It is difficult to obtain an appropriate stereoscopic effect.

  Therefore, in the present embodiment, when the distance to the subject is relatively short, the signal processing and control unit 11 can easily obtain an appropriate stereoscopic effect by adding a background image to the left and right of the subject. A captured image is obtained.

  FIG. 4 is an explanatory diagram for explaining background images added to the left and right of the subject. FIG. 4A shows a state in which a subject corresponding to FIG. 4B and 4C show the distance distribution at each point on the horizontal line in the imaging range, with the horizontal position in the imaging range on the horizontal axis and (1 / distance (L)) on the vertical axis. Is. In FIGS. 4B and 4C, infinity is represented as zero.

  FIG. 4B shows the distance from the photographing device 1 of FIG. 4A to each point on the horizontal line (two-dot chain line) including the subject 21. As shown in FIG. 4B, the distance to the subject 21 is the closest among the subjects in the imaging range, and the distance to the background portion of the wall 23o is relatively large. The distance between the left and right background portions of the subject 21 is almost the same at any position. That is, it is not necessary to give a stereoscopic effect to such a background portion.

  Therefore, in the present embodiment, by using such a background portion having a sufficiently small distance difference as an image missing portion, the image size is made constant regardless of the distance to the subject. Although the diverted portion does not appear three-dimensionally, it can be seen that it is the background, and there is no problem in expressing the feeling of the main subject popping out. However, such a measure is not appropriate when there is a distance change in the background as shown in FIG.

  The distance determination unit 11d of the signal processing and control unit 11 determines the distance from the imaging device 1 in each part of the imaging range. The determination result of the distance determination unit 11d is given to the main subject determination unit 11b and the background determination unit 11c. The main subject determination unit 11b determines the main subject in the imaging range based on the determination result of the distance determination unit 11d. For example, the main subject determination unit 11b determines an object having the smallest distance from the photographing device 1 as the main subject.

  Moreover, the background determination part 11c determines the background part in an imaging range based on the determination result of the distance determination part 11d. For example, the background determination unit 11c determines a subject that is a surrounding subject in the imaging range and has a relatively large distance from the photographing device 1 as a background portion. Further, the background determination unit 11c constituting the 3D effect determination unit determines whether or not to divert the background part to the image missing part based on whether or not the change in the distance from the imaging device 1 is small in the determined background part. judge.

  5 and 6 are explanatory diagrams showing an example of a method for determining whether or not the background portion is diverted to the image missing portion.

  The background determination unit 11c has a small change in the distance from the photographing device 1 in the background portion when the distance between the both ends of the background portion between the main subject and the imaging range end portion is substantially the same. It is determined that the image can be diverted to the missing part. Circles in FIG. 5 indicate positions used for determination.

  That is, the background determination unit 11c calculates the difference in distance between the circled portions in FIG. 5 for the background portion on the left side of the main subject. When the obtained difference is smaller than the predetermined threshold, the background determination unit 11c determines that the change in the distance from the imaging device 1 is small in the background portion and can be used for the image missing portion. The background determination unit 11c can similarly determine whether or not the background portion on the right side of the main subject can be used.

  FIG. 5A shows an example in which it is determined that the change in the distance from the imaging device 1 is small in the background portion and that it can be used for the image missing portion. FIG. 5B shows an example in which it is determined that the background portion on the left side of the main subject can be diverted to the image missing portion, but the background portion on the right side of the main subject is from the photographing device 1 in the background portion. In this example, it is determined that the change in the distance is relatively large and cannot be used for the missing image portion.

  FIG. 6 shows an example of a distance determination method. FIG. 6 shows an example of a method in the case where the imaging unit 2 includes two imaging elements 2a, and the distance to the subject is the same depending on the distance between the imaging positions on the imaging element 2a of the three subjects 30 to 32. Whether or not. The subjects 30 and 31 have the same distance from the photographing device 1, and the distance between the images of the subjects 30 and 31 formed on the left and right imaging elements 2a is the same distance V1 in the left and right imaging elements 2a. For the subjects 31 and 32 having different distances from the photographing device 1, the distance between the images of the subjects 31 and 32 formed on the left and right imaging elements 2a is different in the left and right imaging elements 2a, and V2 and V3, respectively. It is.

  The background determination unit 11c determines whether or not the distances from the photographing device 1 to the subject at the two positions are substantially the same by comparing the distance between the images of each subject between the left and right imaging elements 2a. Can do.

  The common shooting range determination unit 11f constituting the 3D effect determination unit determines the common shooting range of the left and right images. The common shooting range determination unit 11f obtains the size of the missing image portion based on the determination result of the common shooting range, and gives it to the auxiliary image generation unit 11g. If the size of the missing image portion is within a predetermined threshold, the common shooting range determination unit 11f determines that the generation of the auxiliary image is unnecessary, and the size of the missing image portion is 0. May be output.

  For example, the common shooting range determination unit 11f may determine that the generation of the auxiliary image is unnecessary when the main subject ratio is smaller than 1/3 of the screen size. For example, the common shooting range determination unit 11f determines the main subject by face detection or the like, and if the ratio occupied by the main subject is 1/3 or less, the background is reflected in the remaining 2/3, and there is a possibility that the perspective can be fully enjoyed. Do not perform background composition. For example, the subject at a distance l1 in FIG. 2 is likely to have a large face and no background information, but the subject at a distance l2 is sufficiently reflected in the background, and perspective information can be obtained. Note that the size of the determination threshold is set to 1/3 of the screen size, of course, depending on the parallax, the convergence angle, etc. When the parallax is large, there is no difference between the left and right images, and the effect of synthesis is also reduced. Therefore, for example, it may be ½ of the screen size.

  The auxiliary image generation unit 11g generates an auxiliary image based on the background portion determined to be divertable by the background determination unit 11c to the image missing part. The auxiliary image generation unit 11g obtains the size of the auxiliary image to be diverted from the common shooting range determination unit 11f to the image missing portion, and cuts out the necessary size from the left and right background portions of the main subject to generate an auxiliary image. The auxiliary image generation unit 11g may generate an auxiliary image using one of the left and right background portions of the main subject.

  The combining unit 11h combines the image portion of the main subject in the common shooting range and the auxiliary image. Thereby, the 3D image from the synthesis unit 11h has the same size regardless of the distance to the subject. In addition, even when the distance to the main subject is short, the background is added to the left and right of the central main subject, so that a three-dimensional image can be obtained.

  Note that the common imaging range determination unit 11f determines whether or not to synthesize a background image corresponding to the image missing portion depending on whether or not the size of the image missing portion is within a predetermined threshold. The size of the image missing part is related to the parallax of the left and right images, that is, the stereoscopic effect. That is, the common shooting range determination unit 11f determines whether or not the image should have a sufficient stereoscopic effect, that is, whether or not the background image corresponding to the missing image portion is synthesized by determining the 3D effect. ing. Therefore, whether or not to synthesize a background image corresponding to an image missing portion may be determined by another circuit that performs 3D effect determination instead of the common imaging range determination unit 11f.

  Next, the operation of the embodiment configured as described above will be described with reference to FIGS. FIG. 7 is a flowchart showing an operation flow in the case of having two image sensors 2a as the imaging unit 2, and FIG. 8 is a flowchart for determining whether or not the background portion can be diverted to the image missing portion. FIG. 11 is a flowchart showing an operation flow when the imaging unit 2 has only one image sensor 2a.

  When the photographing apparatus 1 is turned on, the signal processing and control unit 11 determines whether or not the photographing mode is in step S11 of FIG. When the shooting mode is not set, the signal processing and control unit 11 determines whether or not the playback mode is set in step S12.

  If the signal processing and control unit 11 determines that the reproduction mode is instructed in step S12, the process proceeds from step S12 to step S13, and reads the information of the list of files recorded by the recording unit 8, A list display is displayed on the display unit 7.

  When the user selects a file at the time of displaying the file list (step S14), the signal processing and control unit 11 reads out the selected file via the recording / playback control unit 11a, performs a decoding process, and outputs an image signal. Then, the audio signal is reproduced (step S15). The signal processing and control unit 11 gives the reproduced image signal and audio signal to the display unit 7 for display.

  If an end operation is performed when the file list is displayed, the signal processing and control unit 11 moves the process from step S14 to step S16 and ends the playback mode.

  If the signal processing and control unit 11 determines in step S11 that the shooting mode is instructed, it determines in step S21 whether or not the 3D mode is instructed. If not in the 3D mode, the signal processing and control unit 11 displays a through image on the display screen of the display unit 7 in step S22. That is, the signal processing and control unit 11 takes a captured image from the imaging unit 2, performs predetermined signal processing, and then outputs the captured image to the display unit 7 by the display control unit 11e. In this way, a through image is displayed on the display screen of the display unit 7.

  In step S23, the signal processing and control unit 11 determines whether or not a photographing operation has been performed. Here, when the photographer presses the release button, the process proceeds from step S23 to step S24, and photographing is performed. In step S <b> 25, the signal processing and control unit 11 performs predetermined signal processing on the captured image from the imaging unit 2, encodes it into an image file, and then provides the recording unit 8 for recording.

  If the signal processing and control unit 11 determines in step S21 that the photographer has instructed the 3D mode, the signal processing and control unit 11 acquires left and right images in step S31. The next steps S32 to S39 show processing for synthesizing a background image as an auxiliary image to an image in the common shooting range. First, in step S32, it is determined whether a background image needs to be added.

  The common shooting range determination unit 11f determines the common shooting range of the left and right images. The common shooting range determination unit 11f obtains the size of the image missing portion based on the determination result of the common shooting range. For example, if the size of the missing image portion is within a predetermined threshold, the common shooting range determination unit 11f determines that the generation of the auxiliary image is unnecessary, and the size of the missing image portion is 0. The information indicating is output. In this case, the processes in steps S32 to S39 are skipped. For example, the common shooting range determination unit 11f determines that the generation of the auxiliary image is unnecessary when the main subject ratio is smaller than 1/3 of the screen.

  When it is determined by the common shooting range determination unit 11f that a background image needs to be added, the distance determination unit 11d measures the distance to the subject, and the distance measurement result is used as the main subject determination unit 11b and the background determination unit. To 11c. The main subject determination unit 11b determines the main subject based on the distance measurement result (step S33), and the background determination unit 11c determines the background portion based on the distance measurement result (step S34).

  Next, in steps S35 and S36, the background determination unit 11c determines whether or not the background portion can be used for the image missing portion. In the example of FIG. 7, the left background of the left image is first determined. In the description of FIG. 5, an example has been described in which it is determined whether or not the background portion can be diverted to the image missing portion based on whether or not the distance change between the left and right background portions of the main subject is relatively small.

  Furthermore, in the present embodiment, an example will be described in which it is determined whether or not the background portion can be diverted to the image missing portion according to the flow of FIG. 8 in addition to the determination method of FIG. 9 and 10 are explanatory diagrams for explaining the flow of FIG. Note that it may be determined whether the background portion can be diverted to the image missing portion only by using the flow of FIG. 8 without adopting the determination method of FIG.

  The background determination unit 11c aligns the left and right images in step S51 of FIG. In this case, the background image is matched. FIG. 9 shows a method for determining an image portion to be matched. FIG. 9 shows the distribution of distances to the subject on the horizontal line, with the horizontal axis representing the position on a predetermined horizontal line within the imaging range and the vertical axis representing 1 / L (distance). In FIG. 9, the solid line indicates the right image, and the broken line indicates the left image. In the right image shown by the solid line, the imaging part extends to the right side of the shooting range compared to the left image, and in the left image shown by the broken line, the imaging part extends to the left side of the shooting range compared to the right image. The center in the left-right direction is the subject existing portion, and the left and right are the background portions.

In the background determination unit 11c, a portion where the distance between the right image and the left image changes in the background portion is likely to be a background portion of the same pattern. Therefore, the background determination unit 11c aligns the right image and the left image so that the portions where the distance of the background portion similarly changes in the right image and the left image match.
Next, the background determination unit 11c corrects the missing image portion of one of the left and right images by using the other image. FIG. 10A shows the imaging positions of the subjects 41 to 43, and FIG. 10B shows the distance distribution of each of the subjects 41 to 43 in FIG. In FIG. 10, the solid line indicates the right image, and the broken line indicates the left image.

  As shown in FIG. 10B, an image missing portion of the right image exists on the left end side of the imaging range, and an image missing portion of the left image exists on the right end portion side of the imaging range. In order to correct the image missing portion of one image of the left and right images with the background image at the image end portion of the other image, an image corresponding to the image missing portion of one image is present at the image end portion of the other image. It is necessary to exist. In the present embodiment, the change in the distance between both ends of the background portion is relatively small, and the pixel value of the image missing portion of one image is substantially equal to the pixel value of the image end portion of the other image, whereby the other Are determined to be divertable.

  That is, the background determination unit 11c determines whether the change in the distance of the left background portion of the main subject is relatively small for the left image, and then the pixel value of the left image at the same position as the left end of the right image. Is obtained (step S52). When the pixel value at the left end of the right image is substantially the same as the pixel value at the corresponding position in the left image, an image corresponding to an image obtained by extending the left end of the right image is present at the left end of the left image. It can be estimated. In step S53, the background determination unit 11c compares the pixels at corresponding pixel positions, and determines whether or not the difference between the pixel values is within a predetermined threshold (step S54).

  The background determination unit 11c has a relatively small change in the distance of the left background portion of the main subject in the left image, and the difference between the left end pixel of the right image and the pixel value of the left image at the same position of the pixel is a predetermined threshold value. If it is within the range, it is determined that the left background portion of the left image can be diverted to the left image missing portion of the right image, and the process proceeds to step S37.

  The common shooting range determination unit 11f outputs the size of the auxiliary image corresponding to the missing image portion to the auxiliary image generation unit 11g, and the auxiliary image generation unit 11g starts the common shooting range determination unit from the left background portion of the left image. The auxiliary image is generated by cutting out the size based on the output of 11f. This auxiliary image is given to the combining unit 11h, and the combining unit 11h combines the auxiliary image with the left side of the right image (step S37).

  If the background determination unit 11c determines in step S35 that the distance change of the left background portion of the left image is relatively large, or if the determination in step S54 is NO, in step S36, It is determined whether the right background part of the right image can be used as the right image of the left image. That is, the background determination unit 11c first determines whether or not the change in distance in the right background portion of the right image is relatively small. When the distance change is small, the background determination unit 11c repeats the processes of steps S51 to S54 in FIG.

  That is, the background determination unit 11c obtains the pixel value of the right image at the same position as the right end of the left image after aligning the left and right images in step S51 (step S52). Next, in step S53, the background determination unit 11c compares the pixels at corresponding pixel positions, and determines whether or not the pixel value difference is within a predetermined threshold (step S54).

  The background determination unit 11c has a relatively small change in the distance of the right background portion of the main subject in the right image, and the difference between the pixel at the right end of the left image and the pixel value of the right image at the same position of the pixel is a predetermined threshold value. If it is within the range, it is determined that the right background portion of the right image can be diverted to the right image missing portion of the left image, and the process proceeds to step S38.

  The auxiliary image generation unit 11g generates an auxiliary image by cutting out the size based on the output of the common imaging range determination unit 11f from the right background portion of the right image. This auxiliary image is given to the synthesizing unit 11h, and the synthesizing unit 11h synthesizes the auxiliary image on the right side of the left image (step S38).

  In steps S35 and 36, if the background determination unit 11c determines that the distance change between the left and right background portions of the left and right images is relatively large, or if the determination in step S54 is NO, the auxiliary image is synthesized. Not done. In this case, a warning display indicating that an image is missing is displayed in step S39.

  In addition, when 2D imaging using only one image sensor 2a is used as the imaging unit 2, the flow of FIG. 11 is performed. In FIG. 11, the same steps as those in FIG. In FIG. 11, when the 3D mode is instructed, the signal processing and control unit 11 determines whether or not a photographing operation has been performed in step S45. When the release button is pressed by the photographer, the signal processing and control unit 11 performs continuous shooting (step S46). The photographer operates the release button while moving the photographing device 1 horizontally, for example. Thereby, the right image and the left image can be acquired by the signal processing and the continuous shooting of the control unit 11.

  The auxiliary image generation process in steps S31 to S39 is the same as that in FIG. In FIG. 11, when an auxiliary image is generated, in step S47, the signal processing and control unit 11 synthesizes the auxiliary image with an image in the common shooting range and performs 3D display. The user confirms the 3D display, and performs a recording operation when determining that the desired 3D shooting has been performed. As a result, the process proceeds from step S48 to step S25, and the synthesized image from the synthesis unit 11h is recorded in the recording unit 8 by the recording / playback control unit 11a.

  As described above, in the present embodiment, the background portion of the subject is cut out to generate an auxiliary image, and the auxiliary image is combined with the image in the common shooting range to obtain a 3D image, and the distance to the subject is relatively small. Even when there are few areas where the background appears on the screen, such as when the subject is close, it is possible to add background images to the left and right of the subject, emphasizing that the subject subject is close, and obtaining an appropriate perspective. Can do. Further, whether or not the background portion is diverted to the image missing portion is determined depending on whether or not the distance change of the background portion as the auxiliary image is relatively small, and the 3D effect and naturalness are deteriorated by diversion. Can be prevented.

  In the above embodiment, the example in which the background image at one end of the other image is combined with the image missing portion at one end of one of the left and right images has been described. It is also possible to combine both background images of the other image in the image missing portion.

    DESCRIPTION OF SYMBOLS 1 ... Imaging device, 2 ... Imaging part, 3 ... Operation determination part, 7 ... Display part, 8 ... Recording part, 11 ... Signal processing and control part, 11a ... Recording / reproduction control part, 11b ... Main subject determination part, 11c ... Background determination unit, 11d ... distance determination unit, 11e ... display control unit, 11f ... common shooting range determination unit, 11g ... auxiliary image generation unit, 11h ... synthesis unit.

Claims (3)

An imaging unit that obtains a plurality of images with different parallaxes by one or more imagings;
It is determined for each background part whether or not the change in the distance between the left and right background parts of the subject in the central part of the plurality of images obtained by the imaging unit is smaller than a predetermined threshold, and the background determined to be small A background determination unit that determines that the portion can be diverted;
An imaging apparatus, comprising: a combining unit that combines the background portion determined to be divertable based on the determination result of the background determination unit with the image in the center portion. Determining a common photographing range of the plurality of images, determine the size of the image missing component based on the common shooting range of the determination result, the size of the image missing portion in the case within a predetermined threshold, the combining unit The photographing apparatus according to claim 1, further comprising a common photographing range determination unit configured to determine whether or not to combine. The background determination unit further determines whether or not the background portion can be diverted based on a comparison between pixels at corresponding pixel positions in the background portions in the plurality of images.
The photographing apparatus according to claim 1, wherein the photographing apparatus is characterized.

Images