JP5929037B2 - Imaging device - Google Patents

Description

  The present invention relates to an imaging apparatus that can capture a stereoscopic image.

  Some imaging methods are known in an imaging apparatus capable of capturing a stereoscopic image. For example, an imaging apparatus that includes two imaging optical systems and generates a stereoscopic image by synthesizing two images captured by the imaging optical systems is known (see, for example, Patent Document 1). Alternatively, although there is only one imaging optical system, the imaging optical system captures two images in which a parallax is artificially created by sliding the imaging optical system, and combines the two images to generate a stereoscopic image. An apparatus is known (see, for example, Patent Document 2).

  A device including two image pickup optical systems, such as the image pickup device of Patent Document 1, has difficulty in downsizing. In addition, a device that slides the imaging optical system, such as the imaging device disclosed in Patent Document 2, requires a special operation (sliding operation of the imaging optical system) that is different from the conventionally known imaging operation. It becomes difficult to operate.

  The present invention has been made in view of the above-described problems, and is an imaging apparatus including a main imaging optical system and a sub imaging optical system, by utilizing image data acquired by the sub imaging optical system. An object of the present invention is to provide an imaging apparatus capable of generating and recording a three-dimensional image from a two-dimensional image captured by a main imaging optical system.

An image pickup apparatus that picks up a main image using an image pickup lens including a zoom lens, the distance map image generating means for calculating a distance to a subject using a plurality of distance measuring optical systems and generating a distance map image; and Depth information image generation that calculates image feature amounts of the main image and the distance map image, and generates a depth information image in which the distance of the distance map image is associated with the main image using the calculated image feature amount When the zoom lens is at a telephoto position, the main image and the distance mapping image are compared, and a complementary image supplemented with the distance map image is generated in an image area not included in the main image. It has a complementary image generation unit, a three-dimensional image generation means for generating a three-dimensional parallax images from the depth information images, the said three dimensional viewing when the main image is captured It generates and records an image, characterized in that.

  According to the present invention, downsizing is easy, and a stereoscopic image can be captured by the same imaging operation as a conventional imaging operation. Further, according to the present invention, a stereoscopic image can be captured even when the main imaging optical system performs telephoto shooting.

1 is an external perspective view showing an embodiment of an imaging apparatus according to the present invention. It is a functional block diagram which shows the example of the said imaging device It is a figure which shows the principle of the distance calculation by the ranging optical system which the said imaging device has. It is a figure which shows the example of the ranging area set to the ranging image sensor which the said imaging device has. It is a figure which shows the example of the distance map which the said imaging process produces | generates. It is a flowchart which shows the example of the imaging process which the said imaging device performs. It is a figure which shows the image which concerns on the production | generation process of a stereo image in the imaging process which the said imaging device performs. It is a figure which shows the image of the production | generation process of the stereo image which concerns at the time of telephotographing in the imaging process which the said imaging device performs. It is a figure which shows the image which concerns on the adjustment process of a stereo image intensity | strength in the imaging process which the said imaging device performs.

  Hereinafter, embodiments of an imaging apparatus according to the present invention will be described with reference to the drawings. FIG. 1 is an external perspective view showing an embodiment of an imaging apparatus according to the present invention. In FIG. 1, an imaging apparatus 1 has a main imaging optical system 10 including an imaging lens and a ranging optical system 20 including a ranging lens on the front surface of the casing, and a release button 30 on the upper surface of the casing. Do it.

  Next, functional blocks of the imaging apparatus 1 will be described with reference to FIG. 2, the imaging apparatus 1 includes a main imaging optical system 10, a distance measuring optical system 20, a two-dimensional image acquisition unit 42, a distance map acquisition unit 43, an image feature amount calculation unit 44, and a three-dimensional information synthesis unit. 45, an image recording unit 46, a RAM 47 that functions as a work area in the processing of each unit, and a CPU 48 that performs control related to the operation of the imaging apparatus 1.

  The main imaging optical system 10 includes an imaging lens 11 and a main imaging element 110. The imaging lens 11 is a lens that forms a subject image on the light receiving surface of the main imaging element 110. The main image sensor 110 has a light receiving surface composed of light receiving elements arranged two-dimensionally, and outputs an image signal (main image signal) according to the subject image formed on the light receiving surface via the imaging lens 11. To do.

  The distance measuring optical system 20 faces the distance measuring lens 22, the distance measuring lens 21 and the distance measuring lens 22 which are arranged with a predetermined interval, the sub imaging element 210 which is arranged opposite to the distance measuring lens 21, and the distance measuring lens 22. And a sub-imaging element 220 to be arranged. The distance measuring lens 21 is a lens that forms a subject image on the light receiving surface of the sub-image sensor 210, and the distance measuring lens 22 is a lens that forms a subject image on the light-receiving surface of the sub image sensor 220. Each of the sub image sensor 210 and the sub image sensor 220 has a light receiving surface composed of light receiving elements arranged two-dimensionally. The sub image sensor 210 and the sub image sensor 220 output image signals (sub image signals) corresponding to the subject image formed on the respective light receiving surfaces. The interval between the sub-imaging elements is a sufficient interval for an image obtained by the sub-image signal to secure a parallax necessary for calculating a distance described later.

  The two-dimensional image acquisition unit 42 includes means for performing signal processing on the main image signal and acquiring a main image that is a two-dimensional subject image (capturing the main image).

  The distance map acquisition unit 43 performs signal processing on the sub-image signal, calculates a distance to the subject, and acquires (generates) a two-dimensional distance map image based on the calculated distance. Have.

  The two sub-imaging elements 210 and 220 of the distance measuring optical system 20 are arranged at predetermined intervals according to the divergence of the optical axes of the distance measuring lens 21 and the distance measuring lens 22. The light receiving surfaces are arranged on the same straight line. As shown in FIG. 3, a subject 81 and a subject 82, which are two subjects having different distances to the light receiving surface of each sub image sensor 210 and the sub image sensor 220, are formed on the light receiving surface of the sub image sensor 210. The positions where images are formed on the light receiving surface of the sub-image sensor 220 are different. The distance map acquisition unit 43 performs triangulation using the difference between the imaging positions of the two sub-imaging elements, and calculates the distance to each subject.

  Here, a method for generating a distance map image from an image obtained via the distance measuring optical system 20 will be described. FIG. 4 is a diagram schematically showing a plurality of distance measuring areas 60 set on the light receiving surface 210a of the sub imaging element 210. As shown in FIG. In FIG. 4, the light receiving surface 210a has 25 distance measuring areas 60 in which the light receiving surface 210a is divided into 5 rows and 5 rows.

  For each distance measuring area 60, the distance is calculated using the triangulation described above. A distance map image is obtained by distributing the distances calculated for each distance measuring area 60 in two dimensions. An example of the distance map image is shown in FIG. The numbers in each distance measurement area 60 shown in FIG. 5 exemplify the reciprocal of the distance calculated for each distance measurement area 60. That is, in FIG. 5, the distance measuring area 60 having a larger number indicates that the distance to the distance measuring object (subject) is closer. By using the reciprocal of the distance, the arithmetic processing can be performed even when the distance cannot be calculated (when the distance is set to “0”).

  Returning to FIG. The image feature amount calculation unit 44 calculates an image feature amount between the main image acquired by the main imaging optical system 10 and the distance map image acquired by the distance measuring optical system 20.

  The three-dimensional information synthesis unit 45 generates a two-dimensional depth information image in which the main image and the distance of the distance map image are associated with each other using each image feature amount calculated by the image feature amount calculation unit 44, and this two-dimensional depth information. A stereoscopic image that is a three-dimensional parallax image is generated from the image.

  The image recording unit 46 is means for recording the stereoscopic image generated by the stereoscopic information combining unit 45. The image recording unit 46 is, for example, an external storage device included in the imaging device 1. Note that the image recording unit 46 also records the main image acquired by the main imaging optical system 10.

  Next, an example of an imaging process executed in the imaging apparatus according to the present invention will be described with reference to the flowchart of FIG. In FIG. 6, each processing step is expressed as S101, S102,. The imaging apparatus 1 always calculates the distance to the subject by the distance measuring optical system 20 during operation. The calculated distance becomes information for focus adjustment of the main imaging optical system 10 (S101). The imaging apparatus 1 repeats the process S101 until the user presses the release button 30 (NO in S102).

  When the release button 30 is pressed by the user (YES in S102), the imaging apparatus 1 causes the two-dimensional image from the subject image formed on the light receiving surface of the main imaging element 110 via the imaging lens 11 of the main imaging optical system 10. (Main image) is acquired (S103). In response to pressing of the release button 30 (YES in S102), a distance map image is generated from the distance measurement result obtained in S101 (S104).

  Subsequently, the image feature amount calculation unit 44 calculates the image feature amount of the main image and the image feature amount of the distance map image (S105). Subsequently, the three-dimensional information synthesis unit 45 associates the main image and the image block of the distance map image with each image feature amount and uses each image block of the distance map image for each image block of the main image. A process for associating the distances (mapping process) is performed to generate a two-dimensional depth information image (S106).

  Subsequently, the imaging device 1 generates a stereoscopic image from the generated two-dimensional depth information image and records it in the image recording unit 46 (S107).

  As described above, the imaging apparatus 1 according to the present embodiment calculates the distance to the subject used for focusing of the imaging optical system with respect to the main image captured by the imaging optical system, even if a plurality of imaging optical systems are not provided. Using the sub-image acquired by the ranging optical system, depth information can be given to the main image, and a stereoscopic image can be generated from the main image associated with the depth information.

  Here, the above-described stereoscopic image generation processing flow executed by the imaging apparatus 1 will be described with reference to the image diagram of FIG. In FIG. 7, reference numeral 30 indicates an example of a main image. Reference numeral 31 indicates an example of a distance map image. The main image 30 is two-dimensional image data that does not include information regarding the depth direction of each subject included in the image. The distance map image 31 generated using the distance to the subject obtained by the sub imaging element 210 and the sub imaging element 220 has information on the distance with respect to the image of each subject included in the main image. It is an image.

  Since the resolution of the main image 30 and the distance map image 31 is different and the resolution of the main image 30 is higher, the image feature amount represented by the SIFT feature of the main image 30 and the distance map image 31 is calculated, and the main image 30 A process (mapping) that associates the image block included in 30 with the image block included in the distance map image 31 and associates information related to the distance included in the distance map image 31 with each image block included in the main image 30. Process) (S106 in FIG. 6).

  By this mapping processing (S106 in FIG. 6), a two-dimensional depth information image 32 in which information (distance) regarding the depth direction is added to the main image which is two-dimensional image data is generated.

  A binocular parallax is calculated for the two-dimensional depth information image 32, and a left-eye image 33 and a right-eye image 34 are generated. That is, the stereoscopic image has a left-eye image 33 and a right-eye image 34. In this embodiment, a binocular parallax image is obtained by using a main image 30 acquired by the main image sensor 110 and a two-dimensional image having depth information of the same size as the main image 30 using geometric transformation such as affine transformation. Convert to (stereo image).

  The imaging apparatus according to the present embodiment can generate the same stereoscopic image as when the zoom lens included in the main imaging optical system 10 is at the telephoto end, even when the zoom lens is at the wide end. FIG. 8 shows an example of the angle of view 50 of the main image related to the main imaging optical system 10 and the angle of view 52 of the distance map image related to the distance measuring optical system 20 when the zoom lens is at the wide end. As shown in FIG. 8A, when the zoom lens is at the wide end, the field angle 50 of the main image and the field angle 52 of the distance map image are the same, and the range of the subject included in each image is the same. It is.

  FIG. 8B shows an example of the field angle 51 of the main image and the field angle 52 of the distance map image when the zoom lens included in the main imaging optical system 10 is at the zoom end (telephoto end). . As shown in FIG. 8B, the field angle 51 of the main image when the zoom lens is at the telephoto end is narrower than the field angle 50 when the zoom lens is at the wide end. Therefore, as shown in FIG. 8B, when the zoom lens is at the telephoto end, the field angle 51 of the main image and the field angle 52 of the distance map image are different, and the field angle 51 is smaller than the field angle 52. Narrow. In other words, when the zoom lens is at the telephoto end, there is a subject 54 that is not included in the main image but is included in the distance map image.

  When mapping the main image and the distance map image, it is necessary to be able to determine the correspondence between the image block included in the main image and the image block included in the distance map image. When the zoom lens is at the telephoto end, the mapping between the main image and the distance map image can be performed in the same manner as when the zoom lens is on the wide angle side by calculating the image feature amount of the main image and the distance map image. . That is, which part in the image is matched can be performed for the image within the angle of view 51, and a two-dimensional depth information image can be generated for the image within the angle of view 51. Therefore, a stereoscopic image can be generated for the image within the angle of view 51. When the user desires a stereoscopic image including the subject 54 that is out of the view angle 51, the subject 54 that is out of the view angle 51 is supplemented by the distance map image.

  The complementary processing is executed by a complementary image generation unit (not shown) to generate a complementary image. When the zoom lens is at the telephoto end, the complementary image generating means compares the main image having the angle of view 51 with the distance mapping image having the angle of view 52, and among the images included in the angle of view 52, An image region corresponding to the corner 51 is specified, and an image that is included in the distance mapping image and protrudes from the image region is arranged around the image in the field of view 51 to generate a complementary image.

  The image feature amount of the generated complementary image is calculated, the image feature amount of the distance map image is calculated, and the two image feature amounts are used to associate the distance of the distance map image with the complementary image 2 A dimension depth information image is generated. The two-dimensional depth information image is generated by calculating the image feature amount of the main image and the distance map image for the image within the angle of view 51, and the distance of the distance map image is simply associated with the complemented portion. You may generate by. A three-dimensional parallax image is generated from the generated two-dimensional depth image to generate a stereoscopic image. Thus, according to the imaging apparatus 2 according to the present embodiment, depth information can be appropriately added to the main image captured with the zoom lens at the telephoto end, and the user desires Can generate a stereoscopic image including a subject 54 that deviates from the angle of view 51.

  In an imaging apparatus that can capture a stereoscopic image using two conventional main imaging optical systems, when the zoom magnification is increased, the angle of view becomes narrower, and the overlapping range of the angles of view of the two imaging optical systems becomes narrower. As a result, the range in which the parallax is generated is narrowed, and in some cases, the range in which the angle of view is overlapped is eliminated, and the range in which the parallax is generated may be eliminated. If the parallax information is small, sufficient distance information to the subject cannot be obtained and a stereoscopic image cannot be acquired. In addition, if the distance between the two optical systems is sufficiently separated and there is a sufficient range of parallax even after zooming, a stereoscopic image can be acquired, but in such a configuration, It is necessary to accurately synchronize the zoom positions of the two main imaging optical systems. In this respect, the imaging apparatus according to the present invention solves the problems of the conventional imaging apparatus, and can generate a stereoscopic image even when performing zoom photography.

  In addition, the imaging apparatus according to the present invention acquires a main image that is two-dimensional image data using the main imaging optical system 10, and adds information related to the distance to the acquired main image. An image is generated. Therefore, in this stereoscopic image generation process, the three-dimensional intensity can be adjusted by adjusting the correlation between the distance information of the image related to the subject and the distance information of the image related to the background.

  FIG. 9 is an image diagram illustrating an example of a three-dimensional intensity adjustment process related to a stereoscopic image generated by the imaging apparatus according to the present embodiment. In FIG. 9, the horizontal axis represents the distance between the subject and the background included in the main image. In FIG. 9, an image block related to the subject is represented by reference numeral 71, and an image block related to the background is represented by reference numeral 72.

  FIG. 9A shows an example of the subject image block 71 and the background image block 72 when the three-dimensional intensity is not changed. In the stereoscopic image, “increasing the three-dimensional intensity” means that the subject image block 71 and the background image block 72 corresponding to the foreground are relatively separated from each other in a normal state. The separation of the subject image block 71 and the background image block 72 means that the difference between the distance related to the subject image block 71 and the distance related to the background image block 72 becomes large. As a result, the image can be deeply felt. That is, as shown in FIG. 9B, when the subject image block 71 and the background image block 72 are separated, the three-dimensional intensity increases.

  On the contrary, as shown in FIG. 9C, when the subject image block 71 and the background image block 72 are brought close to each other, the three-dimensional intensity is reduced. That is, if the difference in distance between the subject image block 71 and the background image block 72 is reduced, the three-dimensional intensity can be reduced.

  In the adjustment process for increasing the three-dimensional intensity, for example, in the distance map image generation process (S104), a process for determining whether or not a predetermined threshold is exceeded with respect to the distance of each image block calculated in the process. Do. Furthermore, a process of adding a predetermined value may be performed for the distance of the image block exceeding the threshold. This series of processing results in a relatively large difference in distance between image blocks, and the three-dimensional intensity can be increased.

  Further, for example, in the distance map image generation process (S104), a determination process as to whether or not a predetermined threshold is exceeded with respect to the distance of each image block calculated in the process, and an image block that does not exceed the threshold For the distance, a process of subtracting a predetermined value may be performed. This series of processing results in a relatively large difference in distance between image blocks, and the three-dimensional intensity can be increased.

  The adjustment process for weakening the three-dimensional intensity may be performed in reverse to the adjustment process for increasing the three-dimensional intensity. For example, in the distance map image generation process (S104), a determination process as to whether or not a predetermined threshold is exceeded with respect to the distance of each image block calculated in the process, and an image block distance that does not exceed the threshold. On the other hand, a process of adding a predetermined value may be performed. This series of processing results in a relatively small difference in distance between image blocks, and the three-dimensional intensity can be weakened.

  Further, for example, in the distance map image generation process (S104), a determination process for determining whether or not the distance of each image block calculated in the process exceeds a predetermined threshold value, and an image block exceeding the threshold value For the distance, a process of subtracting a predetermined value may be performed. This series of processing results in a relatively small difference in distance between image blocks, and the three-dimensional intensity can be weakened.

  As described above, the imaging apparatus according to the present invention can generate a three-dimensional stereoscopic image using the data related to the distance acquired by the ranging optical system and the image data of the subject acquired by the main imaging optical system. it can. Therefore, in the imaging apparatus according to the present invention, it is necessary to provide a plurality of main imaging optical systems, and a three-dimensional image can be obtained without performing a special operation such as sliding the main imaging optical system during shooting processing.

DESCRIPTION OF SYMBOLS 10 Main imaging optical system 20 Distance measuring optical system 42 Two-dimensional image acquisition part 43 Distance map acquisition part 44 Three-dimensional information composition part 45 Image recording part

JP 2010-177921 A JP 2010-239564 A

Claims (4)

An imaging apparatus that captures a main image using an imaging lens including a zoom lens,
Distance map image generation means for calculating a distance to a subject using a plurality of distance measuring optical systems and generating a distance map image;
A depth information image that calculates image feature amounts of the main image and the distance map image, and generates a depth information image that associates the distance of the distance map image with the main image using the calculated image feature amount. Generating means;
Complement image generation for comparing the main image and the distance mapping image when the zoom lens is at a telephoto position, and generating a complementary image supplemented with the distance map image in an image area not included in the main image Means,
Stereoscopic image generation means for generating a three-dimensional parallax image from the depth information image,
Generating and recording the three-dimensional parallax image when the main image is captured;
Imaging device characterized by that   The imaging apparatus according to claim 1, wherein a sub-imaging device included in the distance measuring optical system has a maximum number of pixels smaller than that of an imaging device used for imaging the main image. The distance map image generation means includes
The imaging apparatus according to claim 1, wherein the distance is calculated using triangulation. The imaging lens has a focus lens,
The imaging apparatus according to claim 1, further comprising an autofocus unit configured to move the focus lens to a focus position corresponding to the distance.

Images