WO2011129387A1

WO2011129387A1 - Image capture device

Info

Publication number: WO2011129387A1
Application number: PCT/JP2011/059243
Authority: WO
Inventors: 大森圭祐; 若林保孝
Original assignee: シャープ株式会社
Priority date: 2010-04-14
Filing date: 2011-04-14
Publication date: 2011-10-20
Also published as: JP2011239372A

Abstract

In an image capture device that generates a high-resolution image by employing a plurality of image capture assemblies to composite a capture image, memory usage increases when the process is carried out in hardware. Disclosed is an image capture device comprising at least three image capture assemblies, further comprising an optical assembly and an image capture element. The image capture assemblies are positioned in the pixel array direction of the image capture elements. A photographic subject, which is within an image capture distance of a prescribed range, is captured such that a sampling phase in the array direction is shifted by 0.2-0.8 pixels, relative to an image captured by a first image capture assembly, by any of the other image capture assemblies.

Description

Imaging device

The present invention relates to an imaging apparatus using a plurality of imaging systems.

In recent years, imaging devices such as digital still cameras and digital video cameras have increased in the number of pixels and the resolution has been increased, and the optical system such as a lens has become large in order to obtain sufficient optical performance. There is a problem of becoming.

As a method of improving the resolution of the image without changing the size of the optical system and the image sensor, the relative positional relationship between the optical system and the image sensor is shifted in time, or the light is separated into a plurality using a prism, A plurality of pieces of image information in which the positional relationship between the subject image and the pixels of the image pickup device is shifted from each other is acquired by making the light incident on a plurality of image pickup devices, and the high resolution is obtained by combining these pieces of image information. A technique called “pixel shifting” for acquiring an image is known. For example, Patent Document 1 includes a displacement unit that displaces the image sensor as illustrated in FIG. 21, and displaces the relative positional relationship between the subject image and the image sensor in an oblique direction with respect to the arrangement direction of the pixels of the image sensor. Thus, a technique for improving the resolution in the horizontal direction and the vertical direction has been proposed. Here, the horizontal direction and the vertical direction mean pixel arrangement directions as shown in FIG.

Also, a pixel shifting technique has been proposed for a compound eye system using a plurality of imaging systems. Patent Document 2 proposes a technique for improving the resolution by arranging a plurality of imaging systems so as to realize pixel shifting in a direction perpendicular to the pixel arrangement direction.

In order to deepen the understanding of the pixel shifting technique in the compound eye system, the pixel shifting will be described with reference to FIGS.

FIG. 23 shows a situation where a common subject 2301 is imaged by the two

imaging systems

2302 and 2303. FIG. 24 shows a region to be imaged by the imaging system 2302 by a grid 2401 and a region to be imaged by the imaging system 2303 by a grid 2402 (broken line), and each of the smallest squares is imaged by one pixel. Indicates the area. In FIG. 24, the areas imaged by the pixels of the two imaging systems are almost overlapped, and the sampling phases are aligned. On the other hand, in FIG. 25, the area captured by each pixel of the imaging elements of the two imaging systems is shifted by 3/2 pixels in the horizontal direction and 1/2 pixel in the vertical direction, and the sampling phase is shifted. The sampling phase here means the relative positional relationship between the image forming point on the image sensor at a certain point of the subject and the pixel in a plurality of imaging systems. For example, in FIG. 25, images are taken with a 3/2 pixel shift in the horizontal direction, and the horizontal sampling phase shift is 1/2 pixel as a decimal part. Pixel shift means a state in which the sampling phase is shifted.

Next, a change in sampling phase when the imaging distance changes in the compound eye method will be described with reference to FIG. In FIG. 26, an imaging system 2601 includes an optical system 2602 and an imaging element 2603, and an imaging system 2605 includes an optical system 2606 and an imaging element 2607. The optical system 2602 and the optical system 2606 are arranged on the same plane, and their optical axes are perpendicular to the plane, and FIG. 26 is a view of the state from above. In FIG. 26, the subject is located at a point P0 or P1 on the optical axis of the optical system 2602. The subject image of the subject located at the point P0 is formed at the center of the pixel 2604 of the image sensor 2603 and is formed at the center of the pixel 2608 of the image sensor 2607. That is, the sampling phases are in a uniform state. When the subject changes from the imaging distance H0 at P0 to the imaging distance H1 at P1, the imaging position on the imaging device 2607 shifts, and the difference in the shift amount becomes u0-u1.
u0−u1 = f × B × ((1 / H0) − (1 / H1)) (1)
It can be expressed. Here, f is a focal length, and B is a baseline length. Here, the base line length B is a distance between the principal point Q of the optical system 2602 and the principal point R of the optical system 2606.

For example, if the focal length f is 5 mm, the baseline length B is 12 mm, the imaging distance H0 is 700 mm, and the imaging distance H1 is 725 mm, the difference in the shift amount of the imaging position is about 3 μm from the equation (1). When the pixel pitch of the image sensor is 6 μm, the sampling phase is shifted by about ½ pixel by changing the imaging distance from 700 mm to 725 mm. Assuming that the sampling phase is shifted by ½ pixel at an imaging distance of 700 mm, when the imaging distance is changed to 725 mm, 1/2 + 1/2 = 1, which is an integer, that is, the sampling phases are aligned, resulting in high resolution. It becomes impossible to become. In order to obtain a state in which the sampling phase is shifted regardless of the imaging distance, the focal length f or the base line length B may be reduced so as not to shift the imaging position from the equation (1). The base line length B is the distance between the principal points of the optical systems of the two image pickup systems. As shown in FIG. 27, the arrangement direction of the pixels of the image pickup device of the reference image pickup system is the horizontal direction and the vertical direction. The horizontal component of the length B is expressed as a horizontal baseline length BH, and the vertical component is expressed as a vertical baseline length BV.

In Patent Document 2, by arranging a plurality of imaging systems so that the baseline length in the horizontal direction and the baseline length in the vertical direction are reduced, the change in the sampling phase is small even if the imaging distance changes, and it does not depend on the imaging distance. A technique for shifting pixels is proposed.

28, 29, the compound eye system disclosed in Patent Document 2 will be described. As shown in FIG. 28, in Patent Document 2, the

optical systems

301a, 301b, and 301c are arranged in the same plane so that the

optical axes

300a, 300b, and 300c are parallel, and the optical system is rotated in the plane. Thus, the imaging position of the subject image is shifted to realize pixel shifting. At this time, as shown in FIG. 29, since the deviation between the optical axis 300b and the optical axis 300c in the Y direction (shown in the figure) is sufficiently small, that is, the baseline length in the Y direction is sufficiently small, the imaging distance changes. Even in this case, the change in the sampling phase in the Y direction can be kept small.

Regarding the X direction (described in the figure), since the base line length in the X direction is large, the sampling phase is uniform depending on the imaging distance, but the base line length in the X direction of the optical system 301a and the optical system 301b is sufficiently small. Even if the distance changes, the change in the sampling phase in the X direction can be kept small.

As described above, Patent Document 2 proposes a technique that arranges an imaging system so that a change in sampling phase with a change in imaging distance is small, and realizes pixel shifting regardless of the imaging distance.

JP-A-10-304235 International Publication WO2007 / 013250

However, in order to obtain a high-resolution image by compound-eye pixel shift, it is necessary to calculate the shift amount (parallax) between images in each image sensor. In Patent Document 2, the amount of displacement of the image sensor in the pixel shifting direction is about ½ pixel regardless of the imaging distance, but the baseline length is large in the direction perpendicular to the pixel shifting direction, and the distance between the images depends on the imaging distance. Since the amount of deviation changes, it is necessary to know the amount of deviation. For example, there is a method called stereo matching. Stereo matching is based on one image on multiple images taken from the same subject and the corresponding point (projection point of the same spatial point) is assigned to each pixel in the image in the other image. It is to search from. Stereo matching includes various methods such as region-based matching for obtaining corresponding points using template matching and feature-based matching for extracting feature points such as edges and corners. A high-resolution image is generated by arranging each pixel at a corresponding point on the high-resolution image based on the shift amount between the images thus obtained. At this time, as shown in Patent Document 2, when a high-resolution image is generated by synthesizing images of imaging systems arranged at different heights in the vertical direction, corresponding points are shifted in the vertical direction in the image. Located in place. Therefore, when reading the image from the upper left of the image during processing by hardware, the timing to reach the region where the corresponding point exists in each imaging system is different, so it is necessary to hold the data, and the line memory or frame memory It is necessary to use.

In addition, in the configuration of Patent Document 2, in order to increase the resolution in all the horizontal and vertical directions, two eyes that are shifted by 1/2 pixel in the horizontal and vertical directions are necessary, and the cameras are juxtaposed in the horizontal or vertical direction. A thin configuration cannot be realized in one direction.

The present invention has been made in view of such circumstances, and each camera is arranged in one horizontal or vertical direction so that the amount of memory used is reduced when it is implemented as hardware, depending on the distance of the subject. An object of the present invention is to provide an imaging apparatus capable of increasing the resolution.

The present invention has been made to solve the above-described problems.
The first technical means of the present invention is:
The image pickup system includes at least three image pickup systems each having an optical system and an image pickup element. The image pickup system is arranged in a pixel arrangement direction of the image pickup element and has an image pickup distance within a predetermined range in a picked-up image of the first image pickup system. A subject is imaged with a sampling phase in the arrangement direction shifted from 0.2 to 0.8 pixels with respect to a captured image of the first imaging system by any other imaging system. .

The second technical means is
In the first technical means, the imaging systems are arranged at substantially equal intervals in the arrangement direction, and the optical axes of the imaging systems are substantially parallel.

The third technical means is
In the first or second technical means, a subject within an imaging distance within a predetermined range is sampled in a direction perpendicular to the arrangement direction with respect to a captured image of any imaging system by any other imaging system. It is characterized in that the image is taken with a phase shift of 0.2 to 0.8 pixels.

The fourth technical means is
In the first or second technical means, a subject within an imaging distance within a predetermined range is sampled in a direction perpendicular to the arrangement direction with respect to a captured image of any imaging system by any other imaging system. The image is picked up with a phase shift of approximately 0.5 pixels.

The fifth technical means is
In the third or fourth technical means, a subject within an imaging distance within a predetermined range is sampled in a direction perpendicular to the arrangement direction with respect to a captured image of any imaging system by any other imaging system. It is characterized in that imaging is performed with almost no phase shift.

The sixth technical means is
In the first or second technical means, a subject within an imaging distance within a predetermined range is captured in a direction perpendicular to the arrangement direction by any two other imaging systems with respect to a captured image of any imaging system. The sampling phase is imaged with a shift of approximately 1/3 pixels.

The seventh technical means includes at least three image pickup systems having an optical system and an image pickup device, the image pickup systems are arranged on a substantially straight line, a plane normal including the straight line, an optical axis of the image pickup system, and Are substantially parallel, the straight line and one arrangement direction of the pixels of the imaging element are substantially parallel, and an object at an imaging distance within a predetermined range is captured by the imaging system of any of the imaging systems. It is characterized by being configured so as to be imaged with the sampling phase of the direction shifted.

According to an eighth technical means, in the seventh technical means, an object at an imaging distance within a predetermined range is set to have a sampling phase in the arrangement direction of 0.2 to 0.8 by any one of the imaging systems. It is configured to be imaged with a pixel shift.

Ninth technical means is characterized in that, in the seventh technical means, the optical axes of the imaging system are arranged at equal intervals in the arrangement direction.

According to a tenth technical means, in any one of the seventh to ninth technical means, an object at an imaging distance within a predetermined range is moved in a direction perpendicular to the arrangement direction by any one of the imaging systems. It is characterized by being configured so as to be imaged with the sampling phase shifted.

In an eleventh technical means according to any one of the seventh to ninth technical means, a subject at an imaging distance within a predetermined range is perpendicular to the arrangement direction by any one of the imaging systems. The sampling phase is configured to be imaged with a deviation of 0.2 to 0.8 pixels.

In a twelfth technical means according to any one of the seventh to ninth technical means, a subject at an imaging distance within a predetermined range is perpendicular to the arrangement direction by any one of the imaging systems. The sampling phase is configured to be imaged with a shift of 0.5 pixels.

According to a thirteenth technical means, in the eleventh technical means or the twelfth technical means, an object at an imaging distance within a predetermined range is moved in a direction perpendicular to the arrangement direction by any one of the imaging systems. The sampling phase is configured to be imaged without deviation.

According to a fourteenth technical means, in any one of the tenth and eleventh technical means, an object at an imaging distance within a predetermined range is moved in a direction perpendicular to the arrangement direction by any three imaging systems of the imaging system. The sampling phase is configured to be imaged with a shift of 1/3 pixel at a time.

According to the present invention, in an imaging device that captures a common subject using a plurality of imaging systems, the imaging systems are arranged side by side in either one of the horizontal and vertical directions, and a plurality of obtained images are combined to form a pixel. When generating a large number of images with high resolution, it is possible to reduce the amount of memory used. Moreover, it becomes possible to reduce in thickness by juxtaposing in one direction.

It is a figure which shows the structure of 1st embodiment of this invention. It is a figure explaining the relationship between the arrangement direction of the pixel of an image sensor, and an X-axis, a Y-axis, and a Z-axis. It is a figure explaining arrangement of three image sensors. It is XZ top view which looked at an imaging device from the top. It is a graph which shows the shift amount of the image formation position at the time of changing imaging distance. It is a graph which shows the imaging distance dependence of a sampling phase. It is a graph which shows the result of having calculated the shift of sampling phase. FIG. 7 is a graph showing a result of overlapping the sampling phase shift of FIG. 5 and the sampling phase shift of FIG. 6; It is a graph which shows the result of having changed the display range of the vertical axis | shaft of FIG. 7 from 0.25 pixel to 0.75 pixel. It is the figure which looked at the

image sensors

106 and 109 from the optical axis direction. It is a figure which shows the other example of arrangement | positioning of three imaging systems. It is a figure which shows the further another example of arrangement | positioning of three imaging systems. It is a figure which shows the further another example of arrangement | positioning of three imaging systems. It is a figure which shows a mode that the two imaging systems juxtaposed in the perpendicular direction are image | photographing a to-be-photographed object. It is a figure which shows the captured image at the time of imaging in the state of FIG. It is a figure which shows a mode that two imaging systems juxtaposed in the horizontal direction are image | photographing a to-be-photographed object. It is a figure which shows the captured image at the time of imaging in the state of FIG. It is a graph which shows the shift | offset | difference amount of the sampling phase in the case of QR = 11.7mm. It is a graph which shows the deviation | shift amount of a sampling phase in case QR = 11.0mm. It is a figure which shows the image imaged with two imaging systems juxtaposed in the horizontal direction. It is a figure which shows the image imaged with two imaging systems juxtaposed in the horizontal direction. It is a figure explaining the technique of patent document 1. FIG. It is a figure explaining the horizontal direction and vertical direction with respect to an image sensor. It is a figure which shows a mode that the common to-be-photographed object is imaged with two imaging systems. It is the figure which showed the field which two imaging systems picturize with a lattice. It is a figure explaining the state of pixel shifting. It is a figure explaining the change of a sampling phase when imaging distance changes in a compound eye system. It is a figure explaining the baseline length BH of a horizontal direction, and the baseline length BV of a perpendicular direction. It is a figure explaining the compound eye system currently indicated by patent documents 2. FIG. It is a figure explaining the compound eye system currently indicated by patent documents 2. FIG. It is a graph which shows the result of having verified high resolution by two imaging systems.

<First embodiment>
Hereinafter, a first embodiment of the present invention will be described with reference to the drawings. FIG. 1 is a diagram showing the configuration of this embodiment. FIG. 1 is a diagram illustrating an arrangement of three imaging systems provided in the imaging apparatus according to the present embodiment. The imaging system 101 includes an optical system 102 and an imaging element 103, the imaging system 104 includes an optical system 105 and an imaging element 106, and the imaging system 107 includes an optical system 108 and an imaging element 109. The three

optical systems

101, 104, and 107 are arranged on substantially the same plane, and their optical axes are substantially parallel. The three

image sensors

103, 106, and 109 are disposed on substantially the same plane, and the plane on which the optical system is disposed and the plane on which the image sensor is disposed are substantially parallel.

In the present embodiment, as shown in FIG. 2A, the arrangement direction of the pixels of the image sensor 106 is the horizontal direction (X axis) and the vertical direction (Y axis), and the direction parallel to the optical axis is the imaging distance Z axis. Is the principal point of the optical system 105. In addition, as illustrated in FIG. 2B, the arrangement direction of the pixels of the

imaging elements

103 and 109 is substantially parallel to the arrangement direction of the pixels of the imaging element 106. The pixel pitches in the horizontal direction of the

image sensors

103, 106, and 109 are equal, and the pixel pitch in the vertical direction is also equal. In addition, the readout direction of the pixels of each image sensor is assumed to be the X direction.

FIG. 3 is an XZ plan view of the imaging device shown in FIG. 1 as viewed from above. The relationship between the sampling phases in the horizontal direction of the

imaging systems

104 and 107 will be described with reference to FIG. As shown in FIG. 3, the subject at the point P0 where Z = H0 forms an image at the center of the pixel 106a of the image sensor 106 in the imaging system 104, and in the center of the pixel 109a of the image sensor 109 in the imaging system 107. An image is formed. That is, at the imaging distance H0, the horizontal sampling phases of the

imaging systems

104 and 107 are aligned. Although the sampling phase is changed by changing the relative positional relationship between the optical system and the image sensor, in FIG. 3, when the sampling phases in the horizontal direction of the

imaging systems

104 and 107 are aligned at the imaging distance P0. To do. Next, a change in sampling phase when the imaging distance is changed will be described.

In FIG. 4, the focal length f is 5 mm, the base length B (= RS) is 12 mm, the imaging distance H0 is 500 mm, and the shift amount of the imaging position when the imaging distance is changed from H0 is calculated from the equation (1). The results are shown. The horizontal axis represents the shooting distance, and the vertical axis represents the shift amount of the imaging position. The shift amount changes as the imaging distance changes.

Next, the sampling phase shift amount is calculated. The sampling phase can be calculated from the value obtained by dividing the shift amount by the pixel pitch of the image sensor and the value after the decimal point of the sum of the deviation amounts of the sampling phase at the imaging distance H0. For example, in FIG. 4, the shift amount at the imaging distance H1 (= 600 mm) is calculated from the equation (1):
u0−u1 = 5 × 12 × ((1/500) − (1/600)) = 0.02 (mm)
When the pixel pitch is 6 μm and the shift amount is divided by the pixel pitch,
0.02 ÷ 0.006 ≒ 3.33 (pixels)
Thus, there is a shift of 3.33 pixels from the imaging distance H0 (= 500 mm). Assuming that the sampling phase is shifted by 0.5 pixels at the imaging distance H0, the amount of sampling phase shift at the imaging distance H1 is
3.33 + 0.5 = 3.83 (pixels)
Thus, 0.83 pixels below the decimal point is the sampling phase shift amount at the imaging distance H1. When the sampling phase shift amount is a negative value according to the above calculation, a value obtained by adding 1 is the sampling phase shift amount.

Fig. 5 shows the imaging distance dependence of the sampling phase. The pixel pitch in the horizontal direction is 6 μm. It is assumed that the sampling phases are aligned at the reference imaging distance H0 (= 500 mm). It can be seen that the sampling phase changes when the imaging distance is changed. Therefore, the

imaging systems

104 and 107 have a distance at which the sampling phases are aligned according to the imaging distance, and there are distances at which the resolution cannot be increased depending on the imaging distance.

Therefore, in the present embodiment, an imaging area that cannot be increased in resolution by the

imaging systems

104 and 107 is increased in resolution by the

imaging systems

104 and 101, thereby realizing an increase in resolution over the entire imaging area.

As described above, since the sampling phases of the

imaging systems

104 and 107 are aligned at the reference imaging distance H0, the imaging element 103 is arranged so that the sampling phases of the

imaging systems

104 and 101 are shifted by 1/2 pixel. In FIG. 3, the subject at the point P0 forms an image at the center of the pixel 106a of the image sensor 106 in the image pickup system 104, but forms an image between the pixel 103a of the image sensor 103 and the image sensor 103b in the image pickup system 101. The sampling phase is shifted by 1/2 pixel.

Next, the change in sampling phase when the imaging distance is changed will be described.

In FIG. 6, the shift amount of the imaging position when the imaging distance is changed from H0 when the focal length f is 5 mm, the base length B (= QR) is 12 mm, the imaging distance H0 is 500 mm, is calculated from the equation (1). The results of calculating the sampling phase shift are shown below. It can be seen that at the reference imaging distance H0 (= 500 mm), the sampling phase is ½ pixel shifted, and the sampling phase changes when the imaging distance is changed.

Next, FIG. 7 shows the result of overlapping the sampling phase shift between the

imaging systems

104 and 107 shown in FIG. 5 and the sampling phase shift between the

imaging systems

104 and 101 shown in FIG. The sampling phase shift between the

imaging systems

104 and 107 is indicated by a thin line, and the sampling phase shift between the

imaging systems

104 and 101 is indicated by a bold line. As can be seen from FIG. 7, at the imaging distance where one sampling phase is aligned, the other sampling phase is shifted, and at least one of the imaging distances is in a pixel shifted state. FIG. 8 shows the display range of the vertical axis in FIG. 7 changed from 0.25 pixel to 0.75 pixel. It can be seen that at any imaging distance, either one of the sampling phase shift amounts is always within the range of 0.25 to 0.75 pixels.

Next, we will describe the amount of sampling phase shift required for higher resolution. FIG. 30 shows the result of verification of high resolution using two imaging systems. The chart for resolution measurement was imaged with two imaging systems arranged horizontally, the two images were synthesized, and the horizontal resolution of the synthesized image was evaluated. The horizontal axis of the figure shows the imaging distance from the imaging system to the chart, and the chart was placed at a total of 26 points in order from 100 cm to 125 cm in 1 cm increments. The vertical axis in the figure indicates the horizontal resolution of the composite image. The vertical axis also shows the sampling phase deviation. The sampling phase shift amount was calculated by the above-described method using the focal length, pixel pitch, imaging distance, and the like of the imaging system. From the figure, it can be seen that a sampling phase shift of 0.20 pixels to 0.80 pixels is necessary to sufficiently improve the resolution. Therefore, all three imaging systems are used, and at any imaging distance, either one of the sampling phase shift amounts is always within the range of 0.25 pixel to 0.75 pixel. The resolution can be increased at an imaging distance of.

In this embodiment, the horizontal baseline length QR of the

imaging systems

104 and 101 and the horizontal baseline length RS of the

imaging systems

104 and 107 are both equal to 12 mm, and the focal length f is also equal to 5 mm. By making the base line length and the focal length equal, the amount of change in the shift amount accompanying the change in the imaging distance becomes equal, and imaging with the same sampling phase as that of a certain imaging system (imaging system 104 in this case) at the reference distance. By providing a system (here, the imaging system 107) and an imaging system (here, the imaging system 101) that is shifted by 1/2 pixel of the sampling phase, it is possible to shift the pixels at all imaging distances so that the resolution can be increased. It becomes possible to pick up an image.

Next, we will discuss the high resolution in the vertical direction. FIG. 9 is a view of the

image sensors

106 and 109 as viewed from the optical axis direction. R ′ and S ′ indicate points where the optical axes of the

imaging systems

104 and 107 intersect with the imaging device, and the heights of the optical axes are the same. The image sensor 109 is arranged so that the sampling phase is shifted by 1/2 pixel in the vertical direction with respect to the image sensor 106. Since the heights of the optical axes coincide with each other, the base line length in the vertical direction is zero, and the sampling phase does not change even when the imaging distance is changed according to the equation (1). Accordingly, the sampling phase in the vertical direction is shifted by 1/2 pixel at all imaging distances, and high resolution can be achieved at all imaging distances. Similarly to the horizontal direction, in order to increase the resolution in the vertical direction, the sampling phase of at least two of the three imaging systems only needs to be shifted from 0.20 to 0.80 pixels at which the resolution can be increased. .

With the configuration as described above, by arranging three imaging systems in the horizontal direction, high resolution can be achieved both in the vertical direction and in the horizontal direction regardless of the imaging distance.

Note that, in order to increase the resolution in the vertical direction, the sampling phase of at least two imaging systems out of the three imaging systems may be shifted from 0.20 to 0.80 pixels where the resolution can be increased. The arrangement of the two imaging systems is not limited to the above. For example, as shown in FIG. 10, the configuration in which the sampling phases of the three imaging systems are shifted by 1/3 pixels may be adopted, or as shown in FIG. 11, the sampling phase of one imaging system with respect to the reference imaging system. May be shifted by 1/2 pixel, and the sampling phase of the other imaging system may be aligned. As shown in FIG. 12, the sampling phases of the two imaging systems are both 1/2 pixels relative to the reference imaging system. A configuration may be adopted in which the sampling phases of the two imaging systems are aligned.

Next, calculation of shift amounts (parallax) of a plurality of images necessary for obtaining a high-resolution image by compound-eye pixel shifting will be described. In FIG. 13, two

imaging systems

1301 and 1302 are juxtaposed in the vertical direction to image the subject 1303. FIG. 14 shows a captured image 1401 of the imaging system 1301 and a captured image 1402 of the imaging system 1302 when the image is captured in the state of FIG. The subject 1303 is imaged with a shift in the vertical direction. When a corresponding point is searched from the captured image 1401 based on the captured image 1402 by hardware processing, for example, a point corresponding to the pixel 1404 in the captured image 1402 indicated by shading is the pixel 1403 in the captured image 1401. However, if the corresponding point is searched from the upper left pixel 1405 indicated by diagonal lines, it is necessary to hold data until the corresponding point 1403 is reached, and it is necessary to use a line memory or a frame memory. .

On the other hand, in FIG. 15, two

imaging systems

1501 and 1502 are juxtaposed in the horizontal direction to image the subject 1503. FIG. 16 shows a captured image 1601 of the imaging system 1501 and a captured image 1602 of the imaging system 1502 when the image is captured in the state of FIG. The subject 1503 is imaged with a shift in the horizontal direction. When a corresponding point is searched from the photographed image 1601 based on the photographed image 1602 by hardware processing, for example, a point corresponding to the pixel 1604 in the photographed image 1602 indicated by shading is the pixel 1603 in the photographed image 1601. However, since there is no deviation in the vertical direction, it is not necessary to search for the corresponding point from the upper left pixel 1605 indicated by the diagonal line, and the horizontal line which is the leftmost pixel on the line having the same height as the pixel 1603 which is the corresponding point. It is only necessary to search from the pixel 1606 shown, and the amount of data held until reaching the corresponding point can be significantly reduced as compared with the case where the imaging systems are juxtaposed vertically.

Further, by juxtaposing the three image pickup systems in the horizontal direction, it is possible to reduce the thickness in the vertical direction as compared with the conventional arrangement in which the vertical and horizontal directions are juxtaposed.

<Second Embodiment>
Next, a second embodiment of the present invention will be described. In the first embodiment, by arranging three imaging systems having the same focal length so that the baseline lengths in the horizontal direction are equal, a configuration capable of increasing the resolution in both the vertical and horizontal directions regardless of the imaging distance is realized. .

In the second embodiment, a configuration capable of increasing the resolution regardless of the imaging distance even when an error occurs in the position accuracy in the actual manufacturing process of the imaging device will be described.

If the three imaging systems are not arranged at equal intervals, the amount of shift accompanying the change in the imaging distance is different, but if the difference in the baseline length is small, the influence on the sampling phase will be small, and the imaging distance will be reduced. Regardless, the resolution can be increased.

For example, in FIG. 3, when the focal length f of the three imaging systems is 5 mm, the base lengths RS and QR are 12 mm, and the range of the imaging distance to be increased in resolution is 1000 mm to 3000 mm, An allowable error of the base line length QR for imaging so that the resolution can be achieved is obtained. The sampling phase of the imaging system 104 and the imaging system 107 is shifted by ½ pixel at an imaging distance of 1500 mm, and the sampling phase of the imaging system 101 and the imaging system 104 is aligned. When QR is changed by 1 mm from the state of QR = 12 mm, in the range of QR = 11.7 mm to QR = 12.4 mm, the sampling phases of the imaging system 104 and the imaging system 107 between the imaging distances of 1000 mm and 3000 mm, At least one of the sampling phases of the imaging system 101 and the imaging system 104 is shifted by 0.20 to 0.80 pixels, and it can be said that the resolution can be increased regardless of the imaging distance. FIG. 17 shows the amount of sampling phase shift when QR = 11.7 mm, and FIG. 18 shows the amount of sampling phase shift when QR = 11.0 mm. In FIG. 18, the two sampling phase shift amounts are not within the range of 0.20 to 0.80 pixels at the imaging distances of 2320 mm and 2980 mm indicated by circles, and the resolution cannot be increased. Yes. On the other hand, in FIG. 17, at any distance, either one of the two sampling phase shift amounts is in the range of 0.20 to 0.80 pixels, and the resolution can be increased.

As described above, in the second embodiment, even when there is an error in the baseline length QR of the

imaging systems

101 and 104 and the baseline length RS of the

imaging systems

104 and 107, the error is suppressed within an allowable range. As a result, it is possible to increase the resolution regardless of the imaging distance within the range of the predetermined imaging distance.

Note that the sampling phase varies depending not only on the base line length but also on the relative positional relationship between the optical axis and the image sensor, the focal length, and the pixel pitch, and also depends on the range of the imaging distance for which high resolution is desired. The error range cannot be defined only by the baseline length, but if the relative positional relationship between the optical axis of the optical system and the image sensor, the focal length, the pixel pitch, and the range of the imaging distance to be increased in resolution are determined. The allowable range can be calculated from the shift amount obtained from the equation (1).

In the present embodiment, three imaging systems in which at least one of the two sampling phase shift amounts falls within a range of 0.20 to 0.80 pixels capable of high resolution within a predetermined imaging distance. By arranging, the horizontal resolution can be improved regardless of the imaging distance.

Further, with respect to the vertical direction, by adopting the same configuration as that of the first embodiment, high resolution can be achieved regardless of the imaging distance.

<Third embodiment>
Next, a third embodiment of the present invention will be described. In the third embodiment, as shown in FIG. 11 and FIG. 12, among the three imaging systems arranged in the horizontal direction, the vertical sampling phases of the two imaging systems are aligned, and the vertical direction of the other imaging system is aligned. The sampling phase is shifted by 1/2 pixel with respect to the other two imaging systems.

In the present invention, the sampling phase shift is almost constant regardless of the imaging distance in the vertical direction where the baseline length is small, but the sampling phase shift varies depending on the imaging distance in the horizontal direction where the baseline length is large. The shift amount (parallax) must be obtained using a method such as stereo matching. When searching for corresponding points in the stereo matching method, if there is a deviation in the sampling phase, there is no perfectly corresponding corresponding point, and a predetermined corresponding point exists between pixels.

This will be described with reference to FIG. FIG. 19 shows

images

1901 and 1902 captured by two imaging systems juxtaposed in the horizontal direction. When a point corresponding to the pixel 1903 in the captured image 1902 is searched from the captured image 1901, the corresponding point exists between the pixel 1904 and the pixel 1905.

Furthermore, the case where there is a deviation in the sampling phase in the vertical direction will be described with reference to FIG. FIG. 20 shows

images

2001 and 2002 captured by two imaging systems juxtaposed in the horizontal direction, and the sampling phase in the vertical direction is also shifted by ½ pixel. In FIG. 20, a point corresponding to the pixel 2003 in the captured image 2002 exists between the four

pixels

2004, 2005, 2006, and 2007 in the captured image 2001.

Thus, when searching for corresponding points, if there is a shift in the sampling phase, there is no corresponding matching point, and the calculation accuracy may be reduced. In the third embodiment, the sampling phases in the vertical direction of the two imaging systems are aligned, and when searching for corresponding points, it is possible to search from at least two images that are not shifted in the vertical direction, and the calculation accuracy is high. improves. By improving the calculation accuracy, it is possible to generate an image with higher resolution by reducing the error when generating each pixel on the corresponding point based on the shift amount between images and generating on the high resolution image. It becomes possible.

<Fourth embodiment>
Next, a fourth embodiment of the present invention will be described. In the fourth embodiment, as shown in FIG. 10, the vertical sampling phases of the three imaging systems are shifted by 1/3 pixels. As described above, when the resolution is increased by the two imaging systems, the optimum condition is that the sampling phase is shifted by 1/2 pixel, which is because the sampling interval is the narrowest. For example, if the sampling phase in the vertical direction is shifted by 0.3 pixels, one of the vertical directions is shifted by 0.3 pixels, while the other is shifted by 0.7 pixels, and the sampling interval Is wide. If the vertical sampling phase is shifted by ½ pixel, the vertical direction is ½ pixel. Therefore, when three imaging systems are provided, a state in which the sampling phase is shifted by 1/3 pixel is a configuration in which the sampling interval is the shortest, which is suitable for high resolution.

As described above, according to the present invention, three imaging systems are arranged in the pixel arrangement direction, and a subject within a predetermined range of imaging distance is subjected to a sampling phase which is a condition for high resolution by any two imaging systems. By adopting a configuration in which an image is captured with a shift of 0.2 to 0.8 pixels, the resolution is improved in both the horizontal and vertical directions, and the hardware is provided by arranging the three imaging systems in one direction. Thus, it is possible to reduce a processing amount and a memory amount necessary for performing image processing such as parallax calculation.

The present invention can be used for an imaging apparatus using a plurality of imaging systems.

101, 104, 107, 1301, 1302, 1501, 1502

Imaging system

102, 105, 108

Optical system

103, 106, 109

Imaging device

1303, 1503

Subject

1401, 1402, 1601, 1602, 1901, 1902, 2001, 2002 Captured

image

1403, 1404, 1405, 1603, 1604, 1605, 1606, 1903, 1904, 1905, 2003, 2004, 2005, 2006, 2007 pixels

Claims

The image pickup system includes at least three image pickup systems each having an optical system and an image pickup element. The image pickup system is arranged in a pixel arrangement direction of the image pickup element and has an image pickup distance within a predetermined range in a picked-up image of the first image pickup system. A subject is imaged with a sampling phase in the arrangement direction shifted from 0.2 to 0.8 pixels with respect to a captured image of the first imaging system by any other imaging system. Imaging device.
2. The imaging apparatus according to claim 1, wherein the imaging systems are arranged at substantially equal intervals in the arrangement direction, and the optical axes of the imaging systems are substantially parallel.
A subject within an imaging distance of a predetermined range has a sampling phase of 0.2 to 0.8 pixels in a direction perpendicular to the arrangement direction by any of the other imaging systems with respect to a captured image of any imaging system. The imaging apparatus according to claim 1, wherein the imaging is performed with a shift.
A subject within an imaging distance within a predetermined range is imaged with a sampling phase in a direction perpendicular to the arrangement direction shifted by approximately 0.5 pixels by any other imaging system with respect to a captured image of any imaging system. The imaging apparatus according to claim 1 or 2, wherein
A subject within an imaging distance of a predetermined range is imaged with a sampling phase in a direction perpendicular to the arrangement direction with almost no deviation from one of the other imaging systems with respect to a captured image of any imaging system. The imaging apparatus according to claim 3 or 4, wherein the imaging apparatus is characterized.
A subject within a predetermined imaging distance is shifted by approximately 1/3 of the sampling phase in the direction perpendicular to the arrangement direction with respect to the captured image of any imaging system by any two other imaging systems. The imaging apparatus according to claim 1, wherein the imaging apparatus captures an image.