JP6192222B2 - Imaging device - Google Patents

Description

  The present invention relates to a photographing apparatus.

  Integral photography is known as one of the aerial image reproduction methods (see, for example, Patent Document 1). A photographing apparatus to which this integral photography is applied photographs a light beam from a subject incident through a lens array and generates an integral image. A display device to which integral photography is applied generates a light beam by displaying an integral image on a display unit, and generates a spatial image by passing the light beam through a lens array. That is, integral photography is a reproduction method in which a display device generates a light beam similar to a light beam coming from an actual subject.

JP 2001-228570 A

  The resolution of the aerial image by integral photography depends on the number of light beams emitted from the display device through the lens array. The number of light rays emitted from the display device, that is, the number of light rays recorded in the integral image depends on the resolution of the photographing device. Therefore, it is effective to increase the resolution of the photographing apparatus as one method for increasing the definition of the aerial image by integral photography.

  The present invention has been made in view of the above circumstances, and an object of the present invention is to provide a photographing apparatus that generates integral image data for obtaining a high-definition aerial image by integral photography.

  [1] In order to solve the above-described problem, an imaging device according to one embodiment of the present invention includes a plurality of imaging elements arranged in a matrix with a gap between imaging surfaces, and the plurality of imaging elements. Corresponding to each imaging surface, a plurality of lens arrays provided with a gap, and the value of the first interpolation target pixel in the interpolation target element image for the virtual element lens corresponding to the gap between the lens arrays, The values of a plurality of pixels obtained through the viewpoint and principal points of element lenses around the virtual element lens for each of a plurality of viewpoints on a line passing through one interpolation target pixel and the principal point of the virtual element lens An interpolation processing unit that calculates based on the average value.

[2] In the photographing apparatus according to [1], the interpolation processing unit includes a plurality of pixels for each of the plurality of viewpoints in an order away from the principal point of the virtual element lens with respect to the interpolation target element image. In the average value, the average value when the change rate of the average value is equal to or less than the threshold value is set as the value of the first interpolation target pixel.
[3] In the photographing apparatus according to [1], the interpolation processing unit calculates an average value of a plurality of pixel values for each of the plurality of viewpoints as a value of the first interpolation target pixel. To do.
[4] In the photographing apparatus according to any one of [1] to [3], the interpolation processing unit performs first interpolation in the interpolation target element image from an interpolation target pixel corresponding to a gap between the imaging surfaces. The value of the second interpolation target pixel excluding the target pixel is calculated by pixel interpolation processing.

  According to the present invention, it is possible to generate integral image data for obtaining a high-definition aerial image by integral photography.

1 is a diagram illustrating an outline of a configuration of an imaging display system to which an imaging apparatus according to an embodiment of the present invention is applied. It is a side view at the time of seeing the multi lens imaging array shown in FIG. 1 in the X-axis direction. FIG. 2 is a diagram (XY plan view) schematically showing a gap between imaging surfaces in a multi-lens imaging unit when the multi-lens imaging array shown in FIG. 1 is viewed in the Z-axis direction from the subject side. FIG. 3 is a diagram (XY plan view) schematically showing a gap between imaging lens arrays in a multi-lens imaging unit when the multi-lens imaging array shown in FIG. 1 is viewed from the subject side in the Z-axis direction. FIG. 5 is a diagram showing both the gap between the imaging surfaces shown in FIG. 3 and the gap between the imaging lens arrays shown in FIG. 4. It is a figure for demonstrating the process which an image processing apparatus calculates the value of the interpolation object pixel in the interpolation object element image with respect to a virtual element lens. It is a graph which shows the result of the calculation for an image processing apparatus to obtain | require the 1st interpolation object pixel in an interpolation object element image from two element images adjacent to an interpolation object element image.

Hereinafter, embodiments for carrying out the present invention will be described in detail with reference to the drawings.
FIG. 1 is a diagram showing an outline of the configuration of an imaging display system to which an imaging apparatus according to an embodiment of the present invention is applied. As shown in the figure, the imaging display system 1 includes a multi-lens imaging array 10, an image processing device (interpolation processing unit) 20, a projection device 60, a diffusion plate 70, and a display lens array 80. In the photographing display system 1, the photographing device includes a multi-lens imaging array 10 and an image processing device 20.

  FIG. 1 shows a subject S photographed by the multi-lens imaging array 10. In the same figure, three axes (X axis, Y axis, and Z axis) orthogonal to each other are shown. The X axis is an axis parallel to the horizontal direction of each imaging surface of the plurality of imaging elements included in the multi-lens imaging array 10. The Y axis is an axis parallel to the vertical direction of each imaging surface of the plurality of imaging elements included in the multi-lens imaging array 10. The Z axis is an axis perpendicular to the imaging surface of each imaging device of the multi-lens imaging array 10. That is, the Z axis is an axis parallel to the main optical axis of the photographing apparatus.

  The multi-lens imaging array 10 includes, for example, nine multi-lens imaging units 100. As will be described in detail later, each multi-lens imaging unit 100 includes an imaging lens array (lens array) and the imaging element. As shown in FIG. 1, the multi-lens imaging array 10 includes a multi-lens so that the imaging surfaces of the imaging elements in each of the nine multi-lens imaging units 100 are parallel to a plane including the X axis and the Y axis (XY plane). A predetermined gap is provided between the image pickup units 100, and nine multi-lens image pickup units 100 are arranged in three in each of the X-axis direction and the Y-axis direction. In the present embodiment, each multi-lens imaging unit 100 is distinguished by attaching a subscript (two-digit number) representing a two-dimensional coordinate to each of the nine multi-lens imaging units 100 provided in a matrix. (Hereafter, the same description may be made.)

  The multi-lens imaging array 10 is not limited to nine and may include a plurality of multi-lens imaging units 100. The multi-lens imaging array 10 is provided with a predetermined gap between the multi-lens imaging units 100, and the plurality of multi-lens imaging units 100 are arranged in at least one of the X-axis direction and the Y-axis direction. It may be configured.

  The multi-lens imaging unit 100 passes the luminous flux coming from the subject S through the imaging lens array, and photoelectrically converts the luminous flux that has passed through the imaging lens array by the imaging element to generate element image group data. The element image is an image obtained from the light flux from the element lens in the imaging lens array. Therefore, the element image group is an image group obtained from the light flux from the imaging lens array. The multi-lens imaging array 10 operates nine multi-lens imaging units 100 simultaneously to generate nine element image group data, and the nine element image group data is used as an element image group data set in the image processing apparatus 20. To supply. This element image group data set is a data set of element images corresponding to one frame.

  The image processing apparatus 20 takes in the element image group data set supplied by the multi-lens imaging array 10. The image processing apparatus 20 synthesizes the nine element image group data included in the element image group data set in accordance with the matrix arrangement of nine (3 × 3) multi-lens imaging units 100, thereby 1 Element image group data for a frame is generated.

  Specifically, when the image processing apparatus 20 synthesizes the nine element image group data, for example, the first interpolation target pixel in the interpolation target element image for the virtual element lens corresponding to the gap between the imaging lens arrays. For each of a plurality of viewpoints on a line passing through the first interpolation target pixel and the principal point of the virtual element lens, the viewpoint and the principal points of the element lenses around the virtual element lens Is calculated based on the average value of the values of a plurality of pixels obtained through. Then, the image processing device 20 calculates the value of the second interpolation target pixel excluding the first interpolation target pixel in the interpolation target element image from the interpolation target pixel corresponding to the gap between the imaging surfaces by pixel interpolation processing. To do. Then, the image processing device 20 combines the nine element image group data, the value of the first interpolation target pixel, and the value of the second interpolation target pixel to thereby generate one frame of element image group data. Is generated.

  The image processing apparatus 20 supplies the generated element image group data for one frame to the projection apparatus 60 as integral image data. Details of the process (synthesizing process) in which the image processing apparatus 20 combines the elemental image group data sets to obtain integral image data will be described later. The image processing device 20 is realized by an arithmetic processing device, such as a computer device, on which a central processing unit (CPU) is mounted.

  The photographing display system 1 includes a display device. This display device includes a projection device 60, a diffusion plate 70, and a display lens array 80. This display apparatus takes in integral image data generated by the photographing apparatus according to the present embodiment, and displays this integral image data as integral photography.

  The projection device 60 takes in integral image data supplied by the image processing device 20. The projection device 60 converts the integral image data into image light and projects the image light toward the diffusion plate 70. The projection device 60 is realized by, for example, a projector device.

  The diffusing plate 70 is a flat plate having both light transmission and light diffusing properties. The diffusion plate 70 transmits the image light projected by the projection device 60 while diffusing from one surface (the surface on the projection device 60 side) and reaches the other surface. As a result, the integral image appears on the other surface of the diffusion plate 70.

  The display lens array 80 is an element lens group configured by two-dimensionally arranging a plurality of element lenses regularly (for example, in a square lattice shape) so that the optical axes are parallel to each other. Each element lens is, for example, a convex lens. The display lens array 80 is provided at a position where the focal plane of each element lens coincides with the other surface of the diffusion plate 70. By passing the light beam from the integral image displayed on the other surface of the diffuser plate 70 through the display lens array 80, a spatial image corresponding to the subject S is generated.

  In the imaging display system 1, the positional relationship between the plurality of element lenses and pixels included in the nine imaging lens arrays 100 is equal to the positional relationship between the plurality of element lenses included in the display lens array 80 and the pixels.

FIG. 2 is a side view of the multi-lens imaging array 10 shown in FIG. 1 when viewed in the X-axis direction. FIG. 2 shows a state in which the multi-lens imaging units 100 ₁₁ , 100 ₁₂ , and 100 ₁₃ are fixed to the substrate unit 103 among the nine multi-lens imaging units 100 included in the multi-lens imaging array 10. Referring to FIGS. 1 and 2 together, nine multi-lens imaging units 100 ₁₁ , 100 ₁₂ , 100 ₁₃ , 100 ₂₁ , 100 ₂₂ , 100 ₂₃ , 100 ₃₁ , 100 ₃₂ , and 100 ₃₃ are planes of the substrate unit 103. It is fixed to.

In FIG. 2, the multi-lens imaging unit 100 ₁₁ includes an imaging lens array 101 ₁₁ and an imaging element 102 ₁₁ . The multi-lens imaging unit 100 ₁₂ includes an imaging lens array 101 ₁₂ and an imaging element 102 ₁₂ . The multi-lens imaging unit 100 ₁₃ includes an imaging lens array 101 ₁₃ and an imaging element 102 ₁₃ . Although not shown in the figure, the imaging lens array 100 and the imaging element 102 are also used for the multi-lens imaging units 100 ₂₁ , 100 ₂₂ , 100 ₂₃ , 100 ₃₁ , 100 ₃₂ , and 100 ₃₃ , as in the figure. Is provided.

  The imaging lens array 101 is an element lens group configured by regularly arranging a plurality of element lenses in a two-dimensional manner so that the optical axes are parallel to each other. When the imaging lens array 101 is viewed from the Z-axis direction, the plurality of element lenses are two-dimensionally arranged in a square lattice pattern. Each element lens of the imaging lens array 101 is realized by, for example, a gradient index (GRIN) lens.

  Specifically, a gradient index lens having the following specifications is used as an element lens of the imaging lens array 101. That is, a refractive index distribution type lens having a specification satisfying the following formula (1) is used as an element lens, where L is the length in the optical axis direction of the refractive index distribution type lens, P is the meandering period of the refractive index distribution type lens. . However, in Formula (1), n is a positive integer.

By using a gradient index lens satisfying the formula (1) as an element lens, an erect real image of the subject S existing sufficiently far from one end face of the gradient index lens can be obtained on the other end face or It can appear outside this end face.
In order to cause an erect real image of the subject S existing sufficiently far from one end surface of the gradient index lens to appear on the other end surface, a gradient index lens having a specification satisfying the following formula (2) is used. Use. However, in Formula (2), n is a positive integer.

  FIG. 2 schematically illustrates an example in which the multi-lens imaging unit 100 is configured using a gradient index lens in the case where n is 1 in Formula (2). By using a gradient index lens satisfying Expression (2) as the element lens, crosstalk between element images can be suppressed.

  The image sensor 102 photoelectrically converts a light beam obtained from each element lens of the imaging lens array 101 to generate element image group data. The image sensor 102 is realized by a solid-state image sensor such as a CCD (Charged Coupled Device) image sensor or a CMOS (Complementary Metal Oxide Semiconductor) image sensor, for example.

  In the multi-lens imaging unit 100, the imaging lens array 101 is provided so that the focal plane of each element lens coincides with the imaging surface of the imaging element 102. For example, when a gradient index lens satisfying the formula (2) is used as an element lens of the imaging lens array 101, the focal plane of the gradient index lens is the subject S side of the gradient index lens. Can be matched with the end face on the opposite side. Therefore, in this case, as shown in FIG. 2, the multi-lens imaging unit 100 can be configured by bringing the end surface opposite to the subject S side in the imaging lens array 101 into contact with the imaging surface of the imaging element 102. By using the multi-lens imaging unit 100 configured as described above, it is possible to realize an imaging apparatus that does not include a condenser lens or a diffusion plate. Therefore, according to the photographing apparatus, it is possible to suppress deterioration in image quality by not including the condenser lens and the diffusion plate, and it is possible to reduce the size of the apparatus.

FIG. 3 is a diagram (XY plan view) schematically showing a gap between imaging surfaces in the multi-lens imaging unit 100 when the multi-lens imaging array 10 shown in FIG. 1 is viewed in the Z-axis direction from the subject S side. It is. The figure shows four multi-lens imaging units 100 ₁₁ , 100 ₂₁ , 100 ₁₂ , 100 ₂₂ adjacent to each other in the multi-lens imaging array 10. Further, FIG. 4 shows an imaging lens array in which element lenses are arranged in a square lattice pattern in the XY plane and pixels on the imaging surface in each multi-lens imaging unit 100.

  In FIG. 3, the pixel density with respect to the element lens is low, that is, the pixel is large with respect to the element lens. However, this is for convenience of explanation, and the pixel density with respect to the element lens is actually higher. . In addition, the imaging lens array in the figure is configured by arranging element lenses without gaps in the X-axis direction and the Y-axis direction. In this case, the lens pitch in each of the X-axis direction and the Y-axis direction is equal to the diameter of the element lens. equal. The lens pitch is the size between the center points of adjacent element lenses. Further, the image pickup device in FIG. 2 is a pixel in which pixels are arranged in a lattice pattern without a gap. In this case, the pixel pitch in each of the X-axis direction and the Y-axis direction is equal to the pixel width. The pixel pitch is a size between the center points of adjacent pixels.

As shown in FIG. 3, in the XY plane, between the imaging plane of the imaging surface and the multi-lens imaging unit _{100 21} of the multi-lens imaging unit _{100 11,} and the imaging surface of the multi-lens imaging unit _{100 12} and a multi-lens imaging unit 100 A gap in the X-axis direction is provided between the ₂₂ imaging surfaces. The size P _X of the gap of the X-axis direction is a positive integer multiple of the X-axis direction pixel pitch in the imaging plane. The figure shows that the gap size P _X in the X-axis direction is 8 times the X-axis direction pixel pitch in the imaging plane.

Similarly, in the XY plane, between the imaging surface of the multi-lens imaging unit _{100 11} and a multi-lens imaging unit _{100 12} imaging surface, and the imaging surface of the multi-lens imaging unit _{100 21} of the imaging surface and the multi-lens imaging unit _{100 22} Is provided with a gap in the Y-axis direction. The size P _Y of the gap in the Y-axis direction is a positive integer multiple of the pixel pitch in the Y-axis direction on the imaging surface. This figure shows that the size P _Y of the gap in the Y-axis direction is 8 times the pixel pitch in the Y-axis direction on the imaging surface.

Note that the magnitude P _X and Y-axis direction of the gap size P _Y of the X-axis direction gap may not be equal may be equal.

4 is a diagram (XY plane) schematically showing a gap between imaging lens arrays in the multi-lens imaging unit 100 when the multi-lens imaging array 10 shown in FIG. 1 is viewed from the subject S side in the Z-axis direction. Figure). As shown in FIG. 4, in the XY plane, between the imaging lens array of multi-lens imaging lens array and the multi-lens imaging unit 100 ₂₁ of the imaging unit 100 _11, and the multi-lens imaging unit 100 ₁₂ imaging lens array and between the imaging lens array of multi-lens imaging unit 100 _22, a gap in the X-axis direction is provided. The size L _X of the gap in the X-axis direction is a positive integer multiple of the lens pitch of the imaging lens array. This figure shows that the size L _X of the gap in the X-axis direction is twice the lens pitch of the imaging lens array.

Similarly, in the XY plane, the multi-lens between the imaging lens array of the imaging unit 100 ₁₁ imaging lens array and the multi-lens imaging unit 100 _12, and the multi-lens imaging unit 100 ₂₁ imaging lens array and the multi-lens imaging between the imaging lens array parts 100 _22, the gap in the Y-axis direction is provided. The size L _Y of the gap in the Y-axis direction is a positive integer multiple of the lens pitch of the imaging lens array. This figure shows that the size L _Y of the gap in the Y-axis direction is twice the lens pitch of the imaging lens array.

  The element lens virtually provided corresponding to the gap between the imaging lens arrays is referred to as a virtual element lens. That is, in the example of FIG. 4, there are two rows of virtual element lenses in the gap in the X-axis direction, and two rows of virtual element lenses in the gap in the Y-axis direction.

Note that the size L _Y in the X-axis direction of the gap size L _X and Y-axis direction gap may not be equal may be equal.

FIG. 5 is a diagram showing both the gap between the imaging surfaces shown in FIG. 3 and the gap between the imaging lens arrays shown in FIG. As shown in FIG. 5, in the four multi-lens imaging units 100 ₁₁ , 100 ₂₁ , 100 ₁₂ , 100 ₂₂ arranged adjacent to each other in the XY plane, gaps between imaging surfaces in the X-axis direction and imaging lenses The gap between arrays is not the same. Similarly, the gap between the imaging surfaces in the Y-axis direction and the gap between the imaging lens arrays are not the same. According to FIG. 2 and FIG. 5, the nine multi-lens imaging units 100 have a size of the gap between the imaging surfaces of the imaging element 102 between the multi-lens imaging units 100 as a positive integer multiple of the pixel pitch on the imaging surface, And it fixes to the plane of the board | substrate part 103 so that the magnitude | size of the clearance gap between the lens arrays 101 for imaging satisfy | fills the positive integer multiple of a lens pitch.

  Here, the composition processing of the element image group data set by the image processing apparatus 20 will be described in detail. The image processing apparatus 20 takes in the element image group data set from the multi-lens imaging array 10, and the nine element image group data included in the element image group data set is a gap corresponding to a positive integer multiple of the pixel pitch shown in FIG. Is stored in the storage unit built in the device itself. For example, the storage unit is a memory capable of storing integral image data for at least one frame. That is, nine element image group data are stored in the storage unit corresponding to the matrix arrangement of the imaging surface of the imaging element 102.

  Then, the image processing device 20 uses the value of the first interpolation target pixel in the interpolation target element image for the virtual element lens corresponding to the gap between the imaging lens arrays shown in FIG. For each of a plurality of viewpoints on a line passing through the principal point of the virtual element lens, based on the average value of the values of a plurality of pixels obtained through the viewpoint and the principal points of the element lenses around the virtual element lens To calculate. In the integral photography photography / display method, light ray information from a subject is recorded, and a subject image (spatial image) is reproduced based on the recorded light ray information, so that it depends on the viewpoint distance from the display lens array. The size of the subject image changes. The above-described method for calculating the value of the first interpolation target pixel uses the characteristic of this integral photography, and the value of the pixel (first interpolation target pixel in the interpolation target element image) that could not be acquired on the photographing side. Is a method of calculating This will be described more specifically with reference to FIG.

  FIG. 6 is a diagram for explaining processing in which the image processing apparatus 20 calculates the value of the interpolation target pixel in the interpolation target element image for the virtual element lens. In the figure, between the element image A and the element image C, there is a pixel area of the interpolation target element image B for one element image. The principal point PA is a principal point of the element lens corresponding to the element image A. The principal point PC is a principal point of the element lens corresponding to the element image C. The principal point PB is the principal point of the virtual element lens corresponding to the interpolation target element image B. The figure shows an example in which the image processing device 20 calculates the value of the third interpolation target pixel B (3) in the interpolation target element image B. As shown in FIG. 4, the plurality of element lenses in the imaging lens array are arranged on a plane (XY plane) that is perpendicular to the main optical axis of the imaging apparatus, and therefore the line-of-sight direction in each element image is It gradually changes according to the arrangement of the element lenses. That is, there is a high probability that the correlation between images is high between adjacent element images, or between the element image of interest and the surrounding element images.

  Taking this statistical property and taking the interpolation target pixel B (3) of FIG. 6 as an example, the image processing apparatus 20 uses the interpolation target pixel B (3) of the interpolation target element image B and the virtual element lens. For each of n viewpoints (viewpoint 1, viewpoint 2,..., Viewpoint n) on the line 5 passing through the principal point PB, the value of the pixel obtained through the viewpoint and the principal point PA, and the viewpoint and the principal point. The average value of the pixel values obtained through the point PC is calculated. Specifically, the image processing apparatus 20 receives the value of the pixel A (1) in the element image A reached by the line 1A passing through the viewpoint 1 and the principal point PA and the line 1C passing through the viewpoint 1 and the principal point PC. An average value B (3) [1] with the value of the pixel C (5) in the element image C to be calculated is calculated. Similarly, in the image processing apparatus 20, the value of the pixel A (2) in the element image A reached by the line 2A passing through the viewpoint 2 and the principal point PA and the line 2C passing through the viewpoint 2 and the principal point PC arrive. An average value B (3) [2] with the value of the pixel C (4) in the element image C is calculated.

  After calculating the average value for each of the n viewpoints, the image processing apparatus 20 calculates the average value B (3) [1 for each of the n viewpoints in the order of moving away from the principal point PB with respect to the interpolation target element image B. ], B (3) [2],..., B (3) [n], the average value B (3) [k] when the average rate of change is equal to or less than a predetermined threshold value. The value is determined as the value of the interpolation target pixel B (3). Then, the image processing apparatus 20 stores the calculated value of the interpolation target pixel B (3) in a corresponding address area in the storage unit.

  That is, the image processing device 20 uses the optical axis direction connecting the first interpolation target pixel in the interpolation target element image and the principal point of the virtual element lens corresponding to the interpolation target element image as a reference, and in this optical axis direction, For each of a plurality of viewpoint positions, an average value of pixel values corresponding to the viewpoint position in an element image around the interpolation target element image (for example, an element image adjacent to the interpolation target element image) is calculated. The most probable value from the average value is determined as the value of the first interpolation target pixel. Thereby, the image processing apparatus 20 can generate | occur | produce an interpolation object element image with high precision.

  FIG. 7 is a graph illustrating a calculation result for the image processing apparatus 20 to obtain the first interpolation target pixel in the interpolation target element image from two element images adjacent to the interpolation target element image. In the graph of the figure, the horizontal axis is the viewpoint distance (unit: meters) from the display surface, and the vertical axis is the pixel value. In this graph, the solid line is the value of the first pixel included in one of the element images adjacent to the interpolation target element image (left side as viewed from the viewing side) corresponding to the first interpolation target pixel. The broken line is the value of the second pixel included in the other element image (right side as viewed from the viewing side) adjacent to the interpolation target element image corresponding to the first interpolation target pixel. A one-dot chain line is an average value of the value of the first pixel and the value of the second pixel calculated by the image processing device 20 according to the viewpoint position. The image processing apparatus 20 calculates the average value (for example, “217”) when the average rate of change is equal to or less than a predetermined threshold value from the calculation result represented by the alternate long and short dash line, as the value of the first interpolation target pixel. Get as.

  Further, the image processing device 20 removes the first interpolation target pixel in the interpolation target element image from the interpolation target pixels corresponding to the gap between the imaging surfaces shown in FIG. 3 (eight times the pixel pitch of the imaging surface). The value of the interpolation target pixel is calculated by interpolation processing of adjacent pixels or surrounding pixels. For example, the image processing apparatus 20 calculates the value of the second interpolation target pixel by linearly interpolating the values of surrounding pixels with respect to the second interpolation target pixel. Specifically, for example, when obtaining the value of the second interpolation target pixel from four pixels adjacent to the second interpolation target pixel, the image processing device 20 calculates the average value of the pixel values of these four pixels. The value of the second interpolation target pixel is obtained. Then, the image processing apparatus 20 stores the calculated value of the second interpolation target pixel in the corresponding address area in the storage unit.

  As described above, the imaging apparatus according to the present embodiment corresponds to each of the plurality of imaging elements 102 arranged in a matrix with a gap between the imaging surfaces, and the imaging surfaces of the plurality of imaging elements 102. And a plurality of imaging lens arrays 101 provided with gaps. In addition, the imaging apparatus uses the value of the first interpolation target pixel in the interpolation target element image for the virtual element lens corresponding to the gap between the imaging lens array 101 as the principal point of the first interpolation target pixel and the virtual element lens. For each of a plurality of viewpoints on a line passing through the image processing device 20 based on an average value of a plurality of pixel values obtained through the viewpoint and principal points of element lenses around the virtual element lens. .

  In this way, by configuring the imaging apparatus using a plurality of multi-lens imaging units 100 having a predetermined resolution, it is possible to highly accurately interpolate pixels in a region where light from a subject cannot be obtained, An imaging apparatus having a resolution that is extremely higher than the above-described resolution can be realized.

  Further, in the photographing apparatus, the image processing device 20 changes the average value in the average value of the plurality of pixels for each of the plurality of viewpoints in the order of moving away from the principal point of the virtual element lens with respect to the interpolation target element image. The average value when the rate is equal to or lower than the threshold value may be the value of the first interpolation target pixel. With this configuration, the imaging apparatus can interpolate the interpolation target element image with high accuracy based on the surrounding element images of the interpolation target element image.

  Further, in the photographing apparatus, the image processing device 20 calculates the value of the second interpolation target pixel excluding the first interpolation target pixel in the interpolation target element image from the interpolation target pixels corresponding to the gap between the imaging surfaces, from among the pixels. You may calculate by an insertion process.

[Other embodiments]
The image processing apparatus 20 calculates the value of the first interpolation target pixel in the interpolation target element image for the virtual element lens corresponding to the gap between the imaging lens array 101, and the principal point of the first interpolation target pixel and the virtual element lens. For each of a plurality of viewpoints on a line that passes through and obtained by calculating an average value from an average value of a plurality of pixel values obtained through the viewpoint and principal points of element lenses around the virtual element lens Also good. Specifically, taking the interpolation target pixel B (3) of FIG. 6 as an example, the image processing apparatus 20 calculates the average values B (3) [1], B (3) [2], n for each of the n viewpoints. .., B (3) [n] is calculated as an average value, and the average value is determined as the value of the interpolation target pixel B (3).

  Also with this configuration, the photographing apparatus can generate integral image data for obtaining a high-definition aerial image by integral photography.

  In the embodiment described above, the imaging lens array 101 and the display lens array are configured by arranging element lenses in a square lattice pattern. The arrangement pattern of the element lenses in these lens arrays is not limited to a square lattice arrangement, and may be a delta arrangement, for example. When the element lenses are arranged in a delta shape without gaps, assuming that the diameter of the element lens is D, for example, the lens pitch in the X-axis direction is D for each row, and D / 2 between adjacent rows. The lens pitch in the Y-axis direction is (√3) × D / 2. The lens array constituted by the delta arrangement has a higher density of element lenses than the lens array constituted by the square lattice arrangement. Therefore, an imaging display system using a lens array configured by a delta arrangement can obtain high-quality interpolation target element images and interpolation target pixels when interpolation processing is executed.

  The combination of the element lens of the imaging lens array and the element lens of the display lens array may be as follows. That is, either the imaging side or the display side is a gradient index lens, and the other is a convex lens. Alternatively, either one is a convex lens and the other is a concave lens. Alternatively, both the imaging side and the display side may be convex lenses or gradient index lenses, and image processing for inverting the depth of each element image in the image processing apparatus may be executed.

  In the above-described embodiment, the image processing device 20 is an example of the process of calculating the value of the interpolation target pixel in the interpolation target element image with respect to the virtual element lens. The present invention can also be applied to a case where the element lens of the lens array 101 is scratched or dust is attached.

  Further, some functions of the image processing apparatus 20 may be realized by a computer. In this case, a program for realizing the control function is recorded on a computer-readable recording medium, the program recorded on the recording medium is read into a computer system, and the computer system executes the function. May be realized. The computer system includes an operating system (OS) and hardware of peripheral devices. The computer-readable recording medium is a portable recording medium such as a flexible disk, a magneto-optical disk, an optical disk, or a memory card, and a storage device such as a magnetic hard disk or a solid state drive provided in the computer system. Furthermore, a computer-readable recording medium dynamically holds a program for a short time, such as a computer network such as the Internet, and a communication line when transmitting a program via a telephone line or a cellular phone network. In addition, a server that holds a program for a certain period of time, such as a volatile memory inside a computer system serving as a server device or a client in that case, may be included. Further, the above program may be for realizing a part of the above-described functions, and further, may be realized by combining the above-described functions with a program already recorded in the computer system. Good.

  As described above, the photographing apparatus according to the embodiment can generate integral image data for obtaining a high-definition aerial image by integral photography.

  As mentioned above, although embodiment of this invention was explained in full detail with reference to drawings, the specific structure is not restricted to that embodiment, The design of the range which does not deviate from the summary of this invention, etc. are included.

DESCRIPTION OF SYMBOLS 1 Image | photographing display system 10 Multi lens imaging array 20 Image processing apparatus (interpolation processing part)
60 Projector 70 Diffusion plate 80 Display lens array 100 ₁₁ , 100 ₂₁ , 100 ₃₁ , 100 ₁₂ , 100 ₂₂ , 100 ₃₂ , 100 ₁₃ , 100 ₂₃ , 100 ₃₃ Multi-lens imaging unit 101 ₁₁ , 101 ₁₂ , 101 ₁₃ Lens array 102 ₁₁ , 102 ₁₂ , 102 ₁₃ Image sensor 103 Substrate part

Claims (4)

A plurality of imaging elements arranged in a matrix with a gap between imaging surfaces;
A plurality of lens arrays provided with gaps corresponding to the imaging surfaces of the plurality of imaging elements;
A plurality of viewpoints on a line passing through the first interpolation target pixel and the principal point of the virtual element lens, the value of the first interpolation target pixel in the interpolation target element image for the virtual element lens corresponding to the gap between the lens arrays. For each, an interpolation processing unit that calculates based on an average value of a plurality of pixel values obtained through the viewpoint and a principal point of an element lens adjacent to the virtual element lens;
An imaging device comprising: In the average value of a plurality of pixel values for each of the plurality of viewpoints in the order of moving away from the principal point of the virtual element lens with respect to the interpolation target element image, the interpolation processing unit has a change rate of the average value. The average value when the value is equal to or less than the threshold is the value of the first interpolation target pixel,
The imaging device according to claim 1. The interpolation processing unit calculates an average value of a plurality of pixel values for each of the plurality of viewpoints as a value of the first interpolation target pixel;
The imaging device according to claim 1. The interpolation processing unit calculates a value of a second interpolation target pixel excluding the first interpolation target pixel in the interpolation target element image from the interpolation target pixel corresponding to the gap between the imaging surfaces by pixel interpolation processing. To
The imaging device according to any one of claims 1 to 3.

Images