JP2015186189A - Imaging apparatus, control method thereof, and image reproducing device - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an imaging apparatus and an imaging method which prevent light-field camera information quantity reduction and accuracy deterioration, and to provide an image reproducing device.SOLUTION: In a light-field camera, a stop is set to an open stop (first stop value) (step 31) and a subject is imaged (step 32). First imaging signal data obtained by imaging are temporarily stored in a memory. Continuously, the stop is set to a second stop value of a less aperture amount than the open stop (step 33) and the subject is imaged (step 34). Second imaging signal data obtained by imaging are also temporarily stored in the memory. The first imaging signal data and the second imaging signal data are stored in one file. Since an information quantity of the first imaging signal data is more, in a case where re-focusing cannot be performed from the second imaging signal data, the re-focusing can be performed by using the first imaging signal data.

Description

  The present invention relates to an imaging apparatus, a control method thereof, and an image reproduction apparatus.

  A camera called a light field camera has been realized. In the light field camera, the image data obtained from the image sensor includes information on the light traveling direction in addition to the light intensity distribution on the light receiving surface. As a result, observation images from a plurality of viewpoints and directions can be reconstructed. In the light field camera, the amount of information and the accuracy deteriorate due to changes in the position and size of the pupil of the imaging optical system. For this reason, there is one that reduces these information amount reduction and accuracy deterioration (Patent Document 1).

JP 2013-48309

  However, in the device described in Patent Document 1, it is necessary to change the distance between the microlens array and the image sensor in accordance with the pupil region of the imaging optical system. Is required, which increases the hardware load and cost, making it difficult to implement as a general-purpose digital camera. In addition, the pixel described in Patent Document 1 has a problem that the amount of information itself is small because there is a pixel called a dead zone where a light ray does not enter.

  An object of the present invention is to prevent a decrease in the amount of information and a deterioration in accuracy of a light field camera without increasing the hardware load.

  An image pickup apparatus according to the present invention is arranged between an image pickup lens, an aperture, an image pickup element in which photoelectric conversion elements are arranged, an image pickup lens and an image pickup element, and one microlens is provided for a plurality of photoelectric conversion elements of the image pickup element. A microlens array and a diaphragm for allowing light to enter a photoelectric conversion element that is located at the same position relative to the pupil position of the imaging lens through which the light from the subject passes among the plurality of photoelectric conversion elements. First imaging control means for capturing an image with the image sensor as the first aperture value, a first aperture for capturing the image with the image sensor as the second aperture value, which is an aperture amount smaller than the aperture amount of the first aperture value. 2 imaging control means, first imaging signal data (first imaging signal, first image data) obtained from the imaging device by imaging based on the first imaging control means, and Recording control means for recording second image pickup signal data (second image pickup signal, second image data) obtained from the image pickup device by image pickup based on the second image pickup control means in one file is provided. Features.

  The present invention also provides an imaging method. In other words, an imaging lens, an aperture, an imaging element in which photoelectric conversion elements are arranged, and an imaging lens and an imaging element are arranged, and one microlens is assigned to a plurality of photoelectric conversion elements of the imaging element. Control of an imaging device including a microlens array that causes a light beam to be incident on a photoelectric conversion element existing at the same position relative to a pupil position of an imaging lens through which a light beam from a subject passes among a plurality of photoelectric conversion elements In the method, the first imaging control means causes the imaging device to take an image with the aperture as the first aperture value, and the second imaging control means sets the aperture to an aperture amount smaller than the aperture value of the first aperture value. As the second aperture value, the image pickup device picks up an image, and the recording control unit uses the first image pickup signal data and the first image pickup signal data obtained from the image pickup device by the image pickup based on the first image pickup control unit. It is intended to record the second image pickup signal data obtained from the imaging device by the imaging based on the second imaging control means in a single file.

  The first aperture value is, for example, the aperture value of the aperture stop.

  On the light receiving surface of the photoelectric conversion element, a red filter having a characteristic of transmitting a red light component, a green filter having a characteristic of transmitting a green light component, a blue filter having a characteristic of transmitting a blue light component, or A cyan filter having a characteristic of transmitting a cyan light component, a magenta filter having a characteristic of transmitting a magenta light component, or a yellow filter having a characteristic of transmitting a yellow light component may be formed. In this case, the recording control means will record a signal obtained from the photoelectric conversion element on which the green filter or the yellow filter is formed as a first image signal data in one file.

  There may be further provided a phase difference correlation calculating means for calculating a phase difference correlation obtained from the first image pickup signal obtained in the image pickup device by imaging based on the first recording control means. In this case, the recording control means will record the phase difference correlation calculated by the phase difference correlation calculation means in one file.

  Refocus image generation means for generating a refocus image in which a designated portion is focused out of the subject image captured by the image sensor using the first imaging signal data and the second imaging signal data. You may prepare.

  A stereoscopic image generation unit that generates a stereoscopic image using the first imaging signal data and the second imaging signal data may be further provided.

  Reading means for reading the first imaging signal data recorded by the imaging device and the recorded second imaging signal data, and the first imaging signal data and the second imaging signal data read by the reading means And refocus image generation means for generating a refocus image in which a specified portion is in focus among the subject images represented by the first image signal data or the second image signal data. .

  According to the present invention, the aperture is set to the first aperture value based on the first imaging control means, and the subject is imaged by the imaging device. Subsequently, based on the second imaging control means, the aperture is set to a second aperture value that is smaller than the aperture value of the first aperture value, and the subject is imaged by the imaging device. The first imaging signal data obtained by imaging based on the first imaging control means and the second imaging signal data obtained by imaging based on the second imaging control means are recorded in one file. Since the first imaging signal data and the second imaging signal data obtained by imaging with different aperture values are obtained, by using the first imaging signal data, without increasing the hardware load, Reduces information volume and accuracy degradation.

It is a block diagram which shows the electric constitution of a light field camera. It is a front view of a micro lens array. It is a front view of an image sensor. It is a front view of an image sensor seen through a microlens array. The relationship between an imaging lens, a microlens array, and an imaging device is shown. The pupil region of the imaging lens is shown. The relationship between a microlens and a photoelectric conversion element is shown. The file structure is shown. It is a flowchart which shows the process sequence of a light field camera. A region where light rays generated on the light receiving surface of the image sensor do not enter is shown. It is an example of the image put together by multi-viewpoint image data. It is an example of the image in which a part is blurred. It is an example of the image in which a part is blurred. It is an example of the image from which the blur is eliminated. It is a front view of an image sensor seen through a microlens array. It is an example of an image. It is a front view of an image sensor seen through a microlens array.

  FIG. 1 is a block diagram showing an electrical configuration of a light field camera (imaging device) 1 according to an embodiment of the present invention.

  The overall operation of the light field camera 1 is controlled by the control device 20.

  The light field camera 1 includes an image pickup element 5 in which an aperture 2A, an image pickup lens 2, and a photoelectric conversion element are arranged. A microlens array 3 is disposed between the imaging lens 2 and the imaging device 5.

  FIG. 2 is a front view of the microlens array 3.

  A plurality of microlenses 4 are arranged in the microlens array 3 in the horizontal direction and the vertical direction.

  FIG. 3 is a front view of the image sensor 5.

  In the image sensor 5, photoelectric conversion elements 6 are arranged in the horizontal direction and the vertical direction. In this embodiment, one photoelectric conversion element group (a plurality of photoelectric conversion elements) composed of a total of 25 (not necessarily 25) in the horizontal direction and the vertical direction (not necessarily 5). 6) One microlens 4 is assigned to 7.

  FIG. 4 is a front view of the image sensor 5 viewed from the subject side.

  As described above, it can be seen that one microlens 4 is assigned to each photoelectric conversion element group 7 composed of 25 photoelectric conversion elements 6.

  FIG. 5 shows a state in which the light beam that has passed through the imaging lens 2 enters the photoelectric conversion element 6 of the imaging element 5.

  The light beam from the subject is refracted by the imaging lens 2 and enters the photoelectric conversion element 6 of the imaging element 5 through the microlens 4 constituting the microlens array 3. The imaging lens 2 is divided into five pupil regions a1 to a5 in the vertical direction in FIG. As shown in FIG. 6, the imaging lens 2 is divided into 25 pupil regions in consideration of the horizontal direction and the vertical direction, but in FIG. 5, since it is viewed from the side, five pupil regions divided in the vertical direction for convenience. a1 to a5 are represented. The light beam L1 that has passed through the pupil region a1 enters the photoelectric conversion element p1 in the photoelectric conversion element group 7. Similarly, light rays L2, L3, L4 and L5 that have passed through pupil region a2, pupil region a3, pupil region a4 or pupil region a5 are incident on photoelectric conversion elements p2, p3, p4 and p5 of photoelectric conversion element group 7, respectively. To do. The same applies not only to the specific photoelectric conversion element group 7 but also to all the photoelectric conversion element groups 7 of the imaging element 5.

  FIG. 6 shows a state where the pupil region of the imaging lens 2 is divided into 25 photoelectric conversion elements 6 constituting the photoelectric conversion element group 7.

  The imaging lens 2 includes 25 pupil regions a11 to a15 (pupil region a1), a21 to a25 (pupil region a2), a31 to a35 (pupil region a3), a41 to a45 (pupil region a4), and a51 to a55 ( The pupil region is divided into 25 pupil regions a5).

  FIG. 7 shows the relationship between the microlens 4 and the 25 photoelectric conversion elements 6 constituting the photoelectric conversion element group 7.

  The 25 photoelectric conversion elements 6 constituting the photoelectric conversion element group 7 are denoted by reference numerals p11 to p15 (photoelectric conversion element p1), p21 to p25 (photoelectric conversion element p2), p31 to p35 (photoelectric conversion element p3), and p41 to p45. (Photoelectric conversion element p4) and p51 to p55 (photoelectric conversion element p5).

  Light rays that have passed through the pupil regions a11, a12, a13, a14, or a15 of the imaging lens 2 are incident on the photoelectric conversion elements p11, p12, p13, p14, or p15 by the microlens array 3. Similarly, the light beam that has passed through the pupil regions a21, a22, a23, a24, or a25 of the imaging lens 2 is incident on the photoelectric conversion elements p21, p22, p23, p24, or p25 by the microlens array 3. Light rays that have passed through the pupil regions a31, a32, a33, a34, or a35 enter the photoelectric conversion elements p31, p32, p33, p34, or p35 by the microlens array 3, and the pupil regions a41, a42, a43 of the imaging lens 2 , A44 or a45 is incident on the photoelectric conversion elements p41, p42, p43, p44 or p45 by the microlens array 3 and passes through the pupil regions a51, a52, a53, a54 or a55 of the imaging lens 2. The light beam is incident on the photoelectric conversion elements p51, p52, p53, p54 or p55 by the microlens array 3. The pupil regions a11 to a15, a21 to a25, a31 to a35, a41 to a45 and a51 to a55 of the imaging lens 2 as described above, the photoelectric conversion elements p11 to p15, p21 to p25, and p31 to p35 of the photoelectric conversion element group 7. , P41 to p45 and p51 to p55 are not limited to the specific photoelectric conversion element group 7, but are established for all the photoelectric conversion element groups 7 included in the imaging element 5. As described above, the microlens array 3 is present at the same position relative to the pupil position of the imaging lens 2 through which the light beam from the subject passes among the plurality of photoelectric conversion elements 6 constituting the photoelectric conversion element group 7. The light is incident on the photoelectric conversion element 6 that performs the above operation.

  Returning to FIG. 1, when the imaging mode is set by the switch included in the operation device 17, live view imaging is performed, and the images are relatively located at the same position by imaging control by the control device (imaging control means) 20. A signal obtained from the photoelectric conversion element 6 becomes an output signal of the imaging element 5. The image signal output from the image sensor 5 is subjected to analog signal processing such as correlated double sampling, signal amplification, and analog / digital conversion in the analog signal processing circuit 11, and is output as digital image data. The digital image data output from the analog signal processing circuit 11 is subjected to digital signal processing such as reference level adjustment in the digital signal processing circuit 12. The digital image data output from the digital signal processing circuit 12 is subjected to predetermined image processing in the image processing device 13. The digital image data output from the image processing device 13 is given to the display device 18 via the control device 20.

  In the case of a light field camera, light rays from different viewpoints (different directions) are incident on each of the plurality of photoelectric conversion elements 6 constituting the photoelectric conversion element group 7, and thus are obtained from all the photoelectric conversion elements 6. When a live view image is displayed using a signal, the entire image becomes blurred. In this embodiment, a live view image represented by a signal obtained from the photoelectric conversion element 6 on which only light rays from a specific viewpoint (specific direction) are incident is displayed by the control device (display control unit) 20 on the display device 18. Is displayed on the display screen. A sharp live view image 30 is displayed.

  When a shutter release button included in the operation device 17 is pressed, the aperture 2A is set to a first aperture value (for example, an open aperture value, but not an open aperture value) by the control device 20. The subject is imaged by the imaging device 5 under the control of the control device (first imaging control means) 20 with the aperture 2A set to the first aperture value. The first imaging signal data is obtained from the imaging device 5 by imaging. The obtained first imaging signal data is given to the memory 14 via the analog signal processing circuit 11, the digital signal processing circuit 12, and the image processing device 13, and is temporarily stored.

  The aperture 2A is set to a second aperture value by the control device 20. The aperture amount of the second aperture value is smaller than the aperture amount of the first aperture value, and may be an aperture value designated by the user, or the first imaging signal data or the first image data obtained as described above. It may be an aperture value determined based on imaging signal data obtained from live view imaging performed before one imaging signal data is obtained. Whether the aperture value is specified by the user or the aperture value determined based on the imaging signal data obtained from live view imaging, the aperture amount is smaller than the aperture value of the first aperture value. Thus, the second aperture value is determined. When the second aperture value is designated by the user, or when the second aperture value is determined based on imaging signal data obtained from live view imaging, the second aperture value is determined (designated) first. The first aperture value may be determined such that the aperture amount is larger than the aperture amount of the determined second aperture value. When the second aperture value is an open aperture value, imaging with the first aperture value (or imaging with the second aperture value) may not be performed.

  The subject is imaged by the imaging device 5 under the control of the control device (second imaging control means) 20 with the aperture 2A set to the second aperture value. Second imaging signal data is obtained from the imaging device 5 by imaging. The obtained second imaging signal data is given to the memory 14 via the analog signal processing circuit 11, the digital signal processing circuit 12, and the image processing device 13 and stored temporarily, as with the first imaging signal data. Is done.

  The first imaging signal data and the second imaging signal data temporarily stored in the memory 14 are read from the memory 14 and stored (recorded) in one file in the image processing device 13. Such a file is recorded on the memory card 16 by the recording / playback control device (recording control means) 15. The recording / reproducing control device 15 can also read a file recorded on the memory card 16. Since the first imaging signal data and the second imaging signal data described above are stored in the file, the first imaging signal data and the second imaging signal data are read from the memory card 16 (reading means). ). As will be described later, among the subject image represented by the first imaging signal data or the second imaging signal data by the control device 15 from the read first imaging signal data and second imaging signal data. , A refocus image in which the designated portion is in focus (not blurred) is generated (refocus image generation means).

  FIG. 8 is an example of a file structure (data structure) for storing (recording) the first imaging signal data and the second imaging signal data.

  The file includes a header area for storing file management information and an image data recording area. Distance data (distance map) is stored in the header area. Based on the first imaging signal data obtained by imaging with the first aperture value, distance data (distance map) representing the distance from the camera 1 to the subject is obtained in the image processing device 13 for each pixel constituting the image. Generated. The generated distance data is stored in the header area. The first image signal data and the second image signal data are recorded in the image data recording area. However, the first imaging signal data may be recorded in the header area.

  FIG. 9 is a flowchart showing a processing procedure of the light field camera 1.

  When a shutter command is given to the light field camera 1 by a shutter release button or the like, the control device 20 sets the aperture 2A to the first aperture value (open aperture value) (step 31). The subject is imaged with the first aperture value (step 32), and the first imaging signal data is obtained as described above. The obtained first imaging signal data is temporarily stored in the memory 14.

  Subsequently, the control device 20 sets the aperture 2A to the second aperture value (step 33). As described above, the opening amount of the second aperture value is smaller than the opening amount of the first aperture value. The subject is imaged with the second aperture value (step 34), and the second imaging signal data is obtained as described above. The obtained second imaging signal data is temporarily stored in the memory 14.

  The first imaging signal data and the second imaging signal data stored in the memory 14 are read by the control device 20, and the read first imaging signal data and second imaging signal data are in one file. Is recorded (stored) (step 35). It should be noted that the first image signal data is temporarily stored in the memory 14 as the first aperture value for the Live View shooting before the shutter release button is pressed, and the second aperture value is set after the shutter release button is pressed. 2 is temporarily recorded in the memory 14, and if the aperture amount of the first aperture value is larger than the aperture amount of the second aperture value, the read first image signal data and the second The imaging signal data may be recorded in one file.

  FIG. 10 shows a portion where no light beam is incident on the light receiving surface of the image sensor 5.

  A region 8 indicated by a circle indicates a region where light beams that have passed through the imaging lens 2 and one microlens 4 are incident on the light receiving surface of the imaging device 5. These areas 8 increase as the amount of opening of the diaphragm 2 increases, and decrease as the aperture of the diaphragm 2 decreases. On the light receiving surface of the image sensor 5, a region 9 (shown by hatching) excluding these regions 8 is a region where no light beam is incident because the diaphragm 2 is narrowed (because the aperture amount is reduced).

  Depending on the opening amount of the second aperture value (when the opening amount is small), there may be a region 9 where no light beam is incident on the light receiving surface of the image sensor 5 as shown in FIG. In this case, since the information from the area 9 cannot be obtained, there arises a problem that the distance information to each subject is lowered, and a sharp image cannot be obtained when refocusing is performed based on the second imaging signal data. Or the accuracy may deteriorate. However, in this embodiment, since imaging is performed with the first aperture value having a larger aperture amount than the aperture amount of the second aperture value, even if there is a region 9 that is not incident in imaging with the second aperture value. By refocusing, a sharp image can be obtained without blurring a desired portion.

  FIG. 11 is an example of a multi-viewpoint image represented by the second imaging signal data obtained by imaging with the second aperture value.

  Although the multi-viewpoint image 40 has an aperture amount smaller than the open aperture value, it is represented by light rays from various directions, so the image is totally blurred.

  When an image in which a desired main subject is not blurred is generated from such a multi-viewpoint image 40, the refocus position of the main subject is designated by the user. The distance to the designated main subject is read from distance data (a distance map, which is stored in a file in association with the second imaging signal data, that is, multi-viewpoint image data).

  As shown in FIGS. 12 and 13, two images 41 and 44 in which the main subject exists at a distance close to the read distance are generated from the second imaging signal data. FIG. 12 shows an image 41 in which the person 42 in the back side from the camera 1 is in focus at the time of imaging, and the person 43 in front is not in focus. FIG. 13 shows an image 44 in which the person 42 in the back side from the camera 1 at the time of imaging is not in focus and the person 43 in front is in focus.

  Further, when the aperture amount is small and the area 8 in FIG. 10 is small, the depth of field is deep, so that a focused image can be obtained. However, in this case, since the distance information to each subject is small, Depending on the distance, the blurring process may not be possible. Even in such a case, by obtaining the distance information with the first aperture value (open aperture), the blurring process can be performed according to the distance to the subject.

  The characteristics will be further explained with an example. When shooting a moving subject such as a flow of water, an image showing the atmosphere in which the water is flowing cannot be obtained unless the shutter is opened for a predetermined time. If the shutter is left for a predetermined time in a bright state (such as when shooting a waterfall landscape in a clear state), a so-called whiteout (overexposed) image will result. If the image is taken with a high-speed shutter with an open aperture), the second aperture value has no overexposure (overexposure), and the shutter can be opened for a predetermined time to obtain the second image signal data, the first image signal data By using the distance information in the image to blur only a specific area according to the distance, it is difficult for a normal digital camera to blur only the area where water flows and focus the background. Image processing can also be facilitated.

  By refocusing the image 41 and the image 44 shown in FIGS. 12 and 13, an image 50 in which the persons 42 and 43 are not blurred is obtained as shown in FIG.

  As described above, when the two images 41 and 44 are generated, the second imaging signal data cannot be obtained from the region 9 (see FIG. 10) where the light does not enter. For this reason, in this embodiment, the first imaging signal data is used as the data to be obtained in such a region 9. As a result, two images 41 and 44 that do not lack the amount of information are generated.

  FIG. 15 is a front view of the image sensor 5 as seen from the subject side, and corresponds to FIG. FIG. 16 is an example of an image represented by a part of the second imaging signal data.

  Of the photoelectric conversion elements 6A and 6B indicated by hatching, an image 50A indicated by a broken line in FIG. 16 is obtained by the second imaging signal data obtained from one photoelectric conversion element 6A. Similarly, an image 50B indicated by a solid line in FIG. 16 is obtained by the second imaging signal data obtained from the other photoelectric conversion element 6B. When the image 50A and the image 50B are displayed at the same time, as shown in FIG. 16, the adult hair portion is shifted by an amount corresponding to the phase difference Δ. The phase difference Δ can be calculated by performing the correction process of the phase difference correlation between the images 50A and 50B. When the image 50A or the image 50B is shifted by the phase difference Δ, an image without deviation (no blur) in the adult hair portion is obtained. Needless to say, if the image is not shifted by the second imaging signal data obtained from all the photoelectric conversion elements 6, the entire image 40 is blurred as shown in FIG.

  At the time of imaging the second imaging signal data, as shown in FIG. 10, a photoelectric conversion element 6 in which no light enters may be generated. Therefore, the second imaging signal data from the photoelectric conversion element 6 in which no light enters. I can't get it. For this reason, the above-described images 50A and 50B represented by the second imaging signal data to be obtained from the photoelectric conversion element 6 where no light beam is incident may not be obtained. In this embodiment, an image that cannot be obtained at the time of imaging the second imaging signal data is formed from the first imaging signal data, and the image is shifted by the image processing device 13 (refocus image generating means), A refocus image in which a designated portion (for example, an adult face portion) is focused is generated. Thus, as shown in FIG. 10, there is no second imaging signal data for the region 9 where no light beam is incident, and therefore the first imaging signal data is used as data to be obtained from the region 9. In the first imaging signal data, as shown in FIG. 10, there is no (small) area 9 in which no light enters, so that images obtained from all the photoelectric conversion elements 6 can be shifted to generate a blur-free image. In addition, since the distance information to the subject is known, it is possible to blur other than the main subject.

  In this embodiment, the phase difference correlation is calculated for each pixel constituting the image represented by the first imaging signal data from the first imaging signal data obtained in the imaging device 5 by imaging with the first aperture value. (Value, data) can be calculated by the control device 20 (phase difference correlation calculating means). The calculated phase difference correlation is recorded in the above-mentioned one file as distance data (distance map). Of course, the calculated phase difference correlation itself may be recorded in a file. When an image is picked up with the second aperture value, there is a photoelectric conversion element 6 in which no light beam is incident as shown in FIG. 10, so that it corresponds to all the pixels constituting the image represented by the second image pickup signal data. The phase difference correlation may not be calculated. For this reason, there may be a portion where the distance is not known in the subject represented by the image represented by the second imaging signal data. However, since the first imaging signal data is picked up using the first aperture value whose aperture is larger than the second aperture value, it is a subject portion whose distance is not known from the second imaging signal data. However, the distance is known from the first image signal data. Such distance data (distance map, phase difference correlation) is recorded in the file as described above.

  When the second aperture value is a small aperture with a small aperture amount, the depth of field becomes deep, so that a sharp and clear image can be obtained regardless of the distance to the subject. However, since the distance to the subject can be known from the distance map of the first imaging signal data, a part of the image can be blurred using the image processing device 13 according to the distance.

  Further, an image 50A as shown in FIG. 16 obtained from a photoelectric conversion element having parallax in the horizontal direction (for example, 6A and 6B shown in FIG. 15 and not limited to 6A and 6B if there is parallax in the horizontal direction). By using the left-eye image and the image 50B as the right-eye image, a three-dimensional image can be generated from the images 50A and 50B by the control device 20 (stereoscopic image generating means). In generating a stereoscopic image, data obtained from the photoelectric conversion elements 6 present in the same column may be shifted in the vertical direction. In the case where a stereoscopic image is generated from the second imaging signal data, there is no second imaging signal data in the region 9 where no light ray is incident as shown in FIG. Instead, a stereoscopic image can be generated using the first imaging signal data. Further, since the distance to all parts (most parts) of the subject represented by the stereoscopic image is known, it is possible to give a parallax corresponding to the distance to the stereoscopic image.

  FIG. 17 is a front view of the image sensor 5 as seen from the subject side, and corresponds to FIG.

  A filter having a light transmission characteristic common to the plurality of photoelectric conversion elements 6 (photoelectric conversion element group 7) assigned to the microlens 4 is formed. In FIG. 17, a symbol R is a filter having a characteristic of transmitting a red light component, a symbol B is a filter having a characteristic of transmitting a blue light component, and a symbol G is a characteristic of transmitting a green light component. It is a filter having. In FIG. 17, the filters are Bayer array, but other filters may be used.

  Thus, when a filter is formed on the light receiving surface of the image sensor 5 (light receiving surface of the photoelectric conversion element 6), the first image signal data obtained by imaging with the first aperture value is green. Only those obtained from the photoelectric conversion element 6 on which the filter G having the characteristic of transmitting the light component is formed may be stored in the file as described above. The human eye is most sensitive to the wavelength region corresponding to the green color, and the photoelectric conversion elements 6 on which the filters G are formed are arranged at equal intervals from FIG. Used as imaging signal data.

  In the example shown in FIG. 17, a filter that transmits a red light component, a filter that transmits a blue light component, and a filter that transmits a green light component are formed in the photoelectric conversion element 6. A filter that transmits light, a filter that transmits a magenta light component, and a filter that transmits a yellow light component may be formed in the photoelectric conversion element 6. In that case, the phase difference correlation will be calculated using the first imaging signal data obtained from the photoelectric conversion element 6 in which a filter that transmits the yellow light component is formed.

1 Light field camera (imaging device)
2 Image sensor 2A Aperture 3 Micro lens array
15 Recording / playback control device (recording control means, reading means)
20 Control device (first imaging control means, second imaging control means, phase difference correlation calculation means, refocus image generation means, stereoscopic image generation means)

Claims (8)

Imaging lens,
Aperture,
An image sensor in which photoelectric conversion elements are arranged;
One microlens is allocated to the plurality of photoelectric conversion elements of the image pickup element, which is disposed between the image pickup lens and the image pickup element, and light rays from a subject are out of the plurality of photoelectric conversion elements. A microlens array in which the light beam is incident on a photoelectric conversion element existing at the same position relative to the pupil position of the imaging lens passing therethrough,
First imaging control means for imaging with the imaging element, with the aperture as a first aperture value;
Based on the second imaging control means for imaging by the imaging element, and the first imaging control means, with the aperture set as a second aperture value having an aperture amount smaller than the aperture amount of the first aperture value. Recording control means for recording the first imaging signal data obtained from the imaging element by imaging and the second imaging signal data obtained from the imaging element by imaging based on the second imaging control means, in one file;
An imaging apparatus comprising: The first aperture value is the aperture value of the aperture stop,
The imaging device according to claim 1. On the light receiving surface of the photoelectric conversion element,
A red filter that transmits red light components, a green filter that transmits green light components, a blue filter that transmits blue light components, or a light filter that transmits cyan light A cyan filter having, a magenta filter having a characteristic of transmitting a magenta light component, or a yellow filter having a characteristic of transmitting a yellow light component is formed.
The recording control means is
A signal obtained from a photoelectric conversion element on which a green filter or a yellow filter is formed is recorded in one file as first imaging signal data.
The imaging device according to claim 1 or 2. A phase difference correlation calculating means for calculating a phase difference correlation obtained from a first image pickup signal obtained in the image pickup device by image pickup based on the first recording control means;
The recording control means is
The phase difference correlation calculated by the phase difference correlation calculating means is recorded in the one file.
The imaging device according to any one of claims 1 to 3. Refocused image generation for generating a refocused image in which a designated portion is focused out of the subject image captured by the imaging device, using the first imaging signal data and the second imaging signal data means,
The imaging device according to any one of claims 1 to 4, further comprising: Stereoscopic image generating means for generating a stereoscopic image using the first imaging signal data and the second imaging signal data;
The imaging device according to claim 4, further comprising: Reading means for reading the first imaging signal data and the second imaging signal data recorded by the imaging device according to claim 1, and the first imaging signal data read by the reading means; Refocusing that generates a refocused image in which a specified portion is in focus among the subject image represented by the first imaging signal data or the second imaging signal data from the second imaging signal data Image generation means,
An image reproducing apparatus comprising: An imaging lens, an aperture, an imaging device in which photoelectric conversion elements are arranged, and a microlens allocated to a plurality of photoelectric conversion elements of the imaging device, arranged between the imaging lens and the imaging element. And a microlens array that causes the light beam to enter a photoelectric conversion element that is located at the same position relative to the pupil position of the imaging lens through which the light beam from the subject passes among the plurality of photoelectric conversion elements. In the control method of the imaging device,
The first imaging control means causes the imaging device to take an image with the aperture as the first aperture value,
The second imaging control means causes the imaging device to capture an image of the aperture as a second aperture value having an aperture smaller than the aperture of the first aperture value,
The recording control means has a first imaging signal data obtained from the imaging element by imaging based on the first imaging control means and a second imaging obtained from the imaging element by imaging based on the second imaging control means. Record signal data in one file,
Control method of imaging apparatus.

Images