JP2017092529A - Imaging apparatus and imaging method - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an imaging apparatus capable of assisting photographing, so as to obtain an excellent reconstructed image.SOLUTION: The imaging apparatus includes imaging means for acquiring image data capable of generating a reconstructed image whose focal position is readjusted, reconstruction means for generating the reconstructed image whose focal position is readjusted from the image data, detection means for detecting a distance up to a subject, and control means for allowing display means to display a screen indicating a positional relation between the subject and a predetermined refocus surface on the basis of the distance up to the subject and the position of the predetermined refocus surface or for allowing the display means to display the screen indicating the focusing degree of the subject in the reconstructed image on the basis of the distance between a refocus surface and the subject.SELECTED DRAWING: Figure 8

Description

  The present invention relates to an imaging apparatus and an imaging method.

In recent years, an imaging apparatus capable of acquiring not only light intensity distribution but also information on the incident direction of light, that is, a light field camera has been proposed.
When a microlens array is arranged between a photographing lens and an image sensor and one microlens is made to correspond to a plurality of pixels, light that has passed through the microlens is acquired by a plurality of pixels in each incident direction (patent) Reference 1).

  By applying a technique called light field photography to the image signal thus obtained, that is, light field information, an arbitrary virtual image plane, that is, an image focused on a refocus plane, It can be reconstructed after shooting. Since the light field camera does not need to be in focus at the time of shooting, the shooting can be performed without missing a photo opportunity.

JP 2007-4471 A

However, the conventional technique may not always obtain a sufficiently good reconstructed image.
An object of the present invention is to provide an imaging apparatus and an imaging method that can assist imaging so that a good reconstructed image can be obtained.

  According to an aspect of the present invention, an imaging unit that acquires image data capable of generating a reconstructed image whose focus position has been readjusted, and the reconstructed image whose focus position has been readjusted as the image Based on the reconstruction means generated from the data, the detection means for detecting the distance to the subject, the distance to the subject, and the position of the predetermined refocus surface, the subject and the predetermined refocus surface A screen showing the positional relationship of the subject is displayed on the display means, or a screen showing the degree of focus of the subject in the reconstructed image is displayed on the display means based on the distance between the refocus plane and the subject There is provided an imaging device characterized by comprising control means for controlling the image pickup device.

  According to the present invention, a screen showing the positional relationship between the subject and the predetermined refocus plane or a screen showing the degree of focus of the subject in the reconstructed image is displayed, so that a good reconstructed image can be obtained. You can assist shooting.

1 is a perspective view illustrating an appearance of an imaging apparatus according to a first embodiment of the present invention. 1 is a block diagram illustrating a configuration of an imaging apparatus according to a first embodiment of the present invention. It is a top view which shows a part of imaging part. It is the schematic which shows the relationship between an imaging lens, a micro lens array, and an imaging part. It is the schematic which shows the positional relationship of an imaging device, a refocus surface, and a to-be-photographed object. It is a figure for demonstrating refocusing. It is a figure for demonstrating refocusing. It is a figure which shows the example of the imaging | photography assistance screen in the imaging device by 1st Embodiment of this invention. 3 is a flowchart illustrating an operation of the imaging apparatus according to the first embodiment of the present invention. It is a figure for demonstrating control of an aperture value. It is a figure which shows the example of the imaging | photography assistance screen in the imaging device by 2nd Embodiment of this invention. It is a figure which shows the example of the imaging | photography assistance screen in the imaging device by 3rd Embodiment of this invention.

Hereinafter, preferred embodiments of the present invention will be described with reference to the drawings.
[First Embodiment]
An imaging apparatus according to a first embodiment of the present invention will be described with reference to FIGS. FIG. 1 is a perspective view showing an appearance of the imaging apparatus according to the first embodiment of the present invention.
The imaging apparatus 100 according to the present embodiment is a digital camera (electronic camera), more specifically, a light field camera.
As shown in FIG. 1, a display unit 28 for displaying images and various information is provided on the back surface of the imaging apparatus 100. Each unit of the imaging apparatus 100 is provided with an operation unit 70 for receiving various operations by the user. The operation unit 70 includes operation members such as various switches, various buttons, and a touch panel. The shutter button 61 is for instructing photographing and constitutes a part of the operation unit 70. The mode switch 60 is for switching various modes, and constitutes a part of the operation unit 70. The controller wheel 73 is an operation member that can be rotated and constitutes a part of the operation unit 70. The power switch 72 is for switching the power on / off.

  The imaging apparatus 100 is provided with a connector 112 for connecting the imaging apparatus 100 and other devices by a connection cable 111. Further, the imaging apparatus 100 is provided with a slot 203 for storing the recording medium 200. Examples of the recording medium 200 include a memory card. The recording medium 200 stored in the slot 203 can communicate with the imaging device 100. The slot 203 is provided with a lid 202.

FIG. 2 is a block diagram illustrating the configuration of the imaging apparatus 100 according to the present embodiment.
The lens unit 150 includes a photographing lens 103. The distance measuring sensor 104 is for acquiring distance information of the subject space imaged by the imaging unit 22. As a technique for acquiring distance information, for example, a time of flight (TOF) method can be used. The time-of-flight method is a method of detecting the distance based on the time until the irradiated laser light is reflected by the subject and returned. In addition, the technique used when acquiring distance information is not limited to the time-of-flight method. In the imaging apparatus 100 according to the present embodiment, distance information is acquired at a resolution corresponding to the pixel pitch of the imaging unit 22, and the acquired distance information is held in the memory 32. The distance measuring sensor 104 and the system control unit 50 described later can function together as a detection unit that detects the distance from the imaging device 100 to the subject 810 (see FIG. 5).

  A shutter 101 is disposed in front of the imaging unit 22. The imaging unit 22 includes a CCD image sensor, a CMOS image sensor, or the like that converts an optical image into an electrical signal. An A / D converter (Analog to Digital Converter) 23 converts an analog signal output from the imaging unit 22 into a digital signal. The imaging unit 22 functions as an imaging unit that acquires image data that can generate a reconstructed image with the focus position readjusted.

  The image processing unit 24 performs resize processing such as predetermined pixel interpolation and reduction, color conversion processing, and the like on the data from the A / D converter 23 or the data from the memory control unit 15. Further, the image processing unit 24 performs a predetermined calculation process using the image data obtained by the imaging unit 22. The system control unit 50 performs exposure control and the like based on the calculation result obtained by the image processing unit 24. Specifically, the system control unit 50 performs automatic exposure (AE) processing, flash pre-flash (EF) processing, and the like. The image processing unit 24 further performs a predetermined calculation process using the image data obtained by the imaging unit 22, and an auto white balance (AWB) process based on the calculation result obtained by the calculation process. Also do. The image processing unit 24 also performs processing for generating a reconstructed image (refocus image) reconstructed so as to focus on a desired refocus plane using the light field RAW data held in the memory 32. The image processing unit 24 can function as a reconstruction unit that generates a reconstructed image whose focus position is readjusted from the image data acquired by the imaging unit 22. The refocus plane position is instructed by the system control unit 50 or the like.

  Data output from the A / D converter 23 is written into the memory 32 via the image processing unit 24 and the memory control unit 15. Specifically, image data acquired by the imaging unit 22 and converted into digital data by the A / D converter 23 is stored in the memory 32. Image data to be displayed on the display unit 28 is also stored in the memory 32. The memory 32 has a storage capacity sufficient to store a predetermined number of still images, a moving image and sound for a predetermined time.

The memory 32 also serves as an image display memory (video memory). A D / A converter (Digital to Analog Converter) 13 converts image display data stored in the memory 32 into an analog signal and supplies the analog signal to the display unit 28. Thus, the image display data stored in the memory 32 is displayed by the display unit 28 via the D / A converter 13.
As the display unit 28, for example, a liquid crystal display (LCD) is used. The display unit 28 performs display according to the analog signal from the D / A converter 13.

The signal digitized by the A / D converter 23, that is, the digital signal is stored in the memory 32 after being subjected to predetermined signal processing by the image processing unit 24. The digital signal stored in the memory 32 is converted into an analog form by the D / A converter 13 and sequentially transferred to the display unit 28, whereby the display unit 28 functions as an electronic viewfinder that displays a through image.
As the display unit 28, a 3D display or the like corresponding to stereoscopic display can be used. In this case, display data corresponding to stereoscopic display is generated by the image processing unit 24, the display data is written in the memory 32, and the display data is transferred to the display unit 28 via the D / A converter 13. 3D display is performed.

Image data composed of light field RAW data is input to the compression / decompression processing unit 35, and the compression / decompression processing unit 35 performs compression processing and expansion processing on the input image data. Examples of the compression processing method include a JPEG (Joint Photographic Experts Group) method.
The nonvolatile memory 56 is an electrically erasable and recordable memory. For example, an EEPROM (Electrically Erasable Programmable Read-Only Memory) is used. The nonvolatile memory 56 stores programs, constants, and the like for operating the system control unit 50.

The system control unit 50 controls the entire imaging apparatus 100 according to the present embodiment. As the system control unit 50, for example, a CPU (Central Processing Unit) is used. Various processes are performed by the imaging apparatus 100 by the system control unit 50 executing the program recorded in the nonvolatile memory 56. As the system memory 52, for example, a RAM (Random Access Memory) is used. The system memory 52 stores constants and variables for operating the system control unit 50. Further, the program read from the nonvolatile memory 56 is also developed in the system memory 52. The system control unit 50 also performs display control by controlling the memory 32, the D / A converter 13, the display unit 28, and the like.
The system timer 53 is a time measuring unit that measures the time used for various controls and the time of a built-in clock.

The touch panel 36 is disposed on the upper surface of the display unit 28. As a detection method in the touch panel 36, a resistance film method, a capacitance method, or the like can be used. The position where the touch panel 36 is touched with a finger is detected by the system control unit 50 and converted into coordinates or the like.
The mode switch 60, the shutter button 61, the shutter switch 62, and the operation unit 70 are operation means for inputting various operation instructions to the system control unit 50. The mode switch 60 is for switching the operation mode of the system control unit 50 to any one of a still image recording mode, a moving image recording mode, a reproduction mode, and the like. The shutter switch 62 is turned on when the shutter button 61 is being operated, that is, in a so-called half-pressed state (shooting preparation instruction), and generates a first shutter switch signal SW1. In response to the first shutter switch signal SW1, the system control unit 50 starts operations such as auto focus (AF) processing, automatic exposure processing, auto white balance processing, and flash pre-flash processing.

  The shutter switch 62 is turned on when the operation of the shutter button 61 is completed, that is, when it is fully pressed (shooting instruction), and generates a second shutter switch signal SW2. The system control unit 50 starts a series of operations from reading the pixel signal from the imaging unit 22 to writing the image data to the recording medium 200 by the second shutter switch signal SW2.

  Each operation member of the operation unit 70 is also used when various operation icons displayed on the display unit 28 are selected and operated. Various functions are appropriately assigned to each operation member of the operation unit 70 for each scene, and each operation member of the operation unit 70 functions as various function buttons. Examples of the function buttons include an end button, a return button, an image advance button, a jump button, a narrowing button, and an attribute change button.

  One of the plurality of operation members constituting the operation unit 70 is a menu button. When the menu button is pressed, a menu screen for performing various settings is displayed on the display unit 28. The user can make various settings by operating a four-way button, a SET button, or the like while viewing the menu screen displayed on the display unit 28.

  The controller wheel 73 is used when a selection item is instructed. When the controller wheel 73 is rotated, an electrical pulse signal corresponding to the operation amount is output from the controller wheel 73. The system control unit 50 controls each unit of the imaging apparatus 100 based on the pulse signal output from the controller wheel 73. The system control unit 50 can determine the angle at which the controller wheel 73 is rotated, the number of times the controller wheel 73 is rotated, and the like based on the pulse signal. As the controller wheel 73, for example, a dial type controller wheel that physically rotates in response to a rotation operation by a user and generates a pulse signal can be used. The controller wheel 73 is not limited to a dial type controller wheel. For example, as the controller wheel 73, a controller wheel configured by a touch sensor can be used. The controller wheel 73 configured by the touch sensor does not physically rotate the controller wheel 73 itself, detects the rotation operation of the user's finger on the touch sensor, and outputs a pulse signal or the like corresponding to the rotation operation of the finger. To do. Such a controller wheel 73 is also referred to as a touch wheel.

  The power supply control unit 80 includes a battery detection circuit (not shown), a DC-DC converter (not shown), a switch circuit (not shown) that switches a block to be energized, and the like. The power control unit 80 detects the presence / absence of a battery, the type of battery, the remaining battery level, and the like. Further, the power supply control unit 80 controls the DC-DC converter based on these detection results and instructions from the system control unit 50, and requires necessary voltages for each unit of the imaging apparatus 100 including the recording medium 200. Supply only for a period.

As the power supply unit 30, for example, a primary battery such as an alkaline battery or a lithium battery, or a secondary battery such as a NiCd battery, a NiMH battery, or a Li battery can be used. The power supply unit 30 may be an AC adapter or the like.
The interface 18 is for inputting and outputting data and the like with the recording medium 200. The recording medium 200 is a memory card or the like, and is configured by a semiconductor memory, a magnetic disk, or the like.

  FIG. 3 is a plan view showing a part of the imaging unit 22. As shown in FIG. 3, a pixel array in which a plurality of unit pixels 210 are arranged in a matrix is provided on the imaging surface of the imaging unit 22. In FIG. 3, a portion surrounded by a thick line corresponds to one unit pixel 210. The unit pixel 210 includes a plurality of pixels (divided pixels) 211. Each divided pixel 211 is provided with a photoelectric conversion element (photoelectric conversion unit). Here, a case where one unit pixel 210 is configured by a total of 25 divided pixels 211 of 5 rows × 5 columns is shown as an example. A microlens array 102 including a plurality of microlenses 1020 is arranged on the imaging surface of the imaging unit 22. Each microlens 1020 corresponds to each unit pixel 210. A plurality of divided pixels 211 constituting one unit pixel 210 are associated with one microlens 1020.

  FIG. 4 is a schematic diagram showing a relationship among the photographing lens 103, the microlens array 102, and the imaging unit 22. As shown in FIG. FIG. 4 is a diagram when observed from a direction perpendicular to the optical axis. As shown in FIG. 4, light from each of the pupil regions L <b> 1 to L <b> 5 of the exit pupil of the photographing lens 103 passes through one microlens 1020, and each photoelectric conversion element p <b> 1 of the unit pixel 210 of the imaging unit 22. Are received by .about.p5. One photoelectric conversion element p <b> 1 to p <b> 5 is provided for each of the plurality of divided pixels 211 constituting the unit pixel 210. For example, photodiodes are used as the photoelectric conversion elements p1 to p5. By using the signal obtained by each divided pixel 211 of the imaging unit 22 and reconstructing the image at an arbitrary focal position (refocus plane), an image focused on an arbitrary subject can be generated. it can.

Next, the positional relationship among the imaging apparatus 100, the refocus surfaces 803 to 805, and the subject 810 will be described with reference to FIG.
FIG. 5 is a schematic diagram illustrating a positional relationship among the imaging apparatus 100, the refocus surfaces 803 to 805, and the subject 810. As shown in FIG. 5, a subject 810 to be photographed is located in front of the imaging apparatus 100. FIG. 5 shows a refocus surface 803, a refocus surface 804, and a refocus surface 805.

Next, the generation of a reconstructed image will be described with reference to FIGS. FIG. 6 is a diagram for explaining generation of a reconstructed image. FIG. 6 shows a case where reconstruction is performed so that the refocus plane 803 is in focus.
FIG. 6 shows virtual pixels corresponding to the refocus plane 803, that is, reconstructed pixels P1 to P5. Here, in order to simplify the description, the description will be given focusing on the five unit pixels 210a to 210e, but in reality, the imaging unit 22 is provided with a large number of unit pixels 210. Further, here, for the sake of simplicity, five divided pixels a1 to a5, b1 to b5, c1 to c5, d1 to d5, and e1 to e5 are shown for each unit pixel 210a to 210e. Yes. However, in practice, each of the unit pixels 210a to 210e is provided with 25 divided pixels. Microlenses 1020a to 1020e are positioned above the unit pixels 210a to 210e.
Light beams r1 to r5 that are emitted from the respective pupil regions L1 to L5 of the photographing lens 103 and converge on the reconstructed pixel P3 that is a virtual pixel enter the divided pixels c1 to c5 through the microlens 1020c, respectively. .

By adding the pixel signals (pixel values) output from the respective divided pixels c1 to c5, it is possible to calculate the pixel value S3 in the reconstructed pixel P3. For example, the pixel value S3 in the reconstructed pixel P3 can be calculated by using the following equation (1).
S3 = Σ (pixel value of ck) (1)
However, k = 1-5.
The pixel values S1, S2, S4, and S5 in the reconstructed pixels P1, P2, P4, and P5, which are other virtual pixels, can be calculated in the same manner as described above.

  Next, a case where reconstruction is performed so that the refocus surface 804 is focused will be described with reference to FIG. FIG. 7 is a diagram for explaining generation of a reconstructed image. FIG. 7 shows a case where reconstruction is performed so that the refocus plane 804 is in focus.

  The luminous fluxes r1 to r5 emitted from the respective pupil regions L1 to L5 of the photographing lens 103 and converged on the reconstructed pixel P3 which is a virtual pixel are transmitted to the unit pixels 210a to 210e via the microlenses 1020a to 1020e, respectively. Each incident. Specifically, the light beams r1, r2, d4, and r5 from the pupil regions L1, L2, L4, and L5 are incident so as to straddle a plurality of divided pixels. More specifically, the light beam r5 from the pupil region L5 is incident so as to straddle both the divided pixel a2 and the divided pixel a3. Further, the light beam r4 from the pupil region L4 is incident on the divided pixel b2 and the divided pixel b3. Further, the light beam r2 from the pupil region L2 is incident on the divided pixel d3 and the divided pixel d4. In addition, the light beam r1 from the pupil region L1 is incident on the divided pixel e3 and the divided pixel e4. On the other hand, the light beam r3 from the pupil region L3 does not enter the plurality of divided pixels but enters one divided pixel c3.

The pixel value S3 in the reconstructed pixel P3 can be calculated by weighted addition of pixel signals (pixel values) from the respective divided pixels a2, a3, b2, b3, c3, d3, d4, e3, and e4. It is. For example, the pixel value S3 in the reconstructed pixel P3 can be calculated by using the following equation (2).
S3 = w1 · a2 + w2 · a3 + w3 · b2 + w4 · b3 + c3 + w3 · d4 + w4 · d3 + w2 · e3 + w1 · e4 (2)
However, w1-w4 has shown the weighting coefficient.
The pixel values S1, S2, S4, and S5 in the reconstructed pixels P1, P2, P4, and P5, which are other virtual pixels, can be calculated in the same manner as described above.
As described above, the divided pixels related to the reconstructed pixels P1 to P5 are selected from the divided pixels a1 to e5 provided in the imaging unit 22, and the pixel values of the selected divided pixels are weighted as necessary. By performing addition, a reconstructed image can be generated.

  However, if the calculation of the weighting coefficient for all the reconstructed pixels is performed when the reconstructed image is generated, the processing load when the reconstructed image is generated becomes enormous. In order to reduce the processing load at the time of generating a reconstructed image, it is preferable to previously hold information necessary for generating a reconstructed image in the nonvolatile memory 56 or the like. For this reason, in the present embodiment, information necessary for generating a reconstructed image is stored in advance in the nonvolatile memory 56. Examples of information required when generating a reconstructed image include a weighting coefficient used when performing weighted addition. In addition, as information required when generating a reconstructed image, information on divided pixels related to each reconstructed pixel can be given.

  By the way, it is difficult to previously store such information at all positions from the closest side to the infinity side in the nonvolatile memory 56 because the amount of information is enormous. Therefore, in the present embodiment, such information at discrete positions is held in the nonvolatile memory 56 in advance. Specifically, as information used when a reconstructed image is generated so that the refocus plane 803 is in focus, the following information is exemplified. For example, there is information that the divided pixels related to the reconstructed pixel P3 are c1 to c5. Further, for example, information indicating that the weighting coefficients w1 to w4 are all 1 is mentioned. These pieces of information are stored in advance in the nonvolatile memory 56 or the like.

  According to the present embodiment, information required when generating a reconstructed image is stored in advance in the nonvolatile memory 56 or the like, and information required when generating a reconstructed image is refocused. There is no need to calculate. For this reason, according to this embodiment, the processing load at the time of refocusing can be reduced.

  Next, a photographing assist screen 800 that is a screen for assisting photographing will be described with reference to FIG. FIG. 8 is a schematic diagram illustrating an example of a shooting assist screen 800 in the imaging apparatus according to the present embodiment. Note that the shooting assist screen 800 is displayed on the display unit 28. An image signal (video signal) for displaying the photographing assist screen 800 on the display unit 28 is generated by the system control unit 50 or the like.

  As shown in FIG. 8, a reconstructed image 801 generated by reconstruction is displayed on the photographing assist screen 800. Here, a case where the reconstructed image 801 reconstructed so as to be focused on the refocus plane 803 (see FIG. 5) is displayed while being sequentially updated will be described as an example. In addition, here, a case where three subjects (subject images) 810 to 812 are included in the captured image will be described as an example.

  As shown in FIG. 8, the photographing assist screen 800 includes a figure 100 ′ schematically showing the imaging device 100 and figures 803 ′ to 805 ′ schematically showing refocus surfaces 803 to 805 (see FIG. 5). Is displayed. Here, for convenience of explanation, a figure 100 ′ schematically showing the imaging device 100 may be referred to as an imaging device. Further, figures 803 'to 805' schematically showing the refocus surfaces 803 to 805, respectively, may be referred to as refocus surfaces. For the refocus planes 803 to 805, information necessary for generating a reconstructed image is held in the nonvolatile memory 56 in advance. Here, three refocus surfaces 803 to 805 in which information necessary for generating a reconstructed image is stored in advance in the nonvolatile memory 56 are shown. The number is not limited to three and can be set as appropriate.

  In addition, as shown in FIG. 8, graphics 810 ′ to 812 ′ schematically showing subjects 810 to 812 are also displayed on the photographing assist screen 800. Here, for convenience of explanation, figures 810 ′ to 812 ′ schematically showing the subjects 810 to 812 may be referred to as subjects. The distance from the imaging device 100 to each of the subjects 810 to 812 is detected using the distance measuring sensor 104 as described above. Based on the distance detected using the distance measuring sensor 104, positions of figures 810 ′ to 812 ′ schematically showing the subject are set.

  When the user touches any of the figures 810 ′ to 812 ′ schematically showing the subject with a finger, the system control unit 50 detects a signal from the touch panel 36 disposed on the upper surface of the display unit 28. Based on a signal from the touch panel 36, the system control unit 50 detects which of the plurality of graphics 810 ′ to 812 ′ schematically showing the subject is touched by the user's finger. The system control unit 50 displays a rectangular frame 820 so as to surround the subject corresponding to the touched figure. FIG. 8 shows an example where the figure 811 ′ corresponding to the subject 811 is touched by the user's finger.

  The positions of the subjects 810 and 812 corresponding to the figures 810 ′ and 812 ′ are substantially equal to the positions of the refocus planes 804 and 805, respectively. For this reason, for the subjects 810 and 812 corresponding to the figures 810 ′ and 812 ′, a refocus image with a high degree of focus (focus level) is obtained by using information stored in the nonvolatile memory 56 in advance. be able to. On the other hand, the position of the subject 811 corresponding to the figure 811 ′ is far away from both the refocus plane 804 and the refocus plane 805. For this reason, when refocusing on the subject 811, information previously stored in the nonvolatile memory 56 cannot be used. For this reason, when reconstruction is performed so that the subject 811 is in focus, there is a possibility that a sufficient degree of focus cannot be obtained. In order to obtain a sufficient degree of focus even when the subject 811 is refocused, for example, the following is effective. That is, it is effective to move the subject 811 near any one of the refocus surfaces 803 to 805 in which information necessary for generating a reconstructed image is held in the nonvolatile memory 56 in advance. . However, it is not easy for the user to grasp which subject is far away from the refocus surfaces 803 to 805 without assistance. Therefore, in the present embodiment, the system control unit 50 associates the figures 810 ′ to 812 ′ with the subjects 810 to 812, respectively. When any of the figures 810 ′ to 812 ′ is touched with a finger by the user, the system control unit 50 displays a display for making the subject associated with the touched figure stand out. At 800. For example, when the figure 811 ′ is touched by the user's finger, the system control unit 50 displays a rectangular frame 820 surrounding the subject 811 corresponding to the figure 811 ′ on the photographing assist screen 800. As a result, the user can grasp which subjects 810 to 812 are far away from the refocus planes 803 to 805. The user can urge the subject 811 far away from the refocus planes 803 to 805 to move. The subject 811 moves near the refocus planes 803 to 805 in accordance with an instruction from the user. As a result, a reconstructed image with a high degree of focus can be obtained even when reconstruction is performed so that any subject 810 to 812 is in focus.

  Further, information 821 indicating the distance between the refocus planes 803 to 805 and the subject 811, that is, a text display may be provided on the photographing assist screen. FIG. 8 shows an example in which the refocus plane 804 exists at a position 6 m ahead of the subject 811 and the refocus plane 805 exists at a position 4 m behind the subject 811. If information 821 indicating the distance between the refocus planes 803 to 805 and the subject 811 is provided on the photographing assist screen, the user can accurately grasp the amount of movement that the subject 811 should move, and more quickly. It becomes possible to perform photography.

  The system control unit 50 can function as a control unit that displays, using the display unit 28, a photographing assist screen that indicates the positional relationship between the subjects 810 to 812 and the predetermined refocus surfaces 803 to 805.

Next, the operation of the imaging apparatus according to the present embodiment will be described with reference to FIG. FIG. 9 is a flowchart illustrating the operation of the imaging apparatus according to the present embodiment.
In step S901, the user instructs the start of shooting. The instruction to start shooting is performed by the user operating the operation unit 70. In step S902, a light field image (light field data) is acquired. In step S903, a reconstructed image reconstructed so as to be focused on a predetermined refocus plane is generated from the light field image acquired in step S902, and the generated reconstructed image 801 is displayed on the photographing assist screen 800. To do.

  In step S904, graphics 810 ′ to 812 ′ indicating the subject are displayed on the photographing assist screen 800 based on the distance information indicating the positions of the subjects 810 to 812. The distance information indicating the positions of the subjects 810 to 812 is acquired using the distance measuring sensor 104 and held in the memory 32 as described above. In addition, refocus planes 803 to 805 (see FIG. 8) in which information necessary for generating a reconstructed image is stored in advance in the nonvolatile memory 56 are displayed on the photographing assist screen 800.

  In step S905, it is detected whether or not figures 810 'to 812' schematically showing the subject have been touched by the user's finger. When it is detected that any of the figures 810 ′ to 812 ′ is touched by the user's finger (YES in step S905), the process proceeds to step S906. If no touch is detected on the figures 810 'to 812' (NO in step S905), step S905 is repeated.

  In step S906, a rectangular frame 820 surrounding the subject 811 associated with the touched figure 811 ′ is displayed on the photographing assist screen. In addition, the distance between the subject 811 corresponding to the touched figure 811 ′ and the refocus planes 804 and 805 is calculated by the system control unit 50, and information 821 indicating the distance is displayed on the photographing assist screen 800. .

  As described above, according to the present embodiment, the positional relationship between the subjects 810 to 812 and the refocus planes 803 to 805 is displayed on the photographing assist screen. For this reason, photographing can be assisted so that a good reconstructed image can be obtained.

[Second Embodiment]
An imaging apparatus and an imaging method according to the second embodiment of the present invention will be described with reference to FIGS. The same components as those of the imaging apparatus according to the first embodiment shown in FIGS. 1 to 9 are denoted by the same reference numerals, and description thereof is omitted or simplified.
The imaging apparatus according to the present embodiment displays the focusing range 860 in consideration of the aperture value 850 on the imaging assist screen 800.
The aperture value control performed in the present embodiment will be described with reference to FIG. FIG. 10 is a diagram for explaining aperture value control.
If a part of the light beams r1 to r5 from the pupil regions L1 to L5 is reduced, the aperture value can be increased. Therefore, in the imaging apparatus 100 according to the present embodiment, the aperture value is adjusted by adjusting the weighting coefficient used when the plurality of pixel values output from the plurality of divided pixels 211 included in the unit pixel 210 are weighted and added. To do. For example, the aperture value may be increased by setting the weighting coefficient relating to the light beam r1 from the pupil region L1 and the light beam r5 from the pupil region L5 to zero. When the weighting coefficient related to the light beam r1 from the pupil region L1 and the light beam r5 from the pupil region L5 is set to zero, the pixel value S3 of the reconstructed pixel P3 is calculated by the following equation (3). Can do.
S3 = w3 · b2 + w4 · b3 + c3 + w3 · d4 + w4 · d3 (3)

  Increasing the aperture value increases the depth of field. Therefore, if the aperture value is increased, the range 950 (see FIG. 10) in which a high degree of focus is obtained when reconstruction is performed using information stored in the nonvolatile memory 56 in advance is increased. Can do. The aperture value can be adjusted by a user operating an operation member provided in the operation unit 70.

  Here, the case where the weighting coefficient relating to the light beam r1 from the pupil region L1 and the light beam r5 from the pupil region L5 is set to zero has been described as an example, but the present invention is not limited to this. For example, it is possible to increase the aperture value by reducing the weighting coefficient related to the light beam r1 from the pupil region L1 and the light beam r5 from the pupil region L5. That is, the aperture value can be adjusted by appropriately adjusting the weighting coefficient used when weighted addition of a plurality of pixel values respectively output from the plurality of divided pixels 211 included in the unit pixel 210.

  A shooting assist screen 800 displayed in the imaging apparatus of the present embodiment will be described with reference to FIG. FIG. 11 is a diagram illustrating an example of a shooting assist screen 800 in the imaging apparatus according to the present embodiment.

  As shown in FIG. 11, an aperture value 850 is displayed on the photographing assist screen 800. Here, a case where the aperture value 850 is set to F5 is shown as an example. In addition, a reconstructed image 801 corresponding to the set aperture value 850 is displayed on the photographing assist screen 800. Further, a focusing range 860 corresponding to the aperture value 850 is shown on the photographing assist screen 800. In FIG. 11, the focus range 860 is shown using arrows.

In the case shown in FIG. 11, the subject 811 and the subject 812 are located within the focusing range 860 of the refocus plane 805. A subject 810 is located near the refocus plane 804. Therefore, the user can easily grasp that a good reconstructed image can be obtained even when the reconstructed image is generated so that any of the subjects 810 to 812 is in focus.
As described above, the focusing range 860 corresponding to the aperture value 850 may be displayed on the photographing assist screen 800.

[Third Embodiment]
An imaging apparatus and an imaging method according to the third embodiment will be described with reference to FIG. The same components as those of the imaging apparatus and imaging method according to the first or second embodiment shown in FIGS. 1 to 11 are denoted by the same reference numerals, and description thereof will be omitted or simplified.

  FIG. 12 is a diagram showing a photographing assist screen 800 in the imaging apparatus according to the present embodiment. As shown in FIG. 12, in this embodiment, graphics 880 to 882 indicating the degree of focus of each of the subjects 810 to 812 in the reconstructed image 801 are displayed on the photographing assist screen 800. Here, the degree of focus when the reconstructed image 801 is generated so that the refocus plane 803 is in focus is shown. The figures 880 to 882 indicating the degree of focus are shown using dotted lines. The longer the dotted line is, the higher the degree of focusing. A figure 803 ′ schematically showing the refocus plane 803 is indicated by a thick solid line. Graphics 810 ′ to 812 ′ schematically showing the subjects 810 to 812 are also displayed on the photographing assist screen 800. Figures 810 ′ to 812 ′ schematically showing the subject are associated with the subjects 810 to 812. For example, the system control unit 50 calculates the degree of focus of each of the subjects 810 to 812 based on the distance between the refocus plane 803 and each of the subjects 810 to 812.

In this way, when the figures 880 to 882 indicating the degree of focus of the respective subjects 810 to 812 are displayed on the photographing assist screen 800, how much the respective subjects 810 to 812 are in focus in the reconstructed image 801. The user can easily grasp this.
As described above, the graphics 880 to 882 indicating the degree of focus of each of the subjects 810 to 812 may be displayed on the photographing assist screen 800.

  Here, the focus level when the reconstructed image 801 is generated so that the refocus plane 803 is in focus is shown, but the present invention is not limited to this. The degree of focus of each of the subjects 810 to 812 when the reconstructed image 801 is generated so that the other refocus planes 804 and 805 are focused can be displayed in the same manner as described above. The refocus planes 803 to 805 can be selected by operating the operation unit 70 by the user.

As mentioned above, although embodiment of this invention was described, this invention is not limited to these embodiment.
For example, you may make it combine the said embodiment suitably.
(Other examples)
The present invention supplies a program that realizes one or more functions of the above-described embodiments to a system or apparatus via a network or a storage medium, and one or more processors in a computer of the system or apparatus read and execute the program This process can be realized. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

13 ... D / A converter 15 ... memory control unit 18 ... I / F
22 ... Imaging unit 23 ... A / D converter 24 ... Image processing unit 28 ... Display unit 30 ... Power supply unit 32 ... Memory 35 ... Compression / decompression processing unit 36 ... Touch panel 50 ... System control unit 56 ... Non-volatile memory 60 ... Mode Switch 61 ... Shutter button 62 ... Shutter switch 70 ... Operation unit 72 ... Power switch 80 ... Power control unit 101 ... Shutter 102 ... Microlens array 103 ... Shooting lens 104 ... Distance sensor 150 ... Lens unit 200 ... Recording medium

Claims (14)

Imaging means for acquiring image data capable of generating a reconstructed image with the focus position readjusted;
Reconstructing means for generating the reconstructed image with the focus position readjusted from the image data;
Detection means for detecting the distance to the subject;
Based on the distance to the subject and the position of the predetermined refocus surface, a screen showing a positional relationship between the subject and the predetermined refocus surface is displayed on the display means, or the refocus surface and the An imaging apparatus comprising: control means for causing the display means to display a screen indicating a degree of focus of the subject in the reconstructed image based on a distance to the subject. The screen showing the positional relationship between the subject and the predetermined refocus plane includes a first graphic schematically showing the subject and a second graphic schematically showing the predetermined refocus surface. The imaging apparatus according to claim 1, further comprising: The screen showing the degree of focus of the subject includes a first figure schematically showing the subject and a third figure showing the degree of focus of the subject. The imaging device described in 1. When the operation is performed on the first graphic, the control unit makes the subject image associated with the first graphic that is the target of the operation stand out in the reconstructed image. The image pickup apparatus according to claim 2, wherein display is performed. A touch panel disposed on the display means;
The imaging apparatus according to claim 4, wherein the operation on the first graphic is performed by touching the touch panel. The imaging apparatus according to claim 4, wherein the display for making the subject image stand out in the reconstructed image is performed by displaying a frame so as to surround the subject image. The imaging device according to any one of claims 1 to 6, wherein the control unit further displays a figure indicating an in-focus range according to an aperture value on the screen. The imaging means includes a pixel array in which a plurality of unit pixels each including a plurality of photoelectric conversion units are arranged in a matrix, a plurality of microlenses arranged above the pixel array, and one microlens A microlens array arranged to correspond to one unit pixel,
The reconfiguring means adjusts the aperture value by adjusting a weighting coefficient used when weighted addition of a plurality of pixel values respectively output from the plurality of photoelectric conversion units included in the unit pixel. The imaging apparatus according to claim 7, wherein the imaging apparatus is characterized. 9. The imaging apparatus according to claim 1, wherein information indicating the distance between the subject and the predetermined refocus plane is further displayed on the screen. The 4th figure which shows an imaging device typically is further displayed on the screen. An imaging device given in any 1 paragraph of Claims 1-9 characterized by things. The imaging apparatus according to any one of claims 1 to 10, further comprising a storage unit that stores information required to generate the reconstructed image on the predetermined refocus plane. The imaging means includes a pixel array in which a plurality of unit pixels each including a plurality of photoelectric conversion units are arranged in a matrix, a plurality of microlenses arranged above the pixel array, and one microlens A microlens array arranged to correspond to one unit pixel,
The reconstruction unit generates the reconstructed image by weighted addition of a plurality of pixel values respectively output from the plurality of photoelectric conversion units included in the unit pixel,
The imaging apparatus according to claim 11, wherein the information necessary for generating the reconstructed image on the predetermined refocus plane includes a weighting coefficient used for the weighted addition. Detecting the distance to the subject;
Based on the distance to the subject and the position of the predetermined refocus plane, a screen showing a positional relationship between the subject and the predetermined refocus plane is displayed, or between the refocus plane and the subject And a step of displaying a screen showing the degree of focus of the subject in the reconstructed image based on the distance between them. On the computer,
Detecting the distance to the subject;
Based on the distance to the subject and the position of the predetermined refocus plane, a screen showing a positional relationship between the subject and the predetermined refocus plane is displayed, or between the refocus plane and the subject And a step of displaying a screen indicating the degree of focus of the subject in the reconstructed image based on the distance between the two.