JP2017161941A - Reproducer, imaging apparatus, and control method of reproducer - Google Patents

Abstract

PROBLEM TO BE SOLVED: To easily grasp the range that is in focus in an image generated in accordance with the focal position after photographing and the change in throttle.SOLUTION: A reproducer for generating an image from light field information and displaying it includes: focus commanding means for commanding the focal position of the image including a virtual focal position; generating means for generating an image from the light field information in accordance with the focal position commanded by the focus commanding means; and displaying means for displaying the image generated by the generating means and an indication showing the focus state.SELECTED DRAWING: Figure 4

Description

  The present invention relates to a playback device, an imaging device, a playback device control method, a program, and a storage medium.

  2. Description of the Related Art In recent years, there have been proposed imaging devices that perform various operations on data obtained by an imaging device and perform various digital image processing to output various images. Patent Document 1 discloses an imaging apparatus that simultaneously acquires a two-dimensional intensity distribution of light in a subject space and light angle information, that is, images with different viewpoints. Here, the two-dimensional intensity distribution of light and the angle information of the light are combined to be called light space information (= light field information), and the three-dimensional information of the subject space can be obtained by acquiring the light space information. . In this imaging apparatus, the focus position of an image called refocusing can be changed by acquiring light space information and performing image reconstruction processing after shooting.

  On the other hand, a display method when a so-called refocus operation is performed on multi-viewpoint images acquired at the same time has also been proposed. Patent Document 2 proposes a system that interactively displays an image corresponding to a user operation. Specifically, in the system of Patent Document 2, refocus processing is performed so as to focus on the indicated location in accordance with an instruction from the pointing device, and the result is presented to the user.

JP 2007-4471 A US Patent Application Publication No. 2008/0131019

  However, the imaging devices and systems disclosed in Patent Documents 1 and 2 can display an image in response to an operation such as refocusing, but can intuitively grasp the in-focus range. It wasn't. Furthermore, when observing on a low resolution monitor, it is not easy to confirm the in-focus range.

  The present invention has been made in view of the above-described problems, and an object of the present invention is to make it possible to easily grasp a focus state in an image generated in accordance with a change in focus position after shooting.

  The present invention is a playback device that generates and displays an image from light field information, and includes a focus instruction unit that indicates a focus position of an image including a virtual focus position, and the focus instruction from the light field information. The image processing apparatus includes generation means for generating an image according to a focus position designated by the means, and display means for displaying a display indicating a focus state together with the image generated by the generation means.

  According to the present invention, it is possible to easily grasp the focus state in an image generated according to a change in the focus position after shooting.

It is a figure which shows the structure of a digital camera. It is a figure which shows the structure of a micro lens array. It is a figure which shows the example of an imaging optical system. It is a figure which shows the example which displays a focus state. It is a figure for demonstrating the calculation for a focus determination. It is a flowchart which shows the process of this embodiment.

Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings. In the drawings, the same members are denoted by the same reference numerals, and redundant description is omitted.
A playback apparatus according to this embodiment will be described with reference to FIGS. Here, a case where an imaging device (digital camera) is applied as a playback device will be described.
FIG. 1A is a central sectional view of the digital camera.
The digital camera has a camera body 1 and a lens unit 2 that can be attached to and detached from the camera body 1. The camera body 1 includes an imaging device 6, a rear display device as the display unit 9, a quick return mechanism 14, a finder display unit 16, and the like. The lens unit 2 includes a lens that forms the photographing optical system 3 along the optical axis 4, a lens system control circuit 12, and the like. The camera body 1 and the lens unit 2 are electrically connected by an electrical contact 11.

FIG. 1B is a block diagram showing an electrical configuration of the digital camera.
The digital camera including the camera body 1 and the lens unit 2 has an imaging system, an image processing system, a recording / reproducing system, and a control system. The imaging system is configured to include the imaging optical system 3 and the imaging element 6. The image processing system includes an image processing unit 7. The recording / reproducing system includes a memory unit 8 and a display unit 9. The control system includes a camera system control circuit 5, an operation detection unit 10, a lens system control circuit 12, and a lens driving unit 13. The lens driving unit 13 can drive a focus lens, a shake correction lens, a diaphragm, and the like.

  In the imaging system, light from an object is imaged on the imaging surface of the imaging element 6 via the imaging optical system 3. A so-called microlens array (hereinafter referred to as MLA) in which microlenses are arranged in a lattice pattern is disposed on the surface of the image sensor 6. MLA is an example of light beam control means. Details of the functions and arrangement of the MLA will be described later with reference to FIG. Since the focus evaluation amount / appropriate exposure amount is obtained from the image sensor 6, the imaging optical system 3 is appropriately adjusted based on this signal, so that an appropriate amount of object light is exposed to the image sensor 6, and A subject image is formed in the vicinity of the image sensor 6.

The image processing unit 7 includes an A / D converter, a white balance circuit, a gamma correction circuit, an interpolation calculation circuit, and the like. The image processing unit 7 can include functions for developing and generating an image for recording. In the present embodiment, the camera system control circuit 5 has these functions. .
The memory unit 8 includes a processing circuit necessary for recording in addition to a recording unit that actually records image data. The memory unit 8 outputs to the recording unit and generates and stores an image to be output to the display unit 9. In addition, the memory unit 8 compresses images, moving images, sounds, and the like using a predetermined method.

  The camera system control circuit 5 generates and outputs a timing signal at the time of imaging. The camera system control circuit 5 controls the imaging system, the image processing system, and the recording / reproducing system according to user operations. Specifically, the camera system control circuit 5 detects the depression of a shutter release button (not shown) by the operation detection unit 10, thereby driving the image sensor 6, the operation of the image processing unit 7, and the compression processing of the memory unit 8. Control etc. The camera system control circuit 5 controls the state of each segment for displaying information by the display unit 9 such as a liquid crystal monitor.

  Next, the adjustment operation of the optical system by the control system will be described. The camera system control circuit 5 receives a signal from the image sensor 6 via the image processing unit 7 and obtains an appropriate focal position and aperture position based on the received signal. The camera system control circuit 5 transmits a command to the lens system control circuit 12 via the electrical contact 11, and the lens system control circuit 12 controls the lens driving unit 13 according to the received command. In addition, a camera shake detection sensor (not shown) is connected to the lens system control circuit 12. In the camera shake correction mode, the lens system control circuit 12 appropriately controls the camera shake correction lens via the lens driving unit 13 based on the signal from the camera shake detection sensor.

  The digital camera shown in FIG. 1 can reproduce an image. The camera system control circuit 5 displays the acquired image on the display unit 9 when the operation detection unit 10 detects switching to the reproduction mode. As will be described later, the camera system control circuit 5 reconstructs and displays an image according to a focus position (focus position) and an aperture value (aperture) using a plurality of images having different parallaxes. . At this time, the camera system control circuit 5 controls the developing method according to the instruction of the focus position and the instruction of the aperture value.

FIG. 2 is a diagram for explaining a main part of the photographing optical system of the present embodiment. Although the present invention can be applied to other photographing optical systems, the other photographing optical systems will be described with reference to FIG.
First, in order to realize the present invention, it is necessary to acquire subject images from a plurality of viewpoints. In the present embodiment, an MLA is disposed in the vicinity of the imaging plane of the image sensor 6 for obtaining angle information, and a plurality of pixels are associated with one microlens constituting the MLA.

FIG. 2A is a diagram schematically illustrating the relationship between the image sensor and the MLA. FIG. 2B is a schematic diagram showing the correspondence between the pixels of the image sensor and the MLA. FIG. 2C is a diagram showing that the pixels arranged under the MLA are associated with a specific pupil region by the MLA.
As shown in FIG. 2A, an MLA 18 is provided on the image sensor 6. The front principal point of the MLA 18 is arranged in the vicinity of the imaging plane of the photographing optical system 3. FIG. 2A is a view of the MLA as seen from the front and side of the digital camera. When viewed from the front, the MLA 18 is arranged so that the lens covers the pixels on the image sensor 6. In FIG. 2 (a), the microlenses constituting the MLA are shown to be easy to see, but in actuality, each microlens is only several times as large as a pixel (see FIG. 2 (described later)). see b)). One pixel corresponds to one photoelectric conversion unit.

FIG. 2B is a partially enlarged view seen from the front of FIG. A grid-like frame shown in FIG. 2B indicates each pixel of the image sensor 6. On the other hand, each microlens constituting the MLA 18 is indicated by thick circles 20, 21, 22, and 23.
As shown in FIG. 2B, a plurality of pixels are assigned to one microlens. Here, 5 rows × 5 columns = 25 pixels are provided for one microlens. That is, the size of each microlens is 5 × 5 times the size of the pixel.

FIG. 2C is a cross-sectional view of a part of the image sensor 6 and a part of the microlens 20 cut along a line II shown in FIG. 2C shows the pixels 20a, 20b, 20c, 20d, and 20e of the image sensor 6, and the upper side shows the exit pupil plane of the photographing optical system 3. FIG. Originally, when the exit pupil plane is aligned with the pixel direction shown in the lower side of FIG. 2C, the exit pupil plane is perpendicular to the plane of FIG. 2C. Is changing.
In FIG. 2 (c), one-dimensional projection / signal processing will be described for the sake of simplicity, but the camera system control circuit 5 can be easily extended to two dimensions. Note that the pixels 20a to 20e in FIG. 2C have a positional relationship corresponding to the pixels 20a to 20e in FIG.

  As shown in FIG. 2C, each of the pixels 20 a to 20 e is designed to be conjugate with a specific region on the exit pupil plane of the photographing optical system 3 by the micro lens 20. In the example of FIG. 2C, the pixel 20a and the region 30a correspond to the pixel 20b and the region 30b, the pixel 20c and the region 30c, the pixel 20d and the region 30d, and the pixel 20e and the region 30e, respectively. That is, only the light beam that has passed through the region 30a on the exit pupil plane of the photographing optical system 3 enters the pixel 20a. The same applies to the other pixels. Further, the adjacent pixels 21a to 21e also correspond to the same region on the exit pupil plane. That is, the pixel 21a corresponds to the region 30a, the pixel 21b corresponds to the region 30b, the pixel 21c corresponds to the region 30c, the pixel 21d corresponds to the region 30d, and the pixel 21e corresponds to the region 30e.

  As a result, the camera system control circuit 5 can acquire angle information from the passing area on the pupil plane and the positional relationship on the image sensor 6. In order to generate different images from a plurality of viewpoints from the optical system described in FIG. 2, the camera system control circuit 5 arranges pixels corresponding to the same pupil plane of each microlens. Thereby, the camera system control circuit 5 can generate different images at a plurality of viewpoints, which are input images.

Next, another photographing optical system will be described with reference to FIG.
As described above, in order to apply the present invention, it is necessary to acquire a plurality of images with different viewpoints. Such an optical system capable of acquiring a plurality of different viewpoint images is known as a plenoptic camera or a so-called multi-lens camera. Note that the images with different viewpoints are equivalent to the light space information described in Patent Document 1.
FIG. 3 is a diagram schematically illustrating a state in which a light beam from an object (subject) forms an image on the image sensor 6. FIG. 3A corresponds to the photographing optical system described in FIG. 2, and is an example in which the MLA 18 is disposed in the vicinity of the imaging surface of the photographing optical system 3. FIG. 3B shows an example in which the MLA 18 is arranged closer to the object than the imaging plane of the photographing optical system 3. FIG. 3C shows an example in which the MLA 18 is arranged on the side farther from the object than the imaging plane of the photographing optical system 3. FIG. 3D is an example showing a so-called multi-eye optical system.

3A to 3D show an object plane 51, arbitrary points 51a and 51b on the object, and a pupil plane 52 of the photographing optical system. 3A to 3C show pupil regions 30a to 30e of the photographing optical system and specific microlenses 61, 62, 71, 72, 73, 81, 82, 83, and 84 on the MLA 18. Has been. 3B and 3C show a virtual imaging plane 50 of the main lens as a conjugate plane with the object 51. FIG. 3D shows the imaging devices 6a, 6b, 6c constituting the multi-eye optical system and the pupil regions 31a to 35a of the main lenses of the multi-eye optical system.
3 (a) to 3 (d), the light beam emitted from the point 51a on the object and passing through the photographing optical system or the multi-view optical system is indicated by a solid line, and is emitted from the point 51b on the object and the photographing optical system or A light beam passing through the multi-eye optical system is indicated by a one-dot chain line.

  In FIG. 3A, as described with reference to FIG. 2, the MLA 18 is disposed in the vicinity of the imaging plane of the photographing optical system 3, so that the imaging element 6 and the pupil plane 52 of the photographing optical system 3 have a conjugate relationship. is there. Further, the object plane 51 and the MLA 18 are in a conjugate relationship. Therefore, the light beam emitted from the point 51a on the object reaches the micro lens 61, the light beam emitted from the point 51b reaches the micro lens 62, and the light beam that has passed through each of the regions 30a to 30e is below the micro lens. To the corresponding pixels provided in.

  In FIG. 3B, the light beam from the photographing optical system 3 is imaged by the MLA 18, and the image sensor 6 is provided on the imaging surface. By arranging in this way, the object plane 51 and the imaging device 6 are in a conjugate relationship. The light beam emitted from the point 51a on the object and passed through the region 30a on the pupil plane reaches the micro lens 71, and the light beam emitted from the point 51a on the object and passed through the region 30c on the pupil plane reaches the micro lens 72. To do. The light beam emitted from the point 51b on the object and passed through the region 30a on the pupil plane reaches the micro lens 72, and the light beam emitted from the point 51b on the object and passed through the region 30c on the pupil plane is micro lens 73. To reach. The light beam that has passed through each microlens reaches a corresponding pixel provided under the microlens. In this way, images are formed at different positions depending on the point on the object and the passing area on the pupil plane. By rearranging these at positions on the virtual imaging plane 50 of the main lens, the same information as in FIG. 3A can be obtained. That is, information about the pupil region (incident angle) that has passed through and the position on the image sensor 6 can be obtained.

  In FIG. 3C, the light beam from the imaging optical system 3 is re-imaged by MLA 18 (referred to as being imaged again when the light beam once imaged is diffused) and imaged on the imaging surface. Element 6 is provided. By arranging in this way, the object plane 51 and the imaging device 6 are in a conjugate relationship. The light beam emitted from the point 51a on the object and passed through the region 30a on the pupil plane reaches the microlens 82, and the light beam emitted from the point 51a on the object and passed through the region 30c on the pupil plane reaches the microlens 81. To do. The light beam emitted from the point 51b on the object and passed through the region 30a on the pupil plane reaches the micro lens 84, and the light beam emitted from the point 51b on the object and passed through the region 30c on the pupil plane reaches the micro lens 83. To do. The light beam that has passed through each microlens reaches a corresponding pixel provided under the microlens. Similar to FIG. 3B, the same information as FIG. 3A can be obtained by rearranging the main lens at the position on the virtual imaging plane 50. That is, information about the pupil region (incident angle) that has passed through and the position on the image sensor 6 can be obtained.

  In FIG. 3D, similar information can be obtained not by the main lens and MLA but by a so-called multi-eye optical system. In this multi-view optical system, it can be considered that the main lenses 31a to 35a are arranged directly in the pupil regions 30a to 30e in FIG. The images of the image sensors 6a, 6b, and 6c are images having different so-called viewpoints.

FIG. 3 shows an example in which the position information and the angle information can be acquired using the MLA (phase modulation element) as the pupil dividing means. Other optical configurations can be used as long as they can be obtained. For example, a method of inserting a mask (gain modulation element) with an appropriate pattern into the optical path of the photographing optical system may be used.
Another method may be a method of acquiring different images from a plurality of viewpoints by time division. In this method, a large number of images that are separated in time are obtained by a digital camera, and different images from a plurality of viewpoints are obtained by using camera shake or user swing.

Next, a case where an image is developed from a plurality of images with different parallaxes according to a user's operation using the digital camera according to the present embodiment and displayed on the display unit 9 will be described with reference to FIGS.
FIG. 4A shows the positional relationship between the digital camera and the subject. In FIG. 4A, there are a subject 110 and a subject 111 that are different in distance from the digital camera in a range 102 photographed by the camera body 1 and the lens unit 2, respectively.
FIG. 4B is a perspective view showing the appearance of the digital camera. The lens unit 2 is provided with a focus instruction unit 120 that can be operated by a user. The focus instruction unit 120 is an example of a focus instruction unit. The focus instruction unit 120 includes, for example, a first ring member that can rotate along the outer periphery of the lens unit 2, and the rotation position of the first ring member is detected by the operation detection unit 10.
In addition, the camera body 1 is provided with an aperture value instruction unit 121 that can be operated by the user. The aperture value instruction unit 121 is an example of an aperture instruction unit. The aperture value instruction unit 121 includes, for example, a rotatable second ring member, and the operation detection unit 10 detects the rotational position of the second ring member.

  FIG. 4C is a diagram showing a subject image acquired by photographing the state shown in FIG. FIG. 4C shows a case where all subjects are acquired sharply, ignoring the focus state. In an image 103 shown in FIG. 4C, an image of a subject 111 that is relatively far from a subject 110 that is relatively close to the digital camera is acquired.

  In the following, a case is considered where a digital camera acquires a plurality of images with different parallaxes so that exposure times overlap at least. The present invention is applied when displaying an image after image acquisition and when manipulating the image. Since a plurality of images having different parallax have obtained information equivalent to so-called ray space information, it is possible to change the focus position (refocus) and change the aperture value after obtaining the image. The camera system control circuit 5 generates an image in which the refocus and the aperture value are adjusted in accordance with the operation through the focus instruction unit 120 and the aperture value instruction unit 121 by the user. Since the image generation method is disclosed in Patent Document 1 and Patent Document 2, the description thereof is omitted.

  In the present embodiment, a case where the focus instruction unit 120 is used as an operation unit for refocusing and the aperture value instruction unit 121 is used as an operation unit for changing an aperture value will be described. That is, the camera system control circuit 5 changes the focus position according to the operation of the focus instruction unit 120, focuses on the subject in front or behind, and changes the aperture value according to the operation of the aperture value instruction unit 121. Adjust the depth of field. The operation detection unit 10 detects operations performed by the focus instruction unit 120 and the aperture value instruction unit 121.

FIG. 4D to FIG. 4G are diagrams illustrating examples of display images. The camera system control circuit 5 overlays and displays a display indicating the focus state and a display indicating that the focus state is changed on the image 103. Here, the focus state display includes a distance measurement frame 130 and a focus detection frame 131 in focus. The display indicating that the focus state changes includes a distance measuring frame 132 that is focused by narrowing down.
First, FIG. 4D shows a state in which the subject 111 in the distance is in focus and the subject 110 in the vicinity is not in focus (blurred). On the other hand, FIG. 4E shows a state in which the subject 110 located near is in focus and the subject 111 located far away is out of focus (blurred). FIG. 4F shows a state where both the near subject 110 and the far subject 111 are in focus. FIG. 4G shows a state in which the object 110 located near is in focus and the object 111 located far away is out of focus (blurred), as in FIG. 4E. This shows a case where changes in focus state due to narrowing down are displayed simultaneously.

  In this embodiment, since the image acquired by the digital camera can be refocused, the focus position is changed after the image is acquired, and different focus positions are obtained as shown in FIGS. 4 (d) and 4 (e). An image can be generated. This process is performed by the camera system control circuit 5 in accordance with an operation via the focus instruction unit 120. In FIG. 4D to FIG. 4G, the distance measurement frame 130 is displayed as an overlay on the generated image. The distance measurement frame corresponds to the range of an image to be cut out when a part of the image is cut out to calculate the focus state of the subject. Since the digital camera acquires a plurality of images with different parallaxes, the distance to the subject can be measured based on the principle of triangulation. By using this, it is possible to determine whether or not each subject is in focus at the currently displayed focus position. Specifically, this process can be determined by SAD calculation described later with reference to FIG.

  In the example shown in FIGS. 4D to 4G, the in-focus range is indicated by the in-focus range frame 131. In FIG. 4D to FIG. 4G, the distance measurement frame 131 is indicated by a thick line. However, the distance measurement frame 131 only needs to be distinguished from a distance measurement frame in an out-of-focus range. For example, it may be colored. . In FIG. 4D, the distance measurement frame 131 corresponding to the far object 111 is shown by a thick line, and in FIG. 4E, the distance measurement frame 131 corresponding to the nearby object 110 is shown by a thick line. Yes.

  Next, consider a case where an operation for changing the aperture value, that is, a so-called aperture operation is performed. When narrowed down, the depth of field becomes deep, and even subjects with different distances are in focus. Here, the case where the blur is equal to or less than the permissible circle of confusion is assumed to be in focus. FIG. 4 (f) schematically shows a narrowed state, in which both the subject 110 and the subject 111 are in focus, and the distance measurement frame 131 corresponding to both is indicated by a thick line. As described with reference to FIG. 4F, it can be seen that the range in focus changes when the aperture value is changed. That is, as the depth direction changes, the in-focus range in the screen also changes. FIG. 4 (g) shows this intuitively. In FIG. 4G, the distance measuring frame 132 that is focused by changing the aperture is indicated by hatching. Actually, the focusing frame 132 in focus can be displayed in a different color or the like so as to be distinguishable between the ranging frame 130 and the ranging frame 131. FIG. 4G shows that the subject 110 in front is in focus, but the subject 111 is also focused by narrowing down. As described above, the range that is in focus by changing the aperture value is presented by a method different from the range that is currently in focus.

  In the method disclosed in Patent Document 2, it is necessary for the user to determine whether or not the image is in focus by looking at the image, but it is easy to accurately determine the focus state on a low-resolution monitor or the like. is not. On the other hand, in the present embodiment, the in-focus range is displayed by overlaying, so that the in-focus range is displayed intuitively and easily. Therefore, the user can easily grasp that the subject is in focus even with a low-resolution monitor. Also, since the focus state presentation method as shown in FIG. 4 is also used during the aiming operation of the conventional camera, it is easy for the user who has used the conventional camera to understand.

Next, processing and flowcharts for determining the focus will be described with reference to FIGS.
The camera system control circuit 5 displays the focus position to be displayed first using a value such as a setting at the time of shooting, a default setting, or a previous setting value. FIG. 5 specifically describes an evaluation value calculation method and a display method according to the evaluation value when the user instructs the focus instruction unit 120 and the aperture value instruction unit 121 to change the focus state.

FIG. 5A is a diagram for explaining a method of calculating a focus evaluation value in a specific distance measurement frame. FIG. 5B is a diagram illustrating the relationship between the focus evaluation value and the focus position. FIG. 6 is a flowchart showing the reproduction process.
In FIG. 5A, an image 141 of a certain viewpoint, an image 142 of a viewpoint different from the image 141, a focusing frame 143 of interest, a corresponding ranging frame 144, an arrow 145 as a change direction of an image clipping position, an adder 151, an absolute value calculator 152, and an adder 153, respectively.
An image 141 shown in FIG. 5A shows an image at a specific viewpoint among images having different viewpoints. Here, the viewpoint image corresponding to the pixel 20e in FIG. 2 will be described. Similarly, the image 142 will be described as a viewpoint image corresponding to the pixel 20a in FIG.
The camera system control circuit 5 pays attention to the distance measuring frame 143 of the image 141 of a specific viewpoint. The camera system control circuit 5 cuts out the focused image and compares it with the image 142 of another viewpoint. In FIG. 5A, the cut-out image is enlarged and displayed downward.

  Several methods are conceivable for the comparison for obtaining the focus evaluation value. Here, the description will be made on the assumption that the difference absolute value integration (SAD = Sum of Absolute Difference) is used. SAD is defined by the following equation.

  In Equation 1, X (i) indicates a signal within a focusing frame in which an image of a specific viewpoint is focused, and Y (i) indicates a signal cut out from an image of another viewpoint. i is a subscript corresponding to the pixel to be compared. For example, in an 8 × 8 region, i is a value from 1 to 64. Therefore, this means that the differences of all the pixels are integrated. As is clear from the definition shown in Formula 1, in SAD, the minimum value is obtained when the degree of similarity is high (an image of a specific viewpoint and an image of another viewpoint are very similar). Here, the value of SAD is used as the focus evaluation value (focus evaluation information).

In FIG. 5A, Equation 1 is shown in a block diagram. As shown in FIG. 5A, the adder 151 calculates a difference between the focused distance measurement frame 143 and the image 144 cut out from another viewpoint. Next, after the absolute value calculator 152 acquires the absolute value of the calculated difference, the adder 153 performs addition for all the pixels.
As described above, a specific viewpoint image 141 and other viewpoint images 142 correspond to the pixels 20e and 20a in FIG. 3, respectively. Since there is a so-called epipolar constraint between images acquired at different viewpoints at the same time, it is sufficient to search only in that direction. For example, when comparing images obtained from the pixel 20a and the pixel 20e, since the viewpoint position is shifted only in the X direction, the search direction can be limited to the X direction. In FIG. 5A, the search direction is schematically indicated by an arrow 145.

  Therefore, the camera system control circuit 5 can obtain a local minimum value somewhere by calculating the SAD while changing the position where the image is cut out from the image 142 of another viewpoint in the direction of the arrow 145. The location of the minimum value is the position where the two images are in agreement and in focus.

  FIG. 5B is a diagram illustrating a relationship between the image cut-out position and the focus evaluation value. The horizontal axis indicates the position where the image is cut out from the image 142 of another viewpoint, and the vertical axis indicates the focus evaluation value (SAD). The camera system control circuit 5 can obtain a value as shown in FIG. 5B by calculating the SAD while changing the cutout position of the image in the direction of the arrow 145 as described above. In the example of FIG. 5B, calculation is performed at eight cutout positions, which are indicated by evaluation points 161 to 168, respectively. Here, the evaluation score 165 is a minimum value. At this time, the position corresponding to the evaluation point 165 is the position where the focus is best. Note that the value of the evaluation score 161 to the evaluation score 168 and the shape of the graph depend on the subject.

  At this time, the camera system control circuit 5 sets the range of a certain horizontal width in FIG. 5B around the evaluation point 165 that is in focus as the focus range that is determined to be in focus. The width of the in-focus range is determined based on the allowable circle of confusion and the F number. The camera system control circuit 5 can set the allowable circle of confusion according to the method of using the output image. For example, the camera system control circuit 5 sets the permissible circle of confusion to a suitable size according to the viewing method such as the display unit 9 and the print resolution, or increases according to the input or selection of the permissible circle of confusion by a user operation in advance. It is preferable to set the thickness. In the present embodiment, the setting of the permissible circle of confusion by the camera system control circuit 5 corresponds to the setting of the focus threshold value for determining the focus state. That is, the camera system control circuit 5 corresponds to an example of threshold setting means.

  When the camera system control circuit 5 sets a large permissible circle of confusion (such as viewing on a low-resolution monitor or printing with a small size such as an L version), the focus in FIG. Increase the in-focus range (the width) that is determined to be correct. On the other hand, when the permissible circle of confusion is set small (such as viewing on a high-resolution monitor or printing in a large size such as A4 size), the same F number is in focus in FIG. Decrease the focusing range (width).

The camera system control circuit 5 can calculate a focus evaluation value as shown in FIG. 5B for each distance measurement frame. Further, the focus position of the display image is not necessarily in a focused state in all the distance measurement frames. For example, as described with reference to FIG. 4, when there are subjects with different distances, one is often focused and the other is often out of focus.
The relationship between the evaluation value in FIG. 5B and the display in FIG. 4 will be described. Here, in order to simplify the description, the camera system control circuit 5 will be described as being capable of generating two images with the F number of the display image being F2.8 and F5.6. As for the focus position of the display image, three cases of focus position 1, focus position 2, and focus position 3 shown in FIG. 5B will be described.
Consider a case where the focus position of the display image is at the focus position 1 in FIG. At this position, it can be considered that the focus is in focus at any aperture value of F2.8 and F5.6. Therefore, the camera system control circuit 5 determines that the focus is in focus regardless of whether the setting of the display image is F2.8 or F5.6, and overlays and displays a display indicating in-focus.

  Next, consider the case where the focus position of the display image is at the focus position 2 in FIG. At this position, F5.6 can be considered to be in focus, but F2.8 cannot be considered to be in focus. Therefore, when the display image setting is F5.6, the camera system control circuit 5 determines that the camera is in focus and overlays and displays a display indicating in-focus. On the other hand, when the setting of the display image is F2.8, it is determined that the image is not in focus, and the display indicating the focus is not overlaid and displayed. However, the camera system control circuit 5 changes the aperture value by the aperture value instruction unit 121 in the case where the range in focus is displayed by changing the aperture value as shown in FIG. Displayed as the range in which the focus state changes.

Finally, consider the case where the focus position of the display image is at the focus position 3 in FIG. At this position, it cannot be regarded that the aperture is in focus at any aperture value of F2.8 and F5.6. Therefore, the camera system control circuit 5 determines that the focus is not in focus regardless of whether the display image setting is F2.8 or F5.6, and does not overlay the display indicating the focus.
By performing the processing as described above, the display as shown in FIG. 4D to FIG. 4G can be realized.

FIG. 6 is a flowchart showing processing of the digital camera according to the present embodiment.
In step S101, the camera system control circuit 5 starts processing.
In step S <b> 102, the camera system control circuit 5 waits for an interrupt by a user operation via the operation detection unit 10. If there is a user operation, the process proceeds to step S103, and if there is no user operation, the process waits in step S102.

In step S103, the camera system control circuit 5 determines whether or not the user operation is an end operation. In the case of an end operation, the process proceeds to step S104 and the process ends. If it is not an end operation, the process proceeds to step S105.
In step S105, the camera system control circuit 5 determines whether or not the user operation is an operation for changing the aperture or an operation for changing the focus. Specifically, the camera system control circuit 5 determines whether or not the aperture value instruction unit 121 or the focus instruction unit 120 is an operation for changing the aperture value or the focus position via the operation detection unit 10. If there is an operation to change the aperture or focus, the process proceeds to step S106. If it is not an operation to change the aperture or focus, processing is performed according to a user operation (not shown) and the process returns to step S102.

In step S <b> 106, the camera system control circuit 5 reads the focus threshold value from the memory unit 8, for example. This focus threshold value is an allowable circle of confusion set in advance by the camera system control circuit 5 according to the viewing method, user input, and the like in step S107.
In step S108, the camera system control circuit 5 performs development and updates the display image. Specifically, the camera system control circuit 5 develops and displays the image on the display unit 9 in accordance with the operation for changing the aperture or focus determined in step S105.

  In step S109, the camera system control circuit 5 determines whether or not the current focus frame (ranging point) is in focus, that is, whether or not the subject is in focus. If the subject is in focus, the process proceeds to step S110. If the subject is not in focus, the process proceeds to step S111. Here, as described with reference to FIG. 5B, the camera system control circuit 5 determines whether the focus is in focus, the focus position and aperture value of the display image, the allowable circle of confusion, and the focus of the distance measurement frame. The determination can be made based on the position information. When the process proceeds from step S105 to step S106, the focus position and aperture value of the display image are determined by the user operation, and an allowable circle of confusion is set in step S106. Therefore, in step S109, the camera system control circuit 5 calculates the focus evaluation value as described above with reference to FIG. 5B, obtains the in-focus position of the distance measurement frame, and can be regarded as being in focus. It can be determined whether or not.

In step S110, the camera system control circuit 5 performs in-focus display. Specifically, the camera system control circuit 5 overlays a distance measuring frame 131 that can identify that the camera is in focus on the display image, and displays it on the display unit 9.
In step S111, the camera system control circuit 5 determines whether or not the focus detection frame is in focus by narrowing down. When focusing is achieved by narrowing down, the process proceeds to step S112, and when focusing is not achieved, the process proceeds to step S113. Here, as described with reference to FIG. 5B, the camera system control circuit 5 determines whether or not the focusing frame is focused by narrowing down, as described with reference to FIG. The determination can be made based on the information on the focus position of the distance measurement frame. For example, as shown in FIG. 5B, when the focus position of the display image is “focus position 2” and the current display image setting is F2.8, the evaluation point 163 is in focus. Not determined to be. However, the evaluation point 163 is determined to be in focus because the focus range that is determined to be in focus is expanded by narrowing down. As described above, the camera system control circuit 5 determines whether or not the focus is achieved by an operation of changing the aperture value by the user.

In step S112, the camera system control circuit 5 performs a display (narrowing focus display) that can identify that focusing is achieved by narrowing down. Specifically, the camera system control circuit 5 overlays a display frame on the display unit 9 and displays the range-finding frame 132 that can identify that the subject is in focus by narrowing down.
In step S113, the camera system control circuit 5 determines whether or not the processing from step S109 to step S112 has been performed for all the distance measurement frames. When the processing is completed for all the ranging frames, the process returns to step S102, and the camera system control circuit 5 waits for a user operation. If it has not been completed, the process returns to step S109, and the camera system control circuit 5 pays attention to the next distance measurement frame and repeats until all the distance measurement frames have been processed.

  As described above, according to the present embodiment, by displaying the focus state on the image generated in accordance with the change of the focus and the aperture after shooting, the user can select the range in focus. Intuitively and easily grasped.

  In the present embodiment, the case where the distance measuring frame 132 that can identify that the subject is in focus by narrowing down is displayed in the flowchart of FIG. 6, but the present invention is not limited to this case. For example, when displaying only the currently focused range, step S111 and step S112 may be omitted, and if “N” in step S109, the process proceeds to step S113.

As mentioned above, although this invention was demonstrated with various embodiment, this invention is not limited only to these embodiment, A change etc. are possible within the scope of the present invention.
In the above-described embodiment, the case where the present invention is applied to a digital camera has been described as an example. However, the present invention is not limited to this case, and the present invention can also be used for a playback apparatus.
The present invention can also be realized by executing the following processing. That is, this is a process in which a program that realizes the functions of the above-described embodiments is supplied to a digital camera via a network or various storage media, and a digital camera computer (camera system control circuit 5 or the like) reads and executes the program. .

  1: Camera body 2: Lens unit 5: Camera system control circuit 6: Image sensor 9: Display unit 18: Micro lens array (MLA) 20: Micro lens 110: Subject 111: Subject 120: Focus instruction unit 121: Aperture value instruction Parts 130-132: Distance measurement frame

Claims (20)

A playback device that generates and displays an image from light field information,
Focus instruction means for instructing a focus position of an image including a virtual focus position;
Generating means for generating an image from the light field information according to a focus position instructed by the focus instruction means;
Display means for displaying a display indicating a focus state together with the image generated by the generating means;
A playback apparatus comprising: Including an aperture instruction means for instructing aperture setting of an image, including a virtual aperture,
2. The reproduction according to claim 1, wherein the generation unit generates an image from the light field information according to a focus position instructed by the focus instruction unit and an aperture setting instructed by the aperture instruction unit. apparatus.   The playback apparatus according to claim 1, wherein the display unit performs display indicating a focus state of a region of a part of the image generated by the generation unit.   4. The display device according to claim 1, wherein the display unit displays a focus state by differentiating a display color when the focus is achieved and a display color when the focus is not achieved. 5. The reproducing apparatus as described.   5. The display unit according to claim 1, wherein the display unit is capable of displaying a frame set in the image generated by the generation unit and displaying a focus state in an area corresponding to the frame. The playback apparatus according to any one of the above.   6. The reproducing apparatus according to claim 1, wherein the display unit is capable of displaying an area in which an in-focus image can be generated from the light field information.   The display means displays, on the image generated by the generating means, a display indicating an area where an in-focus image can be generated from the light field information together with a display indicating the focus state. The reproduction apparatus according to claim 1, wherein the reproduction apparatus is a reproduction apparatus.   8. The playback apparatus according to claim 1, wherein the display of the focus state by the display unit is performed based on a phase difference obtained from the light field information.   The reproduction apparatus according to claim 1, wherein the light field information includes a signal obtained from a multi-view optical system including a plurality of imaging elements.   9. The light field information according to claim 1, wherein the light field information includes a plurality of microlenses and a signal obtained from an imaging device in which a plurality of photoelectric conversion units are assigned to each microlens. 10. Playback device.   The playback apparatus according to claim 1, wherein the display indicating the focus state indicates a subject at which distance is in focus.   The playback apparatus according to claim 1, wherein the display indicating the focus state indicates an area in focus in the image. Imaging means;
The playback device according to any one of claims 1 to 12,
An imaging device comprising:   14. The imaging apparatus according to claim 13, wherein the light field information includes a signal imaged by controlling a focal position or a diaphragm position based on a signal obtained from the imaging unit. A ring member rotatable along the outer periphery of a lens barrel including an imaging optical system for guiding a light beam to the imaging means;
The imaging apparatus according to claim 13 or 14, wherein the focus instruction unit instructs a focus position of an image by detecting rotation of the ring member. A control method of a playback device that generates and displays an image from light field information,
A focus instruction step for indicating a focus position of an image including a virtual focus position;
A generation step for generating an image from the light field information according to the focus position instructed by the focus instruction step;
A display step for displaying a display indicating a focus state together with the image generated by the generation step;
A control method for a playback apparatus, comprising: A playback device that generates and displays an image from light field information,
A diaphragm instruction means for instructing an image aperture setting including a virtual diaphragm,
Focus instruction means for instructing a focus position of an image including a virtual focus position;
Generating means for generating an image from the light field information in accordance with a focus position instructed by the focus instruction means and an aperture setting instructed by the aperture instruction means;
Display means for displaying a display indicating a focus state together with the image generated by the generating means;
A playback apparatus comprising: A control method of a playback device that generates and displays an image from light field information,
An aperture instruction step for instructing aperture setting of an image including a virtual aperture,
A focus instruction step for indicating a focus position of an image including a virtual focus position;
A generation step for generating an image from the light field information according to the focus position instructed in the focus instruction step and the aperture setting instructed in the aperture instruction step;
A display step for displaying a display indicating a focus state together with the image generated by the generation step;
A control method for a playback apparatus, comprising:   The program for making a computer perform each step of the control method of the reproducing | regenerating apparatus of Claim 16 or 18.   A computer-readable storage medium storing a program for causing a computer to execute each step of the playback apparatus control method according to claim 16.

Images