JP5875839B2 - Plenoptic camera - Google Patents

Description

  The present invention relates to a technique for reconstructing and displaying light field image data generated by imaging with a plenoptic camera.

  In a plenoptic camera, image data including three-dimensional image information about a subject can be recorded by recording the distribution of the direction of light rays incident on the photographing optical system of the camera. Here, the three-dimensional image information means that information in the depth direction is included in addition to the two-dimensional image information obtained by a normal camera. By processing image data including three-dimensional image information, extracting light rays from any direction among light rays incident on the photographing optical system, and reconstructing a two-dimensional image, for example, any photographing distance It is possible to generate an image focused on the subject located at.

  As a plenoptic camera that is relatively small and easy to use for general users, a microlens array composed of a large number of microlenses arranged in two dimensions is placed near the light-receiving surface of the image sensor and formed by a photographic lens. There is known a type in which a subject image is divided by a microlens array and guided onto an image sensor.

  When an image based on image data obtained by photographing with such a plenoptic camera is reproduced on the camera, an image generation process called “refocus” is performed, An image focused on a subject located at a desired shooting distance is generated. Such image generation processing can also be performed by a personal computer, an online service, or the like.

  Patent Document 1 discloses a technique for generating and displaying an image for display by performing image generation processing corresponding to a focus value designated by a user using image data obtained by a plenoptic camera. The

Japanese Patent Laid-Open No. 2008-21878

  An example of post-processing image data obtained with a plenoptic camera and generating an image focused on a desired subject will be described as an example. However, it is not always possible to generate an image in focus. In other words, depending on the focus adjustment position (distance) of the photographic lens and the depth of field of the photographic lens that were set when shooting with the plenoptic camera, the subject is too far or too close In some cases, an in-focus image cannot be generated.

  Therefore, when processing is to be performed later, an image focused on a desired subject may not be generated. That is, even in the plenoptic camera, there is a limit to the range in which an in-focus image can be obtained later. For example, the photographer may be wondering whether it is better to focus on the flower in front and blur the background, or to focus on the background with the flower in front blurred. When shooting with a plenoptic camera, if you know later that you will get an image that is both in focus, the user will focus on the composition without worrying about the focus position. can do. However, as described above, there is a limit to the distance range that can be focused later even with an image taken with a plenoptic camera. Therefore, in order to ensure that the desired image can be obtained later, the photographer must eventually change the focus adjustment position of the photographic lens, change the shooting distance, or change the focal length of the photographic lens. However, it may be necessary to take multiple shots.

  The present invention has been made in view of the above problems, and what kind of finished image can be obtained when post-processing is performed on an image obtained by photographing with a plenoptic camera. Is intended to be presented at the time of shooting.

According to an aspect of the present invention, a plenoptic camera comprises:
A reconstruction processing unit that performs a reconstruction process based on a predetermined reconstruction parameter for a light field image that can also obtain information on the depth direction of the subject, and generates a completed image;
A display unit capable of displaying an image;
A reconfiguration parameter setting unit that selects and sets the values of the extreme poles in a range that can be taken as a reconfiguration parameter when performing the reconfiguration process in the reconfiguration processing unit;
From the time of photographing a predetermined period, the image generated by the reconstruction processor in response to one value among the values of the poles that are set in the reconstruction parameter setting unit, in response to the other value A display processing unit that performs processing for simultaneously displaying the image generated by the reconstruction processing unit on the display unit.

  According to the present invention, it is possible to confirm at the time of shooting what kind of finished image can be obtained when post-processing is performed on an image obtained by photographing with a plenoptic camera. Become.

It is a figure which illustrates notionally an example of the optical structure of a plenoptic camera. It is a figure which illustrates notionally another example of the optical structure of a plenoptic camera. It is a figure which illustrates notionally the process at the time of reconstructing a light field image and obtaining a completed image. It is a block diagram explaining the internal structure of the plenoptic camera which concerns on embodiment of this invention. It is a figure explaining the example which produces | generates the completion image which changed the object depth with the plenoptic camera which concerns on the 1st Embodiment of this invention, and displays the display image based on this completion image on a display part. (A), (b) is an example in which images with different depths of field are displayed according to the user's operation, and (c), a plurality of images with different depths of field are simultaneously displayed on the display unit. It is a figure explaining an example. It is a figure explaining the example which produces | generates the completion image which changed the viewpoint with the plenoptic camera which concerns on the 2nd Embodiment of this invention, and displays the display image based on this completion image on a display part, (A), (b) is an example in which images with different viewpoint positions are displayed according to user operations, and (c) is an example in which a plurality of images with different viewpoint positions are simultaneously displayed on the display unit. It is a figure explaining. It is a figure explaining the example which produces | generates the completion image which changed the focusing distance with the plenoptic camera which concerns on the 3rd Embodiment of this invention, and displays the display image based on this completion image on a display part. Yes, (a) and (b) are examples in which images with different in-focus distances are displayed according to user operations, and (c) is a plurality of images with different in-focus distances displayed simultaneously on the display unit. It is a figure explaining an example.

  1 and 2 are conceptual diagrams for explaining an arrangement example of optical elements of a plenoptic camera according to an embodiment of the present invention. 1 and 2 differ in the arrangement position of the microlens array along the optical axis direction of the photographing lens. In FIG. 1 and FIG. 2, each component is drawn in a size different from the actual size.

  1 and 2, the microlens array 104 is configured by arranging a large number of microlenses in two dimensions. Each microlens constituting the microlens array 104 guides subject light to a partial region on the light receiving surface of the image sensor 106. Each partial region includes a plurality of pixels arranged two-dimensionally. 1 and 2, the photographing lens 102 is shown as a single lens element, but the photographing lens 102 may be composed of a plurality of lens elements. The taking lens 102 may be a single focus lens or a lens whose focal length can be changed. The taking lens 102 may be of a fixed focus, but it is desirable that the taking lens 102 be configured to be able to perform focus adjustment (focusing). In FIGS. 1 and 2, the numbers of microlenses and image sensors are different from the actual ones.

  FIG. 1 shows an example in which a microlens array 104 is disposed so as to be positioned on a focal plane (imaging plane) of the photographing lens 102 and an image sensor 106 is disposed behind the microlens array 104. Here, if the focal length of each microlens constituting the microlens array 104 is f, the microlens array 104 captures an image so that the distance between the main surface of the microlens and the light receiving surface of the image sensor 106 is f. Positioned and fixed with respect to the element 106. As shown in FIG. 1, the subject light emitted from the photographing lens 102 toward the microlens array 104 is guided onto the light receiving surface of the image sensor 106 through each microlens constituting the microlens array 104.

  In FIG. 2, the microlens array 104 is arranged so that the focal plane (imaging plane) of the photographing lens 102 and the object point of each microlens constituting the microlens array 104 coincide with each other, and the image sensor 106 is arranged behind the microlens array 104. An example in which is arranged will be shown. The microlens array 104 is positioned with respect to the image sensor 106 so that the image point of the microlens and the light receiving surface of the image sensor 106 coincide. In other words, when the object distance of the microlens is a, the image distance is b, and the focal distance is f, each component is positioned in a positional relationship that satisfies (1 / a) + (1 / b) = (1 / f). Is done.

  With the configuration shown in FIG. 1 or FIG. 2, what optical path the light emitted from the subject, transmitted through the photographing lens 102 and the microlens array 104 and incident on each pixel on the image sensor 106 has passed through. It becomes possible to specify for each pixel. In other words, conventional cameras obtain color and brightness information for each pixel, while plenoptic cameras obtain information on the incident direction of light rays in addition to obtaining color and brightness information for each pixel. Can be obtained. The image sensor 106 may be a monochrome image sensor or an image sensor in which an on-chip color filter is disposed on a pixel. In this specification, the image sensor 106 includes an on-chip color filter having a Bayer array. The image data read out from the image sensor 106 and demosaiced is referred to as a light field image in this specification.

  FIG. 3 is a diagram conceptually illustrating a processing procedure for performing a reconstruction process on a light field image so that the light field image can be displayed as a normal image. In FIG. 3, all are shown as erect images in order to make the drawing easy to see. A primary image PI is formed by the photographing lens 102, and a secondary image SI is formed on the light receiving surface of the image sensor 106 by the microlens array 104. The secondary image SI is picked up by the image pickup device 106 and subjected to demosaic processing to generate a light field image LFI. In FIG. 3, a portion of the light field image LFI extracted and enlarged is shown as the light field image LFI.

  A reconstruction process based on a predetermined reconstruction parameter is performed on the light field image LFI to generate a reconstruction image RI. Then, by performing a reconstruction process on the entire light field image LFI, a completed image FRI desired by the user is generated. The completed image FRI is a bitmap image that can generate an image that can be viewed by the user through processing such as display and printing. When performing the reconstruction process to generate the completed image FRI, by setting various reconstruction parameters according to the purpose, the subject located at the desired shooting distance was focused as described in detail later An image, an image from a desired depth of field, or a desired viewpoint position can be generated as a completed image FRI.

  In the process of reconstructing an image, by determining which pixel position in the light field image LFI is to be selected, it is determined which light path from the object field to reach the image sensor 106 is selected. Can be determined. That is, by selecting and reconstructing light rays that reach the image sensor 106 through a desired optical path from the light field image LFI, for example, an image in which an image of a subject located at a specific shooting distance becomes sharper is generated. Is possible.

  Also, corresponding to each subject located at different shooting distances within the shooting range, an image in which the images of each subject are sharper is obtained and synthesized to obtain an image with a desired depth of field. Can be generated. Furthermore, it is possible to obtain an image with a deeper depth of field by selecting and reconstructing a light beam that has passed through a portion with a large F value (portion with a small aperture diameter) of the photographing lens 102.

  Further, by setting a partial area within the aperture stop of the photographing lens 102, that is, a sub-aperture area, and extracting and reconstructing information corresponding to the light beam passing through the sub-aperture area from the light field image, It is possible to obtain an image corresponding to a subject photographed from a specific viewpoint position. For example, by obtaining a circular area smaller than the aperture of the photographing lens 102 as a sub-opening area and setting the center position of the sub-opening area to a different position off the axis of the photographing lens 102, images of different viewpoints can be obtained. Can do. For example, when the aperture of the photographing lens 102 is viewed from the rear of the camera 100 and the sub aperture areas are set to the upper side, the lower side, the right side, and the left side, images from different viewpoints can be obtained. The range in which the viewpoint can be moved depends on the size of the stop aperture (exit pupil diameter) of the photographing lens 102. That is, as the diameter of the aperture opening is larger, the position of the sub-opening region can be greatly changed, and the range in which the viewpoint can be moved is expanded. Further, if the size of the sub-opening area to be set is made as small as possible, the viewpoint movable range can be expanded accordingly.

  FIG. 4 is a block diagram illustrating a configuration of a plenoptic camera 100 (hereinafter simply referred to as camera 100) according to an embodiment of the present invention. The camera 100 includes a photographing lens 102, a microlens array 104 (the microlens array is denoted as MLA in FIG. 4), an image sensor 106, and an analog front end 108 (analog front end in FIG. 4). Is represented as AFE), and a lens driving unit 110. The camera 100 further includes a CPU 122, a storage unit 124, an operation unit 130, a display unit 140, a display processing unit 142, a trigger input unit 144, a system bus 150, a reconstruction parameter setting unit 160, and image processing. Unit 170. The image recording medium 120 is configured by a flash memory card or the like and is configured to be detachable from the camera 100.

  The photographing lens 102 may be fixed to the camera 100 or may be configured to be detachable from the camera 100. As described above, the photographing lens 102 may be a single focus lens, a variable focal length lens, a fixed focus lens, or a focus adjustable lens. Here, it is assumed that the photographing lens 102 is a lens with adjustable focus. The lens driving unit 110 controls the movement of the focus adjustment lens in the photographing lens 102.

  An image obtained by photographing with the camera 100 is recorded on the image recording medium 120. The CPU 122 comprehensively controls the operation of the camera 100. The storage unit 124 includes a ROM 126 and a RAM 128. The ROM 126 is configured by a flash memory or the like, and stores firmware, camera control parameters, status information, and the like that need to be retained even when power is not supplied to the camera 100. The RAM 128 is configured by an SDRAM or the like having a relatively high access speed, and is configured to be accessible from both the CPU 122 and the image processing unit 170, and functions as a work area for the CPU 122 and the image processing unit 170. The reconstruction parameter setting unit 160, the trigger input unit 144, and the display processing unit 142 can be realized by the CPU 122 executing a program loaded in the RAM 128.

  The operation unit 130 includes one or more of a slide switch, a push button switch, a touch panel, a dial switch, and the like, and the CPU 122 detects an operation by the user. Then, the CPU 122 controls the camera 100 so that the operation according to the user's operation is performed. The display unit 140 includes a TFT color liquid crystal display device and a backlight device, and is configured to display a color image, characters, icons, and the like. The display unit 140 may be composed of a self-luminous display device such as an organic EL color display panel. In the present embodiment, it is assumed that a touch panel is provided on the surface of display unit 140.

  The image sensor 106 may be either a CCD image sensor or a CMOS image sensor. The analog front end 108 performs amplification, noise reduction processing, A / D conversion processing, and the like on the analog image output from the image sensor 106 to generate a digital image. In this specification, the image sensor 106 is a CMOS image sensor, and an analog front end 108 is built in the image sensor 106 so that a digital image can be output.

  The image processing unit 170 performs processing such as demosaic processing, white balance adjustment, noise reduction, and saturation / contrast adjustment on the image output from the image sensor 106. The image processing unit 170 also includes a reconstruction processing unit 172. The reconstruction processing unit 172 performs a reconstruction process, which will be described later, on the light field image after the demosaic process to generate a completed image FRI. The display processing unit 142 generates a display image based on the completed image FRI and displays it on the display unit 140. The display processing unit 142 also performs processing for displaying a preview image (live view image) or a postview image on the display unit 140 when the camera 100 is operating in the recording mode.

  The trigger input unit 144 issues a control signal to the reconstruction processing unit 172 when a trigger described below is input. When the reconstruction processing unit 172 receives this control signal, execution of processing for reconstructing the light field image obtained at that time and generating a completed image FRI is started.

  The trigger input by the trigger input unit 144 may be, for example, a signal input from a timer (not shown) that emits a signal when a certain time has elapsed after the power is turned on and the operation mode is switched to the recording mode. it can. Alternatively, when the camera 100 is performing a preview display operation, it can be detected that the user has operated the operation unit 130 and this can be used as a trigger. The operation of the operation unit 130 by the user is an operation of a touch panel, a push button switch, a diamond switch, or the like for starting execution of a process for displaying a display image based on the final image FRI obtained by the reconstruction process. It is possible. Alternatively, it is also possible to input as a trigger that a signal is emitted from the timer when a predetermined time has elapsed after the start of post-view display. It is also possible to detect that the user has operated the operation unit 130 during post-view display and use this as a trigger.

  The reconfiguration parameter setting unit 160 sets a reconfiguration parameter that is a parameter for specifying the processing content when the reconfiguration processing is performed by the reconfiguration processing unit 172. The reconstruction parameter setting process in the reconstruction parameter setting unit 160 will be described in detail later.

  The camera 100 may be capable of capturing a still image or may be capable of capturing a moving image. Hereinafter, the operation of the camera 100 will be described as an example of capturing a still image.

− First embodiment −
The reconstructed image display process performed by the camera 100 according to the first embodiment of the present invention will be described below. FIG. 5 is a diagram showing a display unit 140 provided in the camera 100, and a dial switch 134 and a seesaw switch 132 as operation units. These display unit 140, dial switch 134, and seesaw switch 132 are provided on the back surface of the camera 100. In addition, it is assumed that a touch panel is provided on the surface of the display unit 140. The display unit 140 displays a preview image (live view image), a post-view image, or a display image based on the completed image FRI generated by the reconstruction process and a GUI display 500 or the like. In the present embodiment, it is assumed that a GUI display 500 simulating a slide-type volume is displayed on a display image.

  The user operates the dial switch 134 and the seesaw switch 132 when a preview display similar to that performed by a conventional camera is being performed. At this time, the trigger input unit 180 inputs, as a trigger, an operation for starting execution of processing for displaying a reconstructed image, and issues a command for starting execution of the reconfiguration processing to the reconfiguration processing unit 172. . The reconstruction processing unit 172 performs reconstruction processing on the light field image obtained at the time of command input or thereafter. Then, a display image based on the generated completed image FRI is displayed on the display unit 140 together with the GUI display 500.

  The completed image FRI may be generated by performing reconstruction processing on the light field image obtained by performing the photographing operation. In that case, the trigger input unit 180 inputs as a trigger that the user has operated the dial switch 134, the seesaw switch 132, or the touch panel while the post-view display after the photographing operation is being performed. Further, the trigger input unit 180 may input a signal issued from a timer when a predetermined time has elapsed after the start of the post-view display. Furthermore, the trigger input unit 180 can input as a trigger that the user operated the dial switch 134, the seesaw switch 132, or the touch panel during the post-view display.

  When the user touches the displayed portion of the slide lever 502 with a finger or the like and slides (drags) along the vertical direction of the display unit 140, the display position of the slide lever 502 moves up and down. Configuration parameters change. Although FIG. 5 shows an example where the slide lever 502 is slid up and down, it may be slid left and right. Also, the display position of the slide lever 502 changes in response to the dial switch 134 being rotated clockwise or counterclockwise, or the seesaw switch 312 is tilted up or down or left and right. The configuration parameter may also be changed. In that case, a touch panel becomes unnecessary.

  FIG. 5 shows a case where the depth of field of the image generated by the reconstruction process is changed and displayed. FIG. 5A shows an example in which an image with a shallow depth of field is displayed on the display unit 140 in accordance with the slide lever 502 being slid upward and positioned on the “shallow” side. In FIG. 5A, it is assumed that images of the three-wheeled flowers FL1, FL2, and FL3 are displayed on the display unit 140. The flower FL1 is located at a shooting distance relatively close to the camera 100, the flower FL3 is located at a relatively long shooting distance, and the flower FL2 is located at an intermediate shooting distance. Hereinafter, the distances at which the flowers FL1, FL2, and FL3 are located are referred to as short distance, medium distance, and long distance.

  FIG. 5A shows that only the flower FL1 located at a short distance forms a sharp image, and the images of the flowers FL2 and FL3 located at a middle distance and a long distance are blurred. That is, an image with a shallow depth of field is displayed. That is, in accordance with the user positioning the slide lever 502 on the “shallow” side, the reconstruction parameter setting unit 160 performs reconstruction so that the depth of field of the image obtained by the reconstruction process becomes shallow. Set the parameters. The reconstruction processing unit 172 performs a reconstruction process based on the reconstruction parameter set by the reconstruction parameter setting unit 160 on the light field image to generate a completed image FRI.

  FIG. 5B shows an example in which an image with a deep depth of field is displayed on the display unit 140 in accordance with the slide lever 502 being slid down and positioned on the “depth” side. FIG. 5B shows an example in which an image in which all the flowers FL1, FL2, and FL3 located from a short distance to a long distance form a sharp image, that is, an image having a deep depth of field is displayed on the display unit 140. . In response to the user positioning the slide lever 502 on the “depth” side, the reconstruction parameter setting unit 160 sets the reconstruction parameter so that the depth of field of the image obtained by the reconstruction process becomes deeper. Set.

  The reconstruction processing unit 172 performs reconstruction processing based on the reconstruction parameter set by the reconstruction parameter setting unit 160 on the light field image. In this case, the reconstruction processing unit 172 focuses on the image FL1 located at a short distance, the image focused on the flower FL2 located at a medium distance, and the flower FL3 located at a long distance. An image is generated by reconstruction processing. Then, the image processing unit 170 performs a process of synthesizing these images to generate a completed image FRI having a deep depth of field. Alternatively, the reconstruction processing unit 172 completes a deep depth of field by performing processing for extracting and reconstructing light rays that have passed through a portion having a relatively small aperture diameter (small aperture portion) in the photographing lens 102. It is also possible to generate an image FRI.

  In the above, with reference to FIG. 5A and FIG. 5B, the reconfiguration processing unit 172 performs the reconfiguration process based on the reconfiguration parameter set by the reconfiguration parameter setting unit 160 according to the user operation. In the above example, the completed image FRI corresponding to the user operation is generated and the display image is displayed. However, the following processing is also possible. FIG. 5C illustrates an example in which a plurality of images having different depths of field are displayed side by side on the display unit 140. FIG. 5C shows an example in which an image Is having a shallow depth of field and an image Id having a deep depth of field are displayed side by side, but three or more images may be displayed side by side. At this time, the images may be displayed so as to overlap each other, and the size of the displayed images may be different among a plurality of images. Also, a plurality of images may be switched over time and displayed on the display unit 140 in order.

  When the display illustrated in FIG. 5C is performed, the trigger input unit 180 may input a trigger when the user performs an operation of half-pressing the release switch, or the user may input a dial switch. A trigger may be input when the 134 or the seesaw switch 132 is operated. Alternatively, the trigger input unit 180 inputs a signal generated from the timer when a predetermined time has elapsed since the user performed operations such as turning on the power of the camera 100, switching to the shooting mode, and half-pressing. May be. Further, even when the trigger input unit 180 inputs a signal generated from a timer when a user operation is detected during post-view display after shooting, or when a certain time has elapsed since the start of post-view display. Good. Until the display illustrated in FIG. 5C is started or after the display illustrated in FIG. 5C is completed, a preview display similar to that performed by a normal digital camera is performed. It is possible.

  When the display described with reference to FIG. 5C is performed, the extreme values of the range that can be taken as reconstruction parameters when the reconstruction processing unit 172 performs reconstruction processing on the light field image are set. Including the plurality of reconstruction parameters, the reconstruction parameter setting unit 160 sets them. In this embodiment, the extreme values of the range that can be taken as reconstruction parameters when performing reconstruction processing are, for example, the reconstruction parameter with the shallowest depth of field and the reconstruction with the deepest depth of field. It can be a configuration parameter. At this time, the focus adjustment position (focusing distance) currently set in the photographing lens 102, the focal length and aperture value of the photographing lens 102, and the permissible circle of confusion (the sharpness of the image that can be regarded as being in focus substantially). The depth of field can be set. That is, the depth of field is the shallowest when the permissible circle of confusion is minimum (0), and the depth of field is the deepest when the permissible circle of confusion is maximized.

  Based on the reconstruction parameters, the reconstruction processing unit 172 performs reconstruction processing to generate a plurality of (plural types) of completed images FRI, and display images based on these completed images FRI are simultaneously displayed on the display unit 140. Or they are displayed sequentially. As a result, in the present embodiment, a plurality of images including at least the shallowest image and the deepest image are displayed. By viewing these display images, the user can intuitively determine to what extent a completed image FRI in focus can be obtained later or whether an image suitable for the drawing intention can be obtained. You can check it, and you can shoot with confidence.

− Second Embodiment −
FIG. 6 is a diagram showing the display unit 140, the dial switch 134, and the seesaw switch 132 provided in the camera 100, similar to that shown in FIG. 5. It is assumed that a touch panel is provided on the surface of the display unit 140. The display unit 140 displays a GUI display 600 or the like together with a live view image, a post view image, or a display image based on the completed image FRI generated by the reconstruction process. In the present embodiment, the GUI display 600 indicates a range in which the viewpoint can be moved by a broken-line circle, and a currently set viewpoint position within the range is indicated by a cross cursor 602. Here, it is assumed that the display unit 140 is provided on the back surface of the camera 100. In the following description, left and right mean left and right toward the subject. In addition, “up” and “down” mean directions toward the heavens and the earth. FIG. 6 shows a state where an image when the viewpoint is changed is displayed on the display unit 140 by the reconstruction process. The method for obtaining the completed image FRI with the viewpoint changed is as described above. That is, the size and position of a sub-aperture area smaller than the stop aperture is identified within the aperture of the photographic lens, and information corresponding to the light beam passing through the sub-open area is extracted from the light field image and reproduced. By configuring, it is possible to obtain an image with a changed viewpoint.

  The user operates the dial switch 134 and the seesaw switch 132 when a preview display similar to that performed by a conventional camera is being performed. At this time, the trigger input unit 180 inputs, as a trigger, a user's operation for starting execution of processing for displaying a reconstructed image, and sends a command to the reconfiguration processing unit 172 to start execution of reconfiguration processing. To emit. The reconstruction processing unit 172 performs reconstruction processing on the light field image obtained at the time of command input or thereafter. Then, a display image based on the generated completed image FRI is displayed on the display unit 140 together with the GUI display 600.

  The completed image FRI may be generated by performing reconstruction processing on the light field image obtained by performing the imaging operation, as described in the first embodiment. In that case, the trigger input unit 180 inputs as a trigger that the user has operated the dial switch 134, the seesaw switch 132, or the touch panel while the post-view display after the photographing operation is being performed. Further, the trigger input unit 180 may input a signal issued from a timer when a predetermined time has elapsed after the start of the post-view display.

  The display position of the cursor 602 is changed vertically and horizontally within the range of the broken line by the user's touching the part where the cursor 602 is displayed and sliding the display unit 140 along the vertical and horizontal directions. However, the reconstruction parameters also change. Further, the display position of the cursor 602 may be changed and the reconstruction parameter may be changed in accordance with an operation of tilting the seesaw switch 312 in any one of up, down, left and right directions.

  FIG. 6A shows an example in which an image is displayed on the display unit 140 when the viewpoint is moved to the upper right according to the cursor 602 being slid to the upper right. In FIG. 6, as in FIG. 5, it is assumed that a flower FL1 located at a short distance, a flower FL2 located at a medium distance, and a flower FL3 located at a long distance are photographed.

  FIG. 6A shows a state in which the flower FL1 located at a short distance is displayed with a relatively shifted position in the lower left in the screen. That is, as the viewpoint is moved to the upper right, the image of the flower FL1, which is a subject closer to the camera 100, is more strongly affected by the movement of the viewpoint, and the amount of movement of the position of the flower FL1 varies. The amount of movement of the position of the flowers FL2 and FL3 becomes larger. As a result, in FIG. 5B and the like, a part of the stem and leaves of the flower FL2 that is hidden behind the shadow of the flower FL1 is visible in FIG. 6A. .

  That is, as the user positions the cursor 602 on the upper right side, the reconstruction parameter setting unit 160 causes the image obtained by the reconstruction process to be the one when the viewpoint is moved to the upper right side. Set the reconfiguration parameter to. The reconstruction processing unit 172 performs a reconstruction process based on the reconstruction parameter set by the reconstruction parameter setting unit 160 on the light field image to generate a completed image FRI. An example in which a display image based on the completed image FRI is displayed on the display unit 140 is shown in FIG.

  FIG. 6B shows an example in which an image when the viewpoint is moved to the upper left in accordance with the cursor 602 being positioned on the upper left is displayed on the display unit 140. In response to the user positioning the cursor 602 on the upper left side, the reconstruction parameter setting unit 160 reconstructs the image obtained by the reconstruction process so that the image obtained when the viewpoint is moved to the upper left side. Set configuration parameters. The reconstruction processing unit 172 performs reconstruction processing based on the reconstruction parameter set by the reconstruction parameter setting unit 160 on the light field image. In this case, a state in which the flower FL1 located at a short distance is displayed with a relatively shifted position on the lower right in the screen is shown. In other words, as described above, as the viewpoint is moved to the upper left, the image of the flower FL1 that is the subject that is closer to the camera 100 is more greatly affected by the movement of the viewpoint, and the flower FL1 The amount of movement of the image position is larger than the amount of movement of the positions of the other flowers FL2, FL3. As a result, the position of the flower FL1 has moved to the right as compared with that shown in FIG.

  In the above, with reference to FIG. 6A and FIG. 6B, the reconstruction processing unit 172 performs the reconstruction process based on the reconstruction parameter set by the reconstruction parameter setting unit 160 according to the user operation. In the above example, the completed image FRI corresponding to the user operation is generated and the display image is displayed. However, the following processing is also possible. FIG. 6C shows an example in which a plurality of images with different viewpoints are displayed side by side on the display unit 140. In FIG. 6C, an image IUR (Upper Right) when the viewpoint is shifted to the upper right, an image IUL (Upper Left) when the viewpoint is shifted to the upper left, an image ILR (Lower Right) when the viewpoint is shifted to the lower right, and the lower left The image ILL (Lower Left) is displayed side by side, but three or less images or five or more images may be displayed side by side. At this time, the images may be displayed so as to overlap each other, and the size of the displayed images may be different among a plurality of images. Similarly to the first embodiment, a plurality of images may be switched over time and displayed on the display unit 140 sequentially.

  When the display illustrated in FIG. 6C is performed, the trigger input unit 180 inputs a trigger when the user performs an operation of half-pressing the release switch, as described in the first embodiment. Alternatively, a trigger may be input when the user operates the dial switch 134 or the seesaw switch 132. Alternatively, the trigger input unit 180 inputs a signal generated from the timer when a predetermined time has elapsed since the user performed operations such as turning on the power of the camera 100, switching to the shooting mode, and half-pressing. May be. Further, even when the trigger input unit 180 inputs a signal generated from a timer when a user operation is detected during post-view display after shooting, or when a certain time has elapsed since the start of post-view display. Good. Until the display illustrated in FIG. 6C is started or after the display illustrated in FIG. 6C is completed, a preview display similar to that performed by a normal digital camera is performed. It is possible.

  When the display described with reference to FIG. 6C is performed, as in the case of the first embodiment, the reconstruction processing unit 172 performs the reconstruction process on the light field image. A plurality of reconstruction parameters are set by the reconstruction parameter setting unit 160 including the values of the two extreme poles that can be taken as the reconstruction parameters. For example, in this example, the reconstruction parameters corresponding to the positions of the viewpoints at the right end and the left end within the changeable range can be set to the values of both poles. Further, the reconstruction parameters corresponding to the upper end, lower end, upper right, lower left, upper left, and lower right within the changeable range can be set to the values of both poles.

  Based on the reconstruction parameters set as described above, the reconstruction processing unit 172 performs reconstruction processing to generate a plurality of (plural types) of completed images FRI, and display images based on these completed images FRI. Are simultaneously or sequentially displayed on the display unit 140. In the example of FIG. 6C, the image IUR when the viewpoint is shifted to the upper right, the image IUL when shifted to the upper left, the image ILR when shifted to the lower right, and the image ILL when shifted to the lower left are displayed side by side. Is done. By viewing these display images, the user can intuitively check at the time of shooting whether the part hidden behind the subject on the near side and not visible can be seen in the later processing. Shooting can be performed.

  As described above, in the second embodiment, the example of generating the completed image FRI in which the viewpoint is changed from the light field image has been described. However, it is also possible to generate a so-called stereo image. That is, it is possible to generate a set of images (stereo pairs) with parallax set in the left-right direction from the light field images. In this case, the display unit 140 may be configured to allow the user to view a stereoscopic image with the naked eye, or to allow the user to view the stereoscopic image by wearing a glasses-type adapter. Then, the reconstruction parameter set by the reconstruction parameter setting unit 160 can be the amount of parallax between images of a stereo pair, that is, the sense of depth of a displayed stereoscopic image. In the same manner as described with reference to FIG. 6C, when a plurality of stereoscopic images having different feelings of depth are displayed simultaneously or sequentially, the reconstruction parameter setting unit 160 causes the reconstruction processing unit 172 to execute the light field. A plurality of reconstruction parameters are set including the values of the two extreme poles that can be taken as reconstruction parameters when performing reconstruction processing on an image. In this case, as the values of the two extreme poles that can be taken as the reconstruction parameters, for example, the parallax amount corresponds to 0 (depth sense 0) and the parallax amount corresponds to the maximum (depth sense is maximum). Can do. The parallax amount corresponding to the maximum is a reconstruction parameter for generating a pair of completed images FRI corresponding to the leftmost and rightmost viewpoints in a range where the viewpoint can be set.

− Third embodiment −
FIG. 7 is a diagram showing the display unit 140, the dial switch 134, and the seesaw switch 132 provided in the camera 100, similar to those shown in FIGS. 5 and 6. It is assumed that a touch panel is provided on the surface of the display unit 140. On the display unit 140, a GUI display 700 and the like are displayed together with a preview image, a postview image, or a display image based on a completed image generated by reconstruction processing. In the present embodiment, it is assumed that a GUI display 700 simulating a slide-type volume is made. FIG. 7 shows a case where an image as if the in-focus position has been changed is generated and displayed by reconstruction processing.

  The user operates the dial switch 134 and the seesaw switch 132 when a preview display similar to that performed by a conventional camera is being performed. At this time, the trigger input unit 180 inputs, as a trigger, an operation for starting execution of processing for displaying a reconstructed image, and issues a command for starting execution of the reconfiguration processing to the reconfiguration processing unit 172. . The reconstruction processing unit 172 performs reconstruction processing on the light field image obtained at the time of command input or thereafter. Then, a display image based on the generated completed image FRI is displayed on the display unit 140 together with the GUI display 700.

  The completed image FRI may be generated by performing reconstruction processing on the light field image obtained by performing the imaging operation, as described in the first embodiment. In that case, the trigger input unit 180 inputs as a trigger that the user has operated the dial switch 134, the seesaw switch 132, or the touch panel while the post-view display after the photographing operation is being performed. Further, the trigger input unit 180 may input a signal issued from a timer when a predetermined time has elapsed after the start of the post-view display.

  When the user applies a finger or the like to the displayed portion of the slide lever 702 and slides the display unit 140 along the vertical direction, the display position of the slide lever 702 changes up and down, and the reconstruction parameter also changes. To do. Although FIG. 7 shows an example in which the slide lever 702 is slid up and down, it may be slid left and right. Also, the display position of the slide lever 702 changes in response to an operation of tilting the seesaw switch 312 in the vertical direction or the horizontal direction, or an operation of rotating the dial switch 134 in either the clockwise or counterclockwise direction. The configuration parameter may also be changed.

  FIG. 7A shows that the image of the subject located at a farther position in the shooting range becomes sharper as the slide lever 702 is slid upward and positioned on the “far” side. The example displayed on the display part 140 is shown. In the present specification, the distance from the camera 100 to the subject from which a sharp image is obtained in the completed image obtained by the reconstruction process is referred to as “focusing distance”. In FIG. 7, as in FIG. 5, it is assumed that a flower FL1 located at a short distance, a flower FL2 located at a medium distance, and a flower FL3 located at a long distance are photographed. That is, the focusing distance of the flower FL3 is longer (far) than the focusing distance of the flowers FL1 and FL2. FIG. 7A shows how the light field image is reconstructed so that the image of the flower FL3 located at a long distance becomes sharper, a completed image FRI is generated, and the display image is displayed.

  Corresponding to the fact that the slide lever 702 is positioned on the “far” side, the reconstruction parameter setting unit 160 shows that the image obtained by performing the reconstruction process is a sharp image of a subject with a longer (far) focus distance. Set the reconfiguration parameters so that The reconstruction processing unit 172 performs a reconstruction process based on the reconstruction parameter set by the reconstruction parameter setting unit 160 on the light field image, and generates a completed image FRI. Based on this completed image FRI, a display image illustrated in FIG. 7A is displayed on the display unit 140.

  FIG. 7B shows a state in which the user has slid the slide lever 702 downward to be positioned on the “near” side, and an object with a shorter (near) focus distance, that is, an image of the flower FL1. An example in which a sharper image is displayed on the display unit 140 is shown. In response to the user positioning the slide lever 702 on the “near” side, the reconstruction parameter setting unit 160 determines that the image of the subject closer to the in-focus distance is sharp among the images obtained by the reconstruction process. Set the reconfiguration parameters to The reconstruction processing unit 172 performs reconstruction processing based on the reconstruction parameter set by the reconstruction parameter setting unit 160 on the light field image, and generates a completed image FRI in which the image of the flower FL1 is sharp. Based on the completed image FRI, the display image illustrated in FIG. 7B is displayed on the display unit 140. Instead of the operation of the slide lever 702 described above or in addition to the operation of the slide lever 702, the image of the subject specified by the user touching the touch panel in the image displayed on the display unit 140 is sharpened. A reconstruction parameter may be set. At this time, the user may designate a plurality of subjects in the display image, and the reconstruction parameter may be set so that all the images of the designated subjects are sharp.

  7A and 7B, the reconfiguration processing unit 172 performs the reconfiguration process based on the reconfiguration parameter set by the reconfiguration parameter setting unit 160 according to the user operation. The example in which the completed image FRI corresponding to the user operation is generated and the corresponding display image is displayed has been described. However, the following processing can also be performed. FIG. 7C illustrates an example in which a plurality of images that connect sharp images corresponding to subjects with different in-focus distances are displayed side by side on the display unit 140. In FIG. 7C, an image In that connects a sharp image to the flower FL1 that is a subject with a short focusing distance, and an image If that connects a sharp image to the flower FL3 that is a subject with a long focusing distance. Are displayed side by side, but as described with reference to FIG. 5 in the first embodiment, three or more images may be displayed side by side. At this time, the images may be displayed so as to overlap each other, and the size of the displayed images may be different among a plurality of images. In addition, a plurality of images may be switched over time and sequentially displayed on the display unit 140.

  When the display illustrated in FIG. 7C is performed, the trigger input unit 180 inputs a trigger when the user performs an operation of half-pressing the release switch, as described in the first embodiment. Alternatively, a trigger may be input when the user operates the dial switch 134 or the seesaw switch 132. Alternatively, the trigger input unit 180 inputs a signal generated from the timer when a predetermined time has elapsed since the user performed operations such as turning on the power of the camera 100, switching to the shooting mode, and half-pressing. May be. Further, even when the trigger input unit 180 inputs a signal generated from a timer when a user operation is detected during post-view display after shooting, or when a certain time has elapsed since the start of post-view display. Good. Until the display illustrated in FIG. 7C is started or after the display illustrated in FIG. 7C is completed, the same preview display as that performed by a normal digital camera is performed. It is possible.

  When the display described with reference to FIG. 7C is performed, the reconstruction processing unit 172 performs the reconstruction process on the light field image, as described in the first embodiment. A plurality of reconstruction parameters are set by the reconstruction parameter setting unit 160 including the values of the two extreme poles that can be taken as the reconstruction parameters. For example, in this example, the reconstruction parameter corresponding to the shortest and longest in-focus distance within the changeable range can be a bipolar value. At this time, the focus adjustment position (focusing distance) currently set in the photographing lens 102, the focal length and aperture value of the photographing lens 102, and the permissible circle of confusion (the sharpness of the image that can be regarded as being in focus substantially). It is possible to determine the depth of field from (degree) and to set the shortest and longest focus distance.

  Based on the reconstruction parameters set as described above, the reconstruction processing unit 172 performs reconstruction processing to generate a plurality of (plural types) of completed images FRI, and display images based on these completed images FRI. Are simultaneously or sequentially displayed on the display unit 140. In the example of FIG. 7C, images corresponding to the shortest and longest focus distances are displayed side by side. By viewing these display images, the user can intuitively confirm the range of the focus distance at which a sharp image can be obtained, and can shoot with confidence. In addition, by sequentially displaying a plurality of images in which the focusing distance is changed little by little like a moving image, it is also possible to reproduce and display images full of emotions, and users can enjoy new enjoyment in using the camera 100. It is possible to bring

  The example in which the reconstruction process is performed on the light field image of the still image and the display is performed according to the first to third embodiments has been described above, but the same process can be performed on the moving image. It is.

  The present invention is applicable to products such as a digital still camera and a digital movie camera.

  As mentioned above, although this invention was demonstrated based on some embodiment, this invention is not limited to embodiment mentioned above, A various deformation | transformation and application are possible within the range of the summary of this invention. Of course.

  Further, various inventions can be extracted from the above embodiment. For example, even if several constituent requirements are deleted from all the constituent requirements shown in the embodiment, if the effect can be obtained, the configuration from which the constituent requirements are deleted can also be extracted as an invention.

100… plenoptic camera (camera)
102 ... Shooting lens 104 ... Micro lens array 106 ... Image sensor 108 ... Analog front end (AFE)
110: Lens driving unit 120 ... Image recording medium 122 ... CPU
124 ... Storage unit 130 ... Operation unit 132 ... Seesaw switch 134 ... Dial switch 140 ... Display unit 150 ... System bus 160 ... Reconfiguration parameter setting unit 170 ... Image processing unit 172 ... Reconfiguration processing unit 500, 600, 700 ... GUI display 502, 702 ... Slide lever 602 ... Cursor

Claims (5)

A reconstruction processing unit that performs a reconstruction process based on a predetermined reconstruction parameter for a light field image that can also obtain information on the depth direction of the subject, and generates a completed image;
A display unit capable of displaying an image;
A reconfiguration parameter setting unit that selects and sets the values of the extreme poles in a range that can be taken as a reconfiguration parameter when performing the reconfiguration process in the reconfiguration processing unit;
From the time of photographing a predetermined period, the image generated by the reconstruction processor in response to one value among the values of the poles that are set in the reconstruction parameter setting unit, in response to the other value A plenoptic camera comprising: a display processing unit that performs a process of simultaneously displaying an image generated by the reconstruction processing unit on the display unit. The plenoptic camera according to claim 1, wherein the reconstruction parameter is a parameter for changing a depth of field. The plenoptic camera according to claim 1, wherein the reconstruction parameter is a parameter for changing a specific viewpoint position. The plenoptic camera according to claim 1, wherein the reconstruction parameter is a parameter for changing a specific shooting distance. 5. The plenoptic camera according to claim 1, further comprising a trigger input unit that inputs a trigger that triggers the start of the reconfiguration processing in the reconfiguration processing unit.