JP5822700B2 - Image capturing method, image capturing apparatus, and program - Google Patents

Description

  The present invention relates to an image capturing method, an image capturing apparatus, and a program for improving the operability of a focusing operation.

  The viewfinder is a part used to determine the composition and focus on the camera. In a single-lens reflex camera, an image passing through a photographing optical system is formed on a focusing screen, and the image is confirmed (a reflex finder). By using the reflex finder, a focusing (focus adjustment) operation can be performed while confirming an image to be actually photographed.

  The focusing screen includes a mat type and a prism type. The mat type is a method of focusing by using a diffusion plate such as ground glass so that a region to be focused in the finder image is sharpened. The prism type is a method in which an image is divided when a focus is not achieved by forming a prism on a part of the focusing screen. As a prism type screen, a split prism in which two wedge-shaped prisms are vertically arranged on a screen and a micro prism in which a large number of small prisms are arranged side by side are known.

  In the mat type, since focusing is determined based on the sharpness of the image, it is necessary to search for a position where the sharpness is highest and to repeat the focusing operation. In addition, the size of the finder image and the visual acuity of the photographer greatly affect the focusing accuracy. However, there is an advantage that any region in the finder image can be focused. On the other hand, the prism type presents an image shift as a focus shift, so that accurate focusing with less hesitation than the mat type is possible. However, since the prism is formed on the finder screen, there is a problem that a region where focusing is possible is limited. For example, in most cases, since a prism is formed near the center of the finder image, focusing is possible only in that region. For this reason, mat-type screens have become the mainstream, especially for single-lens reflex cameras.

  In the digital camera, an electronic viewfinder method in which an image to be taken is displayed on a liquid crystal monitor provided in the camera body is the mainstream. Therefore, basically a finder image similar to the mat type can be obtained. However, due to problems such as low resolution of a liquid crystal monitor for displaying a finder image or low visibility of a liquid crystal finder outdoors, many methods for presenting a focus range more easily have been invented. For example, Patent Document 1 describes a method for enlarging and displaying a region to be focused. Patent Document 2 proposes a method for displaying the degree of focus.

JP 2010-051037 A JP 2002-341236 A

  In recent years, a technology called light field photography has been developed that can acquire images from multiple viewpoints by adding new optical elements to the optical system and adjust the focus position (refocus) later by image processing. ing. As an imaging technique for acquiring a light field, a Plenoptic Camera in which a microlens array is placed behind a main lens and a camera array in which small cameras are arranged are known. When the electronic finder method is adopted as the finder method of such a multi-view camera for photographing such a multi-viewpoint image, it is difficult to secure a fast and accurate focusing by the methods described in Patent Document 1 and Patent Document 2 described above. It is. This is because the above-described problem that the resolution of the liquid crystal monitor and the visual acuity of the photographer greatly affect the focusing accuracy remains as a fundamental problem.

  An object of the present invention is to realize a focus adjustment function like a prism type when an electronic viewfinder method is adopted in a multi-lens camera.

  An imaging apparatus according to the present invention is a multi-view imaging apparatus including a plurality of imaging units and an image display unit that performs live view display that displays a shooting range in real time, and includes a focus area in the live view display. Generating a prism type image using a focus area determining means for determining the position of the image and a parallax image captured by a predetermined imaging unit of the plurality of imaging units, and the generated prism type image is Live view image generation means for generating a live view image arranged at a position determined by the focus area determination means.

  According to the present invention, it is possible to realize a quick and accurate focusing operation at an arbitrary position in a finder image in a multi-view camera adopting an electronic finder system.

FIG. 3A is a front overview of a multi-view imaging apparatus that captures a multi-viewpoint image according to the first embodiment, and FIG. 1 is a diagram illustrating an external appearance of a multi-viewpoint photographing apparatus in Embodiment 1. FIG. (A) is a block diagram showing a circuit configuration of a multi-view camera according to the first embodiment, and (b) is a block diagram showing an internal configuration of an image processing unit according to the first embodiment. It is a figure which shows an example of arrangement | positioning of the imaging part in a camera array. It is a flowchart which shows the flow of a live view image generation process. (A) is an example in which the focus area is arranged at the center of the screen, and (b) is a diagram showing an example in which the focus area is arranged slightly on the upper left of the screen. 3 is a diagram illustrating a relationship between a split prism and a use imaging unit in Embodiment 1. FIG. It is a figure which shows an example of the selection candidate of the division direction of the split prism image displayed by a camera setting menu. It is explanatory drawing which expressed the content of the synthetic | combination process visually. It is a flowchart which shows the flow of a refocus process. (A) is a figure which shows the relationship between a virtual aperture parameter and a weighting coefficient, and is a figure which shows the positional relationship of a target imaging part and a center imaging part. It is a flowchart which shows the flow of a live view image generation process. It is a figure explaining the correlation with each prism area | region and an imaging part. It is explanatory drawing which expressed the content of the synthetic | combination process visually. It is an example of the live view image which made the range of a split prism image stand out from the periphery by reducing the brightness | luminance of parts other than a prism type image. It is a figure which shows an example of the imaging part used when employ | adopting a microprism image. It is an example of the live view image which made the range of a split prism image conspicuous from the circumference by making sharpness of parts other than a prism type image low. 12 is a flowchart illustrating a flow of processing for automatically determining a split direction of a split prism image according to a third embodiment.

[Example 1]
FIG. 1 is a perspective view showing an overview of a multi-view imaging apparatus (hereinafter simply referred to as “multi-view camera”) according to the present embodiment, which captures a multi-viewpoint image. 1 shows an overview of the front, and FIG. 1B shows an overview of the back.

  A camera array in which a plurality of imaging units 101 are arranged is provided in front of the multi-view camera body 100, and images (multi-view images) captured from a plurality of viewpoints can be obtained at the same time. A release button 106 for instructing the start of shooting is provided on the upper part of the multi-lens camera body 100. The imaging unit 101 includes a photographic lens 102, an optical filter 103, an imaging sensor 104, and a peripheral circuit 105. A card slot 113 for inserting an external recording medium (for example, a semiconductor memory card) is provided on the side surface of the multi-lens camera body 100.

  On the back of the multi-lens camera main body 100, an image display unit 107, a cross arrangement switch 108, a power button 109, a menu button 110, an enter button 111, and a cancel button 112 are provided. In this embodiment, since no optical finder is provided, a framing operation (confirmation of focus and composition) is performed using the image display unit 107. The image display unit 107 performs live view display for displaying the shooting range in real time, and displays a camera setting menu when the menu button 110 is pressed. When the menu button 110 is pressed with the camera setting menu displayed, the display returns to the live view display state. The image display unit 107 includes a touch panel 209, and a position designated by the photographer during live view display can be set as a focus area.

  In the camera setting menu displayed by the menu button 110, an item is first selected, and the state of the selected item is changed with the OK button 111, or the item is returned to the item selection without being changed with the cancel button 112. As items that can be changed in the camera setting menu, in addition to the split direction of the split prism image, which will be described later, an aperture value, a shutter speed, and the like used by each imaging unit 101 during shooting can be specified. In the present embodiment, the aperture value, shutter speed, shooting distance, and zoom position are all set to the same setting for each imaging unit 101 constituting the camera array, but needless to say, they may be set individually. Yes.

  FIG. 2A is a block diagram illustrating a circuit configuration of the multi-view camera 100 according to the present embodiment.

  The CPU 200 controls the operation of the entire camera, including processing of image data output from the imaging unit 101, display control of the LCD monitor 208, acquisition of input data from the touch panel 209, and power control instructions for each imaging unit.

  The switch (SW1) 205 is turned on when the release button 106 (see FIG. 1) is half-pressed, and when the switch (SW1) 205 is turned on, the multi-lens camera 100 enters a shooting preparation state. The switch (SW2) 206 is turned on when the release button 106 is pressed to the end (fully pressed state), and when the switch (SW2) 206 is turned on, the multi-view camera 100 starts taking a still image. After the exposure time determined in the shooting preparation state has elapsed, the multi-lens camera ends still image shooting.

  As illustrated in FIG. 1, the imaging unit 101 includes a photographing lens 102, an optical filter 103, an imaging sensor 104, and a peripheral circuit 105. The peripheral circuit 105 includes an imaging unit control circuit 201 and an image processing circuit 202. The imaging unit control circuit 201 performs lens control, shutter control, and the like. For example, as lens control, communication with the photographic lens 102, drive control of the photographic lens 102 when changing the photographic distance, drive control of the diaphragm blades, and the like are performed. In addition, appropriate exposure is performed on the image sensor 104 by shutter control. Image data obtained by the image sensor 104 is processed by the image processing circuit 202 and transmitted to the buffer memory 203 for performing distance estimation processing, image synthesis processing, and the like. The image processing unit 204 performs various types of image processing on the image data stored in the buffer memory 203, and generates image data corresponding to the output format. Details of the image processing unit 204 will be described later.

The backlight illumination 207, the LCD monitor 208, and the touch panel 209 constitute an image display unit 107. The storage device 210 is, for example, a semiconductor memory card that can be attached to and detached from the camera body by the card slot 111, a hard disk drive that is connected by an external terminal (not shown), and the like.
FIG. 3 is a diagram illustrating an example of the arrangement of the imaging units 101 in the camera array. Each rectangle represents the imaging unit 101, and the central imaging unit 101 illustrated in gray is hereinafter referred to as a center imaging unit. Of course, the arrangement of the imaging unit 101 is not limited to that shown in FIGS.

  In this embodiment, the case of a camera array in which each imaging unit is arranged as shown in FIG. 3A will be described as an example.

  FIG. 4 is a flowchart showing a flow of processing for generating an image (hereinafter referred to as “live view image”) used for live view display in the image processing unit 204. In the live view image, an image imitating a prism type such as a split prism (hereinafter referred to as “prism type image”) is displayed at the position determined as the focus area. As described above, the prism type includes a micro prism and a split prism. The difference between the two is only whether the prism area described later is finely divided or divided into two. Hereinafter, a case where a live view image in which an image similar to a split prism (hereinafter referred to as “split prism image”) is displayed as a prism type image in the focus area will be described as an example. Of course, live view display is performed using a live view image in which an image simulating a microprism (hereinafter referred to as a “microprism image”) is displayed, or a live view image using both a split prism image and a microprism image. May be.

  FIG. 2B is a block diagram illustrating an internal configuration of the image processing unit 204 according to the present embodiment.

  The image processing unit 204 includes a focus area determination unit 220, a division direction determination unit 230, an imaging unit determination unit 240, and a live view image generation unit 250. This process is realized by the CPU 200 executing a program read from a ROM (not shown) into the RAM.

  In step 401, the focus area determination unit 220 determines a focus area. Here, the focus area is an area indicating which position in the finder image (in the live view image) is to be focused. When the user designates via the touch panel 209, the focus area is determined at a position centered on the designated position. When there is no user designation, the focus area is determined at a default position (for example, the center of the screen). Then, the split prism image is displayed in the area determined as the focus area. FIG. 5 is a diagram showing an example of the position of the focus area. FIG. 5A shows the case where the focus area is arranged at the center of the screen, and FIG. 5B shows the case where it is arranged slightly on the left of the screen. Show.

  In step 402, the division direction determination unit 230 determines the division direction of the split prism image. The user can select one arbitrary direction from a plurality of selection candidates on the camera setting menu. FIG. 6 is a diagram illustrating an example of selection candidates for the split direction of the split prism image displayed in the camera setting menu. In FIG. 6, there are four types of split prism image division directions (i) to (iv) as selection candidates, and the imaging units used for each division direction are associated with each other, and each is represented by a black square. It is shown. When the user designates a predetermined direction in the camera setting menu, the designated direction is determined as the split direction of the split prism image. If the user omits this setting, the division direction of the split prism image is determined at a default position (for example, the horizontal direction).

  In step 403, the imaging unit determination unit 240 uses the predetermined imaging unit (see FIG. 6) as a captured image imaging unit used for generating a live view image according to the determined split direction of the split prism image. decide.

  In step 404, the live view image generation unit 250 synthesizes the determined captured images of the imaging unit to generate a live view image. FIG. 7 is an explanatory diagram visually expressing the contents of the synthesis process in this step. FIG. 7A shows a live view image generated when the horizontal direction shown in FIG. 6I is determined as the split direction of the split prism image. FIG. 7B shows a live view image generated when the direction of 45 degrees to the right shown in (ii) of FIG. 6 is determined as the split direction of the split prism image. In FIG. 7B, there is no imaging unit indicated by a black square on the line 716 indicating the division direction of the designated split prism image. Therefore, an interpolated image of images taken by two imaging units adjacent to the line 716, specifically, an interpolated image of images taken by the imaging units 712 and 713, and an image taken by the imaging units 714 and 715 A live view image is generated using the interpolated image.

<Live view image generation processing>
Live view image generation processing is based on the concept of refocus processing in light field photography. Therefore, the refocus process will be described first, and then the live view image generation process will be described.

Refocusing process The refocusing process is a process for synthesizing multi-viewpoint images and generating an image with an arbitrary shooting distance and an arbitrary depth of field. Specifically, based on the virtual aperture parameter for determining the depth of field, the virtual shooting distance corresponding to the shooting distance, and the imaging device information, each multi-viewpoint image is shifted and weighted and added to obtain an output image. . FIG. 8 is a flowchart showing the flow of the refocus process.

First, a weighting coefficient for a captured image of each imaging unit is calculated according to a Gaussian function based on the virtual aperture parameter (step 801). FIG. 9A shows the relationship between the virtual aperture parameter and the weighting coefficient. In FIG. 9A, w _m represents a weighting factor in an arbitrary target imaging unit P _m , and L _m represents a distance from the center imaging unit P _c to the target imaging unit P _m ((b) in FIG. 9). reference). It can be seen that the depth of field becomes deeper as the standard deviation value that determines the Gaussian function is smaller and becomes shallower as the standard deviation value is larger.

Next, the shift amount of the captured image by each imaging unit is calculated based on the virtual imaging distance (step 802). Horizontal shift amount Δi of the target image I _m when the virtual object distance was d (m, d) and vertical shift amount .DELTA.j (m, d) is represented by the following formula (1).

Here, W and H are the horizontal and vertical image sizes of the image, θ _w is the horizontal viewing angle of the imaging unit, and θ _h is the vertical viewing angle of the imaging unit. Further, (s _m , t _m ) is the position coordinate of the target imaging unit P _m on the xy plane of FIG. 9B, and (s ′, t ′) is the position of the center imaging unit P _c on the xy plane. Coordinates.

  Finally, the image shifted according to the shift amount calculated in step 802 is combined by applying the weighting factor calculated in step 801 (step 803). An output image H obtained by the synthesis process is expressed by the following equation.

Incidentally, w _m here means the weighting factor for the captured image I _m by the target imaging unit P _m.

  Through the processing as described above, the captured images of the respective imaging units can be superimposed and combined so that the image shift is smaller as the virtual shooting distance d is closer, and the virtual shooting distance d is in focus. An image can be obtained. In addition, by determining the weighting factor according to the virtual aperture parameter, it is possible to control the amount of blur in the out-of-focus portion.

Live View Image Generation Processing A live view image in which a prism type image (focus area) is displayed at an arbitrary position can be generated by shifting the captured image in the same way as the refocus processing described above. In the refocusing process, the amount of deviation of the captured image by each imaging unit is reduced at the in-focus portion, and the image is synthesized so as to increase as the focus is removed. Similarly, a split prism image that virtually realizes a split prism can be generated by shifting and displaying the captured images by the respective imaging units in accordance with the designated shooting distance.

  FIG. 10 is a flowchart showing the flow of live view image generation processing.

First, the imaging units determined in step 403 of the flowchart of FIG. 4 are associated with each prism area (step 1001). Here, the prism area is a component of the prism type image displayed at the position determined as the focus area. For example, in the case of a split prism image, it refers to each of the two divided areas (hereinafter, one area is described as “prism area R _A ” and the other area as “prism area R _B ”). . FIG. 11 is a diagram illustrating the association between each prism region and the imaging unit. In the example of FIG. 11, the upper half of the prism region R _A right end center of the imaging unit P _A is the prism region R _B of the lower half left center of the imaging unit P _B, respectively associated with.

  Next, as in the refocus process, the shift amount of the captured image by the imaging unit selected in step 403 of the flowchart of FIG. 4 is calculated (step 1002).

When calculating the shift amount, subsequently, to obtain the position of the prism region R _A and the prism region R _B (step 1003).

Finally, the captured images obtained from the respective imaging units are combined to generate a live view image (step 1004). Here, in step 803 in the refocusing process, weighted addition is performed on the captured image of each imaging unit to perform synthesis. In this process, for each area of the live view image to be generated, a composition process using images (pixels) photographed by different imaging units is performed. Specifically, in which area the pixel of the captured image of which imaging unit is used is as follows.
Area of the prism region _RA : Pixels of the captured image I _A of the imaging unit P _A Area of the prism region R _B : Pixels of the captured image I _B of the imaging unit P _B Other areas: Captured image I _C of the center imaging unit P _c Pixels

  When the live view image generated by the synthesis process is S, the pixel value of the pixels constituting S is expressed by the following equation.

FIG. 12 is an explanatory diagram visually representing the contents of the composition process. As described above, the pixels of the captured image I _A of the imaging section P _A is the area of the prism region R _A, the pixels of the captured image I _B of the image pickup unit P _B in the area of the prism region R _B, in other areas Center It is shown that the pixels of the captured image I _C of the imaging unit P _c are used.

  The live view image generated as described above is displayed on the image display unit 107, and live view display is performed.

In the present embodiment, the range (focus area) of the split prism image in the live view image is adjusted by adjusting the brightness of the captured image I _c of the center imaging unit P _c so that the photographer can easily recognize the split prism image. Make it stand out from the surroundings. That is, the captured image I _c used in the pixel in the portion other than the prism-type images, as for example, alpha = 0.7, dimming of the partial. 13, by decreasing the luminance of the captured image I _c, which is an example of a live view image stand out the scope of the split prism image from the surroundings is located in the upper left.

In the case of adopting the microprisms image instead of the split prism image, for example, a prism region R _A and the prism region R _B can be realized microprism image by alternately arranging tiles. Alternatively, the imaging unit is positioned above and below the right and left center imaging unit P _c as shown in (a) of FIG. 14, or FIG. 14 (b) like the center upper right of the image pickup unit P _c, lower right, left A captured image of the imaging unit located diagonally above and diagonally to the left may be used, and four parallax images may be arranged in a tile shape. Further, the split prism image and the micro prism image may be combined, the split prism image may be disposed at the center of the focus area, and the micro prism image may be disposed around the split prism image.

  As described above, according to the image processing method according to the present embodiment, it is possible to perform live view display in which a focus shift can be confirmed as an image shift, and even a multi-view camera adopting an electronic viewfinder method is at a loss. It is possible to perform an accurate focusing operation without any problem.

[Example 2]
In the first embodiment, in order to clarify the focus area range in the live view image, the brightness of the captured image I _c by the center imaging unit P _c is adjusted to make the prism type image in the focus area stand out. A view image was generated. Next, an embodiment in which an image having a high sharpness in the focus area and an image having a low sharpness in the other areas so that the focus area can be clearly recognized will be described as a second embodiment. Note that the difference from the first embodiment is only the content of the process in step 1004 of the flowchart of FIG. 10 and the other parts are common, so the difference will be described here.

In this embodiment, in step 1004 of the flowchart of FIG. 10, prepared image B _c according to the blur filter to the captured image I _c of the center image pickup unit P _c, to generate a live view image S by using this. When the live view image generated by the synthesis process is S, the pixel value of the pixels constituting the is expressed by the following equation.

15, by lowering the sharpness of the captured image I _c, which is an example of a live view image stand out the scope of the split prism image from the surroundings is located in the upper left.

  The image processing method according to the present embodiment also enables live view display in which a focus shift can be confirmed as an image shift, and accurate focusing without any hesitation even in a multi-view camera adopting an electronic viewfinder method. The operation can be performed. In particular, a photographer who is familiar with a silver halide camera can realize a viewfinder with little discomfort.

[Example 3]
In the first embodiment, the split direction of the split prism image is determined based on the user's designation in the camera setting menu. Next, an aspect in which the split direction of the split prism image is automatically determined will be described as a third embodiment. Note that the difference from the first embodiment is only the content of the process in step 402 of the flowchart of FIG. 4, and the other parts are common, so the difference will be described here.

In focusing by a split prism, it is desirable that an edge perpendicular to the split direction of the split prism exists. Therefore, in this embodiment, straight line detection is performed on the portion corresponding to the split prism image in the captured image I _c of the center imaging unit P _c , and the split direction of the split prism image is determined based on the detected straight line. Various known methods (for example, Hough transform) can be applied to the straight line detection method.

  FIG. 16 is a flowchart showing a flow of processing for automatically determining the split direction of the split prism image according to the present embodiment.

  First, the division direction determination unit 230 detects an edge in the prism area (step 1601).

  Then, the division direction determination unit 230 applies a Hough transform to the detected edge to detect a straight line (step 1602).

  Subsequently, the division direction determination unit 230 takes an inner product of the detected straight line and the division direction of the split prism image (step 1603).

  Finally, the division direction determination unit 230 determines the direction that minimizes the inner product as the division direction of the split prism image (step 1604).

  As described above, according to the present embodiment, the division direction of the split prism image is automatically determined based on the result of the straight line detection. Therefore, the user need not pay attention to the split direction of the split prism image, and the focus operation can always be performed in an optimum state.

Claims (10)

A multi-lens imaging device including a plurality of imaging units and an image display unit that performs live view display for displaying a shooting range in real time,
Focus area determining means for determining the position of the focus area in the live view display;
A prism type image is generated using a parallax image captured by a predetermined imaging unit among the plurality of imaging units, and the generated prism type image is arranged at a position determined by the focus area determining unit. And a live view image generating means for generating a live view image.   The imaging apparatus according to claim 1, wherein the live view image generation unit generates a live view image with reduced brightness for a portion other than the prism type image.   The imaging apparatus according to claim 1, wherein the live view image generation unit generates a live view image with a low sharpness for a portion other than the prism type image.   The imaging apparatus according to claim 1, wherein the prism type image is a split prism image that virtually represents a split prism.   The imaging apparatus according to claim 1, wherein the prism type image is a microprism image that virtually represents a microprism. A split direction determining means for determining a split direction of the split prism image;
The division direction and the plurality of imaging units are associated with each other,
5. The live view image generation unit generates the split prism image using the parallax image captured by the associated imaging unit based on the determined division direction. The imaging device described in 1.   The imaging apparatus according to claim 6, wherein the division direction determination unit determines a division direction of the prism type image based on designation from a user.   The division direction determining unit performs straight line detection on a portion corresponding to the split prism image of an image used for generating the prism type image, and determines a split direction of the split prism image based on a result of the straight line detection. The imaging apparatus according to claim 6. A method for focusing in a multi-lens imaging device including a plurality of imaging units and an image display unit that performs live view display for displaying a shooting range in real time,
A focus area determination step for determining a position of the focus area in the live view display;
A prism type image is generated using a parallax image captured by a predetermined imaging unit among the plurality of imaging units, and the generated prism type image is arranged at the position determined in the focus area determination step. A live view image generating step for generating a live view image.   The program for functioning a computer as an imaging device of any one of Claims 1-8.