JP6162971B2 - Image processing apparatus, image processing method, imaging apparatus, and control method thereof - Google Patents

Description

  The present invention relates to an imaging apparatus that determines and displays a refocusable range from refocusable image data.

  2. Description of the Related Art In recent years, an imaging apparatus (light field camera) that can acquire not only the intensity distribution of light but also information on the incident direction of light has been proposed in an imaging apparatus such as an electronic camera.

  For example, according to Non-Patent Document 1, a microlens array is arranged between a photographic lens and an image sensor, and one microlens is associated with a plurality of pixels of the image sensor, thereby allowing light that has passed through the microlens. Are obtained for each incident direction by a plurality of pixels. In addition to generating a normal captured image with respect to the pixel signal (light ray information) acquired in this way, a technique called “Light Field Photography” is applied to any image plane (refocus plane). A focused image can be reconstructed after shooting.

  As a case where the user needs to reconstruct an image by refocusing, there may be a case where the desired subject is not in focus when the image is confirmed after shooting. With conventional cameras, it was necessary to re-shoot when the desired subject was not in focus, but it was desirable without re-taking by reconstructing the refocus plane using the image signal taken with the light field camera. An image in which the subject is in focus is obtained.

  Although the technique described in Patent Document 1 is not a light field camera, it is possible to generate an image in which an arbitrary refocus plane is focused by combining a plurality of images taken at a plurality of focus positions. is there.

JP 2010-268052 A

Ren.Ng and 7 others, “Light Field Photography with a Hand-Held Plenoptic Camera”, Stanford Tech Report CTSR 2005-02

  However, the conventional light field camera has a problem that the user cannot confirm the refocusable range from the refocusable image data before shooting. A subject that is not within the refocusable range cannot generate a refocus image in which the subject is focused from refocusable image data. Therefore, even if the user tries to focus on a desired subject after shooting, there is a possibility that refocusing cannot be performed because the user has shot without placing the subject in the refocus range at the time of shooting.

  The present invention has been made in view of the above-described problems, and an object thereof is to reduce shooting failures by allowing a user to grasp a refocusable range before shooting.

An image processing apparatus according to the present invention includes an acquisition unit that acquires image data that is captured by an imaging unit and can be refocused after being captured, and characteristic information of the imaging unit, and a depth in the refocusable image data. Distance acquisition means for acquiring distance information indicating a distance in a direction; refocus range calculation means for calculating information on a refocusable distance range of the refocusable image data based on the characteristic information; and the distance Image processing means for performing image processing on an image corresponding to the refocusable image data based on the information and information on the refocusable distance range to generate a display image to be displayed on a display means; It is characterized by providing.
An image pickup apparatus according to the present invention includes an image pickup unit including an image pickup element in which a plurality of pixels are arranged for one microlens, and image data that can be refocused after being picked up by the image pickup unit. And acquisition means for acquiring the characteristic information of the imaging means, refocus image generation means for generating a refocus image at a target focal position using the refocusable image data, and the refocusing possible Distance acquisition means for acquiring distance information indicating the distance in the depth direction in correct image data, and refocus range calculation for calculating refocusable distance range information of the refocusable image data based on the characteristic information And refocusing image data based on the distance information and the refocusable distance range information. An image processing means for generating a display image to be displayed on the display means, which can distinguish an area where an image focused by refocusing can be obtained and an area where the image cannot be obtained. .

  According to the present invention, it is possible to reduce shooting failures by allowing the user to grasp the refocusable range before shooting.

1 is a block diagram illustrating a configuration of an imaging apparatus according to an embodiment of the present invention. 6 is a flowchart showing the operation of the imaging apparatus according to the first embodiment. The figure which shows a refocus image and its distance information. The figure explaining the maximum refocus amount. The figure explaining the refocus possible range and the distance information of a to-be-photographed object. The flowchart of display image generation. The figure which shows the image processing area | region for producing | generating a display image from refocus determination information. The figure which shows the display image production | generation of user designation mode. The flowchart of the display operation | movement of a display image.

  Hereinafter, an embodiment of the present invention will be described in detail with reference to the accompanying drawings. FIG. 1 is a block diagram showing a configuration of an imaging apparatus according to an embodiment of the present invention. In FIG. 1, 100 is an imaging device, 101 is an imaging unit, and 102 is a refocus image generation unit.

  Reference numeral 103 denotes a distance acquisition unit, 104 denotes a refocusable range calculation unit, 105 denotes a refocus determination unit, 106 denotes an image processing unit, 107 denotes a display unit, 108 denotes an operation unit, and 109 denotes a person detection unit.

  The imaging unit 101 includes an optical system, an imaging element, and a control system that controls the focus position and aperture, and generates and outputs refocusable image data. In addition, the imaging unit 101 outputs optical system characteristic information (focus position, imaging device specifications, etc.) when acquiring refocusable image data.

  The refocus image generation unit 102 generates a refocus image that is focused on the target focal position of the optical system from the refocusable image data output from the imaging unit 101 and the characteristic information of the optical system.

  The distance acquisition unit 103 divides refocusable image data into a plurality of areas, and outputs distance information from the imaging apparatus 100 to a subject in the area in each area. The refocusable range calculation unit 104 outputs refocusable range information at the time of image data acquisition that can generate a refocused image based on the characteristic information output by the imaging unit 101 (refocus range calculation). The refocus determination unit 105 outputs refocus determination information from the distance information and the refocus range information.

  The operation unit 108 includes buttons, a touch panel, a mode dial, and the like for inputting an instruction from the user to the imaging apparatus 100. The person detection unit 109 detects an area where a person exists from the refocus image and outputs person detection information.

  The image processing unit 106 performs image processing on the refocus image from the refocus determination information, and generates and outputs a display image. As an option, a display image can be generated and output by changing an image processing method based on instruction information or person detection information. The display unit 107 includes a liquid crystal screen and outputs a display image output from the image processing unit 106.

  Next, the operation of the imaging apparatus of the present embodiment will be described using the flowchart of FIG. S200 to S204 will be described with reference to FIG. FIG. 3 is a diagram showing refocus image data and distance information thereof. FIG. 3A shows a refocus image and a subject, and FIG. 3B shows a distance map. 300 is a refocus image, 301 is a subject a, 302 is a subject b, and 303 is a subject c.

  Reference numeral 304 is a distance information map, 305 is distance information of the subject a, 306 is distance information of the subject b, 307 is distance information of the subject c, and 308 is an area constituting the distance map.

  In S200, the imaging unit 101 acquires and outputs refocusable image data. Details of the optical system, the image sensor, and the control system for acquiring refocusable image data are described in Non-Patent Document 1, and thus description thereof is omitted here.

  In S201, the refocus image generation unit 102 generates and outputs a refocus image 300 at the focal position of the optical system from the refocusable image data and the optical system characteristic information. Since the generation of the refocus image is also described in Non-Patent Document 1, the description thereof is omitted here.

  In S202, the distance acquisition unit 103 divides the refocusable image data into regions 308 constituting a plurality of distance maps. The area 308 constituting the distance map may be set for each set of pixels imaged through one microlens 900 shown in FIG. 4 to be described later, but a set of pixels imaged through a plurality of microlenses 900. It is good as a body.

  In S203, the distance acquisition unit 103 calculates the distance to the subject for each region 308 constituting the distance map and creates a distance map. As a method for calculating the distance, an image with parallax may be created from refocusable image data, and may be obtained by using a phase difference for each region 308 constituting the distance map. There is no particular limitation here.

  In this embodiment, in order to make the explanation easy to understand, the distance from the imaging apparatus 100 to each subject is grouped as a short distance, a medium distance, and a long distance, but the distance and the number for grouping are not particularly limited.

  In S204, the distance acquisition unit 103 outputs the distance map as distance information. In step S <b> 205, the refocusable range calculation unit 104 calculates a refocusable range from the characteristic information output by the imaging unit 101 and outputs the refocusable range. The refocusable range can be calculated by adding the maximum refocus amount to the focused distance.

  The calculation of the maximum refocus amount will be described with reference to FIG. FIG. 4 is a diagram for explaining the maximum refocus amount. Reference numeral 900 denotes a microlens, and reference numeral 901 denotes an image sensor. A state in which light is incident on the image sensor 901 through the microlens 900 is shown.

  As shown in FIG. 9, assuming that the angle resolution Δθ, the number of angle divisions Nθ, and the pixel pitch Δx, the maximum refocus amount dmax is calculated by the following equation (1).

dmax = Nθ · Δx / tan (Δθ) (1)
That is, the subject existing within the range of the distance dmax along the optical axis around the focused distance when refocusable image data is taken can be refocused.

  In S206, the refocus determination unit 105 creates refocus determination information from the distance information and the refocus range information and outputs it.

  A method of creating refocus determination information will be described with reference to FIGS. FIG. 5 is a diagram for explaining refocusable range and subject distance information. 600 indicates a refocusable range, 601 indicates a distance da between the subject a301 and the refocusable range, 602 indicates a distance db between the subject b302 and the refocusable range, and 603 indicates a boundary between the subject c303 and the refocusable range. Distance dc.

  Originally, the determination is made for all the areas 308 constituting the distance map, but for the sake of easy understanding, the description will be limited to the subject a301, the subject b302, and the subject c303. The refocus determination information is flag information of 1 when the area 308 constituting the distance map exists in the refocusable range, and 0 when it does not exist (outside the refocusable range). As an option, a difference distance (degree of distance) from the boundary of the refocusable range may be added. In the present embodiment, there will be described two cases where the refocus determination information is only flag information or flag information and difference distance.

  Since the area 308 constituting the distance map in the distance information 305 of the subject a does not exist in the refocusable range 600, the flag information is 0, and if the difference distance is also set to be added, the difference distance da601 is added to the determination information.

  Since the area 308 constituting the distance map in the distance information 306 of the subject b exists in the refocusable range 600, the flag information is 1, and if the difference distance is also set to be added, the difference distance db602 is added to the determination information. The difference distance is selected as the closer of the two boundaries of the refocusable range, but may be reversed.

  Since the area 308 constituting the distance map in the distance information 307 of the subject c does not exist in the refocusable range 600, the flag information is 0, and if the difference distance is also set to be added, the difference distance dc603 is added to the determination information.

  In S207, the image processing unit 106 generates a display image. A display image generation method will be described with reference to a display image generation flowchart of FIG.

  In S400, it is confirmed whether the operation mode of the imaging apparatus is the portrait mode. The mode is selected from the operation unit 108. The portrait mode will be described in S405. When it is in the portrait mode, the process proceeds to S405, and when it is not the portrait mode, the process proceeds to S401.

  In step S401, it is confirmed whether the user designation mode is set. The user designation mode will be described in S403. When it is in the user designation mode, the process proceeds to S403, and when it is not the user designation mode, the process proceeds to S402.

  In S402, the image processing unit 106 generates a display image from the refocus determination information. The generation of the display image will be described with reference to FIG. FIG. 7 is a diagram showing an image processing area for generating a display image from the refocus determination information.

  Reference numeral 500 denotes an image processing area applied to the distance information 305 of the subject a, reference numeral 501 denotes an image processing area applied to the distance information 306 of the subject b, and reference numeral 502 denotes an image processing area applied to the distance information 307 of the subject c. These are called a subject a image processing area 500, a subject b image processing area 501, and a subject c image processing area 502, respectively.

  Image processing will be described. When the refocus determination information is only flag information, the process is divided into a location where the flag information is 1 and a location where the flag information is 0. Where the flag information is 1, the subject b image processing area 501 and where the flag information is 0 are the subject a image processing area 500 and the subject c image processing area 502.

  The flag information is painted with a color having only one location, or is painted with a transparent color. Alternatively, only portions where the flag information is 0 may be black and white (first image processing). Alternatively, image processing such as blurring may be added to at least one of the flag information so that the location where the flag information is 1 and 0 can be seen visually. The above image processing may be reversed at the 0 and 1 locations of the flag information.

  When the refocus determination information includes not only the flag information but also a difference distance, image processing strength may be added according to the difference distance. For example, the difference distance is different between the subject a image processing area 500 and the subject c image processing area 502. For this reason, when a color is added to a location where the flag information is 0, the difference distance da601 is smaller than the difference distance dc603, and therefore the color density and brightness may be smaller than the subject c image processing area 502. As long as the image processing is changed according to the difference distance, the specific contents are not limited.

  In step S <b> 403, an area designated by the user using the operation unit 108 is output. Here, the user designation mode will be described with reference to FIG. FIG. 8 is a diagram showing display image generation in the user designation mode. FIG. 8A shows a refocus image 300 and a selection area 700. FIG. 8B is a diagram showing a region selected in the refocus image 300. Reference numeral 701 denotes a selected area. FIG. 8C shows an example in which a display image is generated from the determination information of the selected area. Reference numeral 702 denotes a comment.

  The user designation mode is a mode in which a user selects a subject on the screen using the operation unit 108 and generates a display image that can confirm whether or not the selected subject can be refocused. The user uses the operation unit 108 to select a selection area 700 including a subject to be confirmed whether refocusing is possible. Here, for example, it is assumed that the user has selected the subject a301.

  In S404, based on the refocus determination information at the same position as the selected area 701, a display image is generated so that it can be determined whether or not the selected area 701 can be refocused. The display image can be generated from the refocus determination information in the same manner as in S402, but only the selected area 701 is subjected to image processing (second image processing). Further, the display image may be generated by combining the comment 702 in the refocus image 300 at an arbitrary position from the refocus determination information. At this time, if there is a difference distance in the determination information in the comment 702, the distance to the refocus range may be displayed to the user by embedding the difference distance in characters (fourth image processing).

  In step S405, the person detection unit 109 detects an area where a person exists from the refocus image 300 and outputs person detection information. In S406, the display image of the area of the person detection information is generated from the refocus determination information so that it can be determined whether or not the area can be refocused. The generation of the display image from the refocus determination information may be the same as that in S402, but the area for image processing is limited to the person detection area.

  In S407, a display image of an area other than the person detection area is generated. In areas other than the person detection area, the brightness is uniformly reduced or black and white by image processing (third image processing). If the imaging apparatus 100 is configured without the operation unit 108 and the person detection unit 109, S400 and S401 are skipped and the process starts from S402. If the imaging apparatus 100 does not include the person detection unit 109, S400 is skipped and the process starts from S401.

  In S208, the display unit 107 displays a display image. The display method will be described with reference to FIG. FIG. 9 is a flowchart of the display image display operation.

  In S800, the image processing unit 106 confirms whether the setting is to be displayed when the display button of the operation unit 108 is pressed. If it is the setting to display, the process proceeds to S802. If it is not set to display, the process proceeds to S801.

  In step S <b> 801, the image processing unit 106 outputs the generated display image to the display unit 107. The display unit 107 displays the input display image until the next input is received. The image processing unit 106 may output the refocused image 300 as a display image without performing image processing after a predetermined time.

  In step S802, it is confirmed whether the display button is pressed. If the button has been pressed, the process proceeds to S801. If the button has not been pressed, the process proceeds to S803. Here, input for a predetermined time may be awaited.

  In step S803, the image processing unit 106 outputs the refocused image as a display image without performing image processing on the refocused image 300. The display unit 107 displays the input display image until the next input is received.

  In step S209, it is confirmed whether shooting has been performed. If it is not photographed, the process returns to S200. If the image has been shot, the flowchart ends.

  As described above, the refocusable range of image data that can be refocused by the user before shooting can be confirmed on the display unit 107. Further, the refocusable range of only a limited subject such as a specific subject or a person can be displayed on the display unit 107.

  As mentioned above, although preferable embodiment of this invention was described, this invention is not limited to these embodiment, A various deformation | transformation and change are possible within the range of the summary.

Claims (22)

An acquisition means for acquiring image data captured by the imaging means and refocusable after being imaged, and characteristic information of the imaging means;
Distance acquisition means for acquiring distance information indicating a distance in a depth direction in the refocusable image data;
Refocus range calculation means for calculating information on a refocusable distance range of the refocusable image data based on the characteristic information;
Image processing means for performing image processing on an image corresponding to the refocusable image data based on the distance information and the refocusable distance range information to generate a display image to be displayed on the display means; ,
An image processing apparatus comprising:   The image processing apparatus according to claim 1, wherein the characteristic information is information related to a focus position of an imaging optical system in which the refocusable image data is captured or a specification of an imaging element.   The image processing means makes it possible to distinguish between an area where an image focused by refocusing can be obtained and an area where it cannot be obtained in an image corresponding to image data that can be refocused by the image processing. Or the image processing apparatus of 2.   4. The image processing according to claim 1, wherein the image processing is processing for changing a color of an area where an image focused by refocusing or an area other than the area is obtained. The image processing apparatus described.   The image according to any one of claims 1 to 4, further comprising refocus image generation means for generating a refocus image at a target focal position using the refocusable image data. Processing equipment. An operation unit that receives an instruction from a user that specifies an image area of the refocusable image data;
Based on the instruction, the image processing unit generates a display image that can distinguish whether or not an image focused by refocusing is obtained in an image corresponding to image data in which the designated region can be refocused. The image processing apparatus according to claim 1, wherein the image processing apparatus is an image processing apparatus. Subject detection means for detecting a specific subject from an image corresponding to the refocusable image data;
The image processing means is a display image capable of distinguishing whether or not an image focused by refocusing is obtained in an image corresponding to refocusable image data in a region where a specific subject is detected by the subject detection means. The image processing apparatus according to claim 1, wherein the image processing apparatus generates an image. The image processing means, for each of a plurality of areas of the refocusable image data, information indicating whether the distance in the depth direction of the area is within the refocusable distance range or outside the refocusable distance range; and The information processing apparatus includes information indicating a degree outside the range when the distance is outside the refocusable distance range, and the image processing unit changes the image processing based on the information outside the range. 8. The image processing device according to any one of items 7. The image processing means generates the display image so that a subject located outside the refocusable distance range can recognize the magnitude of the distance outside the refocusable distance range. The image processing apparatus according to claim 4.   Whether the image processing means can obtain an image focused on each area by comparing the distance information of each area of the refocusable image data with the information on the refocusable distance range. The image processing apparatus according to claim 1, wherein a display image reflecting the determination result is generated. The image processing apparatus according to any one of claims 1 to 10,
The imaging means;
An imaging apparatus comprising:   The imaging apparatus according to claim 11, wherein the imaging unit includes an imaging element having a plurality of microlenses and a plurality of pixels arranged with respect to one microlens. An image pickup means including an image pickup element in which a plurality of pixels are arranged with respect to one microlens;
Acquisition means for acquiring image data captured by the imaging means and refocusable after being imaged, and characteristic information of the imaging means;
Refocus image generation means for generating a refocus image at a target focal position using the refocusable image data;
Distance acquisition means for acquiring distance information indicating a distance in a depth direction in the refocusable image data;
Refocus range calculation means for calculating information on a refocusable distance range of the refocusable image data based on the characteristic information;
Based on the distance information and the information on the refocusable distance range, an image corresponding to the refocusable image data is distinguished from an area where an image focused by refocus can be obtained and an area where the image cannot be obtained. Image processing means for generating a display image to be displayed on the display means,
An imaging apparatus comprising: An acquisition step of acquiring image data captured by the imaging unit and refocusable after being captured, and characteristic information of the imaging unit;
A distance acquisition step of acquiring distance information indicating a distance in a depth direction in the refocusable image data;
A refocus range calculation step of calculating information on a refocusable distance range of the refocusable image data based on the characteristic information;
An image processing step of performing image processing on an image corresponding to the refocusable image data based on the distance information and the refocusable distance range information to generate a display image to be displayed on a display unit; ,
An image processing method comprising: A control method for controlling an image pickup apparatus having an image pickup unit including an image pickup element in which a plurality of pixels are arranged for one microlens,
An acquisition step of acquiring image data captured by the imaging unit and refocusable after being captured, and characteristic information of the imaging unit;
Using the refocusable image data, a refocus image generation step of generating a refocus image at a target focal position;
A distance acquisition step of acquiring distance information indicating a distance in a depth direction in the refocusable image data;
A refocus range calculation step of calculating information on a refocusable distance range of the refocusable image data based on the characteristic information;
Based on the distance information and the information on the refocusable distance range, an image corresponding to the refocusable image data is distinguished from an area where an image focused by refocus can be obtained and an area where the image cannot be obtained. An image processing step of generating a display image to be displayed on the display means,
A method for controlling an imaging apparatus, comprising: An acquisition means for acquiring image data captured by the imaging means and refocusable after being imaged, and characteristic information of the imaging means;
Information acquisition means for acquiring position information indicating positions in a depth direction of a plurality of subjects in the refocusable image data;
Refocus range calculation means for calculating refocusable range information in the depth direction of the refocusable image data based on the characteristic information;
Image processing for generating a display image to be displayed on a display unit by performing image processing on an image corresponding to the refocusable image data based on the position information and the range information in the refocusable depth direction. Means,
An image processing apparatus comprising: The image processing apparatus according to claim 16, wherein the characteristic information is information related to a focus position of an imaging optical system in which the refocusable image data is captured or a specification of an imaging element. The image processing means makes it possible to distinguish between an area where an image focused by refocusing and an area where it cannot be obtained in an image corresponding to image data that can be refocused by the image processing. Or the image processing device according to 17. 19. The image processing according to claim 16, wherein the image processing is processing for changing a color of an area where an image focused by refocusing is obtained or any other area. The image processing apparatus described. The image according to any one of claims 16 to 19, further comprising refocus image generation means for generating a refocus image at a target focal position using the refocusable image data. Processing equipment. 21. The image processing apparatus according to claim 16, wherein the imaging unit includes an imaging device in which a plurality of pixels are arranged with respect to one microlens. An acquisition step of acquiring image data captured by the imaging unit and refocusable after being captured, and characteristic information of the imaging unit;
An information acquisition step of acquiring position information indicating positions in a depth direction of a plurality of subjects in the refocusable image data;
A refocus range calculation step of calculating information on a refocusable depth direction range of the refocusable image data based on the characteristic information;
Image processing for generating a display image to be displayed on a display unit by performing image processing on an image corresponding to the refocusable image data based on the position information and the range information in the refocusable depth direction. Process,
An image processing method comprising:

Images