JP2015175905A - Image processor, image processing method, program, and imaging device - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an imaging device capable of performing display with less load due to a change in a focal position for a photographer in the case of photographing a moving image.SOLUTION: An image processor for processing refocus processing-enabling imaged data obtained by imaging the optical image of a subject formed by a photographic optical system including a focus lens by imaging means includes: imaged data acquisition means for acquiring refocus processing-enabling imaged data; lens information acquisition means for acquiring the information of the position of a focus lens; focal surface determination means for determining the position of the focal surface of the display image of the acquired imaged data on the basis of the acquired information of the position of the focus lens; refocus processing means for generating the image data of a refocus image on the focal surface at the determined position from the imaged data; and display data generation means for generating the display data of the acquired imaged data on the basis of the generated image data of the refocus image.

Description

  The present invention relates to an image processing apparatus, and more particularly, to an image processing apparatus, an image processing method and a program for generating display data of the image data using recombination of image data, and an imaging apparatus to which the image processing apparatus is applied. .

  In recent digital single-lens reflex cameras, an imaging apparatus having a moving image shooting function in addition to a still image shooting function has appeared. These digital single-lens reflex cameras include a product having a function that enables still image shooting during moving images. Also, digital cameras that can shoot movies generally have a liquid crystal display on the back, and the photographer can shoot while observing the subject to be photographed via the liquid crystal display on the back. it can.

  In addition, an autofocus (hereinafter referred to as AF) technique is known that automatically performs a focus adjustment operation for focusing on a predetermined area when shooting moving images and still images. The autofocus technology is roughly classified into a phase difference method and a contrast method.

  The phase difference AF is a method in which a light beam incident from a lens is divided, a phase shift between images generated by the respective light beams is detected, and focus adjustment is performed according to the detected phase shift. . Japanese Patent Application Laid-Open No. 2004-228561 discloses a technique for realizing phase difference AF by performing pupil division on an imaging surface by configuring a plurality of photoelectric conversion units per pixel.

  The contrast method is a method in which an image is acquired while moving the lens, and the lens is driven with a lens position having a peak contrast amplitude of the image as a focus position.

  By the way, a technique capable of obtaining (reconstructing) an image in focus at different distances of the object scene from image data obtained by one shooting is known. In Patent Document 2, a light beam that has passed through different areas of the exit pupil of the imaging optical system is picked up by an image sensor, and image data obtained from the light beams that have passed through the different areas are combined, so that an arbitrary one-time shooting is possible. A technique for generating image data of an image on the image plane is disclosed.

Japanese Patent No. 4027113 JP 2007-4471 A

  However, when AF is performed in an imaging apparatus capable of shooting a moving image, there are the following problems.

  When the phase difference AF is applied to moving image shooting, there is a problem that the quality of the moving image is deteriorated by continuously performing high-speed lens driving and stopping operations.

  On the other hand, when the contrast method is applied to moving image shooting, there is a problem that the quality of the displayed moving image is deteriorated because the operation is performed once past the focus position of the subject in order to determine the contrast peak.

  In addition, in a conventional interchangeable lens camera such as a single-lens reflex camera, it is possible to attach a lens designed mainly for still image shooting, and a lens designed for such a still image is mainly used. The driving speed and driving interval are not necessarily suitable for moving images.

  Therefore, an object of the present invention is to provide an image processing apparatus, an image processing method, a program, and an imaging apparatus capable of simultaneously realizing display and recording with moving image quality and lens control suitable for automatic focus adjustment. It is to be.

  According to the present invention, an image processing apparatus for processing imaging data that can be refocused and obtained by imaging an optical image of a subject formed by an imaging optical system including a focus lens by an imaging unit is provided. Based on the acquired information on the position of the focus lens, on the basis of the acquired information on the position of the focus lens, on the basis of the acquired information on the position of the focus lens, the imaging data acquisition means that acquires the imaging data that can be processed A focal plane determination unit that determines a position of the focal plane, a refocus processing unit that generates image data of the refocus image on the focal plane at the determined position from the imaging data, and image data of the generated refocus image And display data generation means for generating display data of the acquired imaging data.

  According to the present invention, it is possible to display and reproduce high-quality moving images by separately setting the focal plane used for display and recording with respect to the focal plane related to the focus lens position.

1 is a block diagram illustrating a configuration of an imaging apparatus to which an image processing apparatus according to an embodiment of the present invention is applied. The figure for demonstrating the structure of the image pick-up element used with the imaging device of FIG. The figure for demonstrating the relationship between the division pixel and incident light in the image sensor of FIG. The figure which shows the flowchart of the video recording operation | movement at the time of applying the image processing apparatus which concerns on the 1st Example of this invention to the imaging device of FIG. The figure which shows the detailed flowchart of the video recording operation | movement of FIG. The figure which shows the flowchart of the virtual focal plane determination operation | movement which concerns on 1st Example of this invention. The figure for demonstrating the determination structure of the virtual focal plane for a display which concerns on 1st Example of this invention. Diagram for explaining refocus processing The figure which shows the flowchart of the virtual focal plane determination operation | movement concerning 2nd Example of this invention. The figure for demonstrating determination of the virtual focal plane determination operation | movement which concerns on 2nd Example of this invention. The figure which shows the flowchart of the virtual focal plane determination operation | movement concerning the modification of the 2nd Example of this invention.

  Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings.

[Example 1]
Hereinafter, an imaging apparatus to which the image processing apparatus according to the first embodiment of the present invention is applied will be described with reference to FIGS. Examples of the imaging apparatus include a digital camera and a video camera. The image processing apparatus according to the present embodiment is also an information processing apparatus such as a multi-lens camera or a PC that processes output of a camera capable of acquiring light field data. Needless to say, it is applicable.

FIG. 1 is a block diagram illustrating a configuration of an imaging apparatus 100 to which the image processing apparatus according to the present embodiment is applied.
In FIG. 1, reference numeral 101 denotes a photographing optical system including a focus lens, which guides light from a subject to an image sensor 102 through a plurality of lens groups and a diaphragm, and forms an optical image on the image sensor 102. The focus lens included in the photographic optical system 101 is driven based on a drive command from a lens drive control unit (not shown) or a rotation operation of the focus ring, and thereby the focus position can be adjusted.

  A plurality of microlenses are arranged in a grid pattern on the image sensor 102, and a plurality of photoelectric conversion units are provided below each microlens. Each microlens divides the light beam passing through the exit pupil of the photographing optical system 101 into pupils, and the plurality of photoelectric conversion units are designed to receive the light beams divided by the microlens. The detailed configuration of the image sensor 102 will be described later.

  Reference numeral 103 denotes an A / D converter that converts an analog signal output from the image sensor into a digital signal.

  The image processing unit 104 includes a circuit and a program group having an image processing function including a focus detection unit 105, a display virtual focal plane determination unit 106, and a refocus image generation unit 107, which will be described later. The image processing unit 104 performs predetermined image processing on the imaging data output from the A / D conversion unit 103 based on a control command from the camera system control unit 108, and generates image data for recording and display. .

  The focus detection unit 105 performs a correlation calculation based on the imaging data obtained via the A / D conversion unit 103, and calculates an evaluation value used for lens position adjustment. The evaluation value calculated by the focus detection unit 105 is converted into a lens driving command via the camera system control unit 108 and used for driving the photographing optical system 101. Detailed operation of the focus detection unit 105 will be described later.

  The display virtual focal plane determining unit 106 determines the position of the focal plane when forming display image data. A detailed description of the display virtual focal plane determination unit 106 will be described later.

  The refocus image generation unit 107 performs an image on a focal plane (virtual focal plane) different from the actual focal plane (hereinafter, referred to as a refocused image) by combining image data output from the A / D conversion unit 103. Is a circuit that generates Detailed operation of the refocus image generation unit 107 will be described later.

  In addition to controlling the operation of the entire camera, the camera system control unit 108 acquires data obtained by image processing, lens position information of the photographing optical system 101, and the like, and also functions as a data intermediary between the blocks. The control program of the camera system control unit and necessary data are stored in advance in a memory (not shown).

  The operation unit 109 is an operation member such as various switches and dials provided in the imaging apparatus 100, and the photographer can control setting of shooting parameters, shooting operation, and the like using the operation unit.

  The display unit 110 is a liquid crystal display such as an LCD, and displays an image at the time of shooting, image data stored in the storage unit 111, various setting screens, and the like.

  The storage unit 111 is a recording medium such as an SD card or CompactFlash, and records image data generated by the image processing unit 104.

  Next, the configuration of the image sensor 102 included in the imaging apparatus 100 will be described with reference to FIGS. 2 and 3.

  FIG. 2A is an enlarged view of a part of the image sensor 102, and shows a correspondence relationship between the microlens array and the pixels. A circle in the figure indicates the microlens 201. A rectangular area corresponding to the microlens 201 is a pixel. A microlens 201 and pixels are two-dimensionally arranged on the image sensor 102.

Next, the configuration of the pixel will be described.
FIG. 2B is a diagram in which one pixel on the image sensor 102 is observed from above. FIG. 2C is a cross-sectional view of the pixel along the line P-P ′ shown in FIG. In FIG. 2, the same parts are denoted by the same reference numerals.
In FIG. 2, the micro lens 201 is disposed in the vicinity of the imaging plane of the photographing optical system 101 and condenses the light beam output from the photographing optical system. Reference numerals 202 and 203 denote photoelectric conversion units. Hereinafter, the photoelectric conversion units as shown in 202 and 203 are referred to as divided pixels.

  As shown in FIG. 3, the divided pixel 202 and the divided pixel 203 respectively receive light beams that have passed through different regions 302 and 303 on the exit pupil 301 of the photographing optical system 101. In this embodiment, the photoelectric conversion unit is divided into two to form divided pixels. However, the scope of application of the present invention is not limited to this, and the light beams that have passed through different regions on the exit pupil are respectively shown. Any structure that can receive light independently is acceptable.

  Next, the operation of the camera system control unit 108 during moving image shooting will be described with reference to FIGS.

  FIG. 4 is a flowchart of the moving image shooting control operation in the imaging apparatus 100.

  When the moving image shooting start operation is executed by the operation unit 109, the imaging apparatus 100 executes the following moving image recording flowchart. This operation is performed by the camera system control unit 108 executing a control program stored in advance in a memory (not shown).

  In step S <b> 401, the camera system control unit 108 determines whether the operation unit 109 has stopped moving image shooting. If it is determined in step S401 that the moving image shooting stop operation has been performed, the camera system control unit 108 ends the moving image shooting operation. On the other hand, if the stop operation has not been performed in step S401, the process proceeds to step S402.

  In acquisition of imaging data in step S <b> 402, the camera system control unit 108 controls driving of the imaging element 102 and the A / D conversion unit 103 to acquire imaging data of the imaging target region.

  Next, in steps S403 to S405, the camera system control unit 108 controls the image processing unit 104 to execute the processing of each step by parallel operation.

  Details of the recording process in step S403, the display process in step S404, and the focus adjustment process in step S405 will be described later. Note that the processes in steps S403 to S405 are repeated until a moving image shooting stop operation is performed.

  Next, detailed processing contents of steps S403 to S405 will be described.

  FIGS. 5A, 5B, and 5C are flowcharts of the processes from S403 to S405 in FIG. 4, respectively.

First, the recording process in step S403 will be described with reference to FIG.
When the recording process starts, the camera system control unit 108 controls the image processing unit, converts the captured data acquired in step S402 into data of a predetermined format, compresses the data, and generates recording data.

  In step S502, the camera system control unit 108 records the recording data generated in step S501 in the storage unit 111 inserted in a storage medium slot (not shown) of the imaging apparatus 100, and ends the recording process. .

  Next, the display data generation operation in step S404 will be described with reference to FIG. 5B, FIG. 6, and FIG.

  When the display process starts, first, a display virtual focal plane determination process by the display virtual focal plane determination unit 106 is performed in step S511.

  FIG. 6 shows a flowchart of the detailed operation of the display virtual focal plane determination process. This operation is performed by the display virtual focal plane determination unit 106 under the control of the camera system control unit 108.

  When the display virtual focal plane determination process is started, the camera system control unit acquires the current lens position information of the photographing optical system 101 in synchronization with the display timing by acquiring lens information in step S601. The acquired focus lens position is associated with a focal plane position fp (hereinafter simply referred to as a focal plane fp) based on a correspondence table registered in advance in a memory (not shown) in the imaging apparatus 100.

  FIG. 7A is a diagram illustrating an example of the displacement of the focal plane fp at a certain time during moving image shooting. The vertical axis represents the position of the focal plane, and the horizontal axis represents time. The dotted line on the graph is the display timing uniquely determined according to the display frame rate. In the figure, a scene change occurs between the frame Fr (x) and the frame Fr (x + 1), the focus lens causes a steep operation, and the focal plane changes steeply.

In step S602, based on the position of the focal plane fp, the position of the focal plane most suitable for display (referred to as an ideal focal plane) is variably determined with respect to the focal plane fp. The position ip of the ideal focal plane (hereinafter simply referred to as the ideal focal plane ip) is determined by the following Equation 1.
ip (x) = (fp (x−1) + fp (x)) ÷ 2 Equation 1

  In the above formula, x represents display timing (frame). That is, the value obtained by the addition average of the focal plane fp (x) at the current display timing and the focal plane fp (x-1) at the previous display timing (the average value of the focal plane variation) is the ideal focal plane. It is calculated as ip (x).

FIG. 7B is a diagram illustrating a change in the ideal focal plane ip obtained by applying Expression 1 at each display timing in FIG. It can be seen that the change rate of the ideal focal plane ip obtained by applying Equation 1 is lower than the actual change rate of the focal plane position (the gradient of the straight line in the figure). Here, for the sake of explanation, an example has been shown in which the ideal focal plane is obtained by the addition average of the actual focal plane fp and fp one frame before, but the calculation method of the ideal focal plane is not limited to this. Absent. For example, the ideal focal plane may be determined by using a moving average process in which the number of taps and the coefficient are changed as shown in Equation 2 below.
ip (x) = (fp (x−2) + (fp (x−1) × 2) + fp (x)) ÷ 4 Equation 2
Further, as shown in the following Expression 3, the ideal focal plane may be determined by obtaining the root mean square of focal plane positions in a plurality of different frames.
ip (x) = √ ((fp (x−1) ^ 2 + fp (x) ^ 2) ÷ 2) Equation 3

  Further, the focal plane change amount L allowable for display may be defined in advance, and the ideal focal plane may be determined so as to limit the current focal plane fp change amount within the range of the change amount L. . Further, in this embodiment, the example in which the lens position information is used as a means for obtaining the position of the focal plane has been described. However, the scope of the present invention is not limited to this, and other optical information may be used.

  In step S603, a refocusable range is calculated.

  Although the details of the principle of refocus will be described later, the refocus processing is realized by synthesizing image data obtained by the divided pixel groups. Therefore, it can be said that the area where the imaging data of the focused image can be obtained by the refocus processing is the range where the focused image can be obtained by the divided pixels. The range that appears to be in focus by the divided pixels is a range that is within the depth of field. The divided pixels on the image sensor 102 in this embodiment are designed to divide the exit pupil of the photographing optical system 101 into two. Accordingly, the depth of field that can be acquired by the divided pixels is the depth of field in a state in which the depth is reduced by one step from the F value of the photographing optical system 101. In this embodiment, the relationship between the focus lens position and the F value, the focal length, the depth of field, and the focal depth is previously stored as a table in a memory (not shown) in the imaging apparatus 100, and the focus lens position, the F value, The depth of field and the depth of focus can be determined uniquely from the focal length.

FIG. 7C is a diagram conceptually showing the refocus range corresponding to FIG.
The refocus range Rf_Range has a range obtained by combining a depth Rf_far from the image plane to the subject side and a depth Rf_near further rearward from the image plane with respect to the actual focal plane.

Next, in steps S604 to S605, a comparison is made to determine whether or not the calculated ideal focal plane ip is within the refocusable range Rf_Range, and the display virtual focal plane fp_out is determined as follows according to the result.
When ip ≧ Rf_far fp_out = Rf_far
When ip ≦ Rf_near fp_out = Rf_near
When Rf_near <ip <Rf_far fp_out = ip

  This decision rule determines the ideal focal plane as the display virtual focal plane when the ideal focal plane is within the refocus range, and if not, the limit of the refocusable range is determined as the display virtual This rule is determined as the focal plane.

  FIG. 7D is a diagram showing the relationship between the refocusable range Rf_Range and the ideal focal plane ip in a plurality of frames shown in FIG. In the drawing, the ideal focal plane takes a value smaller than Rf_near in the frame Fr (x + 1) and is out of the refocus range.

  FIG. 7E is a diagram showing the display virtual focal plane fp_out determined in accordance with the comparison determination result in step S604 and step S605. The virtual focal plane for display of the frame Fr (x + 1) is at the position of Rf_near (x + 1). In this way, the display virtual focal plane fp_out is determined by steps S601 to S605.

  Next, in step S512 of FIG. 5B, the refocus image generation unit 107 performs a refocus image generation process corresponding to the display virtual focal plane fp_out. The refocus image generation process is realized by the refocus image generation unit 107.

  Here, the principle of refocus will be described.

FIG. 8 is a diagram for explaining the operation of the refocus processing.
8A shows the relationship between incident light and a focal plane in a certain area of the image sensor 102 when imaging. FIG. 8A shows incident light when the focal plane is on the array surface of the microlens 201 for each microlens by a solid line, a broken line, a dotted line, and the like. A focused image is formed on the imaging surface of the imaging device during imaging. The light rays incident on the two divided pixels corresponding to each microlens at that time are indicated by the same type of line. The light beam that has passed through the exit pupil regions 302 and 303 described in FIG. 3 is divided into pupils by a microlens. The luminous flux that has passed through the exit pupil region 302 is received by the divided pixels 802, 804, 806, 808, and 810, and the luminous flux that has passed through the exit pupil region 303 is received by the divided pixels 801, 803, 805, 807, and 809, respectively.

  FIG. 8B is a diagram schematically showing the light flux received by each of the divided pixels in FIG. By adding the divided pixels 801 and 802, 803 and 804, 805 and 806, 807 and 808, 809 and 810 under the same lens, respectively, an image in which the same type of line is formed can be obtained.

  FIG. 8C is a diagram illustrating a correspondence between incident light and divided pixels on the virtual focal plane 1 that generates a refocus image. Each of the divided pixels 801 to 810 has the light ray information shown in FIG.

  FIG. 8D shows an example in which a refocus image on the virtual focal plane 1 is generated by shifting and adding image data generated from the signal received in FIG. Accordingly, by obtaining the imaging data of different line types by shift addition of the imaging data obtained in FIG. 8A, it is equivalent to the image data that can be acquired on the virtual focal plane 1 shown in FIG. Can be handled as data.

  FIGS. 8E and 8F are examples in which addition is performed by shifting in the opposite direction to FIGS. 8C and 8D. As imaging data equivalent to imaging data obtained on the virtual focal plane 2, FIGS. Can be handled.

  By using the shift addition configuration in this way, it is possible to generate a refocus image focused on an arbitrary subject area. Using this refocus processing, a refocus image corresponding to the display virtual focal plane ip determined by the display virtual focal plane determining means of this embodiment is generated.

  Here, the refocus processing has been described using an example in which there are two divided pixels for the microlens. However, the scope of the present invention is not limited to this, and refocusing corresponding to the virtual focal plane for display is performed by performing recombination processing according to the pixel structure such as the number of divided pixels and the configuration of the optical system. It suffices if an image can be obtained.

  Further, for the sake of explanation, an example of the refocus processing by a simple shift addition operation has been shown, but the present invention is not limited to this, and a refocus image at each virtual focal plane may be obtained using another method. . For example, as described in Patent Document 2, processing by weighted addition may be performed.

  In step S513, the image processing unit 104 performs predetermined image processing such as resizing on the refocused image generated based on the display virtual focal plane to generate a display image (display data). .

  Next, in step S514, the camera system control unit 108 outputs the display refocus image generated in step S513 to the display unit 110, and ends the display process.

  Next, the focus adjustment operation in step S405 in FIG. 4 will be described with reference to FIG. This operation is performed by the focus detection unit 105 under the control of the camera system control unit 108.

  When the focus adjustment process is started, a pupil divided image is generated in step S521. As described in the description of the image sensor 102, each of the divided pixels under one microlens receives light beams that have passed through different regions on the exit pupil. Therefore, a pair of pupil-divided images can be obtained by collecting the outputs of the divided pixels having the same positional relationship with respect to each microlens.

  Next, in step S522, a correlation calculation is performed on the pupil division image obtained in step S521. In the correlation calculation, an evaluation value is acquired by integrating the absolute value of the difference value for each overlapping pixel while shifting the pair of pupil-divided images in the horizontal direction around a preset autofocus region. Although an example in which the absolute value of the difference between the pixels is used as the evaluation value has been described here, the evaluation value may be acquired by a sum of squares of the difference value or other methods.

  In step S523, the imaging optical system 101 is driven based on the image shift amount to the position where the evaluation value of the correlation calculation is the lowest, that is, the position where the degree of coincidence of the pupil divided images is the highest, and the focus adjustment process is performed. Complete.

  As described above, according to the present embodiment, it is possible to reduce the focal plane change rate in the display with respect to the focal plane change rate based on the driving of the focus lens suitable for focus adjustment. As a result, it is possible to display a moving image with less burden on the photographer in recognizing the display image while performing the same lens driving as before.

[Example 2]
Next, a second embodiment of the present invention will be described with reference to FIGS.

  In the first embodiment of the present invention, when the focal plane changes particularly greatly by switching the subject distance, the display virtual focal plane within the refocusable range also changes abruptly, and the subject In some cases, the display may be out of focus. For example, this is the case of the display virtual focal plane fp_out (x + 1) with respect to the change of the focal plane from the frame Fr (x) to the frame Fr (x + 1) in FIG. 7E described in the first embodiment. In such a case, in the first embodiment, it is not always possible to display a moving image with less burden on the photographer.

  Therefore, in this embodiment, when the focal plane changes greatly due to the switching of the subject distance, the focal plane for display is set to the focal plane corresponding to the position of the focus lens so that the subject is in focus. The purpose is to display.

  However, when a large change in the focal plane due to switching of the subject distance continues, if the focal plane for display is controlled to a focal plane corresponding to the focus lens position, an unfavorable display is possible for the photographer. .

  Therefore, when a large change in the focal plane due to the switching of the subject distance continues, a moving image with less burden on the photographer is displayed by setting the display focal plane as the virtual focal plane.

  Since the configuration of the image pickup apparatus in the second embodiment is the same as that of the image pickup apparatus 100 described in the first embodiment, description thereof is omitted here. The recording process, display process, and focus adjustment process at the time of moving image shooting shown in FIGS. 5A, 5B, and 5C are the display virtual focal plane determination process shown in step S511 of FIG. 5B. Except for the subroutine, this embodiment is the same as the first embodiment, and a description thereof will be omitted.

The display virtual focal plane determination process according to this embodiment will be described with reference to FIGS. 9 and 10.
FIG. 9 is a flowchart of the display virtual focal plane determination process in this embodiment.
FIGS. 10A, 10B, and 10C are diagrams showing the displacement of the focal plane and the amount of displacement between display frames at a certain time during moving image shooting, and show the displacement in different scenes.
FIGS. 10D, 10E, and 10F show the displacement of the display virtual focal plane in the respective scenes of FIGS. 10A, 10B, and 10C.
FIGS. 9 and 10 correspond to FIGS. 6 and 7 described in the first embodiment, and parts already described are denoted by the same reference numerals and description thereof is omitted here.

  First, display virtual focal plane determination processing in the case where there is no significant change in focal plane due to switching of the subject distance will be described with reference to FIGS. 9, 10A, and 10D.

  When the display virtual focal plane determination process shown in FIG. 9 is started, in step S601 described in the first embodiment, the display virtual focal plane determination unit 106 displays the subject at the current display timing, frame Fr (x). The focal plane position information fp (x) corresponding to the distance information is obtained.

  In step S909, the display virtual focal plane determination unit 106 compares the amount of change in the focal plane position acquired in step S601 within a predetermined time with a preset threshold D (predetermined value) to determine the subject distance. Determine whether a switch has occurred. This determination process will be described with reference to FIG.

  The determination as to whether or not the subject distance is switched in the frame Fr (x) that is the current display timing is between the focal plane fp (x) and the focal plane fp (x−1) at the previous display timing. The comparison is made by comparing the amount of change fp_diff (x) occurring in the threshold value D.

  When the change amount fp_diff (x) is larger than the threshold value D, it is determined that the subject distance is switched, and when the change amount fp_diff (x) is smaller than the threshold value D, it is determined that the subject distance is not switched. The threshold value D is a threshold value for the amount of change in the focal plane within a predetermined time that is predetermined for detecting the switching of the subject distance, and is stored in a memory (not shown) in the imaging apparatus 100.

  In step S910, the display virtual focal plane determination unit 106 determines whether or not the subject distance has been switched in the frame Fr (x-1) which is the previous display timing. In this embodiment, it is assumed that the focal plane change amount fp_diff (x−1) in the frame Fr (x−1) in FIG. Therefore, in step S910, the display virtual focal plane determination unit 106 determines that there is no switching of the subject distance in the frame Fr (x−1), and proceeds to step S602.

  For the change in the focal plane shown in FIG. 10 (a), the display virtual focal plane determining unit 106 performs the processes of steps S602, S603, S604, S606, and S608 in the same manner as in the first example. Thus, the display virtual focal plane fp_out (x−1) shown in FIG. 10D is determined, and the display virtual focal plane determination process is ended.

  Here, if the focal plane variation fp_diff (x) in FIG. 10A is also smaller than the threshold D, the processing in the frame Fr (x + 1) is the same, and the display virtual focal plane shown in FIG. fp_out (x) is determined.

  Next, with reference to FIG. 9, FIG. 10B, and FIG. 10E, a display virtual focal plane determination process when a large change in focal plane due to switching of the subject distance occurs only once within a predetermined time will be described. To do.

  Assume that the focal plane variation fp_diff (x−1) in the frame Fr (x−1) shown in FIG. Accordingly, in step S910, the display virtual focal plane determination unit 106 determines that the subject distance is switched in the frame Fr (x−1), and the process proceeds to step S911.

  In step S911, the display virtual focal plane determination unit 106 determines whether the subject distance is switched in the frame Fr (x). That is, it is determined whether or not the subject distance is continuously switched.

  Since there is no change in the focal plane in the frame Fr (x) in FIG. 10B, in step S911, the display virtual focal plane determining unit 106 determines that the subject distance is not switched in the frame Fr (x), and the step The process proceeds to S912.

  In step S912, the display virtual focal plane determination unit 106 determines the display virtual focal plane fp_out (x−1) shown in FIG. 10E as the focal plane corresponding to the actual focus lens position, and displays the virtual display plane. The focal plane determination process ends.

  Note that the thin line shown in FIG. 10E indicates the virtual focal plane for display when it is determined in each process of step S602 to step S608 described in the first embodiment.

  In the processing for the frame Fr (x + 1) in FIG. 10B, the display virtual focal plane determination unit 106 in step S910 determines that the subject distance has not been switched. Therefore, after step S910, the display virtual focal plane determination unit 106 determines the display virtual focal plane by the processes of steps S602, S603, S604, S606, and S608, and the display virtual focal plane shown in FIG. It becomes like fp_out (x).

  Next, the display virtual focal plane determination process when the subject distance is continuously switched will be described with reference to FIGS. 9, 10C, and 10F.

  Assume that focal plane variations fp_diff (x) and fp_diff (x−1) in the frames Fr (x) and Fr (x−1) shown in FIG. Accordingly, in step S910, the display virtual focal plane determination unit 106 determines that the subject distance is switched in the frame Fr (x−1), and the process proceeds to step S911.

  In step S911, the display virtual focal plane determination unit 106 determines that the subject distance is switched in the frame Fr (x), and the process proceeds to step S602.

  When the switching of the subject distance is continuous, the display virtual focal plane determination unit 106 determines the display virtual focal plane fp_out (x−1) through steps S602 to S608 as in the first embodiment. Then, the display virtual focal plane determination process ends.

  For FIG. 10C, the display virtual focal plane determination unit 106 determines the display virtual focal plane fp_out (x−1) in each process of steps S602, S603, S604, and S605.

  In the process in the frame Fr (x + 1) in FIG. 10C, since there is no change in the focal plane in the frame Fr (x + 1), the subject distance is switched in step S910, and the subject distance is switched in step S911. Is determined to be none, and the process proceeds to step S912.

  Accordingly, in step S912, the display virtual focal plane determination unit 106 determines the display virtual focal plane fp_out (x) shown in FIG. 10E as a focal point corresponding to the actual focus lens position.

  Similarly, in the frame Fr (x + 1) and subsequent frames, the display virtual focal plane determination unit 106 determines whether or not the subject distance has been switched continuously by the display virtual focal plane determination unit 106 shown in FIGS. As shown in f), the display virtual focal plane fp_out is determined.

  As described above, according to the present embodiment, if the subject distance is not switched, the focal plane in the display is controlled to be the virtual focal plane described in the first embodiment, and the subject distance is switched. In the case where a shift occurs, the focal plane is controlled so as to correspond to the focus lens position.

  Further, it is also determined whether or not the subject distance is continuously switched, and when the subject distance is continuously switched, the focal plane in the display is controlled to be a virtual focal plane.

  In other words, in this embodiment, when the focal plane changes particularly greatly due to the switching of the subject distance, the focal plane for display is set to the focal plane corresponding to the focus lens position so that the subject is in focus. The displayed image is displayed.

  On the other hand, for a relatively small focal plane change due to subject tracking, the display focal plane is controlled to be a virtual focal plane, and a moving image can be displayed with less burden on the photographer. It becomes possible.

  Furthermore, when another subject crosses in front of the main subject, the focus on the other subject is temporarily focused, and the continuous change in the focal plane such as focusing on the main subject is reduced. The photographer can display an image suitable for observation.

  Here, for the sake of explanation, an example in which the amount of change in the focal plane within a predetermined time is changed between one frame is shown, but the present invention is not limited to this. For example, a predetermined number of frames or a predetermined time period may be used.

  Further, in this embodiment, the method of determining the focal plane of the display image during moving image shooting has been described. Therefore, the focal plane change amount with respect to the past frame is used in the subject distance switching determination process. However, when the recorded moving image is reproduced, the amount of change in the focal plane with respect to the previous and subsequent display frames may be used.

  Similarly, the amount of change in the focal plane in the past frame is also used in the subject distance switching continuous determination process. However, when the recorded moving image is reproduced, the subject is determined from the amount of change in the focal plane before and after the display frame. You may determine the continuation of distance switching.

  In addition, although the example in which it is determined that the switching of the subject distance is continued when the switching of the subject distance occurs in the next frame of the frame in which the switching of the subject distance has occurred, the present invention is not limited to this. . For example, it may be determined that the switching of the subject distance is continuous when the switching of the subject distance occurs again in an arbitrary plurality of frames from the frame in which the switching of the subject distance has occurred.

  In the present embodiment, an example is shown in which the focal plane for display is controlled to the focal plane corresponding to the focus lens position when the subject distance is switched, but the present invention is not limited to this. Control may be made to a virtual focal plane on the driving direction side of the focal plane corresponding to the focus lens position, as in fp_out (x−1) in FIG. Note that the thin line shown in FIG. 10G indicates the virtual focal plane for display when the subject distance switching described in the present embodiment occurs. This makes it possible to display an in-focus image more quickly when the subject changes.

[Modification of Second Embodiment]
Next, a modification of the second embodiment of the present invention will be described with reference to FIG.
According to this modification, when an object such as an automobile having a high moving speed is photographed as a subject, the focal plane of the display image is controlled to the focal plane corresponding to the focus lens position, and the subject is focused. It is possible to make a display.

  In the second embodiment, the display virtual focal plane determination unit 106 determines switching of the subject distance by comparing the amount of change of the focal plane within a predetermined time with a predetermined threshold D, and switches the subject distance. In the case of continuous, the focal plane in the display was controlled to be a virtual focal plane.

  On the other hand, in the present modification, in addition to the processing of the second embodiment, the display virtual focal plane determination unit 106 continuously switches the subject distance based on the correlation of the focal plane variation fp_diff. Is further provided.

  The configuration of the imaging apparatus in this modification is the same as the configuration of the imaging apparatus 100 described in the first and second embodiments, and thus will not be described here.

  The recording process, display process, and focus adjustment process at the time of moving image shooting shown in FIGS. 5A, 5B, and 5C are the display virtual focal plane determination process shown in step S511 of FIG. 5B. Since operations similar to those of the first and second embodiments are performed except for the subroutine, the description thereof is omitted here.

  FIG. 11 is a flowchart of display virtual focal plane determination processing in the present modification. This flowchart differs from the flowchart of FIG. 9 according to the second embodiment only in that step S1113 is added.

  FIG. 11 corresponds to FIG. 6 and FIG. 9 described in the first and second embodiments, and parts already described are denoted by the same reference numerals and detailed description thereof is omitted.

  As shown in FIG. 11, if the display virtual focal plane determination unit 106 determines in step S911 that the subject distance has been switched, the process proceeds to step S1113. If it is determined that the subject distance has not been switched, step S1113 is performed. The process proceeds to S912.

  In step S <b> 1113, the display virtual focal plane determination unit 106 determines whether there is a correlation in the switching of the subject distance in the frame Fr (x−1) and the frame Fr (x). That is, it is determined whether or not there is a correlation between the focal plane change amount fp_diff (x−1) in the frame Fr (x−1) and the focal plane change amount fp_diff (x) in the frame Fr (x). Details of determination on correlation will be described later.

  If the display virtual focal plane determination unit 106 determines in step S1113 that there is a correlation in the switching of the subject distance, the process proceeds to step S912, and if it is determined that there is no correlation in the switching of the subject distance, step S602. Proceed to

  When the process proceeds to step S912, the display virtual focal plane determination unit 106 determines the display virtual focal plane fp_out (x-1) as the focal plane corresponding to the focus lens position, and ends the display virtual focal plane determination process. To do.

  When the process proceeds to step S602, the display virtual focal plane determination unit 106 determines the display virtual focal plane fp_out (x-1) through steps S602 to S608, and the display virtual focal plane determination process ends.

Next, a description will be given of the determination of the correlation in the switching of the subject distance.
In this modification, the correlation is assumed to be two, that is, the change amount correlation and the direction correlation. First, the change amount correlation will be described.
The change amount correlation is determined that there is a change amount correlation when the difference between the focal plane change amounts fp_diff (x−1) and fp_diff (x) is less than or equal to a threshold value N (predetermined amount or less). If it is greater than the value, it is determined that there is no correlation between the change amounts.
The threshold value N is a predetermined threshold value for determining whether or not there is a correlation in switching between consecutive subject distances, and is not shown in the imaging apparatus 100 as with the threshold value D described in the first embodiment. Stored in the memory.

Next, the correlation of directions will be described.
For example, the amount of change of the focal plane when the focus lens is driven to the close side is defined as + fp_diff, and the amount of change of the focal plane when the focus lens is changed by driving to the infinity side is defined as −fp_diff.
Then, when the signs of the focal plane change amounts fp_diff in the frames Fr (x−1) and Fr (x) are the same, it is determined that there is a correlation of directions, and when the signs are different, there is no correlation of directions. judge.

  In this modification, when the display virtual focal plane determination unit 106 determines that both the amount of change and the direction are correlated in step S <b> 1113, the display virtual focal plane determination unit 106 determines that the switching of the subject distance is correlated. . If it is determined that there is no correlation in either one, it is determined that there is no correlation in the switching of the subject distance. That is, even if it is determined that the focal plane change amount between the frames Fr (x−1) and the frame Fr (x) is large and the switching of the subject distance is continuous, if the focal plane change is correlated, the display is performed. The focal plane is controlled so as to be a focal plane corresponding to the position of the focus lens.

  As described above, in this modification, when an object with a high moving speed such as an automobile is photographed as a subject, the focal plane in the display is the focal plane corresponding to the focus lens position. As a result, while displaying a moving image with less burden on the photographer, it is possible to display a focused image following the subject intended by the photographer.

  The object of the present invention can also be achieved by supplying a storage medium storing software program codes for realizing the functions of the above-described embodiments to a system or apparatus. That is, it goes without saying that the object of the present invention can also be achieved when the computer (or CPU or MPU) of the system or apparatus reads and executes the program code stored in the storage medium.

  In this case, the program code itself read from the storage medium realizes the functions of the above-described embodiments, and the program code itself and the storage medium storing the program code constitute the present invention.

  As a storage medium for supplying the program code, for example, a flexible disk, a hard disk, an optical disk, a magneto-optical disk, a CD-ROM, a CD-R, a magnetic tape, a nonvolatile memory card, a ROM, or the like can be used.

  In addition, the functions of the above-described embodiment can also be realized when an OS (basic system or operating system) operating on the computer performs part or all of the actual processing based on the instruction of the program code read by the computer. Realized. Needless to say, this case is also included in the present invention.

  Furthermore, after the program code read from the storage medium is written to the memory provided in the function expansion board inserted in the computer or the function expansion unit connected to the computer, the processing based on the instruction of the program code is also performed. Included in the invention. That is, it goes without saying that the present invention also includes the case where the CPU or the like provided in the function expansion board or function expansion unit performs part or all of the actual processing based on the instruction of the program code to realize the functions of the above-described embodiment. Yes.

  The above-described embodiments are merely examples of implementation in carrying out the present invention, and the technical scope of the present invention should not be construed as being limited thereto. That is, the present invention can be implemented in various forms without departing from the technical idea or the main features thereof.

Claims (15)

An image processing apparatus for processing imaging data that can be refocused and obtained by imaging an optical image of a subject formed by an imaging optical system including a focus lens with an imaging unit,
Imaging data acquisition means for acquiring imaging data that can be refocused;
Lens information acquisition means for acquiring information on the position of the focus lens;
A focal plane determination means for determining the position of the focal plane of the display image of the acquired imaging data based on the acquired focus lens position information;
Refocus processing means for generating image data of the refocused image on the focal plane at the determined position from the imaging data;
An image processing apparatus, comprising: display data generation means for generating display data of acquired imaging data based on the generated image data of a refocus image.   The focal plane determination means determines the focal plane position of the display image based on the amount of change in the focal plane position corresponding to the acquired focus lens position and the second focal point. The first focal plane is a focal plane based on the average of the amount of change, and the second focal plane is the position of the focus lens. The image processing apparatus according to claim 1, wherein the third focal plane is a focal plane having a change amount steeper than the change amount.   The amount of change of the focal plane corresponding to the position of the focus lens is the position of the focal plane corresponding to the position of the focus lens in a predetermined number of frames or a predetermined time period before or after the frame of the display image. The focal plane determination means sets the focal plane of the display image as the first focal plane when the variation is smaller than a predetermined value, and the display image when the variation is larger than the predetermined value. The image processing apparatus according to claim 2, wherein the focal plane is any one of the second focal plane and the third focal plane.   The image processing according to claim 3, wherein the focal plane determination unit sets the focal plane of the display image as the first focal plane when a change amount larger than the predetermined value is continuous. apparatus.   When the amount of change larger than the predetermined value is continuous, the focal plane determination means determines the focal plane of the display image according to the correlation between the continuous change amount based on the difference between the continuous change amounts and the direction of change. 5. The image processing apparatus according to claim 4, wherein the image processing apparatus determines any one of the position of the first focal plane, the position of the second focal plane, and the position of the third focal plane.   The focal plane determination means sets the focal plane of the display image as the position of the second focal plane when the difference between the variation amounts is equal to or less than a predetermined amount and the direction of the variation is the same, and otherwise The image processing apparatus according to claim 5, wherein a focal plane of the display image is the first focal plane.   The focal plane determination unit further acquires information of the photographing optical system, determines a refocusable range in the refocus processing based on the information of the photographing optical system, and the determined refocusable range 7. The image processing apparatus according to claim 2, wherein the position of the first focal plane and the position of the third focal plane are limited based on the above.   The focal plane determination means determines the determined position of the first focal plane when the position of the determined first focal plane or the position of the third focal plane is not within the refocusable range. The image processing apparatus according to claim 7, wherein the position of the third focal plane is a limit position of the refocusable range.   The image processing apparatus according to claim 1, wherein the imaging data acquisition unit acquires imaging data that can be refocused and reproduced from a recording medium. In a control method of an image processing apparatus that processes imaging data that can be refocused, obtained by imaging an optical image of a subject formed by a photographing optical system including a focus lens by an imaging unit,
An imaging data acquisition step of acquiring imaging data capable of the refocus processing;
A lens information acquisition step of acquiring information of the position of the focus lens;
A focal plane determination step for determining a focal plane position of a display image of the acquired imaging data based on the acquired position information of the focus lens;
A refocus processing step of generating image data of a refocus image on the focal plane at the determined position from the imaging data;
A display data generation step for generating display data of the acquired imaging data based on the image data of the generated refocus image;
In the focal plane determination step, the focal plane position of the display image is determined based on the amount of change in the focal plane position corresponding to the acquired focus lens position. Determining a position of any one of the planes, wherein the first focal plane is a focal plane based on an average of the amount of change, and the second focal plane corresponds to a position of the focus lens. The control method characterized by being. A program for controlling an image processing apparatus that processes imaging data that can be refocused and obtained by imaging an optical image of a subject formed by an imaging optical system including a focus lens with an imaging unit,
Computer
Imaging data acquisition means for acquiring imaging data capable of the refocus processing;
Lens information acquisition means for acquiring information on the position of the focus lens;
A focal plane determination unit that determines a focal plane position of a display image of the acquired imaging data based on the acquired focus lens position information, and the focal plane determination unit sets the focal plane determination unit at a position of the acquired focus lens. Based on the amount of change in the corresponding focal plane position, the focal plane position of the display image is determined as either the first focal plane position or the second focal plane position, and the first focal plane position is determined. Is a focal plane based on the average of the amount of change, and the second focal plane is a focal plane corresponding to the position of the focus lens,
Refocus processing means for generating image data of a refocus image on the focal plane at the determined position from the imaging data;
A program for functioning as display data generation means for generating display data of the acquired imaging data based on the generated image data of the refocus image.   A computer-readable storage medium storing the program according to claim 11.   A program that causes a computer to function as each unit of the image processing apparatus according to any one of claims 1 to 9.   A storage medium storing a program that causes a computer to function as each unit of the image processing apparatus according to any one of claims 1 to 9. A taking optical system including a focus lens;
An imaging unit that captures an optical image of a subject formed by the imaging optical system and generates imaging data that can be refocused;
Lens information acquisition means for acquiring information on the position of the focus lens;
A focal plane determining unit that determines a focal plane position of a display image of the imaging data generated by the imaging unit based on the acquired position information of the focus lens;
Refocus processing means for generating image data of a refocus image on the focal plane at the determined position from the imaging data;
Display data generating means for generating display data of the generated imaging data based on the image data of the generated refocus image;
Display means for displaying the display data,
The focal plane determination means determines the focal plane position of the display image based on the amount of change in the focal plane position corresponding to the acquired focus lens position and the second focal point. The first focal plane is a focal plane based on the average value of the amount of change, and the second focal plane is a focal plane corresponding to the position of the focus lens. An imaging apparatus characterized by that.

Images