JP2015087450A - Imaging device and imaging processing program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To suppress an amount of computation upon calculating an amount of phase differences in a pair of focus detection optical images.SOLUTION: An imaging device according to the present invention comprises: an image sensor; determination means for, when focus detection data is included in partial image data output by the image sensor until a periodic prescribed timing, extracting the focus detection data from the partial image data, and determining whether the extracted focus detection data satisfies a prescribed condition; record means for, when it is determined by the determination means that the extracted focus detection data satisfies the prescribed condition, recording the focus detection data in a storage part, and when it is determined by the determination means that the extracted focus detection data does not satisfy the prescribed condition, not recording the focus detection data in the storage part; and calculation means for calculating an amount of phase differences in a pair of focus detection optical images.

Description

  The present invention relates to an imaging apparatus and an imaging processing program.

  There has been known an imaging device in which a plurality of types of imaging pixels having different spectral sensitivity characteristics are regularly arranged in a two-dimensional manner to capture an image formed by an optical system (see, for example, Patent Document 1). In the imaging device, a part of the imaging pixels along a predetermined direction is configured as a focus detection pixel, and the same spectral sensitivity characteristic as that of the imaging pixel located at the position of the focus detection pixel in a regular arrangement The image pickup pixels and the focus detection pixels are mixedly arranged on the pixel array in a predetermined direction so that the image pickup pixels having the above are arranged between the plurality of focus detection pixels.

Japanese Patent Laid-Open No. 2008-40087

  As the number of focus detection pixels included in the focus detection pixel row of the image sensor increases, the physical amount (for example, the amount of data to be handled, the memory bandwidth, the memory occupation amount, and the calculation) At least one of the quantities) increases. For this reason, there is a possibility that a problem occurs in the processing of the read data. For example, when the user does not designate a focus detection area, a large amount of data must be processed with a limited processing capability, and a defocus amount with low reliability may be calculated.

(1) The imaging apparatus according to claim 1 is responsive to a pair of focus detection optical images of a subject formed by a pair of focus detection light beams that pass through the pair of pupil regions of the optical system from the subject. A plurality of focus detection pixels that output a pair of focus detection data, and a plurality of imaging data corresponding to the imaging optical image formed by the imaging light flux that comes from the subject through the optical system. A plurality of pixels composed of a plurality of imaging pixels are arranged in a two-dimensional plane, and outputs an image data including focus detection data and imaging data, and a predetermined periodic timing among the image data If the partial image data output by the image sensor includes the focus detection data, the focus detection data is extracted from the partial image data, and the extracted focus detection data is predetermined. If the determination means for determining whether or not the condition is satisfied and the focus detection data extracted by the determination means satisfy the predetermined condition, the focus detection data extracted by the determination means is stored in the storage unit. When it is determined that the focus detection data recorded and extracted by the determination unit does not satisfy the predetermined condition, the focus detection data extracted by the determination unit is not recorded in the storage unit, and the recording unit By calculating the phase difference amount of the pair of focus detection optical images based on the focus detection data recorded in the storage unit, and by controlling the optical system based on the phase difference amount calculated by the calculation means An optical system control unit that adjusts the focal position of the optical system and an image generation unit that generates an image based on imaging data are provided.
(2) According to a tenth aspect of the present invention, there is provided an imaging processing program for generating a pair of focus detection optical images of a subject formed by a pair of focus detection light beams coming from the subject through a pair of pupil regions of the optical system. A plurality of focus detection pixels that output a pair of corresponding focus detection data and imaging data corresponding to the imaging optical image formed by the imaging light flux that comes from the subject through the optical system is output. When image data is output from an image sensor that outputs image data including focus detection data and imaging data, a plurality of pixels including a plurality of imaging pixels are arranged in a two-dimensional plane. If focus detection data is included in the partial image data output by the image sensor by a predetermined periodic timing, the focus detection data is extracted from the focus detection data. Determination processing for determining whether or not the focus detection data extracted and extracted satisfies a predetermined condition, and if it is determined that the focus detection data extracted by the determination processing satisfies a predetermined condition, the determination processing extracts If the focus detection data extracted by the determination process is determined not to satisfy the predetermined condition, the focus detection data extracted by the determination process is stored in the storage unit. A recording process that does not record the image, a calculation process that calculates the phase difference amount of the pair of focus detection optical images based on the focus detection data recorded in the storage unit by the recording process, and the optical system control unit controls the optical system Based on the imaging system data and the optical system control process for outputting the phase difference amount calculated by the calculation process to the optical system control unit so as to adjust the focal position of the optical system Characterized in that to execute the image generation processing for generating an image on the computer.

  According to the present invention, even if the number of focus detection pixels included in the focus detection pixel row of the image sensor increases, by appropriately selecting data to be handled among the data output from the focus detection pixels, Therefore, it is possible to suppress a possibility of causing a problem when processing the read data. For example, it is possible to reduce the amount of calculation when calculating the phase difference between the pair of focus detection optical images.

It is a figure which shows the structure of the imaging device which concerns on 1st Embodiment. It is a figure for demonstrating the mode of the imaging light beam which the imaging pixel receives. It is a figure for demonstrating the mode of the light beam for phase difference AF which the pixel for phase difference AF light-receives. It is a figure which shows the example of arrangement | positioning of the pixel arrangement area for phase difference AF on an image sensor, and the pixel arrangement area for imaging. It is a figure which shows an example of the data for phase difference AF for one line which cut the high frequency component, the large threshold value, and the small threshold value. It is a figure which shows another example of the data for phase difference AF for one line which cut the high frequency component, the large threshold value, and the small threshold value. FIGS. 7A and 7B are flowcharts of AF control performed by the control device of the imaging apparatus according to the first embodiment. FIG. 7A illustrates a phase difference AF data recording flow, and FIG. 7B illustrates a phase difference amount calculation flow. It is. It is a figure which shows the example of arrangement | positioning with the pixel arrangement area for phase difference AF on the image sensor of the imaging device which concerns on 2nd Embodiment, and the pixel arrangement area for imaging. It is a figure which shows the structure of the imaging device which concerns on 3rd Embodiment. FIGS. 10A and 10B are flowcharts of AF control performed by the control device of the imaging apparatus according to the third embodiment. FIG. 10A illustrates a phase difference AF data recording flow, and FIG. 10B illustrates an addition processing flow. It is a figure which shows the provision form of the computer program product for the control apparatus of the imaging device by this invention for performing an imaging process.

--- First embodiment ---
In the imaging apparatus 100 according to the first embodiment of the present invention, the phase difference AF data extraction unit 3 extracts the phase difference AF data from the image data output from the image sensor 2. The phase difference AF data extracted by the phase difference AF data extraction unit 3 is recorded in the phase difference AF data temporary storage unit 7 in units of lines by the determination unit 4 of the phase difference AF data extraction unit 3. The recording unit 5 of the phase difference AF data extraction unit 3 determines only the phase difference AF data of some lines according to the determination result in units of lines by the determination unit 4 of the phase difference AF data extraction unit 3. The data is recorded in the phase difference AF data storage unit 6. Using the phase difference AF data recorded in the phase difference AF data storage unit 6, the phase difference amount calculation unit 8 calculates the phase difference amount. Based on the phase difference calculation result by the phase difference calculation unit 8, the optical system control unit 9 controls the optical system 1 to automatically adjust the focus shift.

  The imaging device 100 according to the first embodiment has the configuration shown in FIG. The imaging apparatus 100 includes an optical system 1, an image sensor 2, a control device 110, a phase difference AF data temporary storage unit 7, a phase difference AF data storage unit 6, an optical system control unit 9, and a display unit 13. The control device 110 is a computer including a phase difference AF data extraction unit 3, a phase difference amount calculation unit 8, an interpolation data generation unit 10, an image generation recording unit 11, and a display image generation unit 12. The phase difference AF data extraction unit 3 includes a determination unit 4 and a recording unit 5. The determination unit 4 records the phase difference AF data in the phase difference AF data temporary storage unit 7, and whether or not the recorded phase difference AF data satisfies a predetermined condition, that is, data necessary for calculating the phase difference amount. It is determined in units of lines. Whether the recording unit 5 reads out the phase difference AF data recorded in the phase difference AF data temporary storage unit 7 based on the result determined by the determination unit 4 and records it in the phase difference AF data storage unit 6. Whether or not is controlled in units of lines.

  The optical system 1 includes a plurality of lenses such as a zoom lens, transmits a light beam from a subject, and forms an optical image on the image sensor 2. The image sensor 2 converts optical information included in the formed optical image into electrical information. The electrical information has RGB color components constituting the image data, and can be represented by a depth of 8 bits or 12 bits for each color. The image sensor 2 includes a plurality of pixels including a plurality of imaging pixels 210 illustrated in FIG. 2 and a plurality of pairs of focus detection pixels illustrated in FIG. 3, that is, a plurality of pairs of phase difference AF pixels 213 and 214. For example, as shown in FIG. 4, they are arranged in a two-dimensional plane. The electrical information converted on the phase difference AF pixels 213 and 214 is, as focus detection data, that is, phase difference AF data, for imaging composed of RGB color components of the electrical information converted by the imaging pixel 210. The data is mixed and output from the image sensor 2. Thus, the image sensor 2 outputs image data including phase difference AF data and imaging data. As shown in FIG. 4, in each of the line N and line M of the image sensor 2, the phase difference AF pixel arrangement area is divided into four by the imaging pixel arrangement area. This is because the interpolation data generation unit 10 does not deteriorate the interpolation performance when interpolating and generating the image data at the pixel position for phase difference AF, and the phase difference AF control of this embodiment is divided into four. The entire phase difference AF data for one line is performed as one set of phase difference AF data.

  FIG. 2 is a diagram for explaining the state of the imaging light beam 71 received by the imaging pixel 210. The imaging light beam 71 passes through the optical system 1 from a subject (not shown) and arrives at the imaging pixel 210.

  The photoelectric conversion units 411 of all the imaging pixels 210 arranged on the image sensor 2 receive the imaging light beam 71 transmitted through the optical system 1. The shape of the photoelectric conversion unit 411 is projected onto the pupil region 95 on the optical system 1 that is separated from the microlens 410 by the distance d by the microlens 410 of each imaging pixel 210.

  Accordingly, the photoelectric conversion unit 411 of each imaging pixel 210 receives the imaging light flux 71 that passes through the pupil region 95 and the microlens 410 of each imaging pixel 210, and is connected to each microlens 410 by the imaging light flux 71. An imaging signal corresponding to the intensity of the imaged optical image is output. A set of imaging signals output from the plurality of imaging pixels 210 forms imaging data.

  FIG. 3 is a diagram for explaining the state of phase difference AF light beams (focus detection light beams) 73 and 74 received by the phase difference AF pixels 213 and 214 in comparison with FIG. The phase difference AF light beams 73 and 74 are transmitted from a subject (not shown) through the pair of pupil regions 93 and 94 of the optical system 1 to the phase difference AF pixels 213 and 214, respectively.

  Of all the phase difference AF pixels 213 and 214 arranged on the image sensor 2, the photoelectric conversion unit 413 of the phase difference AF pixel 213 is one of the pair of pupil regions 93 and 94 of the optical system 1. The phase difference AF light beam 73 transmitted through the pupil region 93 is received. The photoelectric conversion unit 414 of the phase difference AF pixel 214 receives the phase difference AF light beam 74 transmitted through the other pupil region 94 of the pair of pupil regions 93 and 94 of the optical system 1. The shape of the photoelectric conversion unit 413 is projected onto one pupil region 93 common to all the phase difference AF pixels 213 on the optical system 1 separated from the microlens 410 by the distance d by the microlens 410 of each phase difference AF pixel 213. Is done. The shape of the photoelectric conversion unit 414 is the same as that of the other pupil region 94 common to all the phase difference AF pixels 214 on the optical system 1 separated from the microlens 410 by the distance d by the microlens 410 of each phase difference AF pixel 214. Projected.

  Accordingly, the photoelectric conversion unit 413 of each phase difference AF pixel 213 receives the phase difference AF light beam 73 passing through one pupil region 93 and the microlens 410 of each phase difference AF pixel 213, and the phase difference AF light beam 73. Thus, a phase difference AF signal (focus detection signal) corresponding to the intensity of the phase difference AF optical image (focus detection optical image) formed on each microlens 410 is output. The photoelectric conversion unit 414 of each phase difference AF pixel 214 receives the phase difference AF light beam 74 that passes through the other pupil region 94 and the micro lens 410 of each phase difference AF pixel 214, and the phase difference AF light beam 74 A phase difference AF signal corresponding to the intensity of the phase difference AF optical image formed on each microlens 410 is output. A set of phase difference AF signals output from the plurality of phase difference AF pixels 213 and a set of phase difference AF signals output from the plurality of phase difference AF pixels 214 constitute a pair of phase difference AFs described later. Form data for use.

  The imaging pixel 210 receives the pupil region obtained by integrating the pair of pupil regions 93 and 94 on the optical system 1 through which the phase difference AF light beams 73 and 74 received by the pair of phase difference AF pixels 213 and 214 pass. This coincides with the pupil region 95 on the optical system 1 through which the photographing light beam 71 passes. On the optical system 1, the phase difference AF light beams 73 and 74 are in a complementary relationship with the photographing light beam 71.

  In the image sensor 2, a pair of phase difference AF pixels 213 and 214 are arranged in a straight line, thereby arranging a pair of phase difference AF pixels. Preferably, such a pair of phase difference AF pixel rows is arranged on a plurality of lines of the image sensor 2. In the pair of phase difference AF pixel rows, the plurality of phase difference AF signals output from the photoelectric conversion units 413 and 414 of the phase difference AF pixels 213 and 214 are a pair of positions corresponding to the pair of pupil regions 93 and 94, respectively. A signal sequence for phase difference AF is formed. Thus, a pair of positions relating to the intensity distribution of the pair of phase difference AF optical images formed on the phase difference AF pixel row by the pair of phase difference AF light beams 73 and 74 passing through the pair of distance measuring pupils 93 and 94, respectively. A pair of phase difference AF data that is a phase difference AF signal train is obtained. By performing phase difference amount calculation processing by correlation calculation on the pair of phase difference AF data, the phase difference amount of the pair of phase difference AF optical images by the so-called pupil division type phase difference detection method is detected. . Further, the amount of driving of the optical system 1 controlled by the optical system control unit 9 is performed by performing a conversion operation according to the proportional relationship between the distance between the center of gravity of the pair of pupil regions 93 and 94 and the distance d described above on the phase difference amount. Is calculated.

  Returning to the description using FIG. The phase difference AF data extraction unit 3 extracts image plane phase difference AF pixels 213 and 214 from partial image data (corresponding to input line data described later) of one line of the image data output from the image sensor 2. The phase difference AF data is extracted based on the arrangement information, and the extracted phase difference AF data is recorded in the phase difference AF data temporary storage unit 7.

  Further, the phase difference AF data extraction unit 3 outputs the image data extracted from the phase difference AF data to the interpolation data generation unit 10. The interpolation data generation unit 10 uses the image data of the peripheral pixel positions of the phase difference AF pixel position from the extracted image data of the phase difference AF data output from the phase difference AF data extraction unit 3. Image data at the pixel position for phase difference AF is generated by interpolation and output to the image generation / recording unit 11 and the display image generation unit 12. The image generation / recording unit 11 records image data in which the image data of the phase difference AF pixel position output from the interpolation data generation unit 10 is generated by interpolation on a recording medium 14 such as a DRAM or a flash memory. The display image generation unit 12 reduces the image data obtained by interpolation generation of the image data of the phase difference AF pixel position output from the interpolation data generation unit 10 to a size to be displayed on the display unit 13 (the output size from the sensor is the display size) If it is the same, the same magnification) process is performed and output to the display unit 13. The display unit 13 is an LCD panel or an organic EL panel, and displays the image data output from the display image generation unit 12. A user of the imaging apparatus 100 performs shooting using the image data displayed on the display unit 13.

  The phase difference AF data temporary storage unit 7 is composed of, for example, an SRAM that can record and hold phase difference AF data for two lines. The phase difference AF data temporary storage unit 7 writes (records) phase difference AF data for one line, and reads phase difference AF data for one line different from the writing line on which the writing is performed. Can be performed simultaneously.

  The phase difference AF data extraction unit 3 extracts the phase difference AF data for one line, and performs writing (recording) processing in the phase difference AF data temporary storage unit 7. The determination unit 4 determines whether or not the phase difference AF data for one line on which the writing process is performed has data necessary for calculating the phase difference amount, and determines whether the phase difference AF data has extreme values (maximum value and minimum value). Value) and a threshold value. That is, the determination unit 4 has a phase difference AF data maximum value in one line larger than the larger one of the two arbitrary threshold values set in advance (hereinafter referred to as a large threshold value), and the position in one line. When the phase difference AF data minimum value is smaller than the smaller one of two preset thresholds (hereinafter referred to as a small threshold value), one line of phase difference AF data is used to calculate the phase difference amount. Determine that you have the necessary data. Note that the determination based on such a predetermined condition is preferably performed after executing a filtering process for cutting a high-frequency component on a plurality of phase difference AF data in one line.

  It is assumed that phase difference AF data for one line corresponding to line N in FIG. 4 is extracted by the phase difference AF data extraction unit 3 and recorded in the phase difference AF data temporary storage unit 7. FIG. 5 shows phase difference AF data for one line obtained by cutting a high frequency component from the phase difference AF data for one line, a large threshold value, and a small threshold value. As is apparent from FIG. 5, the maximum value A> large threshold value and the minimum value B <small threshold value of the phase difference AF data 50 for one line from which the high frequency component is cut. At this time, the determination unit 4 determines that the phase difference AF data 50 in the line N has data necessary for calculating the phase difference amount. The recording unit 5 reads out the phase difference AF data 50 for the line N from the phase difference AF data temporary storage unit 7 based on the determination result in the determination unit 4 and writes it into the phase difference AF data storage unit 6. The phase difference AF data 50 for the line N is recorded in the phase difference AF data storage unit 6.

  It is assumed that the phase difference AF data extraction unit 3 extracts one line of phase difference AF data corresponding to the line M in FIG. 4 and records it in the phase difference AF data temporary storage unit 7. FIG. 6 shows phase difference AF data 60 for one line obtained by cutting a high frequency component from the phase difference AF data for one line, a large threshold value, and a small threshold value. As is apparent from FIG. 6, the maximum value A ≦ the large threshold value of the phase difference AF data 60 for one line from which the high frequency component is cut. At this time, the determination unit 4 determines that the phase difference AF data 60 in the line M does not have data necessary for calculating the phase difference amount. The recording unit 5 does not read the phase difference AF data 60 of the line M recorded in the phase difference AF data temporary storage unit 7 based on the determination result in the determination unit 4.

  In this way, the determination unit 4 determines whether or not the phase difference AF data recorded in the phase difference AF data temporary storage unit 7 has data necessary for calculating the phase difference amount in the order of input lines in units of input lines. Do. The recording unit 5 controls the reading of the phase difference AF data from the phase difference AF data temporary storage unit 7 based on the determination result in the determination unit 4 and the phase difference AF data to the phase difference AF data recording 6. Write (record) control is performed. Based on the determination result of the determination unit 4, the recording unit 5 performs read control and write (record) control, so that the phase difference AF data can be used to locate an input line that does not have data necessary for phase difference calculation. Since control for not writing (not recording) in the phase difference AF data storage unit 6 is performed, the amount of data written in the phase difference AF data storage unit 6 can be reduced.

  The phase difference amount calculation unit 8 reads the phase difference AF data written (recorded) in the phase difference AF data storage unit 6 and, based on the read phase difference AF data, the pair of phase differences described above. The phase difference amount of the AF optical image is calculated. The optical system control unit 9 adjusts the focal position of the optical system 1 by performing AF control of the optical system 1 based on the phase difference amount calculated by the phase difference calculation unit 8.

  FIG. 7 shows an AF control flowchart by the control device 110 of the imaging apparatus 100 according to the first embodiment. FIG. 7A shows a phase difference AF data recording flow of the phase difference AF data extracting unit 3 for recording the phase difference AF data in the phase difference AF data storage unit 6.

  As processing shown in processing step S10, input line data (partial image data) for one line of image data captured by the image sensor 2 is input to the phase difference AF data extraction unit 3 in line order. .

  As processing shown in processing step S20, the phase difference AF data extraction unit 3 determines whether or not the input line data includes phase difference AF data. When the input line data does not include phase difference AF data, the process proceeds to the process shown in process step S60. If the input line data includes phase difference AF data, the phase difference AF data extraction unit 3 proceeds to the process shown in process step S30.

  As the processing shown in the processing step S30, the phase difference AF data extraction unit 3 performs a recording process of extracting the phase difference AF data from the input line data and writing it in the phase difference AF data temporary storage unit 7.

  As processing shown in processing step S40, the determination unit 5 in the phase difference AF data extraction unit 3 determines whether or not the phase difference AF data included in the input line data has data necessary for calculating the phase difference amount. That is, whether or not the above-described predetermined condition is satisfied is determined from the relationship between the phase difference AF data for one line from which the high-frequency component is cut and the large threshold and the small threshold. As a result of the determination, if the amount of data necessary for calculating the phase difference amount does not exist, the processing proceeds to processing step S60. As a result of the determination, if there is data necessary for calculating the phase difference amount, the process proceeds to processing step S50.

  As the processing shown in the processing step S50, the recording unit 5 in the phase difference AF data extracting unit 3 uses the phase difference AF for one line written (recorded) in the phase difference AF data temporary storage unit 7. Data is read, and the read phase difference AF data for one line is written (recorded) in the phase difference AF data storage unit 6.

  As processing shown in processing step S60, the phase difference AF data extraction unit 3 determines whether or not the input line data is the last line of one frame meaning one image, and the input line is one frame. If it is not the last line, the process moves to the process shown in process step S10, and the next line is input. When the input line is the last line of one frame, the process proceeds to the first line input of the next frame, and thereafter, the processing steps S10 to S60 are sequentially repeated for a plurality of frames.

  A calculation flow by the phase difference calculation unit 8 of the phase difference necessary for the optical system control unit 9 to control the optical system 1 will be described. As the processing shown in the processing step S110, the processing priority in a plurality of lines such as the line N and the line M (see FIG. 4) including the phase difference AF pixels 213 and 214 in one screen is determined according to an arbitrary determination procedure. To do. For example, the line N is determined as the highest priority line.

  As the process shown in process step S120, the process target is processed so that the process shown in process step S130 is repeated at a predetermined cycle in order of priority determined in process step S110 until the process shown in process step S130 is positively determined. Change the line sequentially. A plurality of imaging pixels 210 shown in FIG. 2 and a plurality of pairs of focus detection pixels shown in FIG. 3, that is, a plurality of pairs of phase difference AF pixels 213 and 214, are arranged in a two-dimensional plane. The image sensor 2 outputs image data as input line data to the phase difference AF data extraction unit 3 of the control device 110 for each row of the two-dimensional plane. The periodic predetermined timing is a timing at which the input line data is output by the image sensor 2.

  As processing shown in processing step S130, the phase difference amount calculation unit 8 determines whether or not readable phase difference AF data is recorded in the phase difference AF data storage unit 6. If the phase difference AF data is not recorded as a result of the determination, the check is repeated repeatedly at a predetermined timing until the phase difference AF data is recorded in the phase difference AF data storage unit 6 and can be read out. . If the phase difference AF data is recorded as a result of the determination in the process indicated by process step S130, the process proceeds to process step S140.

  As processing shown in processing step S140, the phase difference amount calculation unit 8 reads out the phase difference AF data for one line from the phase difference AF data storage unit 6.

  As processing of processing step S150, the phase difference amount calculation unit 8 calculates the phase difference amount of the pair of phase difference AF optical images described above using the phase difference AF data for one line read out in processing step S140. The phase difference amount is output to the optical system control unit 9. Since the calculation of the phase difference amount using the phase difference AF data is a known technique, it will be omitted. The optical system control unit 9 automatically adjusts the focal position of the optical system 1 based on the phase difference amount calculated by the phase difference amount calculation unit 8. Thereafter, the process proceeds to the next frame, and the processing steps S110 to S150 for a plurality of frames are sequentially repeated thereafter.

  As described above, it is determined whether or not the data for phase difference AF has data necessary for calculating the phase difference amount in units of one line in the order of input lines. As a result of the determination, writing (recording) processing is performed in the phase difference AF data storage unit 6 only when the phase difference AF data of the input line has data necessary for calculating the phase difference amount.

  The imaging device 100 according to the present embodiment includes an image sensor 2, a control device 110, and an optical system control unit 9. The control device 110 includes the phase difference AF data extraction unit 3 including the determination unit 4 and the recording unit 5, the phase difference amount calculation unit 8, and at least one of the image generation recording unit 11 and the display image generation unit 12. Have.

  In the image sensor 2, a pair of subjects imaged on each microlens 410 by a pair of phase difference AF light fluxes 73 and 74 arriving from the subject through the pair of pupil regions 93 and 94 of the optical system 1. Each microlens is composed of a plurality of phase difference AF pixels 213 and 214 that output a pair of phase difference AF data corresponding to the phase difference AF optical image, and a photographing light beam 71 that arrives from the subject through the optical system 1. A plurality of pixels including a plurality of imaging pixels 210 that output imaging data corresponding to an imaging optical image formed on 410 are arranged in a two-dimensional plane, and the image sensor 2 is used for phase difference AF. Image data including data and imaging data is output.

  The determination unit 4 included in the phase difference AF data extraction unit 3 of the control device 110 includes, in the image data, one line of input line data output by the image sensor 2 by a predetermined period, If phase difference AF data is included, the phase difference AF data is extracted from the input line data, and it is determined whether or not the extracted phase difference AF data satisfies a predetermined condition.

  The recording unit 5 included in the phase difference AF data extraction unit 3 of the control device 110 extracts the phase difference AF data extracted by the determination unit 4 when the determination unit 4 satisfies the predetermined condition. The extracted phase difference AF data is recorded in the phase difference AF data storage unit 6. When the phase difference AF data extracted by the determination unit 4 determines that the predetermined condition is not satisfied, the determination unit 4 The extracted phase difference AF data is not recorded in the phase difference AF data storage unit 6.

  The phase difference amount calculation unit 8 calculates the phase difference amount of the pair of phase difference AF optical images based on the phase difference AF data recorded in the phase difference AF data storage unit 6 by the recording unit 5. The optical system control unit 9 adjusts the focal position of the optical system 1 by controlling the optical system 1 based on the phase difference amount calculated by the phase difference amount calculation unit 8. The image generation recording unit 11 and the display image generation unit 12 generate an image based on the above-described imaging data.

  In the case where the user does not specify the focus detection area, in the conventional imaging device, the reliability of the defocus amount is determined after the phase difference amount calculation using all the phase difference AF data is performed. According to imaging apparatus 100 according to the present embodiment, it is possible to remove phase difference AF data that causes defocus amount calculation with low reliability before calculating the defocus amount. It is determined whether or not there is data necessary for phase difference AF in line units in order of input lines by simple calculation using a threshold value, and based on the determination result, the phase difference AF data in line units in order of input lines is determined. Control the extraction. In this way, the phase difference AF is calculated with a smaller amount of data for phase difference AF than that of an imaging apparatus having a conventional image sensor that holds all the data for phase difference AF in one frame. be able to. That is, there are effects of reducing the amount of calculation when calculating the phase difference amount, reducing the occupation rate of the memory space, reducing the amount of data to be read / written, and the power consumption associated therewith.

--- Second Embodiment ---
The imaging apparatus 100 according to the second embodiment of the present invention has the same configuration as the imaging apparatus 100 according to the first embodiment. In the imaging apparatus 100 according to the first embodiment, the determination unit 4 performs threshold determination for each input line of image data captured by the image sensor 2. In the imaging apparatus 100 according to the present embodiment, as shown in FIG. 8, the phase difference AF pixel arrangement area excluding the imaging pixel arrangement area in the partial image data of one line such as the line N or the line M is determined. For example, the area is divided into a plurality of areas including areas W, X, Y, and Z, and the determination unit 4 performs threshold determination for each of the divided areas.

  The one-line partial image data described above includes a plurality of sets of phase difference AF data, and each set of phase difference AF data includes the position of each of the plurality of regions including the areas W, X, Y, and Z described above. Corresponds to phase difference AF data. Therefore, the phase difference AF pixels corresponding to each set of phase difference AF data are arranged across a plurality of columns on a two-dimensional plane in which a plurality of pixels are arranged in the image sensor 2. The determination unit 4 extracts the phase difference AF data for each set, and determines whether the extracted phase difference AF data for each set satisfies a predetermined condition by threshold determination. By doing in this way, in the imaging apparatus 100 according to the present embodiment, the amount of data for phase difference AF held in a smaller unit than the imaging apparatus 100 according to the first embodiment is adjusted, and the phase difference AF is adjusted. Arithmetic can be performed.

--- Third embodiment ---
An imaging apparatus 100 according to the third embodiment of the present invention has the configuration shown in FIG. A description of blocks having the same configuration as that of the imaging apparatus 100 according to the first embodiment is omitted. The phase difference AF data frame adding unit 15 in the control device 110 frames the phase difference AF data arranged at the same pixel position for the phase difference AF data recorded in the phase difference AF data storage unit 6. Addition processing is performed every time. The phase difference AF data subjected to the addition processing is used in the phase difference calculation unit 8 by the phase difference calculation unit 8.

  That is, when the recording unit 5 records the phase difference AF data extracted by the determination unit 4 in the phase difference AF data storage unit 6, the phase difference AF data frame addition unit 15 performs the phase difference AF use. The addition data obtained by adding the data to the phase difference AF data extracted by the determination unit 4 until the previous time is newly recorded in the phase difference AF data storage unit 6 as phase difference AF data. In this way, even if only one phase difference AF data that is insufficient for phase difference calculation can be obtained in one frame, the phase difference AF sufficient for phase difference calculation can be obtained by adding data of a plurality of frames. Data can be obtained.

  The phase difference AF data is written (recorded) in the phase difference AF data storage unit 6 by the determination unit 4 and the recording unit 5 of the phase difference AF data extraction unit 3 based on the relationship between the large threshold value and the small threshold value. Get control. As a result of the determination by the determination unit 4 regarding the writing control of the first frame, the phase difference AF data of the line or area determined to satisfy the predetermined condition is stored not only in the first frame but also in the subsequent frames. Part 6 is recorded. As a result of the determination by the determination unit 4 regarding the writing control of the first frame, the phase difference AF data of the line or area determined not to satisfy the predetermined condition is not only the first frame but also the subsequent frame. It is not recorded in the storage unit 6. In this way, by sharing the determination result for the write control of the first frame among a plurality of frames, the data write amount, the data read amount, and the data calculation amount to the phase difference AF data storage unit 6 can be reduced.

  An initialization process of control for sharing a determination result among a plurality of frames, that is, an initialization process of phase difference AF data recorded in the phase difference AF data storage unit 6 is performed by an initialization process unit (not shown) in the control device 110. Is done by. Such initialization processing by the initialization processing unit may be performed with a user arbitrary timing as a trigger, or may be performed with a timing when a scene is changed by scene detection of image data as a trigger. It may be performed with the case where the amount of motion of the electric motion detection exceeds a certain value as a trigger, or may be performed with the result of determining the evaluation value from the gyro sensor as a trigger. The trigger may be generated from the result of the maximum value and the minimum value of the contrast of the input data and the threshold value.

  FIG. 10 is an AF control flowchart by the control device 110 of the imaging apparatus 100 according to the present embodiment. FIG. 10A shows a phase difference AF data recording flow of the phase difference AF data extracting unit 3 for recording the phase difference AF data in the phase difference AF data storage unit 6. Processing similar to that of the embodiment is omitted. FIG. 10B is a diagram showing an addition processing flow for adding the phase difference AF data arranged at the same pixel position for each frame, which is performed by the phase difference AF data frame addition unit 15 in the control device 110. . This addition processing is performed every time processing step S85 is completed.

  Since processing steps S10 to S50 in FIG. 10A are the same as those in the first embodiment, processing after an affirmative determination in processing step S60 will be described. After an affirmative determination is made in processing step S60, in the second and subsequent frames, in processing step S70, the input line data of the line recorded in the phase difference AF data temporary storage unit 7 in the first frame is extracted from the phase difference AF data. The input line data is input to the output unit 3, and the input line data is recorded in the phase difference AF data temporary storage unit 7 in processing step S75.

  In processing step S80, it is determined whether or not the processing target line is a line that was Y in processing step S40. In the case of affirmative determination, after being written (recorded) in the phase difference AF data storage unit 6 in processing step S85, the processing in processing step S90 is performed, and in the case of negative determination, the processing in processing step S90 is performed immediately. .

  In processing step S90, the phase difference AF data extraction unit 3 determines whether or not the input line data is the last line of one frame meaning one image, and the input line is the last line of one frame. If not, the process proceeds to process step S70, and the next line input is performed. If the input line is the last line of one frame, the process in process step S95 is performed. In process step S95, when a sea change is detected, an affirmative determination is made and the process after process step S10 is performed. When no sea change is detected, a negative determination is made and the process in process step S10 is performed. In other words, the processing after processing step S70 is performed.

  As described above, the addition processing shown in FIG. 10B is performed by the phase difference AF data frame addition unit 15 every time processing step S85 is completed. In processing step S210 for the second and subsequent frames, the phase difference AF data recorded in the phase difference AF data storage unit 6 up to the previous frame and the phase difference AF data of the current frame are added. In the processing step S220, the addition result is written back (recorded) in the phase difference AF data storage unit 6, and the process proceeds to the next frame.

---- Modified example ---
(1) In the embodiment described above, the determination unit 4 included in the phase difference AF data extraction unit 3 of the control device 110 determines the maximum value of the phase difference AF data extracted from the input line data for one line. Is larger than the large threshold value and the minimum value is smaller than the small threshold value, it is determined that the data for phase difference AF for one line has data necessary for calculating the phase difference amount. However, when the difference between the maximum value and the minimum value of the phase difference AF data extracted from the input line data for one line is larger than a predetermined threshold, the determination unit 4 determines that the phase difference AF data for one line is It is good also as determining with having data required for phase difference calculation.

(2) In the above-described embodiment, the imaging pixel 210 includes a plurality of imaging pixels 210 illustrated in FIG. 2 and a plurality of pairs of focus detection pixels illustrated in FIG. 3, that is, a plurality of pairs of phase difference AF pixels 213 and 214. The image sensor 2 in which a plurality of pixels are arranged in a two-dimensional plane outputs image data as input line data to the phase difference AF data extraction unit 3 of the control device 110 for each row of the two-dimensional plane. . In this case, the determination unit 4 periodically determines whether or not the phase difference AF data extracted from the input line data satisfies a predetermined condition. The predetermined predetermined timing indicates that the input line data for one line is the image sensor 2. Is the output timing.

  However, the periodic predetermined timing at which the determination unit 4 determines whether or not the phase difference AF data extracted from the input line data satisfies the predetermined condition is that the input line data for a plurality of lines is received by the image sensor 2 in total. It may be the output timing. By doing so, the phase difference AF pixel arrangement region is not limited to the case where the pixel arrangement region for phase difference AF extends in the horizontal direction of the image sensor 2 as shown in FIGS. The present invention can also be applied to the case where the vertical direction is crossed.

  Alternatively, the image sensor 2 may output the image data as input line data for a plurality of lines to the phase difference AF data extraction unit 3 of the control device 110 for each of a plurality of rows on the two-dimensional plane. In this case, the determination unit 4 periodically determines whether or not the phase difference AF data extracted from the input line data satisfies a predetermined condition. The predetermined predetermined timing is that the input line data for the plurality of lines is the image sensor 2. Is the output timing.

(3) In the above-described embodiment, the determination unit 4 included in the phase difference AF data extraction unit 3 of the control device 110 determines the maximum value of the phase difference AF data extracted from the input line data for one line. Is less than the large threshold value, it is determined that the data for phase difference AF for one line does not have data necessary for calculating the phase difference amount. When the minimum value of the phase difference AF data extracted from the input line data for one line is equal to or greater than the small threshold value, the determination unit 4 has the data necessary for calculating the phase difference amount. It is good also as determining not to do.

  The large threshold value and the small threshold value described above may be determined according to imaging data output from the imaging pixel 210. For example, it can be determined according to the contrast of the entire plurality of frame images obtained in the past.

  Further, when the frame image data is output, it is assumed that the determination unit 4 determines that the phase difference AF data does not have data necessary for calculating the phase difference amount in all lines. At this time, the threshold value control may be relaxed by lowering the large threshold value and raising the small threshold value at the time of determination processing by the determination unit 4 for the phase difference AF data when the next frame image data is output.

(4) As shown in FIG. 11, a computer program product for the control device 110 of the imaging device 100 to execute the imaging processing in each of the above-described embodiments and modifications is a program storage medium 2000 (for example, a CD-ROM). ) Or a communication network 4000 (for example, the Internet), the data is transmitted to the data processing apparatus 1000 (for example, a PC) through a data signal. The imaging apparatus 100 reads an imaging processing program from the data processing apparatus 1000 via communication.

  The data processing apparatus 1000 is provided with an imaging processing program via the program storage medium 2000. Alternatively, the imaging processing program may be provided in the following supply form. The data processing apparatus 1000 has a connection function with the communication network 4000. The server 3000 is a computer that provides an imaging processing program, and stores the imaging processing program in a storage medium such as a hard disk. The server 3000 places the imaging processing program read from the hard disk on a carrier wave as a data signal and transfers it to the data processing apparatus 1000 via the communication network 4000. As described above, the imaging processing program is preferably supplied by various forms of computer program products including provision as data signals via the program storage medium 2000 and the communication network 4000.

1 optical system, 2 image sensor, 3 phase difference AF data extraction unit, 4 determination unit,
5 recording unit, 6 phase difference AF data storage unit, 7 phase difference AF data temporary storage unit,
8 phase difference amount calculation unit, 9 optical system control unit, 10 interpolation data generation unit,
11 image generation recording unit, 12 display image generation unit, 13 display unit, 14 recording medium,
15 Phase difference AF data frame adder, 50, 60 Phase difference AF data,
71 Light flux for photographing, 73, 74 Light flux for phase difference AF, 93, 94, 95 Pupil area,
100 imaging device, 110 control device,
210 pixels for imaging, 231 and 214 pixels for phase difference AF,
410 microlens, 411, 413, 414 photoelectric conversion unit,
1000 data processing device, 2000 program storage medium,
3000 servers, 4000 communication networks

Claims (10)

A plurality of focus detection data corresponding to a pair of phase difference AF optical images of the subject formed by a pair of focus detection light beams coming from the subject through a pair of pupil regions of the optical system A plurality of imaging pixels that output imaging data corresponding to an imaging optical image formed by an imaging light beam that comes from the subject through the optical system through the optical system. An image sensor in which pixels are arranged in a two-dimensional plane, and outputs image data including the focus detection data and the imaging data;
If the focus detection data is included in the partial image data in the image data output by the image sensor by the predetermined periodic timing of the image data, the focus detection is performed from the partial image data. Determination means for extracting the data for use and determining whether or not the extracted focus detection data satisfies a predetermined condition;
When it is determined that the focus detection data extracted by the determination unit satisfies the predetermined condition, the focus detection data extracted by the determination unit is recorded in a storage unit, and is extracted by the determination unit. A recording unit that does not record the focus detection data extracted by the determination unit in the storage unit when it is determined that the extracted focus detection data does not satisfy the predetermined condition;
Calculation means for calculating a phase difference amount between the pair of focus detection optical images based on the focus detection data recorded in the storage unit by the recording means;
Optical system control means for adjusting the focal position of the optical system by controlling the optical system based on the phase difference amount calculated by the calculation means;
An imaging apparatus comprising: an image generation unit configured to generate an image based on the imaging data. The imaging device according to claim 1,
The predetermined condition is that the maximum value of the focus detection data extracted by the determination unit is greater than a first threshold value, and the minimum value of the focus detection data extracted by the determination unit is the first value. An imaging apparatus characterized by being smaller than a second threshold value smaller than the threshold value. The imaging device according to claim 1,
The imaging apparatus according to claim 1, wherein the predetermined condition is that a difference between a maximum value and a minimum value of the focus detection data extracted by the determination unit is larger than a predetermined threshold. The imaging device according to any one of claims 1 to 3,
The image sensor outputs the image data for each row of the two-dimensional plane;
The imaging apparatus according to claim 1, wherein the predetermined timing is a timing at which the partial image data corresponding to each row of the two-dimensional plane is output by the image sensor. The imaging device according to any one of claims 1 to 3,
The image sensor outputs the image data for each of a plurality of rows of the two-dimensional plane;
The imaging apparatus according to claim 1, wherein the predetermined timing is a timing at which the partial image data corresponding to the plurality of rows of the two-dimensional plane is output by the image sensor. The imaging device according to any one of claims 1 to 3,
The image sensor outputs the image data for each row of the two-dimensional plane;
The predetermined timing is a timing at which the partial image data corresponding to a plurality of rows on the two-dimensional plane is output by the image sensor by the image sensor outputting the image data for each row. An imaging device. In the imaging device according to any one of claims 1 to 6,
The partial image data includes a plurality of sets of the focus detection data,
The focus detection data of each set corresponds to a plurality of columns of the two-dimensional plane,
The determination unit extracts the focus detection data of each set, and determines whether the extracted focus detection data of each set satisfies the predetermined condition. In the imaging device according to any one of claims 1 to 7,
The image sensor periodically outputs the image data,
The recording unit records the focus detection data in the partial image data output by the image sensor in the past when recording the focus detection data extracted by the determination unit in the storage unit. An addition apparatus obtained by adding to the focus detection data extracted in step (b) is newly recorded in the storage unit as the focus detection data. In the imaging device according to any one of claims 1 to 8,
The image sensor periodically outputs the image data,
The determination device changes the predetermined condition when the image sensor outputs the image data after determining that the extracted focus detection data does not satisfy the predetermined condition. A plurality of focus detection data corresponding to a pair of focus detection optical images of the subject formed by a pair of focus detection light beams coming from the subject through a pair of pupil regions of the optical system; A plurality of pixels including a focus detection pixel and a plurality of imaging pixels that output imaging data corresponding to an imaging optical image formed by an imaging light flux that has passed through the optical system from the subject. Are arranged in a two-dimensional plane, and the image data is output from an image sensor that outputs image data including the focus detection data and the imaging data. If the partial image data output by the image sensor by the timing includes the focus detection data, the focus detection data is extracted from the focus detection data. And the focus detection data is a determination process of determining whether a predetermined condition is satisfied extracted,
When it is determined that the focus detection data extracted by the determination process satisfies the predetermined condition, the focus detection data extracted by the determination process is recorded in a storage unit, and is extracted by the determination process. When it is determined that the extracted focus detection data does not satisfy the predetermined condition, a recording process that does not record the focus detection data extracted by the determination process in the storage unit;
A calculation process for calculating a phase difference amount between the pair of focus detection optical images based on the focus detection data recorded in the storage unit by the recording process;
An optical system control process for outputting the phase difference amount calculated by the calculation process to the optical system control section so that the optical system control section controls the optical system to adjust the focal position of the optical system; ,
An imaging processing program causing a computer to execute an image generation process for generating an image based on the imaging data.

Images