JP2014023114A - Imaging device and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To solve a problem that an evaluation value cannot be calculated appropriately when dividing and reading output from an imaging element.SOLUTION: The imaging device includes: an imaging element having a plurality of first photoelectric conversion elements and a plurality of second photoelectric conversion elements; a first reading circuit for reading output of the plurality of first photoelectric conversion elements to output a first partial image data which is image data of the image corresponding to the portion to which the plurality of first photoelectric conversion elements are provided; a second reading circuit for reading output of the plurality of second photoelectric conversion elements to output a second partial image data which is image data of the image corresponding to the portion to which the plurality of second photoelectric conversion elements are provided; a first calculation section for calculating a first evaluation value from the first partial image data outputted from the first reading circuit; and a second calculation section for calculating a second evaluation value from a second partial image data output from the second reading circuit.

Description

  The present invention relates to an imaging apparatus and a program.

A CCD image sensor having a plurality of signal output paths is known (for example, see Patent Document 1).
[Prior art documents]
[Patent Literature]
[Patent Document 1] Japanese Patent Application Laid-Open No. 2008-104017

  When the outputs of a plurality of photoelectric conversion elements included in the imaging element are read out separately, there is a problem that the evaluation value cannot be calculated appropriately.

  In the first aspect of the present invention, an imaging apparatus reads an output of a plurality of first photoelectric conversion elements and an imaging element having a plurality of first photoelectric conversion elements and a plurality of second photoelectric conversion elements. A first readout circuit that outputs first partial image data, which is image data of an image corresponding to a portion provided with the first photoelectric conversion element, and outputs of a plurality of second photoelectric conversion elements are read out, and a plurality of second photoelectric conversion elements are read out. From a second readout circuit that outputs second partial image data that is image data of an image corresponding to a portion provided in the photoelectric conversion element, and a first evaluation value based on the first partial image data output from the first readout circuit And a second calculation unit that calculates a second evaluation value from the second partial image data output from the second readout circuit.

  In the second aspect of the present invention, the imaging device reads the outputs of the plurality of first photoelectric conversion elements included in the imaging unit, and image data of an image corresponding to a portion provided with the plurality of first photoelectric conversion elements. The first readout circuit that outputs the first image data and the output of the plurality of second photoelectric conversion elements included in the imaging unit, and the image corresponding to the portion provided in the plurality of second photoelectric conversion elements A second readout circuit that outputs second image data that is data, a first calculator that calculates a first evaluation value from the first image data output from the first readout circuit, and an output from the second readout circuit A second calculation unit that calculates a second evaluation value from the second image data, the first calculation unit outputs the first evaluation value to the second calculation unit, and the second calculation unit outputs the second evaluation value. Output to the first calculator.

  In the third aspect of the present invention, an imaging device includes an imaging element having a plurality of first photoelectric conversion element arrays and a plurality of second photoelectric conversion element arrays that are successively arranged in the first direction, and a plurality of timings. An exposure controller that sequentially exposes the first photoelectric conversion element array and the plurality of second photoelectric conversion element arrays in the first direction, and the plurality of first photoelectric conversion element arrays from the first timing. The first partial image data output from the plurality of first photoelectric conversion element arrays and the second photoelectric conversion element arrays sequentially exposed from the second timing following the first timing are sequentially exposed at And an output control unit that outputs the second partial image data output from the plurality of second photoelectric conversion element arrays as image data of one first image.

  According to a fourth aspect of the present invention, there is provided a program for an imaging apparatus having an imaging element having a plurality of first photoelectric conversion elements and a plurality of second photoelectric conversion elements, wherein the first readout circuit includes a plurality of first photoelectric conversion elements. A step of reading out the output of the photoelectric conversion element and outputting first partial image data which is image data of an image corresponding to a portion provided with the plurality of first photoelectric conversion elements; Reading out the outputs of the two photoelectric conversion elements and outputting second partial image data that is image data corresponding to the portions provided in the plurality of second photoelectric conversion elements; Calculating a first evaluation value from the first partial image data output from the readout circuit; and causing the second calculation unit to calculate a second evaluation value from the second partial image data output from the second readout circuit. And a step of causing issued computer.

  In the fifth aspect of the present invention, the first readout circuit is caused to read out the outputs of the plurality of first photoelectric conversion elements included in the imaging unit, and corresponds to a portion provided with the plurality of first photoelectric conversion elements. A step of outputting first image data which is image data of an image; and a second readout circuit for reading out outputs of a plurality of second photoelectric conversion elements included in the imaging unit and providing the plurality of second photoelectric conversion elements. Outputting second image data that is image data of an image corresponding to the given portion; causing the first calculation unit to calculate a first evaluation value from the first image data output from the first readout circuit; A step of causing the second calculation unit to calculate a second evaluation value from the second image data output from the second readout circuit; a step of outputting the first evaluation value to the second calculation unit; and a second evaluation value Output to the first calculation unit To execute that the steps on a computer.

  According to a sixth aspect of the present invention, there is provided a program for an imaging apparatus having an imaging element having a plurality of first photoelectric conversion element arrays and a plurality of second photoelectric conversion element arrays that are successively arranged in the first direction. Then, from each of the plurality of timings, an exposure control step of sequentially exposing the plurality of first photoelectric conversion element arrays and the plurality of second photoelectric conversion element arrays in the first direction, and a plurality of first photoelectric conversion element arrays from the first timing. First partial image data output from a plurality of first photoelectric conversion elements by sequentially exposing with one photoelectric conversion element, and a plurality of second photoelectric conversion elements from a second timing following the first timing The computer executes an output control step of outputting the second partial image data output from the plurality of second photoelectric conversion element arrays as image data of one first image by sequentially exposing in rows. That.

  It should be noted that the above summary of the invention does not enumerate all the necessary features of the present invention. In addition, a sub-combination of these feature groups can also be an invention.

1 shows an example of a block configuration of a camera 10 according to an embodiment. An example of a block configuration of the image sensor 132 and the image processing engine unit 135 is schematically shown together with a data input / output relationship. An example of an operation flow of the image processing engine unit 135 when imaging is instructed is shown. An example of the operation flow of the image processing engine unit 135 when an image is evaluated is shown. An example of processing information concerning AF evaluation is shown typically. An example of processing information relating to face detection is shown. An example of an operation flow in the image processing engine unit 135 when performing image processing is shown. An example of timing at which each process is performed in the image processing engine unit 135 is schematically shown. A modification of the transfer unit that transfers image data to the image processing engine unit 135 is shown. The part which the output control part 930 outputs additionally is shown typically. An output sequence of column data output from the output control unit 930 to the image processing engine unit 135 is schematically shown. The modification of the operation | movement flow in the image processing engine part 135 in the case of performing image processing is shown. An example of a block configuration when transferring from the image sensor 132 to the image processing engine unit 135 in another transfer mode is schematically shown. An example of readout control of the image sensor 132 and output destination switching control is schematically shown. The modification of transfer control is shown.

  Hereinafter, the present invention will be described through embodiments of the invention, but the following embodiments do not limit the invention according to the claims. Moreover, not all the combinations of features described in the embodiments are essential to the solving means of the invention.

  FIG. 1 shows an example of a block configuration of a camera 10 according to an embodiment. This figure shows a block configuration of the camera 10 with the lens unit 120 mounted. The camera 10 is a single-lens reflex camera as an example of an imaging device.

  The lens unit 120 includes a lens mount having a lens mount contact 121. The camera body 130 includes a camera mount having a camera mount contact 131. When the lens mount and the camera mount are engaged and the lens unit 120 and the camera body 130 are integrated, the lens mount contact 121 and the camera mount contact 131 are connected. The lens MPU 123 is connected to the camera MPU 140 via the lens mount contact 121 and the camera mount contact 131, and controls the lens unit 120 in cooperation with each other while communicating with each other.

  The lens unit 120 includes a lens group 122, a lens driving unit 124, and a lens MPU 123. The subject light passes through the lens group 122 as an optical system of the lens unit 120 along the optical axis and enters the camera body 130. The main mirror 145 can take an oblique state where the main mirror 145 is obliquely provided in the subject light flux centered on the optical axis of the lens group 122 and a retreat state where the main mirror 145 is retracted from the subject light flux.

  When the main mirror 145 is in an oblique state, the main mirror 145 reflects a part of the subject light flux that has passed through the lens group 122. Specifically, the region near the optical axis of the main mirror 145 in the oblique state is formed as a half mirror. Part of the subject luminous flux incident on the region near the optical axis is transmitted and the other part is reflected. The subject luminous flux reflected by the main mirror 145 is guided to the finder unit 147 and observed by the user. The user can check the composition and the like through the finder unit 147. The finder unit 147 may include a display device that presents various types of information including information indicating the setting state of the imaging operation to the user, along with the subject image based on the subject luminous flux.

  Part of the subject luminous flux that has passed through the region near the optical axis of the main mirror 145 is reflected by the sub mirror 146 and guided to the AF unit 142. The AF unit 142 includes a plurality of photoelectric conversion element arrays that receive a subject light beam. The photoelectric conversion element array outputs a signal with a phase match when in a focused state, and outputs a signal with a phase shift when in a front pin state or a rear pin state. The amount of phase shift corresponds to the amount of shift from the focus state. The AF unit 142 detects a phase difference by performing a correlation operation on the output of the photoelectric conversion element array, and outputs a phase difference signal indicating the phase difference to the camera MPU 140.

  The focus state of the lens group 122 is adjusted using the phase difference signal from the AF unit 142 under the control of the camera MPU 140 and the like. For example, the target position of the focus lens included in the lens group 122 is determined by the camera MPU 140 based on the focus state detected from the phase difference signal, and the position of the focus lens is controlled by the control of the lens MPU 123 toward the determined target position. Is done. Specifically, the lens MPU 123 controls the lens driving unit 124 including a focus lens motor as an example, and moves the focus lens constituting the lens group 122. As described above, when the main mirror 145 is down and in the inclined state, the focus state of the lens group 122 is detected by the phase difference detection method, and the focus adjustment is performed. The AF unit 142 is provided with photoelectric conversion element arrays at a plurality of positions respectively corresponding to the plurality of regions in order to adjust the focus state in each of the plurality of regions in the subject image.

  The photometric element 144 receives a part of the light beam guided to the optical viewfinder. The luminance information of the subject detected by the photoelectric conversion element included in the photometric element 144 is output to the camera MPU 140. The camera MPU 140 calculates an AE evaluation value based on the luminance information acquired from the photometric element 144 and uses it for exposure control.

  When the main mirror 145 retracts from the subject light beam, the sub mirror 146 retracts from the subject light beam in conjunction with the main mirror 145. A focal plane shutter 143 as an example of a mechanical shutter is provided on the lens group 122 side of the image sensor 132. When the main mirror 145 is in the retracted state and the focal plane shutter 143 is in the open state, the subject light flux that has passed through the lens group 122 is incident on the light receiving surface of the image sensor 132.

  The imaging element 132 is an example of an imaging unit. The image sensor 132 is a single image sensor. The image sensor 132 captures an image of the subject using the subject light flux that has passed through the lens group 122. Examples of the image sensor 132 include a solid-state image sensor such as a CCD sensor or a CMOS sensor. The image sensor 132 has a plurality of photoelectric conversion elements that receive the subject light flux. The plurality of photoelectric conversion elements output signals corresponding to the amount of accumulated charge generated in each of the photoelectric conversion elements. The image sensor 132 is driven by the drive unit 148 that has received an instruction from the camera MPU 140.

  An analog image signal obtained by photoelectric conversion by each photoelectric conversion element is AD-converted by an AD conversion element provided in the image sensor 132 and output to the image processing engine unit 135 as a digital image signal. The imaging element 132 has a plurality of first photoelectric conversion elements and a plurality of second photoelectric conversion elements. The plurality of first photoelectric conversion elements included in the imaging element 132 are read independently of the plurality of second photoelectric conversion elements, and the respective digital image signals are output to the image processing engine unit 135 in parallel. The image processing engine unit 135 collectively processes partial image data output from each of the plurality of first photoelectric conversion elements and the plurality of second photoelectric conversion elements as one image.

  The image processing engine unit 135 mainly has an image processing function. The image processing engine unit 135 uses at least a part of the memory area of the RAM 136 as an example of the volatile memory as a buffer area for temporarily storing the image data, and performs various processing on the image data stored in the RAM 136. The image processing is performed. Examples of the image processing performed by the image processing engine unit 135 include white balance correction, color interpolation processing, color correction, gamma correction, contour enhancement processing, and image data compression processing. When the image sensor 132 continuously captures images, sequentially output image data is sequentially stored in the buffer area. A plurality of image data obtained by continuously capturing images by the image sensor 132 is sequentially stored in the buffer area as image data of continuous still images or image data of each image constituting the moving image. The RAM 136 also functions as a frame memory that temporarily stores frames when the image processing engine unit 135 processes moving image data.

  The image processing engine unit 135 performs processing for generating image data for recording, processing for generating image data for display, processing for automatic focus adjustment (AF), processing for automatic exposure adjustment (AE), white, and the like. Responsible for image processing including balancing processing and face detection processing. The image processing engine unit 135 calculates various evaluation values such as a contrast evaluation value for AF processing, a white balance evaluation value, and evaluation information for face detection, and performs various controls based on the calculated evaluation information. . For example, the image processing engine unit 135 detects the contrast amount from the image data and supplies it to the camera MPU 140. For example, the image processing engine unit 135 detects the contrast amount from each of a plurality of image data obtained by imaging with the focus lens positioned at different positions in the optical axis direction. The camera MPU 140 adjusts the focus state of the lens group 122 based on the detected contrast amount and the position of the focus lens. For example, the camera MPU 140 determines the target position of the focus lens so as to increase the amount of contrast, and causes the lens MPU 123 to control the position of the focus lens toward the determined target position. As described above, when the main mirror 145 is up and in the retracted state, the focus state of the lens group 122 is detected by the contrast detection method, and the focus adjustment is performed. As described above, the camera MPU 140 performs the focus adjustment of the lens group 122 in cooperation with the image processing engine unit 135 and the lens MPU 123.

  When recording image data, the image processing engine unit 135 converts the image data into image data in a standardized image format. For example, the image processing engine unit 135 performs compression processing for generating still image data obtained by encoding still image data in an encoding format compliant with a standard such as JPEG. In addition, the image processing engine unit 135 converts a plurality of frames into QuickTime, H.264, and the like. H.264, MPEG2, Motion JPEG, etc. A compression process for generating moving image data encoded by an encoding method compliant with a standard such as JPEG is performed.

  The image processing engine unit 135 outputs and records the generated image data such as still image data and moving image data to an external memory 180a and an external memory 180b as an example of a nonvolatile recording medium. For example, the image processing engine unit 135 records a still image file in the external memory 180a and records a moving image file in the external memory 180b. An example of the external memory 180 is a semiconductor memory such as a flash memory. Specifically, as the external memory 180, various memory cards such as an SD memory card, a CF storage card, and an XQD memory card can be exemplified. The image data stored in the RAM 136 is transferred to the external memory 180 through the recording medium IF 150. Further, the image data recorded in the external memory 180 is transferred to the RAM 136 through the recording medium IF 150 and used for processing such as display. Examples of the recording medium IF 150 include a card controller that controls access to the above-described memory card.

  The image processing engine unit 135 generates display image data in parallel with the generation of recording image data. For example, the image processing engine unit 135 generates image data for display to be displayed on the display unit 138 such as a liquid crystal display device during a so-called live view operation. Further, at the time of image reproduction, the image processing engine unit 135 generates image data for display from the image data read from the external memory 180. The generated display image data is converted into an analog signal under the control of the display control unit 137 and displayed on the display unit 138. In addition to image display based on image data obtained by imaging, various menu items and touch panel objects related to various settings of the camera 10 are also displayed without image display based on the image data. The image is displayed on the display unit 138 by being superposed under the control of the display control unit 137.

  The external device IF 152 is responsible for communication with the external device according to the control of the camera MPU 140. The image data recorded in the external memory 180 is transferred to the external device through the external device IF 152. In addition, image data acquired by communication from an external device through the external device IF 152 is recorded in the external memory 180. The external device IF 152 may communicate with the external device by USB communication.

  The operation input unit 141 receives an operation from the user. The operation input unit 141 includes various operation members such as a release button, a playback button, a live view switch, a moving image button, and a power switch. The camera MPU 140 detects that the operation input unit 141 has been operated, and executes an operation corresponding to the operation. For example, the camera MPU 140 controls each unit of the camera 10 so as to execute an imaging operation when a release button is pressed.

  The camera 10 is controlled directly or indirectly by the camera MPU 140 and the image processing engine unit 135, including the control described above. Constants, parameters such as variables, programs, and the like necessary for the operation of the camera 10 are stored in the system memory 139. The system memory 139 is an electrically erasable / storable nonvolatile memory, and is configured by, for example, a flash ROM, an EEPROM, or the like. The system memory 139 stores parameters, programs, and the like so that they are not lost even when the camera 10 is not operating. Parameters, programs, and the like stored in the system memory 139 are expanded in the RAM 136 and used for controlling the camera 10. The image processing engine unit 135, the RAM 136, the system memory 139, the display control unit 137, and the camera MPU 140 in the camera main body 130 are connected to each other via a connection interface 149 such as a bus and exchange various data.

  Each part of the camera main body 130, each part of the lens unit 120, and the external memory 180 are supplied with power from the power supply 190 through the power supply circuit 192 under the control of the camera MPU 140. The power source 190 may be a secondary battery such as a nickel metal hydride battery or a lithium ion battery, a non-rechargeable battery, or the like. A power source 190 as a battery is detachably attached to the camera body 130. The power source 190 may be a commercial power source.

  FIG. 2 schematically illustrates an example of a block configuration of the image sensor 132 and the image processing engine unit 135 together with a data input / output relationship. The image processing engine unit 135 includes a first image processing engine 210 and a second image processing engine 220. The imaging element 132 includes a first photoelectric conversion element 201a, a first photoelectric conversion element 201b,..., A second photoelectric conversion element 202a, a second photoelectric conversion element 202b, the first drive circuit 21, and a second drive circuit 22. Have Note that the plurality of first photoelectric conversion elements 201a, the first photoelectric conversion elements 201b,... May be collectively referred to as the plurality of first photoelectric conversion elements 201 or simply the first photoelectric conversion elements 201. In addition, the plurality of second photoelectric conversion elements 202a, the second photoelectric conversion elements 202b,... May be collectively referred to as the plurality of second photoelectric conversion elements 202 or simply the second photoelectric conversion elements 202. The first photoelectric conversion element and the second photoelectric conversion element 202 are disposed on substantially the same surface of the imaging element 132. As an example, the first photoelectric conversion element and the second photoelectric conversion element 202 are provided in a matrix.

  The first photoelectric conversion element 201 and the plurality of second photoelectric conversion elements 202 each form a plurality of photoelectric conversion element arrays. The first photoelectric conversion element 201 and the second photoelectric conversion element 202 form an n-line photoelectric conversion element array as a whole. Specifically, the first photoelectric conversion element 201 forms continuous m-line photoelectric conversion element arrays L1 to Lm. In addition, the second photoelectric conversion element 202 forms continuous m-line photoelectric conversion element arrays Lm + 1 to Ln. The photoelectric conversion element arrays L1 to Lm and the photoelectric conversion element arrays Lm + 1 to Ln are provided side by side in a specific direction. One photoelectric conversion element array Lm among the plurality of photoelectric conversion element arrays formed by the first photoelectric conversion element 201 is one photoelectric conversion element among the plurality of photoelectric conversion element arrays formed by the second photoelectric conversion element 202. Adjacent to column Lm + 1. Here, n = 2m. As described above, the first photoelectric conversion element 201 and the second photoelectric conversion element 202 form a photoelectric conversion element array having the same number of lines. For the sake of simplicity, the first photoelectric conversion element 201 is provided above the second photoelectric conversion element 202, and a partial area corresponding to the first photoelectric conversion element 201 also in the image area represented by the output image data. Will be described as being located above the partial region corresponding to the second photoelectric conversion element 202.

  The output of the first photoelectric conversion element 201 is read by the first drive circuit 21 and output to the first image processing engine 210 as first partial image data that is a digital image signal. The output of the second photoelectric conversion element 202 is read by the second drive circuit 22 and output to the second image processing engine 220 as second partial image data that is a digital image signal. The first partial image data and the second partial image data are output in parallel. The first drive circuit 21 reads the output of the first photoelectric conversion element 201 and outputs first partial image data that is image data of an image corresponding to the portion where the first photoelectric conversion element 201 is provided. It is an example of a circuit. The second drive circuit 22 reads the output of the second photoelectric conversion element 202 and outputs second partial image data that is image data of an image corresponding to a portion provided in the second photoelectric conversion element 202. It is an example of a circuit.

  As an example, the first photoelectric conversion element 201 is read by starting exposure in the order of photoelectric conversion element arrays Lm,... L4, L3, L2, and L1. In addition, the second photoelectric conversion element 202 starts reading in the order of photoelectric conversion element arrays Lm + 1, Lm + 2, Lm + 3, Lm + 4. Thus, the reading direction of the first photoelectric conversion element 201 may be opposite to the reading direction of the second photoelectric conversion element 202. The drive unit 148 supplies a timing clock that controls the exposure timing and readout timing of the photoelectric conversion elements of each photoelectric conversion element array to the imaging element 132. The first drive circuit 21 and the second drive circuit 22 control the exposure and readout of the corresponding photoelectric conversion element array based on the supplied timing clock.

  The first partial image data read by the first drive circuit 21 is output to the first image processing engine 210. The second partial image data read by the second drive circuit 22 is output to the second image processing engine 220. The first image processing engine 210 calculates a first evaluation value from the first partial image data. Further, the second image processing engine 220 calculates a second evaluation value from the second partial image data. The second image processing engine 220 outputs the second evaluation value to the first image processing engine 210 and causes the first image processing engine 210 to evaluate. The first image processing engine 210 evaluates one image represented by the first partial image data and the second partial image data based on the first evaluation value and the second evaluation value. The first image processing engine 210 outputs the first evaluation value to the second image processing engine 220 and causes the second image processing engine 220 to evaluate. The second image processing engine 220 evaluates one image represented by the first partial image data and the second partial image data based on the first evaluation value and the second evaluation value. Details of functions and operations of the first image processing engine 210 and the second image processing engine 220 will be described later.

  FIG. 3 shows an example of an operation flow of the image processing engine unit 135 when imaging is instructed. This flow is started when the camera MPU 140 detects a user instruction to start a moving image capturing operation. For example, it is started when the pressing of the moving image recording button is detected.

  When this flow starts, the first image processing engine 210 acquires first partial image data from the first drive circuit 21 (step S302). In addition, the first image processing engine 210 acquires second partial image data from the second drive circuit 22 (step S352). The first partial image data and the second partial image data are RAW image data.

  Subsequently, the first image processing engine 210 performs processing related to image evaluation (step S304). The second image processing engine 220 performs processing related to image evaluation (step S354). Here, image evaluation information is exchanged between the first image processing engine 210 and the second image processing engine 220. This process will be described with reference to FIG.

  Subsequently, the first image processing engine 210 performs processing related to image processing (step S306). The second image processing engine 220 performs processing related to image processing (step S354). Here, the second partial image data is transmitted from the second image processing engine 220 to the first image processing engine 210 or the first partial image data is transmitted from the first image processing engine 210 to the second image processing engine 220. The This process will be described with reference to FIG.

  Subsequently, the first image processing engine 210 determines whether or not to end the moving image capturing operation (step S308). Further, the second image processing engine 220 determines whether or not to end the imaging operation (step S358). For example, when receiving an instruction to start a moving image capturing operation, the camera MPU 140 outputs an instruction to end the moving image capturing operation to the first image processing engine 210 and the second image processing engine 220. When the first image processing engine 210 and the second image processing engine 220 obtain an instruction to end the moving image capturing operation from the camera MPU 140, the first image processing engine 210 and the second image processing engine 220 determine to end the moving image capturing operation. When the first image processing engine 210 determines that the moving image capturing operation is not ended, the process proceeds to step S302, and the second image processing engine 220 determines that the moving image capturing operation is not ended. Then, the process proceeds to step S352. When the first image processing engine 210 and the second image processing engine 220 determine that the moving image capturing operation is not ended, the first image processing engine 210 and the second image processing engine 220 end the imaging operation.

  FIG. 4 shows an example of an operation flow related to the image processing engine unit 135 when an image is evaluated. This flow can be applied as the internal flow of steps S304 and S354 of FIG.

  When this flow starts, in step S402, the first image processing engine 210 calculates a first evaluation value from the first partial image data. Specifically, an AF evaluation value, a white balance correction evaluation value, an AE evaluation value, and person detection evaluation information are calculated as the first evaluation value. In step S452, the second image processing engine 220 calculates a second evaluation value from the second partial image data. Specifically, an evaluation value of the same type as that of the first image processing engine 210 is calculated. The AF evaluation value, the AE evaluation value, the white balance evaluation value, and the evaluation information for person detection are an evaluation value for evaluating the focus state, an evaluation value for evaluating the exposure state, and a white balance state, respectively. It is an example of face detection information for evaluating an evaluation value and the position of a person's face.

  In step S <b> 404, the first image processing engine 210 determines the type of evaluation value to be transmitted to the second image processing engine 220. In step S454, the second image processing engine 220 determines the type of evaluation value to be transmitted to the first image processing engine 210. As an example, when the current frame is an odd-numbered frame, the first image processing engine 210 determines an evaluation value other than the person detection evaluation value as an evaluation value to be transmitted to the second image processing engine 220. . Further, the first image processing engine 210 determines the evaluation value for person detection as an evaluation value to be transmitted to the second image processing engine 220 when the current frame is an even-numbered frame. On the other hand, the second image processing engine 220 determines a person detection evaluation value as an evaluation value to be transmitted to the first image processing engine 210 when the current frame is an odd-numbered frame. The second image processing engine 220 determines an evaluation value other than the evaluation value for person detection as an evaluation value to be transmitted to the first image processing engine 210 when the current frame is an even-numbered frame. This process will be described later with reference to FIGS.

  Subsequently, in step S406, the first image processing engine 210 outputs the first evaluation value of the type determined in step S404 to the second image processing engine 220. In step S456, the second image processing engine 220 outputs the second evaluation value of the type determined in step S454 to the first image processing engine 210. In this way, the first image processing engine 210 calculates the first type of evaluation value as the first evaluation value and outputs it to the second image processing engine 220. In addition, the second image processing engine 220 calculates a second type of evaluation value as the second evaluation value and outputs the second type of evaluation value to the first image processing engine 210. Specifically, the first image processing engine 210 calculates a first type of evaluation value as the first evaluation value when the image sensor 132 captures an image at the first timing, and sends the first type of evaluation value to the second image processing engine 220. The second image processing engine 220 outputs the second type of evaluation value as the second evaluation value and outputs the second evaluation value to the first image processing engine 210 when the image sensor 132 captures the image at the first timing. . The first image processing engine 210 calculates the second type of evaluation value as the first evaluation value when the image sensor 132 captures an image at the second timing following the first timing, and performs the second image processing. The second image processing engine 220 outputs to the engine 220, and the second image processing engine 220 calculates the first type of evaluation value as the second evaluation value when the image sensor 132 images at the second timing, and the first image processing engine 210. Output to.

  Subsequently, in step S408, the first image processing engine 210 evaluates the entire image. Specifically, the first image processing engine 210 performs evaluation based on the type of evaluation value received from the second image processing engine 220. For example, when the current frame is an odd-numbered frame, the first image processing engine 210 performs processing for specifying the position of a person in the entire image. In step S458, the second image processing engine 220 evaluates the entire image. Specifically, the second image processing engine 220 performs evaluation based on the type of evaluation value received from the first image processing engine 210. For example, when the current frame is an odd-numbered frame, the second image processing engine 220 calculates the total contrast amount of the entire image. Further, the second image processing engine 220 determines whether or not the subject is in focus based on the total contrast amount. As described above, the first image processing engine 210 is obtained by imaging at the first timing based on the first evaluation value and the second evaluation value when the imaging element 132 images at the first timing. One image represented by the first partial image data and the second partial image data is evaluated. When the image sensor 132 captures an image at the second timing following the first timing, the second image processing engine 220 captures an image at the second timing based on the first evaluation value and the second evaluation value. One image represented by the obtained first partial image data and second partial image data is evaluated.

  Subsequently, in step S460, the second image processing engine 220 transmits the evaluation result of step S458 to the first image processing engine 210. For example, the second image processing engine 220 transmits a determination result indicating the degree of focus to the first image processing engine 210. In step S410, the first image processing engine 210 performs control based on the evaluation result in step S408 and the evaluation result received from the second image processing engine 220. For example, the determination result of the total contrast amount and the in-focus state is transmitted to the camera MPU 140. Further, the first image processing engine 210 performs processing such as displaying a face detection frame based on the position of the identified person. The first image processing engine 210 and the second image processing engine 220 end this flow when the processes of steps S410 and s460 are completed, respectively.

  FIG. 5 schematically shows an example of processing information relating to AF evaluation. First partial image data 501 is output to the first image processing engine 210. The first image processing engine 210 applies a filter that extracts a contrast amount to each image region 510 corresponding to the focus adjustment region, with respect to the first partial image data 501. Second partial image data 502 is output to the second image processing engine 220. The second image processing engine 220 applies a filter for extracting a contrast amount to each image region 520 corresponding to the focus adjustment region with respect to the second partial image data 502. Then, the first image processing engine 210 transmits information indicating the contrast amount of each image region 510 to the second image processing engine 220. The second image processing engine 220 calculates the total contrast amount in the image region 530 based on the contrast amounts in each image region 510 and each image region 520. For example, the second image processing engine 220 calculates the total contrast amount based on the contrast amount of the image area instructed to adjust the focus. Further, the second image processing engine 220 determines whether or not it is in an in-focus state based on the total contrast amount. Then, the first image processing engine 210 acquires the determination result from the second image processing engine 220 and performs various controls based on the determination result in the first image processing engine 210. For example, the first image processing engine 210 causes the display unit 138 to display that it is in focus. In addition, the camera MPU 140 is notified that it is in focus, and the contrast AF control is stopped.

  FIG. 6 shows an example of processing information related to face detection. First partial image data 601 is output to the first image processing engine 210. The first image processing engine 210 generates reduced image data 610 obtained by reducing the first partial image data 601 as an evaluation value for face detection. The second partial image data 602 is output to the second image processing engine 220. The second image processing engine 220 generates reduced image data 620 obtained by reducing the second partial image data 602 as an evaluation value for face detection. Then, the second image processing engine 220 transmits the reduced image data 620 to the first image processing engine 210. The second image processing engine 220 generates reduced image data 630 of the entire image from the reduced image data 610 and the reduced image data 620, and performs face detection image processing on the reduced image data 630.

  As an example, the first image processing engine 210 identifies the position and size of the person's face area 640 by pattern matching or the like. Then, the first image processing engine 210 causes the display unit 138 to display a face detection frame or the like that presents the position where the face is detected to the user based on the position and size of the identified region. Further, each part of the camera 10 may be controlled so that processing for a human face is performed on the face detection area.

  FIG. 7 shows an example of an operation flow in the image processing engine unit 135 when performing image processing. This flow can be applied as the internal flow of steps S306 and S356 of FIG.

  When this flow starts, the first image processing engine 210 determines whether or not to transmit the first partial image data to the second image processing engine 220 (step S702). For example, the first image processing engine 210 determines to transmit the first partial image data to the second image processing engine 220 when the current frame is an odd-numbered frame. If it is determined that the first partial image data is to be transmitted to the second image processing engine 220, the first partial image data is transmitted to the second image processing engine 220 in step S712, and the process proceeds to step S710. If it is determined not to transmit the first partial image data to the second image processing engine 220, the process proceeds to step S704.

  The second image processing engine 220 determines whether or not to transmit the second partial image data to the first image processing engine 210 (step S752). For example, the second image processing engine 220 determines to transmit the second partial image data to the first image processing engine 210 when the current frame is an even-numbered frame. If it is determined that the second partial image data is to be transmitted to the first image processing engine 210, the second partial image data is transmitted to the first image processing engine 210 in step S762, and this flow ends. If it is determined not to transmit the second partial image data to the first image processing engine 210, the process proceeds to step S754.

  In step S <b> 704, the first image processing engine 210 generates whole image data from the first partial image data and the second partial image data transmitted from the second image processing engine 220. Subsequently, the first image processing engine 210 performs image processing on the entire image data (step S706). Then, the first image processing engine 210 performs compression processing on the entire image data after image processing (step S708). As described above, the second image processing engine 220 outputs the second partial image data to the first image processing engine 210, and the first image processing engine 210 is represented by the first partial image data and the second partial image data. Image processing is performed on image data of one image. Specifically, the second image processing engine 220 outputs the second partial image data that is RAW image data to the first image processing engine 210, and the first image processing engine 210 is the first image data that is RAW image data. Image processing including color interpolation processing and compression processing is performed on the image data of one image represented by the partial image data and the second partial image data which is the RAW image data. Then, in step S710, the first image processing engine 210 records the compressed image data obtained by the compression process in the external memory 180, and ends this flow.

  In step S754, the second image processing engine 220 generates whole image data from the second partial image data and the first partial image data transmitted from the first image processing engine 210. Subsequently, the second image processing engine 220 performs image processing on the entire image data (step S756). Here, the second image processing engine 220 performs the same type of image processing as the image processing in step S706. Then, the second image processing engine 220 performs compression processing on the entire image data after image processing (step S758). Then, the second image processing engine 220 transmits the compressed image data obtained by the compression processing to the first image processing engine 210 in step S760. As described above, the first image processing engine 210 outputs the first partial image data to the second image processing engine 220, and the second image processing engine 220 represents the second partial image data and the first partial image data. Image processing is performed on the image data of one image to be processed. Specifically, the second image processing engine 220 performs color interpolation processing and compression on the image data of one image represented by the first partial image data that is RAW image data and the second partial image data that is RAW image data. Image processing including processing is performed, and compressed image data obtained by performing the image processing is output to the first image processing engine 210.

  The first image processing engine 210 transmits the first partial image data to the second image processing engine 220 in step S712, and then records the compressed image data received from the first image processing engine 210 in the external memory 180 in step S710. To do. As described above, the first image processing engine 210 records the compressed image data in the external memory 180.

  FIG. 8 schematically illustrates an example of the timing at which each process is performed in the image processing engine unit 135. In this figure, the processing after time t0 when the reading of the Nth frame, which is an odd frame, is completed will be described.

  When the reading of the Nth frame is completed, the first image processing engine 210 and the second image processing engine 220 each calculate an evaluation value from the corresponding partial image data. Then, the first image processing engine 210 transmits an evaluation value of a type other than for face detection to the second image processing engine 220. The second image processing engine 220 transmits the face detection evaluation value to the first image processing engine 210. When the evaluation value is received, the first image processing engine 210 performs evaluation for face detection from time t1. The second image processing engine 220 performs processing such as determining the total contrast amount, determining the focus state, determining whether the exposure is appropriate, and determining white balance.

  In addition, when the reading of the Nth frame is completed, the first image processing engine 210 transmits the first partial image data to the second image processing engine 220 in parallel with the above-described processing. The second image processing engine 220 generates whole image data from the second partial image data output from the second drive circuit 22 and the first partial image data received from the first image processing engine 210 to perform image processing. And compression processing.

  When the reading of the (N + 1) th frame is completed at time t2, the first image processing engine 210 and the second image processing engine 220 each calculate an evaluation value from the corresponding partial image data. Then, the first image processing engine 210 transmits the evaluation value for face detection to the second image processing engine 220. The second image processing engine 220 transmits evaluation values of a type other than for face detection to the first image processing engine 210. When the evaluation value is received, from time t3, the first image processing engine 210 performs processing such as determining the total contrast, determining the focus state, determining whether the exposure is appropriate, and determining white balance. The second image processing engine 220 performs face detection evaluation.

  When the reading of the (N + 1) th frame is completed, the second image processing engine 220 transmits the second partial image data to the first image processing engine 210 in parallel with the above-described processing. The first image processing engine 210 generates whole image data from the first partial image data output from the first drive circuit 21 and the second partial image data received from the second image processing engine 220 to perform image processing. And compression processing. Thus, when the image sensor 132 continuously captures images at a plurality of timings, the second image processing engine 220 is calculated from the second partial image data when the image sensor 132 captures images at the first timing. The second evaluation value is output to the first image processing engine 210, and the first image processing engine 210 starts from the first partial image data when the image sensor 132 captures an image at the second timing following the first timing. The calculated first evaluation value is output to the second image processing engine 220. The second image processing engine 220 outputs the second partial image data to the first image processing engine 210 when the image sensor 132 captures an image at the first timing, and the first image processing engine 210 When the image is captured at the second timing 132 following the first timing, the first partial image data is output to the second image processing engine 220. As described above, the first image processing engine 210 and the second image processing engine 220 share the calculation of the evaluation value, the processing of the evaluation, the image processing, and the compression processing, so that the load on each engine can be equalized. . In addition, processing that takes a long time for calculation can be performed by one image processing engine.

  FIG. 9 shows a modification of the transfer unit that transfers the partial image data to the image processing engine unit 135. In addition to the functional blocks described in relation to FIGS. 1 to 8 and the like, the camera 10 of this modification includes a first buffer unit 910 and a second buffer between the first drive circuit 21 and the second drive circuit 22. A unit 920 and an output control unit 930.

  In the drawing, a block configuration between the first drive circuit 21 and the second drive circuit 22 and the image processing engine unit 135 is shown together with the image processing engine unit 135. Other functional blocks in the camera 10 have substantially the same functions and operations as the corresponding functional blocks described with reference to FIGS. Here, the description will focus on differences from the functions and operations of the functional blocks of the camera 10 described with reference to FIGS.

  The first buffer unit 910 holds a predetermined number of column data among the column data of the photoelectric conversion element columns sequentially output from the first drive circuit 21. The first buffer unit 910 holds a predetermined number of the latest column data. The second buffer unit 920 holds a predetermined number of column data among the column data of the photoelectric conversion element columns sequentially output from the second drive circuit 22. The second buffer unit 920 holds a predetermined number of the latest column data. The output control unit 930 outputs the column data of the photoelectric conversion element array held in the second buffer unit 920 to the first image processing engine 210, and the column of photoelectric conversion element arrays held in the first buffer unit 910. Data is output to the first image processing engine 210.

  The output control unit 930 includes a predetermined number of photoelectric conversion elements among the column data held by the second buffer unit 920 in addition to the column data of the second photoelectric conversion element columns in the L1 to Lm rows. The column data of the column is output to the first image processing engine 210. Specifically, the output control unit 930 outputs the second photoelectric conversion of the Lm-th row from column data of a predetermined number of column data held by the second buffer unit 920. The data is output to the first image processing engine 210 before the column data of the element column.

  In addition to the column data of the second photoelectric conversion element columns in the (Lm + 1) th to (Ln) th rows, the output control unit 930 also includes a predetermined number of photoelectrical data among the column data held by the first buffer unit 910. The row data of the conversion element row is output to the second image processing engine 220. Specifically, the output control unit 930 outputs the second photoelectric conversion of the (Lm + 1) th row from among the column data held by the first buffer unit 910 and the column data of a predetermined number of photoelectric conversion element columns. Output to the second image processing engine 220 prior to the column data of the element column. That is, image data for a predetermined number of lines is output to the first image processing engine 210 and the second image processing engine 220 in the adjacent portions of the first photoelectric conversion element 201 and the second photoelectric conversion element 202. To.

  FIG. 10 schematically shows a portion additionally output by the output control unit 930. The output control unit 930 additionally outputs column data of the photoelectric conversion element column group 1020 formed by the second photoelectric conversion elements 202 to the first image processing engine 210. The photoelectric conversion element array group 1020 includes the second photoelectric conversion elements 202 in the (Lm + 1) th row, and is formed of photoelectric conversion element arrays that are continuous in the reading direction. Further, the output control unit 930 additionally outputs column data of the photoelectric conversion element column group 1020 formed by the first photoelectric conversion elements 201 to the second image processing engine 220. The photoelectric conversion element array group 1020 includes the first photoelectric conversion elements 201 in the Lm-th row, and is formed of photoelectric conversion element arrays that are continuous in the reading direction.

  As described above, the column data of the photoelectric conversion element columns included in the photoelectric conversion element column group 1010 or the photoelectric conversion element column group 1020 is output to the first image processing engine 210 and the second image processing engine 220 in an overlapping manner. The first image processing engine 210 is represented by column data of photoelectric conversion element columns included in the photoelectric conversion element column group 1020 and column data of a plurality of photoelectric conversion element columns formed by the first photoelectric conversion element 201. Process image data for one image. Specifically, the first image processing engine 210 uses the column data of the photoelectric conversion element columns included in the photoelectric conversion element column group 1020 and the column data of the photoelectric conversion element columns L1 to Lm formed by the first photoelectric conversion element 201. A first evaluation value is calculated from image data of one represented image. The second image processing engine 220 is represented by column data of photoelectric conversion element columns included in the photoelectric conversion element column group 1010 and column data of a plurality of photoelectric conversion element columns formed by the second photoelectric conversion element 202. Process image data for one image. Specifically, the second image processing engine 220 includes column data of photoelectric conversion element columns included in the photoelectric conversion element column group 1010 and a plurality of photoelectric conversion element columns Lm + 1 to Ln formed by the second photoelectric conversion element 202. A second evaluation value is calculated from the image data of one image represented by the data. For this reason, the first image processing engine 210 and the second image processing engine 220 can apply image filters over a plurality of columns. For example, a two-dimensional image filter can be applied. Note that the number of photoelectric conversion element columns included in the photoelectric conversion element column group 1010 and the photoelectric conversion element column group 1020 is determined by the number of image filter columns applied by the first image processing engine 210 and the second image processing engine 220. May be. For example, when a 4 × 4 image filter is applied, each of the photoelectric conversion element array group 1010 and the photoelectric conversion element array group 1020 may include at least two photoelectric conversion element arrays.

  FIG. 11 schematically illustrates an output sequence of column data output from the output control unit 930 to the image processing engine unit 135. In this example, a configuration in which two photoelectric conversion element arrays are included in each of the photoelectric conversion element array group 1010 and the photoelectric conversion element array group 1020 will be described. The first buffer unit 910 includes line buffers 1110-1 to 11-5 that hold column data for 5 lines. The second buffer unit 920 includes line buffers 1120-1 to 1120-5 that hold column data for five lines.

  The line buffer 1110 stores column data in the order of the line buffer 1110-1, the line buffer 1110-2, the line buffer 1110-3,. After the column data is stored in the line buffer 1110-5, the next column data is stored in the line buffer 1110-1, and the column data is stored again in the above order. Similarly, the line buffer 1120 stores the column data in the order of the line buffer 1120-1, the line buffer 1120-2, the line buffer 1120-3..., And after the column data is stored in the line buffer 1110-5. The next column data is stored in the line buffer 1120-1. This figure shows a state at a timing when four columns of data are stored in the line buffer 1110 and the line buffer 1120 after the start of reading.

  Specifically, at the first transfer timing, the column data of the Lm row of the first photoelectric conversion element 201 is stored in the line buffer 1110-1, and the column data of the Lm + 1 row of the second photoelectric conversion element 202 is stored. Are stored in the line buffer 1120-1. At the subsequent second transfer timing, the column data of the (Lm-1) th row of the first photoelectric conversion element 201 is stored in the line buffer 1110-2, and the column data of the (Lm + 2) th row of the second photoelectric conversion element 202 is It is stored in the buffer 1120-2.

  Subsequently, at the third transfer timing, the column data of the (Lm-2) th row of the first photoelectric conversion element 201 is stored in the line buffer 1110-3, and the column data of the (Lm + 3) th row of the second photoelectric conversion element 202 is stored. Are stored in the line buffer 1120-3. At this transfer timing, the output control unit 930 outputs the column data of the (Lm−1) th row stored in the line buffer 1110-2 to the second image processing engine 220, and is stored in the line buffer 1120-2. The column data of the (Lm + 2) th row is output to the first image processing engine 210. The column data output to the first image processing engine 210 is stored in the buffer area as column data of the lowermost line in the first partial image data. Further, the column data output to the second image processing engine 220 is stored in the buffer area as column data of the uppermost line in the second partial image data.

  Subsequently, at the fourth transfer timing, the column data of the (Lm-3) th row of the first photoelectric conversion element 201 is stored in the line buffer 1110-4, and the column data of the (Lm + 4) th row of the second photoelectric conversion element 202 is stored. Are stored in the line buffer 1120-4. At this transfer timing, the output control unit 930 outputs the Lm-th column data stored in the line buffer 1110-1 to the second image processing engine 220, and stores it in the line buffer 1120-1. The column data of Lm + 1 rows is output to the first image processing engine 210. The column data output to the first image processing engine 210 is stored in the buffer area as column data of the line one line above the bottom in the first partial image data. Further, the column data output to the second image processing engine 220 is stored in the buffer area as column data of the line one line below the uppermost part in the second partial image data.

  The column data of the (Lm−4) th row of the first photoelectric conversion element 201 is stored in the line buffer 1110-5, and the column data of the (Lm + 5) th row of the second photoelectric conversion element 202 is stored in the line buffer 1120-5. . At this transfer timing, the output control unit 930 outputs the column data of the (Lm + 1) th row stored in the line buffer 1120-1 to the second image processing engine 220, and stores it in the first line stored in the line buffer 1110-1. The Lm row column data is output to the first image processing engine 210. The column data output to the first image processing engine 210 is stored in the buffer area as column data of three lines above the bottom in the first partial image data. In addition, the column data output to the second image processing engine 220 is stored in the buffer area as column data of a line three lines below the top in the second partial image data.

  Thereafter, the output control unit 930 sequentially shifts the reading position of the column data from the line buffer 1110 in synchronization with the reading from the first photoelectric conversion element 201 and the second photoelectric conversion element 202, and thereby the first image. Output to the processing engine 210. In addition, the output control unit 930 sequentially shifts the reading position of the column data from the line buffer 1120 and outputs it to the second image processing engine 220. Thus, when the transfer of the column data from the line buffer 1110 and the line buffer 1120 to the first image processing engine 210 and the second image processing engine 220 is completed, the buffer area of the first image processing engine 210 has the Lmth row. Thus, the first partial image data including the output of the photoelectric conversion element row two rows below the photoelectric conversion element row is stored. In addition, the buffer area of the second image processing engine 220 stores the second partial image data including the outputs of the photoelectric conversion element columns that are two rows higher than the (Lm + 1) th photoelectric conversion element columns.

  FIG. 12 shows a modification of the operation flow in the image processing engine unit 135 when image processing is performed. This flow can be applied as the internal flow of steps S306 and S356 of FIG. When applying this modification, in S302 of FIG. 3, the first image processing engine 210 corresponds to the first photoelectric conversion element 201 and the second photoelectric conversion element 202 belonging to the data photoelectric conversion element column group 1020. Get the image data of the part. The image data is referred to as first extended partial image data. In S <b> 352 of FIG. 3, the second image processing engine 220 acquires image data of images corresponding to the first photoelectric conversion element 201 and the second photoelectric conversion element 202 belonging to the photoelectric conversion element array group 1010. The image data is referred to as second extended partial image data.

  When this flow starts, the first image processing engine 210 performs image processing on the first extended partial image data (step S1202). In this process, the first image processing engine 210 applies a filter, performs color interpolation processing on the RAW image data, and converts the image data into YUV format image data. Various image processing such as contour enhancement processing may be performed. Then, the first image processing engine 210 performs compression processing on the first extended partial image data after the image processing (step S1204), and proceeds to step S1206. As an example, in step S1204, the first image processing engine 210 generates JPEG image data.

  In the second image processing engine 220, in step S1202, the second image processing engine 220 performs image processing on the second extended partial image data (step S1252). In this processing, the same type of image processing as that performed on the first extended partial image data in step S1202 is performed. Then, the second image processing engine 220 performs compression processing on the second expanded partial image data after the image processing (step S1254). In this processing, the same type of compression processing as that performed on the first extended partial image data in step S1202 is performed. Then, the second image processing engine 220 transmits the compressed partial image data obtained by compressing the second extended partial image data in step S1254 to the first image processing engine 210 (step S1256).

  In the first image processing engine 210, the compressed partial image data obtained by compressing the first extended partial image data in step S1206 and the compressed partial image data received from the second image processing engine 220 are merged. In this process, the first image processing engine 210 generates JPEG format data. Then, the first image processing engine 210 records the compressed image data of the entire image obtained by merging in the external memory 180, and ends this flow.

  According to this modification, the first image processing engine 210 and the second image processing engine 220 also use a two-dimensional filter in the image region at the boundary between the first photoelectric conversion element 201 and the second photoelectric conversion element 202. Image processing can be performed. Therefore, the first image processing engine 210 and the second image processing engine 220 can perform various image processing such as color interpolation processing and conversion processing to YUV signals. Therefore, the first image processing engine 210 and the second image processing engine 220 can share image processing. Further, as described in this flow, the second image processing engine 220 transmits the partial image data after compression to the first image processing engine 210, so that the amount of data transferred to the first image processing engine 210 is reduced. be able to.

  In this figure, the case where the compressed partial image data is transmitted to the first image processing engine 210 has been described. However, partial image data in YUV format may be transmitted to the first image processing engine 210. Then, the first image processing engine 210 may merge the partial image data in the YUV format. Note that the image data of the entire image obtained by merging may be compressed by the first image processing engine 210 and recorded in the external memory 180. The first image processing engine 210 may generate image data for display from the image data of the entire image obtained by merging and display it on the display unit 138 via the display control unit 137.

  When generating display image data and recording compressed image data, the first image processing engine 210 is responsible for generating compressed image data, and the second image processing engine 220 is responsible for generating image data for display. May be. When the second image processing engine 220 is responsible for generating image data for display, the image data obtained by the image processing in step S1202 in the figure is transmitted to the second image processing engine 220, and the second image processing engine 220 At 220, image data for the entire display image may be generated. In this case, the first image processing engine 210 may generate image data that has been subjected to pixel thinning processing in accordance with the number of display pixels of the display unit 138 and transmit the image data to the second image processing engine 220.

  FIG. 13 schematically shows an example of a block configuration for transferring from the image sensor 132 to the image processing engine unit 135 in another transfer mode. The camera 10 of this example further includes an output switching unit 1300 in addition to the functional blocks described with reference to FIGS. In this transfer mode, the first partial image data and the second partial image data output by starting exposure at the same timing under the control of the output switching unit 1300 are the partial image data of different frames as the image processing engine unit 135. Is output. The output switching unit 1300 switches the output destination of the first partial image data between the first image processing engine 210 and the second image processing engine 220 for each frame. The output switching unit 1300 switches the output destination of the second partial image data between the first image processing engine 210 and the second image processing engine 220 for each frame.

  FIG. 14 schematically shows an example of reading control of the image sensor 132 and output destination switching control. In the example of this figure, it is assumed that the exposure of the first photoelectric conversion element 201 and the second photoelectric conversion element 202 is started at times t1, t2, and t3. In this transfer mode, the photoelectric conversion element arrays L1 to Lm and the second photoelectric conversion element arrays Lm + 1 to Ln are sequentially exposed in a specific direction in which the photoelectric conversion element arrays L1 to Ln are arranged from each of a plurality of timings. Specifically, the first photoelectric conversion element 201 starts exposure in the order of L1, L2, L3..., And the second photoelectric conversion element 202 performs exposure in the order of Lm + 1, Lm + 2, Lm + 3. Be started. That is, reading of the photoelectric conversion element array of the first photoelectric conversion element 201 and the photoelectric conversion element array of the second photoelectric conversion element 202 is sequentially started in the same direction. And when photoelectric conversion element row | line | column Lm exposes sequentially among photoelectric conversion element row | line | columns L1-Lm, it exposes last. Of the photoelectric conversion element arrays Lm + 1 to Ln, the photoelectric conversion element array Lm + 1 adjacent to the photoelectric conversion element array Lm is first exposed when sequentially exposed.

  The first partial image data output from the first photoelectric conversion element 201 after the exposure is started at time t1 is output to the first image processing engine 210 under the control of the output switching unit 1300, and an image in the upper region of the Nth frame. Is stored in the buffer area. Further, the second partial image data output from the second photoelectric conversion element 202 after the exposure is started from the time t1 is output to the second image processing engine 220 under the control of the output switching unit 1300, and the N-1th frame. The image data of the upper area is stored in the buffer area.

  From the time t2 after the exposure of the photoelectric conversion element rows Lm is started from the time t1, the first photoelectric conversion element rows L1 to Lm and the second photoelectric conversion element rows Lm + 1 to Ln are sequentially exposed. The time t2 may be a time before the time when the exposure of the photoelectric conversion element array Lm ends. The second partial image data output from the second photoelectric conversion element 202 after the exposure is started at time t2 is output to the first image processing engine 210 under the control of the output switching unit 1300, and an image in the lower region of the Nth frame. Is stored in the buffer area. Then, the first image processing engine 210 performs image processing and encoding on the image data of the Nth frame, and is used for display on the display unit 138 and recording in the external memory 180.

  Similarly, for the partial image data after time t3, the output destination of the image data is alternately switched under the control of the output switching unit 1300. For this reason, the second partial image data obtained by starting the exposure from time t3 is output to the second image processing engine 220, and the second image processing engine 220 performs image processing and image processing on the image data of the (N + 1) th frame. Encoding is performed for display on the display unit 138 and recording in the external memory 180.

  As described above, according to the present transfer mode, the output switching unit 1300 performs the sequential exposure with the photoelectric conversion element arrays L1 to Lm from the first timing to output the first output from the plurality of photoelectric conversion element arrays L1 to Lm. 1 partial image data, and second partial image data output from the photoelectric conversion element arrays Lm + 1 to Ln by sequentially exposing the photoelectric conversion element arrays Lm + 1 to Ln from the second timing following the first timing, Output as image data of one first image. In addition, the output switching unit 1300 sequentially exposes the plurality of photoelectric conversion element arrays L1 to Lm from the second timing, and outputs the third partial image data output from the photoelectric conversion element arrays L1 to Lm. The second partial image data output from the second photoelectric conversion element arrays Lm + 1 to Ln by sequentially exposing with the photoelectric conversion element arrays Lm + 1 to Ln from the third timing subsequent to the second timing is the second following the first image. Output as image data. The first image processing engine 210 and the second image processing engine 220 perform image processing on the image data of the first image and the image data of the second image, respectively. The output switching unit 1300 sets the output destinations of the partial image data output from the photoelectric conversion element arrays L1 to Lm and the partial image data output from the photoelectric conversion element arrays Lm + 1 to Ln as the first image processing engine 210 and the second image data. A selector that sequentially switches with the image processing engine 220 may be provided. According to this transfer control example, the readout time can be shortened by dividing the photoelectric conversion element into two and reading in parallel. The output of the first photoelectric conversion element 201 and the second photoelectric conversion element 202 can be image-processed by one image processing engine while realizing high-speed reading. For this reason, it is not necessary to redundantly supply the column data of the boundary region between the first photoelectric conversion element 201 and the second photoelectric conversion element 202 to each image processing engine. Further, since the image data output to each of the first image processing engine 210 and the second image processing engine 220 is image data of one whole image, the image processing function such as sensor correction processing in the image processing engine is greatly increased. There is no need to change.

  Further, even when a subject moves and rolling distortion occurs in the subject image, so-called rolling distortion becomes distortion in a certain direction. Since the direction of rolling distortion is constant over the entire area of one entire image, moving image data with less discomfort for the viewer can be obtained.

  FIG. 15 shows a modification of transfer control. In the present modification, each of the first drive circuit 21 and the second drive circuit 22 outputs an identification signal for identifying which of the first image processing engine 210 and the second image processing engine 220 is to perform image processing to the photoelectric conversion element. The partial image data output from the columns L1 to Lm and the partial image data output from the photoelectric conversion element columns Lm + 1 to Ln are output in association with each other. The first drive circuit 21 and the second drive circuit 22 determine the acceptance of the partial image data based on the identification information output from the first image processing engine 210 and the second image processing engine 220 based on the identification information. The partial image data handled by the image processing engine 210 and the second image processing engine 220 are switched. Specifically, the outputs of the first drive circuit 21 and the second drive circuit 22 are supplied to the first image processing engine 210 and the second image processing engine 220, respectively. Here, the first drive circuit 21 and the second drive circuit 22 output partial image data in association with an identification code for identifying each and a synchronization code for identifying an output image processing engine.

  When image processing is performed in the sequence illustrated in FIG. 14, the first drive circuit 21 outputs the synchronization code SyncA in association with the partial image data obtained by starting exposure from time t1. On the other hand, the second drive circuit 22 outputs a synchronization code SyncB in association with the partial image data obtained by starting exposure from time t1. Then, the first drive circuit 21 outputs a synchronization code SyncB in association with the partial image data obtained by starting exposure from time t2. The second drive circuit 22 outputs a synchronization code SyncA in association with the partial image data obtained by starting exposure from time t2.

  The first image processing engine 210 receives the partial image data output in association with the synchronization code SyncA and stores it in the buffer area. The second image processing engine 220 receives the partial image data output in association with SyncB and stores it in the buffer area. Therefore, it is determined that the first partial image data obtained by starting exposure from time t1 is output from the first drive circuit 21 in association with SyncA, and the first image processing engine 210 performs the first partial image data. Image data is received and stored in the buffer area as image data of a partial area below the Nth frame. Then, it is determined that the second partial image data obtained by starting the exposure from time t2 is output from the second drive circuit 22 in association with SyncA, and the first image processing engine 210 performs the second partial image. The data is received and stored in the buffer area as image data of a partial area below the Nth frame.

  In the above example, the case where the imaging operation is performed by the so-called rolling shutter method has been described. In the case of performing a so-called global shutter operation, such as when capturing a still image, the first photoelectric conversion element 201 and the second photoelectric conversion are performed after resetting the accumulated charges of the first photoelectric conversion element 201 and the second photoelectric conversion element 202. Reading may be performed in the order of the elements 202, and the output of the first photoelectric conversion element 201 and the output from the second photoelectric conversion element 202 may be output to the first image processing engine 210. That is, when performing an imaging operation for generating image data of one still image, the first photoelectric conversion element arrays L1 to Lm sequentially expose in the first direction from the exposure start timing, and the first exposure start timing. Are sequentially exposed in the direction in which the photoelectric conversion element arrays L1 to Ln are arranged in the second photoelectric conversion element arrays Lm + 1 to Ln. The output switching unit 1300 first exposes the first partial image data output from the photoelectric conversion element arrays L1 to Lm by sequentially exposing with the photoelectric conversion element arrays L1 to Lm from the timing after the first exposure start timing, and continues. The partial image data output from the photoelectric conversion element arrays Lm + 1 to Ln by sequentially exposing the photoelectric conversion element arrays Lm + 1 to Ln from the timing may be output as still image data.

  Further, as described with reference to FIGS. 1 to 12 and the like, various processes such as the evaluation value calculation process may be shared by the first image processing engine 210 and the second image processing engine 220. For example, the calculation may be shared between the first image processing engine 210 and the second image processing engine 220 for each type of processing. For example, the first image processing engine 210 may perform face detection, and the second image processing engine 220 may perform template matching.

  According to the transfer mode described above, readout control for reading out the photoelectric conversion elements included in the image sensor 132 in multiple sets in parallel without greatly changing the design of the first image processing engine 210 and the second image processing engine 220. It can be made to correspond.

  In the above description, the imaging element 132 has two sets of photoelectric conversion elements. However, the imaging element 132 may have three or more sets of photoelectric conversion elements. The image processing engine unit 135 may include the same number of image processing engines as the number of sets of photoelectric conversion elements. Moreover, as an arrangement mode of the set of photoelectric conversion elements, not only an aspect in which the photoelectric conversion elements are arranged separately in the vertical direction as described in this example but also various arrangement modes can be applied. For example, the first photoelectric conversion element and the second photoelectric conversion element may be alternately provided in at least one of the row direction and the column direction.

  In the above description, the processing described as the operation of the camera MPU 140 is realized by the camera MPU 140 controlling each hardware of the camera 10 according to a program. Further, the processing realized by the image processing engine unit 135 in the above description can be realized by a processor. For example, the processing described as the operation of the image processing engine unit 135 is realized by the processor controlling each hardware of the camera 10 according to the program. That is, the processing described in relation to the camera 10 of the present embodiment is performed by the processor operating in accordance with the program to control each hardware, so that each hardware including the processor, the memory, and the like cooperates with the program. It can be realized by operating. That is, the process can be realized by a so-called computer device. The computer device may load a program for controlling the execution of the above-described process, operate according to the read program, and execute the process. The computer device can load the program from a computer-readable recording medium storing the program.

  In the present embodiment, the camera 10 with the lens unit 120 attached is taken as an example of an imaging apparatus. However, the imaging device is a concept including the camera body 130 to which the lens unit 120 is not attached. As an imaging device, in addition to a single-lens reflex camera that is an example of an interchangeable lens camera, a compact digital camera that is an example of a non-interchangeable camera, a mirrorless camera, a video camera, a mobile phone with an imaging function, imaging Various electronic devices having an imaging function, such as a portable information terminal with a function and an entertainment device such as a game machine with an imaging function, can be applied.

  As mentioned above, although this invention was demonstrated using embodiment, the technical scope of this invention is not limited to the range as described in the said embodiment. It will be apparent to those skilled in the art that various modifications or improvements can be added to the above-described embodiment. It is apparent from the scope of the claims that the embodiments added with such changes or improvements can be included in the technical scope of the present invention.

  The order of execution of each process such as operations, procedures, steps, and stages in the apparatus, system, program, and method shown in the claims, the description, and the drawings is particularly “before” or “prior to”. It should be noted that the output can be realized in any order unless the output of the previous process is used in the subsequent process. Regarding the operation flow in the claims, the description, and the drawings, even if it is described using “first”, “next”, etc. for convenience, it means that it is essential to carry out in this order. It is not a thing.

DESCRIPTION OF SYMBOLS 10 Camera, 20 Lens unit, 121 Lens mount contact, 122 Lens group, 123 Lens MPU, 124 Lens drive part, 130 Camera body, 131 Camera mount contact, 132 Image sensor, 135 Image processing engine part, 136 RAM, 137 Display control Unit, 138 display unit, 139 system memory, 140 camera MPU, 141 operation input unit, 142 AF unit, 143 focal plane shutter, 144 photometric element, 145 main mirror, 146 sub mirror, 147 finder unit, 148 drive unit, 149 connection interface 150 recording medium IF, 180 external memory, 190 power supply, 192 power supply circuit

Claims (26)

An imaging device having a plurality of first photoelectric conversion elements and a plurality of second photoelectric conversion elements;
A first readout circuit that reads out the outputs of the plurality of first photoelectric conversion elements and outputs first partial image data that is image data of an image corresponding to a portion provided with the plurality of first photoelectric conversion elements;
A second readout circuit that reads out the outputs of the plurality of second photoelectric conversion elements and outputs second partial image data that is image data of an image corresponding to a portion provided in the plurality of second photoelectric conversion elements;
A first calculator that calculates a first evaluation value from the first partial image data output from the first readout circuit;
An imaging apparatus comprising: a second calculation unit that calculates a second evaluation value from the second partial image data output from the second readout circuit. The first evaluation unit that evaluates one image represented by the first partial image data and the second partial image data based on the first evaluation value and the second evaluation value. Imaging device. The second evaluation unit that evaluates one image represented by the first partial image data and the second partial image data based on the first evaluation value and the second evaluation value. Imaging device. The image sensor continuously images at a plurality of timings,
The first evaluation unit is obtained by imaging at the first timing based on the first evaluation value and the second evaluation value when the imaging element images at the first timing. Evaluating one image represented by the first partial image data and the second partial image data;
The second evaluation unit picks up an image at the second timing based on the first evaluation value and the second evaluation value when the image pickup device picks up an image at the second timing following the first timing. The imaging apparatus according to claim 3, wherein one image represented by the first partial image data and the second partial image data obtained by performing the evaluation is evaluated. The first calculation unit calculates a first type of evaluation value as the first evaluation value;
The imaging apparatus according to claim 3, wherein the second calculation unit calculates a second type of evaluation value as the second evaluation value. The image sensor continuously images at a plurality of timings,
The first calculation unit calculates a first type of evaluation value as the first evaluation value when the image sensor has imaged at a first timing;
The second calculation unit calculates a second type of evaluation value as the second evaluation value when the image sensor has imaged at the first timing,
The first calculation unit calculates the second type of evaluation value as the first evaluation value when the imaging element images at a second timing following the first timing,
The imaging apparatus according to claim 3, wherein the second calculation unit calculates the first type of evaluation value as the second evaluation value when the imaging device captures an image at the second timing. The first evaluation value and the second evaluation value are an AF evaluation value for evaluating a focus state, an exposure evaluation value for evaluating an exposure state, a white balance evaluation value for evaluating a white balance state, The imaging apparatus according to claim 1, comprising at least one of face detection information for evaluating a face position. A second processing unit having the second calculation unit and a second output unit for outputting the second partial image data;
A first image processing unit configured to perform image processing on image data of one image represented by the first partial image data and the second partial image data output by the second output unit; The imaging device according to any one of claims 1 to 7, further comprising a first processing unit. The first processing unit further includes a first output unit that outputs the first partial image data to the second processing unit,
The second processing unit further includes a second image processing unit that performs image processing on image data of one image represented by the second partial image data and the first partial image data output by the first output unit. The imaging apparatus according to claim 8. The image sensor continuously images at a plurality of timings,
The second output unit outputs the second partial image data to the first image processing unit when the imaging element captures an image at a first timing,
The said 1st output part outputs the said 1st partial image data to the said 2nd image process part, when the said image pick-up element imaged at the 2nd timing following the said 1st timing. Imaging device. The first output unit outputs the first partial image data, which is RAW image data, to the second processing unit,
The second image processing unit performs image processing including color interpolation processing and compression processing on image data of one image represented by the second partial image data which is the first partial image data and RAW image data,
The second output unit outputs the compressed image data obtained by the second image processing unit performing the image processing to the first processing unit,
The imaging apparatus according to claim 10, wherein the first processing unit further includes a recording control unit that records the compressed image data on a recording medium. The second output unit outputs the second partial image data, which is RAW image data, to the first processing unit,
The first image processing unit performs image processing including color interpolation processing and compression processing on image data of one image represented by the first partial image data and the second partial image data which are RAW image data. The imaging device according to any one of claims 8 to 11. A first readout circuit that reads out outputs of a plurality of first photoelectric conversion elements included in the imaging unit and outputs first image data that is image data of an image corresponding to a portion provided with the plurality of first photoelectric conversion elements. When,
Second readout for reading out outputs of a plurality of second photoelectric conversion elements included in the imaging unit and outputting second image data that is image data corresponding to portions provided in the plurality of second photoelectric conversion elements. Circuit,
A first calculator that calculates a first evaluation value from the first image data output from the first readout circuit;
A second calculation unit that calculates a second evaluation value from the second image data output from the second readout circuit;
The first calculation unit outputs the first evaluation value to the second calculation unit,
The second calculation unit is an imaging apparatus that outputs the second evaluation value to the first calculation unit. The plurality of first photoelectric conversion elements and the plurality of second photoelectric conversion elements each form a plurality of photoelectric conversion element arrays,
One of the plurality of photoelectric conversion element arrays formed by the plurality of first photoelectric conversion elements is one of the plurality of photoelectric conversion element arrays formed by the plurality of second photoelectric conversion elements. Adjacent to one second photoelectric conversion element array,
The imaging device
A first buffer unit that holds column data of a predetermined number of photoelectric conversion element columns sequentially output from the first readout circuit;
A second buffer unit that holds column data of a predetermined number of photoelectric conversion element columns sequentially output from the second readout circuit;
Column data of the second photoelectric conversion element array held in the second buffer section is output to the first calculation section, and column data of the first photoelectric conversion element array held in the first buffer section is output. The imaging apparatus according to claim 1, further comprising: an output control unit that outputs the output to the first calculation unit. The first calculation unit is configured based on image data of one image represented by column data of the second photoelectric conversion element array and column data of the plurality of photoelectric conversion element arrays formed by the plurality of first photoelectric conversion elements. The imaging device according to claim 14, wherein the first evaluation value is calculated. An imaging device having a plurality of first photoelectric conversion element rows and a plurality of second photoelectric conversion element rows that are successively arranged in the first direction;
An exposure control unit that sequentially exposes the first photoelectric conversion element array and the plurality of second photoelectric conversion element arrays in the first direction from each of a plurality of timings;
The first partial image data output from the plurality of first photoelectric conversion element arrays by sequentially exposing the plurality of first photoelectric conversion element arrays from the first timing, and the first timing following the first timing. The second partial image data output from the plurality of second photoelectric conversion element arrays is output as image data of one first image by sequentially exposing the plurality of second photoelectric conversion element arrays from the timing of 2. And an output control unit. The output control unit is configured to sequentially expose the plurality of first photoelectric conversion element arrays from the second timing to output third partial image data output from the plurality of first photoelectric conversion element arrays, and The fourth partial image data output from the plurality of second photoelectric conversion element arrays by sequentially exposing with the plurality of second photoelectric conversion element arrays from a third timing following the second timing, The image pickup apparatus according to claim 16, wherein the image pickup apparatus outputs image data of a second image following the image. A first image processing unit that performs image processing on image data of the first image;
The imaging apparatus according to claim 17, further comprising: a second image processing unit that performs image processing on image data of the second image. The output control unit outputs the partial image data output from the plurality of first photoelectric conversion element arrays and the output destination of the partial image data output from the plurality of second photoelectric conversion element arrays to the first image processing unit. The imaging apparatus according to claim 18, further comprising a selector that sequentially switches between the second image processing unit and the second image processing unit. The output control unit outputs an identification signal for identifying which of the first image processing unit and the second image processing unit to perform image processing, partial image data output from the plurality of first photoelectric conversion element arrays, and The image pickup apparatus according to claim 18, wherein the image pickup apparatus outputs the image data in association with the partial image data output from the plurality of second photoelectric conversion element arrays. Among the plurality of first photoelectric conversion element arrays, the photoelectric conversion element array that is exposed last when sequentially exposed is first exposed when the plurality of second photoelectric conversion element arrays are sequentially exposed. Adjacent to the photoelectric conversion element array,
The exposure control unit, from the second timing after starting exposure of the photoelectric conversion element row exposed last when the plurality of first photoelectric conversion element rows are exposed from the first timing, The imaging device according to any one of claims 16 to 20, wherein the plurality of first photoelectric conversion element arrays and the plurality of second photoelectric conversion element arrays are sequentially exposed. When the exposure control unit is exposed from the first timing with the plurality of first photoelectric conversion element arrays, from the second timing before the exposure of the photoelectric conversion element array exposed last is completed, The imaging apparatus according to claim 21, wherein the plurality of first photoelectric conversion element arrays and the plurality of second photoelectric conversion element arrays are sequentially exposed. Among the plurality of first photoelectric conversion element arrays, the photoelectric conversion element array that is exposed last when sequentially exposed is first exposed when the plurality of second photoelectric conversion element arrays are sequentially exposed. Adjacent to the photoelectric conversion element array,
When performing an imaging operation for generating image data of one still image, the exposure control unit sequentially exposes the plurality of first photoelectric conversion element arrays in the first direction from a fourth timing, From the fifth timing following the fourth timing, the plurality of second photoelectric conversion element arrays are sequentially exposed in the first direction,
The output control unit, the first partial image data output from the plurality of first photoelectric conversion element rows by sequentially exposing the plurality of first photoelectric conversion element rows from the fourth timing, and The second partial image data output from the plurality of second photoelectric conversion element arrays by sequentially exposing the plurality of second photoelectric conversion element arrays from the fifth timing is used as image data of the still image. The imaging device according to any one of claims 16 to 22, wherein the imaging device outputs the image. A program for an imaging apparatus having an imaging element having a plurality of first photoelectric conversion elements and a plurality of second photoelectric conversion elements,
The first readout circuit reads out the outputs of the plurality of first photoelectric conversion elements, and outputs first partial image data that is image data of an image corresponding to a portion provided with the plurality of first photoelectric conversion elements. Steps,
The second readout circuit reads out the outputs of the plurality of second photoelectric conversion elements, and outputs second partial image data that is image data of an image corresponding to a portion provided in the plurality of second photoelectric conversion elements. Steps,
Causing the first calculation unit to calculate a first evaluation value from the first partial image data output from the first readout circuit;
A program for causing a computer to execute a step of causing a second calculation unit to calculate a second evaluation value from the second partial image data output from the second readout circuit. A first image which is image data of an image corresponding to a portion provided with the plurality of first photoelectric conversion elements by causing the first reading circuit to read the outputs of the plurality of first photoelectric conversion elements included in the imaging unit. Outputting data; and
Second output is image data of an image corresponding to a portion provided in the plurality of second photoelectric conversion elements by causing the second reading circuit to read outputs of the plurality of second photoelectric conversion elements included in the imaging unit. Outputting image data; and
Causing the first calculator to calculate a first evaluation value from the first image data output from the first readout circuit;
Causing the second calculator to calculate a second evaluation value from the second image data output from the second readout circuit;
Outputting the first evaluation value to the second calculation unit;
A program for causing a computer to execute the step of outputting the second evaluation value to the first calculation unit. A program for an imaging apparatus having an imaging element having a plurality of first photoelectric conversion element rows and a plurality of second photoelectric conversion element rows that are successively arranged in a first direction,
An exposure control step of sequentially exposing the plurality of first photoelectric conversion element arrays and the plurality of second photoelectric conversion element arrays in the first direction from each of a plurality of timings;
The first partial image data output from the plurality of first photoelectric conversion element arrays by sequentially exposing the plurality of first photoelectric conversion element arrays from the first timing, and the first timing following the first timing. The second partial image data output from the plurality of second photoelectric conversion element arrays is output as image data of one first image by sequentially exposing the plurality of second photoelectric conversion element arrays from the timing of 2. A program for causing a computer to execute an output control step.

Images