WO2017057493A1

WO2017057493A1 - Imaging device and image processing device

Info

Publication number: WO2017057493A1
Application number: PCT/JP2016/078684
Authority: WO
Inventors: 孝塩野谷; 敏之神原; 直樹關口
Original assignee: 株式会社ニコン
Priority date: 2015-09-30
Filing date: 2016-09-28
Publication date: 2017-04-06
Also published as: JP6516015B2; JPWO2017057493A1

Abstract

This imaging device is provided with: an imaging element which has an imaging region for imaging a subject, said region having arranged therein first pixels, which output a signal generated from a photoelectrically converted charge, and second pixels, different from the first pixels, which output a signal generated from a photoelectrically converted charge; a setting unit which sets an imaging condition of a first region of the imaging region, in which the first pixels are arranged, and sets an imaging condition of a second region of the imaging region, different from the first region, in which the second pixels are arranged; a selection unit which selects pixels for use in interpolation of the first pixels in the first region in which the first imaging condition was set by the setting unit, said selection being made from the second pixels in the second region in which a second imaging condition different from the first imaging condition was set by the setting unit and the second pixels in the second region in which a third imaging condition different from the second imaging condition was set by the setting unit; and a detection unit which detects at least part of the subject imaged in the imaging region using a signal which, outputted from the first pixels in the first region in which the first imaging condition was set, is interpolated by means of the signal outputted from the second pixels selected by the selection unit.

Description

Imaging apparatus and image processing apparatus

The present invention relates to an imaging device and an image processing device.

An imaging apparatus equipped with an imaging element capable of setting different imaging conditions for each screen area is known (see Patent Document 1). However, when using image data generated in areas with different imaging conditions, there is a problem in that it cannot be performed in the same manner as when using image data generated in an area with the same imaging conditions.

Japanese Unexamined Patent Publication No. 2006-197192

According to the first aspect of the present invention, the imaging device includes a first pixel that outputs a signal generated by the photoelectrically converted charge, and a first pixel that outputs a signal generated by the photoelectrically converted charge. A second pixel different from the image pickup element having an image pickup area for picking up a subject, and an image pickup condition of the first area in which the first pixel is arranged in the image pickup area, A setting unit that sets an imaging condition of a second region that is different from the first region in which the second pixel is arranged in the imaging region, and the first that is set as the first imaging condition by the setting unit The pixels used for interpolation of the first pixel in one area are set to the second pixel in the second area set by the setting unit to a second imaging condition different from the first imaging condition, and the setting unit The third imaging condition is different from the second imaging condition. The first imaging condition interpolated by the selection unit selected from the second pixels of the second region determined and the signal output from the second pixel selected by the selection unit. A detection unit configured to detect at least a part of a subject imaged in the imaging region using a signal output from the first pixel of the set first region.
According to the second aspect of the present invention, the imaging device includes a first pixel that outputs a signal generated by the photoelectrically converted charge, and a first pixel that outputs a signal generated by the photoelectrically converted charge. And a third pixel that outputs a signal generated by photoelectrically converted charges, and a first imaging element having a first imaging region that images a subject. A second imaging element different from the first imaging element having a second imaging area for imaging a subject, and a pixel used for interpolation of the first pixel, the second pixel and the third pixel The image picked up in the first imaging region using the signal output from the first pixel, interpolated by the signal output from the pixel selected by the selection unit selected from the pixels and the pixel selected by the selection unit At least part of the subject And a detection unit for output.
According to the third aspect of the present invention, the imaging device includes a first pixel that outputs a signal generated by the photoelectrically converted charge, and a first pixel that outputs a signal generated by the photoelectrically converted charge. A second pixel different from each other, an imaging element having an imaging area for imaging a subject, an imaging condition of the first area in which the first pixel is arranged in the imaging area, A setting unit that sets an imaging condition of a second region that is different from the first region in which the second pixel is arranged in the imaging region, and the first that is set as the first imaging condition by the setting unit The second pixel in the second region, the pixel used for the signal processing of the signal output from the first pixel in the region set to a second imaging condition different from the first imaging condition by the setting unit; What is the second imaging condition by the setting unit? The second pixel of the second region set in the third imaging condition is selected from among the selection unit, and the signal processed by the signal output from the second pixel selected by the selection unit And a detection unit that detects at least a part of a subject imaged in the imaging region using a signal output from the first pixel of the first region set in the first imaging condition.
According to the fourth aspect of the present invention, the imaging device includes a first pixel that generates a signal based on photoelectrically converted charges, and a second pixel that is different from the first pixel that generates a signal based on photoelectrically converted charges. And a first imaging element having a first imaging region for imaging a subject and a third pixel for generating a signal by photoelectrically converted charges, and imaging the subject. A second imaging element having a second imaging area that is different from the first imaging element, and a pixel used for signal processing of a signal output from the first pixel, the second pixel and the third pixel A selection unit to be selected from among them, and a signal processed by a signal output from the pixel selected by the selection unit, and a subject imaged in the first imaging region using a signal output from the first pixel Detect at least some That includes a detecting unit.
According to the fifth aspect of the present invention, the image processing apparatus uses the pixels used for interpolation of the first pixels arranged in the first area, among the imaging areas of the imaging element set in the first imaging condition. Of the imaging areas set to a second imaging condition different from the first imaging conditions, the second pixel arranged in the second area and a third imaging condition different from the second imaging condition are set. The selection unit selected from among the second pixels arranged in the second region, and the first imaging condition interpolated by a signal output from the second pixel selected by the selection unit A detection unit configured to detect at least a part of a subject imaged in the imaging region using a signal output from the first pixel of the set first region.
According to the sixth aspect of the present invention, the image processing apparatus includes, in the first imaging region, a pixel used for interpolation of the first pixel arranged in the first imaging region of the first imaging element. A selection unit that selects from among a second pixel that is different from one pixel and a third pixel that is disposed in a second imaging region of a second imaging element that is different from the first imaging element, and the selection unit selects A detection unit that detects at least a part of the subject imaged in the first imaging region using the signal output from the first pixel interpolated by the signal output from the pixel that has been output.
According to the seventh aspect of the present invention, the image processing apparatus performs signal processing of a signal output from the first pixel arranged in the first area among the imaging areas of the imaging element set in the first imaging condition. Among the imaging regions set to the second imaging condition different from the first imaging condition, the pixel used for the second imaging is different from the second pixel arranged in the second region and the third imaging different from the second imaging condition The second pixel arranged in the second region set as a condition, a selection unit to select from, and the signal processed by the signal output from the second pixel selected by the selection unit, A detection unit configured to detect at least a part of a subject imaged in the imaging region using a signal output from the first pixel in the first region set in the first imaging condition.
According to the eighth aspect of the present invention, in the image processing device, a pixel used for signal processing of a signal output from the first pixel arranged in the first imaging region of the first imaging element is stored in the first imaging region. A selection unit that selects between a second pixel that is different from the first pixel and a third pixel that is arranged in a second imaging region of a second imaging element different from the first imaging element; Detection for detecting at least a part of a subject imaged in the first imaging region using a signal output from the first pixel, which is signal-processed by a signal output from the pixel selected by the selection unit A section.

It is a block diagram which illustrates the composition of the camera by an embodiment. It is sectional drawing of a laminated type image pick-up element. It is a figure explaining the pixel arrangement | sequence and unit area | region of an imaging chip. It is a figure explaining the circuit in a unit area. It is a figure which shows typically the image of the to-be-photographed object imaged on the image pick-up element of a camera. It is a figure which illustrates the setting screen of imaging conditions. 7A is a diagram illustrating the vicinity of the boundary of the first region in the live view image, FIG. 7B is an enlarged view of the vicinity of the boundary, FIG. 7C is an enlarged view of the target pixel and the reference pixel, and FIG. 7D is an enlarged view of the corresponding reference pixel in the processing image data. FIG. 8A is a diagram illustrating the arrangement of photoelectric conversion signals output from the pixels, FIG. 8 (9) is a diagram illustrating the interpolation of the G color component image data, and FIG. 8C is the G after the interpolation. It is a figure which illustrates the image data of a color component. 9A is a diagram obtained by extracting image data of the R color component from FIG. 8A, FIG. 9B is a diagram illustrating interpolation of the color difference component Cr, and FIG. 9C is an image of the color difference component Cr. It is a figure explaining the interpolation of data. 10A is a diagram obtained by extracting B color component image data from FIG. 8A, FIG. 10B is a diagram illustrating interpolation of the color difference component Cb, and FIG. 10C is an image of the color difference component Cb. It is a figure explaining the interpolation of data. It is a figure which illustrates the position of the pixel for focus detection in an imaging surface. It is the figure which expanded the one part area | region of the focus detection pixel line. It is the figure which expanded the focus point. FIG. 14A is a diagram illustrating a template image representing an object to be detected, and FIG. 14B is a diagram illustrating a live view image and a search range. It is a figure which illustrates about the relationship between the timing of imaging of the image data for a live view image, and the timing of imaging of the processing image data, and FIG. 15A alternately shows the live view image and the processing image data. FIG. 15B illustrates the case where the image data for processing is captured at the start of the display of the live view image, and FIG. 15C illustrates the image data for processing when the display of the live view image ends. It is a figure which illustrates the case where it images. It is a flowchart explaining the flow of the process which sets an imaging condition for every area and images. FIGS. 17A to 17C are diagrams illustrating the arrangement of the first region and the second region on the imaging surface of the imaging device. It is a block diagram which illustrates the composition of the camera by modification 2. FIG. 10 is a block diagram illustrating a configuration of an imaging system according to Modification 6. It is a figure explaining supply of the program to a mobile device.

A digital camera will be described as an example of an electronic device equipped with the image processing apparatus according to this embodiment. The camera 1 (FIG. 1) is configured to be able to capture images under different conditions for each region of the imaging surface of the image sensor 32a. The image processing unit 33 performs appropriate processing in areas with different imaging conditions. Details of the camera 1 will be described with reference to the drawings.

<Explanation of camera>
FIG. 1 is a block diagram illustrating the configuration of a camera 1 according to an embodiment. In FIG. 1, the camera 1 includes an imaging optical system 31, an imaging unit 32, an image processing unit 33, a control unit 34, a display unit 35, an operation member 36, and a recording unit 37.

The imaging optical system 31 guides the light flux from the object scene to the imaging unit 32. The imaging unit 32 includes an imaging element 32a and a driving unit 32b, and photoelectrically converts an object image formed by the imaging optical system 31. The imaging unit 32 can capture images under the same conditions over the entire imaging surface of the imaging device 32a, or can perform imaging under different conditions for each region of the imaging surface of the imaging device 32a. Details of the imaging unit 32 will be described later. The drive unit 32b generates a drive signal necessary for causing the image sensor 32a to perform accumulation control. An imaging instruction such as a charge accumulation time for the imaging unit 32 is transmitted from the control unit 34 to the driving unit 32b.

The image processing unit 33 includes an input unit 33a and a processing unit 33b. Image data acquired by the imaging unit 32 is input to the input unit 33a. The processing unit 33b performs predetermined image processing on the main image data using image data captured separately from the main image data when the main imaging is performed with different imaging conditions between different regions. , Generate an image. Image processing includes, for example, color interpolation processing, pixel defect correction processing, edge enhancement processing, noise reduction processing, white balance adjustment processing, gamma correction processing, display luminance adjustment processing, saturation adjustment processing, and the like. .

The control unit 34 is constituted by a CPU, for example, and controls the overall operation of the camera 1. For example, the control unit 34 performs a predetermined exposure calculation based on the photoelectric conversion signal acquired by the imaging unit 32, the charge accumulation time (exposure time) of the imaging element 32a necessary for proper exposure, and the aperture of the imaging optical system 31. The exposure conditions such as the value and ISO sensitivity are determined and instructed to the drive unit 32b. In addition, image processing conditions for adjusting saturation, contrast, sharpness, and the like are determined and instructed to the image processing unit 33 according to the imaging scene mode set in the camera 1 and the type of the detected subject element. The detection of the subject element will be described later.

The control unit 34 includes an object detection unit 34a, a setting unit 34b, an imaging control unit 34c, and an AF calculation unit 34d. These are realized as software by the control unit 34 executing a program stored in a nonvolatile memory (not shown). However, these may be configured by an ASIC or the like.

The object detection unit 34a performs a known object recognition process, and from the image acquired by the imaging unit 32, a person (person's face), an animal such as a dog or a cat (animal face), a plant, a bicycle, an automobile , Detecting a subject element such as a vehicle such as a train, a building, a stationary object, a landscape such as a mountain or a cloud, or a predetermined specific object. The setting unit 34b divides the imaging screen by the imaging unit 32 into a plurality of regions including the subject element detected as described above.

The setting unit 34b further sets imaging conditions for a plurality of areas. Imaging conditions include the exposure conditions (charge accumulation time, gain, ISO sensitivity, frame rate, etc.) and the image processing conditions (for example, white balance adjustment parameters, gamma correction curves, display brightness adjustment parameters, saturation adjustment parameters, etc.) ). As the imaging conditions, the same imaging conditions can be set for all of the plurality of areas, or different imaging conditions can be set for the plurality of areas.

The imaging control unit 34c controls the imaging unit 32 (imaging element 32a) and the image processing unit 33 by applying imaging conditions set for each region by the setting unit 34b. Thereby, it is possible to cause the imaging unit 32 to perform imaging under different exposure conditions for each of the plurality of regions, and for the image processing unit 33, images with different image processing conditions for each of the plurality of regions. Processing can be performed. Any number of pixels may be included in the region, for example, 1000 pixels or 1 pixel. Further, the number of pixels may be different between regions.

The AF calculation unit 34d controls an automatic focus adjustment (autofocus: AF) operation for focusing on a corresponding subject at a predetermined position (called a focus point) on the imaging screen. The AF calculation unit 34d sends a drive signal for moving the focus lens of the imaging optical system 31 to the in-focus position based on the calculation result. The process performed by the AF calculation unit 34d for automatic focus adjustment is also referred to as a focus detection process. Details of the focus detection process will be described later.

The display unit 35 reproduces and displays the image generated by the image processing unit 33, the image processed image, the image read by the recording unit 37, and the like. The display unit 35 also displays an operation menu screen, a setting screen for setting imaging conditions, and the like.

The operation member 36 is composed of various operation members such as a release button and a menu button. The operation member 36 sends an operation signal corresponding to each operation to the control unit 34. The operation member 36 includes a touch operation member provided on the display surface of the display unit 35.

The recording unit 37 records image data or the like on a recording medium including a memory card (not shown) in response to an instruction from the control unit 34. The recording unit 37 reads image data recorded on the recording medium in response to an instruction from the control unit 34.

<Description of Laminated Image Sensor>
A laminated image sensor 100 will be described as an example of the image sensor 32a described above. FIG. 2 is a cross-sectional view of the image sensor 100. The imaging element 100 includes an imaging chip 111, a signal processing chip 112, and a memory chip 113. The imaging chip 111 is stacked on the signal processing chip 112. The signal processing chip 112 is stacked on the memory chip 113. The imaging chip 111, the signal processing chip 112, the signal processing chip 112, and the memory chip 113 are electrically connected by a connection unit 109. The connection unit 109 is, for example, a bump or an electrode. The imaging chip 111 captures a light image from a subject and generates image data. The imaging chip 111 outputs image data from the imaging chip 111 to the signal processing chip 112. The signal processing chip 112 performs signal processing on the image data output from the imaging chip 111. The memory chip 113 has a plurality of memories and stores image data. Note that the image sensor 100 may include an image pickup chip and a signal processing chip. When the imaging device 100 is configured by an imaging chip and a signal processing chip, a storage unit for storing image data may be provided in the signal processing chip or may be provided separately from the imaging device 100. .

As shown in FIG. 2, the incident light is incident mainly in the positive direction of the Z axis indicated by the white arrow. Further, as shown in the coordinate axes, the left direction of the paper orthogonal to the Z axis is the X axis plus direction, and the front side of the paper orthogonal to the Z axis and X axis is the Y axis plus direction. In the following several figures, the coordinate axes are displayed so that the orientation of each figure can be understood with reference to the coordinate axes in FIG.

The imaging chip 111 is, for example, a CMOS image sensor. Specifically, the imaging chip 111 is a backside illumination type CMOS image sensor. The imaging chip 111 includes a microlens layer 101, a color filter layer 102, a passivation layer 103, a semiconductor layer 106, and a wiring layer 108. The imaging chip 111 is arranged in the order of the microlens layer 101, the color filter layer 102, the passivation layer 103, the semiconductor layer 106, and the wiring layer 108 in the positive Z-axis direction.

The microlens layer 101 has a plurality of microlenses L. The microlens L condenses incident light on the photoelectric conversion unit 104 described later. The color filter layer 102 includes a plurality of color filters F. The color filter layer 102 has a plurality of types of color filters F having different spectral characteristics. Specifically, the color filter layer 102 includes a first filter (R) having a spectral characteristic that mainly transmits red component light and a second filter (Gb, Gr) that has a spectral characteristic that mainly transmits green component light. ) And a third filter (B) having a spectral characteristic that mainly transmits blue component light. In the color filter layer 102, for example, a first filter, a second filter, and a third filter are arranged in a Bayer arrangement. The passivation layer 103 is made of a nitride film or an oxide film, and protects the semiconductor layer 106.

The semiconductor layer 106 includes a photoelectric conversion unit 104 and a readout circuit 105. The semiconductor layer 106 includes a plurality of photoelectric conversion units 104 between a first surface 106a that is a light incident surface and a second surface 106b opposite to the first surface 106a. The semiconductor layer 106 includes a plurality of photoelectric conversion units 104 arranged in the X-axis direction and the Y-axis direction. The photoelectric conversion unit 104 has a photoelectric conversion function of converting light into electric charge. In addition, the photoelectric conversion unit 104 accumulates charges based on the photoelectric conversion signal. The photoelectric conversion unit 104 is, for example, a photodiode. The semiconductor layer 106 includes a readout circuit 105 on the second surface 106b side of the photoelectric conversion unit 104. In the semiconductor layer 106, a plurality of readout circuits 105 are arranged in the X-axis direction and the Y-axis direction. The readout circuit 105 includes a plurality of transistors, reads out image data generated by the electric charges photoelectrically converted by the photoelectric conversion unit 104, and outputs the image data to the wiring layer 108.

The wiring layer 108 has a plurality of metal layers. The metal layer is, for example, an Al wiring, a Cu wiring, or the like. The wiring layer 108 outputs the image data read by the reading circuit 105. The image data is output from the wiring layer 108 to the signal processing chip 112 via the connection unit 109.

Note that the connection unit 109 may be provided for each photoelectric conversion unit 104. Further, the connection unit 109 may be provided for each of the plurality of photoelectric conversion units 104. When the connection unit 109 is provided for each of the plurality of photoelectric conversion units 104, the pitch of the connection units 109 may be larger than the pitch of the photoelectric conversion units 104. In addition, the connection unit 109 may be provided in a peripheral region of the region where the photoelectric conversion unit 104 is disposed.

The signal processing chip 112 has a plurality of signal processing circuits. The signal processing circuit performs signal processing on the image data output from the imaging chip 111. The signal processing circuit includes, for example, an amplifier circuit that amplifies the signal value of the image data, a correlated double sampling circuit that performs noise reduction processing of the image data, and analog / digital (A / D) conversion that converts the analog signal into a digital signal. Circuit etc. A signal processing circuit may be provided for each photoelectric conversion unit 104.

Further, a signal processing circuit may be provided for each of the plurality of photoelectric conversion units 104. The signal processing chip 112 has a plurality of through electrodes 110. The through electrode 110 is, for example, a silicon through electrode. The through electrode 110 connects circuits provided in the signal processing chip 112 to each other. The through electrode 110 may also be provided in the peripheral region of the imaging chip 111 and the memory chip 113. Note that some elements constituting the signal processing circuit may be provided in the imaging chip 111. For example, in the case of an analog / digital conversion circuit, a comparator that compares an input voltage with a reference voltage may be provided in the imaging chip 111, and circuits such as a counter circuit and a latch circuit may be provided in the signal processing chip 112.

The memory chip 113 has a plurality of storage units. The storage unit stores image data that has been subjected to signal processing by the signal processing chip 112. The storage unit is a volatile memory such as a DRAM, for example. A storage unit may be provided for each photoelectric conversion unit 104. In addition, the storage unit may be provided for each of the plurality of photoelectric conversion units 104. The image data stored in the storage unit is output to the subsequent image processing unit.

FIG. 3 is a diagram for explaining the pixel array and the unit area 131 of the imaging chip 111. In particular, a state where the imaging chip 111 is observed from the back surface (imaging surface) side is shown. For example, 20 million or more pixels are arranged in a matrix in the pixel region. In the example of FIG. 3, four adjacent pixels of 2 pixels × 2 pixels form one unit region 131. The grid lines in the figure indicate the concept that adjacent pixels are grouped to form a unit region 131. The number of pixels forming the unit region 131 is not limited to this, and may be about 1000, for example, 32 pixels × 32 pixels, more or less, or one pixel.

As shown in the partially enlarged view of the pixel area, the unit area 131 in FIG. 3 includes a so-called Bayer array composed of four pixels of green pixels Gb, Gr, blue pixels B, and red pixels R. The green pixels Gb and Gr are pixels having a green filter as the color filter F, and receive light in the green wavelength band of incident light. Similarly, the blue pixel B is a pixel having a blue filter as the color filter F and receives light in the blue wavelength band, and the red pixel R is a pixel having a red filter as the color filter F and having a red wavelength band. Receives light.

In the present embodiment, a plurality of blocks are defined so as to include at least one unit region 131 per block. That is, the minimum unit of one block is one unit area 131. As described above, of the possible values for the number of pixels forming one unit region 131, the smallest number of pixels is one pixel. Therefore, when one block is defined in units of pixels, the minimum number of pixels among the number of pixels that can define one block is one pixel. Each block can control pixels included in each block with different control parameters. In each block, all the unit areas 131 in the block, that is, all the pixels in the block are controlled under the same imaging condition. That is, photoelectric conversion signals having different imaging conditions can be acquired between a pixel group included in a certain block and a pixel group included in another block. Examples of the control parameters include a frame rate, a gain, a thinning rate, the number of addition rows or addition columns to which photoelectric conversion signals are added, a charge accumulation time or accumulation count, a digitization bit number (word length), and the like. The imaging device 100 can freely perform not only thinning in the row direction (X-axis direction of the imaging chip 111) but also thinning in the column direction (Y-axis direction of the imaging chip 111). Furthermore, the control parameter may be a parameter in image processing.

FIG. 4 is a diagram for explaining a circuit in the unit region 131. In the example of FIG. 4, one unit region 131 is formed by four adjacent pixels of 2 pixels × 2 pixels. As described above, the number of pixels included in the unit region 131 is not limited to this, and may be 1000 pixels or more, or may be a minimum of 1 pixel. The two-dimensional position of the unit area 131 is indicated by reference signs A to D.

The reset transistor (RST) of the pixel included in the unit region 131 is configured to be turned on and off individually for each pixel. In FIG. 4, a reset wiring 300 for turning on / off the reset transistor of the pixel A is provided, and a reset wiring 310 for turning on / off the reset transistor of the pixel B is provided separately from the reset wiring 300. Similarly, a reset line 320 for turning on and off the reset transistor of the pixel C is provided separately from the

reset lines

300 and 310. A dedicated reset wiring 330 for turning on and off the reset transistor is also provided for the other pixels D.

The pixel transfer transistor (TX) included in the unit region 131 is also configured to be turned on and off individually for each pixel. In FIG. 4, a transfer wiring 302 for turning on / off the transfer transistor of the pixel A, a transfer wiring 312 for turning on / off the transfer transistor of the pixel B, and a transfer wiring 322 for turning on / off the transfer transistor of the pixel C are separately provided. Also for the other pixels D, a dedicated transfer wiring 332 for turning on / off the transfer transistor is provided.

Furthermore, the pixel selection transistor (SEL) included in the unit region 131 is also configured to be turned on and off individually for each pixel. In FIG. 4, a selection wiring 306 for turning on / off the selection transistor of the pixel A, a selection wiring 316 for turning on / off the selection transistor of the pixel B, and a selection wiring 326 for turning on / off the selection transistor of the pixel C are separately provided. Also for the other pixels D, a dedicated selection wiring 336 for turning on and off the selection transistor is provided.

Note that the power supply wiring 304 is commonly connected from the pixel A to the pixel D included in the unit region 131. Similarly, the output wiring 308 is commonly connected to the pixel D from the pixel A included in the unit region 131. Further, the power supply wiring 304 is commonly connected between a plurality of unit regions, but the output wiring 308 is provided for each unit region 131 individually. The load current source 309 supplies current to the output wiring 308. The load current source 309 may be provided on the imaging chip 111 side or may be provided on the signal processing chip 112 side.

By individually turning on and off the reset transistor and the transfer transistor in the unit region 131, the charge accumulation including the charge accumulation start time, the accumulation end time, and the transfer timing is controlled from the pixel A to the pixel D included in the unit region 131. can do. In addition, by individually turning on and off the selection transistors in the unit region 131, the photoelectric conversion signals of the pixels A to D can be output via the common output wiring 308.

Here, a so-called rolling shutter system is known in which charge accumulation is controlled in a regular order with respect to rows and columns for the pixels A to D included in the unit region 131. When a column is designated after selecting a pixel for each row by the rolling shutter method, photoelectric conversion signals are output in the order of “ABCD” in the example of FIG.

Thus, by configuring the circuit with the unit region 131 as a reference, the charge accumulation time can be controlled for each unit region 131. In other words, it is possible to output photoelectric conversion signals at different frame rates between the unit regions 131. Further, in the imaging chip 111, the unit area 131 included in another block is rested while the unit area 131 included in a part of the block is charged (imaged), so that a predetermined block of the imaging chip 111 can be used. Only the imaging can be performed, and the photoelectric conversion signal can be output. Furthermore, it is also possible to switch a block (accumulation control target block) where charge accumulation (imaging) is performed between frames, sequentially perform imaging with different blocks of the imaging chip 111, and output a photoelectric conversion signal.

As described above, the output wiring 308 is provided corresponding to each of the unit areas 131. Since the image pickup device 100 includes the image pickup chip 111, the signal processing chip 112, and the memory chip 113, each chip is arranged in the surface direction by using the electrical connection between the chips using the connection portion 109 for the output wiring 308. The wiring can be routed without increasing the size.

<Block control of image sensor>
In the present embodiment, an imaging condition can be set for each of a plurality of blocks in the imaging device 32a. The control unit 34 (imaging control unit 34c) associates the plurality of regions with the block and causes the imaging to be performed under an imaging condition set for each region.

FIG. 5 is a diagram schematically showing an image of a subject formed on the image sensor 32a of the camera 1. The camera 1 photoelectrically converts the subject image to obtain a live view image before an imaging instruction is given. The live view image refers to a monitor image that is repeatedly imaged at a predetermined frame rate (for example, 60 fps).

The control unit 34 sets the same imaging condition over the entire area of the imaging chip 111 (that is, the entire imaging screen) before the setting unit 34b divides the area. The same imaging condition refers to setting a common imaging condition for the entire imaging screen. For example, even if there is a variation in apex value of less than about 0.3, it is regarded as the same. The imaging conditions set to be the same throughout the imaging chip 111 are determined based on the exposure conditions corresponding to the photometric value of the subject luminance or the exposure conditions manually set by the user.

5, an image including a person 61a, an automobile 62a, a bag 63a, a mountain 64a, and

clouds

65a and 66a is formed on the imaging surface of the imaging chip 111. The person 61a holds the bag 63a with both hands. The automobile 62a stops at the right rear of the person 61a.

<Division of area>
Based on the live view image, the control unit 34 divides the screen of the live view image into a plurality of regions as follows. First, a subject element is detected from the live view image by the object detection unit 34a. The subject element is detected using a known subject recognition technique. In the example of FIG. 5, the object detection unit 34a detects a person 61a, a car 62a, a bag 63a, a mountain 64a, a cloud 65a, and a cloud 66a as subject elements.

Next, the setting unit 34b divides the live view image screen into regions including the subject elements. In the present embodiment, the region including the person 61a is defined as the first region 61, the region including the automobile 62a is defined as the second region 62, the region including the bag 63a is defined as the third region 63, and the region including the mountain 64a is defined as the fourth region. The region 64, the fifth region including the cloud 65a will be referred to as a region 65, and the region including the cloud 66a will be described as a sixth region 66.

<Setting imaging conditions for each block>
When the setting unit 34b divides the screen into a plurality of areas, the control unit 34 causes the display unit 35 to display a setting screen as illustrated in FIG. In FIG. 6, a live view image 60a is displayed, and an imaging condition setting screen 70 is displayed on the right side of the live view image 60a.

The setting screen 70 lists frame rate, shutter speed (TV), and gain (ISO) in order from the top as an example of setting items for imaging conditions. The frame rate is the number of frames of a live view image acquired per second or a moving image recorded by the camera 1. Gain is ISO sensitivity. The setting items for the imaging conditions may be added as appropriate in addition to those illustrated in FIG. When all the setting items do not fit in the setting screen 70, other setting items may be displayed by scrolling the setting items up and down.

In the present embodiment, the control unit 34 sets the region selected by the user among the regions divided by the setting unit 34b as a target for setting (changing) the imaging condition. For example, in the camera 1 capable of touch operation, the user taps the display position of the main subject for which the imaging condition is to be set (changed) on the display surface of the display unit 35 on which the live view image 60a is displayed. For example, when the display position of the person 61 a is tapped, the control unit 34 sets the area 61 including the person 61 a in the live view image 60 a as an imaging condition setting (change) target area and emphasizes the outline of the area 61. To display.

In FIG. 6, an area 61 in which the outline is emphasized and displayed (bold display, bright display, display with a different color, display with a broken line, blink display, etc.) indicates an area for which the imaging condition is to be set (changed). . In the example of FIG. 6, it is assumed that a live view image 60a in which the outline of the region 61 is emphasized is displayed. In this case, the region 61 is a target for setting (changing) the imaging condition. For example, in the camera 1 capable of touch operation, when the user taps the shutter speed (TV) display 71, the control unit 34 displays the current shutter speed for the highlighted area (area 61). The set value is displayed on the screen (reference numeral 68).
In the following description, the camera 1 is described on the premise of a touch operation. However, the imaging condition may be set (changed) by operating a button or the like constituting the operation member 36.

When the upper icon 71a or the lower icon 71b of the shutter speed (TV) is tapped by the user, the setting unit 34b increases or decreases the shutter speed display 68 from the current setting value according to the tap operation. An instruction is sent to the imaging unit 32 (FIG. 1) so as to change the imaging condition of the unit area 131 (FIG. 3) of the imaging element 32a corresponding to the displayed area (area 61) in accordance with the tap operation. The decision icon 72 is an operation icon for confirming the set imaging condition. The setting unit 34b performs the setting (change) of the frame rate and gain (ISO) in the same manner as the setting (change) of the shutter speed (TV).

Although the setting unit 34b has been described as setting the imaging condition based on the user's operation, the setting unit 34b is not limited to this. The setting unit 34b may set the imaging condition based on the determination of the control unit 34 without being based on a user operation. For example, when the overexposure or underexposure occurs in the area including the subject element having the maximum luminance or the minimum luminance in the image, the setting unit 34b cancels the overexposure or underexposure based on the determination of the control unit 34. The imaging conditions may be set in.
For the area that is not highlighted (the area other than the area 61), the set imaging conditions are maintained.

Instead of highlighting the outline of the area for which the imaging condition is set (changed), the control unit 34 displays the entire target area brightly, increases the contrast of the entire target area, or displays the entire target area. May be displayed blinking. Further, the target area may be surrounded by a frame. The display of the frame surrounding the target area may be a double frame or a single frame, and the display mode such as the line type, color, and brightness of the surrounding frame may be appropriately changed. In addition, the control unit 34 may display an indication of an area for which an imaging condition is set, such as an arrow, in the vicinity of the target area. The control unit 34 may darkly display a region other than the target region for which the imaging condition is set (changed), or may display a low contrast other than the target region.

As described above, when an image capturing condition for each region is set and a release button (not shown) constituting the operation member 36 or a display (release icon) for instructing start of imaging is operated, the control unit 34 is operated. By controlling the imaging unit 32, imaging (main imaging) is performed under imaging conditions set for each of the divided areas. In the following description, the divided areas are referred to as a first area 61 to a sixth area 66 (see FIG. 7A), and the first imaging condition to the sixth imaging 66 are included in the first area 61 to the sixth area 66, respectively. A condition shall be set. The image processing unit 33 performs image processing on the image data acquired by the imaging unit 32. This image data is image data recorded in the recording unit 37 and is hereinafter referred to as main image data. Note that the image capturing unit 32 performs image processing on the main image data when acquiring the main image data, and image data used when performing various detection processes and setting processes for capturing the main image data. (Hereinafter referred to as processing image data) is acquired at a timing different from that of the main image data. Details of the image processing will be described later.

<Processing image data>
In the main image data, the processing image data is an image of a target pixel included in a region (hereinafter referred to as a boundary portion) in the vicinity of a boundary that forms a boundary with another region in a certain region (for example, the first region 61). Used when processing. Further, the processing image data is used in focus detection processing, subject detection processing, and imaging condition setting processing. The setting unit 34b of the control unit 34 captures an area wider than the first area 61 for processing image data for the first area 61 (hereinafter referred to as first processing image data). Hereinafter, it is set in the image pickup device 32a as a processing image pickup region). In this case, the setting unit 34b sets, for example, the entire area of the imaging surface of the imaging element 32a as the processing imaging area. Further, the setting unit 34b sets a first imaging condition that is an imaging condition set in the first region 61 as the imaging condition of the first processing image data. Similarly, the setting unit 34b sets the processing imaging areas for the second to sixth processing image data to the entire area of the imaging surface of the imaging element 32a. The setting unit 34b sets the imaging conditions set in the second area 62 to the sixth area 66 as imaging conditions for the second to sixth processing image data, respectively.
The timing for capturing the processing image data will be described later.
In the following, the processing image data will be described separately for the case where it is used for image processing, the case where it is used for focus detection processing, the case where it is used for subject detection processing, and the case where it is used for exposure condition setting processing.

1. When used for image processing A case where image data for processing is used for image processing will be described. The processing unit 33b of the image processing unit 33, when the image processing for the main image data acquired by applying different imaging conditions between the divided regions is predetermined image processing, the main image located at the boundary of the region Image processing is performed on the data using the processing image data. The predetermined image processing is processing for calculating data of a target position to be processed in an image with reference to data of a plurality of reference positions around the target position (hereinafter referred to as a target range). This includes defect correction processing, color interpolation processing, contour enhancement processing, noise reduction processing, and the like.

The image processing is performed to alleviate the uncomfortable feeling that appears in the image after the image processing due to the difference in imaging conditions between the divided areas. In general, when the target position is located at the boundary of the divided area, data in which the same imaging condition as the target position data is applied to a plurality of reference positions in the target range, and different imaging conditions from the target position data May be mixed with data to which. In the present embodiment, the reference position data to which the same imaging condition is applied is referred to as reference position data to which the same imaging condition is applied, and the reference position data to which the same imaging condition is applied is referred to as it is. Based on the idea that it is preferable to calculate data, image processing is performed as follows.

FIG. 7A is a diagram illustrating a predetermined range 80 including the first region 61 and the fourth region 64 that makes the boundary of the first region 61 in the live view image 60a. In this example, it is assumed that the first imaging condition is set in the first area 61 including at least a person, and the fourth imaging condition is set in the fourth area 64 including a mountain. FIG. 7B is an enlarged view of the predetermined range 80 in FIG. The image data from the pixels on the image sensor 32a corresponding to the first area 61 for which the first image capturing condition is set is shown in white, and the image data on the image sensor 32a corresponding to the fourth area 64 for which the fourth image capturing condition is set. The image data from the pixel is shaded. In FIG. 7B, the image data from the target pixel P is located on the first region 61 and in the vicinity of the boundary 81 between the first region 61 and the fourth region 64, that is, the boundary portion. Pixels around the target pixel P (eight pixels in this example) included in the target range 90 (for example, 3 × 3 pixels) centered on the target pixel P are set as reference pixels Pr. FIG. 7C is an enlarged view of the target pixel P and the reference pixels Pr1 to Pr8. The position of the target pixel P is the target position, and the positions of the reference pixels Pr1 to Pr8 surrounding the target pixel P are reference positions. In the following description, the reference symbol Pr is given to collectively refer to reference pixels.

The processing unit 33b of the image processing unit 33 performs image processing using the image data of the reference pixel Pr as it is. That is, the processing unit 33b performs image processing such as interpolation using all data of all reference pixels Pr of the target pixel P. However, when the first imaging condition applied at the time of imaging in the target pixel P and the fourth imaging condition applied at the time of imaging in the reference pixels Pr around the target pixel P are different from each other, the processing unit 33b The image data of the fourth imaging condition on the data is not used. In FIG. 7C, the image data output from the target pixel P and the reference pixels Pr1 to Pr6 for which the first imaging condition is set is shown in white, and is output from the reference pixels Pr7 and Pr8 for which the fourth imaging condition is set. The image data to be processed is indicated by diagonal lines. In the present embodiment, the processing unit 33b does not use the image data of the fourth imaging condition, that is, the image data output from the reference pixels Pr7 and Pr8 indicated by diagonal lines in the image processing. Further, the processing unit 33b refers to the first processing image data generated by setting the first imaging condition instead of the image data of the reference pixels Pr7 and Pr8 of the main image data captured under the fourth imaging condition. Image data of the pixels Pr7 and Pr8 is used for image processing.

FIG. 7D shows the pixels corresponding to the target pixel P and the reference pixel Pr of the main image data shown in FIG. 7C among the first processing image data, that is, the coordinates on the imaging surface of the imaging element 32a. Image data from pixels having the same value is shown. Since the first processing image data is captured under the first imaging condition, in FIG. 7D, pixel data output from the target pixel P and the reference pixel Pr is shown on a white background. The processing unit 33b selects the image data of the reference pixels Pr7 and Pr8 in the first processing image data instead of the image data of the reference pixels Pr7 and Pr8 in the main image data. The processing unit 33b refers to the image data of the reference pixels Pr1 to Pr6 in the main image data and the reference pixels Pr7 and Pr8 in the first processing image data, and determines the image data of the target pixel P of the main image data. calculate. That is, the processing unit 33b calculates the image data of the pixel of interest P using different image data captured with the same imaging condition set.

An example in which the first imaging condition and the fourth imaging condition are different is shown below.
(Example 1)
An example will be described in which only the ISO sensitivity differs between the first imaging condition and the fourth imaging condition, the ISO sensitivity of the first imaging condition is 100, and the ISO sensitivity of the fourth imaging condition is 800. Of the main image data, the target pixel P is acquired under the first imaging condition (ISO sensitivity 100). In this case, the processing unit 33b of the image processing unit 33 includes the image data of the reference pixels Pr1 to Pr6 of the main image data acquired under the first imaging condition among the reference pixels Pr, and the first acquired with the ISO sensitivity of 100. Image processing is performed using the image data of the reference pixels Pr7 and Pr8 (FIG. 7D) of the processing image data. The processing unit 33b does not use the image data of the reference pixels Pr7 and Pr8 acquired under the fourth imaging condition (ISO sensitivity 800) among the reference pixels Pr acquired under the first imaging condition.

(Example 2)
An example in which only the shutter speed is different between the first imaging condition and the fourth imaging condition, the shutter speed of the first imaging condition is 1/1000 second, and the shutter speed of the fourth imaging condition is 1/100 second. To Of the main image data, the target pixel P is acquired under the first imaging condition (shutter speed 1/1000 second). In this case, the processing unit 33b of the image processing unit 33 uses the image data of the reference pixels Pr1 to Pr6 of the main image data acquired at the shutter speed 1/1000 second out of the reference pixel Pr and the shutter speed at 1/1000 second. Image processing is performed using the image data of the reference pixels Pr7 and Pr8 of the acquired first processing image data. The processing unit 33b does not use the image data of the reference pixels Pr7 and Pr8 of the main image data acquired under the fourth imaging condition (shutter speed 1/100 seconds).

(Example 3)
An example in which only the frame rate is different between the first imaging condition and the fourth imaging condition (the charge accumulation time is the same), the frame rate of the first imaging condition is 30 fps, and the frame rate of the fourth imaging condition is 60 fps. To Of the main image data, the target pixel P is acquired under the first imaging condition (30 fps). In this case, the processing unit 33b of the image processing unit 33 refers to the image data of the reference pixels Pr1 to Pr6 of the main image data acquired at 30 fps out of the reference pixels Pr and the first processing image data acquired at 30 fps. Image processing is performed using the image data of the pixels Pr7 and Pr8. The processing unit 33b does not use the image data of the reference pixels Pr7 and Pr8 of the main image data acquired under the fourth imaging condition (60 fps).

On the other hand, the processing unit 33b of the image processing unit 33, when the first imaging condition applied at the time of imaging at the target pixel P and the imaging condition applied at all the reference pixels Pr around the target pixel P are the same. Uses data of the reference pixel Pr acquired under the first imaging condition for image processing. That is, the processing unit 33b uses the data of the reference pixel Pr as it is for image processing.
As described above, even if there are some differences in the imaging conditions, the imaging conditions are regarded as the same.

Note that the processing unit 33b performs the image data of the reference pixel Pr acquired under the first imaging condition and the first data acquired under the first imaging condition on the image data of the target pixel P acquired under the first imaging condition. The present invention is not limited to performing image processing using the image data of the reference pixel Pr of one processing image data. For example, the processing unit 33b does not use the image data of the reference pixels Pr1 to Pr8 of the main image data for the image data of the target pixel P acquired under the first imaging condition, but the reference pixels of the first processing image data. Image processing may be performed using image data of Pr1 to Pr8. That is, the processing unit 33b uses the image data of the reference pixel Pr imaged under the same imaging condition as the imaging condition set when the pixel of interest P is imaged, to the image data of the pixel of interest P. Image processing may be performed. *

<Example of image processing>
An example of image processing performed using the processing image data will be described.
(1) Imaging Pixel Defect Correction Process In the present embodiment, the imaging pixel defect correction process is one of image processes performed during imaging. In general, the image sensor 100, which is a solid-state image sensor, may generate pixel defects during the manufacturing process or after manufacturing, and output abnormal level data. Therefore, the processing unit 33b of the image processing unit 33 performs image processing on the image data output from the imaging pixel in which the pixel defect has occurred, so that the image data at the position of the imaging pixel in which the pixel defect has occurred is inconspicuous. Like that.

An example of the imaging pixel defect correction process will be described. For example, the processing unit 33b of the image processing unit 33 sets the pixel at the position of the pixel defect recorded in advance in a non-volatile memory (not shown) in the main image data as the target pixel P (processing target pixel), and the target pixel. A pixel (eight pixels in this example) around the target pixel P included in the target range 90 (for example, 3 × 3 pixels) centered on P is set as a reference pixel Pr.

The processing unit 33b of the image processing unit 33 calculates the maximum value and the minimum value of the image data in the reference pixel Pr. When the image data output from the target pixel P exceeds these maximum value or minimum value, the processing unit 33b starts from the target pixel P. Max and Min filter processing is performed to replace the output image data with the maximum value or the minimum value. Such processing is performed on the image data from the imaging pixels in which all the pixel defects whose position information is recorded in the nonvolatile memory (not shown) are generated.

In the present embodiment, the processing unit 33b of the image processing unit 33 refers to a pixel to which an imaging condition different from the imaging condition applied to the target pixel P at the time of imaging (the first imaging condition in the example of FIG. 7) is applied. When included in the pixel Pr (Pr7, Pr8 in FIG. 7), the image data of the reference pixel Pr (Pr7, Pr8 in FIG. 7) of the first processing image data is selected. Thereafter, the generation unit 33c of the image processing unit 33 uses the Max and Min filters described above using the image data of the reference pixels Pr1 to Pr6 of the main image data and the image data of the reference pixels Pr7 and Pr8 of the first processing image data. Process.

(2) Color Interpolation Processing In the present embodiment, color interpolation processing is one of image processing performed at the time of imaging. As illustrated in FIG. 3, in the imaging chip 111 of the imaging element 32 a, green pixels Gb and Gr, a blue pixel B, and a red pixel R are arranged in a Bayer array. The processing unit 33b of the image processing unit 33 lacks image data of a color component different from the color component of the color filter F arranged at each pixel position, so that the insufficient color component with reference to the image data of the surrounding pixel positions The color interpolation processing for generating the image data is performed.

An example of color interpolation processing will be described. FIG. 8A is a diagram illustrating the arrangement of image data output from the image sensor 32a. Corresponding to each pixel position, it has one of R, G, and B color components according to the rules of the Bayer array.
<G color interpolation>
The processing unit 33b of the image processing unit 33 that performs the G color interpolation refers to the image data of the four G color components at the reference positions around the target position, with the positions of the R color component and the B color component in turn as the target position. The G color component image data at the target position is generated. For example, when the G color component image data is generated at the target position indicated by the thick frame (second row, second column) in FIG. 8B, the four G color component image data G1 located in the vicinity of the target position. See G4. The image processing unit 33 (generation unit 33c) sets, for example, (aG1 + bG2 + cG3 + dG4) / 4 as image data of the G color component at the target position. Note that a to d are weighting coefficients provided according to the distance between the reference position and the target position and the image structure.

8A to 8C, for example, the first imaging condition is applied to the left and upper regions with respect to the thick line, and the first imaging condition is applied to the right and lower regions with respect to the thick line. Assume that different imaging conditions are applied. The processing unit 33b of the image processing unit 33 applies an imaging condition different from the first imaging condition applied to the target position to the reference position corresponding to the G color component image data G4 indicated by the oblique lines in FIG. Therefore, as the image data G4, the image data of the reference pixel Pr of the first processing image data is used. Thus, the processing unit 33b of the image processing unit 33 calculates the image data of the G color component at the target position using the image data of the reference pixels G1 to G4 to which the first imaging condition is applied.

As shown in FIG. 8C, the processing unit 33b of the image processing unit 33 generates image data of the G color component at the position of the B color component and the position of the R color component in FIG. The image data of the G color component can be obtained at each pixel position.

<R color interpolation>
FIG. 9A is a diagram obtained by extracting R color component image data from FIG. The processing unit 33b of the image processing unit 33 uses the color difference component shown in FIG. 9B based on the G color component image data shown in FIG. 8C and the R color component image data shown in FIG. 9A. Cr image data is calculated.

The processing unit 33b of the image processing unit 33 is located in the vicinity of the target position when generating image data of the color difference component Cr at the target position indicated by, for example, the thick frame (second row and second column) in FIG. 9B. Reference is made to the image data Cr1 to Cr4 of the four color difference components. The image processing unit 33 (processing unit 33b) uses, for example, (eCr1 + fCr2 + gCr3 + hCr4) / 4 as image data of the color difference component Cr at the target position. Note that e to h are weighting coefficients provided according to the distance between the reference position and the target position and the image structure.

Similarly, the processing unit 33b of the image processing unit 33 generates image data of the color difference component Cr at the target position indicated by the thick frame (second row and third column) in FIG. 9C, for example, in the vicinity of the target position. Reference is made to the image data Cr2, Cr4 to Cr6 of the four color-difference components located at. The processing unit 33b of the image processing unit 33 uses, for example, (qCr2 + rCr4 + sCr5 + tCr6) / 4 as image data of the color difference component Cr at the target position. Note that q to t are weighting coefficients provided according to the distance between the reference position and the target position and the image structure. In this way, image data of the color difference component Cr is generated for all pixels.

In FIG. 9A to FIG. 9C, for example, the first imaging condition is applied to the left and upper regions with respect to the thick line, and the first imaging condition is applied to the right and lower regions with respect to the thick line. Assume that different imaging conditions are applied. An imaging condition different from the first imaging condition applied to the target position (second row and second column) is applied to the reference position corresponding to the image data Cr2 of the color difference component Cr indicated by hatching in FIG. 9B. Yes. The image processing unit 33 (processing unit 33b) uses the image data of the reference pixel Pr of the first processing image data for the image data Cr2. Thereafter, the processing unit 33b of the image processing unit 33 calculates the image data of the color difference component Cr at the target position.
Further, the first imaging different from the imaging condition applied to the target position (second row and third column) at the reference position corresponding to the image data Cr4 and Cr5 of the color difference component Cr indicated by hatching in FIG. 9C. Conditions are applied. The image processing unit 33 (processing unit 33b) uses the image data of the reference pixel Pr of the processing image data for the image data Cr4 and Cr5. Thereafter, the processing unit 33b of the image processing unit 33 calculates the image data of the color difference component Cr at the target position.

The processing unit 33b of the image processing unit 33 obtains the image data of the color difference component Cr at each pixel position, and then adds the image data of the G color component shown in FIG. 8C corresponding to each pixel position. The image data of the R color component can be obtained at each pixel position.

<B color interpolation>
FIG. 10A is a diagram in which image data of the B color component is extracted from FIG. The processing unit 33b of the image processing unit 33 uses the color difference component shown in FIG. 10B based on the image data of the G color component shown in FIG. 8C and the image data of the B color component shown in FIG. Cb image data is calculated.

The processing unit 33b of the image processing unit 33 is located in the vicinity of the target position when generating image data of the color difference component Cb at the target position indicated by, for example, the thick frame (third row, third column) in FIG. Reference is made to the image data Cb1 to Cb4 of the four color difference components. The processing unit 33b of the image processing unit 33 uses, for example, (uCb1 + vCb2 + wCb3 + xCb4) / 4 as the image data of the color difference component Cb at the target position. Note that u to x are weighting coefficients provided according to the distance between the reference position and the target position and the image structure.

Similarly, the processing unit 33b of the image processing unit 33 generates the image data of the color difference component Cb at the target position indicated by the thick frame (third row and fourth column) in FIG. 10C, for example, in the vicinity of the target position. Reference is made to the image data Cb2, Cb4 to Cb6 of the four color-difference components located at. The processing unit 33b of the image processing unit 33 uses, for example, (yCb2 + zCb4 + αCb5 + βCb6) / 4 as image data of the color difference component Cb at the target position. Note that y, z, α, and β are weighting coefficients provided in accordance with the distance between the reference position and the target position and the image structure. In this way, image data of the color difference component Cb is generated for all pixels.

10 (a) to 10 (c), for example, the first imaging condition is applied to the left and upper regions with respect to the thick line, and the first imaging condition is applied to the right and lower regions with respect to the thick line. Assume that different imaging conditions are applied. The processing unit 33b of the image processing unit 33 is applied to the target position (third row, third column) at the reference position corresponding to the image data Cb1 and Cb3 of the color difference component Cb indicated by the oblique lines in FIG. Since the first imaging condition different from the imaging condition is applied, the image data of the reference pixel Pr of the processing image data is used for the data Cb1 and Cb3. Thereafter, the generation unit 33c of the image processing unit 33 calculates the image data of the color difference component Cb at the position of interest.
When calculating the image data of the color difference component Cb at the position of interest (third row, fourth column) in FIG. 10C, it corresponds to the image data Cb2, Cb4 to Cb6 of the four color difference components located in the vicinity of the position of interest. The same imaging conditions as the target position are applied to the reference position. The generation unit 33c of the image processing unit 33 calculates the image data of the color difference component Cb at the target position.

The processing unit 33b of the image processing unit 33 obtains the image data of the color difference component Cb at each pixel position, and then adds the image data of the G color component shown in FIG. 8C corresponding to each pixel position. The image data of the B color component can be obtained at each pixel position.

(3) Outline Enhancement Process An example of the outline enhancement process will be described. The processing unit 33b of the image processing unit 33 performs, for example, a known linear filter calculation using a kernel of a predetermined size centered on the target pixel P (processing target pixel) in one frame image. When the kernel size of a sharpening filter that is an example of a linear filter is N × N pixels, the position of the target pixel P is the target position, and the positions of (N ² −1) reference pixels surrounding the target pixel P are referred to. Position.
The kernel size may be N × M pixels.

The processing unit 33b of the image processing unit 33 performs a filter process for replacing the data in the target pixel P with the linear filter calculation result, for example, from the upper horizontal line to the lower horizontal line of the frame image, and the target pixel on each horizontal line. Shift from left to right.

In the present embodiment, the processing unit 33b of the image processing unit 33 performs the first imaging when the reference pixel Pr includes a pixel to which an imaging condition different from the first imaging condition applied to the target pixel P is applied. For pixels to which imaging conditions different from the conditions are applied, the image data of the reference pixel Pr of the first processing image data is used. Thereafter, the generation unit 33c of the image processing unit 33 performs the linear filter processing described above.

(4) Noise reduction processing An example of noise reduction processing will be described. The processing unit 33b of the image processing unit 33 performs, for example, a known linear filter calculation using a kernel of a predetermined size centered on the target pixel P (processing target pixel) in one frame image. When the kernel size of the smoothing filter which is an example of the linear filter is N × N pixels, the position of the target pixel P is the target position, and the positions of the (N ² −1) reference pixels surrounding the target pixel P are referred to. Position.
The kernel size may be N × M pixels.

In the present embodiment, the processing unit 33b of the image processing unit 33 performs processing when the reference pixel Pr includes a pixel to which an imaging condition different from the first imaging condition applied to the target pixel P during imaging is included. For pixels to which an imaging condition different from the one imaging condition is applied, the image data of the reference pixel Pr of the first processing image data is used. Thereafter, the processing unit 33b of the image processing unit 33 performs the linear filter processing described above.

In the above description, the setting unit 34b has been described as setting the entire area of the imaging surface of the imaging element 32a as a processing imaging area, but the present embodiment is not limited to this example. The setting unit 34b may set a partial area of the imaging surface of the imaging element 32a as a processing imaging area. For example, the setting unit 34b sets an area corresponding to the first area 61 of the main image data as the processing imaging area, and applies the first imaging condition. In this case, the setting unit 34b sets, as the processing imaging region, the first region 61 and an outer region that is spread outward by, for example, a predetermined number of pixels from the outer periphery of the first region 61. The processing unit 33b divides the area on the imaging surface of the imaging element 32a in correspondence with each of the second area 62 to the sixth area 66 for areas other than the processing imaging area corresponding to the first area 61, and 2 imaging conditions to 6th imaging conditions are applied. In other words, the setting unit 34b applies the first imaging condition to the processing imaging area on the imaging surface including the area where the first imaging condition is set and the area where the other imaging conditions are set, and the imaging for processing is performed. An imaging condition different from the first imaging condition is applied to an area other than the area.
For each of the second to sixth processing image data, the setting unit 34b similarly uses the processing imaging region set as an area wider than the second region 62 to the sixth region 66, and uses the processing unit 32b. Let's take an image.

Alternatively, the setting unit 34b sets each region set in the main image data and the above-described outer region of each region from the image data captured by setting the entire region of the imaging surface of the image sensor 32a as the processing imaging region. It is also possible to extract the image data relating to the image data for processing. For example, the setting unit 34b sets the first imaging condition and sets the first area 61 of the main image data and the outside of the first area 61 from the image data captured using the entire area of the imaging surface of the imaging element 32a. Image data extracted from the region is generated as first processed image data. Also for the second processing image data to the sixth processing image data, the setting unit 34b applies the second imaging condition to the sixth imaging condition and applies each of the image data captured using the entire area of the imaging surface. The image data from the areas corresponding to the second area 62 to the sixth area 66 may be extracted as processing image data.

Further, the setting unit 34b is not limited to the one that sets the processing imaging area corresponding to each area set in the main image data. A processing imaging area may be set in advance in a partial area of the imaging surface of the imaging element 32a. For example, when the processing imaging area is set near the center of the imaging surface of the imaging device 32a, the main subject is positioned when the person is imaged at the center of the screen as in portraits. It is possible to generate image data for processing in an area where there is a high possibility of this. In this case, the size of the imaging region for processing may be changeable based on a user operation, or may be fixed at a preset size.

2. When performing focus detection processing A case where processing image data is used for focus detection processing will be described. The setting unit 34b performs focus detection processing using processing image data that is captured with the same imaging condition set in the entire area of the imaging surface of the imaging element 32a. This is because when the focus point of the AF operation, that is, the focus detection area is divided into the first and second regions having different imaging conditions, the accuracy of the focus detection processing by the AF calculation unit 34d may be reduced. For example, image data to which different imaging conditions are applied may be mixed in image data for focus detection that detects an image shift amount (phase difference) in an image. In this embodiment, rather than detecting image shift amount (phase difference) using image data to which different imaging conditions are applied as it is, image data having no difference between image data due to differences in imaging conditions is used. Based on the idea that it is preferable to detect the amount of image shift (phase difference), focus detection processing using image data for processing is performed.

<Example of focus detection processing>
In the AF operation of the present embodiment, for example, the subject corresponding to the focus point selected by the user from a plurality of focus points on the imaging screen is focused. The AF calculation unit 34d of the control unit 34 detects the image shift amounts (phase differences) of the plurality of subject images due to the light beams that have passed through different pupil regions of the imaging optical system 31, thereby reducing the defocus amount of the imaging optical system 31. calculate. The AF calculation unit 34d of the control unit 34 moves the focus lens of the imaging optical system 31 to a position where the defocus amount is zero (allowable value or less), that is, a focus position.

FIG. 11 is a diagram illustrating the position of the focus detection pixel on the imaging surface of the imaging device 32a. In the present embodiment, focus detection pixels are discretely arranged along the X-axis direction (horizontal direction) of the imaging chip 111. In the example of FIG. 11, 15 focus detection pixel lines 160 are provided at a predetermined interval. The focus detection pixels constituting the focus detection pixel line 160 output a photoelectric conversion signal for focus detection. In the imaging chip 111, normal imaging pixels are provided at pixel positions other than the focus detection pixel line 160. The imaging pixel outputs a live view image or a photoelectric conversion signal for recording.

FIG. 12 is an enlarged view of a part of the focus detection pixel line 160 corresponding to the focus point 80A shown in FIG. In FIG. 12, a red pixel R, a green pixel G (Gb, Gr), and a blue pixel B, a focus detection pixel S1, and a focus detection pixel S2 are illustrated. The red pixel R, the green pixel G (Gb, Gr), and the blue pixel B are arranged according to the rules of the Bayer arrangement described above.

The square area illustrated for the red pixel R, the green pixel G (Gb, Gr), and the blue pixel B indicates a light receiving area of the imaging pixel. Each imaging pixel receives a light beam passing through the exit pupil of the imaging optical system 31 (FIG. 1). That is, the red pixel R, the green pixel G (Gb, Gr), and the blue pixel B each have a square-shaped mask opening, and light passing through these mask openings reaches the light-receiving portion of the imaging pixel. .
In addition, the shape of the light receiving region (mask opening) of the red pixel R, the green pixel G (Gb, Gr), and the blue pixel B is not limited to a quadrangle, and may be, for example, a circle.

The semicircular region exemplified for the focus detection pixel S1 and the focus detection pixel S2 indicates a light receiving region of the focus detection pixel. That is, the focus detection pixel S1 has a semicircular mask opening on the left side of the pixel position in FIG. 12, and the light passing through the mask opening reaches the light receiving portion of the focus detection pixel S1. On the other hand, the focus detection pixel S2 has a semicircular mask opening on the right side of the pixel position in FIG. 12, and the light passing through the mask opening reaches the light receiving portion of the focus detection pixel S2. As described above, the focus detection pixel S1 and the focus detection pixel S2 respectively receive a pair of light beams passing through different areas of the exit pupil of the imaging optical system 31 (FIG. 1).

Note that the position of the focus detection pixel line 160 in the imaging chip 111 is not limited to the position illustrated in FIG. Further, the number of focus detection pixel lines 160 is not limited to the example of FIG. Further, the shape of the mask opening in the focus detection pixel S1 and the focus detection pixel S2 is not limited to a semicircular shape. For example, a rectangular light receiving region (mask opening) in the imaging pixel R, the imaging pixel G, and the imaging pixel B is used. Part) may be a rectangular shape divided in the horizontal direction.

Further, the focus detection pixel line 160 in the imaging chip 111 may be a line in which focus detection pixels are arranged along the Y-axis direction (vertical direction) of the imaging chip 111. An imaging element in which imaging pixels and focus detection pixels are two-dimensionally arranged as shown in FIG. 12 is known, and detailed illustration and description of these pixels are omitted.

In the example of FIG. 12, the configuration in which the focus detection pixels S1 and S2 each receive one of the pair of focus detection light beams, the so-called 1PD structure, has been described. Instead of this, the focus detection pixels may be configured to receive both of a pair of light beams for focus detection, that is, a so-called 2PD structure. By using the 2PD structure, the photoelectric conversion signal obtained from the focus detection pixel can be used as a recording photoelectric conversion signal.

The AF calculation unit 34d of the control unit 34 is a pair that passes through different regions of the imaging optical system 31 (FIG. 1) based on the focus detection photoelectric conversion signals output from the focus detection pixel S1 and the focus detection pixel S2. The amount of image misalignment (phase difference) between the pair of images due to the luminous flux is detected. Then, the defocus amount is calculated based on the image shift amount (phase difference). Such defocus amount calculation by the pupil division phase difference method is well known in the field of cameras, and thus detailed description thereof is omitted.

The focus point 80A (FIG. 11) is selected by the user at a position corresponding to the predetermined range 80 in the first area 61 in the live view image 60a illustrated in FIG. FIG. 13 is an enlarged view of the focus point 80A. A white background pixel indicates that the first imaging condition is set, and a shaded pixel indicates that the fourth imaging condition is set. In FIG. 13, the position surrounded by the frame 170 corresponds to the focus detection pixel line 160 (FIG. 11).

The AF calculation unit 34d of the control unit 34 normally performs focus detection processing using the image data of the focus detection pixels indicated by the frame 170 as they are. However, when image data to which the first imaging condition is applied and image data to which the fourth imaging condition is applied are mixed in the image data surrounded by the frame 170, the AF calculation unit 34d of the control unit 34 A focus detection process is performed using the first processing image data to which the imaging condition is applied. In this case, the AF calculation unit 34d of the control unit 34 uses image data corresponding to the range surrounded by the frame 170 in the first processing image data.

An example in which the first imaging condition and the fourth imaging condition are different is shown below.
(Example 1)
An example will be described in which only the ISO sensitivity differs between the first imaging condition and the fourth imaging condition, the ISO sensitivity of the first imaging condition is 100, and the ISO sensitivity of the fourth imaging condition is 800. Part of the image data surrounded by the frame 170 is acquired under the first imaging condition (ISO sensitivity 100), and the rest is acquired under the fourth imaging condition (ISO sensitivity 800). In this case, the AF calculation unit 34d of the control unit 34 performs focus detection processing using image data corresponding to a range surrounded by the frame 170 in the first processing image data to which the first imaging condition is applied. .

(Example 2)
An example in which only the shutter speed is different between the first imaging condition and the fourth imaging condition, the shutter speed of the first imaging condition is 1/1000 second, and the shutter speed of the fourth imaging condition is 1/100 second. To Part of the image data enclosed by the frame 170 is acquired under the first imaging condition (shutter speed 1/1000 seconds), and the rest is acquired under the fourth imaging condition (shutter speed 1/100 seconds). In this case, the AF calculation unit 34d of the control unit 34 corresponds to the image data corresponding to the range surrounded by the frame 170 in the first processing image data to which the first imaging condition (shutter speed 1/1000 second) is applied. Is used to perform focus detection processing.

(Example 3)
An example in which only the frame rate is different between the first imaging condition and the fourth imaging condition (the charge accumulation time is the same), the frame rate of the first imaging condition is 30 fps, and the frame rate of the fourth imaging condition is 60 fps. To Part of the image data surrounded by the frame 170 is acquired under the first imaging condition (30 fps), and the rest is acquired under the fourth imaging condition (60 fps). In this case, the AF calculation unit 34d of the control unit 34 uses the image data corresponding to the range surrounded by the frame 170 in the first processing image data to which the first imaging condition (30 fps) is applied. Process.

On the other hand, the AF calculation unit 34d of the control unit 34 may not use the processing image data when the imaging conditions applied in the image data surrounded by the frame 170 are the same. That is, the AF calculation unit 34d of the control unit 34 performs the focus detection process using the image data of the focus detection pixels indicated by the frame 170 as they are.
As described above, even if there are some differences in the imaging conditions, the imaging conditions are regarded as the same.

In the above description, the focus detection process using the pupil division phase difference method is exemplified. However, in the case of the contrast detection method in which the focus lens of the imaging optical system 31 is moved to the in-focus position based on the contrast of the subject image. Can be done in the same way.

When the contrast detection method is used, the control unit 34 moves the focus lens of the imaging optical system 31 and outputs image data output from the imaging pixels of the imaging element 32a corresponding to the focus point at each position of the focus lens. Based on this, a known focus evaluation value calculation is performed. Then, the position of the focus lens that maximizes the focus evaluation value is obtained as the focus position.

The control unit 34 normally performs a focus evaluation value calculation using the image data output from the imaging pixel corresponding to the focus point as it is. However, when the image data corresponding to the focus point includes image data to which the first imaging condition is applied and image data to which an imaging condition different from the first imaging condition is mixed, the control unit 34 A focus evaluation value calculation is performed using the first processing image data to which one imaging condition is applied.

In the above description, the setting unit 34b has been described as setting the entire area of the imaging surface of the imaging element 32a as a processing imaging area, but the present embodiment is not limited to this example. The setting unit 34b may set a partial area of the imaging surface of the imaging element 32a as a processing imaging area. For example, the setting unit 34b sets a region including the frame 170, a region corresponding to the range including the focus point, and the vicinity of the center of the imaging surface of the imaging element 32a as the processing imaging region.
Alternatively, the setting unit 34b sets an entire area on the imaging surface of the imaging element 32a as a processing imaging area, and an area corresponding to a range including the frame 170 or a range including the focus point from the captured image data. May be extracted as image data for processing.

3. FIG. 14A is a diagram illustrating a template image representing an object to be detected, and FIG. 14B illustrates a live view image 60 (a) and a search range 190. It is a figure to do. The object detection unit 34a of the control unit 34 detects an object (for example, the bag 63a which is one of the subject elements in FIG. 5) from the live view image. The object detection unit 34a of the control unit 34 may set the range in which the object is detected as the entire range of the live view image 60a. However, in order to reduce the detection process, a part of the live view image 60a may be used as the search range 190. Good.

When the search range 190 is divided by a plurality of regions having different imaging conditions, the object detection unit 34a of the control unit 34 obtains processing image data that has been captured with the same imaging conditions set in all the regions of the imaging element 32a. And subject detection processing.
In general, when the search range 190 used for detecting the subject element includes a boundary between two regions, image data to which different imaging conditions are applied may be mixed in the image data of the search range 190. In the present embodiment, rather than performing detection of subject elements using image data to which different imaging conditions are applied as they are, image data having no difference between image data due to differences in imaging conditions is used in the search range 190. Based on the idea that it is preferable to detect the subject element, subject detection processing using the processing image data is performed.

A case will be described in which the bag 63a which is the belonging of the person 61a is detected in the live view image 60a illustrated in FIG. The object detection unit 34a of the control unit 34 sets the search range 190 in the vicinity of the region including the person 61a. The first area 61 including the person 61a may be set as the search range. As described above, the third imaging condition is set in the third region 63.

The object detection unit 34a of the control unit 34 performs subject detection processing using the image data constituting the search range 190 as it is when the search range 190 is not divided by two regions having different imaging conditions. However, if the image data in the search range 190 includes image data to which the first imaging condition is applied and image data to which the third imaging condition is applied, the object detection unit 34a of the control unit 34 Processing image data is used. In this case, the subject detection unit 34a of the control unit 34 performs subject detection processing using the third processing image data imaged under the third imaging condition applied to the third region 63 corresponding to the bag 63a. Note that the set imaging conditions are the same as (Example 1) to (Example 3) described above for the case of performing focus detection processing.

You may apply the image data of the search range 190 mentioned above with respect to the search range used in order to detect a specific subject like a person's face, and the area | region used for determination of an imaging scene.
Further, not only the search range used for the pattern matching method using the template image for the image data of the search range 190 described above, but also in the search range when detecting the feature amount based on the color or edge of the image. May apply.

In addition, by performing a known template matching process using image data of a plurality of frames having different acquisition times such as a live view image, a region similar to the tracking target in the previously acquired frame image is acquired later. You may apply to the tracking process of the mobile body searched from the obtained frame image. In this case, when different imaging conditions are mixedly applied in the search range set for the frame image acquired later, the control unit 34 searches for the search range in the processing image data. Tracking processing is performed using the image data.

The same applies to the case where a known motion vector is detected using image data of a plurality of frames having different acquisition times. When different imaging conditions are applied in the detection region used for detecting the motion vector in the image data, the control unit 34 detects the detection region used for detecting the motion vector in the processing image data. A motion vector is detected using the image data.

In the above description, the setting unit 34b has been described as setting the entire area of the imaging surface of the imaging element 32a as a processing imaging area, but the present embodiment is not limited to this example. The setting unit 34b may set a partial area of the imaging surface of the imaging element 32a as a processing imaging area. For example, the setting unit 34b uses the range including the search range 190, the region corresponding to the range including the detection range used for detecting the motion vector, and the vicinity of the center of the imaging surface of the imaging element 32a as the processing imaging region. Set.
Alternatively, the setting unit 34b sets a range including the search range 190 and a detection range used for detecting a motion vector from image data captured by setting the entire area of the imaging surface of the imaging element 32a as a processing imaging area. The processing image data may be generated by extracting the included range.

4). When setting the imaging conditions When the setting unit 34b of the control unit 34 divides the area of the imaging screen and sets different imaging conditions between the divided areas, the exposure condition is determined by newly performing photometry, For image data located near the boundary of the region, processing image data is used. For example, when the photometric range set in the center of the imaging screen includes the boundary of the divided areas, image data to which different imaging conditions are applied may be mixed in the photometric range image data. In the present embodiment, the exposure calculation processing is performed using image data having no difference between the image data due to the difference in the imaging conditions, rather than performing the exposure calculation processing using the image data to which the different imaging conditions are applied as it is. Based on the idea that it is preferable, exposure calculation processing using processing image data is performed.

The setting unit 34b of the control unit 34 performs the exposure calculation process using the image data constituting the photometric range as it is when the photometric range is not divided by a plurality of areas having different imaging conditions. However, if image data to which the first imaging condition is applied and image data to which the fourth imaging condition is applied are mixed in the image data in the photometric range (for example, an area including the boundary 80 in FIG. 7). The setting unit 34b of the control unit 34 uses processing image data. In this case, the setting unit 34 b of the control unit 34 performs the exposure calculation process using the first processing image data imaged under the first imaging condition applied to the first region 61. Note that the set imaging conditions are the same as (Example 1) to (Example 3) described above for the case of performing focus detection processing.

Not only the photometric range when performing the exposure calculation process described above, but also the photometric (colorimetric) range used when determining the white balance adjustment value and the necessity of emission of the auxiliary photographing light by the light source that emits the auxiliary photographing light are determined. The same applies to the photometric range performed at the time, and further to the photometric range performed at the time of determining the light emission amount of the photographing auxiliary light by the light source.

Further, in the case where the readout resolution of the image signal is different between the areas obtained by dividing the imaging screen, the area used for the determination of the imaging scene performed when determining the readout resolution for each area can be similarly handled. .

In the above description, the setting unit 34b has been described as setting the entire area of the imaging surface of the imaging element 32a as a processing imaging area, but the present embodiment is not limited to this example. The setting unit 34b may set a partial area of the imaging surface of the imaging element 32a as a processing imaging area. For example, the setting unit 34b sets a region corresponding to a range including the photometric range or a vicinity of the center of the imaging surface of the imaging element 32a as the processing imaging region.
Alternatively, the setting unit 34b sets the entire area of the imaging surface of the image sensor 32a as a processing imaging area and extracts a range that includes a photometric range from the captured image data to obtain each processing image data. It may be generated.

The timing for generating the processing image data used in the various processes described above will be described. Hereinafter, the timing for generating processing image data used for image processing and the timing for generating processing image data used for focus detection processing, subject detection processing, and exposure condition setting processing (hereinafter referred to as detection / setting processing). Separately, it explains.

<Generation of processing image data used for image processing>
The imaging control unit 34c causes the processing image data to be captured at a timing different from the timing at which the imaging unit 32 captures the main image data. In the present embodiment, the imaging control unit 34c causes the imaging unit 32 to capture the processing image data when the live view image is displayed or when the operation member 36 is operated. Further, the imaging control unit 34c outputs information on the imaging conditions set in the processing image data by the setting unit 34b when instructing imaging of the processing image data. Hereinafter, description will be made separately on imaging of processing image data when displaying a live view image and imaging of processing image data when operating the operation member 36.

(1) At the time of displaying a live view image The imaging control unit 34c causes the imaging unit 32 to capture image data for processing after an operation for instructing the display start of the live view image is performed by the user. In this case, the imaging control unit 34c causes the imaging unit 32 to capture the processing image data for each predetermined period while the live view image is displayed. For example, in the frame rate of the live view image, the imaging control unit 34c, for example, at the timing of capturing an even frame or the next timing of capturing a 10 frame live view image, instead of an instruction to capture the live view image. Then, a signal instructing imaging of the processing image data is output to the imaging unit 32.
At this time, the imaging control unit 34c causes the imaging unit 32 to capture the processing image data under the imaging conditions set by the setting unit 34b.

FIG. 15 is used to illustrate the relationship between the timing of capturing image data for a live view image and the timing of capturing image data for processing. FIG. 15A shows a case where imaging of a live view image and imaging of processing image data are alternately performed every other frame. It is assumed that the first imaging condition to the third imaging condition are set for the first area 61 to the third area 63 (FIG. 7A) by the user's operation. At this time, the first processing image data D1 in which the first imaging condition is set and the second imaging condition in which the second imaging condition is set are used for processing each of the first area 61 to the third area 63 of the main image data. The second processing image data D2 and the third processing image data D3 for which the third imaging condition is set are captured.

The imaging control unit 34c instructs the imaging unit 32 to capture the live view image LV1 of the Nth frame, and the control unit 34 causes the display unit 35 to display the live view image LV1 obtained by the imaging. At the timing of imaging of the (N + 1) th frame, the imaging control unit 34c instructs the imaging unit 32 to image the first processing image data D1 to which the first imaging condition is applied. The imaging control unit 34c records the captured first processing image data D1 on a predetermined recording medium (not shown). In this case, the control unit 34 causes the display unit 35 to display the live view image LV1 captured at the imaging timing of the Nth frame as the live view image of the (N + 1) th frame. That is, the display of the live view image LV1 of the previous frame is continued.

At the timing of imaging the live view image LV2 of the (N + 2) th frame, the imaging control unit 34c instructs the imaging unit 32 to image the live view image LV2 of the (N + 2) th frame. The control unit 34 switches from displaying the live view image LV1 on the display unit 35 to displaying the live view image LV2 obtained by imaging the (N + 2) th frame. At the timing of imaging of the (N + 3) th frame, the imaging control unit 34c causes the imaging unit 32 to image the second processing image data D2 to which the second imaging condition is applied, and records the captured second processing image data D2. To do. Also in this case, the control unit 34 continues to display the live view image LV <b> 2 captured at the imaging timing of the (N + 2) th frame as the live view image of the (N + 3) th frame on the display unit 35.

In the N + 4th frame, as in the case of the Nth and N + 2th frames, the imaging control unit 34c causes the imaging unit 32 to capture the live view image LV3, and the control unit 34 displays the captured live view image LV3 on the display unit. 35. In the (N + 5) th frame, the imaging control unit 34c causes the imaging unit 32 to capture the third processing image data D3 to which the third imaging condition is applied. At this time, similarly, the control unit 34 continues to display the live view image LV3 in the (N + 4) th frame on the display unit 35. In the subsequent frames, the control unit 34 repeatedly performs the processes in the Nth to N + 5th frames.

When the setting unit 34b changes the imaging condition based on the detection result of the object detection unit 34a or the calculation result of the AF calculation unit 34d, the imaging control unit 34c captures image data for processing (see FIG. The imaging unit 32 may perform imaging by applying imaging conditions newly set in the 15th (a) (N + 1, N + 3, N + 5 frames).

Note that the imaging control unit 34c may cause the imaging unit 32 to capture the processing image data before starting to display the live view image. For example, when the user turns on the power of the camera 1 or when an operation for instructing to start displaying the live view image is performed, a signal instructing the imaging unit 32 to image the processing image data is output. The imaging control unit 34c instructs the imaging unit 32 to capture the live view image when imaging of the first to third processing image data is completed.

For example, as illustrated in FIG. 15B, the imaging control unit 34c captures the first processing image data D1 under the first imaging condition and the second imaging condition under the timing at which the first to third frames are imaged. The second processing image data D2 and the third processing image data D3 under the third imaging condition are respectively captured. After the fourth frame, the imaging control unit 34c causes the live view images LV1, LV2, LV3,... To be captured, and the control unit 34 causes the display unit 35 to sequentially display the live view images LV1, LV2, LV3,.

Further, the imaging control unit 34c may cause the imaging unit 32 to capture the processing image data at the timing when the user performs an operation to end the display of the live view image while the live view image is displayed. That is, when an operation signal corresponding to an operation for instructing the end of display of the live view image is input from the operation member 36, the imaging control unit 34c outputs a signal for instructing the imaging unit 32 to end the imaging of the live view image. When the imaging unit 32 finishes capturing the live view image, the imaging control unit 32c outputs a signal instructing imaging of the processing image data to the imaging unit 32.

For example, as shown in FIG. 15C, when an operation to end the display of the live view image is performed in the Nth frame, the imaging control unit 34c sets the first imaging condition in the (N + 1) th to (N + 3) th frames. The first processing image data D1, the second processing image data D2 under the second imaging condition, and the third processing image data D3 under the third imaging condition are each captured. In this case, the control unit 34 may display the live view image LV1 captured in the Nth frame on the display unit 35 during the period of the (N + 1) th to (N + 3) th frames, or may not display the live view image. Also good.

Further, for all frames of the live view image, the imaging control unit 34c may capture the processing image data. In this case, the setting unit 34b sets different imaging conditions for each frame in the entire area of the imaging surface of the imaging element 32a. The control unit 34 displays the generated processing image data on the display unit 35 as a live view image.

When the composition of the image being captured changes while the live view image is displayed, the imaging control unit 34c may capture the processing image data. For example, the imaging control is performed when the position of the subject element detected by the setting unit 34b of the control unit 34 based on the live view image is shifted by a predetermined distance or more compared to the position of the subject element detected in the previous frame. The unit 34c may instruct to capture the processing image data.

(2) Operation of the operation member 36 The operation of the operation member 36 for imaging the processing image data includes half-pressing of the release button by the user, that is, an operation for instructing preparation for imaging, full-pressing operation of the release button, That is, an operation for instructing the main imaging can be given.

(2-1) Release Button Half-Pressing When the user performs a half-pressing operation on the release button, that is, an operation for instructing preparation for imaging, an operation signal is output from the operation member 36. This operation signal is output from the operation member 36 during a period when the release button is pressed halfway by the user. When an operation signal corresponding to the start of an operation for instructing preparation for imaging is input from the operation member 36, the imaging control unit 34c of the control unit 34 outputs a signal for instructing imaging of the processing image data to the imaging unit 32. That is, the imaging unit 32 captures the processing image data in response to the start of an operation instructing the preparation for imaging by the user.

Note that the imaging control unit 34c may cause the imaging unit 32 to capture the processing image data at a timing when the operation of the release button by the user ends, for example, by shifting from a half-press operation to a full-press operation. That is, the imaging control unit 36c may output a signal for instructing imaging to the imaging unit 32 at a timing when an operation signal corresponding to an operation for instructing preparation for imaging is not input from the operation member 36.

Further, the imaging control unit 34c may cause the imaging unit 32 to capture the processing image data while the release button is pressed halfway by the user. In this case, the imaging control unit 34c can output a signal instructing imaging to the imaging unit 32 at every predetermined period. Thus, the processing image data can be captured while the release button is half-pressed by the user. Alternatively, the imaging control unit 34c may output a signal for instructing imaging to the imaging unit 32 in accordance with the timing at which the live view image is captured. In this case, the imaging control unit 34c instructs the imaging of the processing image data at the timing of capturing an even frame, for example, at the next timing after capturing the live view image of 10 frames, of the frame rate of the live view image. May be output to the imaging unit 32.
Further, when the processing image data is captured while the live view image is displayed, the processing image data may not be captured based on the half-press operation of the release button.

(2-2) Release button full-press operation When the user performs a full-press operation of the release button, that is, an operation for instructing main imaging, an operation signal is output from the operation member 36. When an operation signal corresponding to an operation for instructing main imaging is input from the operation member 36, the imaging control unit 34c of the control unit 34 outputs a signal for instructing main imaging to the imaging unit 32. After the main image data is picked up by the main image pickup, the image pickup control unit 34c outputs a signal instructing to pick up the processing image data. That is, after an operation for instructing imaging is performed by the user, the imaging unit 32 captures the main image data by the main imaging and then captures the processing image data.
Note that the imaging control unit 34c may cause the imaging unit 32 to capture the processing image data before capturing the main image data. Further, when the processing image data is captured while the live view image is displayed, the processing image data may not be captured based on the half-press operation of the release button.

Note that the operation of the operation member 36 for capturing the processing image data is not limited to a half-press operation or a full-press operation of the release button. For example, when an operation related to imaging other than the operation of the release button is performed by the user, the imaging control unit 34c may instruct imaging of the processing image data. As operations related to imaging, there are, for example, an operation for changing an imaging magnification, an operation for changing an aperture, an operation related to focus adjustment (for example, selection of a focus point), and the like. When the change operation is completed and the new setting is confirmed, the imaging control unit 34c causes the imaging unit 32 to capture the processing image data. As a result, even when main imaging is performed under new settings, it is possible to generate processing image data imaged under the same conditions as the main imaging.

Note that when an operation is performed on the menu screen, the imaging control unit 34c may capture the processing image data. This is because, when an operation related to imaging is performed from the menu screen, there is a high possibility that new settings will be made for the actual imaging. In this case, the imaging unit 32 captures the processing image data while the menu screen is open. The processing image data may be captured at predetermined intervals, or may be captured at a frame rate when capturing a live view image.

When an operation not related to imaging, for example, an operation for reproducing and displaying an image, an operation during reproduction display, or an operation for clock adjustment is performed, the imaging control unit 34c captures image data for processing. Does not output the instruction signal. That is, the imaging control unit 34c does not cause the imaging unit 32 to capture the processing image data. Thereby, it is possible to prevent the image data for processing from being imaged when the possibility that a new setting for the main imaging is performed is low or the possibility that the main imaging is performed is low.

When the operation member 36 has a dedicated button for instructing to capture the processing image data, the imaging control unit 34c causes the imaging unit 32 to capture the processing image data when the dedicated button is operated by the user. Instruct. When the operation member 36 continues to output the operation signal while the user operates the dedicated button, the imaging control unit 34c performs the predetermined period during the period when the dedicated button is operated. Further, the image data for processing may be imaged by the imaging unit 32, or the image data for processing may be imaged when the operation of the dedicated button is completed. As a result, the processing image data can be captured at a timing desired by the user.
Further, when the power of the camera 1 is turned on, the imaging control unit 34c may instruct the imaging unit 32 to capture the processing image data.

In the camera 1 of the present embodiment, all of the exemplified methods may be applied to capture the processing image data, or at least one method may be performed, and the user selects from each method. A method may be performed. Selection by the user can be performed from a menu screen displayed on the display unit 35, for example.

<Generation of processing image data used for detection / setting processing>
The imaging control unit 34c images the processing image data used for the detection / setting process at various timings similar to the timing for generating the processing image data used for the image processing described above. That is, the same image data can be used as processing image data used for detection / setting processing or processing image data used for image processing. Hereinafter, a case where the processing image data used for the detection / setting process is generated at a timing different from the timing of generating the processing image data used for the image processing will be described.

When the user fully presses the release button, the imaging control unit 34c causes the imaging unit 32 to capture the processing image data before imaging the main image data. In this case, a result detected using the latest processing image data imaged immediately before the main imaging or a set result can be reflected in the main imaging.

Even when an operation not related to imaging, for example, an operation for reproducing and displaying an image, an operation during reproduction display, or an operation for clock adjustment is performed, the imaging control unit 34c captures image data for processing. Let In this case, the imaging control unit 34c may generate one frame of processing image data, or may generate a plurality of frames of processing image data.

In the above description, the case where the processing image data is used for the image processing, the focus detection processing, the subject detection processing, and the exposure setting processing is described as an example. However, the processing image data is limited to the processing image data used for all processing. What is used for at least one of the above processes is included in the present embodiment. Which process the image data for processing is used for may be configured so that the user can select and determine from the menu screen displayed on the display unit 35.

<Description of flowchart>
FIG. 16 is a flowchart for explaining the flow of processing for setting an imaging condition for each area and imaging. When the main switch of the camera 1 is turned on, the control unit 34 activates a program that executes the process shown in FIG. In step S10, the control unit 34 causes the display unit 35 to start live view display, and proceeds to step S20.

Specifically, the control unit 34 instructs the imaging unit 32 to start acquiring a live view image, and causes the display unit 35 to sequentially display the acquired live view image. As described above, at this time, the same imaging condition is set for the entire imaging chip 111, that is, the entire screen.
Note that when the AF operation is set during live view display, the AF calculation unit 34d of the control unit 34 performs focus detection processing to focus on the subject element corresponding to a predetermined focus point. Control the behavior.
If the setting for performing the AF operation is not performed during live view display, the AF calculation unit 34d of the control unit 34 performs the AF operation when the AF operation is instructed later.

In step S20, the object detection unit 34a of the control unit 34 detects the subject element from the live view image and proceeds to step S30. In step S30, the setting unit 34b of the control unit 34 divides the screen of the live view image into regions including subject elements, and proceeds to step S40.

In step S <b> 40, the control unit 34 displays an area on the display unit 35. As illustrated in FIG. 6, the control unit 34 highlights an area that is a target for setting (changing) the imaging condition among the divided areas. In addition, the control unit 34 displays the imaging condition setting screen 70 on the display unit 35 and proceeds to step S50.
When the display position of another main subject on the display screen is tapped with the user's finger, the control unit 34 sets an area including the main subject as an area for setting (changing) the imaging condition. Change and highlight.

In step S50, the control unit 34 determines whether an AF operation is necessary. The control unit 34, for example, when the focus adjustment state changes due to the movement of the subject, when the position of the focus point is changed by a user operation, or when execution of an AF operation is instructed by a user operation, An affirmative decision is made in step S50 and the process proceeds to step S70. If the focus adjustment state does not change, the position of the focus point is not changed by the user operation, and the execution of the AF operation is not instructed by the user operation, the control unit 34 makes a negative determination in step S50 and proceeds to step 60. .

In step S70, the control unit 34 performs the AF operation and returns to step S40. The control unit 34 that has returned to step S40 repeats the same processing as described above based on the live view image acquired after the AF operation.

In step S60, the setting unit 34b of the control unit 34 sets an imaging condition for the highlighted area in accordance with a user operation, and proceeds to step S80. That is, imaging conditions are set for each of a plurality of areas. Note that the display transition of the display unit 35 and the setting of the imaging conditions according to the user operation in step S60 are as described above.

In step S80, the control unit 34 determines whether to capture the processing image data while the live view image is displayed. If it is set to capture the processing image data while the live view image is being displayed, an affirmative determination is made in step S80 and the process proceeds to step S90. If the setting for capturing the processing image data is not made during the display of the live view image, a negative determination is made in step S80, and the process proceeds to step S100 described later.

In step S90, the imaging control unit 34c of the control unit 34 instructs the imaging unit 32 to perform imaging of the processing image data for each set imaging condition at a predetermined period during imaging of the live view image. Then, the process proceeds to step S100. Note that the processing image data captured at this time is stored in a storage medium (not shown). In step S100, the control unit 34 determines the presence / absence of an imaging instruction. When the release button constituting the operation member 36 or the display icon for instructing imaging is operated, the control unit 34 makes a positive determination in step S100 and proceeds to step S110. When the imaging instruction is not performed, the control unit 34 makes a negative determination in step S100 and returns to step S60.

In step S110, the control unit 34 performs imaging processing of the processing image data and the main image data. That is, the imaging control unit 34c captures the processing image data for each of the different imaging conditions set in step S60, and performs the actual imaging with the imaging conditions set for each of the areas in step S60. To obtain the main image data and proceed to step S120. Note that, when processing image data used for detection / setting processing is captured, the processing image data is captured before capturing the main image data. If the processing image data is captured in step S90, the processing image data may not be captured in step S110.

In step S120, the imaging control unit 34c of the control unit 34 sends an instruction to the image processing unit 33, and predetermined processing is performed using the processing image data obtained in step S90 or step S110 on the main image data obtained by the imaging. The image processing is performed, and the process proceeds to step S130. Image processing includes the pixel defect correction processing, color interpolation processing, contour enhancement processing, and noise reduction processing.

In step S130, the control unit 34 sends an instruction to the recording unit 37, records the image data after the image processing on a recording medium (not shown), and proceeds to step S140.
In step S140, the control unit 34 determines whether an end operation has been performed. When the end operation is performed, the control unit 34 makes a positive determination in step S140 and ends the process illustrated in FIG. If the end operation is not performed, the control unit 34 makes a negative determination in step S140 and returns to step S20. When returning to step S20, the control unit 34 repeats the above-described processing.

In the example described above, the main imaging is performed under the imaging conditions set in step S60, and the processing is performed on the main image data using the processing image data obtained in step S90 or S110. However, when different imaging conditions are set for each region at the time of capturing the live view image, the focus detection process, the subject detection process, and the imaging condition setting process are obtained while the live view image is displayed in step S90. This is performed based on the processed image data.

In the above description, the multilayer image sensor 100 is illustrated as the image sensor 32a. However, if the imaging condition can be set for each of a plurality of blocks in the image sensor (imaging chip 111), the image sensor 32a is not necessarily configured as a multilayer image sensor. do not have to.

According to the embodiment described above, the following operational effects can be obtained.
(1) The camera 1 includes an input unit 33a and an object detection unit 34a. The input unit 33a captures the light image of the subject incident on the first region of the image capturing unit 32 under the first image capturing condition, and the light image of the subject incident on the second region of the image capturing unit 32 differs from the first image capturing condition. The main image data imaged under the second imaging condition and the processing image data obtained by imaging the light image of the subject incident on the first area and the second area under the third imaging condition are input. The object detection unit 34b detects an object of the subject from the processing image data. As a result, even when different imaging conditions are set for each region on the main image data, the camera 1 detects the subject element using the processing image data in which the same imaging conditions are set. It can be carried out. For example, it is possible to suppress a decrease in detection accuracy of the subject element due to discontinuity or the like appearing in the search range 190 due to a difference in imaging conditions for each region.

(2) The object detection unit 34b performs detection using image data corresponding to a partial region of the imaging surface of the imaging element 32a, that is, image data of the search range 190, among the processing image data. Thereby, it is possible to suppress a decrease in detection accuracy of the subject element due to a difference in imaging conditions for each region.

(3) The processing image data is image data obtained by imaging using a partial area of the imaging surface of the imaging element 32a corresponding to the search range 190. As a result, image data in which the same imaging condition is set for the search range 190 can be acquired as processing image data, so that it is possible to suppress a decrease in detection accuracy of the subject element due to a difference in the imaging condition for each region.

(4) The processing image data excludes a part of the region when the boundary between at least the first region and the second region set in the image sensor 32a divides a part of the region corresponding to the search range 190. Imaging is performed by setting the first imaging condition in the first area, the second imaging condition in the second area excluding some areas, and the third imaging condition in some areas. Thereby, since image data in which the same imaging condition is set for the search range 190 can be acquired, it is possible to suppress a decrease in detection accuracy of the subject element due to a difference in the imaging condition for each region.

(5) The central portion of the image pickup surface of the image pickup device 32 a or the region at the designated position is a partial region corresponding to the search range 190. Thereby, since image data in which the same imaging condition is set for the search range 190 can be acquired, it is possible to suppress a decrease in detection accuracy of the subject element due to a difference in the imaging condition for each region.

(6) The region at the position designated by the user is a range including the search range 190. As a result, the image data in the search range 190 is captured with the same imaging condition set, so that it is possible to suppress a decrease in detection accuracy of the subject element due to a difference in the imaging condition for each region.

(7) The imaging control unit 34c controls the timing of performing the main imaging by the imaging unit 32 and the timing of imaging the processing image data. As a result, it is possible to acquire processing image data used to detect the subject element.

(8) The imaging control unit 34c causes the imaging unit 32 to capture the processing image data between the frames of the live view image, and causes at least one frame of the processing image data to be captured. By capturing the processing image data while capturing the live view image, the detection result of the subject element using the processing image data can be used for capturing the main image data.

(9) The imaging control unit 34c captures the live view image before the imaging unit 32 starts imaging the live view image in the imaging preparation state or after the imaging unit 32 starts imaging the live view image. Every time the predetermined number of frames are performed, the processing image data is captured to capture at least one frame of processing image data. By capturing the processing image data while capturing the live view image, the detection result of the subject element can be used when capturing the main image data.

(10) When there is an operation for instructing the main imaging while the live view image is displayed, the imaging control unit 34c outputs at least one frame of processing image data before the imaging unit 32 performs the main imaging. Let's take an image. Processing image data can be captured at the stage of transition from live view image capturing to main capturing, and the main image data can be captured using the detection results of the subject elements using the processing image data.

(11) When the operation member 36 is operated by the user and an operation on the imaging optical system such as an operation to change the imaging magnification or an operation to change the aperture is performed by the user, the imaging control unit 34c The unit 32 is caused to capture at least one frame of processing image data. As a result, when there is a high possibility that the imaging conditions are changed to the current state for the main imaging, the detection result of the subject element using the processing image data is obtained by imaging the processing image data in advance. It can be reflected in the data.

(12) The imaging control unit 34c causes the imaging unit 32 to capture at least one frame of processing image data while an operation for instructing preparation for main imaging is being performed. Thus, the processing image data is captured before the main imaging, and the main image data can be captured based on the detection result of the subject element using the processing image data.

(13) When there is an operation for instructing the main imaging, the imaging control unit 34c causes the imaging unit 32 to capture at least one frame of processing image data before the imaging unit 32 performs the main imaging. As a result, the processing image data can be captured before the main imaging, and the main image data can be captured using the detection result of the subject element using the processing image data.

(14) The imaging control unit 34c causes the imaging unit 32 to capture at least one frame of processing image data when there is an operation for instructing imaging of the processing image data. As a result, processing image data suitable for detection of subject elements can be obtained by operating a dedicated button at a timing desired by the user, such as when the user determines that there is a high possibility that the imaging conditions will change. You can take an image.

(15) The imaging unit 32 makes the imaging conditions common or different in the plurality of frames when imaging the processing image data in a plurality of frames. As a result, even when the imaging conditions are different among the plurality of divided areas, the processing image data used for each area can be captured.

(16) The setting unit 34b determines the imaging condition of the next frame based on the processing image data acquired in the previous frame of the plurality of frames when the imaging conditions are different in the plurality of frames of the processing image data. Thereby, even when the imaging condition changes during the display of the live view image, it is possible to capture the processing image data suitable for the detection of the subject element.

(17) When there is an operation for designating the imaging condition, the setting unit 34b sets the designated imaging condition and causes the processing image data to be imaged. Thereby, it is possible to capture image data for processing suitable for detection of the subject element with respect to the main image data to which the imaging condition desired by the user is applied.

(18) The setting unit 34b sets at least one of the first imaging condition and the second imaging condition based on the target detected by the object detection unit 34a. Thereby, it is possible to set an optimal imaging condition for the main imaging.

The following modifications are also within the scope of the present invention, and one or a plurality of modifications can be combined with the above-described embodiment.
(Modification 1)
FIGS. 17A to 17C are diagrams illustrating the arrangement of the first region and the second region on the imaging surface of the imaging device 32a. According to the example of FIG. 17A, the first region is configured by even columns, and the second region is configured by odd columns. That is, the imaging surface is divided into even columns and odd columns.

According to the example of FIG. 17B, the first region is configured by odd rows, and the second region is configured by even rows. That is, the imaging surface is divided into odd rows and even rows.
According to the example of FIG. 17C, the first region is configured by blocks of even rows in odd columns and blocks of odd rows in even columns. In addition, the second region is configured by even-numbered blocks in even columns and odd-numbered blocks in odd columns. That is, the imaging surface is divided into a checkered pattern. A first imaging condition is set in the first area, and a second imaging condition different from the first imaging condition is set in the second area.

In any case of FIGS. 17A to 17C, the first image based on the photoelectric conversion signal read from the first region and the photoelectric conversion signal read from the image pickup element 32a that has picked up an image of one frame, and Second images based on the photoelectric conversion signals read from the second region are respectively generated. According to the first modification, the first image and the second image are captured at the same angle of view and include a common subject image.

In the first modification, the control unit 34 uses the first image for display and the second image as processing image data. Specifically, the control unit 34 causes the display unit 35 to display the first image as a live view image. The control unit 34 uses the second image as processing image data. That is, the processing unit 33b performs image processing using the second image, the object detection unit 34a performs subject detection processing using the second image, and the AF calculation unit 34d performs focus detection processing using the second image. And the exposure calculation process is performed by the setting unit 34b using the second image.

In addition, you may change the area | region which acquires a 1st image, and the area | region which acquires a 2nd image for every frame. For example, in the Nth frame, the first image from the first region is captured as a live view image, the second image from the second region is captured as processing image data, and in the N + 1th frame, the first image is processed as processing image data. The second image may be captured as a live view image, and this operation may be repeated in subsequent frames.

1. As an example, the control unit 34 captures a live view image under the first imaging condition, and sets the first imaging condition to a condition suitable for display by the display unit 35. The first imaging condition is the same for the entire imaging screen. On the other hand, the control unit 34 captures the processing image data under the second imaging condition, and sets the second imaging condition to a condition suitable for the focus detection process, the subject detection process, and the exposure calculation process. The second imaging condition is also made the same for the entire imaging screen.
When the conditions suitable for the focus detection process, the subject detection process, and the exposure calculation process are different, the control unit 34 may change the second imaging condition set in the second area for each frame. For example, the second imaging condition of the first frame is a condition suitable for the focus detection process, the second imaging condition of the second frame is a condition suitable for the subject detection process, and the second imaging condition of the third frame is the exposure calculation process. Conditions suitable for In these cases, the second imaging condition in each frame is the same for the entire imaging screen.

2. As another example, the control unit 34 may change the first imaging condition on the imaging screen. The setting unit 34b of the control unit 34 sets different first imaging conditions for each region including the subject element divided by the setting unit 34b. On the other hand, the control unit 34 makes the second imaging condition the same for the entire imaging screen. The control unit 34 sets the second imaging condition to a condition suitable for the focus detection process, the subject detection process, and the exposure calculation process. However, the conditions suitable for the focus detection process, the subject detection process, and the exposure calculation process are set. If they are different, the imaging conditions set in the second area may be different for each frame.

3. As another example, the control unit 34 may change the second imaging condition on the imaging screen while making the first imaging condition the same on the entire imaging screen. For example, a different second imaging condition is set for each region including the subject element divided by the setting unit 34b. Even in this case, if the conditions suitable for the focus detection process, the subject detection process, and the exposure calculation process are different, the imaging conditions set in the second region may be different for each frame.

4). Furthermore, as another example, the control unit 34 makes the first imaging condition different on the imaging screen and makes the second imaging condition different on the imaging screen. For example, the setting unit 34b sets different first imaging conditions for each region including the subject element divided, and the setting unit 34b sets different second imaging conditions for each region including the subject element divided.

17A to 17C, the area ratio between the first region and the second region may be different. For example, the control unit 34 sets the ratio of the first region to be higher than that of the second region based on the operation by the user or the determination of the control unit 34, or sets the ratio of the first region to the second region as shown in FIG. As shown in FIG. 17C, the setting is made equal, or the ratio of the first area is set lower than that of the second area. By making the area ratios different between the first region and the second region, the first image is made to have a higher definition than the second image, the resolutions of the first image and the second image are made equal, or the second image is Compared to the first image, it can be made higher definition.

For example, in the region X surrounded by the thick line in FIG. 17C, the first image is made higher in definition than the second image by averaging the image signals from the blocks in the second region. Thereby, it is possible to obtain image data equivalent to the case where the processing imaging area is enlarged or reduced in accordance with the change of the imaging conditions.

(Modification 2)
In the above-described embodiment, the example in which the setting unit 34b of the control unit 34 detects the subject element based on the live view image and divides the screen of the live view image into regions including the subject element has been described. In the second modification, the control unit 34 may divide the region based on the output signal from the photometric sensor when the photometric sensor is provided separately from the image sensor 32a.

FIG. 18 is a block diagram showing a main configuration of the second modification. The camera 1 includes a photometric sensor 38 in addition to the configuration in the embodiment shown in FIG. The control unit 34 divides the foreground and the background based on the output signal from the photometric sensor 38. Specifically, the live view image acquired by the image sensor 32 b is determined to be the foreground area corresponding to the area determined to be the foreground from the output signal from the photometry sensor 38 and the background from the output signal from the photometry sensor 38. Is divided into a background area corresponding to the selected area.

Further, the control unit 34 arranges the first area and the second area on the imaging surface of the imaging element 32a as illustrated in FIGS. 17A to 17C with respect to the foreground area. On the other hand, the control unit 34 arranges only the first area on the imaging surface of the imaging element 32a with respect to the background area. The control unit 34 uses the first image for display and the second image for detection.

According to the second modification, by using the output signal from the photometric sensor 38, it is possible to divide the live view image acquired by the image sensor 32b. In addition, a first image for display and a second image for detection can be obtained for the foreground area, and only a first image for display can be obtained for the background area. Even when the imaging environment of the subject changes during the display of the live view image, the foreground area and the background area can be newly set by performing area division using the output from the photometric sensor 38. it can. Note that the present invention is not limited to the example in which the photometric sensor 38 detects the imaging environment of the subject. Also good.

(Modification 3)
In the third modification, the image processing unit 33 does not impair the outline of the subject element in the above-described image processing (for example, noise reduction processing). In general, smoothing filter processing is employed when noise reduction is performed. When the smoothing filter is used, the boundary of the subject element may be blurred while the noise reduction effect.

Therefore, the processing unit 33b of the image processing unit 33 compensates for the blur of the subject element boundary by performing contrast adjustment processing in addition to or together with noise reduction processing, for example. In the third modification, the processing unit 33b of the image processing unit 33 sets a curve that draws an S shape as a density conversion (gradation conversion) curve (so-called S-shaped conversion). The processing unit 33b of the image processing unit 33 increases the number of gradations of bright data (and dark data) by extending the gradation portions of bright data and dark data by performing contrast adjustment using S-shaped conversion. At the same time, the number of gradations is reduced by compressing the intermediate gradation image data. As a result, the number of image data having a medium brightness is reduced, and data classified as either bright / dark is increased. As a result, blurring of the boundary of the subject element can be compensated.
According to the third modification, blurring of the boundary of the subject element can be compensated by clearing the contrast of the image.

(Modification 4)
A plurality of image processing units 33 may be provided, and image processing may be performed in parallel. For example, image processing is performed on the image data captured in the region B of the imaging unit 32 while performing image processing on the image data captured in the region A of the imaging unit 32. The plurality of image processing units 33 may perform the same image processing or different image processing. That is, the same parameters are applied to the image data of the region A and the region B, and the same image processing is performed, or the different parameters are applied to the image data of the region A and the region B to perform different image processing. You can do it.

In the case where a plurality of image processing units 33 are provided, image processing is performed by one image processing unit on the data to which the first imaging condition is applied, and another image is applied to the data to which the second imaging condition is applied. Image processing may be performed by the processing unit. The number of image processing units is not limited to the above two, and for example, the same number as the number of imaging conditions that can be set may be provided. That is, each image processing unit takes charge of image processing for each region to which different imaging conditions are applied. According to the modification 4, it is possible to proceed in parallel with imaging under different imaging conditions for each area and image processing for image data of an image obtained for each area.

(Modification 5)
In the above description, the camera 1 has been described as an example, but it may be configured by a high function mobile phone 250 (FIG. 20) having a camera function like a smartphone or a mobile device such as a tablet terminal.

(Modification 6)
In the above-described embodiment, the camera 1 in which the imaging unit 32 and the control unit 34 are configured as a single electronic device has been described as an example. Instead, for example, the imaging unit 1 and the control unit 34 may be provided separately, and the imaging system 1B that controls the imaging unit 32 from the control unit 34 via communication may be configured.
Hereinafter, an example in which the imaging device 1001 including the imaging unit 32 is controlled from the control device 1002 including the control unit 34 will be described.

FIG. 19 is a block diagram illustrating the configuration of an imaging system 1B according to Modification 6. In FIG. 19, the imaging system 1 </ b> B includes an imaging device 1001 and a display device 1002. The imaging device 1001 includes a first communication unit 1003 in addition to the imaging optical system 31 and the imaging unit 32 described in the above embodiment. The display device 1002 includes a second communication unit 1004 in addition to the image processing unit 33, the control unit 34, the display unit 35, the operation member 36, and the recording unit 37 described in the above embodiment.

The first communication unit 1003 and the second communication unit 1004 can perform bidirectional image data communication using, for example, a well-known wireless communication technology or optical communication technology.
Note that the imaging device 1001 and the display device 1002 may be connected by a wired cable, and the first communication unit 1003 and the second communication unit 1004 may perform bidirectional image data communication.

In the imaging system 1B, the control unit 34 controls the imaging unit 32 by performing data communication via the second communication unit 1004 and the first communication unit 1003. For example, by transmitting and receiving predetermined control data between the imaging device 1001 and the display device 1002, the display device 1002 divides the screen into a plurality of regions based on the images as described above, or the divided regions. A different imaging condition is set for each area, or a photoelectric conversion signal photoelectrically converted in each area is read out.

According to the modified example 6, since the live view image acquired on the imaging device 1001 side and transmitted to the display device 1002 is displayed on the display unit 35 of the display device 1002, the user is positioned away from the imaging device 1001. Remote control can be performed from a certain display device 1002.
The display device 1002 can be configured by a high-function mobile phone 250 such as a smartphone, for example. In addition, the imaging device 1001 can be configured by an electronic device including the above-described stacked imaging element 100.
In addition, although the example which provides the control part 34 of the display apparatus 1002 with the object detection part 34a, the setting part 34b, the imaging control part 34c, and the AF calculating part 34d was demonstrated, the object detection part 34a, the setting part 34b, and imaging A part of the control unit 34c and the AF calculation unit 34d may be provided in the imaging apparatus 1001.

(Modification 7)
The program is supplied to the above-described mobile device such as the camera 1, the high-function mobile phone 250, or the tablet terminal by infrared communication or short-range wireless communication from the personal computer 205 storing the program as illustrated in FIG. 20, for example. Can be sent to mobile devices.

The program may be supplied to the personal computer 205 by setting a recording medium 204 such as a CD-ROM storing the program in the personal computer 205 or by a method via the communication line 201 such as a network. You may load. When passing through the communication line 201, the program is stored in the storage device 203 of the server 202 connected to the communication line.

Also, the program can be directly transmitted to the mobile device via a wireless LAN access point (not shown) connected to the communication line 201. Further, a recording medium 204B such as a memory card storing the program may be set in the mobile device. Thus, the program can be supplied as various forms of computer program products, such as provision via a recording medium or a communication line.

As long as the characteristics of the present invention are not impaired, the present invention is not limited to the above-described embodiments, and other forms conceivable within the scope of the technical idea of the present invention are also included in the scope of the present invention. .

The above-described embodiments and modifications also include the following imaging device and image processing device.
(1-1) a first pixel that outputs a signal generated by photoelectrically converted charge and a second pixel that is different from the first pixel and outputs a signal generated by photoelectrically converted charge An imaging element having an imaging area for imaging a subject, an imaging condition of a first area in which the first pixel is arranged in the imaging area, and the second of the imaging areas. A setting unit for setting an imaging condition of a second region different from the first region in which the pixels are arranged, and interpolation of the first pixel of the first region set to the first imaging condition by the setting unit The second pixel in the second region set to a second imaging condition different from the first imaging condition by the setting unit, and a third pixel different from the second imaging condition by the setting unit. The second area of the second region set in the imaging condition The first region of the first region set in the first imaging condition, interpolated by a signal output from the second pixel selected by the selection unit and the second pixel selected by the selection unit. An imaging apparatus comprising: a detection unit that detects at least a part of a subject imaged in the imaging area using a signal output from one pixel.
(1-2) The selection unit uses the first imaging condition and the second imaging condition as pixels used for interpolation of the first pixel in the first region set by the setting unit as the first imaging condition. And the difference between the first imaging condition and the third imaging condition, the second pixel in the second region set to the second imaging condition by the setting unit, and the setting unit The imaging apparatus according to (1-1), wherein the second pixel in the second area set as the third imaging condition is selected from the first pixel and the second pixel.
(1-3) The selection unit uses the first imaging condition and the second imaging condition as pixels used for interpolation of the first pixel in the first area set by the setting unit as the first imaging condition. The imaging device according to (1-2), wherein the second pixel in the second region set to an imaging condition with a small difference between the first imaging condition and the third imaging condition is selected.
(1-4) The selection unit uses the first imaging condition and the second imaging condition as pixels used for interpolation of the first pixel in the first area set by the setting unit as the first imaging condition. And the difference between the first imaging condition and the third imaging condition, the second pixel in the second region set to the second imaging condition by the setting unit, and the setting unit The imaging apparatus according to (1-1), wherein the second pixel in the second area set as the third imaging condition is selected from the first pixel and the second pixel.
(1-5) The selection unit selects, as the first imaging condition and the second imaging condition, pixels used for interpolation of the first pixel in the first area set by the setting unit as the first imaging condition. The imaging device according to (1-4), wherein the second pixel in the second region set to an imaging condition with a small difference between the first imaging condition and the third imaging condition is selected.
(1-6) A storage unit that stores a signal output from the second pixel is provided, and the setting unit sets the second area to the second area before setting the second imaging condition in the second area. The three imaging conditions are set, and the storage unit stores the signals output from the second pixels in the second area set by the setting unit as the third imaging conditions (1-1) to (1 The imaging apparatus according to any one of -5).
(1-7) The imaging apparatus according to (1-6), wherein the setting unit sets the first imaging condition in the first area after setting the third imaging condition in the second area.
(1-8) A storage unit that stores a signal output from the second pixel is provided, and the setting unit sets the second imaging condition in the second region, and then sets the second region to the third region. The imaging unit is set, and the storage unit stores the signal output from the second pixel in the second region set to the second imaging condition by the setting unit (1-1) to (1- 5) The imaging device according to any one of
(1-9) The imaging apparatus according to (1-8), wherein the setting unit sets the first imaging condition in the first area before setting the third imaging condition in the second area.
(1-10) The first pixel includes a first photoelectric conversion unit that photoelectrically converts light incident through the filter having the first spectral characteristic, and the second pixel is different from the first spectral characteristic. The imaging apparatus according to any one of (1-1) to (1-9), further including a second photoelectric conversion unit that photoelectrically converts light incident through the filter having the second spectral characteristic.
(1-11) A plurality of the first pixels are arranged in the first region, and a plurality of the second pixels are arranged in the second region. (1-1) to (1-10) The imaging device according to any one of the above.
(1-12) In the first area, a single first pixel is arranged, and in the second area, a single second pixel is arranged (1-1) to (1-10) The imaging device according to any one of the above.
(1-13) a first pixel that outputs a signal generated by photoelectrically converted charge, and a second pixel that is different from the first pixel and outputs a signal generated by photoelectrically converted charge A first imaging element having a first imaging area that images the subject and a third pixel that outputs a signal generated by the photoelectrically converted charge is disposed and images the subject. A second imaging element having a second imaging area that is different from the first imaging element and a selection unit that selects a pixel used for interpolation of the first pixel from the second pixel and the third pixel And detecting at least a part of the subject imaged in the first imaging area using the signal output from the first pixel interpolated by the signal output from the pixel selected by the selection unit And photography Image device.
(1-14) Imaging conditions of the first area in which the first pixel is arranged in the first imaging area, and the first area in which the second pixel is arranged in the first imaging area The imaging apparatus according to (1-13), further including: a setting unit configured to set an imaging condition for a second area different from the imaging condition for the second imaging area where the third pixel is arranged.
(1-15) The selection unit differs from the first imaging condition by the setting unit in a pixel used for interpolation of the first pixel in the first region set by the setting unit as the first imaging condition. The second pixel of the second area set to the second imaging condition, and the third pixel of the second imaging area set to the third imaging condition different from the second imaging condition by the setting unit (1-14).
(1-16) The selection unit selects pixels used for interpolation of the first pixel in the first region set as the first imaging condition by the setting unit, as the first imaging condition and the second imaging condition. And the difference between the first imaging condition and the third imaging condition, the second pixel in the second region set to the second imaging condition by the setting unit, and the setting unit The imaging device according to (1-15), wherein the third pixel in the second imaging region set in the third imaging condition is selected by the first imaging condition.
(1-17) The selection unit selects pixels used for interpolation of the first pixel in the first region set as the first imaging condition by the setting unit, as the first imaging condition and the second imaging condition. And the second pixel of the second region set to the second imaging condition by the setting unit, which is set to an imaging condition with a small difference between the first imaging condition and the third imaging condition. And the third pixel in the second imaging region set in the third imaging condition by the setting unit. (1-16).
(1-18) The selection unit selects pixels used for interpolation of the first pixel in the first area set as the first imaging condition by the setting unit, as the first imaging condition and the second imaging condition. And the difference between the first imaging condition and the third imaging condition, the second pixel in the second region set to the second imaging condition by the setting unit, and the setting unit The imaging device according to (1-15), wherein the third pixel in the second imaging region set in the third imaging condition is selected by the first imaging condition.
(1-19) The selection unit uses the first imaging condition and the second imaging condition as pixels used for interpolation of the first pixel in the first area set by the setting unit as the first imaging condition. And the first imaging condition and the third imaging condition, the second pixel of the second region set to the second imaging condition by the setting unit set to an imaging condition with a small difference And the third pixel in the second imaging region set in the third imaging condition by the setting unit, according to (1-18).
(1-20) The first pixel includes a first photoelectric conversion unit that photoelectrically converts light incident through a filter having a first spectral characteristic, and the second pixel and the third pixel include the first pixel. The imaging apparatus according to any one of (1-13) to (1-19), further including a second photoelectric conversion unit that photoelectrically converts light incident through a filter having a second spectral characteristic different from the spectral characteristic.
(1-21) In the first region, a plurality of the first pixels are arranged, and in the second region, a plurality of the second pixels are arranged (1-13) to (1-20) The imaging device according to any one of the above.
(1-22) In the first area, a single first pixel is arranged, and in the second area, a single second pixel is arranged (1-13) to (1-20) The imaging device according to any one of the above.
(1-23) A first pixel that outputs a signal generated by photoelectrically converted charges and a second pixel that is different from the first pixel that outputs a signal generated by photoelectrically converted charges are arranged An imaging element having an imaging area for imaging a subject, an imaging condition of a first area in which the first pixel is arranged in the imaging area, and the second pixel in the imaging area Is output from the first pixel of the first region set to the first imaging condition by the setting unit, and a setting unit for setting the imaging condition of the second region different from the first region in which is arranged The pixel used for signal processing of the received signal is the second pixel in the second region set by the setting unit to a second imaging condition different from the first imaging condition, and the second imaging condition by the setting unit. Is set to a different third imaging condition The selection unit selected from the second pixel in the second region, and the first imaging condition processed by the signal output from the second pixel selected by the selection unit is set. An imaging device comprising: a detection unit that detects at least a part of a subject imaged in the imaging region using a signal output from the first pixel in the first region.
(1-24) An area in which a first pixel that generates a signal using photoelectrically converted charges and a second pixel that is different from the first pixel and generates a signal using photoelectrically converted charges are arranged. A first imaging element having a first imaging area for imaging a subject and a second imaging area for imaging the subject, wherein the third pixel for generating a signal by photoelectrically converted charges is disposed. A second imaging element different from the first imaging element; a selection unit that selects a pixel used for signal processing of a signal output from the first pixel from the second pixel and the third pixel; and A detection unit for detecting at least a part of a subject imaged in the first imaging region using a signal output from the first pixel, which is signal-processed by a signal output from the pixel selected by the selection unit; An imaging device comprising Place.
(1-25) Second imaging that is different from the first imaging condition is used for interpolation of the first pixel arranged in the first area among the imaging areas of the imaging device set in the first imaging condition. Among the imaging areas set as conditions, the second pixels arranged in the second area and the second pixels arranged in the second area set in the third imaging condition different from the second imaging condition The first region of the first region set in the first imaging condition, interpolated by a signal output from the second pixel selected by the selection unit and the second pixel selected by the selection unit. An image processing apparatus comprising: a detection unit that detects at least a part of a subject imaged in the imaging region using a signal output from one pixel.
(1-26) A pixel used for interpolation of the first pixel arranged in the first imaging region of the first imaging element is a second pixel different from the first pixel arranged in the first imaging region; A third pixel arranged in a second imaging region of a second imaging element different from the first imaging element, a selection unit to select from, and interpolation by a signal output from the pixel selected by the selection unit And a detection unit that detects at least a part of the subject imaged in the first imaging region using the signal output from the first pixel.
(1-27) A pixel used for signal processing of a signal output from the first pixel arranged in the first area among the imaging areas of the image sensor set in the first imaging condition is defined as the first imaging condition. Among the imaging areas set to the second imaging condition different from the second imaging condition, the second pixels arranged in the second area and the second area set to the third imaging condition different from the second imaging condition The second pixel arranged, a selection unit selected from the above, and the signal set by the signal output from the second pixel selected by the selection unit and set in the first imaging condition An image processing apparatus comprising: a detection unit that detects at least a part of a subject imaged in the imaging area using a signal output from the first pixel in the first area.
(1-28) A pixel used for signal processing of a signal output from the first pixel arranged in the first imaging region of the first imaging element is the first pixel arranged in the first imaging region. From a different second pixel and a third pixel arranged in a second imaging region of a second imaging element different from the first imaging element, a selection unit selected from the pixels selected by the selection unit An image processing apparatus comprising: a detection unit configured to detect at least a part of a subject imaged in the first imaging region using a signal output from the first pixel, which is signal-processed by the output signal.

Further, the above-described embodiments and modifications also include the following subject detection device and imaging device.
(2-1) The optical image of the subject incident on the first region of the imaging unit is captured under the first imaging condition, and the optical image of the subject incident on the second region of the imaging unit is defined as the first imaging condition. An input unit for inputting first image data captured under different second imaging conditions, and second image data obtained by imaging a light image of the subject incident on the first region and the second region under a third imaging condition; And a detection unit that detects an object of the subject from the second image data.
(2-2) In the subject detection device according to (2-1), the detection unit detects the object from data in a region corresponding to a partial region of the imaging unit in the second image data. A subject detection device to detect.
(2-3) The subject detection device according to (2-1), wherein the second image data is captured using a partial region of the imaging unit.
(2-4) In the subject detection apparatus according to (2-3), the second image data includes at least a boundary between the first region and the second region set in the imaging unit as the partial region. When dividing, the first imaging condition is set in the first area excluding the partial area, the second imaging condition is set in the second area excluding the partial area, and the partial area is set. A subject detection apparatus that is imaged by setting the third imaging condition in each of the above.
(2-5) In the subject detection device according to any one of (2-2) to (2-4), the partial region is a subject that is a central portion of the imaging unit or a region at a designated position. Detection device.
(2-6) The subject detection device according to (2-5), wherein the designated position area includes a detection area for detecting the object.
(2-7) The light of the subject incident by setting the first imaging condition in the first area of the imaging surface and setting the second imaging condition different from the first imaging condition in the second area of the imaging surface First imaging for capturing an image is performed to generate first image data, and second imaging for capturing a light image of an incident subject by setting a third imaging condition in the first area and the second area. An imaging apparatus comprising: an imaging unit that performs second image data generation and a detection unit that detects an object of the subject from the second image data generated by the imaging unit.
(2-8) In the imaging device according to (2-7), the detection unit detects the object from data in a region corresponding to a partial region of the imaging surface in the second image data. An imaging device.
(2-9) The imaging apparatus according to (2-7), wherein the imaging unit generates the second image data using a partial region of the imaging surface.
(2-10) In the imaging device according to (2-9), in the imaging unit, the boundary between at least the first area and the second area set on the imaging surface divides the partial area In addition, the first imaging condition is excluded in the first area excluding the partial area, the second imaging condition in the second area excluding the partial area, and the second imaging condition in the partial area. An imaging device that generates the second image data by setting three imaging conditions.
(2-11) In the imaging device according to any one of (2-8) to (2-10), the partial area is a central portion of the imaging surface or an area at an instructed position. .
(2-12) The imaging apparatus according to (2-11), wherein the designated position area includes a detection area for detecting the object.
(2-13) In the imaging device according to any one of (2-7) to (2-12), control for controlling timing of performing the first imaging and timing of performing the second imaging by the imaging unit An imaging apparatus further comprising a unit.
(2-14) The imaging apparatus according to (2-13), further including a display processing unit that causes the display unit to display an image based on the image data generated by the imaging unit, wherein the imaging unit includes an imaging preparation state An imaging device that performs third imaging for generating the image data of a plurality of frames in the display, and the display processing unit displays the images of the plurality of frames generated by the third imaging on the display unit.
(2-15) In the imaging device according to (2-14), the control unit causes the imaging unit to perform the second imaging between frames of the third imaging, so that the first imaging of at least one frame is performed. An imaging device that generates two image data.
(2-16) In the imaging device according to (2-15), the control unit may cause the imaging unit to start the third imaging in the imaging preparation state or before the imaging unit starts the third imaging. An imaging device that performs the second imaging and generates the second image data of at least one frame every time the third imaging is performed for a predetermined number of frames after starting the operation.
(2-17) In the imaging device according to (2-15), when the control unit performs an operation to instruct the first imaging, at least 1 before the imaging unit performs the first imaging. An imaging apparatus that causes the imaging unit to generate the second image data of a frame.
(2-18) The imaging apparatus according to any one of (2-15) to (2-17), further including an operation member for the imaging optical system, wherein the control unit finishes the operation on the operation member, An imaging apparatus that causes the imaging unit to perform the second imaging and generate the second image data of at least one frame.
(2-19) In the imaging apparatus according to (2-18), the control unit receives the second image data of at least one frame while an operation for instructing preparation for the first imaging is performed. An imaging device to be generated by the imaging unit.
(2-20) In the imaging device according to (2-19), when the control unit performs an operation to instruct the first imaging, at least 1 before the imaging unit performs the first imaging. An imaging apparatus that causes the imaging unit to generate the second image data of a frame.
(2-21) In the imaging apparatus according to any one of (2-13) to (2-20), when the control unit performs an operation to instruct the second imaging, An imaging device that performs two imaging operations to generate the second image data of at least one frame.
(2-22) In the imaging device according to any one of (2-7) to (2-21), when the second imaging is performed for a plurality of frames, the imaging unit sets the third imaging condition as described above. An imaging device that is common or different in a plurality of frames.
(2-23) In the imaging device according to (2-22), in the case where the third imaging condition is changed in the plurality of frames, the imaging unit generates the first image generated in a previous frame of the plurality of frames. An imaging apparatus that sets the third imaging condition for the next frame based on two image data.
(2-24) In the imaging apparatus according to any one of (2-7) to (2-23), when the imaging unit performs an operation of specifying an imaging condition, the specified imaging condition is set. An imaging apparatus configured to generate the second image data set as the third imaging condition.
(2-25) In the imaging device according to any one of (2-7) to (2-23), the imaging unit includes an environment detection unit that detects a change in environment when the subject is imaged, and the imaging device The imaging device sets the third imaging condition based on the environment of the subject to be imaged after the change detected by the environment detection unit.
(2-26) In the imaging device according to any one of (2-7) to (2-25), based on the object detected by the detection unit, the first imaging condition and the second imaging An imaging apparatus including a setting unit that sets at least one of conditions.

The disclosure of the following priority application is hereby incorporated by reference.
Japanese patent application 2015 No. 195136 (filed on September 30, 2015)

DESCRIPTION OF SYMBOLS 1 ... Camera 1B ... Imaging system 32 ...

Imaging part

32a, 100 ... Imaging element 33 ... Image processing part 33a ... Input part 33b ... Processing part 34 ... Control part 34a ... Object detection part 34b ... Setting part 34c ... Imaging control part 34d ... AF calculation unit 35 ... display unit 38 ... photometric sensor 90 ... attention range 100 ... stacked imaging element 1001 ... imaging device 1002 ... display device P ... attention pixel Pr ... reference pixel

Claims

In a region where a first pixel that outputs a signal generated by photoelectrically converted charge and a second pixel that outputs a signal generated by photoelectrically converted charge and that is different from the first pixel are arranged. An imaging device having an imaging area for imaging a subject;
An imaging condition of a first area in which the first pixel is arranged in the imaging area, and an imaging condition of a second area different from the first area in which the second pixel is arranged in the imaging area, A setting section for setting,
The pixels used for interpolation of the first pixels in the first region set as the first imaging condition by the setting unit are set to the second imaging condition different from the first imaging condition by the setting unit. A selection unit that selects between the second pixel in two regions and the second pixel in the second region set to a third imaging condition different from the second imaging condition by the setting unit;
The imaging region using a signal output from the first pixel of the first region set in the first imaging condition, interpolated by a signal output from the second pixel selected by the selection unit A detection unit for detecting at least a part of the subject imaged in
An imaging apparatus comprising:
The selection unit uses a difference between the first imaging condition and the second imaging condition as a pixel to be used for interpolation of the first pixel in the first region set by the setting unit as the first imaging condition. Due to the difference between the first imaging condition and the third imaging condition, the second pixel in the second region set as the second imaging condition by the setting unit, and the third imaging by the setting unit. The imaging device according to claim 1, wherein the imaging device is selected from among the second pixels in the second region set as a condition.
The selection unit includes pixels used for interpolation of the first pixel in the first region set by the setting unit as the first imaging condition, the first imaging condition, the second imaging condition, and the first The imaging device according to claim 2, wherein the second pixel in the second region set to an imaging condition with a small difference between the imaging condition and the third imaging condition is selected.
The selection unit uses a difference between the first imaging condition and the second imaging condition as a pixel to be used for interpolation of the first pixel in the first region set by the setting unit as the first imaging condition. Due to the difference between the first imaging condition and the third imaging condition, the second pixel in the second area set as the second imaging condition by the setting unit, and the third imaging by the setting unit. The imaging device according to claim 1, wherein the imaging device is selected from among the second pixels in the second region set as a condition.
The selection unit includes pixels used for interpolation of the first pixel in the first region set by the setting unit as the first imaging condition, the first imaging condition, the second imaging condition, and the first The imaging apparatus according to claim 4, wherein the second pixel in the second region set to an imaging condition with a small difference between the imaging condition and the third imaging condition is selected.
A storage unit for storing a signal output from the second pixel;
The setting unit sets the second area to the third imaging condition before setting the second imaging condition to the second area,
The said memory | storage part memorize | stores the signal output from the said 2nd pixel of the said 2nd area | region set to the said 3rd imaging condition by the said setting part. Imaging device.
The imaging apparatus according to claim 6, wherein the setting unit sets the first imaging condition in the first area after setting the third imaging condition in the second area.
A storage unit for storing a signal output from the second pixel;
The setting unit sets the second area to the third imaging condition after setting the second imaging condition to the second area,
The said memory | storage part memorize | stores the signal output from the said 2nd pixel of the said 2nd area | region set to the said 2nd imaging condition by the said setting part. Imaging device.
The imaging apparatus according to claim 8, wherein the setting unit sets the first imaging condition in the first area before setting the third imaging condition in the second area.
The first pixel includes a first photoelectric conversion unit that photoelectrically converts light incident through a filter having a first spectral characteristic;
The said 2nd pixel has a 2nd photoelectric conversion part which photoelectrically converts the light which injected through the filter of the 2nd spectral characteristic different from the said 1st spectral characteristic. The imaging device described.
The first region includes a plurality of the first pixels,
The imaging device according to any one of claims 1 to 10, wherein a plurality of the second pixels are arranged in the second region.
In the first region, a single first pixel is arranged,
The imaging device according to any one of claims 1 to 10, wherein a single second pixel is arranged in the second region.
In a region where a first pixel that outputs a signal generated by photoelectrically converted charge and a second pixel that outputs a signal generated by photoelectrically converted charge and that is different from the first pixel are arranged. A first imaging device having a first imaging area for imaging a subject;
A second imaging element different from the first imaging element, having a second imaging area that is an area in which third pixels that output signals generated by photoelectrically converted charges are arranged and images a subject;
A selection unit that selects a pixel used for interpolation of the first pixel from the second pixel and the third pixel;
A detection unit for detecting at least a part of a subject imaged in the first imaging region using a signal output from the first pixel interpolated by a signal output from the pixel selected by the selection unit; ,
An imaging apparatus comprising:
An imaging condition of a first area where the first pixel is arranged in the first imaging area and a second different from the first area where the second pixel is arranged in the first imaging area. The imaging apparatus according to claim 13, further comprising a setting unit configured to set an imaging condition for an area and an imaging condition for the second imaging area in which the third pixel is arranged.
The selection unit sets a pixel used for interpolation of the first pixel in the first region set as the first imaging condition by the setting unit to a second imaging condition different from the first imaging condition by the setting unit. The second pixel in the second area set and the third pixel in the second imaging area set to a third imaging condition different from the second imaging condition by the setting unit are selected. The imaging device according to claim 14.
The selection unit uses a difference between the first imaging condition and the second imaging condition as a pixel to be used for interpolation of the first pixel in the first region set by the setting unit as the first imaging condition. Due to the difference between the first imaging condition and the third imaging condition, the second pixel in the second region set as the second imaging condition by the setting unit, and the third imaging by the setting unit. The imaging device according to claim 15, wherein the imaging device is selected from among the third pixels of the second imaging region set as a condition.
The selection unit includes pixels used for interpolation of the first pixel in the first region set by the setting unit as the first imaging condition, the first imaging condition, the second imaging condition, and the first The second pixel of the second region set to the second imaging condition by the setting unit, which is set to an imaging condition with a small difference between the imaging condition and the third imaging condition, and the setting unit The imaging device according to claim 16, wherein the imaging device is selected from among the third pixels of the second imaging region set as the third imaging condition.
The selection unit uses a difference between the first imaging condition and the second imaging condition as a pixel to be used for interpolation of the first pixel in the first region set by the setting unit as the first imaging condition. Due to the difference between the first imaging condition and the third imaging condition, the second pixel in the second area set as the second imaging condition by the setting unit, and the third imaging by the setting unit. The imaging device according to claim 15, wherein the imaging device is selected from among the third pixels of the second imaging region set as a condition.
The selection unit includes pixels used for interpolation of the first pixel in the first region set by the setting unit as the first imaging condition, the first imaging condition, the second imaging condition, and the first The second pixel of the second region set to the second imaging condition by the setting unit, which is set to an imaging condition with a small difference between the imaging condition and the third imaging condition, and the setting unit The imaging device according to claim 18, wherein the imaging device is selected from among the third pixels of the second imaging region set as the third imaging condition.
The first pixel includes a first photoelectric conversion unit that photoelectrically converts light incident through a filter having a first spectral characteristic;
The said 2nd pixel and the said 3rd pixel have a 2nd photoelectric conversion part which photoelectrically converts the light which injected through the filter of the 2nd spectral characteristic different from the said 1st spectral characteristic. The imaging device according to any one of the above.
The first region includes a plurality of the first pixels,
The imaging device according to any one of claims 13 to 20, wherein a plurality of the second pixels are arranged in the second region.
In the first region, a single first pixel is arranged,
The imaging device according to any one of claims 13 to 20, wherein a single second pixel is arranged in the second region.
This is an area where a first pixel that outputs a signal generated by photoelectrically converted charges and a second pixel that is different from the first pixel that outputs a signal generated by photoelectrically converted charges are arranged. An imaging device having an imaging region for imaging the subject,
An imaging condition of a first area in which the first pixel is arranged in the imaging area, and an imaging condition of a second area different from the first area in which the second pixel is arranged in the imaging area, A setting section for setting,
The second imaging condition that is different from the first imaging condition by the setting unit is used for the signal processing of the signal output from the first pixel in the first region that is set by the setting unit as the first imaging condition. The second pixel of the second area set to 2 and the second pixel of the second area set to a third imaging condition different from the second imaging condition by the setting unit A selection section to
The imaging using the signal output from the first pixel in the first region set in the first imaging condition, which is signal-processed by the signal output from the second pixel selected by the selection unit A detection unit for detecting at least a part of the subject imaged in the region;
An imaging apparatus comprising:
A subject is imaged in an area where a first pixel that generates a signal based on photoelectrically converted charges and a second pixel that generates a signal based on photoelectrically converted charges and that is different from the first pixels are arranged. A first imaging device having a first imaging region;
A second imaging element that is different from the first imaging element and has a second imaging area that is an area where a third pixel that generates a signal by photoelectrically converted charges is arranged and images a subject;
A selection unit that selects a pixel used for signal processing of a signal output from the first pixel from the second pixel and the third pixel;
A detection unit that detects at least a part of a subject imaged in the first imaging region using a signal output from the first pixel, which is signal-processed by a signal output from the pixel selected by the selection unit. When,
An imaging apparatus comprising:
Among the imaging regions of the image sensor set in the first imaging condition, a pixel used for interpolation of the first pixel arranged in the first region is set to a second imaging condition different from the first imaging condition. A second pixel arranged in the second region of the imaging region, and a second pixel arranged in the second region set in a third imaging condition different from the second imaging condition. A selection section to choose from,
The imaging region using a signal output from the first pixel of the first region set in the first imaging condition, interpolated by a signal output from the second pixel selected by the selection unit A detection unit for detecting at least a part of the subject imaged in
An image processing apparatus comprising:
A pixel used for interpolation of the first pixel arranged in the first imaging area of the first imaging element, a second pixel different from the first pixel arranged in the first imaging area, and the first imaging element A third pixel arranged in the second imaging region of the second imaging element different from the selection unit for selecting from among,
A detection unit for detecting at least a part of a subject imaged in the first imaging region using a signal output from the first pixel interpolated by a signal output from the pixel selected by the selection unit; ,
An image processing apparatus comprising:
Among the imaging regions of the imaging device set in the first imaging condition, a pixel used for signal processing of a signal output from the first pixel arranged in the first region is different from the first imaging condition. Among the imaging areas set in the imaging condition, the second pixels arranged in the second area and the second pixel arranged in the second area set in the third imaging condition different from the second imaging condition. A selection unit for selecting from two pixels;
The imaging using the signal output from the first pixel in the first region set in the first imaging condition, which is signal-processed by the signal output from the second pixel selected by the selection unit A detection unit for detecting at least a part of the subject imaged in the region;
An image processing apparatus comprising:
A pixel used for signal processing of a signal output from the first pixel arranged in the first imaging region of the first imaging element is a second pixel different from the first pixel arranged in the first imaging region. A third pixel arranged in a second imaging region of a second imaging element different from the first imaging element, and a selection unit for selecting from among,
A detection unit that detects at least a part of a subject imaged in the first imaging region using a signal output from the first pixel, which is signal-processed by a signal output from the pixel selected by the selection unit. When,
An image processing apparatus comprising: