JP2022107233A - Image sensor, imaging device, and signal processing method - Google Patents

Abstract

To determine an appropriate blend rate in accordance with a feature of a subject.SOLUTION: Included are: a pixel array unit in which a first pixel for performing readout with a first light-reception sensitivity and a second pixel for performing readout with a second light-reception sensitivity lower than the first light-reception sensitivity are two-dimensionally arrayed; a comprehensive feature detecting unit which, on the basis of a first signal output from the first pixel set for a first exposure time and a second signal output from the second pixel set for a second exposure time different from the first exposure time, detects, as a comprehensive feature, a feature for a pixel region including a plurality of the first pixels and a plurality of the second pixels; and a blend ratio determining unit for determining a blend ratio of the first signal and the second signal on the basis of a detection result from the comprehensive feature detecting unit.SELECTED DRAWING: Figure 6

Description

The present technology relates to an image sensor, an image pickup device, and a signal processing method for performing HDR (High Dynamic Range) synthesis according to the characteristics of a subject.

A technique for obtaining an HDR image signal by performing HDR composition using a plurality of images having different exposure times is known. In such a technique, the blending ratio of a plurality of image signals having different exposure times is determined according to the characteristics of the subject.
For example, in Patent Document 1 below, the blending ratio is determined so as to use a pixel signal having a short exposure time, which is a signal that is strong against movement, for pixels in which a moving subject is imaged.

Japanese Unexamined Patent Publication No. 2012-257193

However, the subject detected as a moving subject includes a light emitting body that repeatedly blinks periodically. If the blending ratio is determined so as to use a pixel signal having a short exposure time for such a subject, there is a problem that the illuminant cannot be imaged normally.

This technique was made in view of the above circumstances, and aims to determine an appropriate blending ratio according to the characteristics of the subject.

The image sensor according to the present technology has two pixels, a first pixel in which reading is performed by the first light receiving sensitivity and a second pixel in which reading is performed by a second light receiving sensitivity which is lower than the first light receiving sensitivity. The pixel array unit arranged in a dimension, the first signal output from the first pixel set as the first exposure time, and the second exposure time set to a second exposure time different from the first exposure time. A global feature detection unit that detects features of a pixel region including a plurality of the first pixel and a plurality of the second pixels as global features based on a second signal output from the pixels, and the global feature detection unit. It is provided with a blend rate determination unit that determines the blend rate of the first signal and the second signal based on the detection result of the feature detection unit.
The difference in the light receiving sensitivity between the first pixel and the second pixel may be due to the difference in the light receiving area in the pixel, the difference in the length of the exposure time, or the characteristics of the light receiving element. It may be due to the difference between. Further, the global feature is a feature of a subject imaged in a set of pixel regions.

In the above-mentioned image sensor, the global feature may be a feature regarding a change in luminance with the passage of time.
As a subject having a characteristic of a change in brightness with the passage of time, for example, a light emitting body such as an LED (Light Emitting Diode) that blinks in a cycle of several msec to several tens of msec, or a light emitting body that reflects light emitted from the light emitting body is reflected. It can be mentioned as a subject whose brightness is changing, such as an object. A subject with a particularly large change in brightness is more likely to be detected. Further, even when the subject is a moving subject with a large movement, a large change in brightness is likely to be detected.

The global feature detection unit in the image sensor may perform the detection by calculating the likelihood representing the degree to which the imaged subject is a periodic blinking object as the global blinking object likelihood.
As a result, when the subject is a periodic blinking object such as an LED, the global blinking object likelihood is calculated to be high. In particular, the global blinking object likelihood is increased when a periodic blinking object is imaged in most of the target pixel region.

The global feature detection unit in the image sensor described above includes a process of calculating a temporal change amount of the brightness average value of the first pixel for each pixel region as a first pixel change amount, and the pixel region for each. A process of calculating the temporal change amount of the brightness average value of the second pixel as the second pixel change amount, and a process of calculating the difference between the first pixel change amount and the second pixel change amount as the change amount difference value. , May be performed by executing.
The amount of data for the information on the amount of change in the first pixel and the amount of change in the second pixel, which is the amount of change in the average brightness value for each pixel region over time, is smaller than that calculated for each pixel. Similarly, the amount of data is also reduced for the change amount difference value calculated from the change amount of the first pixel and the change amount of the second pixel.

In the above-mentioned image sensor, the global blinking object likelihood may be calculated so that the larger the change amount difference value is, the larger the change amount is, and the smaller the second pixel change amount is, the larger is.
For example, when the exposure time of the first pixel having a high light receiving sensitivity is shorter than the exposure time of the second pixel having a low light receiving sensitivity, the entire exposure time of the first pixel is included in the non-emission state in the periodic blinker. There is. Even in such a case, a part of the exposure time of the second pixel may be included in the light emitting state in the periodic blinker. In such a case, the change amount difference value becomes large.
Further, regarding the amount of change in the second pixel, there is a high possibility that the second pixel having a long exposure time shows a periodic blinking object in a light emitting state. In particular, when the exposure time is longer than the non-light emitting time of the periodic blinking body, the light emitting state of the periodic blinking body is surely imaged, and the amount of change in the second pixel becomes small.

The image sensor described above may include a local feature detection unit that detects features for each pixel using the global features detected by the global feature detection unit.
As a result, the characteristics of the subject are detected for each pixel.

The local feature detection unit in the image sensor described above blinks for each pixel based on the inter-pixel difference value calculated as the difference between the brightness values of the first pixel and the second pixel and the global blinking body likelihood. The body likelihood may be calculated as the local blinking body likelihood.
The pixel-to-pixel difference value and the global blinking object likelihood can be used to estimate the degree to which the subject is a periodic blinking object.

The local blinking body likelihood in the image sensor may be calculated so as to be larger for pixels having a larger difference value between pixels and larger for pixels having a larger global blinking body likelihood.
For example, as a subject having a large difference value between pixels, a moving subject, a periodic blinking object, or the like can be mentioned. Among these subjects, a subject having a large inter-pixel difference value and a high global blinking object likelihood is likely to be a periodic blinking object rather than a moving subject.

The blend rate determination unit in the image sensor may determine the blend rate so that the higher the global blinking likelihood is, the higher the ratio of the second signal is.
Depending on the relationship between the exposure times of the first pixel and the second pixel, the second pixel has a longer exposure time than the first pixel. In this case, it is highly possible that the second pixel is an image of a periodic blinker in a light emitting state.

The blend rate determining unit in the image sensor may determine the blend rate for each pixel so that the higher the local blinking likelihood is, the higher the ratio of the second signal is.
Depending on the relationship between the exposure times of the first pixel and the second pixel, the second pixel has a longer exposure time than the first pixel. In this case, the second pixel is likely to be generated in a state of capturing the periodic blinking body in the light emitting state.

The local feature detection unit in the image sensor can execute a process of calculating the local moving subject likelihood, and the local moving subject likelihood is larger as the pixel-to-pixel difference value is larger. It may be calculated so that the pixel with the smaller global blinking likelihood is larger.
For example, as a subject having a large difference value between pixels, a moving subject, a periodic blinking object, or the like can be mentioned. Among these subjects, a subject having a large inter-pixel difference value and a low global blinking likelihood is likely to be a moving subject.

The global feature detection unit in the image sensor described the process of calculating the temporal change amount of the brightness of the first pixel for each pixel area as the first pixel change amount, and the second pixel area for each pixel area. A process of calculating the temporal change amount of the pixel brightness as the second pixel change amount and a process of calculating the difference between the first pixel change amount and the second pixel change amount as the change amount difference value are executed. The global moving subject likelihood, which indicates the degree to which the imaged subject is a moving subject, may be calculated so as to be larger as the change amount difference value is smaller and larger as the second pixel change amount is larger.
For example, even if the light receiving sensitivities are different, the amount of change in the first signal and the second signal output from the first pixel and the second pixel is large when the subject is a moving subject, so the difference is small. Can be considered.

The image sensor described above includes a local feature detection unit that executes a process of calculating the local moving subject likelihood, which is the moving subject likelihood for each pixel, and the local moving subject likelihood is the first pixel. It may be calculated so that the larger the pixel-to-pixel difference value calculated as the difference in the brightness of the second pixel is larger, and the larger the pixel is, the larger the global moving subject likelihood is.
For example, as a subject having a large difference value between pixels, a moving subject, a periodic blinking object, or the like can be mentioned. Among these subjects, a subject having a large inter-pixel difference value and a high local moving subject likelihood is likely to be a moving subject rather than a periodic blinking object.

The blend rate determination unit in the image sensor may determine the blend rate so that the higher the global moving subject likelihood, the higher the ratio of the first signal.
The first signal is a signal obtained after an exposure time shorter than that of the second signal. Therefore, the image data generated based on the first signal has less blurring of the moving subject than the image data generated based on the second signal.

The blend rate determination unit in the image sensor may determine the blend rate for each pixel so that the higher the local moving subject likelihood is, the higher the ratio of the first signal is.
The first signal is a signal obtained after an exposure time shorter than that of the second signal. Therefore, the image data generated based on the first signal has less blurring of the moving subject than the image data generated based on the second signal.

In the image sensor described above, the first pixel and the second pixel may be configured to have different light receiving sensitivities due to different light receiving areas.
As a result, it is possible to detect the global feature of each pixel region by using the first pixel and the second pixel having different light receiving sensitivities.

In the image sensor described above, the exposure time of the first pixel may be shorter than the exposure time of the second pixel.
This makes it possible to detect, for example, the characteristic of a periodic blinking body that blinks periodically as a global characteristic.

The pixel array unit in the image sensor described above can output a third signal read by a third light receiving sensitivity that is different from both the first light receiving sensitivity and the second light receiving sensitivity, and the global feature detection can be performed. The unit detects a global feature in a pixel region including a plurality of pixels from which the third signal is output based on one of the first signal and the second signal and the third signal. You may.
As a result, the blend rate determination unit determines the blend rate for blending the three pixel signals.

The image pickup apparatus according to the present technology includes an image sensor that performs photoelectric conversion and an optical member that collects the reflected light from the subject on the image sensor, and the image sensor reads out by the first light receiving sensitivity. A pixel array unit in which the first pixel to be read and the second pixel to be read by the second light receiving sensitivity, which is lower than the first light receiving sensitivity, are arranged in two dimensions.
The first signal output from the first pixel, which is the first exposure time, and the second signal, which is output from the second pixel, which is the second exposure time different from the first exposure time. Based on the above, the global feature detection unit that detects features of a pixel region including a plurality of the first pixel and the second pixel as global features, and the global feature detection unit based on the detection results of the global feature detection unit. It is provided with a blend rate determining unit for determining the blend rate of the first signal and the second signal.

In the signal processing method according to the present technology, the first signal output from the first pixel, which is read out by the first light receiving sensitivity according to the first exposure time, and the second exposure time different from the first exposure time. A plurality of the first pixel and the second pixel are each based on the second signal output from the second pixel in which the reading is performed by the second light receiving sensitivity, which is lower than the first light receiving sensitivity. A process including a process of detecting a feature of a included pixel region as a global feature and a process of determining a blend ratio of the first signal and the second signal based on the detection result of the global feature. Is.
The above-mentioned various actions can also be obtained by such an imaging device and a signal processing method.

It is a block diagram which shows the structural example of the image sensor which concerns on this technique. It is the schematic which shows the structural example of a pixel unit. It is the schematic which shows the structural example of the pixel array part. It is a schematic diagram which shows the example in which a plurality of pixel regions are set in a pixel array part. It is a schematic diagram which shows the example which nine pixel regions were set in the pixel array part. It is a block diagram which shows the structural example of a signal processing part. It is a block diagram which shows the structural example of the global feature detection part. It is the schematic which shows the structural example of the pixel array part which has the color filter of the Bayer array. It is a figure for demonstrating an example of detecting a global feature from the change amount difference value and the 2nd pixel change amount. It is a graph for demonstrating the local blinking body likelihood calculated based on the global blinking body likelihood and the inter-pixel difference value. It is a graph for demonstrating the local moving subject likelihood calculated based on the global moving subject likelihood and the inter-pixel difference value. It is a graph which shows the relationship between the amount of light incident on a pixel, and the temporary blend ratio. It is a block diagram which shows the structural example of the image pickup apparatus in the 1st application example. It is a block diagram which shows the configuration example of the in-vehicle system in the 2nd application example. It is a block diagram which shows the structural example of the signal processing part in the example which uses three or more signals.

Hereinafter, embodiments according to the present technology will be described in the following order with reference to the accompanying drawings.
<1. Image sensor configuration>
<2. Signal processing unit configuration and processing content>
<3. First application example>
<4. Second application example>
<5. Example of using three or more signals>
<6. Modification example>
<7. Summary>
<8. This technology>

<1. Image sensor configuration>
FIG. 1 shows the configuration of the image sensor 1 according to the embodiment of the present technology.
The image sensor 1 includes a pixel array unit 2, a first read circuit 3, a second read circuit 4, and a signal processing unit 5.

The pixel array unit 2 is formed by arranging a plurality of pixel units 6 in the row direction and the column direction. The pixel unit 6 is configured to include a plurality of pixels G. The pixel G has a photoelectric conversion element, and obtains an electric signal according to the amount of received light. In the present embodiment, an example in which one pixel unit 6 includes two pixels G will be described. However, the pixel unit 6 may be configured to include three or more pixels G.

A configuration example of the pixel unit 6 is shown in FIG.
The pixel unit 6 includes a first pixel G1 and a second pixel G2 having different light receiving sensitivities.

The first pixel G1 has a larger light receiving area than the second pixel G2. Further, the exposure time of the first pixel G1 is shorter than the exposure time of the second pixel G2. As an example, the exposure time of the first pixel G1 is 8 msec, and the exposure time of the second pixel G2 is 11 msec.

The light receiving sensitivity of the first pixel G1 is made higher than the light receiving sensitivity of the second pixel G2. Therefore, the first pixel G1 is more likely to be saturated than the second pixel G2.

The first pixel G1 and the second pixel G2 receive the reflected light reflected by the subject (imaging subject), perform photoelectric conversion, and output it as a pixel signal in the subsequent stage. The pixel signal output from the first pixel G1 is referred to as the first signal S1, and the pixel signal output from the second pixel G2 is referred to as the second signal S2.

In the following description, the difference in light receiving sensitivity between the first pixel G1 and the second pixel G2 will be described with reference to an example based on the light receiving area and the exposure time, but other cases may also be considered. For example, the difference in light receiving sensitivity between the first pixel G1 and the second pixel G2 may be based only on the light receiving area, may be based only on the length of the exposure time, or may be based on the light receiving element (PD). : Photodiode) may be based only on the difference in characteristics. Of course, it may be based on these multiple factors.

Returning to the description of FIG.
The first read circuit 3 is a signal processing circuit for AD (Analog to Digital) conversion and noise reduction, as well as a configuration for reading the first signal S1 from the first pixel G1 provided in each pixel unit 6 of the pixel array unit 2. Etc. are provided. The first signal S1 is supplied to the signal processing unit 5 via the first read circuit 3.

The second read circuit 4 has a configuration for reading the second signal S2 from the second pixel G2 provided in each pixel unit 6, and also includes a signal processing circuit for AD conversion and noise reduction. The second signal S2 is supplied to the signal processing unit 5 via the second read circuit 4.

The signal processing unit 5 uses the input first signal S1 and second signal S2 to perform a process of detecting a global feature and a process of detecting a local feature.
Further, the signal processing unit 5 determines the blend ratio of the first signal S1 and the second signal S2 by using the detected global feature and local feature, and the first signal S1 and the second signal S1 and the second signal S2 are determined based on the blend ratio. HDR (High Dynamic Range) synthesis is performed by blending the signal S2. As a result, the HDR image signal is output from the signal processing unit 5.

The global feature is a feature of a subject detected for each pixel area GA set as a rectangular area including, for example, a plurality of pixel units 6. Specifically, when the subject imaged in a certain pixel region GA is a light emitting body that blinks periodically, the feature as the subject that blinks periodically is detected. The signal processing unit 5 calculates the degree (high possibility) that the subject is a light emitting body that blinks periodically as a global feature as a likelihood.

In the following example, a subject that blinks periodically is referred to as a "periodic blinking body". The periodic blinking body includes not only a subject that emits light by itself such as an LED (Light Emitting Diode) but also a subject that reflects the light of an LED that repeatedly turns on and off in several msec.
The signal processing unit 5 calculates how much the subject in the pixel region GA is like a periodic blinker as “global blinker likelihood LFL1”. That is, the global blinking object likelihood LFL1 is an index value indicating the degree of the periodic blinking object of the subject in the pixel region GA. If the subject imaged in the pixel region GA is an LED, the global blinker likelihood LFL1 becomes high. Further, the more the LEDs are imaged in the many pixel units 6 in the pixel region GA, the higher the global blinker likelihood LFL1 becomes.

Further, the signal processing unit 5 calculates how much the subject has the characteristics as a moving subject for each pixel region GA as "global moving subject likelihood LFM1". That is, the global moving subject likelihood LFM1 is an index value indicating the degree of the moving subject of the subject. If the subject imaged in the pixel region GA is a moving subject, the global moving subject likelihood LFM1 becomes high. Further, the more the moving subject is imaged in many pixel units 6 in the pixel region GA, the higher the global moving subject likelihood LFM1 becomes.

It can be said that the detection of a periodic blinking object or a moving subject detects a subject whose brightness value changes in the pixel region GA with the passage of time. That is, these global features can be rephrased as features regarding the change in brightness with the passage of time.

The signal processing unit 5 treats the plurality of pixel units 6 as the pixel region GA in order to detect the global feature. For example, the pixel units 6 arranged two-dimensionally as shown in FIG. 3 are divided into several rectangular regions as shown in FIG. 4, and each rectangular region is treated as a pixel region GA.
For example, as shown in FIG. 5, when each of the vertical direction and the horizontal direction is divided into three pixel region GAs, nine pixel region GAs are set. The signal processing unit 5 detects global features for each of the nine pixel region GAs.

The local feature detected by the signal processing unit 5 is a feature of the subject detected for each pixel unit 6. That is, the feature of the subject detected for the first pixel G1 of the pixel unit 6 is treated as the feature of the subject in the pixel unit 6 to which the first pixel G1 belongs. The same applies to the second pixel G2.

The signal processing unit 5 treats the feature as a periodic blinker and the feature as a moving subject detected in the pixel unit 6 as local features. Further, the signal processing unit 5 calculates the local blinking body likelihood LFL2 and the local moving subject likelihood LFM2 as index values indicating the degree of the local feature. Local features are detected using global features and the like.

<2. Signal processing unit configuration and processing content>
A configuration example of the signal processing unit 5 is shown in FIG. The signal processing unit 5 includes an inter-pixel difference detection unit 7, a global feature detection unit 8, a local feature detection unit 9, a blend rate determination unit 10, and a blend unit 11.

The pixel-to-pixel difference detection unit 7 detects the difference in brightness between the first pixel G1 and the second pixel G2 for each pixel unit 6, and calculates the difference in brightness as the “pixel-to-pixel difference value GD”. The inter-pixel difference value GD can be effectively used in the subsequent processing because the exposure times of the first pixel G1 and the second pixel G2 are different.
The inter-pixel difference value GD is used in the local feature detection unit 9 to select whether the subject has a feature as a periodic blinker or a feature as a moving subject. Details will be described later.
The amount of data can be compressed by expressing the difference value GD between pixels with a small number of bits such as 256 steps from 0 to 255.

As described above, the global feature detection unit 8 detects the feature of the subject imaged in the pixel region GA as the global feature. Specifically, the global feature detection unit 8 calculates the global blinking body likelihood LFL1 and the global moving subject likelihood LFM1 for each pixel region GA as global features. Depending on the pixel region GA, only the global blinking body likelihood LFL1 may be calculated, or only the global moving subject likelihood LFM1 may be calculated.
In the present embodiment, the global blinker likelihood LFL1 and the global moving subject likelihood LFM1 are calculated as values in the range of 0.0 to 1.0. The global blinking body likelihood LFL1 and the global moving subject likelihood LFM1 indicate that the higher the numerical value, the higher the feature of the subject imaged in the target pixel region GA.

Here, a more specific configuration example of the global feature detection unit 8 is shown in FIG.
The global feature detection unit 8 includes a first brightness calculation unit 21A that performs processing related to the first signal S1, a region-specific brightness average value first calculation unit 22A, a first memory 23A, and a region-specific brightness change first calculation unit 24A. I have.

Further, the global feature detection unit 8 includes a second brightness calculation unit 21B that performs processing related to the second signal S2, a region-specific brightness average value second calculation unit 22B, a second memory 23B, and a region-specific brightness change second calculation unit. It is equipped with 24B.

Further, the global feature detection unit 8 calculates the feature amount for each pixel region GA using the signals output from the region-by-region luminance change first calculation unit 24A and the region-by-region luminance change second calculation unit 24B. The quantity calculation unit 25 is provided.

The first luminance calculation unit 21A calculates the luminance value for each first pixel G1 based on the first signal S1. As an example, each pixel unit 6 in the pixel array unit 2 has a Bayer array color filter. Specifically, as illustrated in FIG. 8, the pixel array unit 2 has a pixel unit 6R as an R pixel having an R (red) color filter and a G pixel having a G (green) color filter. It has a pixel unit 6G and a pixel unit 6B as a B pixel having a B (blue) color filter.

The pixel unit 6R has a first pixel G1R and a second pixel G2R. Similarly, the pixel unit 6G has a first pixel G1G and a second pixel G2G, and the pixel unit 6B has a first pixel G1B and a second pixel G2B.

The first luminance calculation unit 21A treats four pixel units 6 composed of two pixel units 6 vertically and horizontally as one pixel set GS for the pixel array unit 2 arranged as shown in FIG. 8, and sets the pixels. The brightness value L1S for the first pixel G1 is calculated for each GS. For example, the luminance value L1S for the pixel set GS is calculated using the following equation (1).

Luminance value of pixel set GS L1S = α × L1R + β × L1G + γ × L1B ... Equation (1)
however,
Luminance value of first pixel G1R = L1R
Luminance value of first pixel G1G = L1G
Luminance value of first pixel G1B = L1B
And.

Various coefficients α, β, and γ can be considered. For example, the coefficient α = 2, the coefficient β = 1, and the coefficient γ = 2. Alternatively, the coefficient α = 0.4, the coefficient β = 0.2, and the coefficient γ = 0.4 may be set.

The calculated luminance value L1S is a luminance value for the first pixel G1 in one pixel set GS, but the luminance value L1S is the luminance value L1R of the first pixel G1R, the luminance value L1G of the first pixel G1G, and the first. It may be regarded as the brightness value L1B of the pixel G1B.

In summary, the first luminance calculation unit 21A calculates the luminance values L1R, L1G, L1B for each pixel unit 6, or the luminance value L1S for each pixel set GS.

The region-specific luminance average value first calculation unit 22A calculates the luminance average value L1A of the first pixel G1 for each pixel region GA. For example, when the pixel region GA includes n pixel set GS, the brightness value L1S for the first pixel G1 calculated for each pixel set GS is added and divided by n to obtain the brightness. The average value L1A is calculated.

The calculated brightness average value L1A is output to the area-by-region brightness change first calculation unit 24A and stored in the first memory 23A.

The first memory 23A is provided to store the brightness average value L1A one frame before.
The region-by-region brightness change first calculation unit 24A acquires the brightness average value L1A one frame before from the first memory 23A, and acquires the brightness average value L1A for the current frame from the region-by-region brightness average value first calculation unit 22A. .. The region-by-region brightness change first calculation unit 24A calculates the difference between the brightness average value L1A one frame before and the brightness average value L1A of the current frame. As a result, the brightness change of the first pixel G1 is calculated as the first pixel change amount G1V for each pixel region GA. The calculated first pixel change amount G1V is output to the feature amount calculation unit 25 for each area.

The second brightness calculation unit 21B, the area-by-region brightness average value second calculation unit 22B, the second memory 23B, and the area-by-region luminance change second calculation unit 24B are processing units that perform processing related to the second signal S2, and each of them is a second. 1 The same processing as that of the brightness calculation unit 21A, the area-by-region brightness average value first calculation unit 22A, the first memory 23A, and the area-by-region brightness change first calculation unit 24A is performed.

Specifically, the second luminance calculation unit 21B calculates the luminance value L2S for the second pixel G2 for each pixel set GS. For example, the luminance value L2S for the pixel set GS is calculated using the following equation (2).

Luminance value of pixel set GS L2S = α'x L2R + β'x L2G + γ'x L2B ... Equation (2)
however,
Luminance value of second pixel G2R = L2R
Luminance value of second pixel G2G = L2G
Luminance value of second pixel G2B = L2B
And.

The coefficients α', β', and γ'may be, for example, a coefficient α'= 2, a coefficient β'= 1, a coefficient γ'= 2, a coefficient α'= 0.4, and a coefficient β'= 0. .2, the coefficient γ'= 0.4 may be set.

The second calculation unit 22B of the brightness average value for each region calculates the brightness average value L2A of the second pixel G2 for each pixel region GA.

The calculated brightness average value L2A is output to the region-by-region brightness change second calculation unit 24B and stored in the second memory 23B.

The region-by-region brightness change second calculation unit 24B acquires the brightness average value L2A one frame before from the second memory 23B, and acquires the brightness average value L2A for the current frame from the region-by-region brightness average value second calculation unit 22B. .. The second calculation unit 24B of the brightness change for each region calculates the difference between the brightness average value L2A one frame before and the brightness average value L2A of the current frame. As a result, the second pixel change amount G2V of the second pixel G2 is calculated for each pixel region GA. The calculated second pixel change amount G2V is output to the feature amount calculation unit 25 for each area.

The feature amount calculation unit 25 for each area uses the first pixel change amount G1V for each pixel area GA for the first pixel G1 and the second pixel change amount G2V for each pixel area GA for the second pixel G2 to use the pixel area GA. Calculate the feature amount for each.

Specifically, the feature amount calculation unit 25 for each area calculates the absolute value of the difference between the first pixel change amount G1V and the second pixel change amount G2V for the pixel area GA as the change amount difference value VD (≧ 0). ..
Then, the feature amount calculation unit 25 for each area calculates the feature amount for each pixel area GA based on the change amount difference value VD and the second pixel change amount G2V.

For example, the feature amount is calculated so that the larger the change amount difference value VD and the smaller the second pixel change amount G2V, the higher the global blinker likelihood LFL1.

In the pixel region GA in which the change amount difference value VD is large, it is shown that the change in the brightness value of the subject can be captured only by either one of the first pixel G1 and the second pixel G2. Specifically, the first pixel change amount G1V can be rephrased as a change in the first signal S1 output from the first pixel G1 having a short exposure time. Since the exposure time of the first pixel G1 is short, when the subject is a periodic blinker, the LED in the non-light emitting state is imaged at the timing one frame before, and the LED in the light emitting state is imaged at the timing of the current frame. There is. In this case, the amount of change in the first pixel G1V with respect to the first pixel G1 becomes large.

On the other hand, the second pixel change amount G2V can be rephrased as a change in the second signal S2 output from the second pixel G2 having a long exposure time. Since the exposure time of the second pixel G2 is long, when the subject is a periodic blinker, there is a high possibility that the LED in the light emitting state is imaged at both the timing one frame before and the timing of the current frame. Therefore, the second pixel change amount G2V becomes small. In particular, when the exposure time of the second pixel G2 is longer than the length of the non-emission time of the periodic blinker, there is no frame in which only the non-emission time of the periodic blinker is imaged, so that the amount of change in the second pixel is changed. G2V is small.

Therefore, when the subject is a periodic blinker, the change amount difference value VD becomes large. Therefore, it can be estimated that there is a high possibility that the periodic blinking body is imaged in the pixel region GA in which the change amount difference value VD is large and the second pixel change amount G2V is small.

It should be noted that the global blinking body likelihood LFL1 calculated by the feature amount calculation unit 25 for each region has strength and weakness of certainty.
For example, in the feature amount calculation unit 25 for each region, the larger the change amount difference value VD and the smaller the second pixel change amount G2V, the higher the global blinker likelihood LFL1. Calculate LFL1.

Further, the area-by-region feature amount calculation unit 25 calculates the global moving subject likelihood LFM1 as the feature amount for each pixel region GA.

For example, the feature amount is calculated so that the smaller the change amount difference value VD and the larger the second pixel change amount G2V, the higher the global moving subject likelihood LFM1.

The amount of change in the brightness value of the first pixel change amount G1V and the second pixel change amount G2V is large when the subject is a moving subject, although there is a difference in exposure time. That is, since both the first pixel change amount G1V and the second pixel change amount G2V are large, the change amount difference value VD is small.

Therefore, it can be estimated that there is a high possibility that a moving subject has been imaged in the pixel region GA in which the change amount difference value VD is small and the second pixel change amount G2V is large.

The global moving subject likelihood LFM1 calculated by the feature amount calculation unit 25 for each region has strength and weakness of certainty.
For example, the feature amount calculation unit 25 for each region increases the global moving subject likelihood LFM1 so that the smaller the change amount difference value VD and the larger the second pixel change amount G2V, the larger the global moving subject likelihood. Calculate LFM1.

The global blinking body likelihood LFL1 and the global moving subject likelihood LFM1 calculated by the feature amount calculation unit 25 for each region have values in a range of 0.0 to 1.0. Specifically, it will be described with reference to FIG.

The graph shown in FIG. 9 shows the global characteristics of each pixel region GA with the change amount difference value VD as the horizontal axis and the second pixel change amount G2V as the vertical axis.
The pixel region GA in which the periodic blinker is imaged is distributed in the region AR1 near the horizontal axis of the graph. Further, the pixel region GA in which the moving subject is photographed is distributed in the region AR2 near the vertical axis of the graph.

Further, as for the global blinking body likelihood LFL1, the global blinking body likelihood LFL1 is increased as the pixel region GA having a larger change amount difference value VD and the pixel region GA having a smaller second pixel change amount G2V. That is, as shown in FIG. 9, the global blinker likelihood LFL1 of the point P1 located in the region AR1 is 1.0, and the global blinker likelihood LFL1 of the point P2 located in the region AR1 is 0. It is said to be 0.1.

Similarly, as for the global moving subject likelihood LFM1, the larger the pixel region GA having a smaller change amount difference value VD and the larger the pixel region GA having a larger second pixel change amount G2V, the higher the global moving subject likelihood LFM1. That is, the global moving subject likelihood LFM1 of the point P3 located in the region AR2 is 1.0, and the global moving subject likelihood LFM1 of the point P4 located in the region AR2 is 0.1.

It should be noted that the pixel region GA having a larger change amount difference value VD and the pixel region GA having a smaller second pixel change amount G2V do not increase the global blinking likelihood LFL1 but are located near the center of the region AR1. The global blinking likelihood LFL1 for the pixel region GA may be 1.0.

Similarly, the pixel region GA having a larger change amount difference value VD and the pixel region GA having a smaller second pixel change amount G2V do not increase the global blinking likelihood LFL1 but are located near the center of the region AR2. The global moving subject likelihood LFM1 for the located pixel region GA may be 1.0.

The region-by-region feature amount calculation unit 25 outputs the global blinking body likelihood LFL1 and the global moving subject likelihood LFM1 for each pixel region GA.

Returning to the description of FIG.
The local feature detection unit 9 performs a process of detecting the feature of the subject for each pixel unit 6 as a local feature. The global feature detected by the global feature detection unit 8 is the one that detects the feature of the subject in the pixel region GA, and for the specific pixel unit 6 in the pixel region GA, the global feature in which the feature of the subject is detected is detected. It may not be the case. In such a case, if the blending ratio is determined based on the global characteristics, an inappropriate blend may be performed in the specific pixel unit 6. In view of such circumstances, the local feature detection unit 9 detects the local feature for each pixel G.

Specifically, the local feature detection unit 9 uses the global blinking likelihood LFL1 output from the region-by-region feature amount calculation unit 25 and the pixel-to-pixel difference value GD output from the pixel-to-pixel difference detection unit 7. , The local blinker likelihood LFL2 is calculated. Further, the local moving subject likelihood LFM2 is calculated by using the global moving subject likelihood LFM1 output from the region-by-region feature amount calculation unit 25 and the pixel-to-pixel difference value GD output from the pixel-to-pixel difference detecting unit 7. ..

First, the calculation of the local blinker likelihood LFL2 will be described with reference to FIG.
The graph shown in FIG. 10 shows the data for each pixel unit 6 with the global blinker likelihood LFL1 as the horizontal axis and the pixel-to-pixel difference value GD as the vertical axis. Here, the difference value GD between pixels on the vertical axis is normalized so as to take a value of 0.0 to 1.0.

In order to prevent erroneous determination due to noise or the like, the local feature detection unit 9 has a region in which the global blinker likelihood LFL1 is set to a predetermined threshold Th1 or more and the pixel-to-pixel difference value GD is set to a predetermined threshold Th2 or more. For the pixel unit 6 in which data is plotted on Ar3, it is determined that the subject has a characteristic as a periodic blinker, and the local blinker likelihood LFL2 is calculated.

The local blinker likelihood LFL2 takes any value from 0.0 to 1.0. For example, as shown in FIG. 10, for the pixel unit 6 plotted at the point P5 in which the global blinking body likelihood LFL1 is set to the threshold value Th1 and the pixel-to-pixel difference value GD is set to the threshold value Th2, the local blinking body likelihood is LFL2 is calculated as 0.0. Further, with respect to the pixel unit 6 plotted at the point P6 in which the global blinking body likelihood LFL1 and the pixel-to-pixel difference value GD are both 1.0, it is very likely that the captured subject is a periodic blinking body. Therefore, the local blinker likelihood LFL2 is calculated as 1.0.

In FIG. 10, the local blinker likelihood LFL2 is set to approach 1.0 as the point P5 approaches the point P6.
The local blinking likelihood LFL2 for the pixel unit 6 in which data is plotted in a region other than the region Ar3 may have no value, may be a negative value, or may be 0.0.

Next, the calculation of the local moving subject likelihood LFM2 will be described with reference to FIG.
The graph shown in FIG. 11 shows data for each pixel with the global moving subject likelihood LFM1 as the horizontal axis and the pixel-to-pixel difference value GD as the vertical axis. Here, the difference value GD between pixels on the vertical axis is normalized so as to take a value of 0.0 to 1.0.

The local feature detection unit 9 has a region in which the global moving subject likelihood LFM1 is set to a predetermined threshold Th3 or more and the pixel-to-pixel difference value GD is set to a predetermined threshold Th4 or more in order to prevent erroneous determination due to noise or the like. With respect to the pixel unit 6 in which the data is plotted on Ar4, it is determined that the subject has a characteristic as a moving subject, and the local moving subject likelihood LFM2 is calculated.

The local moving subject likelihood LFM2 takes any value from 0.0 to 1.0. For example, as shown in FIG. 11, the local moving subject likelihood is plotted at the point P7 in which the global moving subject likelihood LFM1 is set to the threshold Th3 and the pixel-to-pixel difference value GD is set to the threshold Th4. LFM2 is calculated as 0.0. Further, with respect to the pixel unit 6 plotted at the point P8 in which the global moving subject likelihood LFM1 and the inter-pixel difference value GD are both 1.0, it is very likely that the captured subject is a moving subject. The local moving subject likelihood LFM2 is calculated as 1.0.

In FIG. 11, the local moving subject likelihood LFM2 is set to approach 1.0 as the point P7 approaches the point P8.
The local moving subject likelihood LFM2 for the pixel unit 6 in which data is plotted in a region other than the region Ar4 may have no value, may be a negative value, or may be 0.0.

In this way, the local feature detection unit 9 performs a process of dropping the global feature of the subject detected in each pixel region GA into the local feature of the pixel unit 6 (or each pixel G).

The local moving subject likelihood LFM2 may be calculated by using the global blinking body likelihood LFL1 without using the global moving subject likelihood LFM1.
For example, as a subject having a large inter-pixel difference value GD, a periodic blinking object or a moving subject can be considered. In such a pixel region GA, when the global blinking body likelihood LFL1 is large, it is highly possible that the subject imaged in the pixel region GA is a periodic blinking body. On the other hand, in the pixel region GA having a large inter-pixel difference value GD and having a low global blinking likelihood LFL1, there is a high possibility that the captured subject is a moving subject. Therefore, it is possible to estimate that the subject imaged by the pixel unit 6 is a moving subject without using the global moving subject likelihood LFM1.

Returning to the description of FIG.
The blend rate determination unit 10 determines the blend rate of the first signal S1 and the second signal S2. In particular, in the present embodiment, the blend rate determination unit 10 determines the blend rate according to the characteristics of the subject for each pixel unit 6.

Therefore, the blend rate determination unit 10 includes a temporary blend rate calculation unit 12 for calculating the temporary blend rate ABF, and a blend rate correction unit 13 for calculating the blend rate BF by correcting the temporary blend rate ABF. ..

The temporary blend rate calculation unit 12 calculates the temporary blend rate in a state where the characteristics of the subject are not taken into consideration as the temporary blend rate ABF. The temporary blend ratio ABF and the blend ratio BF represent the blend ratio of the second signal S2, and take a value from 0.0 to 1.0. That is, when the temporary blend ratio ABF and the blend ratio BF are 0.0, it means that the second signal S2 is not blended at all, and when it is 1.0, the blend ratio of the second signal S2 is 100%. Represents.

The first signal S1 output from the first pixel G1 having high light receiving sensitivity is input to the temporary blend ratio calculation unit 12. The temporary blend rate calculation unit 12 determines the temporary blend rate ABF according to the degree of saturation of the first signal S1. For example, as shown in FIG. 12, the temporary blend ratio ABF is set to 0.0 until the amount of incident light on the pixel G exceeds a predetermined threshold value Th5. Further, when the amount of incident light on the pixel G is equal to or more than a predetermined threshold value Th5 and less than the threshold value Th6, the provisional blend ratio ABF is set to 0.0 or more and less than 1.0. Then, when the amount of incident light on the pixel G exceeds a predetermined threshold value Th6, the temporary blend ratio ABF is fixed at 1.0. That is, when the amount of incident light on the pixel G exceeds the threshold value Th6, it is determined that the first pixel G1 is saturated, and the blend ratio of the first signal S1 becomes 0%.

The blend rate correction unit 13 uses the local blinking body likelihood LFL2 and the local moving subject likelihood LFM2 calculated by the local feature detection unit 9, and the temporary blend rate calculated by the temporary blend rate calculation unit 12. ABF is corrected.

Specifically, when the local blinker likelihood LFL2 is large, the provisional blend ratio ABF is corrected so that the blend ratio BF becomes high, that is, the blend ratio of the second signal S2 becomes high. This is because the second signal S2 is a signal obtained by relatively lengthening the exposure time, and there is a high possibility that the light emitting state of a periodic blinking object such as an LED is imaged.

Further, when the local blinking body likelihood LFL2 is small and the local moving subject likelihood LFM2 is large, the blend ratio BF is tentatively reduced, that is, the ratio of the second signal S2 is tentatively reduced. The blend ratio ABF is corrected. This is because the first signal S1 is a signal obtained by relatively shortening the exposure time, and blurring of a moving subject is suppressed.

As an example, for the pixel G whose local blinker likelihood LFL2 is larger than a predetermined value, the first signal S1 may be excluded by setting the blend ratio BF to 1.0. When the local blinking body likelihood LFL2 is smaller than the predetermined value and the local moving subject likelihood LFM2 is larger than the predetermined value, the blend ratio BF is set to 0.0 to set the second signal S2. May not be included.

The blending unit 11 performs a blending process based on the blending ratio BF determined by the blending ratio correcting unit 13. For example, the blending unit 11 generates an HDR image signal by performing an alpha blending process based on the blending ratio BF. The alpha blend can be expressed by the following equation (3).

HDR image signal = (1-α) × S1 + α × S2 × Gain ... Equation (3)

When the local blinking body likelihood LFL2 is smaller than a predetermined value, the correction is made so that the blend rate BF is 0.0 regardless of the value of the local moving subject likelihood LFM2, that is, the blend rate BF. May be corrected so that

<3. First application example>
In the first application example, a case where the above-mentioned image sensor 1 is applied to the image pickup apparatus 100 will be described.

A configuration example of the image pickup apparatus 100 is shown in FIG. In addition, in FIG. 13, only the main part included in the image pickup apparatus 100 is shown.

The image pickup apparatus 100 includes an optical lens system 101, an image sensor 1 including the above-mentioned signal processing unit 5, an image pickup signal processing unit 102, a control unit 103, a driver unit 104, a storage unit 105, and a communication unit. It includes 106 and a display unit 107.

The optical lens system 101 includes various lenses such as a zoom lens and a focus lens, an aperture mechanism, and the like. Light that has passed through the optical lens system 101 is incident on the pixel array unit 2 included in the image sensor 1.
A part of the optical lens system 101 may be provided in the image sensor 1 like a microlens provided for each pixel G.

The imaging signal processing unit 102 receives the HDR image signal synthesized by HDR from the image sensor 1, and the signal processing required to generate the image data for recording and the signal required to display the image on the display unit 107. Perform processing etc.

The control unit 103 controls the image sensor 1 and the image pickup signal processing unit 102. Further, by sending a control signal to the driver unit 104, various optical members such as various lenses and an aperture mechanism of the optical lens system 101 are controlled.

The driver unit 104 provides a drive signal to the optical member of the optical lens system 101 based on the control signal input from the control unit 103. As a result, the optical member is driven.

The storage unit 105 is composed of, for example, a non-volatile memory or the like. The storage unit 105 stores programs and the like required for each process executed by the control unit 103. Further, the storage unit 105 stores image data such as still image data and moving image data generated by the imaging signal processing unit 102.

The storage unit 105 may be configured as a flash memory built in the image pickup device 100, or may be a memory card (for example, a portable flash memory) that can be attached to and detached from the image pickup device 100, and the memory card can be stored or read. It may be composed of an access unit that performs access for the purpose. Further, it may be realized as an HDD (Hard Disk Drive) or the like as a form built in the image pickup apparatus 100.

The communication unit 106 performs data communication and network communication with an external device by wire or wirelessly. For example, image data (still image data or moving image data) is transmitted to an external display device, recording device, playback device, or the like.
Further, the communication unit 106 may perform communication by various networks such as the Internet, a home network, and a LAN (Local Area Network), and transmit and receive various data to and from a server, a terminal, or the like on the network. ..
Further, the communication unit 106 may function as an output unit that outputs image data to an external device.

The display unit 107 can display a reproduced image of the image data read from the recording medium as the storage unit 105. Further, the display unit 107 may be provided on the image pickup apparatus 100 as a rear monitor, a finder monitor, or the like provided on the back surface of the image pickup apparatus 100, and may be capable of displaying a so-called through image.

The display unit 107 may be in a form that can be attached to and detached from the image pickup apparatus 100.

In the imaging signal processing unit 102 of the present embodiment, the first signal S1 output from the first pixel G1 and the second signal S2 output from the second pixel G2 are appropriately blended according to the characteristics of the subject. The HDR image signal can be received from the image sensor 1. Therefore, by performing various display processing, processing, and the like based on the HDR image signal, the imaging signal processing unit 102 appropriately displays a periodic blinking object such as an LED and suppresses blurring of a moving subject. It is possible to display HDR images and generate image files.

<4. Second application example>
In the second application example, an example in which the HDR image signal is provided from the image sensor 1 described above to the display device and the recognition device that performs the image recognition process will be described.
As such an application example, the in-vehicle system 200 will be mentioned here.

The in-vehicle system 200 includes an optical lens system 201, a driver unit 202, an image sensor 1 including the above-mentioned signal processing unit 5, a display device 300, and a recognition device 400.

Since the optical lens system 201 has the same configuration as the optical lens system 101 described in the first application example, and the driver unit 202 has the same configuration as the driver unit 104 described in the first application example, the description thereof will be omitted. ..

The display device 300 includes monitors and instruments provided at predetermined positions in the vehicle interior such as an instrument panel, and a circuit for controlling the display thereof.
The display device 300 includes a display signal processing unit 301, a display control unit 302, a storage unit 303, a communication unit 304, a display unit 305, and the like.

The display signal processing unit 301 performs signal processing for displaying an image on the display unit 305, and has the same configuration as the imaging signal processing unit 102 described in the first application example.

The display control unit 302 has the same configuration as the control unit 103 described in the first application example. Further, the storage unit 303 and the communication unit 304 have the same configuration as the storage unit 105 and the communication unit 106 described in the first application example. The communication unit 304 is capable of communication using CAN (Controller Area Network).

The display unit 305 is a monitoring device or the like provided at a predetermined position in the vehicle interior, and can display image data captured by the image sensor 1. As a result, the driver and passengers can grasp the situation outside the vehicle through the monitoring device.

The recognition device 400 performs subject recognition processing by performing various processing on the image data of the outside of the vehicle captured by the image sensor 1. The recognition result of the subject by the recognition device 400 is provided to the advanced driver-assistance system (ADAS).
The ADAS realizes a collision damage mitigation braking function, an ACC (Adaptive Cruise Control) function, and the like based on the information acquired from the recognition device 400.

The recognition device 400 includes a recognition signal processing unit 401, a recognition control unit 402, a storage unit 403, and a communication unit 404.

Since the storage unit 403 and the communication unit 404 have the same configuration as the storage unit 105 and the communication unit 106 described in the first application example, detailed description thereof will be omitted.

Data about the captured image is input from the image sensor 1 to the recognition signal processing unit 401. For example, the HDR-combined HDR image signal may be input from the image sensor 1 to the recognition signal processing unit 401, or the first signal S1, the second signal S2, and each pixel G before HDR synthesis may be input. Information of the local blinking object likelihood LFL2 and the local moving subject likelihood LFM2 as the recognition result of the above may be input. Of course, all of this information may be input from the image sensor 1 to the recognition signal processing unit 401.

The recognition signal processing unit 401 recognizes a preceding vehicle, a pedestrian, a traffic light, a sign, or the like based on the information of the local blinking object likelihood LFL2 and the local moving subject likelihood LFM2 output from the image sensor 1. Image recognition processing is performed.
Specifically, based on the information of the local blinking body likelihood LFL2 output from the image sensor 1, it is possible to recognize a traffic light, a tail lamp, etc. as a periodic blinking body, for example, to recognize a preceding vehicle. can. Further, based on the information of the local moving subject likelihood LFM2 output from the image sensor 1, it is possible to recognize a preceding vehicle or a pedestrian as a moving subject.
Further, these image recognition processes may be realized by cooperating with the recognition signal processing unit 401 and the recognition control unit 402.

When the own vehicle is traveling, it may not be possible to estimate the subject as a moving subject based on the local moving subject likelihood LFM2. For example, when the own vehicle is running, a feature such as a moving subject is detected even from an obstacle that is not moving. Therefore, the recognition signal processing unit 401 may perform a process of recognizing a moving subject as an image in consideration of the traveling speed of the own vehicle and the like.
As a result, the recognition signal processing unit 401 can appropriately perform image recognition processing on the subject.

The recognition control unit 402 may recognize a subject such as a preceding vehicle or a pedestrian by executing an image recognition process using, for example, a DNN (Deep Neural Network).
The recognition result obtained by the recognition signal processing unit 401 and the recognition control unit 402 thus obtained is output to the ADAS via the communication unit 404.

By performing appropriate image recognition processing on a subject such as a pedestrian or a preceding vehicle in this way, it is possible to improve the running safety of the vehicle.

<5. Example of using three or more signals>
An example of performing feature detection of a subject using three or more types of pixel signals obtained by three or more readout processes having different light reception sensitivities will be described.

The image sensor 1 may include three types of pixels having different areas, or may be capable of outputting three types of pixel signals using two types of pixels having different areas as shown in FIG. ..

For example, the first pixel G1 included in the image sensor 1 outputs the first signal S1 which is the pixel signal having the highest light receiving sensitivity, and the second pixel G2 is a pixel signal whose light receiving sensitivity is lower than that of the first pixel G1. The second signal S2 is output. Further, the first pixel G1 outputs the third signal S3 having the lowest light receiving sensitivity by shortening the exposure time as compared with the case of outputting the first signal S1 and the second signal S2.

As an example, the first signal S1 is output by setting the exposure time of the first pixel G1 to 8 msec. Further, the second signal S2 is output by setting the exposure time of the second pixel G2 to 11 msec. Further, the third signal S3 is output by setting the exposure time of the first pixel G1 to 4 msec.

FIG. 15 is a configuration example of a signal processing unit 5A that detects features of a subject using three types of pixel signals output from the image sensor 1. The same components as those in FIG. 6 are designated by the same reference numerals, and the description thereof will be omitted as appropriate.

The signal processing unit 5A uses the first signal S1 and the second signal S2 to perform global feature detection and local feature detection, so that the inter-pixel difference detection unit 7, the global feature detection unit 8, and the local feature detection are performed. A unit 9 and a blend rate determining unit 10 are provided. Further, instead of the blend portion 11 shown in FIG. 6, a first blend portion 11A is provided.

Since each of these parts is a processing part that performs the same processing as each part shown in FIG. 6, description thereof will be omitted.

The blend rate determination unit 10 includes a temporary blend rate calculation unit 12 and a blend rate correction unit 13. The blend ratio determining unit 10 determines the ratio of the blend of the second signal S2 to the first signal S1 as the blend ratio BF1.

The first blending unit 11A outputs an HDR image signal by performing an alpha blending process of the first signal S1 and the second signal S2 according to the blending ratio BF1 determined by the blending ratio determining unit 10.

The signal processing unit 5A uses the second signal S2 and the third signal S3 to perform global feature detection and local feature detection, so that the pixel-to-pixel difference detection unit 7A, the global feature detection unit 8A, and the local feature detection are performed. A unit 9A and a blend rate determining unit 10A are provided.

The blend rate determination unit 10A includes a temporary blend rate calculation unit 12A and a blend rate correction unit 13A.

Each of these parts is a processing part that performs the same processing as each part shown in FIG. 6 except that the second signal S2 is used instead of the first signal S1 and the third signal S3 is used instead of the second signal S2. ..
However, the blend ratio determining unit 10A determines the blend ratio of the third signal S3 with respect to the HDR image signal output from the first blend unit 11A as the blend ratio BF2.

The second blending unit 11B performs an α-blending process of the HDR image signal output from the first blending unit 11A and the third signal S3 according to the blending ratio BF2 determined by the blending ratio determining unit 10A.

In this way, by using three or more types of pixel signals, it is possible to widen the dynamic range in HDR composition.

<6. Modification example>
Here, modifications to some of the above-described embodiments will be described.
An example has been described in which the global feature detection unit 8 shown in FIG. 7 is provided with a first memory 23A and a second memory 23B for storing the brightness average value L1A and the brightness average value L2A obtained for each pixel area. .. Further, since the brightness average values L1A and L2A one frame before are stored in each memory, the first pixel change is performed by the region-by-region brightness change first calculation unit 24A and the region-by-region brightness change second calculation unit 24B in the subsequent stage. An example of calculating the quantity G1V and the second pixel change quantity G2V has been described.

In this modification, the first memory 23A and the second memory 23B store the brightness average values L1A and L2A of two frames or more. For example, the brightness average values L1A and L2A one frame before and the brightness average values L1A and L2A two frames before are stored.

Then, the region-by-region luminance change first calculation unit 24A and the region-by-region luminance change second calculation unit 24B perform a process of observing the transition of the luminance change. As a result, the region-by-region luminance change first calculation unit 24A and the region-by-region luminance change second calculation unit 24B can estimate the blinking cycle of the periodic blinker.

If the blinking cycle of the periodic blinking object can be estimated, the exposure time at which the light emitting state of the periodic blinking object can be reliably imaged can be set for the second pixel G2. This makes it possible to generate an HDR image signal in which an LED or the like is reliably captured. In particular, when the process of recognizing the preceding vehicle or the traffic light is executed by the processing unit in the subsequent stage, it is possible to improve the running safety of the vehicle, which is preferable.

As another modification, an example in which a local feature is not detected will be described.
In the above example, the signal processing unit 5 includes both the global feature detection unit 8 and the local feature detection unit 9, and the local blinker likelihood LFL2 and local motion output from the local feature detection unit 9. The case where the blend ratio is determined using the subject likelihood LFM2 has been described.

In this modification, a case where the signal processing unit 5 includes only the global feature detection unit 8 and does not include the local feature detection unit 9 will be described.

The global feature detection unit 8 detects the global feature of the subject for each pixel region GA virtually set in the pixel array unit 2. Here, if the pixel region GA is set as a sufficiently small region, a desired HDR image signal can be obtained by using the characteristics of the subject for each pixel region GA detected by the global feature detection unit 8. In some cases, the blend ratio BF can be determined appropriately.

In this case, the signal processing unit 5 may be configured without the inter-pixel difference detection unit 7 and the local feature detection unit 9 shown in FIG. That is, the blend rate correction unit 13 uses the global blinker likelihood LFL1 and the global moving subject likelihood LFM1 for each pixel region GA output from the global feature detection unit 8, and the blend ratio for each pixel region GA. Calculate BF.

The blending unit 11 outputs an HDR image signal by performing HDR composition using the blend ratio BF for each pixel region GA.
As a result, the configuration of the signal processing unit 5 can be simplified, and the processing load can be reduced.

As yet another modification, it is conceivable to make the thresholds Th1 and Th2 shown in FIG. 10 and the thresholds Th3 and Th4 shown in FIG. 11 variable.
These threshold values are set to suppress erroneous determination that the subject is erroneously determined to be a periodic blinking body or is determined to be a moving subject.

However, depending on the imaging environment, erroneous determinations may still occur frequently. Specifically, it is a dark imaging environment or the like.
When it is estimated that the brightness of the surrounding environment is lower than a certain value based on the output of the brightness sensor, it is conceivable to raise any or all of the threshold values Th1, Th2, Th3, and Th4. By raising each threshold value, the region Ar3 shown in FIG. 10 and the region Ar4 shown in FIG. 11 become narrower, and the local blinking body likelihood LFL2 and the local moving subject likelihood LFM2 for the subject are calculated to be lower.

As a result, the above-mentioned erroneous determination can be suppressed, and an appropriate HDR image signal can be generated.

In addition, another modification will be described.
In the above-mentioned example, the process of determining the blend ratio so that the ratio of the second signal S2 becomes high when the global blinking body likelihood LFL1 or the local blinking body likelihood LFL2 for the subject is high has been described.
Here, when the global blinking body likelihood LFL1 and the local blinking body likelihood LFL2 are low, the blending ratio may be basically determined so that the ratio of the first signal S1 is high. Further, as long as the global blinking body likelihood LFL1 and the local blinking body likelihood LFL2 are not high, the blending ratio may be determined so that the ratio of the second signal S2 is 0%.

As a result, even if the global moving subject likelihood LFM1 and the local moving subject likelihood LFM2 are calculated to be low and it is determined that the moving subject is unlikely to be a moving subject by mistake, the subject is concerned. It is possible to suppress the occurrence of blurring.

<7. Summary>
As described above, the image sensor 1 according to the embodiment has the first pixel G1 in which the reading is performed by the first light receiving sensitivity and the reading by the second light receiving sensitivity which is lower than the first light receiving sensitivity. The first signal S1 and the first exposure time output from the first pixel G1 which is the first exposure time are different from the pixel array unit 2 in which the second pixel G2 is arranged in two dimensions. Based on the second signal S2 output from the second pixel G2, which is the second exposure time, the characteristics of the pixel region GA including a plurality of the first pixel G1 and the second pixel G2 are the overall characteristics. It is provided with a global feature detection unit 8 (8A) for detecting as. Further, the image sensor 1 determines the blend rate BF (BF1, BF2) of the first signal S1 and the second signal S2 based on the detection result of the global feature detection unit 8 (8A). ) And.
The difference in the light receiving sensitivity between the first pixel G1 and the second pixel G2 may be due to the difference in the light receiving area in the pixel, the difference in the length of the exposure time, or the difference in the light receiving element. It may be due to the difference in characteristics. Further, the global feature is a feature of a subject imaged in a set of pixel region GAs.
By detecting the global features for each group of pixel region GAs, the blend ratio BE of the first signal S1 and the second signal S2 can be changed for each pixel region GA, and appropriate HDR synthesis can be performed with a small processing load. Can be done.
Further, by using the signals of the first pixel G1 and the second pixel G2 (first signal S1, second signal S2) having different exposure times for each pixel region GA, a global feature when the subject is a periodic blinker. It is possible to detect global features when the subject is a moving subject.
Further, many subjects are imaged in the pixel array unit 2 with a certain range. That is, few subjects are imaged with only one pixel G. Therefore, by detecting the global feature for each pixel region GA, it is possible to efficiently capture the feature of the subject. Further, when one pixel G fails and a normal pixel signal cannot be output, the pixel signal output from the pixel G may have, for example, a characteristic as a periodic blinker. be. In such a case, it is conceivable that the characteristics of the subject with respect to the pixel G may be erroneously detected. However, as in this configuration, when a global feature is detected together with a plurality of surrounding pixels G, the feature of the subject detected for the failed pixel G is almost ignored, and the feature is erroneously detected. Inappropriate blending based on this is not performed. As a result, it is possible to execute an appropriate blending process even when a failure occurs in the pixel G.

As shown in the description of the signal processing unit 5, the global feature may be a feature regarding the change in luminance with the passage of time.
As a subject having a characteristic of a change in brightness with the passage of time, for example, a light emitting body such as an LED (Light Emitting Diode) that blinks in a cycle of several msec to several tens of msec, or a light emitting body that reflects light emitted from the light emitting body is reflected. It can be mentioned as a subject whose brightness is changing, such as an object. A subject with a particularly large change in brightness is more likely to be detected. Further, even when the subject is a moving subject with a large movement, a large change in brightness is likely to be detected.
By detecting such a subject, it is possible to appropriately blend the first signal S1 and the second signal S2, and it is possible to suppress artifacts generated due to a periodic blinking object or a moving subject. ..

The global feature detection unit 8 (8A) in the image sensor 1 may perform detection by calculating the likelihood representing the degree to which the imaged subject is a periodic blinker as the global blinker likelihood LFL1.
As a result, when the subject is a periodic blinking object such as an LED, the global blinking object likelihood LFL1 is calculated to be high. In particular, the global blinker likelihood LFL1 is increased when a periodic blinker is imaged in most of the target pixel region GA.
In addition to the periodic blinking body, moving subjects can be considered as the subject whose brightness changes, but by calculating the global blinking body likelihood LFL1 as in this configuration, the subject moves with the periodic blinking body. It is possible to distinguish which of the subjects it is.
As a result, the blend ratio BF (BF1, BF2) can be changed according to whether the subject is a periodic blinking body or a moving subject, and appropriate HDR composition according to the subject becomes possible.

As described with reference to FIG. 7 and the like, the global feature detection unit 8 (8A) of the image sensor 1 changes the amount of change over time in the brightness average value of the first pixel G1 for each pixel region GA by the first pixel. The process of calculating as the amount G1V, the process of calculating the temporal change amount of the brightness average value of the second pixel G2 for each pixel area GA as the second pixel change amount G2V, and the first pixel change amount G1V and the second pixel. The detection may be performed by executing a process of calculating the difference of the change amount G2V by VDing the change amount difference value.
The amount of data for the information of the first pixel change amount G1V and the second pixel change amount G2V, which are the temporal changes in the brightness average values L1A and L2A for each pixel region GA, is smaller than that calculated for each pixel G. To. Similarly, the amount of data is also reduced for the change amount difference value VD calculated from the first pixel change amount G1V and the second pixel change amount G2V.
By using such information, it is possible to detect global features with a small processing load.

As described with reference to FIG. 9 and the like, in the image sensor 1, the global blinking likelihood LFL1 is calculated so as to be larger as the change amount difference value VD is larger and larger as the second pixel change amount G2V is smaller. You may.
For example, when the exposure time of the first pixel G1 having a high light receiving sensitivity is shorter than the exposure time of the second pixel G2 having a low light receiving sensitivity, all the exposure times of the first pixel G1 are in a non-light emitting state in the periodic blinker. May be included. Even in such a case, a part of the exposure time of the second pixel G2 may be included in the light emitting state in the periodic blinker. In such a case, the change amount difference value VD becomes large.
Further, with respect to the second pixel change amount G2V, there is a high possibility that the second pixel G2 having a long exposure time shows a periodic blinking object in a light emitting state. In particular, when the exposure time is longer than the non-emission time of the periodic blinking object, the light emitting state of the periodic blinking object is surely imaged, and the second pixel change amount G2V becomes small.
Therefore, when the change amount difference value VD is large and the second pixel change amount G2V is small, the global blinking body likelihood LFL1 appropriately determines that the subject is a periodic blinking body, so that an appropriate blend ratio BF is obtained. (BF1, BF2) can be set.

As described with reference to FIG. 6 and the like, in the image sensor 1, the local feature detection unit 9 that detects the feature for each pixel G using the global feature detected by the global feature detection unit 8 (8A). (9A) may be provided.
As a result, the characteristics of the subject are detected for each pixel G.
Therefore, an appropriate blend ratio BF (BF1, BF2) is determined for each pixel G, and optimum HDR synthesis can be performed.

As described with reference to FIGS. 6 and 10, the local feature detection unit 9 (9A) of the image sensor 1 has a pixel-to-pixel difference calculated as a difference between the brightness values of the first pixel G1 and the second pixel G2. The blinking body likelihood for each pixel G may be calculated as the local blinking body likelihood LFL2 based on the value GD and the global blinking body likelihood LFL1.
The inter-pixel difference value GD and the global blinking object likelihood LFL1 can be used to estimate the degree to which the subject is a periodic blinking object.
As a result, it is possible to detect that the subject is a periodic blinking object, so that the blend ratio BF (BF1, BF2) can be appropriately set for each pixel G.

As described with reference to FIG. 10 and the like, in the image sensor 1, the local blinking body likelihood LFL2 is larger for the pixel G having a larger inter-pixel difference value GD, and is larger for the pixel G having a larger global blinking body likelihood LFL1. It may be calculated so as to be.
For example, as a subject having a large inter-pixel difference value GD, a moving subject, a periodic blinker, or the like can be mentioned. Among these subjects, a subject having a large inter-pixel difference value GD and having a high global blinking object likelihood LFL1 is likely to be a periodic blinking object rather than a moving subject.
As described above, since the local blinking body likelihood LFL2 is an index indicating the certainty that the subject imaged in the pixel G is a periodic blinking body, the blend ratio BF ( BF1, BF2) can be appropriately set for each pixel G.

As described in the modified example, the blend rate determination unit 10 (10A) of the image sensor 1 has a blend rate BF (BF1, BF2) so that the higher the global blinking likelihood LFL1, the higher the ratio of the second signal S2. ) May be determined.
Depending on the relationship between the exposure times of the first pixel G1 and the second pixel G2, the second pixel G2 has a longer exposure time than the first pixel G1. In this case, it is highly possible that the second pixel G2 is imaged with a periodic blinker in a light emitting state.
Therefore, by increasing the ratio of the second signal S2 output from the second pixel G2, it is possible to increase the possibility of obtaining image data in which the periodic blinker is emitting light.

As described above, the blend rate determination unit 10 (10A) of the image sensor 1 has a blend rate BF (BF1, BF1) for each pixel G so that the higher the local blinker likelihood LFL2, the higher the ratio of the second signal S2. BF2) may be determined.
In the above example, the second pixel G2 has a longer exposure time than the first pixel G1. In this case, the second pixel G2 is likely to be generated in a state of capturing the periodic blinking body in the light emitting state.
By increasing the ratio of the second signal S2 based on the height of the local blinker likelihood LFL2 for each pixel G, the possibility of obtaining image data in which the periodic blinker is emitting light is increased. be able to. In particular, by using such a method for determining the blend ratio for each pixel G, the blend ratio is determined as if the periodic blinker is imaged for the pixel G in which the periodic blinker is not imaged. Therefore, it is possible to perform appropriate HDR composition according to the characteristics of the subject for each pixel G.

As described with reference to FIG. 10 and the like, the local feature detection unit 9 (9A) of the image sensor 1 can execute the process of calculating the local moving subject likelihood LFM2, and the local moving subject likelihood LFM2 can be executed. May be calculated so that the larger the pixel-to-pixel difference value GD is, the larger the pixel G is, and the smaller the global blinking likelihood LFL1 is, the larger the pixel G is.
For example, as a subject having a large inter-pixel difference value GD, a moving subject, a periodic blinker, or the like can be mentioned. Among these subjects, a subject having a large inter-pixel difference value GD and a low global blinking likelihood LFL1 is highly likely to be a moving subject.
By appropriately detecting that the subject is a moving subject in this way, the blend ratio BF (BF1, BF2) can be appropriately set for each pixel G. Further, since it is not necessary to calculate an index value or the like indicating the degree to which the subject is a moving subject, it is possible to determine an appropriate blend ratio BF (BF1, BF2) while suppressing an increase in the processing load.

As described with reference to FIGS. 7 and 9, the global feature detection unit 8 (8A) of the image sensor 1 determines the amount of time change in the brightness of the first pixel G1 for each pixel region GA as the first pixel. The process of calculating the change amount G1V, the process of calculating the temporal change amount of the brightness of the second pixel G2 for each pixel area GA as the second pixel change amount G2V, and the first pixel change amount G1V and the second pixel change. The process of calculating the difference of the amount G2V as the change amount difference value VD and the process of calculating the change amount difference value VD are executed, and the global moving subject likelihood LFM1 indicating the degree to which the imaged subject is a moving subject is larger and the first as the change amount difference value VD is smaller. It may be calculated so that the larger the two-pixel change amount G2V is, the larger the amount is.
For example, even if the light receiving sensitivities are different, the amount of change in the first signal S1 and the second signal S2 output from the first pixel G1 and the second pixel G2 is large when the subject is a moving subject. It is possible that the difference will be small.
Therefore, when the change amount difference value VD is small and the second pixel change amount G2V is large, the global moving subject likelihood LFM1 appropriately determines that the subject is a moving subject, so that an appropriate blend ratio BF ( BF1, BF2) can be set.

As described with reference to FIGS. 6 and 11, in the image sensor 1, the local feature detection unit 9 that executes a process of calculating the local moving subject likelihood LFM2, which is the moving subject likelihood for each pixel G. (9A), the local moving subject likelihood LFM2 is larger as the pixel-to-pixel difference value GD calculated as the difference in brightness between the first pixel G1 and the second pixel G2 is larger, and the global moving subject likelihood LFM1 It may be calculated so that the larger the pixel, the larger the pixel.
For example, as a subject having a large inter-pixel difference value GD, a moving subject, a periodic blinker, or the like can be mentioned. Among these subjects, a subject having a large inter-pixel difference value GD and a high local moving subject likelihood LFM2 is likely to be a moving subject rather than a periodic blinking object.
In this way, since it is possible to appropriately detect that the subject is a moving subject, it is possible to appropriately set the blend ratio BF (BF1, BF2) for each pixel G.

As described in the modified example, the blend rate determination unit 10 (10A) of the image sensor 1 has a blend rate BF (BF1, BF2) so that the higher the global moving subject likelihood LFM1, the higher the ratio of the first signal S1. ) May be determined.
The first signal S1 is a signal obtained after an exposure time shorter than that of the second signal S2. Therefore, the image data generated based on the first signal S1 has less blurring of the moving subject than the image data generated based on the second signal S2.
Therefore, by increasing the ratio of the first signal S1 and performing blending, it is possible to obtain image data in which blurring of a moving subject is suppressed.

As described in each of the above examples, the blend rate determining unit 10 (10A) of the image sensor 1 blends each pixel G so that the higher the local moving subject likelihood LFM2, the higher the ratio of the first signal S1. The rate BF (BF1, BF2) may be determined.
The first signal S1 is a signal obtained after an exposure time shorter than that of the second signal S2. Therefore, the image data generated based on the first signal S1 has less blurring of the moving subject than the image data generated based on the second signal S2.
By increasing the ratio of the first signal S1 based on the height of the local moving subject likelihood LFM2 for each pixel G, it is possible to obtain image data in which blurring of the moving subject is suppressed. In particular, by using such a method for determining the blend ratio BF (BF1, BF2) for each pixel G, the second signal is obtained for the other pixel G in which the periodic blinker is imaged while suppressing the blurring of the moving subject. By increasing the ratio of S2, it is possible to obtain image data in which the light emitting state of the periodic blinking object is captured.

As described with reference to FIG. 2 and the like, the first pixel G1 and the second pixel G2 of the image sensor 1 may be configured to have different light receiving sensitivities due to different light receiving areas.
Thereby, the global feature of each pixel region GA can be detected by using the first pixel G1 and the second pixel G2 having different light receiving sensitivities.
Therefore, an appropriate blend ratio BF (BF1, BF2) based on the global characteristics can be determined for each pixel region GA.

As described with reference to FIG. 2 and the like, the exposure time of the first pixel G1 of the image sensor 1 may be shorter than the exposure time of the second pixel G2. For example, the exposure time of the first pixel G1 is 8 msec, and the exposure time of the second pixel G2 is 11 msec.
This makes it possible to detect, for example, the characteristic of a periodic blinking body that blinks periodically as a global characteristic.
Therefore, it is possible to appropriately determine the blend ratio BF (BF1, BF2) based on the global characteristics and perform HDR synthesis.
It is desirable that the exposure time of the second pixel G2 is set based on the blinking cycle of the periodic blinking object. In particular, when it is used as an in-vehicle image sensor 1, the target of the subject as a periodic blinker such as a brake lamp or a traffic light of a preceding vehicle is clarified, so an appropriate exposure time is set. can do.

As described with reference to FIG. 15 in an example using three or more signals, the pixel array unit 2 of the image sensor 1 has a third light receiving sensitivity that is different from both the first light receiving sensitivity and the second light receiving sensitivity. The third signal S3 read by the above may be output. Further, the global feature detection unit 8 (8A) includes a plurality of pixels G to which the third signal S3 is output based on one of the first signal S1 and the second signal S2 and the third signal S3. Global features may be detected for the pixel region GA.
As a result, the blend rate determination unit 10 (10A) determines the blend rate for blending the three pixel signals.
Therefore, the dynamic range in HDR synthesis can be increased.

It should be noted that the effects described in the present specification are merely examples and are not limited, and other effects may be obtained.

<8. This technology>
(1)
A pixel array unit in which a first pixel that is read by a first light receiving sensitivity and a second pixel that is read by a second light receiving sensitivity, which is lower than the first light receiving sensitivity, are arranged two-dimensionally. When,
The first signal output from the first pixel, which is the first exposure time, and the second signal, which is output from the second pixel, which is the second exposure time different from the first exposure time. Based on this, a global feature detection unit that detects features of a pixel region including a plurality of the first pixel and the second pixel as global features, and a global feature detection unit.
An image sensor including a blend rate determining unit that determines the blending rate of the first signal and the second signal based on the detection result of the global feature detecting unit.
(2)
The image sensor according to (1) above, wherein the global feature is a feature regarding a change in brightness with the passage of time.
(3)
The image sensor according to (2) above, wherein the global feature detection unit performs the detection by calculating the likelihood representing the degree to which the imaged subject is a periodic blinking object as the global blinking object likelihood.
(4)
The global feature detection unit calculates a temporal change amount of the brightness average value of the first pixel for each pixel area as a first pixel change amount, and a brightness of the second pixel for each pixel area. The process of calculating the temporal change amount of the average value as the second pixel change amount and the process of calculating the difference between the first pixel change amount and the second pixel change amount as the change amount difference value are executed. The image sensor according to (3) above, wherein the detection is performed by the above.
(5)
The image sensor according to (4) above, wherein the global blinking body likelihood is calculated so that the larger the change amount difference value is, the larger the change amount is, and the smaller the second pixel change amount is, the larger is is calculated.
(6)
The image sensor according to any one of (3) to (5) above, further comprising a local feature detection unit that detects a feature for each pixel using the global feature detected by the global feature detection unit.
(7)
The local feature detection unit locally determines the blinking object likelihood for each pixel based on the pixel-to-pixel difference value calculated as the difference between the brightness values of the first pixel and the second pixel and the global blinking object likelihood. The image sensor according to (6) above, which is calculated as the target blinking body likelihood.
(8)
The image sensor according to (7) above, wherein the local blinking body likelihood is calculated so that a pixel having a larger difference value between pixels has a larger pixel and a pixel having a larger global blinking body likelihood has a larger value.
(9)
The image according to any one of (3) to (8) above, wherein the blend ratio determining unit determines the blend ratio so that the higher the global blinking likelihood is, the higher the ratio of the second signal is. Sensor.
(10)
The image sensor according to (8) above, wherein the blend rate determining unit determines the blend rate for each pixel so that the higher the local blinking likelihood is, the higher the ratio of the second signal is.
(11)
The local feature detection unit
It is possible to execute the process of calculating the local moving subject likelihood.
The local moving subject likelihood is calculated so that the larger the pixel-to-pixel difference value is, the larger the pixel is, and the smaller the global blinking object likelihood is, the larger the pixel is. Either (8) or (10) above. Image sensor described in.
(12)
The global feature detector
The process of calculating the amount of time change in the brightness of the first pixel as the amount of change in the first pixel for each pixel area, and the amount of time change in the brightness of the second pixel for each pixel area are calculated as the amount of change in the second pixel. A process of calculating as a change amount and a process of calculating the difference between the first pixel change amount and the second pixel change amount as a change amount difference value are executed.
The global moving subject likelihood, which indicates the degree to which the imaged subject is a moving subject, is calculated so as to be larger as the change amount difference value is smaller and larger as the second pixel change amount is larger. The image sensor according to any one of (11).
(13)
It is equipped with a local feature detection unit that executes a process to calculate the local moving subject likelihood, which is the moving subject likelihood for each pixel.
The local moving subject likelihood is
It is calculated so that the larger the pixel-to-pixel difference value calculated as the difference in brightness between the first pixel and the second pixel is, and the larger the pixel is, the larger the global moving subject likelihood is. Described image sensor.
(14)
The image according to any one of (12) to (13) above, wherein the blend ratio determining unit determines the blend ratio so that the higher the global moving subject likelihood is, the higher the ratio of the first signal is. Sensor.
(15)
The image sensor according to (13) above, wherein the blend rate determining unit determines the blend rate for each pixel so that the higher the local moving subject likelihood is, the higher the ratio of the first signal is.
(16)
The image sensor according to any one of (1) to (15) above, wherein the first pixel and the second pixel have different light receiving sensitivities due to different light receiving areas.
(17)
The image sensor according to any one of (1) to (16) above, wherein the exposure time of the first pixel is shorter than the exposure time of the second pixel.
(18)
The pixel array unit is capable of outputting a third signal read by a third light receiving sensitivity that is different from both the first light receiving sensitivity and the second light receiving sensitivity.
The global feature detection unit provides a global view for a pixel region including a plurality of pixels to which the third signal is output based on the signal of either one of the first signal and the second signal and the third signal. The image sensor according to any one of the above (1) to (17).
(19)
An image sensor that performs photoelectric conversion and
An optical member that collects the reflected light from the subject on the image sensor is provided.
The image sensor is
A pixel array unit in which a first pixel that is read by a first light receiving sensitivity and a second pixel that is read by a second light receiving sensitivity, which is lower than the first light receiving sensitivity, are arranged two-dimensionally. When,
The first signal output from the first pixel, which is the first exposure time, and the second signal, which is output from the second pixel, which is the second exposure time different from the first exposure time. Based on this, a global feature detection unit that detects features of a pixel region including a plurality of the first pixel and the second pixel as global features, and a global feature detection unit.
An imaging device including a blend rate determining unit that determines the blending rate of the first signal and the second signal based on the detection result of the global feature detecting unit.
(20)
The first signal output from the first pixel, which is read out by the first light receiving sensitivity according to the first exposure time, and the second exposure time, which is different from the first exposure time, are higher than the first light receiving sensitivity. Features of a pixel region containing a plurality of the first pixel and the second pixel based on a second signal output from the second pixel on which reading is performed by a second light receiving sensitivity having a low sensitivity. And the process of detecting as a global feature,
A signal processing method comprising a process of determining a blend ratio of the first signal and the second signal based on a detection result of the global feature.

1 Image sensor 2 Pixel array unit 5, 5A Signal processing unit 8, 8A Global feature detection unit 9, 9A Local feature detection unit 10, 10A Blend rate determination unit G pixel G1 1st pixel G2 2nd pixel GA pixel area S1 1st signal S2 2nd signal S3 3rd signal G1V 1st pixel change amount G2V 2nd pixel change amount VD change amount difference value LFL1 Global blinker likelihood LFL2 Local blinker likelihood LFM1 Global moving subject likelihood LFM2 Local moving subject likelihood GD Pixel-to-pixel difference value BF, BF1, BF2 Blend rate

Claims (20)

A pixel array unit in which a first pixel that is read by a first light receiving sensitivity and a second pixel that is read by a second light receiving sensitivity, which is lower than the first light receiving sensitivity, are arranged two-dimensionally. When,
The first signal output from the first pixel, which is the first exposure time, and the second signal, which is output from the second pixel, which is the second exposure time different from the first exposure time. Based on this, a global feature detection unit that detects features of a pixel region including a plurality of the first pixel and the second pixel as global features, and a global feature detection unit.
An image sensor including a blend rate determining unit that determines the blending rate of the first signal and the second signal based on the detection result of the global feature detecting unit. The image sensor according to claim 1, wherein the global feature is a feature regarding a change in brightness with the passage of time. The image sensor according to claim 2, wherein the global feature detection unit performs the detection by calculating the likelihood representing the degree to which the imaged subject is a periodic blinking object as the global blinking object likelihood. The global feature detection unit calculates a temporal change amount of the brightness average value of the first pixel for each pixel region as a first pixel change amount, and a brightness of the second pixel for each pixel region. The process of calculating the temporal change amount of the average value as the second pixel change amount and the process of calculating the difference between the first pixel change amount and the second pixel change amount as the change amount difference value are executed. The image sensor according to claim 3, wherein the detection is performed by the above. The image sensor according to claim 4, wherein the global blinking body likelihood is calculated so that the larger the change amount difference value is, the larger the change amount is, and the smaller the second pixel change amount is, the larger is is calculated. The image sensor according to claim 3, further comprising a local feature detection unit that detects a feature for each pixel using the global feature detected by the global feature detection unit. The local feature detection unit locally determines the blinking object likelihood for each pixel based on the pixel-to-pixel difference value calculated as the difference between the brightness values of the first pixel and the second pixel and the global blinking object likelihood. The image sensor according to claim 6, which is calculated as the target blinking body likelihood. The image sensor according to claim 7, wherein the local blinking body likelihood is calculated so that the pixel having a larger difference value between pixels has a larger pixel, and the pixel having a larger global blinking body likelihood has a larger value. The image sensor according to claim 3, wherein the blend rate determining unit determines the blend rate so that the higher the global blinking likelihood is, the higher the ratio of the second signal is. The image sensor according to claim 8, wherein the blend rate determining unit determines the blend rate for each pixel so that the higher the local blinking likelihood is, the higher the ratio of the second signal is. The local feature detection unit
It is possible to execute the process of calculating the local moving subject likelihood.
The image sensor according to claim 8, wherein the local moving subject likelihood is calculated so that the pixel having a larger difference value between pixels has a larger likelihood, and the pixel having a smaller global blinking object likelihood has a higher likelihood. The global feature detector
The process of calculating the amount of time change in the brightness of the first pixel as the amount of change in the first pixel for each pixel area, and the amount of time change in the brightness of the second pixel for each pixel area are calculated as the amount of change in the second pixel. A process of calculating as a change amount and a process of calculating the difference between the first pixel change amount and the second pixel change amount as a change amount difference value are executed.
The third aspect of claim 3, wherein the global moving subject likelihood, which indicates the degree to which the imaged subject is a moving subject, is calculated so as to be larger as the change amount difference value is smaller and larger as the second pixel change amount is larger. Image sensor. It is equipped with a local feature detection unit that executes a process to calculate the local moving subject likelihood, which is the moving subject likelihood for each pixel.
The local moving subject likelihood is
The twelfth claim, which is calculated so that a pixel having a larger pixel-to-pixel difference value calculated as a difference in brightness between the first pixel and the second pixel has a larger pixel-to-pixel difference value and a pixel having a larger global moving subject likelihood becomes larger. Image sensor. The image sensor according to claim 12, wherein the blend rate determining unit determines the blend rate so that the higher the global moving subject likelihood is, the higher the ratio of the first signal is. The image sensor according to claim 13, wherein the blend rate determining unit determines the blend rate for each pixel so that the higher the local moving subject likelihood is, the higher the ratio of the first signal is. The image sensor according to claim 1, wherein the first pixel and the second pixel have different light receiving sensitivities due to different light receiving areas. The image sensor according to claim 1, wherein the exposure time of the first pixel is shorter than the exposure time of the second pixel. The pixel array unit is capable of outputting a third signal read by a third light receiving sensitivity that is different from both the first light receiving sensitivity and the second light receiving sensitivity.
The global feature detection unit provides a global view for a pixel region including a plurality of pixels to which the third signal is output based on the signal of either one of the first signal and the second signal and the third signal. The image sensor according to claim 1, which detects a target feature. An image sensor that performs photoelectric conversion and
An optical member that collects the reflected light from the subject on the image sensor is provided.
The image sensor is
A pixel array unit in which a first pixel that is read by a first light receiving sensitivity and a second pixel that is read by a second light receiving sensitivity, which is lower than the first light receiving sensitivity, are arranged two-dimensionally. When,
The first signal output from the first pixel, which is the first exposure time, and the second signal, which is output from the second pixel, which is the second exposure time different from the first exposure time. Based on this, a global feature detection unit that detects features of a pixel region including a plurality of the first pixel and the second pixel as global features, and a global feature detection unit.
An imaging device including a blend rate determining unit that determines the blending rate of the first signal and the second signal based on the detection result of the global feature detecting unit. The first signal output from the first pixel, which is read out by the first light receiving sensitivity according to the first exposure time, and the second exposure time, which is different from the first exposure time, are higher than the first light receiving sensitivity. Based on the second signal output from the second pixel, which is read out by the second light receiving sensitivity, which is a low sensitivity, the characteristics of the pixel region including a plurality of the first pixel and the second pixel are described. Processing to detect as a global feature and
A signal processing method comprising a process of determining a blend ratio of the first signal and the second signal based on a detection result of the global feature.

Images