WO2022149431A1

WO2022149431A1 - Image sensor, imaging device, and signal processing method

Info

Publication number: WO2022149431A1
Application number: PCT/JP2021/046511
Authority: WO
Inventors: 大輝山崎
Original assignee: ソニーセミコンダクタソリューションズ株式会社
Priority date: 2021-01-08
Filing date: 2021-12-16
Publication date: 2022-07-14
Also published as: JP2022107233A

Abstract

An image sensor comprising: a pixel array unit in which a first pixel for performing readout with a first light-reception sensitivity and a second pixel for performing readout with a second light-reception sensitivity lower than the first light-reception sensitivity are two-dimensionally arrayed; a comprehensive feature detecting unit which, on the basis of a first signal output from the first pixel set for a first exposure time and a second signal output from the second pixel set for a second exposure time different from the first exposure time, detects, as a comprehensive feature, a feature for a pixel region including a plurality of the first pixels and a plurality of the second pixels; and a blend ratio determining unit for determining a blend ratio of the first signal and the second signal on the basis of a detection result from the comprehensive feature detecting unit.

Description

Image sensor, image pickup device, signal processing method

This technology relates to an image sensor, an image pickup device, and a signal processing method for performing HDR (High Dynamic Range) synthesis according to the characteristics of the subject.

A technique for obtaining an HDR image signal by performing HDR composition using a plurality of images having different exposure times is known. In such a technique, the blending ratio of a plurality of image signals having different exposure times is determined according to the characteristics of the subject.
For example, in Patent Document 1 below, the blend ratio is determined so as to use a pixel signal having a short exposure time, which is a signal that is strong against movement, for pixels in which a moving subject is imaged.

Japanese Unexamined Patent Publication No. 2012-257193

However, the subject detected as a moving subject includes a luminous body that repeatedly blinks periodically. If the blending ratio is determined so as to use a pixel signal having a short exposure time for such a subject, there is a problem that the light emitter cannot normally image.

This technique was made in view of the above circumstances, and the purpose is to determine an appropriate blending ratio according to the characteristics of the subject.

The image sensor according to the present technology has two pixels, a first pixel in which reading is performed by the first light receiving sensitivity and a second pixel in which reading is performed by a second light receiving sensitivity which is lower than the first light receiving sensitivity. The pixel array unit arranged in a dimension, the first signal output from the first pixel set as the first exposure time, and the second exposure time set to a second exposure time different from the first exposure time. A global feature detection unit that detects features of a pixel region containing a plurality of the first pixel and a plurality of the second pixels as global features based on a second signal output from the pixels, and the global feature detection unit. It is provided with a blend rate determining unit for determining the blend rate of the first signal and the second signal based on the detection result of the feature detection unit.
The difference in the light receiving sensitivity between the first pixel and the second pixel may be due to the difference in the light receiving area in the pixel, the difference in the length of the exposure time, or the characteristics of the light receiving element. It may be due to the difference between. Further, the global feature is a feature of a subject imaged in a set of pixel regions.

In the above-mentioned image sensor, the global feature may be a feature regarding the change in luminance with the passage of time.
As a subject having a characteristic of a change in brightness with the passage of time, for example, a light emitting body such as an LED (Light Emitting Diode) that blinks in a cycle of several msec to several tens of msec, or a light emitting body that reflects light emitted from the light emitting body is reflected. It can be mentioned as a subject whose brightness is changing, such as an object. In particular, a subject with a large change in brightness is more likely to be detected. Further, even when the subject is a moving subject with a large movement, a large change in luminance is likely to be detected.

The global feature detection unit in the image sensor described above may perform the detection by calculating the likelihood representing the degree to which the image pickup subject is a periodic blinker as the global blinker likelihood.
As a result, when the subject is a periodic blinking object such as an LED, the global blinking object likelihood is calculated to be high. In particular, the global blinking object likelihood is increased when a periodic blinking object is imaged in most of the target pixel region.

The global feature detection unit in the image sensor described above performs a process of calculating a temporal change amount of the brightness average value of the first pixel as a first pixel change amount for each pixel area, and the pixel area for each. A process of calculating the temporal change amount of the brightness average value of the second pixel as a second pixel change amount, and a process of calculating the difference between the first pixel change amount and the second pixel change amount as a change amount difference value. , May be performed by executing.
The amount of data for the information on the amount of change in the first pixel and the amount of change in the second pixel, which is the amount of change in the brightness average value for each pixel region over time, is smaller than that calculated for each pixel. Similarly, the amount of data is reduced for the change amount difference value calculated from the change amount of the first pixel and the change amount of the second pixel.

In the above-mentioned image sensor, the global blinking body likelihood may be calculated so as to be larger as the change amount difference value is larger and larger as the second pixel change amount is smaller.
For example, when the exposure time of the first pixel having high light receiving sensitivity is shorter than the exposure time of the second pixel having low light receiving sensitivity, the entire exposure time of the first pixel is included in the non-light emitting state in the periodic blinker. There is. Even in such a case, a part of the exposure time of the second pixel may be included in the light emitting state in the periodic blinker. In such a case, the change amount difference value becomes large.
Further, regarding the amount of change in the second pixel, there is a high possibility that the second pixel having a long exposure time shows a periodic blinker in a light emitting state. In particular, when the exposure time is longer than the non-light emission time of the periodic blinker, the light emission state of the periodic blinker is surely captured, and the amount of change in the second pixel becomes small.

The image sensor described above may include a local feature detection unit that detects features for each pixel using the global features detected by the global feature detection unit.
As a result, the characteristics of the subject are detected for each pixel.

The local feature detection unit in the image sensor described above blinks for each pixel based on the pixel-to-pixel difference value calculated as the difference between the luminance values of the first pixel and the second pixel and the global blinker likelihood. The body likelihood may be calculated as a local blinking body likelihood.
The pixel-to-pixel difference value and the global blinking object likelihood can be used to estimate the degree to which the subject is a periodic blinking object.

The local blinking body likelihood in the above-mentioned image sensor may be calculated so as to be larger for pixels having a larger difference value between pixels and larger for pixels having a larger global blinking body likelihood.
For example, as a subject having a large difference value between pixels, a moving subject, a periodic blinker, or the like can be mentioned. Among these subjects, a subject having a large difference value between pixels and having a high global blinking object likelihood is likely to be a periodic blinking object rather than a moving subject.

The blend ratio determination unit in the image sensor may determine the blend ratio so that the higher the global blinking likelihood is, the higher the ratio of the second signal is.
Depending on the relationship between the exposure time of the first pixel and the exposure time of the second pixel, the second pixel has a longer exposure time than the first pixel. In this case, it is highly possible that the second pixel is an image of a periodic blinker in a light emitting state.

The blend rate determination unit in the image sensor may determine the blend rate for each pixel so that the higher the local blinking likelihood is, the higher the ratio of the second signal is.
Depending on the relationship between the exposure time of the first pixel and the exposure time of the second pixel, the second pixel has a longer exposure time than the first pixel. In this case, the second pixel is likely to be generated in a state of capturing a periodic blinker in a light emitting state.

The local feature detection unit in the image sensor described above is capable of executing a process of calculating the local moving subject likelihood, and the local moving subject likelihood is larger as the pixel-to-pixel difference value is larger. It may be calculated so that the pixel with the smaller global blinker likelihood becomes larger.
For example, as a subject having a large difference value between pixels, a moving subject, a periodic blinker, or the like can be mentioned. Among these subjects, a subject having a large difference value between pixels and having a low global blinking likelihood is highly likely to be a moving subject.

The global feature detection unit in the image sensor described above includes a process of calculating a temporal change in the brightness of the first pixel for each pixel region as a first pixel change, and a second for each pixel region. A process of calculating the temporal change amount of the pixel brightness as the second pixel change amount and a process of calculating the difference between the first pixel change amount and the second pixel change amount as the change amount difference value are executed. The global moving subject likelihood, which indicates the degree to which the imaged subject is a moving subject, may be calculated so as to be larger as the change amount difference value is smaller and larger as the second pixel change amount is larger.
For example, even if the light receiving sensitivities are different, the amount of change in the first signal and the second signal output from the first pixel and the second pixel is large when the subject is a moving subject, so that the difference is small. Can be considered.

The image sensor described above includes a local feature detection unit that executes a process of calculating the local moving subject likelihood, which is the moving subject likelihood for each pixel, and the local moving subject likelihood is the first pixel. It may be calculated so that the pixel having a larger difference value between pixels calculated as the difference in the brightness of the second pixel is larger and the pixel having a larger global moving subject likelihood is larger.
For example, as a subject having a large difference value between pixels, a moving subject, a periodic blinker, or the like can be mentioned. Among these subjects, a subject having a large difference value between pixels and having a high likelihood of a local moving subject is likely to be a moving subject rather than a periodic blinker.

The blend ratio determination unit in the image sensor may determine the blend ratio so that the ratio of the first signal increases as the global moving subject likelihood increases.
The first signal is a signal obtained after an exposure time shorter than that of the second signal. Therefore, the image data generated based on the first signal has less blurring of the moving subject than the image data generated based on the second signal.

The blend rate determination unit in the image sensor may determine the blend rate for each pixel so that the higher the local moving subject likelihood is, the higher the ratio of the first signal is.
The first signal is a signal obtained after an exposure time shorter than that of the second signal. Therefore, the image data generated based on the first signal has less blurring of the moving subject than the image data generated based on the second signal.

In the image sensor described above, the first pixel and the second pixel may be configured to have different light receiving sensitivities due to different light receiving areas.
As a result, it is possible to detect the global feature of each pixel region by using the first pixel and the second pixel having different light receiving sensitivities.

In the image sensor described above, the exposure time of the first pixel may be shorter than the exposure time of the second pixel.
This makes it possible to detect, for example, the characteristic of a periodic blinking body that blinks periodically as a global characteristic.

The pixel array unit in the image sensor described above is capable of outputting a third signal read by a third light receiving sensitivity that is different from either the first light receiving sensitivity or the second light receiving sensitivity, and the global feature detection. The unit detects a global feature in a pixel region including a plurality of pixels from which the third signal is output based on either one of the first signal and the second signal and the third signal. You may.
As a result, the blend rate determination unit determines the blend rate for blending the three pixel signals.

The image pickup apparatus according to the present technology includes an image sensor that performs photoelectric conversion and an optical member that collects the reflected light from the subject on the image sensor, and the image sensor reads out by the first light receiving sensitivity. A pixel array unit in which the first pixel and the second pixel to be read by the second light receiving sensitivity, which is lower than the first light receiving sensitivity, are arranged in two dimensions, and the first exposure time. The first pixel is based on the first signal output from the first pixel and the second signal output from the second pixel having a second exposure time different from that of the first exposure time. The first signal and the second signal are based on the detection results of the global feature detection unit that detects features of a pixel region containing a plurality of the second pixels as global features, and the global feature detection unit. It is provided with a blend rate determination unit for determining the blend rate of the signal.

In the signal processing method according to the present technology, the first signal output from the first pixel, which is read out by the first light receiving sensitivity according to the first exposure time, and the second exposure time different from the first exposure time. A plurality of the first pixel and the second pixel are each based on the second signal output from the second pixel in which the reading is performed by the second light receiving sensitivity, which is lower than the first light receiving sensitivity. A process including a process of detecting a feature of a included pixel region as a global feature and a process of determining a blend ratio of the first signal and the second signal based on the detection result of the global feature. Is.
The above-mentioned various actions can also be obtained by such an image pickup device and a signal processing method.

It is a block diagram which shows the structural example of the image sensor which concerns on this technique. It is a schematic diagram which shows the structural example of a pixel unit. It is a schematic diagram which shows the structural example of a pixel array part. It is a schematic diagram which shows the example in which a plurality of pixel areas are set in a pixel array part. It is a schematic diagram which shows the example which nine pixel areas were set in the pixel array part. It is a block diagram which shows the structural example of a signal processing part. It is a block diagram which shows the structural example of the global feature detection part. It is a schematic diagram which shows the structural example of the pixel array part which has the color filter of the Bayer array. It is a figure for demonstrating an example of detecting a global feature from the change amount difference value and the 2nd pixel change amount. It is a graph for demonstrating the local blinking body likelihood calculated based on the global blinking body likelihood and the inter-pixel difference value. It is a graph for demonstrating the local moving subject likelihood calculated based on the global moving subject likelihood and the inter-pixel difference value. It is a graph which shows the relationship between the amount of light incident on a pixel, and the tentative blend ratio. It is a block diagram which shows the structural example of the image pickup apparatus in the 1st application example. It is a block diagram which shows the configuration example of the in-vehicle system in the 2nd application example. It is a block diagram which shows the structural example of the signal processing part in the example which uses three or more signals.

Hereinafter, embodiments according to the present technology will be described in the following order with reference to the accompanying drawings.
<1. Image sensor configuration>
<2. Signal processing unit configuration and processing content>
<3. First application example>
<4. Second application example>
<5. Example of using three or more signals>
<6. Modification example>
<7. Summary>
<8. This technology>

<1. Image sensor configuration>
FIG. 1 shows the configuration of the image sensor 1 according to the embodiment of the present technology.
The image sensor 1 includes a pixel array unit 2, a first read circuit 3, a second read circuit 4, and a signal processing unit 5.

The pixel array unit 2 is composed of a plurality of pixel units 6 arranged in the row direction and the column direction. The pixel unit 6 is configured to include a plurality of pixels G. The pixel G has a photoelectric conversion element, and obtains an electric signal according to the amount of received light. In the present embodiment, an example in which one pixel unit 6 includes two pixels G will be described. However, the pixel unit 6 may be configured to include three or more pixels G.

FIG. 2 shows a configuration example of the pixel unit 6.
The pixel unit 6 includes a first pixel G1 and a second pixel G2 having different light receiving sensitivities.

The light receiving area of the first pixel G1 is larger than that of the second pixel G2. Further, the exposure time of the first pixel G1 is shorter than the exposure time of the second pixel G2. As an example, the exposure time of the first pixel G1 is 8 msec, and the exposure time of the second pixel G2 is 11 msec.

The light receiving sensitivity of the first pixel G1 is higher than the light receiving sensitivity of the second pixel G2. Therefore, the first pixel G1 is more likely to be saturated than the second pixel G2.

The first pixel G1 and the second pixel G2 receive the reflected light reflected by the subject (imaging subject), perform photoelectric conversion, and output it as a pixel signal in the subsequent stage. The pixel signal output from the first pixel G1 is referred to as the first signal S1, and the pixel signal output from the second pixel G2 is referred to as the second signal S2.

In the following description, the difference in light receiving sensitivity between the first pixel G1 and the second pixel G2 will be described by giving an example based on the light receiving area and the exposure time, but other cases are also conceivable. For example, the difference in light receiving sensitivity between the first pixel G1 and the second pixel G2 may be based only on the light receiving area, may be based only on the length of the exposure time, or may be based only on the length of the exposure time, or the light receiving element (PD). : Photodiode) may be based only on the difference in characteristics. Of course, it may be based on these multiple factors.

Returning to the description of FIG.
The first read circuit 3 is a signal processing circuit for AD (Analog to Digital) conversion and noise reduction, as well as a configuration for reading the first signal S1 from the first pixel G1 provided in each pixel unit 6 of the pixel array unit 2. Etc. are provided. The first signal S1 is supplied to the signal processing unit 5 via the first read circuit 3.

The second read circuit 4 has a configuration for reading the second signal S2 from the second pixel G2 provided in each pixel unit 6, and also includes a signal processing circuit for AD conversion and noise reduction. The second signal S2 is supplied to the signal processing unit 5 via the second read circuit 4.

The signal processing unit 5 performs a process of detecting a global feature and a process of detecting a local feature using the input first signal S1 and second signal S2.
Further, the signal processing unit 5 determines the blend ratio of the first signal S1 and the second signal S2 by using the detected global feature or local feature, and the first signal S1 and the second signal S1 and the second signal S2 are determined based on the blend ratio. HDR (High Dynamic Range) synthesis is performed by blending the signal S2. As a result, the HDR image signal is output from the signal processing unit 5.

The global feature is a feature of a subject detected for each pixel area GA set as a rectangular area including, for example, a plurality of pixel units 6. Specifically, when the subject imaged in a certain pixel region GA is a light emitting body that blinks periodically, the feature as the subject that blinks periodically is detected. The signal processing unit 5 calculates the degree (high possibility) that the subject is a light emitting body that blinks periodically as a global feature as a likelihood.

In the following example, a subject that blinks periodically is referred to as a “periodic blinking body”. The periodic blinking body includes not only a subject that emits light by itself such as an LED (Light Emitting Diode), but also a subject that reflects the light of an LED that repeatedly turns on and off in several msec, for example.
The signal processing unit 5 calculates how much the subject in the pixel region GA is like a periodic blinker as a “global blinker likelihood LFL1”. That is, the global blinking object likelihood LFL1 is an index value indicating the degree of the periodic blinking object of the subject in the pixel region GA. If the subject imaged in the pixel region GA is an LED, the global blinking body likelihood LFL1 becomes high. Further, the more the LEDs are imaged in the many pixel units 6 in the pixel region GA, the higher the global blinker likelihood LFL1 is.

Further, the signal processing unit 5 calculates how much the subject has the characteristics as a moving subject for each pixel region GA as "global moving subject likelihood LFM1". That is, the global moving subject likelihood LFM1 is an index value indicating the degree of the moving subject of the subject. If the subject captured in the pixel region GA is a moving subject, the global moving subject likelihood LFM1 becomes high. Further, the larger the number of pixel units 6 in the pixel region GA, the higher the global moving subject likelihood LFM1.

It can be said that the detection of a periodic blinking object or a moving subject detects a subject whose luminance value changes in the pixel region GA with the passage of time. That is, these global features can be rephrased as features regarding the change in luminance with the passage of time.

The signal processing unit 5 treats the plurality of pixel units 6 as the pixel region GA in order to detect the global feature. For example, the pixel units 6 arranged two-dimensionally as shown in FIG. 3 are divided into several rectangular areas as shown in FIG. 4, and each rectangular area is treated as a pixel area GA.
For example, as shown in FIG. 5, when each of the vertical direction and the horizontal direction is divided into three pixel region GAs, nine pixel region GAs are set. The signal processing unit 5 detects global features for each of the nine pixel region GAs.

The local feature detected by the signal processing unit 5 is a feature of the subject detected for each pixel unit 6. That is, the feature of the subject detected for the first pixel G1 of the pixel unit 6 is treated as the feature of the subject in the pixel unit 6 to which the first pixel G1 belongs. The same applies to the second pixel G2.

The signal processing unit 5 treats the feature as a periodic blinker and the feature as a moving subject detected for the pixel unit 6 as local features. Further, the signal processing unit 5 calculates the local blinking body likelihood LFL2 and the local moving subject likelihood LFM2 as index values indicating the degree of the local feature. Local features are detected using global features and the like.

<2. Signal processing unit configuration and processing content>
FIG. 6 shows a configuration example of the signal processing unit 5. The signal processing unit 5 includes an inter-pixel difference detection unit 7, a global feature detection unit 8, a local feature detection unit 9, a blend rate determination unit 10, and a blend unit 11.

The pixel-to-pixel difference detection unit 7 detects the difference in luminance between the first pixel G1 and the second pixel G2 for each pixel unit 6, and calculates the difference in luminance as the “pixel-to-pixel difference value GD”. The pixel-to-pixel difference value GD can be effectively used in the subsequent processing because the exposure times of the first pixel G1 and the second pixel G2 are different.
The inter-pixel difference value GD is used in the local feature detection unit 9 to select whether the subject has a feature as a periodic blinker or a feature as a moving subject. Details will be described later.
The amount of data can be compressed by expressing the difference value GD between pixels with a small number of bits such as 256 steps from 0 to 255.

As described above, the global feature detection unit 8 detects the feature of the subject imaged in the pixel region GA as the global feature. Specifically, the global feature detection unit 8 calculates the global blinker likelihood LFL1 and the global moving subject likelihood LFM1 for each pixel region GA as global features. Depending on the pixel region GA, only the global blinking body likelihood LFL1 may be calculated, or only the global moving subject likelihood LFM1 may be calculated.
In the present embodiment, the global blinker likelihood LFL1 and the global moving subject likelihood LFM1 are calculated as values in the range of 0.0 to 1.0. The global blinking body likelihood LFL1 and the global moving subject likelihood LFM1 indicate that the higher the numerical value, the higher the characteristic of the subject imaged in the target pixel region GA.

Here, a more specific configuration example of the global feature detection unit 8 is shown in FIG.
The global feature detection unit 8 includes a first brightness calculation unit 21A that performs processing related to the first signal S1, a region-specific brightness average value first calculation unit 22A, a first memory 23A, and a region-specific brightness change first calculation unit 24A. I have.

Further, the global feature detection unit 8 includes a second brightness calculation unit 21B that performs processing related to the second signal S2, a region-specific brightness average value second calculation unit 22B, a second memory 23B, and a region-specific brightness change second calculation unit. It is equipped with 24B.

Further, the global feature detection unit 8 calculates the feature amount for each pixel region GA using the signals output from the region-by-region luminance change first calculation unit 24A and the region-by-region luminance change second calculation unit 24B. The quantity calculation unit 25 is provided.

The first luminance calculation unit 21A calculates the luminance value for each first pixel G1 based on the first signal S1. As an example, each pixel unit 6 in the pixel array unit 2 has a Bayer array color filter. Specifically, as illustrated in FIG. 8, the pixel array unit 2 has a pixel unit 6R as an R pixel having an R (red) color filter and a G pixel having a G (green) color filter. It has a pixel unit 6G and a pixel unit 6B as a B pixel having a B (blue) color filter.

The pixel unit 6R has a first pixel G1R and a second pixel G2R. Similarly, the pixel unit 6G has a first pixel G1G and a second pixel G2G, and the pixel unit 6B has a first pixel G1B and a second pixel G2B.

The first luminance calculation unit 21A treats the four pixel units 6 composed of two pixel units 6 vertically and horizontally as one pixel set GS for the pixel array unit 2 arranged as shown in FIG. 8, and sets the pixels. The luminance value L1S for the first pixel G1 is calculated for each GS. For example, the luminance value L1S for the pixel set GS is calculated using the following equation (1).

Luminance value of pixel set GS L1S = α × L1R + β × L1G + γ × L1B ... Equation (1)
however,
Luminance value of first pixel G1R = L1R
Luminance value of first pixel G1G = L1G
Luminance value of first pixel G1B = L1B
And.

Coefficients α, β, γ can be various. For example, the coefficient α = 2, the coefficient β = 1, and the coefficient γ = 2. Alternatively, the coefficient α = 0.4, the coefficient β = 0.2, and the coefficient γ = 0.4 may be set.

The calculated luminance value L1S is a luminance value for the first pixel G1 in one pixel set GS, but the luminance value L1S is the luminance value L1R of the first pixel G1R, the luminance value L1G of the first pixel G1G, and the first. It may be regarded as the luminance value L1B of the pixel G1B.

In summary, the first luminance calculation unit 21A calculates the luminance values L1R, L1G, L1B for each pixel unit 6 or the luminance values L1S for each pixel set GS.

The area-wise brightness average value first calculation unit 22A calculates the brightness average value L1A of the first pixel G1 for each pixel area GA. For example, when the pixel region GA includes n pixel set GS, the luminance value L1S for the first pixel G1 calculated for each pixel set GS is added and divided by n to obtain the luminance. The average value L1A is calculated.

The calculated luminance average value L1A is output to the area-by-region luminance change first calculation unit 24A and stored in the first memory 23A.

The first memory 23A is provided to store the brightness average value L1A one frame before.
The region-by-region brightness change first calculation unit 24A acquires the brightness average value L1A one frame before from the first memory 23A, and acquires the brightness average value L1A for the current frame from the region-by-region brightness average value first calculation unit 22A. .. The region-by-region brightness change first calculation unit 24A calculates the difference between the brightness average value L1A one frame before and the brightness average value L1A of the current frame. As a result, the luminance change of the first pixel G1 is calculated as the first pixel change amount G1V for each pixel region GA. The calculated first pixel change amount G1V is output to the feature amount calculation unit 25 for each area.

The second brightness calculation unit 21B, the area-specific brightness average value second calculation unit 22B, the second memory 23B, and the area-by-region brightness change second calculation unit 24B are processing units that perform processing related to the second signal S2, respectively. 1 Performs the same processing as the brightness calculation unit 21A, the area-by-region brightness average value first calculation unit 22A, the first memory 23A, and the region-by-region brightness change first calculation unit 24A.

Specifically, the second luminance calculation unit 21B calculates the luminance value L2S for the second pixel G2 for each pixel set GS. For example, the luminance value L2S for the pixel set GS is calculated using the following equation (2).

Luminance value of pixel set GS L2S = α'× L2R + β'× L2G + γ'× L2B ... Equation (2)
however,
Luminance value of second pixel G2R = L2R
Luminance value of second pixel G2G = L2G
Luminance value of second pixel G2B = L2B
And.

The coefficients α', β', and γ'may be, for example, a coefficient α'= 2, a coefficient β'= 1, a coefficient γ'= 2, a coefficient α'= 0.4, and a coefficient β'= 0. .2, the coefficient γ'= 0.4 may be set.

The second calculation unit 22B of the brightness average value for each region calculates the brightness average value L2A of the second pixel G2 for each pixel region GA.

The calculated luminance average value L2A is output to the region-by-region luminance change second calculation unit 24B and stored in the second memory 23B.

The area-by-region brightness change second calculation unit 24B acquires the brightness average value L2A one frame before from the second memory 23B, and acquires the brightness average value L2A for the current frame from the area-by-region brightness average value second calculation unit 22B. .. The second calculation unit 24B for the brightness change for each region calculates the difference between the brightness average value L2A one frame before and the brightness average value L2A of the current frame. As a result, the second pixel change amount G2V of the second pixel G2 is calculated for each pixel region GA. The calculated second pixel change amount G2V is output to the feature amount calculation unit 25 for each area.

The area-by-region feature amount calculation unit 25 uses the first pixel change amount G1V for each pixel area GA for the first pixel G1 and the second pixel change amount G2V for each pixel area GA for the second pixel G2. Calculate the feature amount for each.

Specifically, the area-by-region feature amount calculation unit 25 calculates the absolute value of the difference between the first pixel change amount G1V and the second pixel change amount G2V for the pixel area GA as the change amount difference value VD (≧ 0). ..
Then, the feature amount calculation unit 25 for each area calculates the feature amount for each pixel area GA based on the change amount difference value VD and the second pixel change amount G2V.

For example, the feature amount is calculated so that the larger the change amount difference value VD and the smaller the second pixel change amount G2V, the higher the global blinker likelihood LFL1.

In the pixel region GA where the change amount difference value VD is large, it is shown that the change in the brightness value of the subject can be captured only by either one of the first pixel G1 and the second pixel G2. Specifically, the first pixel change amount G1V can be rephrased as a change in the first signal S1 output from the first pixel G1 having a short exposure time. Since the exposure time of the first pixel G1 is short, when the subject is a periodic blinker, the LED in the non-light emitting state is imaged at the timing one frame before, and the LED in the light emitting state is imaged at the timing of the current frame. There is. In this case, the first pixel change amount G1V for the first pixel G1 becomes large.

On the other hand, the second pixel change amount G2V can be rephrased as a change in the second signal S2 output from the second pixel G2 having a long exposure time. Since the exposure time of the second pixel G2 is long, when the subject is a periodic blinker, there is a high possibility that the LED in the light emitting state is imaged at both the timing one frame before and the timing of the current frame. Therefore, the second pixel change amount G2V becomes small. In particular, when the exposure time of the second pixel G2 is longer than the length of the non-emission time of the periodic blinker, there is no frame in which only the non-emission time of the periodic blinker is imaged, so that the amount of change in the second pixel is changed. G2V is small.

Therefore, when the subject is a periodic blinker, the change amount difference value VD becomes large. Therefore, it can be estimated that there is a high possibility that the periodic blinker is imaged in the pixel region GA in which the change amount difference value VD is large and the second pixel change amount G2V is small.

It should be noted that the global blinking body likelihood LFL1 calculated by the feature amount calculation unit 25 for each region has strength and weakness of certainty.
For example, in the feature amount calculation unit 25 for each region, the larger the change amount difference value VD and the smaller the second pixel change amount G2V, the higher the global blinker likelihood LFL1. Calculate LFL1.

Further, the feature amount calculation unit 25 for each area calculates the global moving subject likelihood LFM1 as the feature amount for each pixel area GA.

For example, the feature amount is calculated so that the smaller the change amount difference value VD and the larger the second pixel change amount G2V, the higher the global moving subject likelihood LFM1.

The first pixel change amount G1V and the second pixel change amount G2V have a large change in luminance value when the subject is a moving subject, although there is a difference in exposure time. That is, since both the first pixel change amount G1V and the second pixel change amount G2V are large, the change amount difference value VD is small.

Therefore, it can be estimated that there is a high possibility that a moving subject has been imaged in the pixel region GA where the change amount difference value VD is small and the second pixel change amount G2V is large.

It should be noted that the global moving subject likelihood LFM1 calculated by the feature amount calculation unit 25 for each region has strength and weakness of certainty.
For example, the feature amount calculation unit 25 for each region increases the global moving subject likelihood LFM1 so that the smaller the change amount difference value VD and the larger the second pixel change amount G2V, the higher the global moving subject likelihood. Calculate LFM1.

The range of values that can be taken for the global blinking body likelihood LFL1 and the global moving subject likelihood LFM1 calculated by the feature amount calculation unit 25 for each region is 0.0 to 1.0. Specifically, it will be described with reference to FIG.

The graph shown in FIG. 9 shows the global characteristics of each pixel region GA with the change amount difference value VD as the horizontal axis and the second pixel change amount G2V as the vertical axis.
The pixel region GA in which the periodic blinker is imaged is distributed in the region AR1 near the horizontal axis of the graph. Further, the pixel region GA in which the moving subject is photographed is distributed in the region AR2 near the vertical axis of the graph.

Further, as for the global blinking body likelihood LFL1, the larger the change amount difference value VD is, the larger the pixel region GA is, and the smaller the second pixel change amount G2V is, the higher the global blinking body likelihood LFL1 is. That is, as shown in FIG. 9, the global blinker likelihood LFL1 of the point P1 located in the region AR1 is 1.0, and the global blinker likelihood LFL1 of the point P2 located in the region AR1 is 0. It is said to be 1.

Similarly, as for the global moving subject likelihood LFM1, the pixel region GA having a smaller change amount difference value VD and the pixel region GA having a larger second pixel change amount G2V have a higher global moving subject likelihood LFM1. That is, the global moving subject likelihood LFM1 of the point P3 located in the region AR2 is 1.0, and the global moving subject likelihood LFM1 of the point P4 located in the region AR2 is 0.1.

It should be noted that the pixel region GA having a larger change amount difference value VD and the pixel region GA having a smaller second pixel change amount G2V do not increase the global blinker likelihood LFL1 but are located near the center of the region AR1. The global blinker likelihood LFL1 for the pixel region GA may be 1.0.

Similarly, the pixel region GA having a larger change amount difference value VD and the pixel region GA having a smaller second pixel change amount G2V do not have a higher global blinker likelihood LFL1 but are located near the center of the region AR2. The global moving subject likelihood LFM1 for the located pixel region GA may be 1.0.

The region-specific feature amount calculation unit 25 outputs the global blinker likelihood LFL1 and the global moving subject likelihood LFM1 for each pixel region GA.

Returning to the description of FIG.
The local feature detection unit 9 performs a process of detecting the feature of the subject for each pixel unit 6 as a local feature. The global feature detected by the global feature detection unit 8 is the one that detects the feature of the subject in the pixel region GA, and the global feature in which the feature of the subject is detected for the specific pixel unit 6 in the pixel region GA. It may not be the case. In such a case, if the blending ratio is determined based on the global characteristics, there is a possibility that inappropriate blending will be performed in the specific pixel unit 6. In view of such circumstances, the local feature detection unit 9 detects the local feature for each pixel G.

Specifically, the local feature detection unit 9 uses the global blinker likelihood LFL1 output from the region-by-region feature amount calculation unit 25 and the pixel-to-pixel difference value GD output from the pixel-to-pixel difference detection unit 7. , The local blinker likelihood LFL2 is calculated. Further, the local moving subject likelihood LFM2 is calculated using the global moving subject likelihood LFM1 output from the region-by-region feature amount calculation unit 25 and the pixel-to-pixel difference value GD output from the pixel-to-pixel difference detecting unit 7. ..

First, the calculation of the local blinking body likelihood LFL2 will be described with reference to FIG.
The graph shown in FIG. 10 shows data for each pixel unit 6 with the global blinker likelihood LFL1 as the horizontal axis and the pixel-to-pixel difference value GD as the vertical axis. Here, the difference value GD between pixels on the vertical axis is normalized so as to take a value of 0.0 to 1.0.

The local feature detection unit 9 has a region in which the global blinking body likelihood LFL1 is set to a predetermined threshold Th1 or more and the pixel-to-pixel difference value GD is set to a predetermined threshold Th2 or more in order to prevent erroneous determination due to noise or the like. For the pixel unit 6 in which the data is plotted on Ar3, it is determined that the subject has the characteristic as a periodic blinker, and the local blinker likelihood LFL2 is calculated.

The local blinker likelihood LFL2 takes any value from 0.0 to 1.0. For example, as shown in FIG. 10, the local blinking body likelihood is plotted at the point P5 in which the global blinking body likelihood LFL1 is set to the threshold value Th1 and the pixel-to-pixel difference value GD is set to the threshold value Th2. LFL2 is calculated as 0.0. Further, with respect to the pixel unit 6 plotted at the point P6 in which the global blinking body likelihood LFL1 and the pixel-to-pixel difference value GD are both 1.0, it is highly possible that the captured subject is a periodic blinking body. Therefore, the local blinker likelihood LFL2 is calculated as 1.0.

In FIG. 10, the local blinker likelihood LFL2 is set to approach 1.0 as the point P5 approaches the point P6.
The local blinker likelihood LFL2 for the pixel unit 6 in which data is plotted in a region other than the region Ar3 may have no value, may be a negative value, or may be 0.0.

Next, the calculation of the local moving subject likelihood LFM2 will be described with reference to FIG.
The graph shown in FIG. 11 shows data for each pixel with the global moving subject likelihood LFM1 as the horizontal axis and the pixel-to-pixel difference value GD as the vertical axis. Here, the difference value GD between pixels on the vertical axis is normalized so as to take a value of 0.0 to 1.0.

The local feature detection unit 9 has a region in which the global moving subject likelihood LFM1 is set to a predetermined threshold Th3 or more and the pixel-to-pixel difference value GD is set to a predetermined threshold Th4 or more in order to prevent erroneous determination due to noise or the like. For the pixel unit 6 in which the data is plotted on Ar4, it is determined that the subject has the characteristics as a moving subject, and the local moving subject likelihood LFM2 is calculated.

The local moving subject likelihood LFM2 takes any value from 0.0 to 1.0. For example, as shown in FIG. 11, the local moving subject likelihood is plotted at the point P7 in which the global moving subject likelihood LFM1 is set to the threshold Th3 and the pixel-to-pixel difference value GD is set to the threshold Th4. LFM2 is calculated as 0.0. Further, with respect to the pixel unit 6 plotted at the point P8 in which the global moving subject likelihood LFM1 and the pixel-to-pixel difference value GD are both 1.0, it is highly possible that the captured subject is a moving subject. The local moving subject likelihood LFM2 is calculated as 1.0.

In FIG. 11, the local moving subject likelihood LFM2 is set to approach 1.0 as the point P7 approaches the point P8.
The local moving subject likelihood LFM2 for the pixel unit 6 in which data is plotted in a region other than the region Ar4 may have no value, may be a negative value, or may be 0.0.

In this way, the local feature detection unit 9 performs a process of dropping the global feature of the subject detected in each pixel region GA into the local feature of the pixel unit 6 (or each pixel G).

The local moving subject likelihood LFM2 may be calculated by using the global blinking body likelihood LFL1 without using the global moving subject likelihood LFM1.
For example, as a subject having a large difference value GD between pixels, a periodic blinking object or a moving subject can be considered. In such a pixel region GA, when the global blinker likelihood LFL1 is large, it is highly possible that the subject imaged in the pixel region GA is a periodic blinker. On the other hand, in the pixel region GA in which the pixel-to-pixel difference value GD is large and the global blinking body likelihood LFL1 is low, there is a high possibility that the captured subject is a moving subject. Therefore, it is possible to estimate that the subject captured by the pixel unit 6 is a moving subject without using the global moving subject likelihood LFM1.

Returning to the description of FIG.
The blend rate determination unit 10 determines the blend rate of the first signal S1 and the second signal S2. In particular, in the present embodiment, the blend rate determination unit 10 determines the blend rate according to the characteristics of the subject for each pixel unit 6.

Therefore, the blend rate determination unit 10 includes a temporary blend rate calculation unit 12 for calculating the temporary blend rate ABF, and a blend rate correction unit 13 for calculating the blend rate BF by correcting the temporary blend rate ABF. ..

The temporary blend rate calculation unit 12 calculates the temporary blend rate in a state where the characteristics of the subject are not taken into consideration as the temporary blend rate ABF. The temporary blend ratio ABF and the blend ratio BF represent the blend ratio of the second signal S2, and take a value from 0.0 to 1.0. That is, when the temporary blend ratio ABF and the blend ratio BF are 0.0, it means that the second signal S2 is not blended at all, and when it is 1.0, the blend ratio of the second signal S2 is 100%. Represents.

The first signal S1 output from the first pixel G1 having high light receiving sensitivity is input to the temporary blend ratio calculation unit 12. The temporary blend rate calculation unit 12 determines the temporary blend rate ABF according to the degree of saturation of the first signal S1. For example, as shown in FIG. 12, the temporary blend ratio ABF is set to 0.0 until the amount of incident light on the pixel G exceeds a predetermined threshold value Th5. Further, when the amount of incident light on the pixel G is a predetermined threshold value Th5 or more and less than the threshold value Th6, the provisional blend ratio ABF is 0.0 or more and less than 1.0. Then, when the amount of incident light on the pixel G exceeds a predetermined threshold value Th6, the temporary blend ratio ABF is fixed at 1.0. That is, when the amount of incident light on the pixel G exceeds the threshold value Th6, it is determined that the first pixel G1 is saturated, and the blend ratio of the first signal S1 becomes 0%.

The blend rate correction unit 13 uses the local blinker likelihood LFL2 and the local moving subject likelihood LFM2 calculated by the local feature detection unit 9, and the temporary blend rate calculated by the temporary blend rate calculation unit 12. ABF is corrected.

Specifically, when the local blinker likelihood LFL2 is large, the provisional blend ratio ABF is corrected so that the blend ratio BF becomes high, that is, the blend ratio of the second signal S2 becomes high. This is because the second signal S2 is a signal obtained by relatively lengthening the exposure time, and there is a high possibility that the light emitting state of a periodic blinker such as an LED is imaged.

Further, when the local blinking body likelihood LFL2 is small and the local moving subject likelihood LFM2 is large, the blend ratio BF is tentatively reduced, that is, the ratio of the second signal S2 is tentatively reduced. The blend ratio ABF is corrected. This is because the first signal S1 is a signal obtained by relatively shortening the exposure time, and blurring of a moving subject is suppressed.

As an example, for the pixel G whose local blinker likelihood LFL2 is larger than a predetermined value, the first signal S1 may be excluded by setting the blend ratio BF to 1.0. When the local blinking body likelihood LFL2 is smaller than the predetermined value and the local moving subject likelihood LFM2 is larger than the predetermined value, the blend ratio BF is set to 0.0 to set the second signal S2. May not be included.

The blend unit 11 performs a blend process based on the blend rate BF determined by the blend rate correction unit 13. For example, the blending unit 11 generates an HDR image signal by performing an alpha blending process based on the blending ratio BF. The alpha blend can be expressed by the following equation (3).

HDR image signal = (1-α) x S1 + α x S2 x Gain ... Equation (3)

When the local blinking body likelihood LFL2 is smaller than a predetermined value, the correction is made so that the blend ratio BF is 0.0 regardless of the value of the local moving subject likelihood LFM2, that is, the blend ratio BF. May be corrected so that

<3. First application example>
In the first application example, the case where the above-mentioned image sensor 1 is applied to the image pickup apparatus 100 will be described.

FIG. 13 shows a configuration example of the image pickup apparatus 100. In addition, in FIG. 13, only the main part included in the image pickup apparatus 100 is shown.

The image pickup apparatus 100 includes an optical lens system 101, an image sensor 1 including the above-mentioned signal processing unit 5, an image pickup signal processing unit 102, a control unit 103, a driver unit 104, a storage unit 105, and a communication unit. It includes 106 and a display unit 107.

The optical lens system 101 includes various lenses such as a zoom lens and a focus lens, an aperture mechanism, and the like. Light that has passed through the optical lens system 101 is incident on the pixel array unit 2 included in the image sensor 1.
A part of the optical lens system 101 may be provided in the image sensor 1 like a microlens provided for each pixel G.

The imaging signal processing unit 102 receives the HDR image signal synthesized by HDR from the image sensor 1, and the signal processing required to generate the image data for recording and the signal required to display the image on the display unit 107. Perform processing etc.

The control unit 103 controls the image sensor 1 and the image pickup signal processing unit 102. Further, by sending a control signal to the driver unit 104, the optical members such as various lenses and the aperture mechanism of the optical lens system 101 are controlled.

The driver unit 104 provides a drive signal to the optical member of the optical lens system 101 based on the control signal input from the control unit 103. This drives the optical member.

The storage unit 105 is composed of, for example, a non-volatile memory or the like. The storage unit 105 stores programs and the like required for each process executed by the control unit 103. Further, the storage unit 105 stores image data such as still image data and moving image data generated by the image pickup signal processing unit 102.

The storage unit 105 may be configured as a flash memory built in the image pickup device 100, or may be stored in or read from a memory card (for example, a portable flash memory) that can be attached to and detached from the image pickup device 100 and the memory card. It may be composed of an access unit that performs access for the purpose. Further, it may be realized as an HDD (Hard Disk Drive) or the like as a form built in the image pickup apparatus 100.

The communication unit 106 performs data communication and network communication with an external device by wire or wirelessly. For example, image data (still image data or moving image data) is transmitted to an external display device, recording device, playback device, or the like.
Further, the communication unit 106 may perform communication by various networks such as the Internet, a home network, and a LAN (Local Area Network), and may transmit and receive various data to and from a server, a terminal, or the like on the network. ..
Further, the communication unit 106 may function as an output unit that outputs image data to an external device.

The display unit 107 can display a reproduced image of the image data read from the recording medium as the storage unit 105. Further, the display unit 107 may be provided on the image pickup apparatus 100 as a rear monitor or a finder monitor provided on the back surface of the image pickup apparatus 100, and may be capable of displaying a so-called through image.

The display unit 107 may be in a form that can be attached to and detached from the image pickup apparatus 100.

In the image pickup signal processing unit 102 of the present embodiment, the first signal S1 output from the first pixel G1 and the second signal S2 output from the second pixel G2 are appropriately blended according to the characteristics of the subject. The HDR image signal can be received from the image sensor 1. Therefore, by performing various display processing, processing, and the like based on the HDR image signal, the imaging signal processing unit 102 appropriately displays periodic blinkers such as LEDs and suppresses blurring of moving subjects. It is possible to display HDR images and generate image files.

<4. Second application example>
In the second application example, an example in which the HDR image signal is provided from the above-mentioned image sensor 1 to the display device and the recognition device that performs the image recognition process will be described.
As an example of such an application, the in-vehicle system 200 will be mentioned here.

The in-vehicle system 200 includes an optical lens system 201, a driver unit 202, an image sensor 1 including the above-mentioned signal processing unit 5, a display device 300, and a recognition device 400.

Since the optical lens system 201 has the same configuration as the optical lens system 101 described in the first application example, and the driver unit 202 has the same configuration as the driver unit 104 described in the first application example, the description thereof will be omitted. ..

The display device 300 includes monitors and instruments provided at predetermined positions in the vehicle interior such as an instrument panel, and a circuit for controlling the display thereof.
The display device 300 includes a display signal processing unit 301, a display control unit 302, a storage unit 303, a communication unit 304, a display unit 305, and the like.

The display signal processing unit 301 performs signal processing for displaying an image on the display unit 305, and has the same configuration as the imaging signal processing unit 102 described in the first application example.

The display control unit 302 has the same configuration as the control unit 103 described in the first application example. Further, the storage unit 303 and the communication unit 304 have the same configuration as the storage unit 105 and the communication unit 106 described in the first application example. The communication unit 304 is capable of communication using CAN (Controller Area Network).

The display unit 305 is a monitor device or the like provided at a predetermined position in the vehicle interior, and can display image data captured by the image sensor 1. This enables the driver and passengers to grasp the situation outside the vehicle via the monitoring device.

The recognition device 400 performs subject recognition processing by performing various processing on the image data of the outside of the vehicle captured by the image sensor 1. The recognition result of the subject by the recognition device 400 is provided to the advanced driver-assistance system (ADAS).
The ADAS realizes a collision damage mitigation braking function, an ACC (Adaptive Cruise Control) function, and the like based on the information acquired from the recognition device 400.

The recognition device 400 includes a recognition signal processing unit 401, a recognition control unit 402, a storage unit 403, and a communication unit 404.

Since the storage unit 403 and the communication unit 404 have the same configuration as the storage unit 105 and the communication unit 106 described in the first application example, detailed description thereof will be omitted.

Data about the captured image is input from the image sensor 1 to the recognition signal processing unit 401. For example, the HDR synthesized HDR image signal may be input from the image sensor 1 to the recognition signal processing unit 401, or the first signal S1, the second signal S2, and each pixel G before HDR synthesis may be input. Information of the local blinking object likelihood LFL2 and the local moving subject likelihood LFM2 as the recognition result of the above may be input. Of course, all of this information may be input from the image sensor 1 to the recognition signal processing unit 401.

The recognition signal processing unit 401 recognizes a preceding vehicle, a pedestrian, a traffic light, a sign, or the like based on the information of the local blinking object likelihood LFL2 and the local moving subject likelihood LFM2 output from the image sensor 1. Image recognition processing is performed.
Specifically, based on the information of the local blinking body likelihood LFL2 output from the image sensor 1, it is possible to recognize a traffic light, a tail lamp, etc. as a periodic blinking body, for example, to recognize a preceding vehicle. can. Further, based on the information of the local moving subject likelihood LFM2 output from the image sensor 1, it is possible to recognize a preceding vehicle or a pedestrian as a moving subject.
Further, these image recognition processes may be realized by cooperating with the recognition signal processing unit 401 and the recognition control unit 402.

When the own vehicle is traveling, it may not be possible to estimate the subject as a moving subject based on the local moving subject likelihood LFM2. For example, while the own vehicle is traveling, a feature such as a moving subject is detected even from an obstacle that is not moving. Therefore, the recognition signal processing unit 401 may perform a process of recognizing a moving subject as an image in consideration of the traveling speed of the own vehicle and the like.
As a result, the recognition signal processing unit 401 can appropriately perform image recognition processing on the subject.

The recognition control unit 402 may recognize a subject such as a preceding vehicle or a pedestrian by executing an image recognition process using, for example, a DNN (Deep Neural Network).
The recognition result obtained by the recognition signal processing unit 401 and the recognition control unit 402 thus obtained is output to the ADAS via the communication unit 404.

By performing appropriate image recognition processing on a subject such as a pedestrian or a preceding vehicle in this way, it is possible to improve the running safety of the vehicle.

<5. Example of using three or more signals>
An example of performing feature detection of a subject using three or more types of pixel signals obtained by three or more readout processes having different light receiving sensitivities will be described.

The image sensor 1 may include three types of pixels having different areas, or may be capable of outputting three types of pixel signals using two types of pixels having different areas as shown in FIG. ..

For example, the first pixel G1 included in the image sensor 1 outputs the first signal S1 which is the pixel signal having the highest light receiving sensitivity, and the second pixel G2 is a pixel signal whose light receiving sensitivity is lower than that of the first pixel G1. The second signal S2 is output. Further, from the first pixel G1, the third signal S3 having the lowest light receiving sensitivity is output by making the exposure time shorter than the case where the first signal S1 and the second signal S2 are output.

As an example, the first signal S1 is output by setting the exposure time of the first pixel G1 to 8 msec. Further, the second signal S2 is output by setting the exposure time of the second pixel G2 to 11 msec. Further, the third signal S3 is output by setting the exposure time of the first pixel G1 to 4 msec.

FIG. 15 is a configuration example of a signal processing unit 5A that detects features of a subject using three types of pixel signals output from the image sensor 1. The same components as those in FIG. 6 are designated by the same reference numerals, and the description thereof will be omitted as appropriate.

The signal processing unit 5A uses the first signal S1 and the second signal S2 to perform global feature detection and local feature detection, so that the pixel-to-pixel difference detection unit 7, the global feature detection unit 8, and the local feature detection are performed. A unit 9 and a blend rate determining unit 10 are provided. Further, the first blending portion 11A is provided instead of the blending portion 11 shown in FIG.

Since each of these parts is a processing part that performs the same processing as each part shown in FIG. 6, the description thereof will be omitted.

The blend rate determination unit 10 includes a temporary blend rate calculation unit 12 and a blend rate correction unit 13. The blend ratio determining unit 10 determines the ratio of the blend of the second signal S2 to the first signal S1 as the blend ratio BF1.

The first blending unit 11A outputs an HDR image signal by performing an alpha blending process of the first signal S1 and the second signal S2 according to the blending ratio BF1 determined by the blending ratio determining unit 10.

The signal processing unit 5A uses the second signal S2 and the third signal S3 to perform global feature detection and local feature detection, so that the pixel-to-pixel difference detection unit 7A, the global feature detection unit 8A, and the local feature detection are performed. A unit 9A and a blend rate determining unit 10A are provided.

The blend rate determination unit 10A includes a temporary blend rate calculation unit 12A and a blend rate correction unit 13A.

Each of these parts is a processing unit that performs the same processing as each part shown in FIG. 6 except that the second signal S2 is used instead of the first signal S1 and the third signal S3 is used instead of the second signal S2. ..
However, the blend ratio determining unit 10A determines the blend ratio of the third signal S3 with respect to the HDR image signal output from the first blend unit 11A as the blend ratio BF2.

The second blending unit 11B performs an alpha blending process of the HDR image signal output from the first blending unit 11A and the third signal S3 according to the blending ratio BF2 determined by the blending ratio determining unit 10A.

In this way, by using three or more types of pixel signals, it is possible to widen the dynamic range in HDR synthesis.

<6. Modification example>
Here, modifications to some of the above-described embodiments will be described.
In the global feature detection unit 8 shown in FIG. 7, an example is described in which a first memory 23A and a second memory 23B for storing the luminance average value L1A and the luminance average value L2A obtained for each pixel area are provided. .. Further, since the brightness average values L1A and L2A one frame before are stored in each memory, the first pixel change is performed by the region-by-region luminance change first calculation unit 24A and the region-by-region luminance change second calculation unit 24B in the subsequent stage. An example of calculating the quantity G1V and the second pixel change amount G2V has been described.

In this modification, the first memory 23A and the second memory 23B store the brightness average values L1A and L2A of two frames or more. For example, the brightness average values L1A and L2A one frame before and the brightness average values L1A and L2A two frames before are stored.

Then, the region-by-region luminance change first calculation unit 24A and the region-by-region luminance change second calculation unit 24B perform processing for observing the transition of the luminance change. As a result, the region-by-region luminance change first calculation unit 24A and the region-by-region luminance change second calculation unit 24B can estimate the blinking cycle of the periodic blinker.

If the blinking cycle of the periodic blinking object can be estimated, the exposure time at which the light emitting state of the periodic blinking object can be reliably imaged can be set for the second pixel G2. This makes it possible to generate an HDR image signal in which an LED or the like is reliably captured. In particular, when the process of recognizing the preceding vehicle or the traffic light is executed by the processing unit in the subsequent stage, it is possible to improve the running safety of the vehicle, which is preferable.

As another modification, an example in which a local feature is not detected will be described.
In the above example, the signal processing unit 5 includes both the global feature detection unit 8 and the local feature detection unit 9, and the local blinker likelihood LFL2 and the local motion output from the local feature detection unit 9. The case where the blend ratio is determined using the subject likelihood LFM2 has been described.

In this modification, a case where the signal processing unit 5 includes only the global feature detection unit 8 and does not include the local feature detection unit 9 will be described.

The global feature detection unit 8 detects the global feature of the subject for each pixel region GA virtually set in the pixel array unit 2. Here, if the pixel region GA is set as a sufficiently small region, a desired HDR image signal can be obtained by using the characteristics of the subject for each pixel region GA detected by the global feature detection unit 8. In some cases, the blend ratio BF can be determined appropriately.

In this case, the signal processing unit 5 may be configured without the inter-pixel difference detection unit 7 and the local feature detection unit 9 shown in FIG. That is, the blend rate correction unit 13 uses the global blinker likelihood LFL1 and the global moving subject likelihood LFM1 for each pixel region GA output from the global feature detection unit 8, and the blend ratio for each pixel region GA. Calculate BF.

The blending unit 11 outputs an HDR image signal by performing HDR composition using the blending ratio BF for each pixel region GA.
As a result, the configuration of the signal processing unit 5 can be simplified, and the processing load can be reduced.

As yet another modification, it is conceivable to make the threshold values Th1 and Th2 shown in FIG. 10 and the threshold values Th3 and Th4 shown in FIG. 11 variable.
These threshold values are set to suppress erroneous determination that the subject is erroneously determined to be a periodic blinker or is determined to be a moving subject.

However, depending on the imaging environment, erroneous determination may still occur frequently. Specifically, it is a dark imaging environment or the like.
When it is estimated that the brightness of the surrounding environment is lower than a certain value based on the output of the brightness sensor, it is conceivable to raise any or all of the threshold values Th1, Th2, Th3, Th4. By increasing each threshold value, the region Ar3 shown in FIG. 10 and the region Ar4 shown in FIG. 11 become narrower, and the local blinking body likelihood LFL2 and the local moving subject likelihood LFM2 for the subject are calculated to be lower.

This makes it possible to suppress the above-mentioned erroneous determination and generate an appropriate HDR image signal.

Further, another modification will be described.
In the above-mentioned example, the process of determining the blend ratio so that the ratio of the second signal S2 becomes high when the global blinking body likelihood LFL1 or the local blinking body likelihood LFL2 for the subject is high has been described.
Here, when the global blinking body likelihood LFL1 or the local blinking body likelihood LFL2 is low, the blending ratio may be basically determined so that the ratio of the first signal S1 is high. Further, as long as the global blinking body likelihood LFL1 and the local blinking body likelihood LFL2 are not high, the blending ratio may be determined so that the ratio of the second signal S2 is 0%.

As a result, even if the global moving subject likelihood LFM1 and the local moving subject likelihood LFM2 are calculated to be low and it is determined that the moving subject is unlikely to be a moving subject by mistake, the subject is concerned. It is possible to suppress the occurrence of blurring.

<7. Summary>
As described above, the image sensor 1 in the embodiment has the first pixel G1 in which the reading is performed by the first light receiving sensitivity and the reading by the second light receiving sensitivity which is lower than the first light receiving sensitivity. The first signal S1 and the first exposure time output from the first pixel G1 which is the first exposure time are different from the pixel array unit 2 in which the second pixel G2 is arranged in two dimensions. Based on the second signal S2 output from the second pixel G2, which is the second exposure time, the feature of the pixel region GA including a plurality of the first pixel G1 and the second pixel G2 is a global feature. It is provided with a global feature detection unit 8 (8A) for detecting as. Further, the image sensor 1 determines the blend ratio BF (BF1, BF2) of the first signal S1 and the second signal S2 based on the detection result of the global feature detection unit 8 (8A). ) And.
The difference in the light receiving sensitivity between the first pixel G1 and the second pixel G2 may be due to the difference in the light receiving area in the pixel, the difference in the length of the exposure time, or the difference in the light receiving element. It may be due to the difference in characteristics. Further, the global feature is a feature of a subject imaged in a set of pixel region GA.
By detecting the global features for each group of pixel region GAs, the blend ratio BE of the first signal S1 and the second signal S2 can be changed for each pixel region GA, and appropriate HDR synthesis can be performed with a small processing load. Can be done.
Further, by using the signals of the first pixel G1 and the second pixel G2 (first signal S1 and second signal S2) having different exposure times for each pixel region GA, a global feature when the subject is a periodic blinker. It is possible to detect global features when the subject is a moving subject.
Further, many subjects are imaged in the pixel array unit 2 with a certain range. That is, few subjects are imaged with only one pixel G. Therefore, by detecting the global feature for each pixel region GA, it is possible to efficiently capture the feature of the subject. Further, when one pixel G fails and a normal pixel signal cannot be output, the pixel signal output from the pixel G may have a feature as, for example, a periodic blinker. be. In such a case, it is conceivable that the characteristics of the subject with respect to the pixel G may be erroneously detected. However, as in this configuration, when a global feature is detected together with a plurality of surrounding pixels G, the feature of the subject detected for the failed pixel G is almost ignored, and the feature is erroneously detected. Inappropriate blending process based on is not executed. This makes it possible to execute an appropriate blending process even when a failure occurs in the pixel G.

As shown in the description of the signal processing unit 5, the global feature may be a feature regarding the change in luminance with the passage of time.
As a subject having a characteristic of a change in brightness with the passage of time, for example, a light emitting body such as an LED (Light Emitting Diode) that blinks in a cycle of several msec to several tens of msec, or a light emitting body that reflects light emitted from the light emitting body is reflected. It can be mentioned as a subject whose brightness is changing, such as an object. In particular, a subject with a large change in brightness is more likely to be detected. Further, even when the subject is a moving subject with a large movement, a large change in luminance is likely to be detected.
By detecting such a subject, it is possible to appropriately blend the first signal S1 and the second signal S2, and it is possible to suppress artifacts generated by a periodic blinker or a moving subject. ..

The global feature detection unit 8 (8A) in the image sensor 1 may perform detection by calculating the likelihood representing the degree to which the image pickup subject is a periodic blinker as the global blinker likelihood LFL1.
As a result, when the subject is a periodic blinking object such as an LED, the global blinking object likelihood LFL1 is calculated to be high. In particular, the global blinker likelihood LFL1 is increased when a periodic blinker is imaged in most of the target pixel region GA.
In addition to the periodic blinking body, moving subjects can be considered as the subject whose brightness changes, but by calculating the global blinking body likelihood LFL1 as in this configuration, the subject moves with the periodic blinking body. It is possible to distinguish which of the subjects it is.
As a result, the blend ratio BF (BF1, BF2) can be changed according to whether the subject is a periodic blinker or a moving subject, and appropriate HDR composition according to the subject becomes possible.

As described with reference to FIG. 7 and the like, the global feature detection unit 8 (8A) of the image sensor 1 changes the amount of change over time in the brightness average value of the first pixel G1 for each pixel region GA by the first pixel. Processing to calculate as the amount G1V, processing to calculate the temporal change amount of the brightness average value of the second pixel G2 for each pixel area GA as the second pixel change amount G2V, and the first pixel change amount G1V and the second pixel. The detection may be performed by executing a process of calculating the difference of the change amount G2V by VDing the change amount difference value.
The amount of data for the information of the first pixel change amount G1V and the second pixel change amount G2V, which are the temporal changes in the brightness average values L1A and L2A for each pixel region GA, is smaller than that calculated for each pixel G. To. Similarly, the amount of data is also reduced for the change amount difference value VD calculated from the first pixel change amount G1V and the second pixel change amount G2V.
By using such information, it is possible to detect global features with a small processing load.

As described with reference to FIG. 9 and the like, in the image sensor 1, the global blinker likelihood LFL1 is calculated so as to be larger as the change amount difference value VD is larger and larger as the second pixel change amount G2V is smaller. You may.
For example, when the exposure time of the first pixel G1 having a high light receiving sensitivity is shorter than the exposure time of the second pixel G2 having a low light receiving sensitivity, all the exposure times of the first pixel G1 are in a non-light emitting state in the periodic blinker. May be included. Even in such a case, a part of the exposure time of the second pixel G2 may be included in the light emitting state in the periodic blinker. In such a case, the change amount difference value VD becomes large.
Further, with respect to the second pixel change amount G2V, there is a high possibility that the second pixel G2 having a long exposure time shows a periodic blinker in a light emitting state. In particular, when the exposure time is longer than the non-light emission time of the periodic blinker, the light emission state of the periodic blinker is surely captured, and the second pixel change amount G2V becomes small.
Therefore, when the change amount difference value VD is large and the second pixel change amount G2V is small, the global blinking body likelihood LFL1 appropriately determines that the subject is a periodic blinking body, so that an appropriate blend ratio BF is obtained. (BF1, BF2) can be set.

As described with reference to FIG. 6 and the like, in the image sensor 1, the local feature detection unit 9 that detects the feature of each pixel G by using the global feature detected by the global feature detection unit 8 (8A). (9A) may be provided.
As a result, the characteristics of the subject are detected for each pixel G.
Therefore, an appropriate blend ratio BF (BF1, BF2) is determined for each pixel G, and optimum HDR synthesis can be performed.

As described with reference to FIGS. 6 and 10, the local feature detection unit 9 (9A) of the image sensor 1 has a pixel-to-pixel difference calculated as a difference between the luminance values of the first pixel G1 and the second pixel G2. The blinking body likelihood for each pixel G may be calculated as the local blinking body likelihood LFL2 based on the value GD and the global blinking body likelihood LFL1.
The pixel-to-pixel difference value GD and the global blinking object likelihood LFL1 can be used to estimate the degree to which the subject is a periodic blinking object.
As a result, it is possible to detect that the subject is a periodic blinker, so that the blend ratio BF (BF1, BF2) can be appropriately set for each pixel G.

As described with reference to FIG. 10 and the like, in the image sensor 1, the local blinking body likelihood LFL2 is larger for the pixel G having a larger pixel-to-pixel difference value GD, and larger for the pixel G having a larger global blinking body likelihood LFL1. It may be calculated so as to be.
For example, as a subject having a large difference value GD between pixels, a moving subject, a periodic blinker, or the like can be mentioned. Among these subjects, a subject having a large inter-pixel difference value GD and having a high global blinking body likelihood LFL1 is likely to be a periodic blinking object rather than a moving subject.
As described above, since the local blinking body likelihood LFL2 is an index indicating the certainty that the subject imaged by the pixel G is a periodic blinking body, the blend ratio BF (blend ratio BF) is based on the local blinking body likelihood LFL2. BF1, BF2) can be appropriately set for each pixel G.

As described in the modified example, the blend ratio determination unit 10 (10A) of the image sensor 1 has a blend ratio BF (BF1, BF2) so that the higher the global blinker likelihood LFL1, the higher the ratio of the second signal S2. ) May be determined.
Depending on the relationship between the exposure times of the first pixel G1 and the second pixel G2, the second pixel G2 has a longer exposure time than the first pixel G1. In this case, it is highly possible that the second pixel G2 is imaged with a periodic blinker in a light emitting state.
Therefore, by increasing the ratio of the second signal S2 output from the second pixel G2, it is possible to increase the possibility of obtaining image data in which the periodic blinker is emitting light.

As described above, the blend ratio determination unit 10 (10A) of the image sensor 1 has a blend ratio BF (BF1, BF1) for each pixel G so that the higher the local blinker likelihood LFL2, the higher the ratio of the second signal S2. BF2) may be determined.
In the above example, the second pixel G2 has a longer exposure time than the first pixel G1. In this case, the second pixel G2 is likely to be generated in a state of capturing a periodic blinker in a light emitting state.
By increasing the ratio of the second signal S2 based on the height of the local blinker likelihood LFL2 for each pixel G, the possibility of obtaining image data in which the periodic blinker is emitting light is increased. be able to. In particular, by using such a method for determining the blend ratio for each pixel G, the blend ratio is determined as if the periodic blinker is imaged for the pixel G in which the periodic blinker is not imaged. Therefore, it is possible to perform appropriate HDR composition according to the characteristics of the subject for each pixel G.

As described with reference to FIG. 10 and the like, the local feature detection unit 9 (9A) of the image sensor 1 can execute the process of calculating the local moving subject likelihood LFM2, and the local moving subject likelihood LFM2 can be executed. May be calculated so that the larger the pixel-to-pixel difference value GD is, the larger the pixel G is, and the smaller the global blinking likelihood LFL1 is, the larger the pixel G is.
For example, as a subject having a large difference value GD between pixels, a moving subject, a periodic blinker, or the like can be mentioned. Among these subjects, a subject having a large inter-pixel difference value GD and having a low global blinking likelihood LFL1 is highly likely to be a moving subject.
By appropriately detecting that the subject is a moving subject in this way, the blend ratio BF (BF1, BF2) can be appropriately set for each pixel G. Further, since it is not necessary to calculate an index value or the like indicating the degree to which the subject is a moving subject, it is possible to determine an appropriate blend ratio BF (BF1, BF2) while suppressing an increase in the processing load.

As described with reference to FIGS. 7 and 9, the global feature detection unit 8 (8A) of the image sensor 1 determines the amount of time change in the brightness of the first pixel G1 for each pixel region GA as the first pixel. The process of calculating the change amount G1V, the process of calculating the temporal change amount of the brightness of the second pixel G2 for each pixel area GA as the second pixel change amount G2V, and the first pixel change amount G1V and the second pixel change. The process of calculating the difference of the amount G2V as the change amount difference value VD is executed, and the global moving subject likelihood LFM1 indicating the degree to which the imaged subject is a moving subject is larger and the first as the change amount difference value VD is smaller. It may be calculated so that the larger the two-pixel change amount G2V is, the larger the amount is.
For example, even if the light receiving sensitivities are different, the amount of change in the first signal S1 and the second signal S2 output from the first pixel G1 and the second pixel G2 is large when the subject is a moving subject. It is possible that the difference will be small.
Therefore, when the change amount difference value VD is small and the second pixel change amount G2V is large, the global moving subject likelihood LFM1 appropriately determines that the subject is a moving subject, so that an appropriate blend ratio BF ( BF1, BF2) can be set.

As described with reference to FIGS. 6 and 11, in the image sensor 1, the local feature detection unit 9 that executes a process of calculating the local moving subject likelihood LFM2, which is the moving subject likelihood for each pixel G. (9A) is provided, and the local moving subject likelihood LFM2 is larger as the pixel-to-pixel difference value GD calculated as the difference in brightness between the first pixel G1 and the second pixel G2 is larger, and the global moving subject likelihood LFM1 is provided. It may be calculated so that the larger the pixel, the larger the pixel.
For example, as a subject having a large difference value GD between pixels, a moving subject, a periodic blinker, or the like can be mentioned. Among these subjects, a subject having a large inter-pixel difference value GD and having a high local moving subject likelihood LFM2 is likely to be a moving subject rather than a periodic blinker.
In this way, since it is possible to appropriately detect that the subject is a moving subject, it is possible to appropriately set the blend ratio BF (BF1, BF2) for each pixel G.

As described in the modified example, the blend ratio determination unit 10 (10A) of the image sensor 1 has a blend ratio BF (BF1, BF2) so that the higher the global moving subject likelihood LFM1, the higher the ratio of the first signal S1. ) May be determined.
The first signal S1 is a signal obtained after a shorter exposure time than the second signal S2. Therefore, the image data generated based on the first signal S1 has less blurring of the moving subject than the image data generated based on the second signal S2.
Therefore, by increasing the ratio of the first signal S1 and performing blending, it is possible to obtain image data in which blurring of a moving subject is suppressed.

As described in each of the above examples, the blend rate determining unit 10 (10A) of the image sensor 1 blends each pixel G so that the higher the local moving subject likelihood LFM2, the higher the ratio of the first signal S1. The rate BF (BF1, BF2) may be determined.
The first signal S1 is a signal obtained after a shorter exposure time than the second signal S2. Therefore, the image data generated based on the first signal S1 has less blurring of the moving subject than the image data generated based on the second signal S2.
By increasing the ratio of the first signal S1 based on the height of the local moving subject likelihood LFM2 for each pixel G, it is possible to obtain image data in which blurring of the moving subject is suppressed. In particular, by using such a method for determining the blend ratio BF (BF1, BF2) for each pixel G, the second signal is obtained for the other pixel G in which the periodic blinker is imaged while suppressing the blurring of the moving subject. By increasing the ratio of S2, it is possible to obtain image data in which the light emitting state of the periodic blinker is captured.

As described with reference to FIG. 2 and the like, the first pixel G1 and the second pixel G2 of the image sensor 1 may be configured to have different light receiving sensitivities due to different light receiving areas.
Thereby, the global feature of each pixel region GA can be detected by using the first pixel G1 and the second pixel G2 having different light receiving sensitivities.
Therefore, an appropriate blend ratio BF (BF1, BF2) based on the global characteristics can be determined for each pixel region GA.

As described with reference to FIG. 2 and the like, the exposure time of the first pixel G1 of the image sensor 1 may be shorter than the exposure time of the second pixel G2. For example, the exposure time of the first pixel G1 is 8 msec, and the exposure time of the second pixel G2 is 11 msec.
This makes it possible to detect, for example, the characteristic of a periodic blinking body that blinks periodically as a global characteristic.
Therefore, it is possible to appropriately determine the blend ratio BF (BF1, BF2) based on the global characteristics and perform HDR synthesis.
It is desirable that the exposure time of the second pixel G2 is set based on the blinking cycle of the periodic blinking object. In particular, when it is used as an in-vehicle image sensor 1, the target of the subject as a periodic blinker such as a brake lamp or a traffic light of a preceding vehicle is clarified, so an appropriate exposure time is set. can do.

As described with reference to FIG. 15 in an example using three or more signals, the pixel array unit 2 of the image sensor 1 has a third light receiving sensitivity that is different from both the first light receiving sensitivity and the second light receiving sensitivity. The third signal S3 read by the above may be output. Further, the global feature detection unit 8 (8A) includes a plurality of pixels G to which the third signal S3 is output based on one of the first signal S1 and the second signal S2 and the third signal S3. Global features may be detected for the pixel region GA.
As a result, the blend rate determination unit 10 (10A) determines the blend rate for blending the three pixel signals.
Therefore, the dynamic range in HDR synthesis can be increased.

It should be noted that the effects described in the present specification are merely examples and are not limited, and other effects may be obtained.

<8. This technology>
(1)
A pixel array unit in which a first pixel that is read by a first light receiving sensitivity and a second pixel that is read by a second light receiving sensitivity, which is lower than the first light receiving sensitivity, are arranged two-dimensionally. When,
The first signal output from the first pixel, which is the first exposure time, and the second signal, which is output from the second pixel, which is the second exposure time different from the first exposure time. Based on this, a global feature detection unit that detects features of a pixel region containing a plurality of the first pixel and the second pixel as global features,
An image sensor including a blend rate determining unit that determines the blending rate of the first signal and the second signal based on the detection result of the global feature detecting unit.
(2)
The image sensor according to (1) above, wherein the global feature is a feature regarding a change in luminance with the passage of time.
(3)
The image sensor according to (2) above, wherein the global feature detection unit performs the detection by calculating the likelihood representing the degree to which the image-receiving subject is a periodic blinker as the global blinker likelihood.
(4)
The global feature detection unit calculates a temporal change amount of the brightness average value of the first pixel for each pixel area as a first pixel change amount, and a processing for calculating the brightness of the second pixel for each pixel area. The process of calculating the temporal change amount of the average value as the second pixel change amount and the process of calculating the difference between the first pixel change amount and the second pixel change amount as the change amount difference value are executed. The image sensor according to (3) above.
(5)
The image sensor according to (4) above, wherein the global blinking body likelihood is calculated so as to be larger as the change amount difference value is larger and larger as the second pixel change amount is smaller.
(6)
The image sensor according to any one of (3) to (5) above, comprising a local feature detection unit that detects a feature for each pixel using the global feature detected by the global feature detection unit.
(7)
The local feature detection unit locally determines the blinking body likelihood for each pixel based on the pixel-to-pixel difference value calculated as the difference between the luminance values of the first pixel and the second pixel and the global blinking body likelihood. The image sensor according to (6) above, which is calculated as the target blinking body likelihood.
(8)
The image sensor according to (7) above, wherein the local blinking body likelihood is calculated so that the pixel having a larger difference value between pixels is larger and the pixel having a larger global blinking body likelihood is larger.
(9)
The image according to any one of (3) to (8) above, wherein the blend ratio determining unit determines the blend ratio so that the higher the global blinking likelihood is, the higher the ratio of the second signal is. Sensor.
(10)
The image sensor according to (8) above, wherein the blend ratio determining unit determines the blend ratio for each pixel so that the higher the local blinker likelihood is, the higher the ratio of the second signal is.
(11)
The local feature detector is
It is possible to execute the process of calculating the likelihood of a local moving subject.
The local moving subject likelihood is calculated so as to be larger for pixels having a larger difference value between pixels and larger for pixels with a smaller global blinking object likelihood. Either (8) or (10) above. Image sensor described in.
(12)
The global feature detector is
The process of calculating the temporal change in the brightness of the first pixel for each pixel area as the first pixel change, and the temporal change in the brightness of the second pixel for each pixel region in the second pixel. A process of calculating as a change amount and a process of calculating the difference between the first pixel change amount and the second pixel change amount as a change amount difference value are executed.
The global moving subject likelihood, which indicates the degree to which the imaged subject is a moving subject, is calculated so as to be larger as the change amount difference value is smaller and larger as the second pixel change amount is larger. The image sensor according to any one of (11).
(13)
It is equipped with a local feature detection unit that executes processing to calculate the local moving subject likelihood, which is the moving subject likelihood for each pixel.
The local moving subject likelihood is
In the above (12), the pixel-to-pixel difference value calculated as the difference in brightness between the first pixel and the second pixel is calculated to be larger as the pixel is larger and the pixel having the global moving subject likelihood is larger. The image sensor described.
(14)
The image according to any one of (12) to (13) above, wherein the blend ratio determining unit determines the blend ratio so that the higher the global moving subject likelihood is, the higher the ratio of the first signal is. Sensor.
(15)
The image sensor according to (13) above, wherein the blend ratio determining unit determines the blend ratio for each pixel so that the higher the local moving subject likelihood is, the higher the ratio of the first signal is.
(16)
The image sensor according to any one of (1) to (15) above, wherein the first pixel and the second pixel have different light receiving sensitivities due to different light receiving areas.
(17)
The image sensor according to any one of (1) to (16) above, wherein the exposure time of the first pixel is shorter than the exposure time of the second pixel.
(18)
The pixel array unit is capable of outputting a third signal read by a third light receiving sensitivity that is different from both the first light receiving sensitivity and the second light receiving sensitivity.
The global feature detection unit is a global feature region for a pixel region including a plurality of pixels to which the third signal is output based on the signal of either one of the first signal and the second signal and the third signal. The image sensor according to any one of the above (1) to (17).
(19)
An image sensor that performs photoelectric conversion and
An optical member that collects the reflected light from the subject on the image sensor is provided.
The image sensor is
A pixel array unit in which a first pixel that is read by a first light receiving sensitivity and a second pixel that is read by a second light receiving sensitivity, which is lower than the first light receiving sensitivity, are arranged two-dimensionally. When,
The first signal output from the first pixel, which is the first exposure time, and the second signal, which is output from the second pixel, which is the second exposure time different from the first exposure time. Based on this, a global feature detection unit that detects features of a pixel region containing a plurality of the first pixel and the second pixel as global features,
An image pickup apparatus including a blend rate determining unit that determines the blending rate of the first signal and the second signal based on the detection result of the global feature detecting unit.
(20)
The first signal output from the first pixel, which is read out by the first light receiving sensitivity according to the first exposure time, and the second exposure time, which is different from the first exposure time, are higher than the first light receiving sensitivity. Features of a pixel region containing a plurality of the first pixel and the second pixel based on a second signal output from the second pixel, which is read out by a second light receiving sensitivity having a low sensitivity. And the process of detecting as a global feature,
A signal processing method comprising a process of determining a blend ratio of the first signal and the second signal based on a detection result of the global feature.

1 Image sensor 2 Pixel array unit 5, 5A

Signal processing unit

8, 8A Global

feature detection unit

9, 9A Local

feature detection unit

10, 10A Blend rate determination unit G pixel G1 1st pixel G2 2nd pixel GA pixel area S1 1st signal S2 2nd signal S3 3rd signal G1V 1st pixel change amount G2V 2nd pixel change amount VD change amount difference value LFL1 global blinker likelihood LFL2 local blinker likelihood LFM1 global moving subject likelihood LFM2 Local moving subject likelihood GD Pixel-to-pixel difference value BF, BF1, BF2 Blend rate

Claims

A pixel array unit in which a first pixel that is read by a first light receiving sensitivity and a second pixel that is read by a second light receiving sensitivity, which is lower than the first light receiving sensitivity, are arranged two-dimensionally. When,
The first signal output from the first pixel, which is the first exposure time, and the second signal, which is output from the second pixel, which is the second exposure time different from the first exposure time. Based on this, a global feature detection unit that detects features of a pixel region containing a plurality of the first pixel and the second pixel as global features,
An image sensor including a blend rate determining unit that determines the blending rate of the first signal and the second signal based on the detection result of the global feature detecting unit.
The image sensor according to claim 1, wherein the global feature is a feature regarding a change in luminance with the passage of time.
The image sensor according to claim 2, wherein the global feature detection unit performs the detection by calculating the likelihood representing the degree to which the image-receiving subject is a periodic blinking object as the global blinking object likelihood.
The global feature detection unit calculates a temporal change amount of the brightness average value of the first pixel for each pixel area as a first pixel change amount, and a processing for calculating the brightness of the second pixel for each pixel area. The process of calculating the temporal change amount of the average value as the second pixel change amount and the process of calculating the difference between the first pixel change amount and the second pixel change amount as the change amount difference value are executed. The image sensor according to claim 3, wherein the detection is performed by the above.
The image sensor according to claim 4, wherein the global blinking body likelihood is calculated so that the larger the change amount difference value is, the larger the change amount is, and the smaller the second pixel change amount is, the larger the change amount is.
The image sensor according to claim 3, further comprising a local feature detection unit that detects a feature for each pixel using the global feature detected by the global feature detection unit.
The local feature detection unit locally determines the blinking body likelihood for each pixel based on the pixel-to-pixel difference value calculated as the difference between the luminance values of the first pixel and the second pixel and the global blinking body likelihood. The image sensor according to claim 6, which is calculated as a target blinking body likelihood.
The image sensor according to claim 7, wherein the local blinking body likelihood is calculated so that the pixel having a larger difference value between pixels is larger and the pixel having a larger global blinking body likelihood is larger.
The image sensor according to claim 3, wherein the blend ratio determining unit determines the blend ratio so that the higher the global blinking likelihood is, the higher the ratio of the second signal is.
The image sensor according to claim 8, wherein the blend ratio determining unit determines the blend ratio for each pixel so that the higher the local blinker likelihood is, the higher the ratio of the second signal is.
The local feature detector is
It is possible to execute the process of calculating the likelihood of a local moving subject.
The image sensor according to claim 8, wherein the local moving subject likelihood is calculated so that the pixel having a larger difference value between pixels has a larger likelihood, and the pixel having a smaller global blinking body likelihood has a higher likelihood.
The global feature detector is
The process of calculating the temporal change in the brightness of the first pixel for each pixel area as the first pixel change, and the temporal change in the brightness of the second pixel for each pixel region in the second pixel. A process of calculating as a change amount and a process of calculating the difference between the first pixel change amount and the second pixel change amount as a change amount difference value are executed.
The third aspect of claim 3 is that the global moving subject likelihood, which indicates the degree to which the imaged subject is a moving subject, is calculated so as to be larger as the change amount difference value is smaller and larger as the second pixel change amount is larger. Image sensor.
It is equipped with a local feature detection unit that executes processing to calculate the local moving subject likelihood, which is the moving subject likelihood for each pixel.
The local moving subject likelihood is
The twelfth claim is calculated so that the larger the pixel-to-pixel difference value calculated as the difference in brightness between the first pixel and the second pixel, the larger the pixel, and the larger the global moving subject likelihood is, the larger the pixel is. Image sensor.
The image sensor according to claim 12, wherein the blend ratio determining unit determines the blend ratio so that the ratio of the first signal increases as the global moving subject likelihood increases.
The image sensor according to claim 13, wherein the blend ratio determining unit determines the blend ratio for each pixel so that the higher the local moving subject likelihood is, the higher the ratio of the first signal is.
The image sensor according to claim 1, wherein the first pixel and the second pixel have different light receiving sensitivities due to different light receiving areas.
The image sensor according to claim 1, wherein the exposure time of the first pixel is shorter than the exposure time of the second pixel.
The pixel array unit is capable of outputting a third signal read by a third light receiving sensitivity that is different from both the first light receiving sensitivity and the second light receiving sensitivity.
The global feature detection unit is a pixel region including a plurality of pixels to which the third signal is output based on the signal of either one of the first signal and the second signal and the third signal. The image sensor according to claim 1, wherein the feature is detected.
An image sensor that performs photoelectric conversion and
An optical member that collects the reflected light from the subject on the image sensor is provided.
The image sensor is
A pixel array unit in which a first pixel that is read by a first light receiving sensitivity and a second pixel that is read by a second light receiving sensitivity, which is lower than the first light receiving sensitivity, are arranged two-dimensionally. When,
The first signal output from the first pixel, which is the first exposure time, and the second signal, which is output from the second pixel, which is the second exposure time different from the first exposure time. Based on this, a global feature detection unit that detects features of a pixel region containing a plurality of the first pixel and the second pixel as global features,
An image pickup apparatus including a blend rate determining unit that determines the blending rate of the first signal and the second signal based on the detection result of the global feature detecting unit.
The first signal output from the first pixel, which is read out by the first light receiving sensitivity according to the first exposure time, and the second exposure time different from the first exposure time, are higher than the first light receiving sensitivity. Based on the second signal output from the second pixel, which is read out by the second light receiving sensitivity, which has a low sensitivity, the characteristics of the pixel region including a plurality of the first pixel and the second pixel are described. Processing to detect as a global feature and
A signal processing method comprising a process of determining a blend ratio of the first signal and the second signal based on a detection result of the global feature.