WO2021103818A1

WO2021103818A1 - Image sensor, control method, camera assembly, and mobile terminal

Info

Publication number: WO2021103818A1
Application number: PCT/CN2020/119387
Authority: WO
Inventors: 唐城; 蓝和; 杨鑫; 李小涛; 徐锐; 张弓; 张海裕; 王文涛; 孙剑波
Original assignee: Oppo广东移动通信有限公司
Priority date: 2019-11-25
Filing date: 2020-09-30
Publication date: 2021-06-03
Also published as: CN111050041A; CN111050041B

Abstract

An image sensor (10), a control method, a camera assembly (40), and a mobile terminal (60). The image sensor (10) comprises panchromatic pixels and color pixels. The color pixels can be exposed to output at least one frame of color original image; and within the time when the color pixels are exposed once to obtain a frame of color original image, at least some of the panchromatic pixels can be exposed multiple times to obtain multiple frames of panchromatic original images, wherein the multiple frames of panchromatic original images can be used for correcting at least one frame of color original image to acquire a target image.

Description

Image sensor, control method, camera assembly and mobile terminal

Priority information

This application requests the priority and rights of the patent application with the patent application number 201911167206.2 filed with the State Intellectual Property Office of China on November 25, 2019, and the full text is incorporated herein by reference.

Technical field

This application relates to the field of imaging technology, and in particular to an image sensor, a control method, a camera assembly, and a mobile terminal.

Background technique

Motion blur is caused by the relative displacement of the imaging system and the subject. The problem of motion blur is particularly prominent in application scenarios such as long exposure time or fast motion of the subject.

Summary of the invention

The embodiments of the present application provide an image sensor, a control method, a camera assembly, and a mobile terminal.

One aspect of the present application provides an image sensor. The image sensor includes panchromatic pixels and color pixels, the color pixels having a narrower spectral response than the panchromatic pixels. The color pixels can be exposed to output at least one frame of color original image; within the time that the color pixels are exposed once to obtain one frame of color original image, at least part of the panchromatic pixels can be exposed multiple times to obtain multiple frames of full color An original image, and multiple frames of the full-color original image can be used to correct at least one frame of the color original image to obtain a target image.

In another aspect, the present application also provides a control method for the image sensor. The image sensor includes panchromatic pixels and color pixels, and the color pixels have a narrower spectral response than the panchromatic pixels. The control method includes: exposing the color pixels and outputting at least one frame of color original image; at least part of the panchromatic pixels are exposed for multiple times within the time that the color pixels are exposed once to obtain one frame of the color original image to obtain multiple Frame full-color original images; and correcting at least one frame of said color original images by using multiple frames of said full-color original images to obtain a target image.

In yet another aspect, the present application also provides a camera assembly. The camera assembly includes a lens and an image sensor. The image sensor can receive light passing through the lens. The image sensor includes panchromatic pixels and color pixels, and the color pixels have a narrower spectral response than the panchromatic pixels. The color pixels can be exposed to output at least one frame of color original image; within the time that the color pixels are exposed once to obtain one frame of color original image, at least part of the panchromatic pixels can be exposed multiple times to obtain multiple frames of full color An original image, and multiple frames of the full-color original image can be used to correct at least one frame of the color original image to obtain a target image.

In another aspect, this application also provides a mobile terminal. The mobile terminal includes a housing and a camera assembly. The camera assembly is combined with the housing. The camera assembly includes a lens and an image sensor. The image sensor can receive light passing through the lens. The image sensor includes panchromatic pixels and color pixels, and the color pixels have a narrower spectral response than the panchromatic pixels. The color pixels can be exposed to output at least one frame of color original image; within the time that the color pixels are exposed once to obtain one frame of color original image, at least part of the panchromatic pixels can be exposed multiple times to obtain multiple frames of full color An original image, and multiple frames of the full-color original image can be used to correct at least one frame of the color original image to obtain a target image.

The additional aspects and advantages of the embodiments of the present application will be partly given in the following description, and part of them will become obvious from the following description, or be understood through the practice of the present application.

Description of the drawings

The above-mentioned and/or additional aspects and advantages of the present application will become obvious and easy to understand from the description of the embodiments in conjunction with the following drawings, in which:

Figure 1 is a schematic diagram of the saturation time of different color channels;

Fig. 2 is a schematic diagram of an image sensor in an embodiment of the present application;

FIG. 3 is a schematic diagram of the connection between pixel circuits and conversion circuits of some pixels in an embodiment of the present application;

4 is a schematic diagram of the connection between the pixel circuit and the conversion circuit of some pixels in an embodiment of the present application;

FIG. 5 is a schematic diagram of the connection of pixel circuits and conversion circuits of some pixels according to another embodiment of the present application;

FIG. 6 is a schematic diagram of a reset method of a pixel circuit in an embodiment of the present application;

FIG. 7 is a schematic diagram of another pixel circuit resetting method in the embodiment of the present application;

FIG. 8 is a schematic diagram of a connection mode of a pixel array and an exposure control line in an embodiment of the present application;

FIG. 9 is a schematic diagram of a pixel array and selection line connection mode in an embodiment of the present application;

FIG. 10 is a schematic diagram of a pixel array in an embodiment of the present application;

FIG. 11 is a schematic diagram of a minimum repeating unit pixel arrangement in an embodiment of the present application;

FIG. 12 is a schematic diagram of another minimum repeating unit pixel arrangement in an embodiment of the present application;

FIG. 13 is a schematic diagram of another minimum repeating unit pixel arrangement in an embodiment of the present application;

FIG. 14 is a schematic diagram of another minimum repeating unit pixel arrangement in an embodiment of the present application;

15 is a schematic diagram of another minimum repeating unit pixel arrangement in an embodiment of the present application;

FIG. 16 is a schematic diagram of another minimum repeating unit pixel arrangement in an embodiment of the present application;

FIG. 17 is a schematic diagram of another minimum repeating unit pixel arrangement in an embodiment of the present application;

FIG. 18 is a schematic diagram of another minimum repeating unit pixel arrangement in an embodiment of the present application;

FIG. 19 is a schematic diagram of another minimum repeating unit pixel arrangement in an embodiment of the present application;

FIG. 20 is a schematic diagram of another minimum repeating unit pixel arrangement in an embodiment of the present application;

FIG. 21 is a schematic diagram of another minimum repeating unit pixel arrangement in an embodiment of the present application;

FIG. 22 is a schematic diagram of another minimum repeating unit pixel arrangement in an embodiment of the present application;

FIG. 23 is a schematic diagram of another minimum repeating unit pixel arrangement in an embodiment of the present application; FIG.

FIG. 24 is a schematic diagram of another minimum repeating unit pixel arrangement in an embodiment of the present application; FIG.

FIG. 25 is a schematic diagram of another minimum repeating unit pixel arrangement in an embodiment of the present application; FIG.

FIG. 26 is a schematic diagram of another minimum repeating unit pixel arrangement in an embodiment of the present application;

FIG. 27 is a schematic diagram of a camera assembly according to some embodiments of the present application;

FIG. 28 is a schematic flowchart of a control method of some embodiments of the present application;

FIG. 29 is a schematic diagram of the control method of the embodiment of the present application;

FIG. 30 is another principle diagram of the control method of the embodiment of the present application;

FIG. 31 is another principle diagram of the control method of the embodiment of the present application;

FIG. 32 is a schematic flowchart of a control method according to some embodiments of the present application;

FIG. 33 is a schematic flowchart of a control method according to some embodiments of the present application;

FIG. 34 is another principle diagram of the control method of the embodiment of the present application;

FIG. 35 is another principle diagram of the control method of the embodiment of the present application;

FIG. 36 is a schematic diagram of a mobile terminal according to some embodiments of the present application.

Detailed ways

The embodiments of the present application are described in detail below. Examples of the embodiments are shown in the accompanying drawings, in which the same or similar reference numerals indicate the same or similar elements or elements with the same or similar functions throughout. The following embodiments described with reference to the drawings are exemplary, and are only used to explain the embodiments of the present application, and should not be understood as limitations on the embodiments of the present application.

Please refer to FIG. 2 and FIG. 8, this application provides an image sensor 10. The image sensor 10 includes panchromatic pixels and color pixels, and the color pixels have a narrower spectral response than the panchromatic pixels. Color pixels can be exposed to output at least one frame of color original image. During the time when the color pixels are exposed once to obtain one frame of color original image, at least part of the panchromatic pixels can be exposed multiple times to obtain multiple frames of panchromatic original image. The multi-frame full-color original image can be used to correct at least one frame of the color original image to obtain the target image.

Referring to FIGS. 3 to 5, in some embodiments, at least part of the pixel circuit 110 of the panchromatic pixel W includes a photoelectric conversion element 117, an exposure control circuit 116, and a selection circuit. The exposure control circuit 116 is electrically connected to the photoelectric conversion element 117, and the exposure control circuit 116 is used to transfer the charge accumulated by the photoelectric conversion element 117 after being irradiated to the floating diffusion unit FD multiple times within the time of one exposure of the color pixel. The selection circuit is used to output analog signals corresponding to the charges in the floating diffusion unit FD multiple times. The image sensor 10 also includes a conversion circuit 16. Each panchromatic pixel in at least part of the panchromatic pixels corresponds to a plurality of conversion circuits 16, and the plurality of conversion circuits 16 are electrically connected to the selection circuit of each panchromatic pixel. An analog signal is correspondingly output to a conversion circuit 16, and each conversion circuit 16 is used to convert the analog signal into a digital signal.

Referring to FIGS. 3 to 5 and 6, in some embodiments, the pixel circuit 110 of at least part of the full-color pixel further includes a reset circuit, which is electrically connected to the floating diffusion unit FD, and the reset circuit is used for exposing the color pixels. In one time, the floating diffusion unit FD is reset every time before the exposure control circuit 116 transfers the charge.

Referring to FIGS. 3 to 5 and 7, in some embodiments, the pixel circuit 110 of at least part of the full-color pixel further includes a reset circuit, which is electrically connected to the floating diffusion unit FD, and the reset circuit is used for exposing the color pixels. In one time, before the exposure control circuit 116 transfers the charge for the first time, the floating diffusion unit FD is reset.

Referring to FIG. 4, in some embodiments, each conversion circuit 16 includes a switch 161 and an analog-to-digital conversion module 162. The switch 161 is electrically connected to the selection circuit, and the switch 161 is used for passing the analog signal from the selection circuit; The digital-to-analog conversion module 162 is electrically connected to the switch 161, and the analog-to-digital conversion module 162 is used to convert an analog signal into a digital signal.

Please refer to FIG. 5. In some embodiments, each conversion circuit 16 includes a switch 161, a capacitor 163 and an analog-to-digital conversion module 162. The switch 161 is electrically connected to the selection circuit, and the switch 161 is used for passing the analog signal from the selection circuit. The capacitor 163 is connected to the switch 161, and the capacitor 163 is used to store the analog signal transmitted through the switch 161. The digital-to-analog conversion module 162 is electrically connected to the switch 161 and the capacitor 163, and the analog-to-digital conversion module 162 is used to convert an analog signal into a digital signal.

Referring to FIG. 11, in some embodiments, the panchromatic pixels W and the color pixels are arranged in a two-dimensional pixel array. The two-dimensional pixel array includes a plurality of minimum repeating units. In the minimum repeating unit, the panchromatic pixels W are arranged In the first diagonal direction D1, the color pixels are arranged in the second diagonal direction D2, and the first diagonal direction D1 is different from the second diagonal direction D2.

Please refer to FIG. 2, FIG. 8 and FIG. 28. The present application provides a control method for the image sensor 10. The image sensor 10 includes panchromatic pixels and color pixels, and the color pixels have a narrower spectral response than the panchromatic pixels. Control methods include:

01: Control the exposure of color pixels and output at least one frame of color original image;

02: Control at least part of the panchromatic pixels to be exposed multiple times within the time that the color pixels are exposed once to obtain one frame of color original image to obtain multiple frames of panchromatic original image; and

03: Use multiple frames of full-color original images to correct at least one frame of color original images to obtain the target image.

Please refer to FIG. 28 and FIG. 32. In some embodiments, step 03: correcting at least one frame of color original image by using multiple frames of panchromatic original image to obtain the target image includes:

031: Calculate correction data based on multi-frame panchromatic original images; and

032: Use the correction data to correct at least one frame of the original color image to obtain the target image.

Referring to FIG. 33, in some embodiments, when only part of the panchromatic pixels are exposed for multiple times within the time that the color pixels are exposed once to obtain one frame of color original image to obtain multiple frames of panchromatic original image, the control method is also include:

04: Expose the remaining panchromatic pixels to output the panchromatic original image;

05: Correct the brightness of the target image based on the panchromatic original image output by the remaining panchromatic pixels.

Please refer to FIG. 2, FIG. 8 and FIG. 27. The present application also provides a camera assembly 40. The camera assembly 40 includes a lens 30 and an image sensor 10. The image sensor 10 can receive light passing through the lens 30. The image sensor 10 includes panchromatic pixels and color pixels, and the color pixels have a narrower spectral response than the panchromatic pixels. Color pixels can be exposed to output at least one frame of color original image. During the time when the color pixels are exposed once to obtain one frame of color original image, at least part of the panchromatic pixels can be exposed multiple times to obtain multiple frames of panchromatic original image. The multi-frame full-color original image can be used to correct at least one frame of the color original image to obtain the target image.

Please refer to FIGS. 27 and 28. In some embodiments, the camera assembly 40 further includes a processing chip 20 for correcting at least one frame of the color original image by using the multi-frame full-color original image to obtain the target image.

Please refer to FIG. 27 and FIG. 32. In some embodiments, the processing chip 20 is further configured to calculate correction data based on multiple frames of full-color original images; and use the correction data to correct at least one frame of color original images to obtain a target image.

Please refer to Figure 27 and Figure 33, when only part of the panchromatic pixels are exposed to multiple frames of panchromatic original images within the time that the color pixels are exposed once to obtain one frame of color original image, the remaining panchromatic pixels are exposed to Output full-color original image. The processing chip 20 is also used to correct the brightness of the target image according to the panchromatic original image output by the remaining panchromatic pixels.

Referring to FIG. 2, FIG. 8, FIG. 27, and FIG. 36, the present application also provides a mobile terminal 60. The mobile terminal 60 includes a housing 50 and a camera assembly 40. The camera assembly 40 is combined with the housing 50. The camera assembly 40 includes a lens 30 and an image sensor 10. The image sensor 10 can receive light passing through the lens 30. The image sensor 10 includes panchromatic pixels and color pixels, and the color pixels have a narrower spectral response than the panchromatic pixels. Color pixels can be exposed to output at least one frame of color original image. During the time when the color pixels are exposed once to obtain one frame of color original image, at least part of the panchromatic pixels can be exposed multiple times to obtain multiple frames of panchromatic original image. The multi-frame full-color original image can be used to correct at least one frame of the color original image to obtain the target image.

The application will be further explained below in conjunction with the drawings.

Motion blur is caused by the relative displacement between the camera assembly and the subject. In order to eliminate motion blur, two independent cameras are usually used in related technologies. One camera acquires one frame of image, and the other camera acquires multiple frames within the time that the camera acquires one frame of image, and then corrects the image according to the multiple frames. One frame of image. However, the hardware system of this method of removing motion blur is more complicated and costly, and operations such as calibration and compensation are required in the back-end processing process, and the algorithm complexity is relatively high.

In addition, in an image sensor containing pixels of multiple colors, the pixels of different colors receive different exposures per unit time (that is, the sensitivity is different, and the sensitivity of pixels with more exposure per unit time is higher). After the color is saturated, some colors have not yet been exposed to the ideal state.

In Figure 1, RGBW (red, green, blue, full color) is taken as an example. See Figure 1. In Figure 1, the horizontal axis is the exposure time, the vertical axis is the exposure, Q is the saturated exposure, LW is the exposure curve of the panchromatic pixel W, LG is the exposure curve of the green pixel G, and LR is the red pixel R The exposure curve of LB is the exposure curve of the blue pixel.

It can be seen from FIG. 1 that the slope of the exposure curve LW of the panchromatic pixel W is the largest, that is, the panchromatic pixel W can obtain more exposure per unit time, and reach saturation at t1. The slope of the exposure curve LG of the green pixel G is the second, and the green pixel is saturated at time t2. The slope of the exposure curve LR of the red pixel R is again the same, and the red pixel is saturated at time t3. The slope of the exposure curve LB of the blue pixel B is the smallest, and the blue pixel is saturated at t4. At t1, the panchromatic pixel W has been saturated, and the exposure of the three pixels R, G, and B has not yet reached the ideal state.

The present application provides an image sensor 10 (shown in FIG. 2). The image sensor 10 is simultaneously arranged with panchromatic pixels with higher sensitivity and color pixels with lower sensitivity than panchromatic pixels, so that at least part of the panchromatic pixels with higher sensitivity can be exposed at a higher frame rate for a period of time. Multi-frame image, the remaining pixels are exposed at a lower frame rate within this period of time to obtain at least one frame of image. The multi-frame image obtained after high frame rate exposure can be used to correct the low frame rate exposure to obtain the image, so as to achieve the low frame rate exposure. Reduce the motion blur of the image obtained by the rate exposure. The image sensor 10 according to the embodiment of the present application can eliminate the motion blur of the image without setting multiple image sensors 10, and the complexity of the hardware system is low. In addition, the motion-blurred image and the corrected image are acquired by the same image sensor 10, no compensation and calibration are needed in the subsequent processing, and the algorithm complexity is also low. In addition, the corrective image is obtained by using high-sensitivity panchromatic pixels. The image has a high signal-to-noise ratio. Using an image with a high signal-to-noise ratio to correct a motion blur image can improve the elimination of motion blur. effect.

Next, first, the basic structure of the image sensor 10 will be introduced. Please refer to FIG. 2, which is a schematic diagram of the image sensor 10 in an embodiment of the present application. The image sensor 10 includes a pixel array 11, a vertical driving unit 12, a control unit 13, a column processing unit 14 and a horizontal driving unit 15.

For example, the image sensor 10 may use a complementary metal oxide semiconductor (CMOS, Complementary Metal Oxide Semiconductor) photosensitive element or a charge-coupled device (CCD, Charge-coupled Device) photosensitive element.

For example, the pixel array 11 includes a plurality of pixels (not shown in the figure) arranged two-dimensionally in an array, and each pixel includes a photoelectric conversion element 117 (shown in FIG. 3). Each pixel converts light into electric charge according to the intensity of the light incident on it.

For example, the vertical driving unit 12 includes a shift register and an address decoder. The vertical drive unit 12 includes readout scanning and reset scanning functions. Readout scanning refers to scanning the pixels of each row and each column, and reading signals from these pixels. For example, the signal output by the pixel in the pixel row that is selected and scanned is transmitted to the column processing unit 14. The reset scan is used to reset the charge, and the photocharge of the photoelectric conversion element 117 is discarded, so that the accumulation of new photocharge can be started.

For example, the signal processing performed by the column processing unit 14 is correlated double sampling (CDS) processing. In the CDS process, the reset level and signal level output from each pixel in the selected pixel row are taken out, and the level difference is calculated. Thus, the signals of the pixels in a row are obtained. The column processing unit 14 may have an analog-to-digital (A/D) conversion function for converting analog pixel signals into a digital format.

For example, the horizontal driving unit 15 includes a shift register and an address decoder. The horizontal driving unit 15 scans the pixel array 11 according to a predetermined rule. Through the selection scanning operation performed by the horizontal driving unit 15, each pixel column is processed by the column processing unit 14, and is output.

For example, the control unit 13 configures timing signals according to the operation mode, and uses various timing signals to control the vertical driving unit 13, the column processing unit 14, and the horizontal driving unit 15 to work together.

The image sensor 10 also includes a filter (not shown) arranged on the pixel array 11. The spectral response of each pixel in the pixel array 11 (that is, the color of light that the pixel can receive) is determined by the color of the filter corresponding to the pixel. The color pixels and panchromatic pixels in this application refer to pixels that can respond to light whose color is the same as the color of the corresponding filter.

FIG. 3 is a schematic diagram of the connection between the pixel circuit 110 of the color pixel and the conversion circuit 16 according to an embodiment of the present application. FIG. 4 is a schematic diagram of the connection between the pixel circuit 110 of at least part of the full-color pixel and the conversion circuit 16 according to an embodiment of the present application. Wherein, when all panchromatic pixels can obtain multiple frames of panchromatic original images within the time when the color pixels obtain one frame of color original images, FIG. 4 is an embodiment of the pixel circuit 110 of all panchromatic pixels connected to the conversion circuit 16 FIG. 3 is a schematic diagram of the connection between the pixel circuits 110 and the conversion circuit 16 of all color pixels in an embodiment. When only part of the panchromatic pixels can obtain multiple frames of panchromatic original images within the time that the color pixels obtain one frame of the original color image, FIG. 4 shows the connection between the pixel circuit 110 of the partial panchromatic pixels and the conversion circuit 16 of an embodiment. For a schematic diagram, FIG. 3 is a schematic diagram of the pixel circuits 110 of all color pixels connected to the conversion circuit 16 and the pixel circuits 110 of the remaining full-color pixels are connected to the conversion circuit 16 according to an embodiment.

As shown in FIGS. 3 and 4, the pixel circuit 110 of each pixel includes a photoelectric conversion element 117 (for example, a photodiode PD), an exposure control circuit 116 (for example, a transfer transistor 112), and a reset circuit (for example, a reset transistor 113). ), an amplifying circuit (for example, the amplifying transistor 114), and a selection circuit (for example, the selection transistor 115). In the embodiment of the present application, the transfer transistor 112, the reset transistor 113, the amplifying transistor 114, and the selection transistor 115 are, for example, MOS transistors, but are not limited thereto. The pixel circuit 110 of each pixel is connected to the column processing unit 14 of FIG. 2. A conversion circuit 16 is provided in the column processing unit 14.

For example, referring to FIGS. 2, 3, and 4, the gate TG of the transfer transistor 112 is connected to the vertical driving unit 12 through an exposure control line (shown in FIG. 8); the gate RG of the reset transistor 113 is connected through a reset control line (not shown in the figure). (Shown) is connected to the vertical driving unit 12; the gate SEL of the selection transistor 115 is connected to the vertical driving unit 12 through a selection line (shown in FIG. 9). The exposure control circuit 116 (for example, the transfer transistor 112) in each pixel circuit 110 is electrically connected to the photoelectric conversion element 117 for transferring the electric potential accumulated by the photoelectric conversion element 117 after being irradiated. For example, the photoelectric conversion element 117 includes a photodiode PD, and the anode of the photodiode PD is connected to the ground, for example. The photodiode PD converts the received light into electric charge. The cathode of the photodiode PD is connected to the floating diffusion unit FD via the exposure control circuit 116 (for example, the transfer transistor 112). The floating diffusion unit FD is connected to the gate of the amplifying transistor 114 and the source of the reset transistor 113.

For example, the exposure control circuit 116 is the transfer transistor 112, and the control terminal TG of the exposure control circuit 116 is the gate of the transfer transistor 112. When a pulse of an active level (for example, VPIX level) is transmitted to the gate of the transfer transistor 112 through the exposure control line, the transfer transistor 112 is turned on. The transfer transistor 112 transfers the charge photoelectrically converted by the photodiode PD to the floating diffusion unit FD. Each time the transfer transistor 112 transfers a charge to the floating diffusion unit FD, it means that one exposure is completed.

For example, the drain of the reset transistor 113 is connected to the pixel power supply VPIX. The source of the reset transistor 113 is connected to the floating diffusion unit FD. Before the charge is transferred from the photodiode PD to the floating diffusion unit FD, a pulse of an effective reset level is transmitted to the gate of the reset transistor 113 via the reset control line, and the reset transistor 113 is turned on. The reset transistor 113 resets the floating diffusion unit FD to the pixel power supply VPIX.

For example, the gate of the amplifying transistor 114 is connected to the floating diffusion unit FD. The drain of the amplifying transistor 114 is connected to the pixel power supply VPIX. After the floating diffusion unit FD is reset by the reset transistor 113, the amplifying transistor 114 outputs the reset level through the output terminal OUT via the selection transistor 115. After the charge of the photodiode PD is transferred by the transfer transistor 112, the amplifying transistor 114 outputs a signal level via the selection transistor 115.

For example, the drain of the selection transistor 115 is connected to the source of the amplifying transistor 114. The source of the selection transistor 115 is connected to the column processing unit 14 in FIG. 2. When the pulse of the active level is transmitted to the gate of the selection transistor 115 through the selection line, the selection transistor 115 is turned on. The signal output by the amplifying transistor 114 is transmitted to the conversion circuit 16 provided in the column processing unit 14 through the selection transistor 115.

For example, the conversion circuit 16 includes a switch 161 and an analog-to-digital conversion module 162. One end of the switch 161 is electrically connected to the selection circuit, that is, is electrically connected to the source of the selection transistor 115. The other end of the switch 161 is electrically connected to the analog-to-digital conversion module 162. The switch 161 is used for passing the signal level (the signal level is an analog signal) from the selection circuit. The analog-to-digital conversion module 162 is used to convert the analog signal into a digital signal.

The pixel circuit 110 shown in FIG. 3 is electrically connected to a conversion circuit 16. The working process of the circuit shown in FIG. 3 is: the reset transistor 113 resets the floating diffusion unit FD, and the photoelectric conversion element 117 receives light and converts the received light into electric charge. When the transfer transistor 112 receives the pulse of the effective level transmitted from the exposure control line, the transfer transistor 112 transfers the charge accumulated after the photoelectric conversion element 117 is irradiated to the floating diffusion unit FD. The amplifying transistor 114 reads and amplifies the analog signal corresponding to the charge in the floating diffusion unit FD, and then outputs the analog signal to the selection transistor 115. The selection transistor 115 outputs an analog signal to the conversion circuit 16, and the conversion circuit 16 converts the analog signal into a digital signal. In this way, multiple digital signals output by multiple color pixels form a color original image.

The pixel circuit 110 shown in FIG. 4 is electrically connected to a plurality of conversion circuits 16. The working process of the circuit shown in FIG. 4 is: the photoelectric conversion element 117 of FIG. 3 receives light and converts the received light into electric charge, and the transfer transistor 112 of FIG. 3 transfers the charge accumulated after the photoelectric conversion element 117 is irradiated to During this period of time (hereinafter referred to as the first exposure time) of the floating diffusion unit FD, the photoelectric conversion element 117 of FIG. 4 continuously receives light and converts the received light into electric charge, and the transfer transistor 112 of FIG. 4 transfers the photoelectric conversion element multiple times 117 The accumulated electric charge after light irradiation goes to the floating diffusion unit FD. Every time the transfer transistor 112 transfers the charge, the amplifying transistor 114 will read and amplify the analog signal corresponding to the charge in the floating diffusion unit FD, and output the acquired analog signal to the selection transistor 115. The selection transistor 115 then outputs the analog signal to a corresponding conversion circuit 16. During the first exposure time, when the effective level pulse is transmitted to the gate of the selection transistor 115 through the selection line for the Nth time, the selection transistor 115 is turned on, the switch 161 of the Nth conversion circuit 16 is closed, and the remaining conversion circuits The switch 161 of 16 is turned off. The analog signal output by the amplifying transistor 114 is transmitted to the N-th conversion circuit 16 through the selection transistor 115, and the analog-to-digital conversion module 162 of the N-th conversion circuit 16 converts the analog signal into a digital signal, where N is a natural number greater than or equal to 1. For example, during the first exposure time, when the pulse of the effective level is transmitted to the gate of the selection transistor 115 through the selection line for the first time, the selection transistor 115 is turned on, the switch 161 in the first selection circuit is closed, and the rest The switch 161 in the selection circuit is turned off, the analog signal output by the amplifying transistor 114 is transmitted to the first conversion circuit 16 through the selection transistor 115, and the analog-to-digital conversion module 162 of the first conversion circuit 16 converts the analog signal into a digital signal. For another example, during the first exposure time, when the pulse of the effective level is transmitted to the gate of the selection transistor 115 through the selection line for the sixth time, the selection transistor 115 is turned on, and the switch 161 in the sixth conversion circuit 16 is closed. , The switch 161 in the remaining conversion circuit 16 is turned off, the analog signal output by the amplifying transistor 114 is transmitted to the sixth conversion circuit 16 through the selection transistor 115, and the analog-to-digital conversion module 162 of the sixth conversion circuit 16 converts the analog signal to digital signal.

The conversion circuit 16 of FIG. 4 will perform an analog-to-digital conversion operation immediately after receiving the analog signal transmitted from the selection transistor 115.

FIG. 5 is a schematic diagram of the connection between the pixel circuit 110 of at least part of the full-color pixel and the conversion circuit 16 according to another embodiment of the present application. Wherein, when all panchromatic pixels can obtain multiple frames of panchromatic original images within the time that the color pixels obtain one frame of color original images, FIG. 5 is an embodiment of the pixel circuit 110 of all panchromatic pixels connected to the conversion circuit 16 FIG. 3 is a schematic diagram of the connection between the pixel circuits 110 and the conversion circuit 16 of all color pixels in an embodiment. When only part of the panchromatic pixels can obtain multiple frames of panchromatic original images within the time that the color pixels obtain one frame of the original color image, FIG. 5 shows the connection between the pixel circuit 110 of the partial panchromatic pixels and the conversion circuit 16 of an embodiment. Schematic diagram, FIG. 3 is a schematic diagram of the connection between the pixel circuit 110 of the remaining pixels (including the remaining panchromatic pixels and all color pixels) and the conversion circuit 16 in the image sensor 10 of an embodiment.

The difference between the conversion circuit 16 shown in FIG. 5 and the conversion circuit 16 shown in FIG. 4 is that the conversion circuit 16 of FIG. 4 only includes a switch 161 and an analog-to-digital conversion module 162, while the conversion circuit 16 of FIG. 5 includes a switch 161, a capacitor 163 and Analog-to-digital conversion module 162. In the conversion circuit 16 of FIG. 5, one end of the switch 161 is electrically connected to the source of the selection transistor 115, the other end of the switch 161 is electrically connected to one end of the capacitor 163, and is electrically connected to the analog-to-digital conversion module 162. One end of the capacitor 163 is connected to the analog-to-digital conversion module 162, and the other end of the capacitor 163 is grounded. The switch 161 is used for passing the analog signal from the selection circuit. The capacitor 163 is used to store the analog signal transmitted through the switch 161. The analog-to-digital conversion module 162 is used to convert analog signals into digital signals. There are multiple conversion circuits 16 shown in FIG. 5. The working process of the circuit shown in FIG. 5 is roughly the same as the working process of the circuit shown in FIG. 4, except that the conversion circuit 16 of FIG. 5 does not need to perform analog-to-digital conversion immediately after receiving the analog signal transmitted by the selection transistor 115. Operation, but first store the analog signal in the capacitor 163. When the panchromatic pixel completes N exposures within the first exposure time, and the capacitors 163 of the N conversion circuits 16 all store analog signals, the N conversion circuits 16 The analog-to-digital conversion module 162 in the corresponding capacitor 163 simultaneously reads the analog signal stored in the corresponding capacitor 163, and the analog-to-digital conversion module 162 in the N conversion circuits 16 simultaneously performs the operation of analog-to-digital conversion.

In an embodiment of the present application, within the time of one exposure of the color pixels (that is, the first exposure time), at least part of the floating diffusion unit FD of the full-color pixel resets the floating diffusion unit before the exposure control circuit 116 transfers the charge. FD. Specifically, referring to FIG. 2, FIG. 3, and FIG. 6, the first exposure time of the photoelectric conversion element 117 in FIG. 3 is T, and the photoelectric conversion element in FIG. 4 starts to be exposed at time t0. During the first exposure time T, the exposure control circuit 116 of FIG. 4 sequentially transfers the charge accumulated by the photoelectric conversion element 117 of FIG. 4 at t1, t2, t3, t4, and t5, and the reset transistor 113 of FIG. 4 controls each exposure The circuit 116 resets the floating diffusion unit FD before the charge is transferred. In this way, the charge corresponding to the analog signal received by the Nth conversion circuit 16 is the time when the photoelectric conversion element 117 transfers the charge at the N-1th time to the Nth transfer by the exposure control circuit. The charge accumulated during the time of the charge. For example, the charge corresponding to the analog signal received by the first conversion circuit 16 is the charge accumulated by the photoelectric conversion element 117 from time t0 to time t1, and the charge corresponding to the analog signal received by the second conversion circuit 16 is the photoelectric conversion element 117 The accumulated charge from time t1 to time t2, etc., and so on. In this reset mode, the charges corresponding to the analog signals received by the multiple conversion circuits 16 are independent of each other. In other words, when at least part of the panchromatic pixels perform multiple exposures within the first exposure time, the time for each exposure is called the second exposure time, and the multiple second exposure times do not overlap.

In an embodiment of the present application, within the time of one color pixel exposure (ie, the first exposure time), at least part of the floating diffusion unit FD of the full color pixel resets the floating diffusion before the exposure control circuit 116 transfers the charge for the first time. Unit FD. Specifically, referring to FIG. 2, FIG. 3, and FIG. 7, the first exposure time of the photoelectric conversion element 117 in FIG. 3 is T, and the photoelectric conversion element in FIG. 4 starts to be exposed at time t0. In the first exposure time T, the exposure control circuit 116 of FIG. 4 sequentially transfers the charge accumulated by the photoelectric conversion element 117 of FIG. 4 at t1, t2, t3, t4, and t5, and the reset transistor 113 of FIG. 4 is used in the exposure control circuit 116 The floating diffusion unit FD is reset before the first charge transfer (that is, time t1), so that the charge corresponding to the analog signal received by the Nth conversion circuit 16 is from the time the photoelectric conversion element 117 starts to expose to the Nth time the exposure control circuit 116 The accumulated charge during the time the charge was transferred. For example, the charge corresponding to the analog signal received by the first conversion circuit 16 is the charge accumulated by the photoelectric conversion element 117 from time t0 to time t1, and the charge corresponding to the analog signal received by the second conversion circuit 16 is the photoelectric conversion element 117 The charge from t0 to t2, and so on. In this reset mode, the charges corresponding to the analog signals received by the multiple conversion circuits 16 are not independent of each other. In other words, when at least part of the panchromatic pixels perform multiple exposures within the first exposure time, the time for each exposure is called the second exposure time, and multiple second exposure times are overlapped. This reset method can avoid the problem that the second exposure time is short and the floating diffusion unit FD is too late to reset.

FIG. 8 is a schematic diagram of the connection mode of the pixel array 11 (shown in FIG. 2), the conversion circuit 16 (shown in FIG. 3 and FIG. 5), and the exposure control line according to an embodiment of the present application, in which all panchromatic pixels are connected to multiple Conversion circuit 16. The pixel array 11 is a two-dimensional pixel array. The two-dimensional pixel array includes a plurality of panchromatic pixels and a plurality of color pixels, wherein the color pixels have a narrower spectral response than the panchromatic pixels. The arrangement of pixels in the pixel array 11 is as follows:

It should be noted that, for the convenience of illustration, only some pixels in the pixel array 11 are shown in FIG. 6, and other surrounding pixels and connections are replaced by ellipsis "...".

As shown in Figure 8,

pixels

1101, 1103, 1106, 1108, 1111, 1113, 1116, and 1118 are full-color pixels W,

pixels

1102, 1105 are pixels A of the first color (for example, red pixels R), and

pixels

1104, 1107 , 1112 and 1115 are the second color pixel B (for example, the green pixel G), and the

pixels

1114 and 1117 are the third color pixel C (for example, the blue pixel Bu). All panchromatic pixels W, namely

pixels

1101, 1103, 1106, 1108, 1111, 1113, 1116, and 1118 adopt the circuit structure shown in Figure 5; all color pixels, namely

pixels

1102, 1105, 1104, 1107, 1112, 1115, Both 1114 and 1117 adopt the image circuit structure shown in Figure 3.

It can be seen from Figure 8 that the control terminal TG of the exposure control circuit in the panchromatic pixel W (

pixels

1101, 1103, 1106, and 1108) is connected to a first exposure control line TX1, and the panchromatic pixel W (1111, 1113, 1116, And 1118) the control terminal TG of the exposure control circuit is connected to another first exposure control line TX1; the control terminal TG of the exposure control circuit in the first color pixel A (pixels 1102 and 1105), the second color pixel B (pixel 1104) , 1107), the control terminal TG of the exposure control circuit is connected to a second exposure control line TX2, the control terminal TG of the exposure control circuit in the second color pixel B (pixels 1112, 1115), the third color pixel C (pixel 1114, In 1117), the control terminal TG of the exposure control circuit is connected to another second exposure control line TX2. Each first exposure control line TX1 can control the exposure of the panchromatic pixels through the first exposure control signal; each second exposure control line TX2 can control the exposure of the color pixels through the second exposure control signal. In order to realize that within the time when the color pixels are exposed to obtain one frame of color original image at a time, the panchromatic pixels can be exposed for multiple times to obtain multiple frames of panchromatic original images.

Please refer to FIG. 2 and FIG. 8, the first exposure control line TX1 and the second exposure control line TX2 are both connected to the vertical driving unit 12 in FIG. 2. The vertical drive unit 12 transmits the first exposure control signal to the control terminal TG of the exposure control circuit connected to the first exposure control line TX1, and transmits the second exposure control signal to the exposure control circuit connected to the second exposure control line TX2 The control terminal TG.

It can be understood that since there are multiple pixel row groups in the pixel array 11, the vertical driving unit 12 connects multiple first exposure control lines TX1 and multiple second exposure control lines TX2. The plurality of first exposure control lines TX1 and the plurality of second exposure control lines TX2 correspond to corresponding pixel row groups.

For example, the first first exposure control line TX1 corresponds to the panchromatic pixels in the first and second rows; the second first exposure control line TX1 corresponds to the panchromatic pixels in the third and fourth rows, so By analogy, the third first exposure control line TX1 corresponds to the panchromatic pixels in the fifth and sixth rows; the fourth first exposure control line TX1 corresponds to the panchromatic pixels in the seventh and eighth rows, and then down The corresponding relationship between the first exposure control line TX1 and the panchromatic pixels further downstream will not be repeated. The signal timings transmitted by different first exposure control lines TX1 are also different, and the signal timings are configured by the vertical driving unit 12.

For example, the first second exposure control line TX2 corresponds to the color pixels in the first and second rows; the second second exposure control line TX2 corresponds to the color pixels in the third and fourth rows, and so on, The third second exposure control line TX2 corresponds to the color pixels in the fifth and sixth rows; the fourth second exposure control line TX2 corresponds to the color pixels in the seventh and eighth rows, and then the second exposure The corresponding relationship between the control line TX2 and the color pixels further downstream will not be repeated. The signal timings transmitted by the different second exposure control lines TX2 are also different, and the signal timings are also configured by the vertical driving unit 12.

FIG. 9 is a schematic diagram of the pixel array 11 (shown in FIG. 2), the conversion circuit 16 (shown in FIG. 3 and FIG. 5), and the connection mode of the selection line according to an embodiment of the present application. The arrangement of the pixel array 11 in FIG. 9 is similar to the arrangement of the pixel array 11 in FIG. 8, and will not be repeated here. It can be seen from Figure 9 that the selection terminal SEL of the selection circuit in the panchromatic pixel W (

pixels

1101, 1103, 1106, and 1108) is connected to a first selection line SEL1, and the panchromatic pixel W (1111, 1113, 1116, and 1118) The selection terminal SEL of the selection circuit in) is connected to another first selection line SEL1; the selection terminal SEL of the selection circuit in the first color pixel A (pixels 1102 and 1105) and the second color pixel B (pixels 1104, 1107) are selected The selection terminal SEL of the circuit is connected to a second selection line SEL2, the selection terminal SEL of the selection circuit in the second color pixel B (pixels 1112, 1115), and the selection terminal of the selection circuit in the third color pixel C (pixels 1114, 1117) SEL is connected to another second selection line SEL2. Each first selection line SEL1 can control the full-color pixel selection circuit through the first selection signal to output an analog signal to the conversion circuit 16 to obtain a multi-frame full-color original image; each second selection line SEL2 can control color pixels (for example, The selection circuits of the first color pixel A and the second color pixel B, the second color pixel B and the third color pixel C) output analog signals to the conversion circuit 16 to obtain a color original image. In this way, independent control of the output analog signals of panchromatic pixels and color pixels can be realized, panchromatic pixels can output multiple frames of panchromatic original images, and color pixels can output color original images.

FIG. 10 is a schematic diagram of the connection between the pixel array 11 (shown in FIG. 2) and the conversion circuit 16 (shown in FIG. 3 and FIG. 5) according to an embodiment of the present application. Among them, the pixel circuit 110 of only part of the panchromatic pixels (that is, the panchromatic pixel W') is electrically connected to a plurality of conversion circuits 16, and the pixel circuit 110 of the remaining panchromatic pixels (that is, the panchromatic pixel W) is only connected to one conversion circuit 16 Electric connection. At this time, the panchromatic pixel W is only exposed once in the first exposure time, and the analog signal output by the panchromatic pixel W once exposure is performed by the conversion circuit 16 to perform analog-to-digital conversion; the panchromatic pixel W'can be exposed during the first exposure time. After multiple exposures, the multiple analog signals outputted by the full-color pixel W′ after multiple exposures are respectively performed by multiple conversion circuits 16 to perform analog-to-digital conversion. The panchromatic pixels W and panchromatic pixels W'shown in FIG. 10 are alternately arranged in a row. At this time, the panchromatic pixels W in the same row are controlled by a first exposure control line, and the panchromatic pixels W'in the same row are controlled by another A first exposure control line control. In other embodiments, the panchromatic pixels W and panchromatic pixels W'can also be arranged alternately in the column. At this time, the panchromatic pixels W in the same column are controlled by a first exposure control line, and the panchromatic pixels W in the same column are controlled by a first exposure control line. 'Controlled by another first exposure control line.

FIGS. 11 to 26 show examples of pixel arrangement in various image sensors 10 (shown in FIG. 2). Among them, the pixel circuit 110 (shown in FIG. 4 or FIG. 5) of each full-color pixel is connected to a plurality of conversion circuits 16 (shown in FIG. 4 or FIG. 5). Referring to FIGS. 2 and 8 to 24, the image sensor 10 includes a plurality of color pixels (for example, a plurality of first color pixels A, a plurality of second color pixels B, and a plurality of third color pixels C) and a plurality of color pixels. A two-dimensional pixel array composed of color pixels W (that is, the pixel array 11 shown in FIG. 2). Among them, color pixels have a narrower spectral response than panchromatic pixels. The response spectrum of the color pixel is, for example, a part of the W response spectrum of the panchromatic pixel. The two-dimensional pixel array includes the smallest repeating unit (Figures 11 to 26 show examples of the smallest repeating unit of pixels in various image sensors 10. The two-dimensional pixel array is composed of multiple smallest repeating units, and the smallest repeating unit is in rows and columns. Copy and arrange on the top. In the smallest repeating unit, the panchromatic pixel W is set in the first diagonal direction D1, the color pixel is set in the second diagonal direction D2, the first diagonal direction D1 and the second diagonal The direction D2 is different. The first exposure time of the at least two panchromatic pixels adjacent in the first diagonal direction D1 is controlled by the first exposure signal, and the second exposure time of the at least two color pixels adjacent in the second diagonal direction D2 The exposure time is controlled by the second exposure signal, so as to realize the independent control of the panchromatic pixel exposure time and the color pixel exposure time. The output of at least two panchromatic pixels adjacent in the first diagonal direction D1 is controlled by the first selection signal, The output of at least two color pixels adjacent in the second diagonal direction D2 is controlled by the second selection signal, thereby achieving independent control of the full-color pixel output analog signal and the color pixel output analog signal. Each minimum repeating unit includes multiple Sub-units, each sub-unit includes multiple single-color pixels (such as multiple first-color pixels A, multiple second-color pixels B, or multiple third-color pixels C) and multiple full-color pixels W. For example, please combine In Figure 8, the pixels 1101-1108 and the pixels 1111-1118 form a minimum repeating unit, where the

pixels

1101, 1103, 1106, 1108, 1111, 1113, 1116, and 1118 are panchromatic pixels, and the

pixels

1102, 1104, 1105, 1107, 1112, 1114, 1115, and 1117 are color pixels.

Pixels

1101, 1102, 1105, and 1106 form a sub-unit, where

pixels

1101, 1106 are full-color pixels, and

pixels

1102, 1105 are single-color pixels (for example, the first color pixel A);

Pixels

1103, 1104, 1107, and 1108 form a sub-unit, where

pixels

1103, 1108 are full-color pixels, and

pixels

1104, 1107 are single-color pixels (for example, second-color pixels B);

pixels

1111, 1112, 1115 and 1116 form a sub-unit, where pixels 1111, 1116 are full-color pixels, and

pixels

1112, 1115 are single-color pixels (for example, the second color pixel B);

pixels

1113, 1114, 1117, and 1118 form a sub-unit, Among them, the

pixels

1113 and 1118 are full-color pixels, and the

pixels

1114 and 1117 are single-color pixels (for example, the third color pixel C).

For example, the number of pixels in the rows and columns of the minimum repeating unit is equal. For example, the minimum repeating unit includes, but is not limited to, a minimum repeating unit of 4 rows and 4 columns, 6 rows and 6 columns, 8 rows and 8 columns, and 10 rows and 10 columns. For example, the number of pixels in the rows and columns of sub-units in the smallest repeating unit is equal. For example, subunits include, but are not limited to, subunits with 2 rows and 2 columns, 3 rows and 3 columns, 4 rows and 4 columns, and 5 rows and 5 columns. This setting helps to balance the resolution and color performance of the image in the row and column directions, and improve the display effect.

For example, FIG. 11 is a schematic diagram of a minimum repeating unit 1181 pixel arrangement in the embodiment of the present application; the minimum repeating unit has 4 rows, 4 columns and 16 pixels, and the subunits have 2 rows, 2 columns and 4 pixels. The arrangement is as follows:

W represents a full-color pixel; A represents a first color pixel among multiple color pixels; B represents a second color pixel among multiple color pixels; C represents a third color pixel among multiple color pixels.

For example, as shown in FIG. 11, the panchromatic pixels W are arranged in the first diagonal direction D1 (that is, the direction connecting the upper left corner and the lower right corner in FIG. 11), and the color pixels are arranged in the second diagonal direction D2 (for example, as shown in FIG. The direction connecting the lower left corner and the upper right corner in 11), the first diagonal direction D1 is different from the second diagonal direction D2. For example, the first diagonal line and the second diagonal line are perpendicular. The first exposure time of at least two panchromatic pixels W adjacent in the first diagonal direction D1 (for example, two panchromatic pixels in the first row, first column and second row and second column from the upper left) is determined by Controlled by the first exposure signal, at least two color pixels adjacent in the second diagonal direction D2 (for example, two color pixels B in the fourth row, first column and third row and second column from the upper left) The second exposure time is controlled by the second exposure signal. The output of at least two full-color pixels W adjacent in the first diagonal direction D1 is controlled by a first selection signal, and the output of at least two color pixels adjacent in the second diagonal direction D2 is controlled by a second selection signal.

It should be noted that the first diagonal direction D1 and the second diagonal direction D2 are not limited to the diagonal, but also include directions parallel to the diagonal. For example, in FIG. 8, the

panchromatic pixels

1101, 1106, 1113 and 1118 are arranged in the first diagonal direction D1, the

panchromatic pixels

1103 and 1108 are also arranged in the first diagonal direction D1, and the panchromatic pixels 1111 and 1116 are also arranged in the first diagonal direction D1; The

color pixels

1104, 1107, 1112, and 1115 are arranged in the second diagonal direction D2, the

first color pixels

1102 and 1105 are also arranged in the second diagonal direction D2, and the

third color pixels

1114 and 1117 are also arranged in the second diagonal direction. The diagonal direction D2, the explanation of the first diagonal direction D1 and the second diagonal direction D2 in FIGS. 12 to 26 below is the same as here. The "direction" here is not a single direction, but can be understood as the concept of a "straight line" indicating the arrangement, and there can be two-way directions at both ends of the straight line.

It should be understood that the orientation or positional relationship indicated by the terms "upper", "lower", "left", "right", etc. here and below is based on the orientation or positional relationship shown in the drawings, and is only for convenience Describe this application and simplify the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as a limitation of the application.

For example, as shown in FIG. 11, the panchromatic pixels in the first row and the second row are connected together by the first exposure control line TX1 in the shape of "W" to realize individual control of the exposure time of the panchromatic pixels. The color pixels (A and B) in the first row and the second row are connected together by a second exposure control line TX2 in a "W" shape to realize individual control of the exposure time of the color pixels. The panchromatic pixels in the third row and the fourth row are connected together by the first exposure control line TX1 in the shape of "W" to realize the individual control of the exposure time of the panchromatic pixels. The color pixels (B and C) in the third row and the fourth row are connected together by a second exposure control line TX2 in a "W" shape to realize individual control of the exposure time of the color pixels. For example, the first exposure signal is transmitted via the first exposure control line TX1, and the second exposure signal is transmitted via the second exposure control line TX2. For example, the first exposure control line TX1 is in the shape of "W" and is electrically connected to the control terminal of the exposure control circuit in two adjacent rows of panchromatic pixels; the second exposure control line TX2 is in the shape of "W" and is connected to the adjacent two rows. The control terminal of the exposure control circuit in the color pixel is electrically connected. For the specific connection manner, please refer to the description of the connection and the pixel circuit in the related parts of FIG. 3 to FIG. 5 and FIG. 8.

It should be noted that the "W" shape of the first exposure control line TX1 and the second exposure control line TX2 does not mean that the physical wiring must be set in strict accordance with the "W" shape, only the connection method corresponds to the full-color pixel and color The arrangement of the pixels is sufficient. For example, the setting of the "W" type exposure control line corresponds to the "W" type pixel arrangement method. This setting method is simple to route, and the pixel arrangement has good resolution and color effects, and realizes full color at low cost. Independent control of pixel exposure time and color pixel exposure time.

Similarly, as shown in FIG. 11, the full-color pixels in the first row and the second row are connected together by a first selection line SEL1 in a "W" shape to realize individual control of the output of the full-color pixels. The color pixels (A and B) of the first row and the second row are connected together by a second selection line SEL2 in a "W" shape to realize individual control of the output of the color pixels. The panchromatic pixels in the third row and the fourth row are connected together by the first selection line SEL1 in the shape of "W" to realize individual control of the output of the panchromatic pixels. The color pixels (B and C) of the third row and the fourth row are connected together by a second selection line SEL2 in a "W" shape to realize individual control of the output of the color pixels. For example, the first selection signal is transmitted via the first selection line SEL1, and the second exposure signal is transmitted via the second selection line SEL2. For example, the first selection line SEL1 has a "W" shape and is electrically connected to the selection terminal of the selection circuit in two adjacent rows of full-color pixels; the second selection line SEL2 has a "W" shape and is connected to the color pixels of two adjacent rows. The selection terminal of the selection circuit is electrically connected. For the specific connection manner, please refer to the description of the connection and the pixel circuit in the related parts of FIG. 3 to FIG. 5 and FIG. 9.

It should be noted that the "W" shape of the first selection line SEL1 and the second selection line SEL2 does not mean that the physical wiring must be set in strict accordance with the "W" shape. It only needs to be connected in a way corresponding to the full-color pixel and the color pixel. Just arrange it. For example, the setting of the "W" type selection line corresponds to the "W" type pixel arrangement method. This setting method is simple to route, and the resolution and color of the pixel arrangement have good effects, so that full-color pixels can be realized at low cost. Independent control of output and color pixel output.

For example, FIG. 12 is a schematic diagram of another minimum repeating unit 1182 pixel arrangement in the embodiment of the present application. The minimum repeating unit is 4 rows, 4 columns and 16 pixels, and the sub-units are 2 rows, 2 columns and 4 pixels. The arrangement is as follows:

For example, as shown in FIG. 12, the panchromatic pixel W is arranged in the first diagonal direction D1 (that is, the direction connecting the upper right corner and the lower left corner in FIG. 12), and the color pixels are arranged in the second diagonal direction D2 (for example, as shown in FIG. The direction where the upper left corner and the lower right corner are connected in 12). For example, the first diagonal line and the second diagonal line are perpendicular. The first exposure time of at least two panchromatic pixels W adjacent in the first diagonal direction D1 (for example, two panchromatic pixels in the first row, second column, and second row, first column from the upper left) is determined by The first exposure signal controls the second at least two color pixels adjacent in the second diagonal direction (for example, two color pixels A in the first row, first column and second row and second column from the upper left) The exposure time is controlled by the second exposure signal. The output of at least two full-color pixels W adjacent in the first diagonal direction D1 is controlled by a first selection signal, and the output of at least two color pixels adjacent in the second diagonal direction D2 is controlled by a second selection signal.

For example, as shown in FIG. 12, the panchromatic pixels in the first row and the second row are connected together by a first exposure control line TX1 in a "W" shape to realize individual control of the exposure time of the panchromatic pixels. The color pixels (A and B) in the first row and the second row are connected together by a second exposure control line TX2 in a "W" shape to realize individual control of the exposure time of the color pixels. The panchromatic pixels in the third row and the fourth row are connected together by the first exposure control line TX1 in the shape of "W" to realize the individual control of the exposure time of the panchromatic pixels. The color pixels (B and C) in the third row and the fourth row are connected together by a second exposure control line TX2 in a "W" shape to realize individual control of the exposure time of the color pixels.

Similarly, as shown in FIG. 12, the full-color pixels in the first row and the second row are connected together by a first selection line SEL1 in a "W" shape, so as to realize individual control of the output of the full-color pixels. The color pixels (A and B) of the first row and the second row are connected together by a second selection line SEL2 in a "W" shape to realize individual control of the output of the color pixels. The panchromatic pixels in the third row and the fourth row are connected together by the first selection line SEL1 in the shape of "W" to realize individual control of the output of the panchromatic pixels. The color pixels (B and C) of the third row and the fourth row are connected together by a second selection line SEL2 in a "W" shape to realize individual control of the output of the color pixels.

For example, FIG. 13 is a schematic diagram of another minimum repeating unit 1183 pixel arrangement in the embodiment of the present application. FIG. 14 is a schematic diagram of another minimum repeating unit 1184 pixel arrangement in the embodiment of the present application. In the embodiment of FIG. 13 and FIG. 14, corresponding to the arrangement of FIG. 11 and FIG. 12, the first color pixel A is a red pixel R; the second color pixel B is a green pixel G; and the third color pixel C is a blue pixel. Color pixel Bu.

It should be noted that, in some embodiments, the response band of the panchromatic pixel W is the visible light band (for example, 400 nm-760 nm). For example, the panchromatic pixel W is provided with an infrared filter to filter out infrared light. In some embodiments, the response wavelength band of the panchromatic pixel W is the visible light wavelength band and the near-infrared wavelength band (for example, 400 nm-1000 nm), which matches the response wavelength band of the photoelectric conversion element 117 (for example, the photodiode PD) in the image sensor 10. For example, the panchromatic pixel W may not be provided with a filter, and the response band of the panchromatic pixel W is determined by the response band of the photodiode, that is, the two match. The embodiments of the present application include, but are not limited to, the above-mentioned waveband range.

For example, FIG. 15 is a schematic diagram of another minimum repeating unit 1185 pixel arrangement in the embodiment of the present application. FIG. 16 is a schematic diagram of another minimum repeating unit 1186 pixel arrangement in the embodiment of the present application. In the embodiment of FIG. 15 and FIG. 16, corresponding to the arrangement of FIG. 11 and FIG. 12, the first color pixel A is a red pixel R; the second color pixel B is a yellow pixel Y; and the third color pixel C is a blue pixel. Color pixel Bu.

For example, FIG. 17 is a schematic diagram of another minimum repeating unit 1187 pixel arrangement in the embodiment of the present application. FIG. 18 is a schematic diagram of another minimum repeating unit 1188 pixel arrangement in the embodiment of the present application. In the embodiment of FIGS. 17 and 18, corresponding to the arrangement of FIGS. 11 and 12, the first color pixel A is magenta pixel M; the second color pixel B is cyan pixel Cy; and the third color pixel C is Yellow pixel Y.

For example, FIG. 19 is a schematic diagram of another minimum repeating unit 1191 pixel arrangement in the embodiment of the present application. The smallest repeating unit is 36 pixels in 6 rows and 6 columns, and the sub-units are 9 pixels in 3 rows, 3 columns, and the arrangement is:

For example, as shown in FIG. 19, the panchromatic pixels in the first row and the second row are connected together by a first exposure control line TX1 in a "W" shape to realize individual control of the exposure time of the panchromatic pixels. The color pixels (A and B) in the first row and the second row are connected together by a second exposure control line TX2 in a "W" shape to realize individual control of the exposure time of the color pixels. The panchromatic pixels in the third row and the fourth row are connected together by the first exposure control line TX1 in the shape of "W" to realize the individual control of the exposure time of the panchromatic pixels. The color pixels (A, B, and C) in the third row and the fourth row are connected together by a second exposure control line TX2 in a "W" shape to realize individual control of the exposure time of the color pixels. The panchromatic pixels in the fifth row and the sixth row are connected together by the first exposure control line TX1 in the shape of "W" to realize individual control of the exposure time of the panchromatic pixels. The color pixels (B and C) of the fifth row and the sixth row are connected together by a second exposure control line TX2 in a "W" shape to realize individual control of the exposure time of the color pixels.

Similarly, as shown in FIG. 19, the full-color pixels in the first row and the second row are connected together by a first selection line SEL1 in a "W" shape, so as to realize individual control of the full-color pixel output. The color pixels (A and B) of the first row and the second row are connected together by a second selection line SEL2 in a "W" shape to realize individual control of the output of the color pixels. The full-color pixels in the third row and the fourth row are connected together by the first selection line SEL1 in the shape of "W" to realize individual control of the full-color pixel output. The color pixels (A, B, and C) of the third row and the fourth row are connected together by a second selection line SEL2 in a "W" shape to realize individual control of the output of the color pixels. The panchromatic pixels in the fifth and sixth rows are connected together by the first selection line SEL1 in the shape of "W" to realize individual control of the panchromatic pixel output. The color pixels (B and C) of the fifth row and the sixth row are connected together by a second selection line SEL2 in a "W" shape to realize individual control of the output of the color pixels.

For example, FIG. 20 is a schematic diagram of another minimum repeating unit 1192 pixel arrangement in the embodiment of the present application. The smallest repeating unit is 36 pixels in 6 rows and 6 columns, and the sub-units are 9 pixels in 3 rows, 3 columns, and the arrangement is:

For example, as shown in FIG. 20, the panchromatic pixels in the first row and the second row are connected together by a first exposure control line TX1 in a "W" shape, so as to realize individual control of the exposure time of the panchromatic pixels. The color pixels (A and B) in the first row and the second row are connected together by a second exposure control line TX2 in a "W" shape to realize individual control of the exposure time of the color pixels. The panchromatic pixels in the third row and the fourth row are connected together by the first exposure control line TX1 in the shape of "W" to realize the individual control of the exposure time of the panchromatic pixels. The color pixels (A, B, and C) in the third row and the fourth row are connected together by a second exposure control line TX2 in a "W" shape to realize individual control of the exposure time of the color pixels. The panchromatic pixels in the fifth row and the sixth row are connected together by the first exposure control line TX1 in the shape of "W" to realize individual control of the exposure time of the panchromatic pixels. The color pixels (B and C) of the fifth row and the sixth row are connected together by a second exposure control line TX2 in a "W" shape to realize individual control of the exposure time of the color pixels.

Similarly, as shown in 20, the full-color pixels in the first row and the second row are connected together by a first selection line SEL1 in a "W" shape to realize individual control of the full-color pixel output. The color pixels (A and B) of the first row and the second row are connected together by a second selection line SEL2 in a "W" shape to realize individual control of the output of the color pixels. The full-color pixels in the third row and the fourth row are connected together by the first selection line SEL1 in the shape of "W" to realize individual control of the full-color pixel output. The color pixels (A, B, and C) of the third row and the fourth row are connected together by a second selection line SEL2 in a "W" shape to realize individual control of the output of the color pixels. The panchromatic pixels in the fifth row and the sixth row are connected together by the first selection line SEL1 in the shape of "W" to realize the individual control of the panchromatic pixel output. The color pixels (B and C) of the fifth row and the sixth row are connected together by a second selection line SEL2 in a "W" shape to realize individual control of the output of the color pixels.

For example, FIG. 21 is a schematic diagram of another minimum repeating unit 1193 pixel arrangement in the embodiment of the present application. FIG. 22 is a schematic diagram of another minimum repeating unit 1194 pixel arrangement in an embodiment of the present application. In the embodiment of FIG. 21 and FIG. 22, corresponding to the arrangement of FIG. 19 and FIG. 20, the first color pixel A is a red pixel R; the second color pixel B is a green pixel G; and the third color pixel C is a blue pixel. Color pixel Bu.

For example, in other embodiments, the first color pixel A is a red pixel R; the second color pixel B is a yellow pixel Y; and the third color pixel C is a blue pixel Bu. For example, in other embodiments, the first color pixel A is a magenta pixel M; the second color pixel B is a cyan pixel Cy; and the third color pixel C is a yellow pixel Y. The embodiments of the present application include but are not limited to this. Please refer to the above description for the specific connection mode of the circuit, which will not be repeated here.

For example, FIG. 23 is a schematic diagram of another minimum repeating unit 1195 pixel arrangement in the embodiment of the present application. The smallest repeating unit is 8 rows, 8 columns and 64 pixels, and the sub-units are 4 rows, 4 columns and 16 pixels. The arrangement is:

For example, as shown in FIG. 23, the panchromatic pixels in the first row and the second row are connected together by the first exposure control line TX1 in the shape of "W" to realize individual control of the exposure time of the panchromatic pixels. The color pixels (A and B) in the first row and the second row are connected together by a second exposure control line TX2 in a "W" shape to realize individual control of the exposure time of the color pixels. The panchromatic pixels in the third row and the fourth row are connected together by the first exposure control line TX1 in the shape of "W" to realize the individual control of the exposure time of the panchromatic pixels. The color pixels (A and B) of the third row and the fourth row are connected together by a second exposure control line TX2 in a "W" shape to realize individual control of the exposure time of the color pixels. The panchromatic pixels in the fifth row and the sixth row are connected together by the first exposure control line TX1 in the shape of "W" to realize individual control of the exposure time of the panchromatic pixels. The color pixels (B and C) of the fifth row and the sixth row are connected together by a second exposure control line TX2 in a "W" shape to realize individual control of the exposure time of the color pixels. The panchromatic pixels in the seventh row and the eighth row are connected together by the first exposure control line TX1 in a "W" shape to realize individual control of the exposure time of the panchromatic pixels. The color pixels (B and C) in the seventh row and the eighth row are connected together by a second exposure control line TX2 in a "W" shape to realize individual control of the exposure time of the color pixels.

Similarly, as shown in FIG. 23, the full-color pixels in the first row and the second row are connected together by a first selection line SEL1 in a "W" shape to realize individual control of the full-color pixel output. The color pixels (A and B) of the first row and the second row are connected together by a second selection line SEL2 in a "W" shape to realize individual control of the output of the color pixels. The full-color pixels in the third row and the fourth row are connected together by the first selection line SEL1 in the shape of "W" to realize individual control of the full-color pixel output. The color pixels (A and B) of the third row and the fourth row are connected together by a second selection line SEL2 in a "W" shape to realize individual control of the output of the color pixels. The panchromatic pixels in the fifth row and the sixth row are connected together by the first selection line SEL1 in the shape of "W" to realize the individual control of the panchromatic pixel output. The color pixels (B and C) of the fifth row and the sixth row are connected together by a second selection line SEL2 in a "W" shape to realize individual control of the output of the color pixels. The panchromatic pixels in the seventh row and the eighth row are connected together by the first selection line SEL1 in the shape of "W" to realize the individual control of the panchromatic pixel output. The color pixels (B and C) of the seventh row and the eighth row are connected together by a second selection line SEL2 in a "W" shape to realize individual control of the output of the color pixels.

For example, FIG. 24 is a schematic diagram of another minimum repeating unit 1196 pixel arrangement in the embodiment of the present application. The smallest repeating unit is 8 rows, 8 columns and 64 pixels, and the sub-units are 4 rows, 4 columns and 16 pixels. The arrangement is:

For example, as shown in FIG. 24, the panchromatic pixels in the first row and the second row are connected together by a first exposure control line TX1 in a "W" shape to realize individual control of the exposure time of the panchromatic pixels. The color pixels (A and B) in the first row and the second row are connected together by a second exposure control line TX2 in a "W" shape to realize individual control of the exposure time of the color pixels. The panchromatic pixels in the third row and the fourth row are connected together by the first exposure control line TX1 in the shape of "W" to realize the individual control of the exposure time of the panchromatic pixels. The color pixels (A and B) of the third row and the fourth row are connected together by a second exposure control line TX2 in a "W" shape to realize individual control of the exposure time of the color pixels. The panchromatic pixels in the fifth row and the sixth row are connected together by the first exposure control line TX1 in the shape of "W" to realize individual control of the exposure time of the panchromatic pixels. The color pixels (B and C) of the fifth row and the sixth row are connected together by a second exposure control line TX2 in a "W" shape to realize individual control of the exposure time of the color pixels. The panchromatic pixels in the seventh row and the eighth row are connected together by the first exposure control line TX1 in a "W" shape to realize individual control of the exposure time of the panchromatic pixels. The color pixels (B and C) in the seventh row and the eighth row are connected together by a second exposure control line TX2 in a "W" shape to realize individual control of the exposure time of the color pixels.

Similarly, as shown in FIG. 24, the full-color pixels in the first row and the second row are connected together by a first selection line SEL1 in a "W" shape to realize individual control of the full-color pixel output. The color pixels (A and B) of the first row and the second row are connected together by a second selection line SEL2 in a "W" shape to realize individual control of the output of the color pixels. The full-color pixels in the third row and the fourth row are connected together by the first selection line SEL1 in the shape of "W" to realize individual control of the full-color pixel output. The color pixels (A and B) of the third row and the fourth row are connected together by a second selection line SEL2 in a "W" shape to realize individual control of the output of the color pixels. The panchromatic pixels in the fifth row and the sixth row are connected together by the first selection line SEL1 in the shape of "W" to realize the individual control of the panchromatic pixel output. The color pixels (B and C) of the fifth row and the sixth row are connected together by a second selection line SEL2 in a "W" shape to realize individual control of the output of the color pixels. The panchromatic pixels in the seventh row and the eighth row are connected together by the first selection line SEL1 in the shape of "W" to realize the individual control of the panchromatic pixel output. The color pixels (B and C) of the seventh row and the eighth row are connected together by a second selection line SEL2 in a "W" shape to realize individual control of the output of the color pixels.

For example, FIG. 26 is a schematic diagram of another minimum repeating unit 1197 pixel arrangement in the embodiment of the present application. FIG. 27 is a schematic diagram of another minimum repeating unit 1198 pixel arrangement in an embodiment of the present application. In the embodiment of FIGS. 26 and 27, corresponding to the arrangement of FIGS. 23 and 24, the first color pixel A is a red pixel R; the second color pixel B is a green pixel G; and the third color pixel C is a blue pixel. Color pixel Bu.

For example, in other embodiments, the first color pixel A is a red pixel R; the second color pixel B is a yellow pixel Y; and the third color pixel C is a blue pixel Bu. For example, the first color pixel A is a magenta pixel M; the second color pixel B is a cyan pixel Cy; and the third color pixel C is a yellow pixel Y. The embodiments of the present application include but are not limited to this. Refer to the above description for the specific connection mode of the circuit, which will not be repeated here.

It can be seen from the above embodiment that, as shown in FIGS. 11 to 26, the image sensor 10 (shown in FIG. 2) includes a plurality of color pixels and a plurality of panchromatic pixels W arranged in a matrix, the color pixels and the panchromatic pixels They are arranged at intervals in the row and column directions.

For example, panchromatic pixels, color pixels, panchromatic pixels, color pixels are alternately arranged in the row direction...

For example, panchromatic pixels, color pixels, panchromatic pixels, and color pixels are alternately arranged in the direction of the column.

Referring to FIGS. 3 to 8, the first exposure control line TX1 is electrically connected to the control terminal TG (for example, the gate of the transfer transistor 112) of the exposure control circuit 116 in the panchromatic pixel W in the 2n-1th row and the 2nth row The second exposure control line TX2 is electrically connected to the control terminal TG (for example, the gate of the transfer transistor 112) of the exposure control circuit 116 in the 2n-1 and 2nth rows of color pixels; n is a natural number greater than or equal to 1.

For example, when n=1, the first exposure control line TX1 is electrically connected to the control terminal TG of the exposure control circuit 116 in the panchromatic pixels W in the first row and the second row; the second exposure control line TX2 is connected to the first row and The control terminal TG of the exposure control circuit 116 in the color pixels in the second row is electrically connected. When n=2, the first exposure control line TX1 is electrically connected to the control terminal TG of the exposure control circuit 116 in the panchromatic pixels W in the third and fourth rows; the second exposure control line TX2 is electrically connected to the third and fourth rows. The control terminal TG of the exposure control circuit 116 in the color pixels of the row is electrically connected. By analogy, I won't repeat them here.

Similarly, please refer to FIGS. 3, 4, 5, and 9, the first selection line SEL1 and the selection terminal SEL of the selection circuit in the full-color pixel W in the 2n-1th row and the 2nth row (for example, the selection transistor 115) is electrically connected; the second selection line SEL2 is electrically connected to the control terminal SEL (for example, the gate of the selection transistor 115) of the selection circuit in the 2n-1th row and the 2nth row of the color pixels; n is greater than or equal to The natural number of 1.

For example, when n=1, the first selection line SEL1 is electrically connected to the selection terminal SEL of the selection circuit in the full-color pixel W in the first row and the second row; the second selection line SEL2 is electrically connected to the first row and the second row. The selection terminal SEL of the selection circuit in the color pixel is electrically connected. When n=2, the first selection line SEL1 is electrically connected to the selection terminal SEL of the selection circuit in the full-color pixels W in the third and fourth rows; the second selection line SEL2 is electrically connected to the color pixels in the third and fourth rows The selection terminal SEL of the middle selection circuit is electrically connected. By analogy, I won't repeat them here.

The image sensor 10 of the embodiment of the present application obtains multiple frames of original images by controlling the panchromatic pixels to be exposed multiple times within the first exposure time, and the multiple frames of original images can be used to calculate the motion information of objects in the scene. Performing blur correction on the color original image according to the motion information can obtain the target image with the motion blur removed.

Please refer to FIG. 27. The present application provides a camera assembly 40. The camera assembly 40 includes a processing chip 20, a lens 30, and the image sensor 10 described in any one of the above embodiments. The image sensor 10 is electrically connected to the processing chip 20. The lens 30 is provided on the optical path of the image sensor 10. The image sensor 10 may receive light passing through the lens 30 to obtain an original image. The processing chip 20 can receive the original image output by the image sensor 10 and perform subsequent processing on the original image.

The present application also provides a control method that can be used for the camera assembly 40 in FIG. 27. As shown in Figure 28, the control method includes:

01: Exposure of color pixels and output at least one frame of color original image;

02: At least part of the panchromatic pixels are exposed multiple times within the time that the color pixels are exposed once to obtain a frame of color original image to obtain multiple frames of panchromatic original image; and

Please refer to FIG. 27 and FIG. 28, the control method of the present application can be implemented by the camera assembly 40. Among them, step 01 and step 02 can be implemented by the image sensor 10. Step 03 can be implemented by the processing chip 20. In other words, the color pixels in the image sensor 10 are exposed and output at least one frame of color original image; at least part of the panchromatic pixels are exposed for multiple times within the time that the color pixels are exposed once to obtain one frame of color original image to obtain multiple frames Full-color original image. The processing chip 20 can correct at least one frame of the color original image by using the multi-frame full-color original image to obtain the target image.

Specifically, please refer to Figure 2, Figure 8, Figure 9, and Figure 29, when the user requests to take a photo, the vertical drive unit 12 in the image sensor 10 will be controlled by the first exposure control line TX1 and the second exposure control line TX2 respectively For the exposure of multiple panchromatic pixels and multiple color pixels in a two-dimensional pixel array, the vertical driving unit 12 controls the output analog signals of the panchromatic pixels and the color pixels to the columns through the first selection line SEL1 and the second selection line SEL2, respectively The processing unit 14 and the conversion circuit 16 in the column processing unit 14 convert the analog signals output by the panchromatic pixels and the color pixels into digital signals. Digital signals corresponding to a plurality of panchromatic pixels form a full-color original image, and color signals corresponding to a plurality of color pixels form a color original image. Each time the user requests to take a photo, the image sensor 10 will output one frame of color original image and multiple frames of full color original image.

As shown in Figure 29, each frame of the full-color original image includes multiple panchromatic pixels W and multiple null pixels N (NULL). The null pixels are neither panchromatic pixels nor color pixels. The position of the empty pixel N in the image can be regarded as no pixel at that position, or the pixel value of the empty pixel can be regarded as zero. Comparing the two-dimensional pixel array with the full-color original image, it can be seen that for each sub-unit in the two-dimensional pixel array, the sub-unit includes two full-color pixels W and two color pixels (color pixel A, color pixel B, or color pixel). Pixel C). The full-color original image also has a sub-unit corresponding to each sub-unit in the two-dimensional pixel array. The sub-unit of the full-color original image includes two full-color pixels W and two empty pixels N, and two empty pixels N The location corresponds to the location of the two color pixels in the subunit of the two-dimensional pixel array.

Similarly, the color original image includes a plurality of color pixels and a plurality of empty pixels N. The empty pixels are neither panchromatic pixels nor color pixels. The position of the empty pixel N in the color original image can be regarded as there is no such position. Pixel, or the pixel value of an empty pixel can be treated as zero. Comparing the two-dimensional pixel array with the color original image, it can be seen that for each sub-unit in the two-dimensional pixel array, the sub-unit includes two panchromatic pixels W and two color pixels. The color original image also has a subunit corresponding to each subunit in the two-dimensional pixel array. The subunit of the color original image includes two color pixels and two empty pixels N. The positions of the two empty pixels N correspond to each other. The position where the two panchromatic pixels W in the subunit of the two-dimensional pixel array are located.

After the processing chip 20 receives the full-color original image and the color original image, it can further process the multi-frame full-color original image to obtain a multi-frame full-color intermediate image, and further process at least one frame of the color original image to obtain at least one frame of color The middle image.

For example, as shown in FIG. 30, for a full-color original image, the full-color original image includes a plurality of subunits, and each subunit includes a plurality of empty pixels N and a plurality of panchromatic pixels. Specifically, each subunit includes two empty pixels N and two panchromatic pixels W, and the processing chip 20 may use all pixels in the subunit including the empty pixels N and the panchromatic pixel W as the panchromatic pixels corresponding to the subunit. For the large pixel W, the processing chip 20 may add the pixel values of all pixels in the subunit including the empty pixel N and the panchromatic pixel W, and use the result of the addition as the pixel of the panchromatic large pixel A corresponding to the subunit. Value, where the pixel value of the empty pixel N can be regarded as zero. Thus, the processing chip 20 can obtain a full-color intermediate image.

For example, for a color original image, the processing chip 20 regards all the pixels of each sub-unit as a single-color large pixel corresponding to a single color in the sub-unit, and outputs the pixel value of the single-color large pixel to obtain a color intermediate image. Specifically, the color original image can be transformed into a color intermediate image in the manner shown in FIG. 31. As shown in FIG. 31, the color original image includes a plurality of sub-units, and each sub-unit includes a plurality of empty pixels N and a plurality of single-color color pixels (also called single-color pixels). Specifically, some sub-units include two empty pixels N and two single-color pixels A, some sub-units include two empty pixels N and two single-color pixels B, and some sub-units include two empty pixels N and Two single-color pixels C. The processing chip 20 may regard all the pixels in the sub-unit including the empty pixel N and the single-color pixel A as the single-color large pixel A corresponding to the single-color A in the sub-unit, and will include the empty pixel N and the single-color pixel B. All the pixels in the sub-unit are regarded as the single-color large pixels B corresponding to the single color B in the sub-unit, and all the pixels in the sub-unit including the empty pixel N and the single-color pixel C are regarded as the single-color pixels in the sub-unit. C corresponds to the single-color large pixel C. For the monochromatic large pixel A, the processing chip 20 may add the pixel values of all pixels in the sub-unit including the empty pixel N and the single-color pixel A, and use the result of the addition as the monochromatic large pixel corresponding to the sub-unit The pixel value of A, where the pixel value of the empty pixel N can be regarded as zero, the same below; the processing chip 20 can add the pixel values of all pixels in the sub-unit including the empty pixel N and the single-color pixel B, and add The result of the addition is taken as the pixel value of the single-color large pixel B corresponding to the sub-unit; the processing chip 20 may add the pixel values of all pixels in the sub-unit including the empty pixel N and the single-color pixel C, and add The result is the pixel value of the single-color large pixel C corresponding to the sub-unit. Thus, the processing chip 20 can obtain the pixel values of a plurality of single large pixels A, the pixel values of a plurality of monochromatic large pixels B, and the pixel values of a plurality of monochromatic large pixels C. The processing chip 20 then forms a pixel array with a lower resolution than a two-dimensional pixel array according to the pixel values of the plurality of monochromatic large pixels A, the pixel values of the plurality of monochromatic large pixels B, and the pixel values of the plurality of monochromatic large pixels C The color intermediate image of the resolution.

After the processing chip 20 obtains the full-color intermediate image and the color intermediate image, the processing chip 20 can use multiple frames of the full-color intermediate image to correct at least one frame of the color intermediate image to obtain the target image.

In the control method of the embodiment of the present application, by controlling the panchromatic pixels to be exposed multiple times within the time of one exposure of the color pixels to obtain multiple frames of panchromatic original images, and using the multiple frames of panchromatic original images to correct the color original images, it is possible to remove Motion blurred target image.

In addition, in the control method of the embodiment of the present application, multiple frames of panchromatic original images are used to correct the color original image. Since the panchromatic original image is acquired by the panchromatic pixels with higher sensitivity, the panchromatic original image has higher signal-to-noise. In contrast, using an image with a high signal-to-noise ratio to correct a motion-blurred image can improve the effect of removing motion blur.

Please refer to FIG. 32. In some embodiments, a point spread function can be used to correct the color original image. Step 03 correcting at least one frame of color original image by using multiple frames of full-color original image to obtain a target image includes:

Please refer to FIG. 27. In some embodiments, both step 031 and step 032 can be implemented by the processing chip 20. In other words, the processing chip 20 can be used to calculate correction data based on multiple frames of full-color original images, and use the correction data to correct at least one frame of color original images to obtain a target image.

Specifically, the processing chip 20 may first calculate the displacement vector between any two adjacent full-color original images according to the multi-frame full-color original images, obtain discrete displacement point information, and fit the displacement information to obtain the displacement curve. Subsequently, the processing chip 20 calculates a speed curve according to the displacement curve, and then calculates a point spread function (ie, correction data) according to the speed curve and the first exposure time. Subsequently, the processing chip 20 corrects the color original image according to the point spread function to obtain a clear target image.

Of course, in other embodiments, methods such as blind image algorithms can also be used to correct the color original image, which is not limited here.

04: Expose the remaining panchromatic pixels to output a panchromatic original image; and

Please refer to FIG. 27 and FIG. 33. In some embodiments, step 04 may be implemented by the image sensor 10, and step 05 may be implemented by the processing chip 20. That is, the remaining full-color pixels of the image sensor 10 are exposed to output a full-color original image. The processing chip can be used to correct the brightness of the target image according to the panchromatic original image output by the remaining panchromatic pixels.

Specifically, referring to Fig. 34, the color pixels are exposed to output one frame of color original image during the first exposure time, and some panchromatic pixels are exposed to output multiple frames of the first panchromatic original image during the first exposure time, and the remaining panchromatic pixels The pixels are exposed to one frame at a time during the first exposure time to output a second full-color original image.

As shown in FIG. 34, each frame of the first panchromatic original image includes a plurality of panchromatic pixels W and a plurality of empty pixels N (NULL), wherein the empty pixels are neither panchromatic pixels nor color pixels. The position of the empty pixel N in the panchromatic original image can be regarded as no pixel at that position, or the pixel value of the empty pixel can be regarded as zero. Comparing the two-dimensional pixel array with the first full-color original image, it can be seen that for each sub-unit in the two-dimensional pixel array, the sub-unit includes two full-color pixels W and two color pixels (color pixel A, color pixel B, Or color pixel C). The first full-color original image also has a sub-unit corresponding to each sub-unit in the two-dimensional pixel array, and the sub-unit of the first full-color original image includes a full-color image for multiple exposures within the first exposure time. Color pixel W'and three empty pixels N. The positions of the three empty pixels N correspond to the two color pixels in the subunit of the two-dimensional pixel array and the panchromatic pixel W for one exposure within the first exposure time. s position.

Similarly, the second panchromatic original image includes a plurality of panchromatic pixels W and a plurality of empty pixels N. The empty pixels are neither panchromatic pixels nor color pixels. A position can be regarded as there is no pixel at that position, or the pixel value of an empty pixel can be regarded as zero. Comparing the two-dimensional pixel array with the second panchromatic original image, it can be known that for each subunit in the two-dimensional pixel array, the subunit includes two panchromatic pixels W and two color pixels. The second full-color original image also has a sub-unit corresponding to each sub-unit in the two-dimensional pixel array, and the sub-unit of the second full-color original image includes a full-color exposure for one exposure within the first exposure time. Pixel W and three empty pixels N. The positions of the three empty pixels N correspond to the two color pixels in the subunit of the two-dimensional pixel array and one panchromatic pixel W'that is used for multiple exposures within the first exposure time. Location.

The color original image is the same as the color original image in the embodiment shown in FIG. 29, and will not be repeated here.

After the processing chip 20 obtains the first panchromatic original image, the second panchromatic original image, and the color original image, it can further process the multiple frames of the first panchromatic original image and the second panchromatic original image to obtain multiple frames of the first panchromatic image. The intermediate image and a frame of second full-color original image, and at least one frame of color original image is further processed to obtain at least one frame of color intermediate image. For example, the processing chip 20 may adopt the method shown in FIG. 30 to realize the conversion of the first full-color original image to the first full-color intermediate image and the conversion of the second full-color original image to the second full-color intermediate image. For example, the processing chip 20 may adopt the method shown in FIG. 31 to realize the conversion of the color original image to the color intermediate image.

After the processing chip 20 obtains the first panchromatic intermediate image and the color intermediate image, the processing chip 20 can use multiple frames of the first panchromatic intermediate image to correct at least one frame of the color intermediate image to obtain the target image. The correction method is the same as that of the embodiment shown in FIG. 32, and will not be repeated here.

After the processing chip 20 obtains the motion-blurred target image, the second panchromatic intermediate image can be used to perform brightness correction on the motion-blurred target image. Specifically, as shown in FIG. 35, the processing chip 20 first separates the color and brightness of the target image for removing motion blur to obtain a color-brightness separated image. In the color-brightness separated image in FIG. 35, L represents brightness, and CLR represents color. Specifically, assuming that the single-color pixel A is the red pixel R, the single-color pixel B is the green pixel G, and the single-color pixel C is the blue pixel Bu, then: (1) the processing chip 20 can de-motion blur the target in the RGB space The image is converted into a color-brightness separated image in YCrCb space. At this time, Y in YCrCb is the brightness L in the color-brightness separated image, and Cr and Cb in YCrCb are the color CLR in the color-brightness separated image; (2) processing chip 20 You can also convert the RGB de-motion blur target image into a color-brightness separated image in Lab space. At this time, L in Lab is the brightness L in the color-brightness separated image, and a and b in Lab are color-brightness separation images. The color CLR in the image. It should be noted that L+CLR in the color-light separation image shown in FIG. 35 does not mean that the pixel value of each pixel is formed by adding L and CLR, but only that the pixel value of each pixel is composed of L and CLR.

Subsequently, the processing chip 20 fuses the brightness of the color-brightness separated image and the brightness of the second full-color intermediate image. For example, the pixel value of each panchromatic pixel W in the second panchromatic intermediate image is the brightness value of each panchromatic pixel, and the processing chip 20 can separate the color brightness from the L of each pixel in the image and the second panchromatic pixel. The W of the panchromatic pixel at the corresponding position in the intermediate image is added to obtain the pixel value after brightness correction. The processing chip 20 forms a brightness-corrected color-brightness separated image according to a plurality of brightness-corrected pixel values, and then uses color space conversion to convert the brightness-corrected color-brightness separated image into a brightness correction target image.

When the monochromatic large pixel A is the red pixel R, the monochromatic large pixel B is the green pixel G, and the monochromatic large pixel C is the blue pixel Bu, the brightness correction color image is an image arranged in a Bayer array, and the processing chip 20 needs to The brightness-corrected color image is subjected to interpolation processing, so that the pixel value of each large monochromatic pixel after brightness correction has three components of R, G, and B at the same time. The processing chip 20 can perform interpolation processing on the brightness-corrected color image to obtain the final target image.

In some embodiments, if the interval between the acquisition times of two consecutive frames of color original images is less than a predetermined value, it is possible to control at least part of the panchromatic pixels to be exposed multiple times during the process of acquiring only one frame of the color original image. Multiple frames of full-color original images are output, and in the process of acquiring another frame of color original images, there is no need to perform the action of controlling multiple exposures of at least part of the full-color pixels. Subsequently, the processing chip 20 can use the acquired multi-frame full-color original image to correct the two-frame color original image. For example, the image sensor 10 acquires two frames of color original images at the same time, wherein the time interval for acquiring the two frames of color original images is 1ms, of which 1ms is only an example, and the time interval can also be 2ms, 2.5ms, 3ms, 5ms, 8ms, 10ms, 15ms, 20ms, etc., there is no limitation here. The processing chip 20 can correct the first frame of color original image and the second frame of color original image by using the multi-frame full-color original image obtained by at least part of the full-color pixel exposure within the time that the color pixel is exposed once to obtain the first frame of color original image, To obtain two frames of the target image with the motion blur removed. It can be understood that when the time interval between the acquisition times of two consecutive color original images is small, the moving speed of the moving object in the scene will not change significantly, and the correction data required for the two color original images are basically the same. Therefore, the two frames of color original images can be corrected directly according to the correction data required for one frame of the color original images, and the de-motion blur effect of the target image can also be achieved. At the same time, since at least part of the panchromatic pixels do not need to be exposed during the acquisition period of another frame of color original image, the power consumption of the image sensor 10 is reduced. In addition, the processing chip 20 only needs to perform the action of calculating the correction data once, which simplifies the complexity of image processing and accelerates the speed of image processing.

Referring to FIG. 36, the present application also provides a mobile terminal 60. The mobile terminal 60 may be a mobile phone, a tablet computer, a notebook computer, a smart wearable device (such as a smart watch, a smart bracelet, a smart glasses, a smart helmet, etc.), a head-mounted display device, a virtual reality device, etc., which are not limited here.

The mobile terminal 60 includes a housing 50 and a camera assembly 40. The housing 50 and the camera assembly 40 are combined. Illustratively, the camera assembly 40 may be mounted on the housing 50. The mobile terminal 60 may also include a processor (not shown). The processing chip 20 and the processor in the camera assembly 40 may be the same processor or two independent processors, which is not limited here.

In the mobile terminal 60 of the embodiment of the present application, the image sensor 10 is simultaneously arranged with panchromatic pixels with higher sensitivity and color pixels with lower sensitivity than panchromatic pixels, so that at least part of the panchromatic pixels with higher sensitivity can be used in a segment. Expose at a higher frame rate within a period of time to obtain multiple frames of images, and the remaining pixels are exposed at a lower frame rate within this period of time to obtain at least one frame of images. The multiple frames of images obtained after high frame rate exposure can be used to correct low frames Rate exposure to obtain the image, in order to achieve the elimination of the motion blur of the image obtained by the low frame rate exposure. The mobile terminal 60 according to the embodiment of the present application can eliminate the motion blur of the image without setting multiple image sensors 10, and the complexity of the hardware system is low. In addition, the motion-blurred image and the corrected image are acquired by the same image sensor 10, no compensation and calibration are needed in the subsequent processing, and the algorithm complexity is also low. In addition, the corrective image is obtained by using high-sensitivity panchromatic pixels. The image has a high signal-to-noise ratio. Using an image with a high signal-to-noise ratio to correct a motion blur image can improve the elimination of motion blur. effect.

In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "exemplary embodiments", "examples", "specific examples" or "some examples" etc. means to combine the described implementations The specific features, structures, materials, or characteristics described by the manners or examples are included in at least one embodiment or example of the present application. In this specification, the schematic representation of the above-mentioned terms does not necessarily refer to the same embodiment or example. Moreover, the described specific features, structures, materials or characteristics can be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art can combine and combine the different embodiments or examples and the features of the different embodiments or examples described in this specification without mutual contradiction.

Any process or method description in the flowchart or described in other ways herein can be understood as a module, segment or part of code that includes one or more executable instructions for implementing specific logical functions or steps of the process And the scope of the preferred embodiments of the present application includes additional implementations, which may not be in the order shown or discussed, including performing functions in a substantially simultaneous manner or in the reverse order according to the functions involved. This should It is understood by those skilled in the art to which the embodiments of the present application belong.

Although the embodiments of this application have been shown and described above, it can be understood that the above embodiments are exemplary and should not be construed as limitations on the application. Those of ordinary skill in the art can comment on the above within the scope of this application. The implementation mode undergoes changes, modifications, replacements and modifications.

Claims

An image sensor, characterized in that the image sensor comprises:

Panchromatic pixels; and

Color pixels, the color pixels having a narrower spectral response than the panchromatic pixels;

The color pixels can be exposed to output at least one frame of color original image;

During the time that the color pixels are exposed once to obtain one frame of color original image, at least part of the panchromatic pixels can be exposed multiple times to obtain multiple frames of panchromatic original images, and multiple frames of the panchromatic original images can be used for correction At least one frame of the color original image is used to obtain a target image.
The image sensor according to claim 1, wherein at least part of the pixel circuits of the panchromatic pixels comprise:

Photoelectric conversion element;

An exposure control circuit, the exposure control circuit is electrically connected to the photoelectric conversion element, and the exposure control circuit is used for the charge accumulated after the photoelectric conversion element is irradiated multiple times within the time of one exposure of the color pixel Transfer to floating diffusion unit;

A selection circuit configured to output analog signals corresponding to the charges in the floating diffusion unit multiple times; and

The image sensor further includes a conversion circuit, each of the panchromatic pixels in at least part of the panchromatic pixels corresponds to a plurality of the conversion circuits, and the plurality of conversion circuits are connected to all of the panchromatic pixels. The selection circuit is electrically connected, and one of the analog signals output by the selection circuit is correspondingly output to one of the conversion circuits, and each of the conversion circuits is used to convert the analog signal into a digital signal.
The image sensor according to claim 2, wherein at least part of the pixel circuit of the panchromatic pixel further comprises: a reset circuit, the reset circuit is electrically connected to the floating diffusion unit, and the reset circuit is used for During one exposure of the color pixels, the floating diffusion unit is reset every time before the exposure control circuit transfers charges.
The image sensor according to claim 2, wherein the pixel circuit of at least part of the panchromatic pixels further comprises:

A reset circuit, the reset circuit is electrically connected to the floating diffusion unit, and the reset circuit is used to reset all the color pixels before the exposure control circuit transfers the charge for the first time during the exposure time of the color pixels.述 Floating diffusion unit.
4. The image sensor according to any one of claims 2-4, wherein each of the conversion circuits comprises:

A switch, the switch is electrically connected to the selection circuit, and the switch is used for passing the analog signal from the selection circuit; and

An analog-to-digital conversion module, the digital-to-analog conversion module is electrically connected to the switch, and the analog-to-digital conversion module is configured to convert the analog signal into the digital signal.
4. The image sensor according to any one of claims 2-4, wherein each of the conversion circuits comprises:

A switch, the switch is electrically connected to the selection circuit, and the switch is used for passing the analog signal from the selection circuit;

A capacitor, the capacitor is connected to the switch, and the capacitor is used to store the analog signal transmitted through the switch;

An analog-to-digital conversion module, the digital-to-analog conversion module is electrically connected to the switch and the capacitor, and the analog-to-digital conversion module is configured to convert the analog signal into the digital signal.
The image sensor according to claim 1, wherein the panchromatic pixels and the color pixels are arranged in the form of a two-dimensional pixel array, and the two-dimensional pixel array includes a plurality of minimum repeating units. In the repeating unit, the panchromatic pixels are arranged in a first diagonal direction, the color pixels are arranged in a second diagonal direction, and the first diagonal direction is different from the second diagonal direction .
A control method for an image sensor, characterized in that the image sensor includes panchromatic pixels and color pixels, and the color pixels have a narrower spectral response than the panchromatic pixels; the control method includes:

The color pixels are exposed and output at least one frame of color original image;

At least part of the panchromatic pixels are exposed for multiple times within the time that the color pixels are exposed once to obtain one frame of the color original image to obtain multiple frames of panchromatic original images; and

At least one frame of the color original image is corrected by using a plurality of frames of the full-color original image to obtain a target image.
8. The control method according to claim 8, wherein said correcting at least one frame of said color original image to obtain a target image using multiple frames of said panchromatic original image comprises:

Calculating correction data based on multiple frames of the full-color original image; and

Using the correction data to correct at least one frame of the color original image to obtain the target image.
The control method according to claim 8, wherein when only part of the panchromatic pixels are exposed multiple times within the time that the color pixels are exposed once to obtain one frame of color original image, to obtain multiple frames of panchromatic original image. When displaying images, the control method further includes:

The remaining panchromatic pixels are exposed to output a panchromatic original image;

Correcting the brightness of the target image according to the panchromatic original image output by the remaining panchromatic pixels.
A camera assembly, characterized in that it comprises:

Lens; and

An image sensor capable of receiving light passing through the lens, and the image sensor includes:

Panchromatic pixels; and

Color pixels, the color pixels having a narrower spectral response than the panchromatic pixels;

The color pixels can be exposed to output at least one frame of color original image;

During the time that the color pixels are exposed once to obtain one frame of color original image, at least part of the panchromatic pixels can be exposed multiple times to obtain multiple frames of panchromatic original images, and multiple frames of the panchromatic original images can be used for correction At least one frame of the color original image is used to obtain a target image.
The camera assembly of claim 11, wherein at least part of the pixel circuits of the panchromatic pixels comprise:

Photoelectric conversion element;

An exposure control circuit, the exposure control circuit is electrically connected to the photoelectric conversion element, and the exposure control circuit is used for the charge accumulated after the photoelectric conversion element is irradiated multiple times within the time of one exposure of the color pixel Transfer to floating diffusion unit;

A selection circuit configured to output analog signals corresponding to the charges in the floating diffusion unit multiple times; and

The image sensor further includes a conversion circuit, each of the panchromatic pixels in at least part of the panchromatic pixels corresponds to a plurality of the conversion circuits, and the plurality of conversion circuits are connected to all of the panchromatic pixels. The selection circuit is electrically connected, and one of the analog signals output by the selection circuit is correspondingly output to one of the conversion circuits, and each of the conversion circuits is used to convert the analog signal into a digital signal.
The camera assembly of claim 12, wherein at least part of the pixel circuit of the panchromatic pixel further comprises: a reset circuit, the reset circuit is electrically connected to the floating diffusion unit, and the reset circuit is used for During one exposure of the color pixels, the floating diffusion unit is reset every time before the exposure control circuit transfers charges.
The camera assembly according to claim 12, wherein the pixel circuit of at least part of the panchromatic pixels further comprises:

A reset circuit, the reset circuit is electrically connected to the floating diffusion unit, and the reset circuit is used to reset all the color pixels before the exposure control circuit transfers the charge for the first time during the exposure time of the color pixels.述 Floating diffusion unit.
The camera assembly according to any one of claims 12-14, wherein each of the conversion circuits comprises:

A switch, the switch is electrically connected to the selection circuit, and the switch is used for passing the analog signal from the selection circuit; and

An analog-to-digital conversion module, the digital-to-analog conversion module is electrically connected to the switch, and the analog-to-digital conversion module is configured to convert the analog signal into the digital signal.
The camera assembly according to any one of claims 12-14, wherein each of the conversion circuits comprises:

A switch, the switch is electrically connected to the selection circuit, and the switch is used for passing the analog signal from the selection circuit;

A capacitor, the capacitor is connected to the switch, and the capacitor is used to store the analog signal transmitted through the switch;

An analog-to-digital conversion module, the digital-to-analog conversion module is electrically connected to the switch and the capacitor, and the analog-to-digital conversion module is configured to convert the analog signal into the digital signal.
The camera assembly according to claim 11, wherein the camera assembly further comprises a processing chip for correcting at least one frame of the color original image by using multiple frames of the panchromatic original image to obtain a target image.
The camera assembly of claim 17, wherein the processing chip is further used for:

Calculating correction data based on multiple frames of the full-color original image; and

Using the correction data to correct at least one frame of the color original image to obtain the target image.
The camera assembly according to claim 17, wherein when only part of the panchromatic pixels are exposed multiple times within the time that the color pixels are exposed once to obtain one frame of color original image During the image, the remaining panchromatic pixels are exposed to output a panchromatic original image;

The processing chip is further configured to correct the brightness of the target image according to the panchromatic original image output by the remaining panchromatic pixels.
The camera assembly according to claim 11, wherein the panchromatic pixels and the color pixels are arranged in a two-dimensional pixel array, and the two-dimensional pixel array includes a plurality of minimum repeating units. In the repeating unit, the panchromatic pixels are arranged in a first diagonal direction, the color pixels are arranged in a second diagonal direction, and the first diagonal direction is different from the second diagonal direction .
A mobile terminal, characterized in that it comprises:

Shell; and

The camera assembly according to any one of claims 11 to 20, which is combined with the housing.