WO2021046691A1 - 图像采集方法、摄像头组件及移动终端 - Google Patents

图像采集方法、摄像头组件及移动终端 Download PDF

Info

Publication number
WO2021046691A1
WO2021046691A1 PCT/CN2019/104974 CN2019104974W WO2021046691A1 WO 2021046691 A1 WO2021046691 A1 WO 2021046691A1 CN 2019104974 W CN2019104974 W CN 2019104974W WO 2021046691 A1 WO2021046691 A1 WO 2021046691A1
Authority
WO
WIPO (PCT)
Prior art keywords
color
pixels
image
pixel
panchromatic
Prior art date
Application number
PCT/CN2019/104974
Other languages
English (en)
French (fr)
Inventor
唐城
周奇群
张弓
Original Assignee
Oppo广东移动通信有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Oppo广东移动通信有限公司 filed Critical Oppo广东移动通信有限公司
Priority to KR1020227009653A priority Critical patent/KR20220051240A/ko
Priority to EP19944661.8A priority patent/EP4020971A4/en
Priority to PCT/CN2019/104974 priority patent/WO2021046691A1/zh
Priority to JP2022515700A priority patent/JP7298020B2/ja
Priority to CN201980097882.9A priority patent/CN114073068B/zh
Publication of WO2021046691A1 publication Critical patent/WO2021046691A1/zh
Priority to US17/584,813 priority patent/US20220150450A1/en

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T3/00—Geometric image transformation in the plane of the image
    • G06T3/40—Scaling the whole image or part thereof
    • G06T3/4015—Demosaicing, e.g. colour filter array [CFA], Bayer pattern
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
    • H04N23/80—Camera processing pipelines; Components thereof
    • H04N23/84—Camera processing pipelines; Components thereof for processing colour signals
    • H04N23/843—Demosaicing, e.g. interpolating colour pixel values
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T3/00—Geometric image transformation in the plane of the image
    • G06T3/40—Scaling the whole image or part thereof
    • G06T3/4007—Interpolation-based scaling, e.g. bilinear interpolation
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T5/00—Image enhancement or restoration
    • G06T5/50—Image enhancement or restoration by the use of more than one image, e.g. averaging, subtraction
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
    • H04N23/50—Constructional details
    • H04N23/54—Mounting of pick-up tubes, electronic image sensors, deviation or focusing coils
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
    • H04N23/70—Circuitry for compensating brightness variation in the scene
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
    • H04N23/70—Circuitry for compensating brightness variation in the scene
    • H04N23/73—Circuitry for compensating brightness variation in the scene by influencing the exposure time
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N25/00—Circuitry of solid-state image sensors [SSIS]; Control thereof
    • H04N25/10—Circuitry of solid-state image sensors [SSIS]; Control thereof for transforming different wavelengths into image signals
    • H04N25/11—Arrangement of colour filter arrays [CFA]; Filter mosaics
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N25/00—Circuitry of solid-state image sensors [SSIS]; Control thereof
    • H04N25/10—Circuitry of solid-state image sensors [SSIS]; Control thereof for transforming different wavelengths into image signals
    • H04N25/11—Arrangement of colour filter arrays [CFA]; Filter mosaics
    • H04N25/13—Arrangement of colour filter arrays [CFA]; Filter mosaics characterised by the spectral characteristics of the filter elements
    • H04N25/133—Arrangement of colour filter arrays [CFA]; Filter mosaics characterised by the spectral characteristics of the filter elements including elements passing panchromatic light, e.g. filters passing white light
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N25/00—Circuitry of solid-state image sensors [SSIS]; Control thereof
    • H04N25/40—Extracting pixel data from image sensors by controlling scanning circuits, e.g. by modifying the number of pixels sampled or to be sampled
    • H04N25/44—Extracting pixel data from image sensors by controlling scanning circuits, e.g. by modifying the number of pixels sampled or to be sampled by partially reading an SSIS array
    • H04N25/441—Extracting pixel data from image sensors by controlling scanning circuits, e.g. by modifying the number of pixels sampled or to be sampled by partially reading an SSIS array by reading contiguous pixels from selected rows or columns of the array, e.g. interlaced scanning
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N25/00—Circuitry of solid-state image sensors [SSIS]; Control thereof
    • H04N25/50—Control of the SSIS exposure
    • H04N25/53—Control of the integration time
    • H04N25/533—Control of the integration time by using differing integration times for different sensor regions
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N25/00—Circuitry of solid-state image sensors [SSIS]; Control thereof
    • H04N25/50—Control of the SSIS exposure
    • H04N25/57—Control of the dynamic range
    • H04N25/58—Control of the dynamic range involving two or more exposures
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N25/00—Circuitry of solid-state image sensors [SSIS]; Control thereof
    • H04N25/70—SSIS architectures; Circuits associated therewith
    • H04N25/702—SSIS architectures characterised by non-identical, non-equidistant or non-planar pixel layout
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N9/00—Details of colour television systems
    • H04N9/77—Circuits for processing the brightness signal and the chrominance signal relative to each other, e.g. adjusting the phase of the brightness signal relative to the colour signal, correcting differential gain or differential phase
    • H04N9/78—Circuits for processing the brightness signal and the chrominance signal relative to each other, e.g. adjusting the phase of the brightness signal relative to the colour signal, correcting differential gain or differential phase for separating the brightness signal or the chrominance signal from the colour television signal, e.g. using comb filter
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/10—Image acquisition modality
    • G06T2207/10024—Color image
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/10—Image acquisition modality
    • G06T2207/10048—Infrared image
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20212—Image combination
    • G06T2207/20221—Image fusion; Image merging

Definitions

  • This application relates to the field of imaging technology, and in particular to an image acquisition method, camera assembly and mobile terminal.
  • Mobile terminals such as mobile phones are often equipped with cameras to realize the camera function.
  • An image sensor is provided in the camera.
  • the image sensor In order to realize the collection of color images, the image sensor is usually provided with color pixels, and the color pixels are arranged in a Bayer array.
  • white pixels with higher sensitivity than color pixels are added to the image sensor in related technologies.
  • This application provides an image acquisition method, camera assembly and mobile terminal.
  • the image sensor includes a two-dimensional pixel array, the two-dimensional pixel array includes a plurality of panchromatic pixels and a plurality of color pixels, the two-dimensional pixel array includes a minimum repeating unit, each of the minimum repeating unit includes a plurality of sub-units Each of the sub-units includes a plurality of single-color pixels and a plurality of full-color pixels;
  • the image acquisition method includes: controlling the exposure of the two-dimensional pixel array to obtain a full-color original image and a color original image; and processing the color For the original image, use all the pixels of each subunit as the single-color large pixels corresponding to the single color in the sub-unit, and output the pixel values of the single-color large pixels to obtain a color intermediate image; process the full Color the original image to obtain a full-color intermediate image; and process the color intermediate image and/or the full-color intermediate image to obtain a target image.
  • the present application also provides a camera assembly.
  • the camera assembly includes an image sensor and a processing chip.
  • the image sensor includes a two-dimensional pixel array, the two-dimensional pixel array includes a plurality of panchromatic pixels and a plurality of color pixels, the two-dimensional pixel array includes a minimum repeating unit, each of the minimum repeating unit includes a plurality of sub-units Each of the sub-units includes a plurality of single-color pixels and a plurality of full-color pixels, and the image sensor is used for exposure to obtain a full-color original image and a color original image.
  • the processing chip is used to process the color original image to use all the pixels of each of the subunits as monochromatic large pixels corresponding to a single color in the subunit, and output the pixels of the monochromatic large pixels Processing the full-color original image to obtain a full-color intermediate image; processing the color intermediate image and/or the full-color intermediate image to obtain a target image.
  • this application also provides a mobile terminal.
  • the mobile terminal includes an image sensor and a processor.
  • the image sensor includes a two-dimensional pixel array, the two-dimensional pixel array includes a plurality of panchromatic pixels and a plurality of color pixels, the two-dimensional pixel array includes a minimum repeating unit, each of the minimum repeating unit includes a plurality of sub-units Each of the sub-units includes a plurality of single-color pixels and a plurality of full-color pixels, and the image sensor is used for exposure to obtain a full-color original image and a color original image.
  • the processor is configured to process the color original image to use all the pixels of each sub-unit as a single-color large pixel corresponding to a single color in the sub-unit, and output the pixel value of the single-color large pixel To obtain a color intermediate image; process the full-color original image to obtain a full-color intermediate image; process the color intermediate image and/or the full-color intermediate image to obtain a target image.
  • Fig. 1 is a schematic diagram of a camera assembly according to an embodiment of the present application
  • Fig. 2 is a schematic diagram of an image sensor in an embodiment of the present application.
  • FIG. 3 is a schematic diagram of the connection mode of the pixel array and the exposure control line in the embodiment of the present application;
  • Figure 4 is a schematic diagram of exposure saturation time for different color channels
  • FIG. 5 is a schematic diagram of a pixel circuit in an embodiment of the present application.
  • Fig. 6 is a schematic diagram of a minimum repeating unit pixel arrangement in an embodiment of the present application.
  • FIG. 7 is a schematic diagram of another minimum repeating unit pixel arrangement in an embodiment of the present application.
  • FIG. 8 is a schematic diagram of another minimum repeating unit pixel arrangement in an embodiment of the present application.
  • FIG. 9 is a schematic diagram of another minimum repeating unit pixel arrangement in an embodiment of the present application.
  • FIG. 10 is a schematic diagram of another minimum repeating unit pixel arrangement in an embodiment of the present application.
  • FIG. 11 is a schematic diagram of another minimum repeating unit pixel arrangement in an embodiment of the present application.
  • FIG. 12 is a schematic diagram of another minimum repeating unit pixel arrangement in an embodiment of the present application.
  • FIG. 13 is a schematic diagram of another minimum repeating unit pixel arrangement in an embodiment of the present application.
  • FIG. 14 is a schematic diagram of another minimum repeating unit pixel arrangement in an embodiment of the present application.
  • FIG. 15 is a schematic diagram of another minimum repeating unit pixel arrangement in an embodiment of the present application.
  • FIG. 16 is a schematic diagram of another minimum repeating unit pixel arrangement in an embodiment of the present application.
  • FIG. 17 is a schematic diagram of another minimum repeating unit pixel arrangement in an embodiment of the present application.
  • FIG. 18 is a schematic diagram of another minimum repeating unit pixel arrangement in an embodiment of the present application.
  • FIG. 19 is a schematic diagram of another minimum repeating unit pixel arrangement in an embodiment of the present application.
  • FIG. 20 is a schematic diagram of another minimum repeating unit pixel arrangement in an embodiment of the present application.
  • FIG. 21 is a schematic diagram of another minimum repeating unit pixel arrangement in an embodiment of the present application.
  • FIG. 22 is a schematic diagram of the principle of an image acquisition method in the related art.
  • FIG. 23 is a schematic flowchart of an image acquisition method according to some embodiments of the present application.
  • FIG. 24 is a schematic diagram of a principle of a light image acquisition method in an embodiment of the present application.
  • FIG. 25 is another schematic diagram of the principle of the optical image acquisition method in the embodiment of the present application.
  • 26 to 29 are schematic flowcharts of image acquisition methods according to some embodiments of the present application.
  • FIG. 30 is another schematic diagram of the principle of the optical image acquisition method in the embodiment of the present application.
  • FIG. 31 is another schematic diagram of the principle of the optical image acquisition method in the embodiment of the present application.
  • FIG. 32 is another schematic diagram of the principle of the optical image acquisition method in the embodiment of the present application.
  • FIG. 33 is another schematic diagram of the principle of the optical image acquisition method in the embodiment of the present application.
  • FIG. 34 is another schematic diagram of the principle of the optical image acquisition method in the embodiment of the present application.
  • FIG. 35 is a schematic diagram of a mobile terminal according to an embodiment of the present application.
  • the image sensor includes a two-dimensional pixel array.
  • the two-dimensional pixel array includes a plurality of panchromatic pixels and a plurality of color pixels.
  • the two-dimensional pixel array includes a minimum repeating unit, each minimum repeating unit includes a plurality of sub-units, and each sub-unit includes a plurality of single-color pixels and a plurality of full-color pixels.
  • the image acquisition method includes: controlling the exposure of the two-dimensional pixel array to obtain the full-color original image and the color original image; processing the color original image to treat all the pixels of each subunit as the single-color large pixels corresponding to the single color in the subunit, And output the pixel value of the monochromatic large pixel to obtain the color intermediate image; process the full-color original image to obtain the full-color intermediate image; process the color intermediate image and/or the full-color intermediate image to obtain the target image.
  • the camera assembly includes an image sensor and a processing chip.
  • the image sensor includes a two-dimensional pixel array.
  • the two-dimensional pixel array includes a plurality of panchromatic pixels and a plurality of color pixels.
  • the two-dimensional pixel array includes a minimum repeating unit, each minimum repeating unit includes a plurality of sub-units, and each sub-unit includes a plurality of single-color pixels and a plurality of full-color pixels.
  • the image sensor is used for exposure to obtain full-color original images and color original images.
  • the processing chip is used to: process the color original image, take all the pixels of each subunit as the single color large pixel corresponding to the single color in the subunit, and output the pixel value of the single color large pixel to obtain the color intermediate image; Color the original image to obtain a full-color intermediate image; process the color intermediate image and/or the full-color intermediate image to obtain the target image.
  • the mobile terminal includes an image sensor and a processor.
  • the image sensor includes a two-dimensional pixel array.
  • the two-dimensional pixel array includes a plurality of panchromatic pixels and a plurality of color pixels.
  • the two-dimensional pixel array includes a minimum repeating unit, each minimum repeating unit includes a plurality of sub-units, and each sub-unit includes a plurality of single-color pixels and a plurality of full-color pixels.
  • the image sensor is used for exposure to obtain full-color original images and color original images.
  • the processor is used to: process the color original image, take all the pixels of each sub-unit as the single-color large pixel corresponding to the single color in the sub-unit, and output the pixel value of the single-color large pixel to obtain the color intermediate image; process the full color
  • the original image is used to obtain a full-color intermediate image; the color intermediate image and/or the full-color intermediate image are processed to obtain the target image.
  • the camera assembly 40 includes an image sensor 10, a processing chip 20 and a lens 30.
  • the image sensor 10 is electrically connected to the processing chip 20.
  • the lens 30 is provided on the optical path of the image sensor 10.
  • the processing chip 20 and the image sensor 10 and the lens 30 may be packaged in the same housing of the camera assembly 40; or the image sensor 10 and the lens 30 are packaged in the housing, and the processing chip 20 is arranged outside the housing.
  • FIG. 2 is a schematic diagram of the image sensor 10 in an embodiment of the present application.
  • the image sensor 10 includes a pixel array 11, a vertical driving unit 12, a control unit 13, a column processing unit 14 and a horizontal driving unit 15.
  • the image sensor 10 may adopt a complementary metal oxide semiconductor (CMOS, Complementary Metal Oxide Semiconductor) photosensitive element or a charge-coupled device (CCD, Charge-coupled Device) photosensitive element.
  • CMOS complementary metal oxide semiconductor
  • CCD Charge-coupled Device
  • the pixel array 11 includes a plurality of pixels (not shown in FIG. 2) arranged two-dimensionally in an array, and each pixel includes a photoelectric conversion element. Each pixel converts light into electric charge according to the intensity of light incident on it.
  • the vertical driving unit 12 includes a shift register and an address decoder.
  • the vertical drive unit 12 includes readout scanning and reset scanning functions. Readout scanning refers to sequentially scanning unit pixels line by line, and reading signals from these unit pixels line by line. For example, the signal output by each pixel in the pixel row that is selected and scanned is transmitted to the column processing unit 14.
  • the reset scan is used to reset the charge, and the photocharge of the photoelectric conversion element is discarded, so that the accumulation of new photocharge can be started.
  • the signal processing performed by the column processing unit 14 is correlated double sampling (CDS) processing.
  • CDS correlated double sampling
  • the reset level and the signal level output from each pixel in the selected pixel row are taken out, and the level difference is calculated.
  • A/D analog-to-digital
  • the horizontal driving unit 15 includes a shift register and an address decoder.
  • the horizontal driving unit 15 sequentially scans the pixel array 11 column by column. Through the selection scanning operation performed by the horizontal driving unit 15, each pixel column is sequentially processed by the column processing unit 14, and is sequentially output.
  • control unit 13 configures timing signals according to the operation mode, and uses multiple timing signals to control the vertical driving unit 13, the column processing unit 14, and the horizontal driving unit 15 to work together.
  • FIG. 3 is a schematic diagram of the connection mode of the pixel array 11 and the exposure control line in the embodiment of the present application.
  • the pixel array 11 is a two-dimensional pixel array.
  • the two-dimensional pixel array includes a plurality of panchromatic pixels and a plurality of color pixels, wherein the color pixels have a narrower spectral response than the panchromatic pixels.
  • the arrangement of pixels in the pixel array 11 is as follows:
  • pixels 1101, 1103, 1106, 1108, 1111, 1113, 1116, and 1118 are full-color pixels W
  • pixels 1102, 1105 are pixels A of the first color (for example, red pixels R)
  • pixels 1104, 1107 , 1112 and 1115 are the second color pixel B (for example, the green pixel G)
  • the pixels 1114 and 1117 are the third color pixel C (for example, the blue pixel Bu).
  • the control terminal TG of the exposure control circuit in the panchromatic pixel W (pixels 1101, 1103, 1106, and 1108) is connected to a first exposure control line TX1, and the panchromatic pixel W (1111, 1113, 1116) , And 1118) the control terminal TG of the exposure control circuit is connected to another first exposure control line TX1; the control terminal TG of the exposure control circuit in the first color pixel A (pixels 1102 and 1105), the second color pixel B (pixel 1104, 1107) the control terminal TG of the exposure control circuit is connected to a second exposure control line TX2, the control terminal TG of the exposure control circuit in the second color pixel B (pixels 1112, 1115), the third color pixel C (pixel 1114) , 1117) The control terminal TG of the exposure control circuit is connected to another second exposure control line TX2.
  • Each first exposure control line TX1 can control the exposure duration of the panchromatic pixel through the first exposure control signal; each second exposure control line TX2 can control the color pixels (such as the first color pixel A and the first color pixel A and the first color pixel A) through the second exposure control signal.
  • pixels of different colors receive different exposures per unit time. After some colors are saturated, some colors have not yet been exposed to an ideal state. For example, exposure to 60%-90% of the saturated exposure may have a relatively good signal-to-noise ratio and accuracy, but the embodiments of the present application are not limited thereto.
  • RGBW red, green, blue, full color
  • the horizontal axis is the exposure time
  • the vertical axis is the exposure
  • Q is the saturated exposure
  • LW is the exposure curve of the panchromatic pixel W
  • LG is the exposure curve of the green pixel G
  • LR is the exposure curve of the red pixel R
  • LB is the exposure curve of the blue pixel.
  • the slope of the exposure curve LR of the red pixel R is again the same, and the red pixel R is saturated at time t3.
  • the slope of the exposure curve LB of the blue pixel B is the smallest, and the blue pixel B is saturated at t4.
  • the panchromatic pixel W has been saturated, but the exposure of the three pixels R, G, and B has not yet reached the ideal state.
  • the exposure time of the four RGBW pixels is jointly controlled.
  • the exposure time of each row of pixels is the same, connected to the same exposure control line, and controlled by the same exposure control signal.
  • the exposure control signal For example, continue to refer to Figure 4, during the time period 0-t1, all four pixels of RGBW can work normally, but in this interval RGB due to the shorter exposure time and less exposure, the image will be displayed with lower brightness and confidence. A phenomenon in which the noise ratio is low, and even the colors are not bright enough.
  • the W pixels are overexposed due to saturation and cannot work, and the exposure data can no longer truly reflect the target.
  • the image sensor 10 (shown in FIG. 2) provided by the present application can reduce the exposure time limit of the panchromatic pixel W and balance the panchromatic pixel by independently controlling the exposure time of the panchromatic pixel W and the exposure time of the color pixel.
  • Exposure of W and color pixels (including but not limited to RGB) to improve image capture quality.
  • Fig. 3 is an example of independently controlling the exposure time of the panchromatic pixel W and the exposure time of the color pixel.
  • the independent exposure control of the panchromatic pixel W and the color pixel is realized through different exposure control lines, thereby improving the image Shooting quality.
  • the exposure curve in FIG. 4 is only an example, and the slope and the relative relationship of the curve will vary according to the different pixel response bands, and the application is not limited to the situation shown in FIG. 4.
  • the slope of the exposure curve of the red pixel R may be lower than the slope of the exposure curve of the blue pixel B.
  • the first exposure control line TX1 and the second exposure control line TX2 are connected to the vertical driving unit 12 in FIG. 2, and the corresponding exposure control signals in the vertical driving unit 12 are transmitted to the pixels in the pixel array 11
  • the control terminal TG of the exposure control circuit is connected to the vertical driving unit 12 in FIG. 2, and the corresponding exposure control signals in the vertical driving unit 12 are transmitted to the pixels in the pixel array 11
  • the control terminal TG of the exposure control circuit is the exposure control circuit.
  • the vertical driving unit 12 connects multiple first exposure control lines TX1 and multiple second exposure control lines TX2.
  • the plurality of first exposure control lines TX1 and the plurality of second exposure control lines TX2 correspond to corresponding pixel row groups.
  • the first first exposure control line TX1 corresponds to the panchromatic pixels in the first and second rows; the second first exposure control line TX1 corresponds to the panchromatic pixels in the third and fourth rows, so
  • the third first exposure control line TX1 corresponds to the panchromatic pixels in the fifth and sixth rows;
  • the fourth first exposure control line TX1 corresponds to the panchromatic pixels in the seventh and eighth rows, and then down
  • the corresponding relationship between the first exposure control line TX1 and the panchromatic pixels downstream will not be repeated here.
  • the signal timings transmitted by different first exposure control lines TX1 are also different, and the signal timings are configured by the vertical driving unit 12.
  • the first second exposure control line TX2 corresponds to the color pixels in the first and second rows; the second second exposure control line TX2 corresponds to the color pixels in the third and fourth rows, and so on,
  • the third second exposure control line TX2 corresponds to the color pixels in the fifth and sixth rows; the fourth second exposure control line TX2 corresponds to the color pixels in the seventh and eighth rows, and then the second exposure
  • the corresponding relationship between the control line TX2 and the downstream color pixels will not be repeated.
  • the timing of the signal transmitted by the different second exposure control line TX2 is also different, and the timing of the signal is also configured by the vertical driving unit 12.
  • FIG. 5 is a schematic diagram of a pixel circuit 110 in an embodiment of the present application.
  • the pixel circuit 110 in FIG. 5 is applied to each pixel in FIG. 3.
  • the working principle of the pixel circuit 110 will be described below in conjunction with FIG. 3 and FIG. 5.
  • the pixel circuit 110 includes a photoelectric conversion element 117 (e.g., photodiode PD), an exposure control circuit 116 (e.g., transfer transistor 112), a reset circuit (e.g., reset transistor 113), and an amplifier circuit (e.g., amplifier The transistor 114) and the selection circuit (for example, the selection transistor 115).
  • the transfer transistor 112, the reset transistor 113, the amplifying transistor 114, and the selection transistor 115 are, for example, MOS transistors, but are not limited thereto.
  • the gate TG of the transfer transistor 112 is connected to the vertical drive unit 12 through an exposure control line; the gate RG of the reset transistor 113 is connected to the vertical drive through a reset control line (not shown in the figure) Unit 12; the gate SEL of the selection transistor 114 is connected to the vertical driving unit 12 through a selection line (not shown in the figure).
  • the exposure control circuit 116 (for example, the transfer transistor 112) in each pixel circuit 110 is electrically connected to the photoelectric conversion element 117 for transferring the potential accumulated by the photoelectric conversion element 117 after being irradiated with light.
  • the photoelectric conversion element 117 includes a photodiode PD, and the anode of the photodiode PD is connected to the ground, for example.
  • the photodiode PD converts the received light into electric charge.
  • the cathode of the photodiode PD is connected to the floating diffusion unit FD via the exposure control circuit 116 (for example, the transfer transistor 112).
  • the floating diffusion unit FD is connected to the gate of the amplifying transistor 114 and the source of the reset transistor 113.
  • the exposure control circuit 116 is the transfer transistor 112, and the control terminal TG of the exposure control circuit 116 is the gate of the transfer transistor 112.
  • an effective level for example, VPIX level
  • the transfer transistor 112 is turned on.
  • the transfer transistor 112 transfers the charge photoelectrically converted by the photodiode PD to the floating diffusion unit FD.
  • the drain of the reset transistor 113 is connected to the pixel power supply VPIX.
  • the source of the reset transistor 113 is connected to the floating diffusion unit FD.
  • the pulse of the effective reset level is transmitted to the gate of the reset transistor 113 via the reset line, and the reset transistor 113 is turned on.
  • the reset transistor 113 resets the floating diffusion unit FD to the pixel power supply VPIX.
  • the gate of the amplifying transistor 114 is connected to the floating diffusion unit FD.
  • the drain of the amplifying transistor 114 is connected to the pixel power supply VPIX.
  • the amplifying transistor 114 After the floating diffusion unit FD is reset by the reset transistor 113, the amplifying transistor 114 outputs the reset level through the output terminal OUT via the selection transistor 115.
  • the amplifying transistor 114 After the charge of the photodiode PD is transferred by the transfer transistor 112, the amplifying transistor 114 outputs a signal level through the output terminal OUT via the selection transistor 115.
  • the drain of the selection transistor 115 is connected to the source of the amplifying transistor 114.
  • the source of the selection transistor 115 is connected to the column processing unit 14 in FIG. 2 through the output terminal OUT.
  • the selection transistor 115 is turned on.
  • the signal output by the amplification transistor 114 is transmitted to the column processing unit 14 through the selection transistor 115.
  • the pixel structure of the pixel circuit 110 in the embodiment of the present application is not limited to the structure shown in FIG. 5.
  • the pixel circuit 110 may have a three-transistor pixel structure, in which the functions of the amplifying transistor 114 and the selecting transistor 115 are performed by one transistor.
  • the exposure control circuit 116 is not limited to a single transfer transistor 112, and other electronic devices or structures with a control terminal to control the conduction function can be used as the exposure control circuit in the embodiment of the present application.
  • the implementation of the single transfer transistor 112 Simple, low cost, and easy to control.
  • the image sensor 10 includes a plurality of color pixels (for example, a plurality of first color pixels A, a plurality of second color pixels B, and a plurality of third color pixels C) and a plurality of full color pixels.
  • a two-dimensional pixel array composed of color pixels W that is, the pixel array 11 shown in FIG. 3).
  • color pixels have a narrower spectral response than panchromatic pixels.
  • the response spectrum of the color pixel is, for example, a part of the W response spectrum of the panchromatic pixel.
  • the two-dimensional pixel array includes the smallest repeating unit ( Figures 6 to 21 show examples of the smallest repeating unit of pixels in various image sensors 10).
  • the two-dimensional pixel array is composed of multiple smallest repeating units.
  • the smallest repeating unit is in rows and columns. Copy and arrange on top.
  • the panchromatic pixels W are arranged in the first diagonal direction D1
  • the color pixels are arranged in the second diagonal direction D2
  • the first diagonal direction D1 is different from the second diagonal direction D2.
  • the first exposure time of at least two full-color pixels adjacent in the first diagonal direction D1 is controlled by the first exposure signal
  • the second exposure time of at least two color pixels adjacent in the second diagonal direction D2 is controlled by the first Two exposure signal control, so as to realize the independent control of the exposure time of panchromatic pixels and the exposure time of color pixels.
  • Each minimum repeating unit includes a plurality of sub-units, and each sub-unit includes a plurality of single-color pixels (for example, a plurality of first-color pixels A, a plurality of second-color pixels B, or a plurality of third-color pixels C) and a plurality of full-color pixels.
  • Color pixel W is a plurality of sub-units
  • the pixels 1101-1108 and the pixels 1111-1118 form a minimum repeating unit, where the pixels 1101, 1103, 1106, 1108, 1111, 1113, 1116, and 1118 are panchromatic pixels, and the pixel 1102 , 1104, 1105, 1107, 1112, 1114, 1115, and 1117 are color pixels.
  • Pixels 1101, 1102, 1105, and 1106 form a sub-unit, where pixels 1101, 1106 are full-color pixels, and pixels 1102, 1105 are single-color pixels (for example, the first color pixel A); pixels 1103, 1104, 1107, 1108 The pixels 1103 and 1108 are full-color pixels, and the pixels 1104 and 1107 are single-color pixels (for example, the second color pixel B); the pixels 1111, 1112, 1115, and 1116 form a sub-unit.
  • pixels 1111, 1116 are full-color pixels, and pixels 1112, 1115 are single-color pixels (for example, second-color pixel B); pixels 1113, 1114, 1117, and 1118 form a sub-unit, among which, pixels 1113 and 1118 are full-color pixels,
  • the pixels 1114 and 1117 are single-color pixels (for example, the third-color pixel C).
  • the number of pixels in the rows and columns of the minimum repeating unit is equal.
  • the smallest repeating unit includes, but is not limited to, a smallest repeating unit of 4 rows and 4 columns, 6 rows and 6 columns, 8 rows and 8 columns, and 10 rows and 10 columns.
  • the number of pixels in the rows and columns of sub-units in the smallest repeating unit is equal.
  • subunits include, but are not limited to, subunits with 2 rows and 2 columns, 3 rows and 3 columns, 4 rows and 4 columns, and 5 rows and 5 columns. This setting helps to balance the resolution and color performance of the image in the row and column directions, and improve the display effect.
  • FIG. 6 is a schematic diagram of a minimum repeating unit 1181 pixel arrangement in an embodiment of the present application; the minimum repeating unit has 4 rows, 4 columns and 16 pixels, and the subunits have 2 rows, 2 columns and 4 pixels.
  • the arrangement is as follows:
  • W represents a full-color pixel
  • A represents a first color pixel among multiple color pixels
  • B represents a second color pixel among multiple color pixels
  • C represents a third color pixel among multiple color pixels.
  • the panchromatic pixels W are arranged in the first diagonal direction D1 (that is, the direction connecting the upper left corner and the lower right corner in FIG. 6), and the color pixels are arranged in the second diagonal direction D2 (for example, as shown in FIG. The direction connecting the lower left corner and the upper right corner in 6), the first diagonal direction D1 is different from the second diagonal direction D2.
  • the first diagonal line and the second diagonal line are perpendicular.
  • the first exposure time of two adjacent panchromatic pixels W in the first diagonal direction D1 (for example, two panchromatic pixels in the first row, first column and second row and second column from the upper left) is determined by the first exposure time
  • One exposure signal control at least two color pixels adjacent in the second diagonal direction D2 (for example, two color pixels B in the fourth row, first column and third row and second column from the upper left)
  • the exposure time is controlled by the second exposure signal.
  • first diagonal direction D1 and the second diagonal direction D2 are not limited to the diagonal, but also include directions parallel to the diagonal.
  • the panchromatic pixels 1101, 1106, 1113 and 1118 are arranged in the first diagonal direction D1
  • the panchromatic pixels 1103 and 1108 are also arranged in the first diagonal direction D1
  • the panchromatic pixels 1111 and 1116 are also arranged in the first diagonal direction D1
  • the color pixels 1104, 1107, 1112, and 1115 are arranged in the second diagonal direction D2
  • the first color pixels 1102 and 1105 are also arranged in the second diagonal direction D2
  • the third color pixels 1114 and 1117 are also arranged in the second diagonal direction.
  • the diagonal direction D2 is not a single direction, but can be understood as the concept of a "straight line” indicating the arrangement, and there can be two-way directions at both ends of the straight line.
  • the panchromatic pixels in the first row and the second row are connected together by a first exposure control line TX1 in a "W" shape, so as to realize individual control of the exposure time of the panchromatic pixels.
  • the color pixels (A and B) of the first row and the second row are connected together by a second exposure control line TX2 in a "W" shape to realize individual control of the exposure time of the color pixels.
  • the panchromatic pixels in the third row and the fourth row are connected together by the first exposure control line TX1 in the shape of "W", so as to realize the individual control of the exposure time of the panchromatic pixels.
  • the color pixels (B and C) in the third row and the fourth row are connected together by a second exposure control line TX2 in a "W" shape to realize individual control of the exposure time of the color pixels.
  • the first exposure signal is transmitted via the first exposure control line TX1
  • the second exposure signal is transmitted via the second exposure control line TX2.
  • the first exposure control line TX1 is in the shape of "W” and is electrically connected to the control terminal of the exposure control circuit in the panchromatic pixels in two adjacent rows
  • the second exposure control line TX2 is in the shape of "W” and is connected to the two adjacent rows.
  • the control terminal of the exposure control circuit in the color pixel is electrically connected.
  • the specific connection method please refer to the description of the connection and the pixel circuit in the relevant part of FIG. 3 and FIG. 5.
  • the "W" shape of the first exposure control line TX1 and the second exposure control line TX2 does not mean that the physical wiring must be set in strict accordance with the "W” shape, only the connection mode corresponds to the full-color pixel and color
  • the arrangement of the pixels is sufficient.
  • the setting of the "W" type exposure control line corresponds to the "W" type pixel arrangement method. This setting method is simple to route, and the pixel arrangement has good resolution and color effects, and realizes full color at low cost. Independent control of pixel exposure time and color pixel exposure time.
  • FIG. 7 is a schematic diagram of another minimum repeating unit 1182 pixel arrangement in the embodiment of the present application.
  • the minimum repeating unit is 4 rows, 4 columns and 16 pixels, and the subunits are 2 rows, 2 columns and 4 pixels.
  • the arrangement is as follows:
  • W represents a full-color pixel
  • A represents a first color pixel among multiple color pixels
  • B represents a second color pixel among multiple color pixels
  • C represents a third color pixel among multiple color pixels.
  • the panchromatic pixel W is arranged in the first diagonal direction D1 (that is, the direction connecting the upper right corner and the lower left corner in FIG. 7), and the color pixels are arranged in the second diagonal direction D2 (for example, as shown in FIG. The direction of the connection between the upper left corner and the lower right corner in 7).
  • the first diagonal line and the second diagonal line are perpendicular.
  • the first exposure time of two adjacent panchromatic pixels W in the first diagonal direction D1 (for example, two panchromatic pixels in the first row, second column, and second row, first column from the upper left) is determined by the first exposure time
  • One exposure signal control second exposure of at least two color pixels adjacent in the second diagonal direction (for example, two color pixels A in the first row, first column and second row and second column from the upper left)
  • the time is controlled by the second exposure signal.
  • the panchromatic pixels in the first row and the second row are connected together by a first exposure control line TX1 in a "W" shape, so as to realize individual control of the exposure time of the panchromatic pixels.
  • the color pixels (A and B) of the first row and the second row are connected together by a second exposure control line TX2 in a "W" shape to realize individual control of the exposure time of the color pixels.
  • the panchromatic pixels in the third row and the fourth row are connected together by the first exposure control line TX1 in the shape of "W", so as to realize the individual control of the exposure time of the panchromatic pixels.
  • the color pixels (B and C) in the third row and the fourth row are connected together by a second exposure control line TX2 in a "W" shape to realize individual control of the exposure time of the color pixels.
  • FIG. 8 is a schematic diagram of another minimum repeating unit 1183 pixel arrangement in the embodiment of the present application.
  • FIG. 9 is a schematic diagram of another minimum repeating unit 1184 pixel arrangement in the embodiment of the present application.
  • the first color pixel A is a red pixel R
  • the second color pixel B is a green pixel G
  • the third color pixel C is a blue pixel.
  • Color pixel Bu is a schematic diagram of another minimum repeating unit 1184 pixel arrangement in the embodiment of the present application.
  • the response band of the panchromatic pixel W is the visible light band (for example, 400 nm-760 nm).
  • the panchromatic pixel W is provided with an infrared filter to filter out infrared light.
  • the response wavelength band of the panchromatic pixel W is the visible light wavelength band and the near-infrared wavelength band (for example, 400 nm-1000 nm), which matches the response wavelength band of the photoelectric conversion element (for example, photodiode PD) in the image sensor 10.
  • the panchromatic pixel W may not be provided with a filter, and the response band of the panchromatic pixel W is determined by the response band of the photodiode, that is, the two match.
  • the embodiments of the present application include but are not limited to the above-mentioned waveband range.
  • FIG. 10 is a schematic diagram of another minimum repeating unit 1185 pixel arrangement in the embodiment of the present application.
  • FIG. 11 is a schematic diagram of another minimum repeating unit 1186 pixel arrangement in the embodiment of the present application.
  • the first color pixel A is a red pixel R
  • the second color pixel B is a yellow pixel Y
  • the third color pixel C is a blue pixel.
  • Color pixel Bu is a schematic diagram of another minimum repeating unit 1185 pixel arrangement in the embodiment of the present application.
  • FIG. 11 is a schematic diagram of another minimum repeating unit 1186 pixel arrangement in the embodiment of the present application.
  • the first color pixel A is a red pixel R
  • the second color pixel B is a yellow pixel Y
  • the third color pixel C is a blue pixel.
  • Color pixel Bu is a blue pixel.
  • FIG. 12 is a schematic diagram of another minimum repeating unit 1187 pixel arrangement in the embodiment of the present application.
  • FIG. 13 is a schematic diagram of another minimum repeating unit 1188 pixel arrangement in the embodiment of the present application.
  • the first color pixel A is magenta pixel M
  • the second color pixel B is cyan pixel Cy
  • the third color pixel C is Yellow pixel Y.
  • FIG. 14 is a schematic diagram of another minimum repeating unit 1191 pixel arrangement in the embodiment of the present application.
  • the smallest repeating unit is 36 pixels in 6 rows and 6 columns, and the sub-units are 9 pixels in 3 rows, 3 columns, and the arrangement is:
  • W represents a full-color pixel
  • A represents a first color pixel among multiple color pixels
  • B represents a second color pixel among multiple color pixels
  • C represents a third color pixel among multiple color pixels.
  • the panchromatic pixels in the first row and the second row are connected together by a first exposure control line TX1 in a "W" shape to realize individual control of the exposure time of the panchromatic pixels.
  • the color pixels (A and B) of the first row and the second row are connected together by a second exposure control line TX2 in a "W" shape to realize individual control of the exposure time of the color pixels.
  • the panchromatic pixels in the third row and the fourth row are connected together by the first exposure control line TX1 in the shape of "W", so as to realize the individual control of the exposure time of the panchromatic pixels.
  • the color pixels (A, B, and C) of the third row and the fourth row are connected together by a second exposure control line TX2 in a "W" shape to realize individual control of the exposure time of the color pixels.
  • the panchromatic pixels in the fifth row and the sixth row are connected together by the first exposure control line TX1 in the shape of "W” to realize the individual control of the exposure time of the panchromatic pixels.
  • the color pixels (B and C) in the fifth row and the sixth row are connected together by a second exposure control line TX2 in a "W" shape to realize individual control of the exposure time of the color pixels.
  • FIG. 15 is a schematic diagram of another minimum repeating unit 1192 pixel arrangement in the embodiment of the present application.
  • the smallest repeating unit is 36 pixels in 6 rows and 6 columns, and the sub-units are 9 pixels in 3 rows, 3 columns, and the arrangement is:
  • W represents a full-color pixel
  • A represents a first color pixel among multiple color pixels
  • B represents a second color pixel among multiple color pixels
  • C represents a third color pixel among multiple color pixels.
  • the panchromatic pixels in the first row and the second row are connected together by the first exposure control line TX1 in a "W" shape, so as to realize individual control of the exposure time of the panchromatic pixels.
  • the color pixels (A and B) of the first row and the second row are connected together by a second exposure control line TX2 in a "W" shape to realize individual control of the exposure time of the color pixels.
  • the panchromatic pixels in the third row and the fourth row are connected together by the first exposure control line TX1 in the shape of "W", so as to realize the individual control of the exposure time of the panchromatic pixels.
  • the color pixels (A, B, and C) of the third row and the fourth row are connected together by a second exposure control line TX2 in a "W" shape to realize individual control of the exposure time of the color pixels.
  • the panchromatic pixels in the fifth row and the sixth row are connected together by the first exposure control line TX1 in the shape of "W” to realize the individual control of the exposure time of the panchromatic pixels.
  • the color pixels (B and C) in the fifth row and the sixth row are connected together by a second exposure control line TX2 in a "W" shape to realize individual control of the exposure time of the color pixels.
  • FIG. 16 is a schematic diagram of another minimum repeating unit 1193 pixel arrangement in the embodiment of the present application.
  • FIG. 17 is a schematic diagram of another minimum repeating unit 1194 pixel arrangement in an embodiment of the present application.
  • the first color pixel A is a red pixel R
  • the second color pixel B is a green pixel G
  • the third color pixel C is a blue pixel.
  • Color pixel Bu is a schematic diagram of another minimum repeating unit 1193 pixel arrangement in the embodiment of the present application.
  • FIG. 17 is a schematic diagram of another minimum repeating unit 1194 pixel arrangement in an embodiment of the present application.
  • the first color pixel A is a red pixel R
  • the second color pixel B is a green pixel G
  • the third color pixel C is a blue pixel.
  • Color pixel Bu is a blue pixel.
  • the first color pixel A is a red pixel R; the second color pixel B is a yellow pixel Y; and the third color pixel C is a blue pixel Bu.
  • the first color pixel A is a magenta pixel M; the second color pixel B is a cyan pixel Cy; and the third color pixel C is a yellow pixel Y.
  • the embodiments of the present application include but are not limited to this. Please refer to the above description for the specific connection mode of the circuit, which will not be repeated here.
  • FIG. 18 is a schematic diagram of another minimum repeating unit 1195 pixel arrangement in the embodiment of the present application.
  • the smallest repeating unit is 8 rows, 8 columns and 64 pixels, and the sub-units are 4 rows, 4 columns and 16 pixels.
  • the arrangement is:
  • W represents a full-color pixel
  • A represents a first color pixel among multiple color pixels
  • B represents a second color pixel among multiple color pixels
  • C represents a third color pixel among multiple color pixels.
  • the panchromatic pixels in the first row and the second row are connected together by a first exposure control line TX1 in a "W" shape, so as to realize individual control of the exposure time of the panchromatic pixels.
  • the color pixels (A and B) of the first row and the second row are connected together by a second exposure control line TX2 in a "W" shape to realize individual control of the exposure time of the color pixels.
  • the panchromatic pixels in the third row and the fourth row are connected together by the first exposure control line TX1 in the shape of "W", so as to realize the individual control of the exposure time of the panchromatic pixels.
  • the color pixels (A and B) in the third row and the fourth row are connected together by a second exposure control line TX2 in a "W" shape to realize individual control of the exposure time of the color pixels.
  • the panchromatic pixels in the fifth row and the sixth row are connected together by the first exposure control line TX1 in the shape of "W” to realize the individual control of the exposure time of the panchromatic pixels.
  • the color pixels (B and C) in the fifth row and the sixth row are connected together by a second exposure control line TX2 in a "W" shape to realize individual control of the exposure time of the color pixels.
  • the panchromatic pixels in the seventh row and the eighth row are connected together by the first exposure control line TX1 in the shape of "W” to realize individual control of the exposure time of the panchromatic pixels.
  • the color pixels (B and C) in the seventh row and the eighth row are connected together by a second exposure control line TX2 in a "W" shape to realize individual control of the exposure time of the color pixels.
  • FIG. 19 is a schematic diagram of another minimum repeating unit 1196 pixel arrangement in the embodiment of the present application.
  • the smallest repeating unit is 8 rows, 8 columns and 64 pixels, and the sub-units are 4 rows, 4 columns and 16 pixels.
  • the arrangement is:
  • W represents a full-color pixel
  • A represents a first color pixel among multiple color pixels
  • B represents a second color pixel among multiple color pixels
  • C represents a third color pixel among multiple color pixels.
  • the panchromatic pixels in the first row and the second row are connected together by the first exposure control line TX1 in the shape of "W” to realize individual control of the exposure time of the panchromatic pixels.
  • the color pixels (A and B) of the first row and the second row are connected together by a second exposure control line TX2 in a "W" shape to realize individual control of the exposure time of the color pixels.
  • the panchromatic pixels in the third row and the fourth row are connected together by the first exposure control line TX1 in the shape of "W", so as to realize the individual control of the exposure time of the panchromatic pixels.
  • the color pixels (A and B) in the third row and the fourth row are connected together by a second exposure control line TX2 in a "W" shape to realize individual control of the exposure time of the color pixels.
  • the panchromatic pixels in the fifth row and the sixth row are connected together by the first exposure control line TX1 in the shape of "W” to realize the individual control of the exposure time of the panchromatic pixels.
  • the color pixels (B and C) in the fifth row and the sixth row are connected together by a second exposure control line TX2 in a "W" shape to realize individual control of the exposure time of the color pixels.
  • the panchromatic pixels in the seventh row and the eighth row are connected together by the first exposure control line TX1 in the shape of "W” to realize individual control of the exposure time of the panchromatic pixels.
  • the color pixels (B and C) in the seventh row and the eighth row are connected together by a second exposure control line TX2 in a "W" shape to realize individual control of the exposure time of the color pixels.
  • FIG. 20 is a schematic diagram of another minimum repeating unit 1197 pixel arrangement in the embodiment of the present application.
  • FIG. 21 is a schematic diagram of another minimum repeating unit 1198 pixel arrangement in an embodiment of the present application.
  • the first color pixel A is a red pixel R
  • the second color pixel B is a green pixel G
  • the third color pixel C is a blue pixel.
  • Color pixel Bu is a schematic diagram of another minimum repeating unit 1197 pixel arrangement in the embodiment of the present application.
  • FIG. 21 is a schematic diagram of another minimum repeating unit 1198 pixel arrangement in an embodiment of the present application.
  • the first color pixel A is a red pixel R
  • the second color pixel B is a green pixel G
  • the third color pixel C is a blue pixel.
  • Color pixel Bu is a blue pixel.
  • the first color pixel A is a red pixel R; the second color pixel B is a yellow pixel Y; and the third color pixel C is a blue pixel Bu.
  • the first color pixel A is a magenta pixel M; the second color pixel B is a cyan pixel Cy; and the third color pixel C is a yellow pixel Y.
  • the embodiments of the present application include but are not limited to this. Please refer to the above description for the specific connection mode of the circuit, which will not be repeated here.
  • the image sensor 10 (shown in FIG. 2) includes a plurality of color pixels and a plurality of panchromatic pixels W arranged in a matrix, the color pixels and the panchromatic pixels They are arranged at intervals in the row and column directions.
  • panchromatic pixels, color pixels, panchromatic pixels, color pixels are alternately arranged in the row direction.
  • panchromatic pixels, color pixels, panchromatic pixels, color pixels are alternately arranged in the column direction.
  • the first exposure control line TX1 is electrically connected to the control terminal TG (for example, the gate of the transfer transistor 112) of the exposure control circuit 116 in the 2n-1th row and the 2nth row of the panchromatic pixel W
  • the second exposure control line TX2 is electrically connected to the control terminal TG (for example, the gate of the transfer transistor 112) of the exposure control circuit 116 in the 2n-1th row and the 2nth row of the color pixels; n is a natural number greater than or equal to 1.
  • the first exposure control line TX1 is electrically connected to the control terminal TG of the exposure control circuit 116 in the panchromatic pixels W in the first row and the second row; the second exposure control line TX2 is connected to the first row and The control terminal TG of the exposure control circuit 116 in the color pixels in the second row is electrically connected.
  • the first exposure control line TX1 is electrically connected to the control terminal TG of the exposure control circuit 116 in the panchromatic pixels W in the third and fourth rows; the second exposure control line TX2 is electrically connected to the third and fourth rows.
  • the control terminal TG of the exposure control circuit 116 in the color pixels of the row is electrically connected.
  • the first exposure time is less than the second exposure time.
  • the ratio of the first exposure time to the second exposure time is one of 1:2, 1:3, or 1:4.
  • the ratio of the first exposure time to the second exposure time can be adjusted to 1:2, 1:3, or 1:4 according to the brightness of the environment.
  • the exposure ratio is the above-mentioned integer ratio or close to the integer ratio, it is beneficial to the setting and control of the timing setting signal.
  • the image sensor when the image sensor is working, the image sensor will fit the pixel value of each panchromatic pixel in the pixel array to other pixels.
  • the original image including only the color pixels is output. Specifically, taking pixel A as a red pixel R, pixel B as a green pixel G, and pixel C as a blue pixel Bu as an example, the column processing unit in the image sensor reads out the pixel values of multiple red pixels R and multiple green pixels.
  • the image sensor After the pixel value of G, the pixel value of multiple blue pixels Bu, and the pixel value of multiple panchromatic pixels W, the image sensor will first fit the pixel value of each panchromatic pixel W to be adjacent to the panchromatic pixel In the red pixel R, green pixel G, and blue pixel Bu, the non-Bayer array arrangement image is converted into the original image output of the Bayer array arrangement for the processing chip to perform subsequent processing on the original image, such as the original image
  • the image is subjected to interpolation processing to obtain a full-color image (the pixel value of each pixel in a full-color image is composed of three components of red, green and blue), etc.
  • the image sensor needs to execute a more complex algorithm, and the amount of calculation is relatively large, and because the Qualcomm platform does not support the processing of images arranged in a non-Bayer array, additional hardware (such as additional Processing chip) to perform the process of converting the image of the non-Bayer array arrangement into the original image of the Bayer array arrangement.
  • additional hardware such as additional Processing chip
  • the image acquisition method includes:
  • the image acquisition method of the present application can be implemented by the camera assembly 40.
  • step 01 can be implemented by the image sensor 10.
  • step 02, step 03, and step 04 can be implemented by the processing chip 20.
  • the image sensor 10 can be exposed to obtain a full-color original image and a color original image.
  • the processing chip 20 may be used to process a color original image, to treat all pixels of each subunit as a single color large pixel corresponding to a single color in the subunit, and output the pixel value of the single color large pixel to obtain a color intermediate image.
  • the processing chip 20 may also be used to process a full-color original image to obtain a full-color intermediate image, and to process a color intermediate image and/or a full-color intermediate image to obtain a target image.
  • the vertical drive unit 12 in the image sensor 10 controls the exposure of multiple panchromatic pixels and multiple color pixels in the two-dimensional pixel array, and the column processing unit 14 will read the pixel value of each panchromatic pixel and the pixel value of each color pixel.
  • the image sensor 10 does not perform the operation of fitting the pixel value of the panchromatic pixel to the pixel value of the color pixel, but directly outputs a panchromatic original image based on the pixel values of multiple panchromatic pixels, and directly based on the pixel values of multiple panchromatic pixels.
  • the pixel value of the pixel outputs a color original image.
  • the panchromatic original image includes a plurality of panchromatic pixels W and a plurality of empty pixels N (NULL).
  • the empty pixels are neither panchromatic pixels nor color pixels.
  • the panchromatic original image is hollow pixels.
  • the position of N can be regarded as no pixel at that position, or the pixel value of an empty pixel can be regarded as zero. Comparing the two-dimensional pixel array with the full-color original image, it can be seen that for each sub-unit in the two-dimensional pixel array, the sub-unit includes two full-color pixels W and two color pixels (color pixel A, color pixel B, or color pixel). Pixel C).
  • the full-color original image also has a sub-unit corresponding to each sub-unit in the two-dimensional pixel array.
  • the sub-unit of the full-color original image includes two full-color pixels W and two empty pixels N, and two empty pixels N
  • the location corresponds to the location of the two color pixels in the subunit of the two-dimensional pixel array.
  • the color original image includes a plurality of color pixels and a plurality of empty pixels N.
  • the empty pixels are neither panchromatic pixels nor color pixels.
  • the position of the empty pixel N in the color original image can be regarded as no Pixel, or the pixel value of an empty pixel can be treated as zero.
  • the sub-unit includes two panchromatic pixels W and two color pixels.
  • the color original image also has a sub-unit corresponding to each sub-unit in the two-dimensional pixel array.
  • the sub-unit of the color original image includes two color pixels and two empty pixels N.
  • the positions of the two empty pixels N correspond to each other.
  • the processing chip 20 After the processing chip 20 receives the full-color original image and the color original image output by the image sensor 10, it can further process the full-color original image to obtain a full-color intermediate image, and further process the color original image to obtain a color intermediate image.
  • the color original image can be transformed into a color intermediate image in the manner shown in FIG. 25.
  • the color original image includes a plurality of sub-units, and each sub-unit includes a plurality of empty pixels N and a plurality of single-color color pixels (also called single-color pixels).
  • some sub-units include two empty pixels N and two single-color pixels A
  • some sub-units include two empty pixels N and two single-color pixels B
  • some sub-units include two empty pixels N and Two single-color pixels C.
  • the processing chip 20 may regard all the pixels in the sub-unit including the empty pixel N and the single-color pixel A as the single-color large pixel A corresponding to the single-color A in the sub-unit, and will include the empty pixel N and the single-color pixel B.
  • the processing chip 20 can form a color intermediate image based on the plurality of monochromatic large pixels A, the plurality of monochromatic large pixels B, and the plurality of monochromatic large pixels C. If the color original image including a plurality of empty pixels N is regarded as an image with the second resolution, the color intermediate image obtained in the manner shown in FIG. 25 is an image with the first resolution. One resolution is smaller than the second resolution.
  • the full-color intermediate image and/or the color intermediate image can be further processed to obtain the target image.
  • the processing chip 20 may only process the full-color intermediate image to obtain the target image; the processing chip 20 may also only process the color intermediate image to obtain the target image; the processing chip 20 may also process the full-color intermediate image and the color intermediate image at the same time to obtain the target image. Target image.
  • the processing chip 20 can determine the processing mode of the two intermediate images according to actual requirements.
  • the image sensor 10 can directly output the full-color original image and the color original image.
  • the subsequent processing of the full-color original image and the color original image is performed by the processing chip 20, and the image sensor 10
  • the image sensor 10 There is no need to perform the operation of fitting the pixel value of the panchromatic pixel W to the pixel value of the color pixel, the calculation amount of the image sensor 10 is reduced, and there is no need to add new hardware to the image sensor 10 to support the image sensor 10 to execute the image.
  • the design of the image sensor 10 can be simplified.
  • step 01 controlling the exposure of the two-dimensional pixel array to obtain a full-color original image and a color original image can be implemented in various ways.
  • step 01 includes:
  • 011 Control the simultaneous exposure of all panchromatic pixels and all color pixels in the two-dimensional pixel array
  • step 011, step 012, and step 013 can all be implemented by the image sensor 10.
  • the image sensor 10 may output the pixel values of all panchromatic pixels to obtain a panchromatic original image, and may also output the pixel values of all color pixels to obtain a color original image.
  • panchromatic pixels and color pixels can be exposed at the same time, wherein the exposure time of the panchromatic pixels can be less than or equal to the exposure time of the color pixels.
  • the exposure start time and the exposure cutoff time of the panchromatic pixel are the same as the exposure start time and the exposure cutoff time of the color pixel, respectively.
  • the exposure start time of the panchromatic pixel is later than or equal to the exposure start time of the color pixel, and the exposure cut-off time of the panchromatic pixel is earlier than the exposure cut-off time of the color pixel; or
  • the exposure start time of the panchromatic pixel is later than the exposure start time of the color pixel, and the exposure cutoff time of the panchromatic pixel is earlier than or equal to the exposure cutoff time of the color pixel.
  • the image sensor 10 After the panchromatic pixels and the color pixels are exposed and evened, the image sensor 10 outputs the pixel values of all the panchromatic pixels to obtain the panchromatic original image, and outputs the pixel values of all the color pixels to obtain the color original image.
  • the full-color original image can be output before the color original image, or; the color original image can be output before the full-color original image; or, the full-color original image and the color original image can be output at the same time.
  • the output order of the two is not limited here. Simultaneous exposure of panchromatic pixels and color pixels can reduce the acquisition time of panchromatic original images and color original images, and speed up the process of acquiring panchromatic original images and color original images.
  • the method of simultaneous exposure of panchromatic pixels and color pixels has great advantages in fast shooting, continuous shooting and other modes that require higher image output speed.
  • step 01 includes:
  • step 014, step 015, and step 016 can all be implemented by the image sensor 10.
  • all the panchromatic pixels and all the color pixels in the image sensor 10 are exposed in a time-sharing manner.
  • the image sensor 10 may output the pixel values of all panchromatic pixels to obtain a panchromatic original image, and may also output the pixel values of all color pixels to obtain a color original image.
  • the panchromatic pixels and the color pixels may be exposed in time sharing, wherein the exposure time of the panchromatic pixels may be less than or equal to the exposure time of the color pixels.
  • the time-sharing exposure method for all panchromatic pixels and all color pixels can be: (1) All panchromatic pixels first perform the exposure for the first exposure time, and wait After all the panchromatic pixels are exposed, all the color pixels perform the second exposure time exposure; (2) All the color pixels perform the second exposure time exposure first, and after all the color pixels are exposed, all the panchromatic pixels perform the second exposure time again.
  • One exposure time of exposure can be: (1) All panchromatic pixels first perform the exposure for the first exposure time, and wait After all the panchromatic pixels are exposed, all the color pixels perform the second exposure time exposure; (2) All the color pixels perform the second exposure time exposure first, and after all the color pixels are exposed, all the panchromatic pixels perform the second exposure time again.
  • the image sensor 10 After the panchromatic pixels and the color pixels are exposed and evened, the image sensor 10 outputs the pixel values of all the panchromatic pixels to obtain the panchromatic original image, and outputs the pixel values of all the color pixels to obtain the color original image.
  • the output mode of the full-color original image and the color original image can be: (1) When the full-color pixel is exposed before the color pixel, the image sensor 10 can output the full-color original image during the color pixel exposure period, or it can output the full-color original image during the color pixel exposure.
  • the full-color original image and the color original image are sequentially output; (2) When the color pixels are exposed before the full-color pixels, the image sensor 10 can output the color original images during the exposure of the full-color pixels, or wait for the full-color pixels After the exposure is completed, the color original image and the full-color original image are output in turn; (3) No matter which of the pan-color pixels and the color pixels is first exposed, the image sensor 10 can output the full-color original image at the same time after all pixels are exposed And color original image.
  • the control logic of the time-sharing exposure of panchromatic pixels and color pixels is relatively simple.
  • the image sensor 10 may simultaneously have the functions of controlling the simultaneous exposure of panchromatic pixels and color pixels, and controlling the time-sharing exposure of panchromatic pixels and color pixels, as shown in FIGS. 26 and 27.
  • the specific exposure mode used by the image sensor 10 in the process of collecting images can be selected according to actual needs. For example, simultaneous exposure can be used in fast shooting, continuous shooting, etc. modes to meet the needs of rapid image output; in ordinary shooting modes, time-sharing exposure can be used to simplify the control logic.
  • the exposure sequence of the panchromatic pixels and the color pixels can be controlled by the control unit 13 in the image sensor 10.
  • the exposure time of the panchromatic pixels can be controlled by the first exposure signal, and the exposure time of the color pixels can be controlled by the second exposure signal.
  • the image sensor 10 may use a first exposure signal to control at least two adjacent panchromatic pixels in a first diagonal direction to expose for a first exposure time, and use a second exposure signal to control at least two panchromatic pixels. At least two adjacent color pixels in the second diagonal direction are exposed at a second exposure time, where the first exposure time may be less than or equal to the second exposure time.
  • the vertical driving unit 12 in the image sensor 10 transmits the first exposure signal through the first exposure control line TX1 to control at least two adjacent panchromatic pixels in the first diagonal direction to be exposed at the first exposure time, and drive vertically.
  • the unit 12 transmits a second exposure signal through the second exposure control line TX2 to control at least two adjacent panchromatic pixels in the second diagonal direction to be exposed at the second exposure time.
  • the image sensor 10 does not perform the process of fitting the pixel values of multiple panchromatic pixels to the pixel values of the color pixels, but directly outputs One full-color original image and one color original image.
  • the image sensor 10 can use the first exposure signal to control the 2n-1th row and the 2nth row of the panchromatic pixels to be exposed for the first exposure time, and use the second exposure signal to control the first exposure time.
  • the color pixels in the 2n-1 row and the 2nth row are exposed at the second exposure time, where the first exposure time may be less than or equal to the second exposure time.
  • the first exposure control line TX1 in the image sensor 10 is connected to the control terminals TG of all panchromatic pixels in the 2n-1 row and the 2nth row
  • the second exposure control line TX2 is connected to the 2n-1 row and the 2nth row.
  • the control terminals TG of all color pixels are connected.
  • the vertical driving unit 12 transmits the first exposure signal through the first exposure control line TX1 to control the panchromatic pixels in the 2n-1th row and the 2nth row to be exposed with the first exposure time, and transmits the second exposure signal through the second exposure control line TX2 To control the color pixels of the 2n-1th row and the 2nth row to be exposed with the second exposure time.
  • the image sensor 10 does not perform the process of fitting the pixel values of multiple panchromatic pixels to the pixel values of the color pixels, but directly outputs One full-color original image and one color original image.
  • the processing chip 20 may determine the relative relationship between the first exposure time and the second exposure time according to the environmental brightness.
  • the image sensor 10 may first control the exposure of panchromatic pixels and output a panchromatic original image, and the processing chip 20 analyzes the pixel values of multiple panchromatic pixels in the panchromatic original image to determine the environmental brightness.
  • the image sensor 10 controls the panchromatic pixels to be exposed at the first exposure time equal to the second exposure time; when the ambient brightness is greater than the brightness threshold, the image sensor 10 controls the panchromatic pixels to be less than the second exposure time.
  • Exposure time is the first exposure time to exposure.
  • the relative relationship between the first exposure time and the second exposure time can be determined according to the brightness difference between the ambient brightness and the brightness threshold. For example, the greater the brightness difference, the greater the first exposure time and the second exposure time.
  • the ratio of the second exposure time is smaller. For example, when the brightness difference is within the first range [a,b), the ratio of the first exposure time to the second exposure time is 1:2; when the brightness difference is within the second range [b,c) , The ratio of the first exposure time to the second exposure time is 1:3; when the brightness difference is greater than or equal to c, the ratio of the first exposure time to the second exposure time is 1:4.
  • step 02 includes:
  • a color intermediate image is formed according to the pixel values of a plurality of monochromatic large pixels, and the color intermediate image has the first resolution.
  • both step 021 and step 022 can be implemented by the processing chip 20.
  • the processing chip 20 can be used to combine the pixel values of all pixels in each subunit to obtain the pixel value of a single large pixel, and to form a color intermediate image based on the pixel values of multiple large monochrome pixels.
  • the image has the first resolution.
  • the color intermediate image has the first resolution.
  • the processing chip 20 may add the pixel values of all pixels in the sub-unit including the empty pixel N and the single-color pixel A, and use the result of the addition as Corresponding to the pixel value of the single-color large pixel A of the sub-unit, the pixel value of the empty pixel N can be regarded as zero, the same below; the processing chip 20 can treat all the sub-units including the empty pixel N and the single-color pixel B The pixel values of the pixels are added, and the result of the addition is taken as the pixel value of the single-color large pixel B corresponding to the sub-unit; the processing chip 20 may add the values of all pixels in the sub-unit including the empty pixel N and the single-color pixel C The pixel values are added, and the result of the addition is used as the pixel value of the single-color large pixel C corresponding to the sub-unit.
  • the processing chip 20 can obtain the pixel values of a plurality of single large pixels A, the pixel values of a plurality of monochromatic large pixels B, and the pixel values of a plurality of monochromatic large pixels C.
  • the processing chip 20 then forms a color intermediate image according to the pixel values of the plurality of monochromatic large pixels A, the pixel values of the plurality of monochromatic large pixels B, and the pixel values of the plurality of monochromatic large pixels C.
  • the single color A is red R
  • the single color B is green G
  • the single color C is blue Bu
  • the color intermediate image is an image arranged in a Bayer array.
  • the manner in which the processing chip 20 obtains the color intermediate image is not limited to this.
  • the different modes correspond to different target images.
  • the processing chip 20 first determines which mode the camera assembly 40 is in, and then performs corresponding processing on the color intermediate image and/or the full-color intermediate image according to the mode of the camera assembly 40 to obtain the target image corresponding to the mode.
  • the target image includes at least four types of target images: a first target image, a second target image, a third target image, and a fourth target image.
  • the modes of the camera assembly 40 include at least: (1) the mode is the preview mode, the target image in the preview mode can be the first target image or the second target image; (2) the mode is the imaging mode, the target image in the imaging mode It can be the second target image, the third target image, or the fourth target image; (3) the mode is both preview mode and low power consumption mode, at this time the target image is the first target image; (4) mode is the preview mode It is a non-low power consumption mode, and the target image is the second target image at this time; (5) The mode is both an imaging mode and a low power consumption mode, and the target image is the second target image or the third target image at this time; (6) The) mode is both an imaging mode and a non-low power consumption mode. At this time, the target image is the fourth target image.
  • step 04 when the target image is the first target image, step 04 includes:
  • step 040 can be implemented by the processing chip 20.
  • the processing chip 20 can be used to perform interpolation processing on each single-color large pixel in the color intermediate image to obtain and output the pixel values of the other two colors except the single color to obtain the first resolution.
  • the first target image can be used to perform interpolation processing on each single-color large pixel in the color intermediate image to obtain and output the pixel values of the other two colors except the single color to obtain the first resolution.
  • the processing chip 20 needs to perform demosaicing (that is, interpolation processing) on the color intermediate image, so that the pixel value of each monochromatic large pixel has three components of R, G, and B at the same time.
  • demosaicing that is, interpolation processing
  • a linear interpolation method may be used to calculate the pixel values of the other two colors of each monochromatic large pixel except for the single color of the monochromatic large pixel.
  • the processing chip 20 After the processing chip 20 calculates the pixel values of the three components of each monochromatic large pixel, it can calculate the final pixel value corresponding to the monochromatic large pixel based on the three pixel values, namely A+B+C, which needs to be explained.
  • A+B+C here does not mean directly adding three pixels to obtain the final pixel value of the monochromatic large pixel, but only means that the monochromatic large pixel includes the three color components of A, B, and C.
  • the processing chip 20 may form a first target image according to the final pixel values of a plurality of monochromatic large pixels.
  • the first target image is a color intermediate image obtained through interpolation processing, and the processing chip 20 does not perform interpolation processing on the color intermediate image. Therefore, the resolution of the first target image is also the first resolution. rate.
  • the processing algorithm for the processing chip 20 to process the color intermediate image to obtain the first target image is relatively simple, and the processing speed is faster.
  • the camera assembly 40 uses the first target image as the preview image when the mode is both the preview mode and the low power consumption mode. To meet the requirement of the preview mode for the output speed, the power consumption of the camera assembly 40 can also be saved.
  • step 03 when the target image is the second target image, step 03 includes:
  • panchromatic intermediate image has the first resolution
  • Step 04 includes:
  • step 031, step 041, step 042 and step 043 can all be implemented by the processing chip 20.
  • the processing chip 20 can be used to process a full-color original image, treat all pixels of each subunit as a full-color large pixel, and output the pixel value of the full-color large pixel to obtain a full-color intermediate image, a full-color intermediate image Has the first resolution.
  • the processing chip 20 can also be used to separate the color and brightness of the color intermediate image to obtain a color-brightness separated image with a first resolution, and to fuse the brightness of the full-color intermediate image and the brightness of the color-brightness separated image to obtain a color and brightness separated image with the first resolution.
  • Brightness-corrected color image, and interpolation processing is performed on each single-color large pixel in the brightness-corrected color image to obtain the pixel values of the other two colors in addition to the single color and output to obtain the second target with the first resolution image.
  • the full-color original image can be transformed into a full-color intermediate image in the manner shown in FIG. 31.
  • the full-color original image includes a plurality of sub-units, and each sub-unit includes two empty pixels N and two pan-color pixels W.
  • the processing chip 20 may regard all pixels in each sub-unit including the empty pixel N and the full-color pixel W as the full-color large pixel W corresponding to the sub-unit. In this way, the processing chip 20 can form a full-color intermediate image based on a plurality of full-color large pixels W. If the full-color original image including multiple empty pixels N is regarded as an image with the second resolution, the full-color intermediate image obtained in the manner shown in FIG. 31 is an image with the first resolution, where , The first resolution is smaller than the second resolution.
  • the processing chip 20 may use all the pixels of each subunit in the full-color original image as the full-color large pixel W corresponding to the sub-unit in the following manner: the processing chip 20 first merges the pixels of all pixels in each sub-unit Value to obtain the pixel value of the panchromatic large pixel W, and then form a panchromatic intermediate image according to the pixel values of the multiple panchromatic large pixels W. Specifically, for each full-color large pixel, the processing chip 20 may add all the pixel values in the sub-units including the empty pixel N and the full-color pixel W, and use the result of the addition as the full-color corresponding to the sub-unit. The pixel value of the large pixel W, where the pixel value of the empty pixel N can be regarded as zero. In this way, the processing chip 20 can obtain the pixel values of a plurality of full-color large pixels W.
  • the processing chip 20 After the processing chip 20 obtains the full-color intermediate image and the color intermediate image, it can perform fusion processing on the full-color intermediate image and the color intermediate image to obtain the second target image.
  • the processing chip 20 first separates the color and brightness of the color intermediate image to obtain the color-brightness separated image.
  • L represents brightness
  • CLR represents color.
  • the processing chip 20 can convert the color intermediate image in the RGB space into Color and brightness separation image in YCrCb space, at this time Y in YCrCb is the brightness L in the color and brightness separation image, and Cr and Cb in YCrCb are the color CLR in the color and brightness separation image; (2)
  • the processing chip 20 can also Convert the RGB color intermediate image to the color-brightness separated image in Lab space.
  • L in Lab is the brightness L in the color-brightness separated image
  • a and b in Lab are the color CLRs in the color-brightness separated image.
  • L+CLR in the color-light separation image shown in FIG. 31 does not mean that the pixel value of each pixel is formed by adding L and CLR, but only that the pixel value of each pixel is composed of L and CLR.
  • the processing chip 20 fuses the brightness of the color-brightness separated image and the brightness of the full-color intermediate image.
  • the pixel value of each panchromatic pixel W in the panchromatic intermediate image is the brightness value of each panchromatic pixel
  • the processing chip 20 may correspond to the L of each pixel in the color-brightness separation image with that in the panchromatic intermediate image. Add the W of the panchromatic pixel at the position to get the pixel value after brightness correction.
  • the processing chip 20 forms a brightness-corrected color-brightness separated image according to a plurality of brightness-corrected pixel values, and then uses color space conversion to convert the brightness-corrected color-brightness separated image into a brightness-corrected color image.
  • the brightness-corrected color image is an image arranged in a Bayer array
  • the processing chip 20 needs to The brightness-corrected color image is subjected to interpolation processing, so that the pixel value of each large monochromatic pixel after brightness correction has three components of R, G, and B at the same time.
  • the processing chip 20 may perform interpolation processing on the brightness-corrected color image to obtain the second target image. For example, a linear interpolation method may be used to obtain the second target image.
  • the linear interpolation process is similar to the interpolation process in the aforementioned step 40. Go into details again.
  • the second target image is a brightness-corrected color image obtained through interpolation processing, and the processing chip 20 does not perform interpolation processing on the brightness-corrected color image. Therefore, the resolution of the second target image is also The first resolution. Since the second target image is obtained by fusing the brightness of the color intermediate image and the brightness of the panchromatic intermediate image, the second target image has a better imaging effect.
  • the mode is the preview mode and the non-low power consumption mode
  • using the second target image as the preview image can improve the preview effect of the preview image.
  • the mode is the imaging mode and the low power consumption mode
  • the second target image is used as the image provided to the user.
  • the camera assembly 40 can be reduced to a certain extent.
  • the power consumption can meet the usage requirements in the low power consumption mode; at the same time, the brightness of the second target image is relatively bright, which can meet the user's brightness requirements for the target image.
  • step 04 includes:
  • both steps 044 and 045 can be implemented by the processing chip 20.
  • the processing chip 20 can be used to interpolate and process the color intermediate image to obtain a color interpolated image with a second resolution.
  • the corresponding subunits in the color interpolated image are arranged in a Bayer array, and the second resolution is greater than The first resolution.
  • the processing chip 20 can also be used to perform interpolation processing on all single-color pixels in the color interpolation image to obtain pixel values of two other colors except the single color and output to obtain a third target image with a second resolution. .
  • the processing chip 20 splits each large monochromatic pixel in the color intermediate image into four color pixels.
  • the four color pixels form a subunit in the color interpolation image, and each subunit It includes three colors of color pixels, which are one color pixel A, two color pixels B, and one color pixel C.
  • the color pixel A is a red pixel R
  • the color pixel B is a green pixel G
  • the color pixel C is a blue pixel Bu
  • the multiple color pixels in each subunit are arranged in a Bayer array. Therefore, the color interpolated image containing multiple subunits is the image arranged in the Bayer array.
  • the processing chip 20 may perform interpolation processing on the color interpolation image to obtain the third target image.
  • a linear interpolation method may be used to obtain the second target image.
  • the linear interpolation process is similar to the interpolation process in the foregoing step 040. Go into details again.
  • the third target image is an image obtained through interpolation processing, and the resolution of the third target image (ie, the second resolution) is greater than the resolution of the color intermediate image (ie, a resolution).
  • the mode is both the preview mode and the non-low power consumption mode
  • the third target image is used as the preview image to obtain a clearer preview image.
  • the mode is both the imaging mode and the low power consumption mode
  • the third target image is used as the image provided to the user. Since the third target does not need to be fused with the panchromatic intermediate image during the formation process, it can be reduced to a certain extent.
  • the power consumption of the camera assembly 40 can also meet the user's requirements for the clarity of the captured image.
  • step 03 when the target image is the fourth target image, step 03 includes:
  • Step 04 includes:
  • step 032, step 046, step 047, step 048, and step 049 can all be implemented by the processing chip 20.
  • the processing chip 20 can be used to interpolate the full-color original image, and obtain the pixel values of all pixels in each sub-unit to obtain a full-color intermediate image with the second resolution.
  • the processing chip 20 can also be used to interpolate and process the color intermediate image to obtain a color interpolation image with a second resolution.
  • the corresponding subunits in the color interpolation image are arranged in a Bayer array, and the second resolution is greater than the first resolution. .
  • the processing chip 20 can also be used to separate the color and brightness of the color interpolation image to obtain a color-brightness separated image with a second resolution, and to fuse the brightness of the full-color interpolation image and the brightness of the color-brightness separated image to obtain a second resolution.
  • the processing chip 20 first needs to perform interpolation processing on the full-color original image of the first resolution to obtain the full-color intermediate image of the second resolution.
  • the full-color original image includes multiple sub-units, each sub-unit includes two empty pixels N and two pan-color pixels W, the processing chip 20 needs to replace each empty pixel N in each sub-unit with a full Color pixel W, and calculate the pixel value of each panchromatic pixel W at the position of the empty pixel N after replacement.
  • the processing chip 20 For each empty pixel N, the processing chip 20 replaces the empty pixel N with a panchromatic pixel W, and determines the replaced panchromatic pixel W according to the pixel values of the remaining panchromatic pixels W adjacent to the replaced panchromatic pixel W The pixel value of the panchromatic pixel W.
  • panchromatic pixels W 2,3 which are adjacent to panchromatic pixels W 2,3.
  • panchromatic pixels W 1,3 panchromatic pixels W 2,2 , panchromatic pixels W 2,4 , and panchromatic pixels W 3,3 in the panchromatic original image.
  • the processing chip 20 sets the panchromatic pixels The pixel value of W 1,3 , the pixel value of panchromatic pixel W 2,2 , the pixel value of panchromatic pixel W 2,4 , and the average value of the pixel value of panchromatic pixel W 3,3 are used as the replaced panchromatic pixel The pixel value of W 2,3.
  • the processing chip 20 After the processing chip 20 obtains the full-color intermediate image and the color intermediate image, it can perform fusion processing on the full-color intermediate image and the color intermediate image to obtain the fourth target image.
  • the processing chip 20 can perform interpolation processing on the color intermediate image of the first resolution to obtain the color interpolation image of the second resolution, as shown in FIG. 33.
  • the specific interpolation method is similar to the interpolation method in step 045, and will not be repeated here.
  • the processing chip 20 can separate the color and brightness of the color interpolation image to obtain a color-brightness separated image.
  • L represents brightness
  • CLR represents color.
  • the processing chip 20 can convert the color interpolation image in the RGB space Is the color and brightness separation image in YCrCb space, at this time Y in YCrCb is the brightness L in the color and brightness separation image, and Cr and Cb in YCrCb are the color CLR in the color and brightness separation image; (2) the processing chip 20 also The RGB color interpolation image can be converted into the color and brightness separation image in Lab space.
  • L in Lab is the brightness L in the color and brightness separation image
  • a and b in Lab are the colors in the color and brightness separation image.
  • CLR L+CLR in the color-light separation image shown in FIG. 33 does not mean that the pixel value of each pixel is formed by adding L and CLR, but only that the pixel value of each pixel is composed of L and CLR.
  • the processing chip 20 may fuse the brightness of the color-brightness separated image and the brightness of the full-color intermediate image.
  • the pixel value of each panchromatic pixel W in the panchromatic intermediate image is the brightness value of each panchromatic pixel
  • the processing chip 20 may correspond to the L of each pixel in the color-brightness separation image with that in the panchromatic intermediate image. Add the W of the panchromatic pixel at the position to get the pixel value after brightness correction.
  • the processing chip 20 forms a brightness-corrected color-brightness separated image according to a plurality of brightness-corrected pixel values, and then converts the brightness-corrected color-brightness separated image into a brightness-corrected color image.
  • the brightness-corrected color image has a second resolution. rate.
  • the brightness correction color image is an image arranged in a Bayer array
  • the processing chip 20 needs to interpolate the brightness correction color image Processing is performed so that the pixel value of each color pixel after the brightness correction has three components of R, G, and B at the same time.
  • the processing chip 20 may perform interpolation processing on the brightness-corrected color image to obtain the fourth target image. For example, linear interpolation may be used to obtain the fourth target image.
  • the linear interpolation process is similar to the interpolation process in the aforementioned step 40. Go into details again.
  • the fourth target image is obtained by fusing the brightness of the color intermediate image and the brightness of the panchromatic intermediate image, and the fourth target image has a larger resolution, the fourth target image has better brightness and clarity.
  • the mode is both the imaging mode and the non-low power consumption mode, using the fourth target image as the image provided to the user can meet the user's quality requirements for the captured image.
  • the image acquisition method may further include acquiring environmental brightness.
  • This step can be implemented by the processing chip 20, and the specific implementation manner is as described above, and will not be repeated here.
  • the environmental brightness is greater than the brightness threshold
  • the first target image or the third target image can be used as the target image
  • the second target image or the fourth target image can be used as the target image. It can be understood that when the ambient brightness is bright, the brightness of the first target image and the second target image obtained only from the color intermediate image is sufficient to meet the user's brightness requirements for the target image, and there is no need to merge the brightness of the full-color intermediate image.
  • the calculation amount of the processing chip 20 can be reduced, but also the power consumption of the camera assembly 40 can be reduced.
  • the brightness of the first target image and the second target image obtained only from the color intermediate image may not meet the user's need for the brightness of the target image.
  • the second target image obtained by fusing the brightness of the full-color intermediate image is combined.
  • the target image or the fourth target image is used as the target image to increase the brightness of the target image.
  • the present application also provides a mobile terminal 90.
  • the mobile terminal 90 may be a mobile phone, a tablet computer, a notebook computer, a smart wearable device (such as a smart watch, a smart bracelet, a smart glasses, a smart helmet, etc.), a head-mounted display device, a virtual reality device, etc., which are not limited here.
  • the mobile terminal 90 includes an image sensor 50, a processor 60, a memory 70, and a case 80, and the image sensor 50, the processor 60, and the memory 70 are all installed in the case 80.
  • the image sensor 50 is connected to the processor 60, and the image sensor 50 may be the image sensor 10 (shown in FIG. 1) described in any one of the above embodiments.
  • the processor 60 can perform the same functions as the processing chip 20 in the camera assembly 40 (shown in FIG. 1). In other words, the processor 60 can implement the functions that can be implemented by the processing chip 20 described in any one of the foregoing embodiments.
  • the memory 70 is connected to the processor 60, and the memory 70 can store data obtained after processing by the processor 60, such as a target image.
  • the processor 60 and the image sensor 50 may be mounted on the same substrate. At this time, the image sensor 50 and the processor 60 can be regarded as a camera assembly 40. Of course, the processor 60 and the image sensor 50 may also be mounted on a different substrate.
  • the image sensor 50 can directly output the full-color original image and the color original image.
  • the subsequent processing of the full-color original image and the color original image is executed by the processor 60, and the image sensor 50 does not need to execute the full-color pixel
  • the operation of fitting the pixel value of W to the pixel value of the color pixel reduces the computational complexity of the image sensor 50, and there is no need to add new hardware to the image sensor 50 to support the image sensor 50 to perform image processing, which can simplify the image sensor 50 design.

Landscapes

  • Engineering & Computer Science (AREA)
  • Multimedia (AREA)
  • Signal Processing (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • Spectroscopy & Molecular Physics (AREA)
  • Color Television Image Signal Generators (AREA)
  • Studio Devices (AREA)
  • Transforming Light Signals Into Electric Signals (AREA)

Abstract

一种图像采集方法、摄像头组件(40)及移动终端(90)。图像传感器(10)包括二维像素阵列。二维像素阵列包括多个全色像素和多个彩色像素。二维像素阵列包括最小重复单元，每个最小重复单元包含多个子单元，每个子单元包括多个单颜色像素及多个全色像素。图像采集方法包括：(01)控制二维像素阵列曝光以获取全色原始图像和彩色原始图像；(02)处理彩色原始图像，以将每个子单元的所有像素作为与该子单元中单颜色对应的单色大像素，并输出单色大像素的像素值以得到彩色中间图像；(03)处理全色原始图像以得到全色中间图像；(04)处理彩色中间图像和/或全色中间图像以获取目标图像。

Description

图像采集方法、摄像头组件及移动终端 技术领域
本申请涉及影像技术领域，特别涉及一种图像采集方法、摄像头组件及移动终端。
背景技术
手机等移动终端当中往往装配有摄像头，以实现拍照功能。摄像头中设置有图像传感器。为了实现彩色图像的采集，图像传感器中通常会设置有彩色像素，彩色像素以拜耳(Bayer)阵列形式排布。为提升图像传感器在黑暗环境下的成像质量，相关技术中将灵敏度比彩色像素高的白色像素加入到图像传感器中。
发明内容
本申请提供一种图像采集方法、摄像头组件及移动终端。
本申请一个方面提供一种图像传感器的图像采集方法。所述图像传感器包括二维像素阵列，所述二维像素阵列包括多个全色像素和多个彩色像素，所述二维像素阵列包括最小重复单元，每个所述最小重复单元包含多个子单元，每个所述子单元包括多个单颜色像素及多个全色像素；所述图像采集方法包括：控制所述二维像素阵列曝光以获取全色原始图像和彩色原始图像；处理所述彩色原始图像，以将每个所述子单元的所有像素作为与该子单元中单颜色对应的单色大像素，并输出所述单色大像素的像素值以得到彩色中间图像；处理所述全色原始图像以得到全色中间图像；处理所述彩色中间图像和/或所述全色中间图像以获取目标图像。
在另一个方面，本申请还提供一种摄像头组件。摄像头组件包括图像传感器及处理芯片。所述图像传感器包括二维像素阵列，所述二维像素阵列包括多个全色像素和多个彩色像素，所述二维像素阵列包括最小重复单元，每个所述最小重复单元包含多个子单元，每个所述子单元包括多个单颜色像素及多个全色像素，所述图像传感器用于曝光以获取全色原始图像和彩色原始图像。所述处理芯片用于：处理所述彩色原始图像，以将每个所述子单元的所有像素作为与该子单元中单颜色对应的单色大像素，并输出所述单色大像素的像素值以得到彩色中间图像；处理所述全色原始图像以得到全色中间图像；处理所述彩色中间图像和/或所述全色中间图像以获取目标图像。
在又一个方面，本申请还提供一种移动终端。移动终端包括图像传感器及处理器。所述图像传感器包括二维像素阵列，所述二维像素阵列包括多个全色像素和多个彩色像素，所述二维像素阵列包括最小重复单元，每个所述最小重复单元包含多个子单元，每个所述子单元包括多个单颜色像素及多个全色像素，所述图像传感器用于曝光以获取全色原始图像和彩色原始图像。所述处理器用于：处理所述彩色原始图像，以将每个所述子单元的所有像素作为与该子单元中单颜色对应的单色大像素，并输出所述单色大像素的像素值以得到彩色中间图像；处理所述全色原始图像以得到全色中间图像；处理所述彩色中间图像和/或所述全色中间图像以获取目标图像。
本申请实施方式的附加方面和优点将在下面的描述中部分给出，部分将从下面的描述中变得明显，或通过本申请的实践了解到。
附图说明
本申请的上述和/或附加的方面和优点可以从结合下面附图对实施方式的描述中将变得明显和容易理解，其中：
图1是本申请实施方式的摄像头组件的示意图；
图2是本申请实施方式中图像传感器的示意图；
图3是本申请实施方式中像素阵列及曝光控制线连接方式的示意图；
图4是不同色彩通道曝光饱和时间的示意图；
图5是本申请实施方式中一种像素电路的示意图；
图6是本申请实施方式中一种最小重复单元像素排布的示意图；
图7是本申请实施方式中又一种最小重复单元像素排布的示意图；
图8是本申请实施方式中又一种最小重复单元像素排布的示意图；
图9是本申请实施方式中又一种最小重复单元像素排布的示意图；
图10是本申请实施方式中又一种最小重复单元像素排布的示意图；
图11是本申请实施方式中又一种最小重复单元像素排布的示意图；
图12是本申请实施方式中又一种最小重复单元像素排布的示意图；
图13是本申请实施方式中又一种最小重复单元像素排布的示意图；
图14是本申请实施方式中又一种最小重复单元像素排布的示意图；
图15是本申请实施方式中又一种最小重复单元像素排布的示意图；
图16是本申请实施方式中又一种最小重复单元像素排布的示意图；
图17是本申请实施方式中又一种最小重复单元像素排布的示意图；
图18是本申请实施方式中又一种最小重复单元像素排布的示意图；
图19是本申请实施方式中又一种最小重复单元像素排布的示意图；
图20是本申请实施方式中又一种最小重复单元像素排布的示意图；
图21是本申请实施方式中又一种最小重复单元像素排布的示意图；
图22是相关技术中的图像采集方法的原理示意图；
图23是本申请某些实施方式的图像采集方法的流程示意图；
图24是本申请实施方式中光图像采集方法的一个原理示意图；
图25是本申请实施方式中光图像采集方法的另一个原理示意图；
图26至图29是本申请某些实施方式的图像采集方法的流程示意图；
图30是本申请实施方式中光图像采集方法的又一个原理示意图；
图31是本申请实施方式中光图像采集方法的再一个原理示意图；
图32是本申请实施方式中光图像采集方法的再一个原理示意图；
图33是本申请实施方式中光图像采集方法的再一个原理示意图；
图34是本申请实施方式中光图像采集方法的再一个原理示意图；
图35是本申请实施方式的移动终端的示意图。
具体实施方式
下面详细描述本申请的实施方式，实施方式的示例在附图中示出，其中，相同或类似的标号自始至终表示相同或类似的元件或具有相同或类似功能的元件。下面通过参考附图描述的实施方式是示例性的，仅用于解释本申请，而不能理解为对本申请的限制。
本申请提供一种图像传感器的图像采集方法。图像传感器包括二维像素阵列。二维像素阵列包括多个全色像素和多个彩色像素。二维像素阵列包括最小重复单元，每个最小重复单元包含多个子单元，每个子单元包括多个单颜色像素及多个全色像素。图像采集方法包括：控制二维像素阵列曝光以获取全色原始图像和彩色原始图像；处理彩色原始图像，以将每个子单元的所有像素作为与该子单元中单颜色对应的单色大像素，并输出单色大像素的像素值以得到彩色中间图像；处理全色原始图像以得到全色中间图像；处理彩色中间图像和/或全色中间图像以获取目标图像。
本还申请提供一种摄像头组件。摄像头组件包括图像传感器及处理芯片。图像传感器包括二维像素阵列。二维像素阵列包括多个全色像素和多个彩色像素。二维像素阵列包括最小重复单元，每个最小重复单元包含多个子单元，每个子单元包括多个单颜色像素及多个全色像素。图像传感器用于曝光以获取全色原始图像和彩色原始图像。处理芯片用于：处理彩色原始图像，以将每个子单元的所有像素作为与该子单元中单颜色对应的单色大像素，并输出单色大像素的像素值以得到彩色中间图像；处理全色原始图像以得到全色中间图像；处理彩色中间图像和/或全色中间图像以获取目标图像。
本还申请提供一种移动终端。移动终端包括图像传感器及处理器。图像传感器包括二维像素阵列。二维像素阵列包括多个全色像素和多个彩色像素。二维像素阵列包括最小重复单元，每个最小重复单元包含多个子单元，每个子单元包括多个单颜色像素及多个全色像素。图像传感器用于曝光以获取全色原始图像和彩色原始图像。处理器用于：处理彩色原始图像，以将每个子单元的所有像素作为与该子单元中单颜色对应的单色大像素，并输出单色大像素的像素值以得到彩色中间图像；处理全色原始图像以得 到全色中间图像；处理彩色中间图像和/或全色中间图像以获取目标图像。
请参阅图1，本申请提供一种摄像头组件40。摄像头组件40包括图像传感器10、处理芯片20及镜头30。图像传感器10与处理芯片20电连接。镜头30设置在图像传感器10的光路上。处理芯片20可以与图像传感器10及镜头30封装在同一个摄像头组件40的壳体内；或者，图像传感器10与镜头30封装在壳体内，处理芯片20设置在壳体外。
请参阅图2，图2是本申请实施方式中的图像传感器10的示意图。图像传感器10包括像素阵列11、垂直驱动单元12、控制单元13、列处理单元14和水平驱动单元15。
例如，图像传感器10可以采用互补金属氧化物半导体(CMOS，Complementary Metal Oxide Semiconductor)感光元件或者电荷耦合元件(CCD，Charge-coupled Device)感光元件。
例如，像素阵列11包括以阵列形式二维排列的多个像素(图2中未示出)，每个像素包括光电转换元件。每个像素根据入射在其上的光的强度将光转换为电荷。
例如，垂直驱动单元12包括移位寄存器和地址译码器。垂直驱动单元12包括读出扫描和复位扫描功能。读出扫描是指顺序地逐行扫描单位像素，从这些单位像素逐行地读取信号。例如，被选择并被扫描的像素行中的每一像素输出的信号被传输到列处理单元14。复位扫描用于复位电荷，光电转换元件的光电荷被丢弃，从而可以开始新的光电荷的积累。
例如，由列处理单元14执行的信号处理的是相关双采样(CDS)处理。在CDS处理中，取出从所选像素行中的每一像素输出的复位电平和信号电平，并且计算电平差。因而，获得了一行中的像素的信号。列处理单元14可以具有用于将模拟像素信号转换为数字格式的模数(A/D)转换功能。
例如，水平驱动单元15包括移位寄存器和地址译码器。水平驱动单元15顺序逐列扫描像素阵列11。通过水平驱动单元15执行的选择扫描操作，每一像素列被列处理单元14顺序地处理，并且被顺序输出。
例如，控制单元13根据操作模式配置时序信号，利用多种时序信号来控制垂直驱动单元13、列处理单元14和水平驱动单元15协同工作。
图3是本申请实施方式中像素阵列11及曝光控制线连接方式的示意图。像素阵列11为二维像素阵列。二维像素阵列包括多个全色像素和多个彩色像素，其中，彩色像素具有比全色像素更窄的光谱响应。像素阵列11中的像素排布为如下方式：
Figure PCTCN2019104974-appb-000001
需要说明的是，为了方便图示说明，图3中仅示出了像素阵列11中的部分像素，周边其它像素及连线以省略号“……”代替。
如图3所示，像素1101、1103、1106、1108、1111、1113、1116、及1118为全色像素W，像素1102、1105为第一颜色像素A(例如红色像素R)，像素1104、1107、1112、1115为第二颜色像素B(例如绿色像素G)，像素1114、1117为第三颜色像素C(例如蓝色像素Bu)。从图3中可以看出，全色像素W(像素1101、1103、1106和1108)中曝光控制电路的控制端TG与一条第一曝光控制线TX1连接，全色像素W(1111、1113、1116、和1118)中曝光控制电路的控制端TG与另一条第一曝光控制线TX1连接；第一颜色像素A(像素1102和1105)中曝光控制电路的控制端TG、第二颜色像素B(像素1104、1107)中曝光控制电路的控制端TG与一条第二曝光控制线TX2连接，第二颜色像素B(像素1112、1115)中曝光控制电路的控制端TG、第三颜色像素C(像素1114、1117)中曝光控制电路的控制端TG与另一条第二曝光控制线TX2连接。每条第一曝光控制线TX1可通过第一曝光控制信号控制全色像素的曝光时长；每条第二曝光控制线TX2可通过第二曝光控制信号控制彩色像素(例如第一颜色像素A和第二颜色像素B、第二颜色像素B和第三颜色像素C)的曝光时长。由此可实现全色像素和彩色像素曝光时长的独立控制。例如，可以实现在全色像素曝光结束时，彩色像素继续曝光，以达到理想的成像效果。
可以理解，在彩色图像传感器中，不同色彩的像素单位时间接收的曝光量不同，在某些色彩饱和后，某些色彩还未曝光到理想的状态。例如，曝光到饱和曝光量的60％-90％可以具有比较好的信噪比和精确度，但本申请的实施例不限于此。
图4中以RGBW(红、绿、蓝、全色)为例说明。图4中横轴为曝光时间、纵轴为曝光量，Q为饱 和的曝光量，LW为全色像素W的曝光曲线，LG为绿色像素G的曝光曲线，LR为红色像素R的曝光曲线，LB为蓝色像素的曝光曲线。从图4中可以看出，全色像素W的曝光曲线LW的斜率最大，也就是说在单位时间内全色像素W可以获得更多的曝光量，在t1时刻即达到饱和。绿色像素G的曝光曲线LG的斜率次之，绿色像素G在t2时刻饱和。红色像素R的曝光曲线LR的斜率再次之，红色像素R在t3时刻饱和。蓝色像素B的曝光曲线LB的斜率最小，蓝色像素B在t4时刻饱和。在t1时刻，全色像素W已经饱和，而R、G、B三种像素曝光还未达到理想状态。
相关技术中，RGBW四种像素的曝光时间是共同控制的。例如，每行像素的曝光时间是相同的，连接于同一曝光控制线，受同一曝光控制信号的控制。例如，继续参见图4，在0-t1时间段，RGBW四种像素都可以正常工作，但在此区间RGB由于曝光时间较短、曝光量较少，在图像显示时会造成亮度较低、信噪比较低、甚至色彩不够鲜艳的现象。在t1-t4时段，W像素由于饱和造成过度曝光，无法工作，曝光量数据已经无法真实反映目标。
基于上述原因，本申请提供的图像传感器10(图2所示)通过独立控制全色像素W的曝光时间和彩色像素的曝光时间，可以减少全色像素W对曝光时间的限制，均衡全色像素W与彩色像素(包括但不限于RGB)的曝光，从而提高图像拍摄质量。图3即为一个示例的独立控制全色像素W的曝光时间和彩色像素的曝光时间的方式，具体为通过不同的曝光控制线实现全色像素W与彩色像素的独立曝光控制，从而可以提高图像拍摄质量。
需要说明的是，图4中的曝光曲线仅为一个示例，根据像素响应波段的不同，曲线的斜率和相对关系会有所变化，本申请不限于图4中所示的情形。例如，当红色像素R响应的波段比较窄时，红色像素R的曝光曲线斜率可能比蓝色像素B曝光曲线的斜率更低。
请参考图2和图3，第一曝光控制线TX1和第二曝光控制线TX2与图2中的垂直驱动单元12连接，将垂直驱动单元12中相应的曝光控制信号传输到像素阵列11中像素的曝光控制电路的控制端TG。
可以理解的是，由于像素阵列11中有多个像素行组，垂直驱动单元12连接多条第一曝光控制线TX1和多条第二曝光控制线TX2。多条第一曝光控制线TX1和多条第二曝光控制线TX2对应于相应的像素行组。
例如，第一条第一曝光控制线TX1对应第一行和第二行中的全色像素；第二条第一曝光控制线TX1对应第三行和第四行中的全色像素，以此类推，第三条第一曝光控制线TX1对应第五行和第六行中的全色像素；第四条第一曝光控制线TX1对应第七行和第八行中的全色像素，再往下的第一曝光控制线TX1与再往下行的全色像素的对应关系不再赘述。不同第一曝光控制线TX1传输的信号时序也会有所不同，该信号时序由垂直驱动单元12配置。
例如，第一条第二曝光控制线TX2对应第一行和第二行中的彩色像素；第二条第二曝光控制线TX2对应第三行和第四行中的彩色像素，以此类推，第三条第二曝光控制线TX2对应第五行和第六行中的彩色像素；第四条第二曝光控制线TX2对应第七行和第八行中的彩色像素，再往下的第二曝光控制线TX2与再往下行的彩色像素的对应关系不再赘述。不同第二曝光控制线TX2传输的信号时序也会有所不同，该信号时序也由垂直驱动单元12配置。
图5是本申请实施方式中一种像素电路110的示意图。图5中像素电路110应用在图3的每个像素中。下面结合图3和图5对像素电路110的工作原理进行说明。
如图5所示，像素电路110包括光电转换元件117(例如，光电二极管PD)、曝光控制电路116(例如，转移晶体管112)、复位电路(例如，复位晶体管113)、放大电路(例如，放大晶体管114)和选择电路(例如，选择晶体管115)。在本申请的实施例中，转移晶体管112、复位晶体管113、放大晶体管114和选择晶体管115例如是MOS管，但不限于此。
例如，参见图2、图3和图5，转移晶体管112的栅极TG通过曝光控制线连接垂直驱动单元12；复位晶体管113的栅极RG通过复位控制线(图中未示出)连接垂直驱动单元12；选择晶体管114的栅极SEL通过选择线(图中未示出)连接垂直驱动单元12。每个像素电路110中的曝光控制电路116(例如，转移晶体管112)与光电转换元件117电连接，用于转移光电转换元件117经光照后积累的电势。例如，光电转换元件117包括光电二极管PD，光电二极管PD的阳极例如连接到地。光电二极管PD将所接收的光转换为电荷。光电二极管PD的阴极经由曝光控制电路116(例如，转移晶体管112)连接到浮动扩散单元FD。浮动扩散单元FD与放大晶体管114的栅极、复位晶体管113的源极连接。
例如，曝光控制电路116为转移晶体管112，曝光控制电路116的控制端TG为转移晶体管112的栅极。当有效电平(例如，VPIX电平)的脉冲通过曝光控制线(例如TX1或TX2)传输到转移晶体管112的栅极时，转移晶体管112导通。转移晶体管112将光电二极管PD光电转换的电荷传输到浮动扩散单元FD。
例如，复位晶体管113的漏极连接到像素电源VPIX。复位晶体管113的源极连接到浮动扩散单元FD。在电荷被从光电二极管PD转移到浮动扩散单元FD之前，有效复位电平的脉冲经由复位线传输到复位晶体管113的栅极，复位晶体管113导通。复位晶体管113将浮动扩散单元FD复位到像素电源VPIX。
例如，放大晶体管114的栅极连接到浮动扩散单元FD。放大晶体管114的漏极连接到像素电源VPIX。在浮动扩散单元FD被复位晶体管113复位之后，放大晶体管114经由选择晶体管115通过输出端OUT输出复位电平。在光电二极管PD的电荷被转移晶体管112转移之后，放大晶体管114经由选择晶体管115通过输出端OUT输出信号电平。
例如，选择晶体管115的漏极连接到放大晶体管114的源极。选择晶体管115的源极通过输出端OUT连接到图2中的列处理单元14。当有效电平的脉冲通过选择线被传输到选择晶体管115的栅极时，选择晶体管115导通。放大晶体管114输出的信号通过选择晶体管115传输到列处理单元14。
需要说明的是，本申请实施例中像素电路110的像素结构并不限于图5所示的结构。例如，像素电路110可以具有三晶体管像素结构，其中放大晶体管114和选择晶体管115的功能由一个晶体管完成。例如，曝光控制电路116也不局限于单个转移晶体管112的方式，其它具有控制端控制导通功能的电子器件或结构均可以作为本申请实施例中的曝光控制电路，单个转移晶体管112的实施方式简单、成本低、易于控制。
图6至图21示出了多种图像传感器10(图2所示)中像素排布的示例。参见图2、及图6至图21，图像传感器10包括由多个彩色像素(例如多个第一颜色像素A、多个第二颜色像素B和多个第三颜色像素C)和多个全色像素W组成的二维像素阵列(也即图3所示的像素阵列11)。其中，彩色像素具有比全色像素更窄的光谱响应。彩色像素的响应光谱例如为全色像素W响应光谱中的部分。二维像素阵列包括最小重复单元(图6至图21示出了多种图像传感器10中像素最小重复单元的示例)，二维像素阵列由多个最小重复单元组成，最小重复单元在行和列上复制并排列。在最小重复单元中，全色像素W设置在第一对角线方向D1，彩色像素设置在第二对角线方向D2，第一对角线方向D1与第二对角线方向D2不同。第一对角线方向D1相邻的至少两个全色像素的第一曝光时间由第一曝光信号控制，第二对角线方向D2相邻的至少两个彩色像素的第二曝光时间由第二曝光信号控制，从而实现全色像素曝光时间和彩色像素曝光时间的独立控制。每个最小重复单元均包括多个子单元，每个子单元包括多个单颜色像素(例如多个第一颜色像素A、多个第二颜色像素B或多个第三颜色像素C)和多个全色像素W。例如，请结合图3和图5，像素1101-1108及像素1111-1118组成一个最小重复单元，其中，像素1101、1103、1106、1108、1111、1113、1116、1118为全色像素，像素1102、1104、1105、1107、1112、1114、1115、1117为彩色像素。像素1101、1102、1105、1106组成一个子单元，其中，像素1101、1106为全色像素，像素1102、1105为单颜色像素(例如为第一颜色像素A)；像素1103、1104、1107、1108组成一个子单元，其中，像素1103、1108为全色像素，像素1104、1107为单颜色像素(例如为第二颜色像素B)；像素1111、1112、1115、1116组成一个子单元，其中，像素1111、1116为全色像素，像素1112、1115为单颜色像素(例如为第二颜色像素B)；像素1113、1114、1117、1118组成一个子单元，其中，像素1113、1118为全色像素，像素1114、1117为单颜色像素(例如为第三颜色像素C)。
例如，最小重复单元行和列的像素数量相等。例如最小重复单元包括但不限于，4行4列、6行6列、8行8列、10行10列的最小重复单元。例如，最小重复单元中的子单元行和列的像素数量相等。例如，子单元包括但不限于，2行2列、3行3列、4行4列、5行5列的子单元。这种设置有助于均衡行和列方向图像的分辨率和均衡色彩表现，提高显示效果。
例如，图6是本申请实施方式中一种最小重复单元1181像素排布的示意图；最小重复单元为4行4列16个像素，子单元为2行2列4个像素，排布方式为：
Figure PCTCN2019104974-appb-000002
Figure PCTCN2019104974-appb-000003
W表示全色像素；A表示多个彩色像素中的第一颜色像素；B表示多个彩色像素中的第二颜色像素；C表示多个彩色像素中的第三颜色像素。
例如，如图6所示，全色像素W设置在第一对角线方向D1(即图6中左上角和右下角连接的方向)，彩色像素设置在第二对角线方向D2(例如图6中左下角和右上角连接的方向)，第一对角线方向D1与第二对角线方向D2不同。例如，第一对角线和第二对角线垂直。第一对角线方向D1相邻的两个全色像素W(例如，从左上方起第一行第一列和第二行第二列的两个全色像素)的第一曝光时间由第一曝光信号控制，第二对角线方向D2相邻的至少两个彩色像素(例如，从左上方起第四行第一列和第三行第二列的两个彩色像素B)的第二曝光时间由第二曝光信号控制。
需要说明的是，第一对角线方向D1和第二对角线方向D2并不局限于对角线，还包括平行于对角线的方向，例如图6中，全色像素1101、1106、1113、及1118设置在第一对角线方向D1，全色像素1103及1108也设置在第一对角线方向D1，全色像素1111及1116也设置在第一对角线方向D1；第二颜色像素1104、1107、1112、及1115设置在第二对角线方向D2，第一颜色像素1102及1105也设置在第二对角线方向D2，第三颜色像素1114及1117也设置在第二对角线方向D2，下文图7至图21中对第一对角线方向D1及第二对角线方向D2的解释与此处相同。这里的“方向”并非单一指向，可以理解为指示排布的“直线”的概念，可以有直线两端的双向指向。
需要理解的是，此处以及下文中的术语“上”、“下”、“左”、“右”等指示的方位或位置关系为基于附图所示的方位或位置关系，仅是为了便于描述本申请和简化描述，而不是指示或暗示所指的装置或元件必须具有特定的方位、以特定的方位构造和操作，因此不能理解为对本申请的限制。
例如，如图6所示，第一行和第二行的全色像素由呈“W”型的第一曝光控制线TX1连接在一起，以实现全色像素曝光时间的单独控制。第一行和第二行的彩色像素(A和B)由呈“W”型的第二曝光控制线TX2连接在一起，以实现彩色像素曝光时间的单独控制。第三行和第四行的全色像素由呈“W”型的第一曝光控制线TX1连接在一起，以实现全色像素曝光时间的单独控制。第三行和第四行的彩色像素(B和C)由呈“W”型的第二曝光控制线TX2连接在一起，以实现彩色像素曝光时间的单独控制。例如，第一曝光信号经由第一曝光控制线TX1传输，第二曝光信号经由第二曝光控制线TX2传输。例如，第一曝光控制线TX1呈“W”型，与相邻两行的全色像素中曝光控制电路的控制端电连接；第二曝光控制线TX2呈“W”型，与相邻两行的彩色像素中曝光控制电路的控制端电连接。具体连接方式可参见前述图3和图5相关部分关于连接和像素电路的描述。
需要说明的是，第一曝光控制线TX1和第二曝光控制线TX2呈“W”型并不是指物理上走线必须严格按照“W”型设置，只需连接方式对应于全色像素和彩色像素的排布即可。例如，“W”型曝光控制线的设置对应“W”型的像素排布方式，这种设置方式走线简单，像素排布的解像力、色彩都有较好的效果，以低成本实现全色像素曝光时间和彩色像素曝光时间的独立控制。
例如，图7是本申请实施方式中又一种最小重复单元1182像素排布的示意图。最小重复单元为4行4列16个像素，子单元为2行2列4个像素，排布方式为：
Figure PCTCN2019104974-appb-000004
W表示全色像素；A表示多个彩色像素中的第一颜色像素；B表示多个彩色像素中的第二颜色像素；C表示多个彩色像素中的第三颜色像素。
例如，如图7所示，全色像素W设置在第一对角线方向D1(即图7中右上角和左下角连接的方向)，彩色像素设置在第二对角线方向D2(例如图7中左上角和右下角连接的方向)。例如，第一对角线和第二对角线垂直。第一对角线方向D1相邻的两个全色像素W(例如，从左上方起第一行第二列和第二行第一列的两个全色像素)的第一曝光时间由第一曝光信号控制，第二对角线方向相邻的至少两个彩色像素(例如，从左上方起第一行第一列和第二行第二列的两个彩色像素A)的第二曝光时间由第二曝光信号控制。
例如，如图7所示，第一行和第二行的全色像素由呈“W”型的第一曝光控制线TX1连接在一起， 以实现全色像素曝光时间的单独控制。第一行和第二行的彩色像素(A和B)由呈“W”型的第二曝光控制线TX2连接在一起，以实现彩色像素曝光时间的单独控制。第三行和第四行的全色像素由呈“W”型的第一曝光控制线TX1连接在一起，以实现全色像素曝光时间的单独控制。第三行和第四行的彩色像素(B和C)由呈“W”型的第二曝光控制线TX2连接在一起，以实现彩色像素曝光时间的单独控制。
例如，图8是本申请实施方式中又一种最小重复单元1183像素排布的示意图。图9是本申请实施方式中又一种最小重复单元1184像素排布的示意图。在图8和图9的实施例中，分别对应图6和图7的排布方式，第一颜色像素A为红色像素R；第二颜色像素B为绿色像素G；第三颜色像素C为蓝色像素Bu。
需要说明的是，在一些实施例中，全色像素W的响应波段为可见光波段(例如，400nm-760nm)。例如，全色像素W上设置有红外滤光片，以实现红外光的滤除。在一些实施例中，全色像素W的响应波段为可见光波段和近红外波段(例如，400nm-1000nm)，与图像传感器10中的光电转换元件(例如光电二极管PD)响应波段相匹配。例如，全色像素W可以不设置滤光片，全色像素W的响应波段由光电二极管的响应波段确定，即两者相匹配。本申请的实施例包括但不局限于上述波段范围。
例如，图10是本申请实施方式中又一种最小重复单元1185像素排布的示意图。图11是本申请实施方式中又一种最小重复单元1186像素排布的示意图。在图10和图11的实施例中，分别对应图6和图7的排布方式，第一颜色像素A为红色像素R；第二颜色像素B为黄色像素Y；第三颜色像素C为蓝色像素Bu。
例如，图12是本申请实施方式中又一种最小重复单元1187像素排布的示意图。图13是本申请实施方式中又一种最小重复单元1188像素排布的示意图。在图12和图13的实施例中，分别对应图6和图7的排布方式，第一颜色像素A为品红色像素M；第二颜色像素B为青色像素Cy；第三颜色像素C为黄色像素Y。
例如，图14是本申请实施方式中又一种最小重复单元1191像素排布的示意图。最小重复单元为6行6列36个像素，子单元为3行3列9个像素，排布方式为：
Figure PCTCN2019104974-appb-000005
W表示全色像素；A表示多个彩色像素中的第一颜色像素；B表示多个彩色像素中的第二颜色像素；C表示多个彩色像素中的第三颜色像素。
例如，如图14所示，第一行和第二行的全色像素由呈“W”型的第一曝光控制线TX1连接在一起，以实现全色像素曝光时间的单独控制。第一行和第二行的彩色像素(A和B)由呈“W”型的第二曝光控制线TX2连接在一起，以实现彩色像素曝光时间的单独控制。第三行和第四行的全色像素由呈“W”型的第一曝光控制线TX1连接在一起，以实现全色像素曝光时间的单独控制。第三行和第四行的彩色像素(A、B和C)由呈“W”型的第二曝光控制线TX2连接在一起，以实现彩色像素曝光时间的单独控制。第五行和第六行的全色像素由呈“W”型的第一曝光控制线TX1连接在一起，以实现全色像素曝光时间的单独控制。第五行和第六行的彩色像素(B和C)由呈“W”型的第二曝光控制线TX2连接在一起，以实现彩色像素曝光时间的单独控制。
例如，图15是本申请实施方式中又一种最小重复单元1192像素排布的示意图。最小重复单元为6行6列36个像素，子单元为3行3列9个像素，排布方式为：
Figure PCTCN2019104974-appb-000006
W表示全色像素；A表示多个彩色像素中的第一颜色像素；B表示多个彩色像素中的第二颜色像素；C表示多个彩色像素中的第三颜色像素。
例如，如图15所示，第一行和第二行的全色像素由呈“W”型的第一曝光控制线TX1连接在一起，以实现全色像素曝光时间的单独控制。第一行和第二行的彩色像素(A和B)由呈“W”型的第二曝光控制线TX2连接在一起，以实现彩色像素曝光时间的单独控制。第三行和第四行的全色像素由呈“W”型的第一曝光控制线TX1连接在一起，以实现全色像素曝光时间的单独控制。第三行和第四行的彩色像素(A、B和C)由呈“W”型的第二曝光控制线TX2连接在一起，以实现彩色像素曝光时间的单独控制。第五行和第六行的全色像素由呈“W”型的第一曝光控制线TX1连接在一起，以实现全色像素曝光时间的单独控制。第五行和第六行的彩色像素(B和C)由呈“W”型的第二曝光控制线TX2连接在一起，以实现彩色像素曝光时间的单独控制。
例如，图16是本申请实施方式中又一种最小重复单元1193像素排布的示意图。图17是本申请实施方式中又一种最小重复单元1194像素排布的示意图。在图16和图17的实施例中，分别对应图14和图15的排布方式，第一颜色像素A为红色像素R；第二颜色像素B为绿色像素G；第三颜色像素C为蓝色像素Bu。
例如，在其它实施方式中，第一颜色像素A为红色像素R；第二颜色像素B为黄色像素Y；第三颜色像素C为蓝色像素Bu。例如，在其它实施方式中，第一颜色像素A为品红色像素M；第二颜色像素B为青色像素Cy；第三颜色像素C为黄色像素Y。本申请的实施例包括但不局限于此。电路具体连接方式参见上文说明，在此不再赘述。
例如，图18是本申请实施方式中又一种最小重复单元1195像素排布的示意图。最小重复单元为8行8列64个像素，子单元为4行4列16个像素，排布方式为：
Figure PCTCN2019104974-appb-000007
W表示全色像素；A表示多个彩色像素中的第一颜色像素；B表示多个彩色像素中的第二颜色像素；C表示多个彩色像素中的第三颜色像素。
例如，如图18所示，第一行和第二行的全色像素由呈“W”型的第一曝光控制线TX1连接在一起，以实现全色像素曝光时间的单独控制。第一行和第二行的彩色像素(A和B)由呈“W”型的第二曝光控制线TX2连接在一起，以实现彩色像素曝光时间的单独控制。第三行和第四行的全色像素由呈“W”型的第一曝光控制线TX1连接在一起，以实现全色像素曝光时间的单独控制。第三行和第四行的彩色像素(A和B)由呈“W”型的第二曝光控制线TX2连接在一起，以实现彩色像素曝光时间的单独控制。第五行和第六行的全色像素由呈“W”型的第一曝光控制线TX1连接在一起，以实现全色像素曝光时间的单独控制。第五行和第六行的彩色像素(B和C)由呈“W”型的第二曝光控制线TX2连接在一起，以实现彩色像素曝光时间的单独控制。第七行和第八行的全色像素由呈“W”型的第一曝光控制线TX1连接在一起，以实现全色像素曝光时间的单独控制。第七行和第八行的彩色像素(B和C)由呈“W”型的第二曝光控制线TX2连接在一起，以实现彩色像素曝光时间的单独控制。
例如，图19是本申请实施方式中又一种最小重复单元1196像素排布的示意图。最小重复单元为8行8列64个像素，子单元为4行4列16个像素，排布方式为：
Figure PCTCN2019104974-appb-000008
Figure PCTCN2019104974-appb-000009
W表示全色像素；A表示多个彩色像素中的第一颜色像素；B表示多个彩色像素中的第二颜色像素；C表示多个彩色像素中的第三颜色像素。
例如，如图19所示，第一行和第二行的全色像素由呈“W”型的第一曝光控制线TX1连接在一起，以实现全色像素曝光时间的单独控制。第一行和第二行的彩色像素(A和B)由呈“W”型的第二曝光控制线TX2连接在一起，以实现彩色像素曝光时间的单独控制。第三行和第四行的全色像素由呈“W”型的第一曝光控制线TX1连接在一起，以实现全色像素曝光时间的单独控制。第三行和第四行的彩色像素(A和B)由呈“W”型的第二曝光控制线TX2连接在一起，以实现彩色像素曝光时间的单独控制。第五行和第六行的全色像素由呈“W”型的第一曝光控制线TX1连接在一起，以实现全色像素曝光时间的单独控制。第五行和第六行的彩色像素(B和C)由呈“W”型的第二曝光控制线TX2连接在一起，以实现彩色像素曝光时间的单独控制。第七行和第八行的全色像素由呈“W”型的第一曝光控制线TX1连接在一起，以实现全色像素曝光时间的单独控制。第七行和第八行的彩色像素(B和C)由呈“W”型的第二曝光控制线TX2连接在一起，以实现彩色像素曝光时间的单独控制。
例如，图20是本申请实施方式中又一种最小重复单元1197像素排布的示意图。图21是本申请实施方式中又一种最小重复单元1198像素排布的示意图。在图20和图21的实施例中，分别对应图18和图19的排布方式，第一颜色像素A为红色像素R；第二颜色像素B为绿色像素G；第三颜色像素C为蓝色像素Bu。
例如，在其它实施方式中，第一颜色像素A为红色像素R；第二颜色像素B为黄色像素Y；第三颜色像素C为蓝色像素Bu。例如，第一颜色像素A为品红色像素M；第二颜色像素B为青色像素Cy；第三颜色像素C为黄色像素Y。本申请的实施例包括但不局限于此。电路具体连接方式参见上文说明，在此不再赘述。
从上述实施例中可以看出，如图6至图21所示，图像传感器10(图2所示)包括矩阵排布的多个彩色像素和多个全色像素W，彩色像素和全色像素在行和列的方向上均间隔排布。
例如，在行的方向上依次交替设置全色像素、彩色像素、全色像素、彩色像素……
例如，在列的方向上依次交替设置全色像素、彩色像素、全色像素、彩色像素……
请结合图3和图4，第一曝光控制线TX1与第2n-1行和第2n行的全色像素W中曝光控制电路116的控制端TG(例如，转移晶体管112的栅极)电连接；第二曝光控制线TX2与第2n-1行和第2n行的彩色像素中曝光控制电路116的控制端TG(例如，转移晶体管112的栅极)电连接；n为大于等于1的自然数。
例如，当n＝1时，第一曝光控制线TX1与第1行和第2行的全色像素W中曝光控制电路116的控制端TG电连接；第二曝光控制线TX2与第1行和第2行的彩色像素中曝光控制电路116的控制端TG电连接。当n＝2时，第一曝光控制线TX1与第3行和第4行的全色像素W中曝光控制电路116的控制端TG电连接；第二曝光控制线TX2与第3行和第4行的彩色像素中曝光控制电路116的控制端TG电连接。以此类推，在此不再赘述。
在一些实施例中，第一曝光时间小于第二曝光时间。在一些实施例中，第一曝光时间与第二曝光时间的比例为1：2，1：3或1：4中的一种。例如，在光线比较暗的环境下，彩色像素更容易曝光不足，可以根据环境亮度调整第一曝光时间与第二曝光时间的比例为1：2，1：3或1：4。例如，曝光比例为上述整数比或接近整数比的情况下，有利于时序的设置信号的设置和控制。
请结合图22，相关技术中，图像传感器的像素阵列若同时包括全色像素及彩色像素，则图像传感器工作时，图像传感器会将像素阵列中的每个全色像素的像素值拟合到其他彩色像素的像素值中，从而输出仅包括彩色像素的原始图像。具体地，以像素A为红色像素R，像素B为绿色像素G，像素C为蓝色像素Bu为例，图像传感器中的列处理单元读出多个红色像素R的像素值、多个绿色像素G的像素值、多个蓝色像素Bu的像素值、及多个全色像素W的像素值后，图像传感器会先将每个全色像素W的像素值拟合到与该全色像素邻近的红色像素R、绿色像素G、及蓝色像素Bu中，再将非拜耳阵列排布的图像转换成拜耳阵列排布的原始图像输出，以供处理芯片对原始图像做后续处理，比如对原始图像做插 值处理以得到全彩图像(全彩图像中每个像素的像素值均由红色、绿色及蓝色三个分量组合而成)等。这一处理方式中，图像传感器需要执行较为复杂的算法，运算量比较大，且由于高通平台不支持非拜耳阵列排布的图像的处理，可能需要在图像传感器中增加额外的硬件(例如额外的处理芯片)来执行将非拜耳阵列排布的图像转换成拜耳阵列排布的原始图像的处理。
为减小图像传感器的运算量，以及避免在图像传感器中增加额外硬件，本申请提供一种图像采集方法。如图23所示，图像采集方法包括：
01：控制二维像素阵列曝光以获取全色原始图像和彩色原始图像；
02：处理彩色原始图像，以将每个子单元的所有像素作为与该子单元中单颜色对应的单色大像素，并输出单色大像素的像素值以得到彩色中间图像；
03：处理全色原始图像以得到全色中间图像；及
04：处理彩色中间图像和/或全色中间图像以获取目标图像。
请参阅图1和图2，本申请的图像采集方法可以由摄像头组件40实现。其中，步骤01可以由图像传感器10实现。步骤02、步骤03及步骤04可以由处理芯片20实现。也即是说，图像传感器10可以曝光以获取全色原始图像和彩色原始图像。处理芯片20可以用于处理彩色原始图像，以将每个子单元的所有像素作为与该子单元中单颜色对应的单色大像素，并输出单色大像素的像素值以得到彩色中间图像。处理芯片20还可以用于处理全色原始图像以得到全色中间图像、以及处理彩色中间图像和/或全色中间图像以获取目标图像。
具体地，请结合图2和图24，在用户请求拍照时，图像传感器10中的垂直驱动单元12会控制二维像素阵列中的多个全色像素和多个彩色像素均曝光，列处理单元14会读出每一个全色像素的像素值以及每一个彩色像素的像素值。图像传感器10不执行将全色像素的像素值拟合到彩色像素的像素值中的操作，而是直接根据多个全色像素的像素值输出一张全色原始图像，并直接根据多个彩色像素的像素值输出一张彩色原始图像。
如图24所示，全色原始图像包括多个全色像素W及多个空像素N(NULL)，其中，空像素既不为全色像素，也不为彩色像素，全色原始图像中空像素N所处位置可视为该位置没有像素，或者可以将空像素的像素值视为零。比较二维像素阵列与全色原始图像可知，对于二维像素阵列中的每一个子单元，该子单元包括两个全色像素W和两个彩色像素(彩色像素A、彩色像素B、或彩色像素C)。全色原始图像中也具有与二维像素阵列中的每一个子单元对应的一个子单元，全色原始图像的子单元包括两个全色像素W和两个空像素N，两个空像素N所处位置对应二维像素阵列子单元中的两个彩色像素所处的位置。
同样地，彩色原始图像包括多个彩色像素及多个空像素N，其中，空像素既不为全色像素，也不为彩色像素，彩色原始图像中空像素N所处位置可视为该位置没有像素，或者可以将空像素的像素值视为零。比较二维像素阵列与彩色原始图像可知，对于二维像素阵列中的每一个子单元，该子单元包括两个全色像素W和两个彩色像素。彩色原始图像中也具有与二维像素阵列中的每一个子单元对应的一个子单元，彩色原始图像的子单元包括两个彩色像素和两个空像素N，两个空像素N所处位置对应二维像素阵列子单元中的两个全色像素W所处的位置。
处理芯片20接收到图像传感器10输出的全色原始图像和彩色原始图像后，可以对全色原始图像作进一步处理得到全色中间图像，并对彩色原始图像作进一步处理得到彩色中间图像。示例地，彩色原始图像可通过图25所示的方式变换为彩色中间图像。如图25所示，彩色原始图像包括多个子单元，每个子单元都包括多个空像素N和多个单颜色的彩色像素(也称单颜色像素)。具体地，某些子单元包括两个空像素N和两个单颜色像素A，某些子单元包括两个空像素N和两个单颜色像素B，某些子单元包括两个空像素N及两个单颜色像素C。处理芯片20可以将包括空像素N和单颜色像素A的子单元中的所有像素作为与该子单元中的单颜色A对应的单色大像素A，将包括空像素N和单颜色像素B的子单元中的所有像素作为与该子单元中的单颜色B对应的单色大像素B，将包括空像素N和单颜色像素C的子单元中的所有像素作为与该子单元中的单颜色C对应的单色大像素C。由此，处理芯片20即可根据多个单色大像素A、多个单色大像素B、及多个单色大像素C形成一张彩色中间图像。如果包括多个空像素N的彩色原始图像视为一张具有第二分辨率的图像，则按照图25所示方式获取的彩色中间图像则为一张具有第一分辨率的图像，其中，第一分辨率小于第二分辨率。处理芯片20得到全色中间图像 以及彩色中间图像后，可以对全色中间图像和/或彩色中间图像作进一步处理以获取目标图像。具体地，处理芯片20可以仅处理全色中间图像以得到目标图像；处理芯片20也可以仅处理彩色中间图像以得到目标图像；处理芯片20还可以同时处理全色中间图像和彩色中间图像以得到目标图像。处理芯片20可以根据实际需求来决定两张中间图像的处理方式。
综上，本申请实施方式的图像采集方法中，图像传感器10可以直接输出全色原始图像和彩色原始图像，对于全色原始图像和彩色原始图像的后续处理由处理芯片20来执行，图像传感器10无需执行将全色像素W的像素值拟合到彩色像素的像素值中的操作，图像传感器10的运算量得到减小，并且无需在图像传感器10中增加新的硬件来支持图像传感器10执行图像处理，可以简化图像传感器10的设计。
在某些实施方式中，步骤01控制二维像素阵列曝光以获取全色原始图像和彩色原始图像可以根据多种方式实现。
请参阅图26，在一个例子中，步骤01包括：
011：控制二维像素阵列中的所有全色像素和所有彩色像素同时曝光；
012：输出所有全色像素的像素值以获取全色原始图像；及
013：输出所有彩色像素的像素值以获取彩色原始图像。
请参阅图1，步骤011、步骤012和步骤013均可以由图像传感器10实现。也即是说，图像传感器10中的所有全色像素和所有彩色像素同时曝光。图像传感器10可以输出所有全色像素的像素值以获取全色原始图像，还可以输出所有彩色像素的像素值以获取彩色原始图像。
请结合图2和图3，全色像素和彩色像素可以同时曝光，其中，全色像素的曝光时间可以小于或等于彩色像素的曝光时间。具体地，在全色像素的第一曝光时间与彩色像素的第二曝光时间相等时，全色像素的曝光起始时刻及曝光截止时刻分别与彩色像素的曝光起始时刻及曝光截止时刻相同。在第一曝光时间小于第二曝光时间时，全色像素的曝光起始时刻晚于或等于彩色像素的曝光起始时刻，且全色像素的曝光截止时刻早于彩色像素的曝光截止时刻；或者，在第一曝光时间小于第二曝光时间时，全色像素的曝光起始时刻晚于彩色像素的曝光起始时刻，且全色像素的曝光截止时刻早于或等于彩色像素的曝光截止时刻。全色像素和彩色像素曝均光完毕后，图像传感器10输出所有全色像素的像素值以获取全色原始图像，并输出所有彩色像素的像素值以获取彩色原始图像。其中，全色原始图像可以先于彩色原始图像输出，或者；彩色原始图像可以先于全色原始图像输出；或者，全色原始图像和彩色原始图像可以同时输出。二者的输出顺序在此不作限定。全色像素和彩色像素同时曝光可以减小全色原始图像及彩色原始图像的获取时间，加快全色原始图像及彩色原始图像获取进程。全色像素和彩色像素同时曝光的方式在快拍、连拍等对出图速度要求较高的模式下具有极大优势。
请参阅图27，在另一个例子中，步骤01包括：
014：控制二维像素阵列中的所有全色像素和所有彩色像素分时曝光；
015：输出所有全色像素的像素值以获取全色原始图像；及
016：输出所有彩色像素的像素值以获取彩色原始图像。
请参阅图1，步骤014、步骤015和步骤016均可以由图像传感器10实现。也即是说，图像传感器10中的所有全色像素和所有彩色像素分时曝光。图像传感器10可以输出所有全色像素的像素值以获取全色原始图像，还可以输出所有彩色像素的像素值以获取彩色原始图像。
具体地，全色像素和彩色像素可以分时曝光，其中，全色像素的曝光时间可以小于或等于彩色像素的曝光时间。具体地，无论第一曝光时间与第二曝光时间是否相等，所有全色像素和所有彩色像素分时曝光的方式均可以是：(1)所有全色像素先执行第一曝光时间的曝光，待所有全色像素曝光完毕后，所有彩色像素再执行第二曝光时间的曝光；(2)所有彩色像素先执行第二曝光时间的曝光，待所有彩色像素曝光完毕后，所有全色像素再执行第一曝光时间的曝光。全色像素和彩色像素曝均光完毕后，图像传感器10输出所有全色像素的像素值以获取全色原始图像，并输出所有彩色像素的像素值以获取彩色原始图像。其中，全色原始图像和彩色原始图像的输出方式可以是：(1)在全色像素先于彩色像素曝光时，图像传感器10可以在彩色像素曝光期间输出全色原始图像，也可以等彩色像素曝光完毕后再依次输出全色原始图像及彩色原始图像；(2)在彩色像素先于全色像素曝光时，图像传感器10可以在全色像素曝光期间输出彩色原始图像，也可以等全色像素曝光完毕后再依次输出彩色原始图像及全色原始图像；(3)无论全色像素和彩色像素中的哪一个优先曝光，图像传感器10可以在所有像素均曝光完毕后，同 时输出全色原始图像和彩色原始图像。本示例中全色像素和彩色像素分时曝光的方式的控制逻辑较为简单。
图像传感器10可以同时具有图26及图27所示的控制全色像素和彩色像素同时曝光、以及控制全色像素和彩色像素分时曝光的功能。图像传感器10在采集图像的过程中具体采用哪一种曝光方式，可以根据实际需求来自主选定。比如，在快拍、连拍等模式下时可以采用同时曝光的方式以满足快速出图的需求；在普通的拍照模式下可以采用分时曝光的方式以简化控制逻辑等。
图26及图27所示的两个示例中，全色像素和彩色像素的曝光顺序可以由图像传感器10中的控制单元13来控制。
图26及图27所示的两个示例中，全色像素的曝光时间可以由第一曝光信号控制，彩色像素的曝光时间可以由第二曝光信号控制。
具体地，请结合图3，作为一个示例，图像传感器10可以用第一曝光信号控制第一对角线方向相邻的至少两个全色像素以第一曝光时间曝光，并用第二曝光信号控制第二对角线方向相邻的至少两个彩色像素以第二曝光时间曝光，其中，第一曝光时间可以小于或等于第二曝光时间。具体地，图像传感器10中的垂直驱动单元12通过第一曝光控制线TX1传输第一曝光信号以控制第一对角线方向相邻的至少两个全色像素以第一曝光时间曝光，垂直驱动单元12通过第二曝光控制线TX2传输第二曝光信号以控制第二对角线方向相邻的至少两个全色像素以第二曝光时间曝光。待所有全色像素及所有彩色像素均曝光完成后，如图24所示，图像传感器10不执行将多个全色像素的像素值拟合到彩色像素的像素值中的处理，而是直接输出一张全色原始图像和一张彩色原始图像。
请结合图2和图6，作为另一个示例，图像传感器10可以用第一曝光信号控制第2n-1行和第2n行的全色像素以第一曝光时间曝光，并用第二曝光信号控制第2n-1行和第2n行的彩色像素以第二曝光时间曝光，其中，第一曝光时间可以小于或等于第二曝光时间。具体地，图像传感器10中的第一曝光控制线TX1与2n-1行和第2n行的所有全色像素的控制端TG连接，第二曝光控制线TX2与2n-1行和第2n行的所有彩色像素的控制端TG连接。垂直驱动单元12通过第一曝光控制线TX1传输第一曝光信号以控制第2n-1行和第2n行的全色像素以第一曝光时间曝光，通过第二曝光控制线TX2传输第二曝光信号以控制第2n-1行和第2n行的彩色像素以第二曝光时间曝光。待所有全色像素及所有彩色像素均曝光完成后，如图24所示，图像传感器10不执行将多个全色像素的像素值拟合到彩色像素的像素值中的处理，而是直接输出一张全色原始图像和一张彩色原始图像。
在某些实施方式中，处理芯片20可以根据环境亮度来确定第一曝光时间与第二曝光时间的相对关系。示例地，图像传感器10可以先控制全色像素曝光并输出一张全色原始图像，处理芯片20分析全色原始图像中多个全色像素的像素值来确定环境亮度。在环境亮度小于或等于亮度阈值时，图像传感器10控制全色像素以等于第二曝光时间的第一曝光时间来曝光；在环境亮度大于亮度阈值时，图像传感器10控制全色像素以小于第二曝光时间的第一曝光时间来曝光。在环境亮度大于亮度阈值时，可以根据环境亮度与亮度阈值之间的亮度差值来确定第一曝光时间与第二曝光时间的相对关系，例如，亮度差值越大，第一曝光时间与第二曝光时间的比例越小。示例地，在亮度差值位于第一范围[a,b)内时，第一曝光时间与第二曝光时间的比例为1:2；在亮度差值位于第二范围[b,c)内时，第一曝光时间与第二曝光时间的比例为1:3；在亮度差值大于或等于c时，第一曝光时间与第二曝光时间的比例为1:4。
请参阅图28，在某些实施方式中，步骤02包括：
021：合并每个子单元中的所有像素的像素值以得到单色大像素的像素值；及
022：根据多个单色大像素的像素值形成彩色中间图像，彩色中间图像具有第一分辨率。
请参阅图1，在某些实施方式中，步骤021及步骤022均可以由处理芯片20实现。也即是说，处理芯片20可以用于合并每个子单元中的所有像素的像素值以得到单色大像素的像素值、以及根据多个单色大像素的像素值形成彩色中间图像，彩色中间图像具有第一分辨率。其中，彩色中间图像具有第一分辨率。
具体地，如图25所示，对于单色大像素A，处理芯片20可以将包括空像素N和单颜色像素A的子单元中的所有像素的像素值相加，并将相加的结果作为对应该子单元的单色大像素A的像素值，其中，空像素N的像素值可以视为零，下同；处理芯片20可以将包括空像素N和单颜色像素B的子单元中的所有像素的像素值相加，并将相加的结果作为对应该子单元的单色大像素B的像素值；处理芯片20可 以将包括空像素N和单颜色像素C的子单元中的所有像素的像素值相加，并将相加的结果作为对应该子单元的单色大像素C的像素值。由此，处理芯片20即可获得多个单个大像素A的像素值、多个单色大像素B的像素值、以及多个单色大像素C的像素值。处理芯片20再根据多个单色大象素A的像素值、多个单色大像素B的像素值、以及多个单色大像素C的像素值形成一张彩色中间图像。如图25所示，当单颜色A为红色R，单颜色B为绿色G，单颜色C为蓝色Bu时，彩色中间图像即为拜耳阵列排布的图像。当然，处理芯片20获取彩色中间图像的方式并不限于此。
在某些实施方式中，请结合图1和图29，当摄像头组件40处于不同的模式时，不同模式对应不同的目标图像。处理芯片20会先判断摄像头组件40处于哪一种模式，再根据摄像头组件40所处的模式对彩色中间图像和/或全色中间图像做相应处理以得到对应该模式的目标图像。目标图像至少包括四类目标图像：第一目标图像、第二目标图像、第三目标图像、第四目标图像。摄像头组件40所处的模式至少包括：(1)模式为预览模式，预览模式下的目标图像可以为第一目标图像或第二目标图像；(2)模式为成像模式，成像模式下的目标图像可以为第二目标图像、第三目标图像或第四目标图像；(3)模式既为预览模式又为低功耗模式，此时目标图像为第一目标图像；(4)模式即为预览模式又为非低功耗模式，此时目标图像为第二目标图像；(5)模式既为成像模式又为低功耗模式，此时目标图像为第二目标图像或第三目标图像；(6)模式既为成像模式又为非低功耗模式，此时目标图像为第四目标图像。
请参阅图29，在一个例子中，当目标图像为第一目标图像时，步骤04包括：
040：对彩色中间图像中的每个单色大像素进行插值处理以获取除该单颜色以外的另外两种颜色的像素值并输出以得到具有第一分辨率的第一目标图像。
请参阅图1，步骤040可以由处理芯片20实现。也即是说，处理芯片20可以用于对彩色中间图像中的每个单色大像素进行插值处理以获取除该单颜色以外的另外两种颜色的像素值并输出以得到具有第一分辨率的第一目标图像。
具体地，请结合图30，假设单色大像素A为红色像素R，单色大像素B为绿色像素G，单色大像素C为蓝色像素Bu，则彩色中间图像为拜耳阵列排布的图像，处理芯片20需要对彩色中间图像执行去马赛克(即插值处理)，以使得每个单色大像素的像素值都同时具有R、G、B三个分量。示例地，可以采用线性插值的方式来计算每个单色大像素的除单色大像素的单颜色以外的另外两种颜色的像素值。以单色大像素C 2,2(“C 2,2”表示从左上方算起第二行第二列的像素C)为例，单色大像素C 2,2仅具有颜色C的分量的像素值P(C 2,2)，还需要计算出单色大像素C位置处颜色A的像素值P(A 2,2)和颜色B的像素值P(B 2,2)，则P(A 2,2)＝α 1·P(A 3,1)+α 2·P(A 3,3)+α 3·P(A 1,3)+α 4·P(A 1,1)，P(B 2,2)＝β 1·P(B 1,2)+β 2·P(B 2,1)+β 3·P(B 2,3)+β 4·P(B 3,2)，其中，α 1～α 4与β 1～β 4均为插值系数，且α 1+α 2+α 3+α 4＝1，β 1+β 2+β 3+β 4＝1。上述P(A 2,2)及P(B 2,2)的计算方式仅为示例，P(A 2,2)及P(B 2,2)还可以通过除线性插值方式以外的其他插值方式计算得到，在此不作限制。
处理芯片20计算出每个单色大像素的三个分量的像素值后，即可根据三个像素值计算出对应该单色大像素的最终的像素值，即A+B+C，需要说明的是，此处的A+B+C并不表示直接将三个像素相加得到单色大像素最终的像素值，仅表示单色大像素包括A、B、C三个色彩分量。处理芯片20可以根据多个单色大像素的最终的像素值形成一张第一目标图像。由于彩色中间图像具有第一分辨率，第一目标图像为彩色中间图像经插值处理得到，处理芯片20未对彩色中间图像做插补处理，因此，第一目标图像的分辨率也为第一分辨率。处理芯片20处理彩色中间图像得到第一目标图像的处理算法较为简单，处理速度较快，摄像头组件40在模式既为预览模式又为低功耗模式时使用第一目标图像作为预览图像，既可以满足预览模式对出图速度的需求，还可以节省摄像头组件40的功耗。
请再参阅图29，在另一个例子中，当目标图像为第二目标图像时，步骤03包括：
031：处理全色原始图像，将每个子单元的所有像素作为全色大像素，并输出全色大像素的像素值以得到全色中间图像，全色中间图像具有第一分辨率；
步骤04包括：
041：分离彩色中间图像的色彩及亮度以得到具有第一分辨率的色亮分离图像；
042：融合全色中间图像的亮度及色亮分离图像的亮度以得到具有第一分辨率的亮度修正彩色图像；及
043：对亮度修正彩色图像中的每个单色大像素进行插值处理以获取除该单颜色以外的另外两种颜 色的像素值并输出以得到具有第一分辨率的第二目标图像。
请参阅图1，步骤031、步骤041、步骤042及步骤043均可以由处理芯片20实现。也即是说，处理芯片20可以用于处理全色原始图像，将每个子单元的所有像素作为全色大像素，并输出全色大像素的像素值以得到全色中间图像，全色中间图像具有第一分辨率。处理芯片20还可以用于分离彩色中间图像的色彩及亮度以得到具有第一分辨率的色亮分离图像、融合全色中间图像的亮度及色亮分离图像的亮度以得到具有第一分辨率的亮度修正彩色图像、以及对亮度修正彩色图像中的每个单色大像素进行插值处理以获取除该单颜色以外的另外两种颜色的像素值并输出以得到具有第一分辨率的第二目标图像。
具体地，全色原始图像可通过图31所示的方式变换为全色中间图像。如图31所示，全色原始图像包括多个子单元，每个子单元都包括两个空像素N和两个全色像素W。处理芯片20可以将每个包括空像素N和全色像素W的子单元中的所有像素作为与该子单元对应的全色大像素W。由此，处理芯片20即可根据多个全色大像素W形成一张全色中间图像。如果包括多个空像素N的全色原始图像视为一张具有第二分辨率的图像，则按照图31所示方式获取的全色中间图像则为一张具有第一分辨率的图像，其中，第一分辨率小于第二分辨率。
作为一个示例，处理芯片20可以通过以下方式将全色原始图像中每个子单元的所有像素作为与该子单元对应的全色大像素W：处理芯片20首先合并每个子单元中的所有像素的像素值以得到全色大像素W的像素值，再根据多个全色大像素W的像素值形成全色中间图像。具体地，对于每个全色大像素，处理芯片20可以将包括空像素N和全色像素W的子单元中的所有像素值相加，并将相加的结果作为对应该子单元的全色大像素W的像素值，其中，空像素N的像素值可以视为零。由此，处理芯片20即可获得多个全色大像素W的像素值。
处理芯片20获得全色中间图像和彩色中间图像后，可以对全色中间图像和彩色中间图像做融合处理以得到第二目标图像。
示例地，如图31所示，处理芯片20首先分离彩色中间图像的色彩及亮度以获取色亮分离图像，图31中色亮分离图像中的L表示亮度，CLR表示色彩。具体地，假设单颜色像素A为红色像素R，单颜色像素B为绿色像素G，单颜色像素C为蓝色像素Bu，则：(1)处理芯片20可以将RGB空间的彩色中间图像转换为YCrCb空间的色亮分离图像，此时YCrCb中的Y即为色亮分离图像中的亮度L，YCrCb中的Cr和Cb即为色亮分离图像中的色彩CLR；(2)处理芯片20也可以将RGB的彩色中间图像转换为Lab空间的色亮分离图像，此时Lab中的L即为色亮分离图像中的亮度L，Lab中的a和b即为色亮分离图像中的色彩CLR。需要说明的是，图31所示色亮分离图像中L+CLR并不表示每个像素的像素值由L和CLR相加而成，仅表示每个像素的像素值是由L和CLR组成。
随后，处理芯片20融合色亮分离图像的亮度以及全色中间图像的亮度。示例地，全色中间图像中每个全色像素W的像素值即为每个全色像素的亮度值，处理芯片20可以将色亮分离图像中每个像素的L与全色中间图像中对应位置的全色像素的W相加，即可得到亮度修正后的像素值。处理芯片20根据多个亮度修正后的像素值形成一张亮度修正后的色亮分离图像，再利用色彩空间转换将亮度修正后的色亮分离图像转换为亮度修正彩色图像。
在单色大像素A为红色像素R，单色大像素B为绿色像素G，单色大像素C为蓝色像素Bu时，亮度修正彩色图像为拜耳阵列排布的图像，处理芯片20需要对亮度修正彩色图像做插值处理，以使得每个修正了亮度后的单色大像素的像素值都同时具有R、G、B三个分量。处理芯片20可以对亮度修正彩色图像做插值处理以得到第二目标图像，示例地，可采用线性插值方式来获取第二目标图像，线性插值过程与前述步骤40中的插值过程类似，在此不再赘述。
由于亮度修正彩色图像具有第一分辨率，第二目标图像为亮度修正彩色图像经插值处理得到，处理芯片20未对亮度修正彩色图像做插补处理，因此，第二目标图像的分辨率也为第一分辨率。由于第二目标图像是融合了彩色中间图像的亮度及全色中间图像亮度得到的，因此第二目标图像具有更好的成像效果。在模式为预览模式又为非低功耗模式时使用第二目标图像作为预览图像，可以提升预览图像的预览效果。在模式为成像模式又为低功耗模式时，使用第二目标图像作为提供给用户的图像，由于第二目标图像是无需经过插补处理计算得到的，一定程度上可以减小摄像头组件40的功耗，能够满足低功耗模式下的使用需求；同时第二目标图像的亮度较亮，可以满足用户对目标图像的亮度要求。
请再参阅图29，在又一个例子中，当目标图像为第三目标图像时，步骤04包括：
044：插补处理彩色中间图像以得到具有第二分辨率的彩色插补图像，彩色插补图像中对应的子单元呈拜耳阵列排布，第二分辨率大于第一分辨率；及
045：对彩色插补图像中的所有单颜色像素进行插值处理以获取除该单颜色以外的另外两种颜色的像素值并输出以得到具有第二分辨率的第三目标图像。
请参阅图1，步骤044和步骤045均可以由处理芯片20实现。也即是说，处理芯片20可以用于插补处理彩色中间图像以得到具有第二分辨率的彩色插补图像，彩色插补图像中对应的子单元呈拜耳阵列排布，第二分辨率大于第一分辨率。处理芯片20还可以用于对彩色插补图像中的所有单颜色像素进行插值处理以获取除该单颜色以外的另外两种颜色的像素值并输出以得到具有第二分辨率的第三目标图像。
具体地，请结合图32，处理芯片20将彩色中间图像中的每一个单色大像素拆分为四个彩色像素，四个彩色像素组成彩色插补图像中的一个子单元，每个子单元中包括三种颜色的彩色像素，分别为一个彩色像素A、两个彩色像素B、及一个彩色像素C。当彩色像素A为红色像素R，彩色像素B为绿色像素G，彩色像素C为蓝色像素Bu时，每个子单元中的多个彩色像素即呈拜耳阵列排布。由此，包含多个子单元的彩色插补图像即为拜耳阵列排布的图像。处理芯片20可以对彩色插补图像做插值处理以得到第三目标图像，示例地，可采用线性插值方式来获取第二目标图像，线性插值过程与前述步骤040中的插值过程类似，在此不再赘述。第三目标图像是经过插补处理得到的图像，第三目标图像的分辨率(即第二分辨率)比彩色中间图像的分辨率(即一分辨率)来得大。在模式即为预览模式又为非低功耗模式时，将第三目标图像作为预览图像，可以得到更为清晰的预览图像。在模式既为成像模式又为低功耗模式时，将第三目标图像作为提供给用户的图像，由于第三目标形成过程中不需要与全色中间图像做亮度融合，可以在一定程度上降低摄像头组件40的功耗，同时又能满足用户对拍摄图像的清晰度要求。
请再参阅图29，在又一个例子中，当目标图像为第四目标图像时，步骤03包括：
032：插补处理全色原始图像，获取每个子单元中的所有像素的像素值以得到具有第二分辨率的全色中间图像；
步骤04包括：
046：插补处理彩色中间图像以得到具有第二分辨率的彩色插补图像，彩色插补图像中对应的子单元呈拜耳阵列排布，第二分辨率大于第一分辨率；
047：分离彩色插补图像的色彩及亮度以得到具有第二分辨率的色亮分离图像；
048：融合全色插补图像的亮度及色亮分离图像的亮度以得到具有第二分辨率的亮度修正彩色图像；及
049：对亮度修正彩色图像中的所有单颜色像素进行插值处理以获取除该单颜色以外的另外两种颜色的像素值并输出以得到具有第二分辨率的第四目标图像。
请参阅图1，步骤032、步骤046、步骤047、步骤048及步骤049均可以由处理芯片20实现。也即是说，处理芯片20可以用于插补处理全色原始图像，获取每个子单元中的所有像素的像素值以得到具有第二分辨率的全色中间图像。处理芯片20还可以用于插补处理彩色中间图像以得到具有第二分辨率的彩色插补图像，彩色插补图像中对应的子单元呈拜耳阵列排布，第二分辨率大于第一分辨率。处理芯片20还可以用于分离彩色插补图像的色彩及亮度以得到具有第二分辨率的色亮分离图像、融合全色插补图像的亮度及色亮分离图像的亮度以得到具有第二分辨率的亮度修正彩色图像、对亮度修正彩色图像中的所有单颜色像素进行插值处理以获取除该单颜色以外的另外两种颜色的像素值并输出以得到具有第二分辨率的第四目标图像。
具体地，处理芯片20首先要对第一分辨率的全色原始图像做插补处理以得到第二分辨率的全色中间图像。请结合图34，全色原始图像包括多个子单元，每个子单元包括两个空像素N和两个全色像素W，处理芯片20需要将每个子单元中的每个空像素N均替换为全色像素W，并计算出替换后位于空像素N所在位置的每个全色像素W的像素值。对于每一个空像素N，处理芯片20将该空像素N替换为全色像素W，并根据与该替换后的全色像素W相邻的其余全色像素W的像素值来确定该替换后的全色像素W的像素值。以图34所示全色原始图像中的空像素N 1,8(“空像素N 1,8”为从左上方算起第一行第八列的空像素N，下同)为例，空像素N 1,8替换为全色像素W 1,8，与全色像素W 1,8相邻的像素为全色原始图像中的全色像素W 1,7以及全色像素W 2,8，作为示例，可以将全色像素W 1,7的像素值和全色像素W 2,8 的像素值的均值作为全色像素W 1,8的像素值。以图34所示全色原始图像中的空像素N 2,3为例，空像素N 2,3替换为全色像素W 2,3，与全色像素W 2,3相邻的全色像素为全色原始图像中的全色像素W 1,3、全色像素W 2,2、全色像素W 2,4、以及全色像素W 3,3，作为示例，处理芯片20将全色像素W 1,3的像素值、全色像素W 2,2的像素值、全色像素W 2,4的像素值、以及全色像素W 3,3的像素值的均值作为替换后的全色像素W 2,3的像素值。
处理芯片20获得全色中间图像和彩色中间图像后，可以对全色中间图像和彩色中间图像做融合处理以得到第四目标图像。
首先，处理芯片20可以对第一分辨率的彩色中间图像做插补处理以得到第二分辨率的彩色插补图像，如图33所示。具体插补方式与步骤045中的插补方式类似，在此不做赘述。
随后，如图33所示，处理芯片20可以分离彩色插补图像的色彩及亮度以获取色亮分离图像，图33中色亮分离图像中的L表示亮度，CLR表示色彩。具体地，假设单颜色像素A为红色像素R，单颜色像素B为绿色像素G，单颜色像素C为蓝色像素Bu，则：(1)处理芯片20可以将RGB空间的彩色插补图像转换为YCrCb空间的色亮分离图像，此时YCrCb中的Y即为色亮分离图像中的亮度L，YCrCb中的Cr和Cb即为色亮分离图像中的色彩CLR；(2)处理芯片20也可以将RGB的彩色插补图像转换为Lab空间的色亮分离图像，此时Lab中的L即为色亮分离图像中的亮度L，Lab中的a和b即为色亮分离图像中的色彩CLR。需要说明的是，图33所示色亮分离图像中L+CLR并不表示每个像素的像素值由L和CLR相加而成，仅表示每个像素的像素值是由L和CLR组成。
随后，如图34所示，处理芯片20可以融合色亮分离图像的亮度以及全色中间图像的亮度。示例地，全色中间图像中每个全色像素W的像素值即为每个全色像素的亮度值，处理芯片20可以将色亮分离图像中每个像素的L与全色中间图像中对应位置的全色像素的W相加，即可得到亮度修正后的像素值。处理芯片20根据多个亮度修正后的像素值形成一张亮度修正后的色亮分离图像，再将亮度修正后的色亮分离图像转换为亮度修正彩色图像，该亮度修正彩色图像具有第二分辨率。
在彩色像素A为红色像素R，彩色像素B为绿色像素G，彩色像素C为蓝色像素Bu时，亮度修正彩色图像为拜耳阵列排布的图像，处理芯片20需要对亮度修正彩色图像做插值处理，以使得每个修正了亮度后的彩色像素的像素值都同时具有R、G、B三个分量。处理芯片20可以对亮度修正彩色图像做插值处理以得到第四目标图像，示例地，可采用线性插值方式来获取第四目标图像，线性插值过程与前述步骤40中的插值过程类似，在此不再赘述。
由于第四目标图像是融合了彩色中间图像的亮度及全色中间图像亮度得到的，且第四目标图像具有较大的分辨率，因此第四目标图像具有更好的亮度和清晰度。在模式为既为成像模式又为非低功耗模式时使用第四目标图像作为提供给用户的图像，可以满足用户对拍摄图像的质量要求。
在某些实施方式中，图像采集方法还可以包括获取环境亮度。该步骤可以由处理芯片20实现，具体实现方式如前所述，在此不再赘述。在环境亮度大于亮度阈值时，可以将第一目标图像或第三目标图像作为目标图像；在环境亮度小于或等于亮度阈值时，可以将第二目标图像或第四目标图像作为目标图像。可以理解，在环境亮度较亮时，仅由彩色中间图像得到的第一目标图像和第二目标图像的亮度已经足够满足用户对目标图像的亮度需求，此时可以无需融合全色中间图像的亮度来提升目标图像的亮度，如此，不仅可以减小处理芯片20的计算量，还可以降低摄像头组件40的功耗。在环境亮度较低时，仅由彩色中间图像得到的第一目标图像和第二目标图像的亮度可能无法满足用户对对目标图像的亮度需求，将融合了全色中间图像的亮度得到的第二目标图像或第四目标图像作为目标图像，可以提升目标图像的亮度。
请参阅图35，本申请还提供一种移动终端90。移动终端90可以是手机、平板电脑、笔记本电脑、智能穿戴设备(如智能手表、智能手环、智能眼镜、智能头盔等)、头显设备、虚拟现实设备等等，在此不做限制。
移动终端90包括图像传感器50、处理器60、存储器70和机壳80，图像传感器50、处理器60和存储器70均安装在机壳80中。其中，图像传感器50与处理器60连接，图像传感器50可以为上述任意一项实施方式所述的图像传感器10(图1所示)。处理器60可以执行与摄像头组件40(图1所示)中的处理芯片20相同的功能，换言之，处理器60可以实现上述任意一项实施方式所述的处理芯片20所能实现的功能。存储器70与处理器60连接，存储器70可以存储处理器60处理后得到的数据，如目 标图像等。处理器60可以与图像传感器50安装在同一个基板上，此时图像传感器50和处理器60可视为一个摄像头组件40。当然，处理器60也可以与图像传感器50安装在不同的基板上。
本申请的移动终端90，图像传感器50可以直接输出全色原始图像和彩色原始图像，对于全色原始图像和彩色原始图像的后续处理由处理器60来执行，图像传感器50无需执行将全色像素W的像素值拟合到彩色像素的像素值中的操作，图像传感器50的运算量得到减小，并且无需在图像传感器50中增加新的硬件来支持图像传感器50执行图像处理，可以简化图像传感器50的设计。
在本说明书的描述中，参考术语“一个实施方式”、“一些实施方式”、“示意性实施方式”、“示例”、“具体示例”或“一些示例”等的描述意指结合所述实施方式或示例描述的具体特征、结构、材料或者特点包含于本申请的至少一个实施方式或示例中。在本说明书中，对上述术语的示意性表述不一定指的是相同的实施方式或示例。而且，描述的具体特征、结构、材料或者特点可以在任何的一个或多个实施方式或示例中以合适的方式结合。此外，在不相互矛盾的情况下，本领域的技术人员可以将本说明书中描述的不同实施例或示例以及不同实施例或示例的特征进行结合和组合。
流程图中或在此以其他方式描述的任何过程或方法描述可以被理解为，表示包括一个或更多个用于实现特定逻辑功能或过程的步骤的可执行指令的代码的模块、片段或部分，并且本申请的优选实施方式的范围包括另外的实现，其中可以不按所示出或讨论的顺序，包括根据所涉及的功能按基本同时的方式或按相反的顺序，来执行功能，这应被本申请的实施例所属技术领域的技术人员所理解。
尽管上面已经示出和描述了本申请的实施方式，可以理解的是，上述实施方式是示例性的，不能理解为对本申请的限制，本领域的普通技术人员在本申请的范围内可以对上述实施方式进行变化、修改、替换和变型。

Claims (80)

  1. 一种图像传感器的图像采集方法，其特征在于，所述图像传感器包括二维像素阵列，所述二维像素阵列包括多个全色像素和多个彩色像素，所述二维像素阵列包括最小重复单元，每个所述最小重复单元包含多个子单元，每个所述子单元包括多个单颜色像素及多个全色像素；所述图像采集方法包括：
    控制所述二维像素阵列曝光以获取全色原始图像和彩色原始图像；
    处理所述彩色原始图像，以将每个所述子单元的所有像素作为与该子单元中单颜色对应的单色大像素，并输出所述单色大像素的像素值以得到彩色中间图像；
    处理所述全色原始图像以得到全色中间图像；及
    处理所述彩色中间图像和/或所述全色中间图像以获取目标图像。
  2. 根据权利要求1所述的图像采集方法，其特征在于，所述控制所述二维像素阵列曝光以获取全色原始图像和彩色原始图像，包括：
    控制所述二维像素阵列中的所有所述全色像素和所有所述彩色像素同时曝光；
    输出所有所述全色像素的像素值以获取所述全色原始图像；及
    输出所有所述彩色像素的像素值以获取所述彩色原始图像。
  3. 根据权利要求1所述的图像采集方法，其特征在于，所述控制所述二维像素阵列曝光以获取全色原始图像和彩色原始图像，包括：
    控制所述二维像素阵列中的所有所述全色像素和所有所述彩色像素分时曝光；
    输出所有所述全色像素的像素值以获取所述全色原始图像；及
    输出所有所述彩色像素的像素值以获取所述彩色原始图像。
  4. 根据权利要求1所述的图像采集方法，其特征在于，在所述最小重复单元中，所述全色像素设置在第一对角线方向，所述彩色像素设置在第二对角线方向，所述第一对角线方向与所述第二对角线方向不同；
    所述控制所述二维像素阵列曝光以获取全色原始图像和彩色原始图像，包括：
    用第一曝光信号控制所述第一对角线方向相邻的至少两个所述全色像素以第一曝光时间曝光；及
    用第二曝光信号控制所述第二对角线方向相邻的至少两个所述彩色像素以第二曝光时间曝光。
  5. 根据权利要求1所述的图像采集方法，其特征在于，所述控制所述二维像素阵列曝光以获取全色原始图像和彩色原始图像，包括：
    用第一曝光信号控制第2n-1行和第2n行的所述全色像素的第一曝光时间；及
    用第二曝光信号控制第2n-1行和第2n行的所述彩色像素的第二曝光时间；
    其中，n为大于等于1的自然数。
  6. 根据权利要求4或5所述的图像采集方法，其特征在于，所述图像采集方法还包括：
    获取环境亮度；
    在所述环境亮度大于亮度阈值时，所述第一曝光时间小于所述第二曝光时间。
  7. 根据权利要求1所述的图像采集方法，其特征在于，所述处理所述彩色原始图像，以将每个所述子单元的所有像素作为与该子单元中单颜色对应的单色大像素，并输出所述单色大像素的像素值以得到彩色中间图像，包括：
    合并每个所述子单元中的所有所述像素的像素值以得到所述单色大像素的像素值；及
    根据多个所述单色大像素的像素值形成所述彩色中间图像，所述彩色中间图像具有第一分辨率。
  8. 根据权利要求7所述的图像采集方法，其特征在于，所述处理所述彩色中间图像和/或所述全色中间图像以获取目标图像，包括：
    对所述彩色中间图像中的每个所述单色大像素进行插值处理以获取除该单颜色以外的另外两种颜色的像素值并输出以得到具有所述第一分辨率的第一目标图像。
  9. 根据权利要求7所述的图像采集方法，其特征在于，所述处理所述全色原始图像以得到全色中间图像，包括：
    处理所述全色原始图像，将每个所述子单元的所有像素作为全色大像素，并输出所述全色大像素的像素值以得到全色中间图像，所述全色中间图像具有所述第一分辨率；
    所述处理所述彩色中间图像和/或所述全色中间图像以获取目标图像，包括：
    分离所述彩色中间图像的色彩及亮度以得到具有所述第一分辨率的色亮分离图像；
    融合所述全色中间图像的亮度及所述色亮分离图像的亮度以得到具有所述第一分辨率的亮度修正彩色图像；及
    对所述亮度修正彩色图像中的每个所述单色大像素进行插值处理以获取除该单颜色以外的另外两种颜色的像素值并输出以得到具有所述第一分辨率的第二目标图像。
  10. 根据权利要求9所述的图像采集方法，其特征在于，所述处理所述全色原始图像，将每个所述子单元的所有像素作为全色大像素，并输出所述全色大像素的像素值以得到全色中间图像，包括：
    合并每个所述子单元中的所有所述像素的像素值以得到所述全色大像素的像素值；及
    根据多个所述全色大像素的像素值形成所述全色中间图像。
  11. 根据权利要求7所述的图像采集方法，其特征在于，所述处理所述彩色中间图像和/或所述全色中间图像以获取目标图像，包括：
    插补处理所述彩色中间图像以得到具有第二分辨率的彩色插补图像，所述彩色插补图像中对应的所述子单元呈拜耳阵列排布，所述第二分辨率大于所述第一分辨率；及
    对所述彩色插补图像中的所有单颜色像素进行插值处理以获取除该单颜色以外的另外两种颜色的像素值并输出以得到具有所述第二分辨率的第三目标图像。
  12. 根据权利要求7所述的图像采集方法，其特征在于，所述处理所述全色原始图像以得到全色中间图像，包括：
    插补处理所述全色原始图像，获取每个所述子单元中的所有像素的像素值以得到具有第二分辨率的全色中间图像；
    所述处理所述彩色中间图像和/或所述全色中间图像以获取目标图像，包括：
    插补处理所述彩色中间图像以得到具有第二分辨率的彩色插补图像，所述彩色插补图像中对应的所述子单元呈拜耳阵列排布，所述第二分辨率大于所述第一分辨率；
    分离所述彩色插补图像的色彩及亮度以得到具有所述第二分辨率的色亮分离图像；
    融合所述全色插补图像的亮度及所述色亮分离图像的亮度以得到具有所述第二分辨率的亮度修正彩色图像；及
    对所述亮度修正彩色图像中的所有单颜色像素进行插值处理以获取除该单颜色以外的另外两种颜色的像素值并输出以得到具有所述第二分辨率的第四目标图像。
  13. 根据权利要求8-12任意一项所述的图像采集方法，其特征在于，所述图像传感器应用于移动终端或摄像头组件，在所述移动终端或所述摄像头组件处于不同模式时，不同的所述模式对应不同的所述目标图像。
  14. 根据权利要求13所述的图像采集方法，其特征在于，
    在所述模式为预览模式时，所述目标图像为所述第一目标图像或所述第二目标图像；
    在所述模式为成像模式时，所述目标图像为第二目标图像、所述第三目标图像或所述第四目标图像。
  15. 根据权利要求13所述的图像采集方法，其特征在于，
    在所述模式既为预览模式又为低功耗模式时，所述目标图像为所述第一目标图像；
    在所述模式既为所述预览模式又为非低功耗模式时，所述目标图像为所述第二目标图像；
    在所述模式既为成像模式又为所述低功耗模式时，所述目标图像为所述第二目标图像或所述第三目标图像；
    在所述模式既为所述成像模式又为所述非低功耗模式时，所述目标图像为所述第四目标图像。
  16. 根据权利要求8-12任意一项所述的图像采集方法，其特征在于，所述图像采集方法，还包括：
    获取环境亮度；
    在所述环境亮度大于亮度阈值时，所述目标图像为所述第一目标图像或所述第三目标图像；
    在所述环境亮度小于所述亮度阈值时，所述目标图像为所述第二目标图像或所述第四目标图像。
  17. 一种摄像头组件，其特征在于，包括：
    图像传感器，所述图像传感器包括二维像素阵列，所述二维像素阵列包括多个全色像素和多个彩色像素，所述二维像素阵列包括最小重复单元，每个所述最小重复单元包含多个子单元，每个所述子单元包括多个单颜色像素及多个全色像素，所述图像传感器用于曝光以获取全色原始图像和彩色原始图像； 及
    处理芯片，所述处理芯片用于：
    处理所述彩色原始图像，以将每个所述子单元的所有像素作为与该子单元中单颜色对应的单色大像素，并输出所述单色大像素的像素值以得到彩色中间图像；
    处理所述全色原始图像以得到全色中间图像；及
    处理所述彩色中间图像和/或所述全色中间图像以获取目标图像。
  18. 根据权利要求17所述的摄像头组件，其特征在于，所述图像传感器中的所有所述全色像素和所有所述彩色像素同时曝光；
    所述图像传感器输出所有所述全色像素的像素值以获取所述全色原始图像，并输出所有所述彩色像素的像素值以获取所述彩色原始图像。
  19. 根据权利要求17所述的摄像头组件，其特征在于，所述图像传感器中的所有所述全色像素和所有所述彩色像素分时曝光；
    所述图像传感器输出所有所述全色像素的像素值以获取所述全色原始图像，并输出所有所述彩色像素的像素值以获取所述彩色原始图像。
  20. 根据权利要求17所述的摄像头组件，其特征在于，在所述最小重复单元中，所述全色像素设置在第一对角线方向，所述彩色像素设置在第二对角线方向，所述第一对角线方向与所述第二对角线方向不同；
    所述图像传感器用第一曝光信号控制所述第一对角线方向相邻的至少两个所述全色像素以第一曝光时间曝光，并用第二曝光信号控制所述第二对角线方向相邻的至少两个所述彩色像素以第二曝光时间曝光。
  21. 根据权利要求17所述的摄像头组件，其特征在于，所述图像传感器用第一曝光信号控制第2n-1行和第2n行的所述全色像素的第一曝光时间，并用第二曝光信号控制第2n-1行和第2n行的所述彩色像素的第二曝光时间；其中，n为大于等于1的自然数。
  22. 根据权利要求20或21所述的摄像头组件，其特征在于，所述处理芯片还用于获取环境亮度；
    在所述环境亮度大于亮度阈值时，所述第一曝光时间小于所述第二曝光时间。
  23. 根据权利要求22所述的摄像头组件，其特征在于，所述第一曝光时间与所述第二曝光时间的比例为1：2，1：3或1：4中的一种。
  24. 根据权利要求20所述的摄像头组件，其特征在于，所述图像传感器还包括：
    第一曝光控制线，与所述第一对角线方向相邻的至少两个所述全色像素中的曝光控制电路的控制端电连接；及
    第二曝光控制线，与所述第二对角线方向相邻的至少两个所述彩色像素中的曝光控制电路的控制端电连接；
    其中，所述第一曝光信号经由所述第一曝光控制线传输，所述第二曝光信号经由所述第二曝光控制线传输。
  25. 根据权利要求24所述的摄像头组件，其特征在于，
    所述第一曝光控制线呈“W”型，与相邻两行的所述全色像素中的曝光控制电路的控制端电连接；
    所述第二曝光控制线呈“W”型，与相邻两行的所述彩色像素中的曝光控制电路的控制端电连接。
  26. 根据权利要求24或25所述的摄像头组件，其特征在于，每个所述像素还包括光电转换元件，其中，所述曝光控制电路与所述光电转换元件电连接，所述曝光控制电路用于转移所述光电转换元件经光照后积累的电势。
  27. 根据权利要求26所述的摄像头组件，其特征在于，所述曝光控制电路为转移晶体管，所述曝光控制电路的控制端为所述转移晶体管的栅极。
  28. 根据权利要求17所述的摄像头组件，其特征在于，所述最小重复单元为4行4列16个像素，排布方式为：
    Figure PCTCN2019104974-appb-100001
    Figure PCTCN2019104974-appb-100002
    其中，W表示所述全色像素；
    A表示所述多个所述彩色像素中的第一颜色像素；
    B表示所述多个所述彩色像素中的第二颜色像素；
    C表示所述多个所述彩色像素中的第三颜色像素。
  29. 根据权利要求17所述的摄像头组件，其特征在于，所述最小重复单元为4行4列16个像素，排布方式为：
    Figure PCTCN2019104974-appb-100003
    其中，W表示所述全色像素；
    A表示所述多个所述彩色像素中的第一颜色像素；
    B表示所述多个所述彩色像素中的第二颜色像素；
    C表示所述多个所述彩色像素中的第三颜色像素。
  30. 根据权利要求17所述的摄像头组件，其特征在于，所述最小重复单元为6行6列36个像素，排布方式为：
    Figure PCTCN2019104974-appb-100004
    其中，W表示所述全色像素；
    A表示所述多个所述彩色像素中的第一颜色像素；
    B表示所述多个所述彩色像素中的第二颜色像素；
    C表示所述多个所述彩色像素中的第三颜色像素。
  31. 根据权利要求17所述的摄像头组件，其特征在于，所述最小重复单元为6行6列36个像素，排布方式为：
    Figure PCTCN2019104974-appb-100005
    其中，W表示所述全色像素；
    A表示所述多个所述彩色像素中的第一颜色像素；
    B表示所述多个所述彩色像素中的第二颜色像素；
    C表示所述多个所述彩色像素中的第三颜色像素。
  32. 根据权利要求17所述的摄像头组件，其特征在于，所述最小重复单元为8行8列64个像素，排布方式为：
    Figure PCTCN2019104974-appb-100006
    Figure PCTCN2019104974-appb-100007
    其中，W表示所述全色像素；
    A表示所述多个所述彩色像素中的第一颜色像素；
    B表示所述多个所述彩色像素中的第二颜色像素；
    C表示所述多个所述彩色像素中的第三颜色像素。
  33. 根据权利要求17所述的摄像头组件，其特征在于，所述最小重复单元为8行8列64个像素，排布方式为：
    Figure PCTCN2019104974-appb-100008
    其中，W表示所述全色像素；
    A表示所述多个所述彩色像素中的第一颜色像素；
    B表示所述多个所述彩色像素中的第二颜色像素；
    C表示所述多个所述彩色像素中的第三颜色像素。
  34. 根据权利要求28-33任意一项所述的摄像头组件，其特征在于，
    所述第一颜色像素A为红色像素R；
    所述第二颜色像素B为绿色像素G；
    所述第三颜色像素C为蓝色像素Bu。
  35. 根据权利要求28-33任意一项所述的摄像头组件，其特征在于，
    所述第一颜色像素A为红色像素R；
    所述第二颜色像素B为黄色像素Y；
    所述第三颜色像素C为蓝色像素Bu。
  36. 根据权利要求28-33任意一项所述的摄像头组件，其特征在于，
    所述第一颜色像素A为品红色像素M；
    所述第二颜色像素B为青色像素Cy；
    所述第三颜色像素C为黄色像素Y。
  37. 根据权利要求17、28-33任意一项所述的摄像头组件，其特征在于，所述全色像素的响应波段为可见光波段。
  38. 根据权利要求17、28-33任意一项所述的摄像头组件，其特征在于，所述全色像素的响应波段为可见光波段和近红外波段，与所述图像传感器中的光电转换元件的响应波段相匹配。
  39. 根据权利要求17所述的摄像头组件，其特征在于，所述处理芯片还用于：
    合并每个所述子单元中的所有所述像素的像素值以得到所述单色大像素的像素值；及
    根据多个所述单色大像素的像素值形成所述彩色中间图像，所述彩色中间图像具有第一分辨率。
  40. 根据权利要求39所述的摄像头组件，其特征在于，所述处理芯片还用于：
    对所述彩色中间图像中的每个所述单色大像素进行插值处理以获取除该单颜色以外的另外两种颜色的像素值并输出以得到具有所述第一分辨率的第一目标图像。
  41. 根据权利要求39所述的摄像头组件，其特征在于，所述处理芯片还用于：
    处理所述全色原始图像，将每个所述子单元的所有像素作为全色大像素，并输出所述全色大像素的像素值以得到全色中间图像，所述全色中间图像具有所述第一分辨率；
    分离所述彩色中间图像的色彩及亮度以得到具有所述第一分辨率的色亮分离图像；
    融合所述全色中间图像的亮度及所述色亮分离图像的亮度以得到具有所述第一分辨率的亮度修正 彩色图像；及
    对所述亮度修正彩色图像中的每个所述单色大像素进行插值处理以获取除该单颜色以外的另外两种颜色的像素值并输出以得到具有所述第一分辨率的第二目标图像。
  42. 根据权利要求41所述的摄像头组件，其特征在于，所述处理芯片还用于：
    合并每个所述子单元中的所有所述像素的像素值以得到所述全色大像素的像素值；及
    根据多个所述全色大像素的像素值形成所述全色中间图像。
  43. 根据权利要求39所述的摄像头组件，其特征在于，所述处理芯片还用于：
    插补处理所述彩色中间图像以得到具有第二分辨率的彩色插补图像，所述彩色插补图像中对应的所述子单元呈拜耳阵列排布，所述第二分辨率大于所述第一分辨率；及
    对所述彩色插补图像中的所有单颜色像素进行插值处理以获取除该单颜色以外的另外两种颜色的像素值并输出以得到具有所述第二分辨率的第三目标图像。
  44. 根据权利要求39所述的摄像头组件，其特征在于，所述处理芯片还用于：
    插补处理所述全色原始图像，获取每个所述子单元中的所有像素的像素值以得到具有第二分辨率的全色中间图像；
    插补处理所述彩色中间图像以得到具有第二分辨率的彩色插补图像，所述彩色插补图像中对应的所述子单元呈拜耳阵列排布，所述第二分辨率大于所述第一分辨率；
    分离所述彩色插补图像的色彩及亮度以得到具有所述第二分辨率的色亮分离图像；
    融合所述全色插补图像的亮度及所述色亮分离图像的亮度以得到具有所述第二分辨率的亮度修正彩色图像；及
    对所述亮度修正彩色图像中的所有单颜色像素进行插值处理以获取除该单颜色以外的另外两种颜色的像素值并输出以得到具有所述第二分辨率的第四目标图像。
  45. 根据权利要求40-44任意一项所述的摄像头组件，其特征在于，所述图像传感器应用于所述摄像头组件，在所述摄像头组件处于不同模式时，不同的所述模式对应不同的所述目标图像。
  46. 根据权利要求45所述的摄像头组件，其特征在于，
    在所述模式为预览模式时，所述目标图像为所述第一目标图像或所述第二目标图像；
    在所述模式为成像模式时，所述目标图像为第二目标图像、所述第三目标图像或所述第四目标图像。
  47. 根据权利要求45所述的摄像头组件，其特征在于，
    在所述模式既为预览模式又为低功耗模式时，所述目标图像为所述第一目标图像；
    在所述模式既为所述预览模式又为非低功耗模式时，所述目标图像为所述第二目标图像；
    在所述模式既为成像模式又为所述低功耗模式时，所述目标图像为所述第二目标图像或所述第三目标图像；
    在所述模式既为所述成像模式又为所述非低功耗模式时，所述目标图像为所述第四目标图像。
  48. 根据权利要求40-44任意一项所述的摄像头组件，其特征在于，所述处理芯片还用于获取环境亮度；
    在所述环境亮度大于亮度阈值时，所述目标图像为所述第一目标图像或所述第三目标图像；
    在所述环境亮度小于所述亮度阈值时，所述目标图像为所述第二目标图像或所述第四目标图像。
  49. 一种移动终端，其特征在于，包括：
    图像传感器，所述图像传感器包括二维像素阵列，所述二维像素阵列包括多个全色像素和多个彩色像素，所述二维像素阵列包括最小重复单元，每个所述最小重复单元包含多个子单元，每个所述子单元包括多个单颜色像素及多个全色像素，所述图像传感器用于曝光以获取全色原始图像和彩色原始图像；及
    处理器，所述处理器用于：
    处理所述彩色原始图像，以将每个所述子单元的所有像素作为与该子单元中单颜色对应的单色大像素，并输出所述单色大像素的像素值以得到彩色中间图像；
    处理所述全色原始图像以得到全色中间图像；及
    处理所述彩色中间图像和/或所述全色中间图像以获取目标图像。
  50. 根据权利要求49所述的移动终端，其特征在于，所述图像传感器中的所有所述全色像素和所有 所述彩色像素同时曝光；
    所述图像传感器输出所有所述全色像素的像素值以获取所述全色原始图像，并输出所有所述彩色像素的像素值以获取所述彩色原始图像。
  51. 根据权利要求49所述的移动终端，其特征在于，所述图像传感器中的所有所述全色像素和所有所述彩色像素分时曝光；
    所述图像传感器输出所有所述全色像素的像素值以获取所述全色原始图像，并输出所有所述彩色像素的像素值以获取所述彩色原始图像。
  52. 根据权利要求49所述的移动终端，其特征在于，在所述最小重复单元中，所述全色像素设置在第一对角线方向，所述彩色像素设置在第二对角线方向，所述第一对角线方向与所述第二对角线方向不同；
    所述图像传感器用用第一曝光信号控制所述第一对角线方向相邻的至少两个所述全色像素以第一曝光时间曝光，并用第二曝光信号控制所述第二对角线方向相邻的至少两个所述彩色像素以第二曝光时间曝光。
  53. 根据权利要求49所述的移动终端，其特征在于，所述图像传感器用第一曝光信号控制第2n-1行和第2n行的所述全色像素的第一曝光时间，并用第二曝光信号控制第2n-1行和第2n行的所述彩色像素的第二曝光时间；其中，n为大于等于1的自然数。
  54. 根据权利要求52或53所述的移动终端，其特征在于，所述处理器还用于获取环境亮度；
    在所述环境亮度大于亮度阈值时，所述第一曝光时间小于所述第二曝光时间。
  55. 根据权利要求54所述的移动终端，其特征在于，所述第一曝光时间与所述第二曝光时间的比例为1：2，1：3或1：4中的一种。
  56. 根据权利要求52所述的移动终端，其特征在于，所述图像传感器还包括：
    第一曝光控制线，与所述第一对角线方向相邻的至少两个所述全色像素中的曝光控制电路的控制端电连接；及
    第二曝光控制线，与所述第二对角线方向相邻的至少两个所述彩色像素中的曝光控制电路的控制端电连接；
    其中，所述第一曝光信号经由所述第一曝光控制线传输，所述第二曝光信号经由所述第二曝光控制线传输。
  57. 根据权利要求56所述的移动终端，其特征在于，
    所述第一曝光控制线呈“W”型，与相邻两行的所述全色像素中的曝光控制电路的控制端电连接；
    所述第二曝光控制线呈“W”型，与相邻两行的所述彩色像素中的曝光控制电路的控制端电连接。
  58. 根据权利要求56或57所述的移动终端，其特征在于，每个所述像素还包括光电转换元件，其中，所述曝光控制电路与所述光电转换元件电连接，所述曝光控制电路用于转移所述光电转换元件经光照后积累的电势。
  59. 根据权利要求58所述的移动终端，其特征在于，所述曝光控制电路为转移晶体管，所述曝光控制电路的控制端为所述转移晶体管的栅极。
  60. 根据权利要求49所述的移动终端，其特征在于，所述最小重复单元为4行4列16个像素，排布方式为：
    Figure PCTCN2019104974-appb-100009
    其中，W表示所述全色像素；
    A表示所述多个所述彩色像素中的第一颜色像素；
    B表示所述多个所述彩色像素中的第二颜色像素；
    C表示所述多个所述彩色像素中的第三颜色像素。
  61. 根据权利要求49所述的移动终端，其特征在于，所述最小重复单元为4行4列16个像素，排布方式为：
    Figure PCTCN2019104974-appb-100010
    其中，W表示所述全色像素；
    A表示所述多个所述彩色像素中的第一颜色像素；
    B表示所述多个所述彩色像素中的第二颜色像素；
    C表示所述多个所述彩色像素中的第三颜色像素。
  62. 根据权利要求49所述的移动终端，其特征在于，所述最小重复单元为6行6列36个像素，排布方式为：
    Figure PCTCN2019104974-appb-100011
    其中，W表示所述全色像素；
    A表示所述多个所述彩色像素中的第一颜色像素；
    B表示所述多个所述彩色像素中的第二颜色像素；
    C表示所述多个所述彩色像素中的第三颜色像素。
  63. 根据权利要求49所述的移动终端，其特征在于，所述最小重复单元为6行6列36个像素，排布方式为：
    Figure PCTCN2019104974-appb-100012
    其中，W表示所述全色像素；
    A表示所述多个所述彩色像素中的第一颜色像素；
    B表示所述多个所述彩色像素中的第二颜色像素；
    C表示所述多个所述彩色像素中的第三颜色像素。
  64. 根据权利要求49所述的移动终端，其特征在于，所述最小重复单元为8行8列64个像素，排布方式为：
    Figure PCTCN2019104974-appb-100013
    其中，W表示所述全色像素；
    A表示所述多个所述彩色像素中的第一颜色像素；
    B表示所述多个所述彩色像素中的第二颜色像素；
    C表示所述多个所述彩色像素中的第三颜色像素。
  65. 根据权利要求49所述的移动终端，其特征在于，所述最小重复单元为8行8列64个像素，排 布方式为：
    Figure PCTCN2019104974-appb-100014
    其中，W表示所述全色像素；
    A表示所述多个所述彩色像素中的第一颜色像素；
    B表示所述多个所述彩色像素中的第二颜色像素；
    C表示所述多个所述彩色像素中的第三颜色像素。
  66. 根据权利要求60-65任意一项所述的移动终端，其特征在于，
    所述第一颜色像素A为红色像素R；
    所述第二颜色像素B为绿色像素G；
    所述第三颜色像素C为蓝色像素Bu。
  67. 根据权利要求60-65任意一项所述的移动终端，其特征在于，
    所述第一颜色像素A为红色像素R；
    所述第二颜色像素B为黄色像素Y；
    所述第三颜色像素C为蓝色像素Bu。
  68. 根据权利要求60-65任意一项所述的移动终端，其特征在于，
    所述第一颜色像素A为品红色像素M；
    所述第二颜色像素B为青色像素Cy；
    所述第三颜色像素C为黄色像素Y。
  69. 根据权利要求49、60-65任意一项所述的移动终端，其特征在于，所述全色像素的响应波段为可见光波段。
  70. 根据权利要求49、60-65任意一项所述的移动终端，其特征在于，所述全色像素的响应波段为可见光波段和近红外波段，与所述图像传感器中的光电转换元件的响应波段相匹配。
  71. 根据权利要求49所述的移动终端，其特征在于，所述处理器还用于：
    合并每个所述子单元中的所有所述像素的像素值以得到所述单色大像素的像素值；及
    根据多个所述单色大像素的像素值形成所述彩色中间图像，所述彩色中间图像具有第一分辨率。
  72. 根据权利要求71所述的移动终端，其特征在于，所述处理器还用于：
    对所述彩色中间图像中的每个所述单色大像素进行插值处理以获取除该单颜色以外的另外两种颜色的像素值并输出以得到具有所述第一分辨率的第一目标图像。
  73. 根据权利要求71所述的移动终端，其特征在于，所述处理器还用于：
    处理所述全色原始图像，将每个所述子单元的所有像素作为全色大像素，并输出所述全色大像素的像素值以得到全色中间图像，所述全色中间图像具有所述第一分辨率；
    分离所述彩色中间图像的色彩及亮度以得到具有所述第一分辨率的色亮分离图像；
    融合所述全色中间图像的亮度及所述色亮分离图像的亮度以得到具有所述第一分辨率的亮度修正彩色图像；及
    对所述亮度修正彩色图像中的每个所述单色大像素进行插值处理以获取除该单颜色以外的另外两种颜色的像素值并输出以得到具有所述第一分辨率的第二目标图像。
  74. 根据权利要求73所述的移动终端，其特征在于，所述处理器还用于：
    合并每个所述子单元中的所有所述像素的像素值以得到所述全色大像素的像素值；及
    根据多个所述全色大像素的像素值形成所述全色中间图像。
  75. 根据权利要求71所述的移动终端，其特征在于，所述处理器还用于：
    插补处理所述彩色中间图像以得到具有第二分辨率的彩色插补图像，所述彩色插补图像中对应的所述子单元呈拜耳阵列排布，所述第二分辨率大于所述第一分辨率；及
    对所述彩色插补图像中的所有单颜色像素进行插值处理以获取除该单颜色以外的另外两种颜色的像素值并输出以得到具有所述第二分辨率的第三目标图像。
  76. 根据权利要求71所述的移动终端，其特征在于，所述处理器还用于：
    插补处理所述全色原始图像，获取每个所述子单元中的所有像素的像素值以得到具有第二分辨率的全色中间图像；
    插补处理所述彩色中间图像以得到具有第二分辨率的彩色插补图像，所述彩色插补图像中对应的所述子单元呈拜耳阵列排布，所述第二分辨率大于所述第一分辨率；
    分离所述彩色插补图像的色彩及亮度以得到具有所述第二分辨率的色亮分离图像；
    融合所述全色插补图像的亮度及所述色亮分离图像的亮度以得到具有所述第二分辨率的亮度修正彩色图像；及
    对所述亮度修正彩色图像中的所有单颜色像素进行插值处理以获取除该单颜色以外的另外两种颜色的像素值并输出以得到具有所述第二分辨率的第四目标图像。
  77. 根据权利要求72-75任意一项所述的移动终端，其特征在于，所述图像传感器应用于所述移动终端，在所述移动终端处于不同模式时，不同的所述模式对应不同的所述目标图像。
  78. 根据权利要求77所述的移动终端，其特征在于，
    在所述模式为预览模式时，所述目标图像为所述第一目标图像或所述第二目标图像；
    在所述模式为成像模式时，所述目标图像为第二目标图像、所述第三目标图像或所述第四目标图像。
  79. 根据权利要求77所述的移动终端，其特征在于，
    在所述模式既为预览模式又为低功耗模式时，所述目标图像为所述第一目标图像；
    在所述模式既为所述预览模式又为非低功耗模式时，所述目标图像为所述第二目标图像；
    在所述模式既为成像模式又为所述低功耗模式时，所述目标图像为所述第二目标图像或所述第三目标图像；
    在所述模式既为所述成像模式又为所述非低功耗模式时，所述目标图像为所述第四目标图像。
  80. 根据权利要求72-75任意一项所述的移动终端，其特征在于，所述处理芯片还用于获取环境亮度；
    在所述环境亮度大于亮度阈值时，所述目标图像为所述第一目标图像或所述第三目标图像；
    在所述环境亮度小于所述亮度阈值时，所述目标图像为所述第二目标图像或所述第四目标图像。
PCT/CN2019/104974 2019-09-09 2019-09-09 图像采集方法、摄像头组件及移动终端 WO2021046691A1 (zh)

Priority Applications (6)

Application Number Priority Date Filing Date Title
KR1020227009653A KR20220051240A (ko) 2019-09-09 2019-09-09 이미지 캡쳐 방법, 카메라 어셈블리 및 모바일 터미널
EP19944661.8A EP4020971A4 (en) 2019-09-09 2019-09-09 IMAGE COLLECTION METHOD, CAMERA ASSEMBLY AND MOBILE TERMINAL
PCT/CN2019/104974 WO2021046691A1 (zh) 2019-09-09 2019-09-09 图像采集方法、摄像头组件及移动终端
JP2022515700A JP7298020B2 (ja) 2019-09-09 2019-09-09 画像キャプチャ方法、カメラアセンブリ及び移動端末
CN201980097882.9A CN114073068B (zh) 2019-09-09 2019-09-09 图像采集方法、摄像头组件及移动终端
US17/584,813 US20220150450A1 (en) 2019-09-09 2022-01-26 Image capturing method, camera assembly, and mobile terminal

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
PCT/CN2019/104974 WO2021046691A1 (zh) 2019-09-09 2019-09-09 图像采集方法、摄像头组件及移动终端

Related Child Applications (1)

Application Number Title Priority Date Filing Date
US17/584,813 Continuation US20220150450A1 (en) 2019-09-09 2022-01-26 Image capturing method, camera assembly, and mobile terminal

Publications (1)

Publication Number Publication Date
WO2021046691A1 true WO2021046691A1 (zh) 2021-03-18

Family

ID=74865928

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2019/104974 WO2021046691A1 (zh) 2019-09-09 2019-09-09 图像采集方法、摄像头组件及移动终端

Country Status (6)

Country Link
US (1) US20220150450A1 (zh)
EP (1) EP4020971A4 (zh)
JP (1) JP7298020B2 (zh)
KR (1) KR20220051240A (zh)
CN (1) CN114073068B (zh)
WO (1) WO2021046691A1 (zh)

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN113676636A (zh) * 2021-08-16 2021-11-19 Oppo广东移动通信有限公司 高动态范围图像的生成方法、装置、电子设备和存储介质

Citations (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2008063351A1 (en) * 2006-11-10 2008-05-29 Eastman Kodak Company Noise reduction of panchromatic and color image
CN101233762A (zh) * 2005-07-28 2008-07-30 伊斯曼柯达公司 具有改善的光敏感度的图像传感器
CN101248659A (zh) * 2005-08-23 2008-08-20 伊斯曼柯达公司 在变化照明条件下捕获图像
CN102369721A (zh) * 2009-03-10 2012-03-07 美商豪威科技股份有限公司 具有合成全色图像的彩色滤光器阵列(cfa)图像
CN102461174A (zh) * 2009-06-05 2012-05-16 全视科技有限公司 具有四个通道的滤色器阵列图案
CN103930923A (zh) * 2011-12-02 2014-07-16 诺基亚公司 用于捕获图像的方法、装置和计算机程序产品
US20140267351A1 (en) * 2013-03-14 2014-09-18 Microsoft Corporation Monochromatic edge geometry reconstruction through achromatic guidance

Family Cites Families (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US8264576B2 (en) * 2007-03-05 2012-09-11 DigitalOptics Corporation Europe Limited RGBW sensor array
US8068153B2 (en) * 2009-03-27 2011-11-29 Omnivision Technologies, Inc. Producing full-color image using CFA image
US9699429B2 (en) * 2012-03-27 2017-07-04 Sony Corporation Image processing apparatus, imaging device, image processing method, and program for reducing noise or false colors in an image
US20140063300A1 (en) * 2012-09-06 2014-03-06 Aptina Imaging Corporation High dynamic range imaging systems having clear filter pixel arrays
JP2017118191A (ja) * 2015-12-21 2017-06-29 ソニー株式会社 撮像素子及びその駆動方法、並びに撮像装置
JP6953297B2 (ja) * 2017-12-08 2021-10-27 キヤノン株式会社 撮像装置及び撮像システム

Patent Citations (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN101233762A (zh) * 2005-07-28 2008-07-30 伊斯曼柯达公司 具有改善的光敏感度的图像传感器
CN101248659A (zh) * 2005-08-23 2008-08-20 伊斯曼柯达公司 在变化照明条件下捕获图像
WO2008063351A1 (en) * 2006-11-10 2008-05-29 Eastman Kodak Company Noise reduction of panchromatic and color image
CN102369721A (zh) * 2009-03-10 2012-03-07 美商豪威科技股份有限公司 具有合成全色图像的彩色滤光器阵列(cfa)图像
CN102461174A (zh) * 2009-06-05 2012-05-16 全视科技有限公司 具有四个通道的滤色器阵列图案
CN103930923A (zh) * 2011-12-02 2014-07-16 诺基亚公司 用于捕获图像的方法、装置和计算机程序产品
US20140267351A1 (en) * 2013-03-14 2014-09-18 Microsoft Corporation Monochromatic edge geometry reconstruction through achromatic guidance

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN113676636A (zh) * 2021-08-16 2021-11-19 Oppo广东移动通信有限公司 高动态范围图像的生成方法、装置、电子设备和存储介质
CN113676636B (zh) * 2021-08-16 2023-05-05 Oppo广东移动通信有限公司 高动态范围图像的生成方法、装置、电子设备和存储介质

Also Published As

Publication number Publication date
JP7298020B2 (ja) 2023-06-26
CN114073068B (zh) 2023-11-03
KR20220051240A (ko) 2022-04-26
CN114073068A (zh) 2022-02-18
JP2022547221A (ja) 2022-11-10
US20220150450A1 (en) 2022-05-12
EP4020971A1 (en) 2022-06-29
EP4020971A4 (en) 2022-09-07

Similar Documents

Publication Publication Date Title
CN110649056B (zh) 图像传感器、摄像头组件及移动终端
WO2021179806A1 (zh) 图像获取方法、成像装置、电子设备及可读存储介质
CN110740272B (zh) 图像采集方法、摄像头组件及移动终端
WO2021063162A1 (zh) 图像传感器、摄像头组件及移动终端
WO2021223590A1 (zh) 图像传感器、控制方法、摄像头组件和移动终端
WO2021196553A1 (zh) 高动态范围图像处理系统及方法、电子设备和可读存储介质
CN111314592B (zh) 图像处理方法、摄像头组件及移动终端
WO2021103818A1 (zh) 图像传感器、控制方法、摄像头组件及移动终端
WO2021179805A1 (zh) 图像传感器、摄像头组件、移动终端及图像获取方法
CN110784634B (zh) 图像传感器、控制方法、摄像头组件及移动终端
CN111447380B (zh) 控制方法、摄像头组件和移动终端
US20220336508A1 (en) Image sensor, camera assembly and mobile terminal
WO2021159944A1 (zh) 图像传感器、摄像头组件及移动终端
WO2022007215A1 (zh) 图像获取方法、摄像头组件及移动终端
WO2021062661A1 (zh) 图像传感器、摄像头组件及移动终端
US20220150450A1 (en) Image capturing method, camera assembly, and mobile terminal
US20220139974A1 (en) Image sensor, camera assembly, and mobile terminal
CN111031297B (zh) 图像传感器、控制方法、摄像头组件和移动终端
US20220279108A1 (en) Image sensor and mobile terminal
CN112235485B (zh) 图像传感器、图像处理方法、成像装置、终端及可读存储介质
CN114424517B (zh) 图像传感器、控制方法、摄像头组件及移动终端
WO2021046690A1 (zh) 图像传感器、摄像头模组、移动终端及图像采集方法

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 19944661

Country of ref document: EP

Kind code of ref document: A1

ENP Entry into the national phase

Ref document number: 2022515700

Country of ref document: JP

Kind code of ref document: A

NENP Non-entry into the national phase

Ref country code: DE

ENP Entry into the national phase

Ref document number: 20227009653

Country of ref document: KR

Kind code of ref document: A

ENP Entry into the national phase

Ref document number: 2019944661

Country of ref document: EP

Effective date: 20220324